I think it is true that if that it was legal and women just walked around topless, it would fade from being taboo and super crazy sexual. Which would be kind of nice, because they are primarily for feeding babies and it is kind of disturbing when you're not allowed to post breastfeeding photos on FB but are allowed to post all sorts of racy, sexual very boob included photos.
There are lots of different body parts in different cultures considered sexual and made to be hidden, things like neck, ankles, etc. It's very much a cultural thing, once you hide it and declare it a special sexy thing, you get this kind of body part obsessed result we have going.