I wouldn't think of taking any of my dogs into a grocery store, but I've taken them into most other stores out here in Montana without a problem. Once, when walking into Staples with one of my Min Pins, (I was holding him,) an employee came over to me and said dogs weren't allowed, then I replied..."Oh, I needed some ink cartridges, but they're not worth leaving my dog in the car to die from heat (over 100 degrees that day,) so I'll buy them online," then turned to leave. As I was walking out, a manager came running over and said he'd be glad to help me and my dog, and he was glad to see I wasn't leaving my dog in the car.
I would not, however, expect to just walk into a department store with my Rottie on a leash. My min pin in my arms isn't going to intimidate other shoppers, but I've seen people at Petsmart intimidated by my Rottie, and they're supposedly "dog people," and he's the most well socialized, perfect gentleman of a dog that I've ever seen, but I prefer not to infringe on other people's rights to shop in peace, and if they see a big dog, and they're scared, I'm definitely infringing. I DO, take him to Petsmart, and any other store that specifically allows dogs, I just won't ever push the envelope with him.
However, when my Rottie was a pup, my wife and I did take him everywhere, and that was in the Philly area where people aren't as tolerant about dogs as they are out here. We were never asked to leave a store, and more often than not, the cashiers wouldn't LET us leave the store, they were too busy petting the cute little puppy.
It all comes down to the type of store and the type of people in the area. I love all of my dogs, but that love doesn't give me the right to infringe on anyone else's rights to enjoy a day shopping. If I take my big dog to Petsmart and people don't like him there, that's too bad, the sign on the front of the store says that dogs are welcome, so they know ahead of time. If I take him into Wal-Mart, and a kid who has severe anxiety over big dogs sees him, freaks out, and freaks his mom out, then it's not fair to them. They thought they were just out shopping, and they had no idea they'd be walking the same aisles as a Rottweiler, and if they hate Rotties, that's their right.