It's ok, yeah, but most people probably choose to take it off and only put it on when they need to.
The main problem with leaving it on all the time is that it could rub your dog's skin raw in the sensitive places it touches, such as under the arms and on the chest. Wearing it for a little while wouldn't be a problem, but wearing it for days on end - particularly sleeping in it - could start irritating his skin.
The other problem is that many dogs can figure out how to chew it off. They can easily reach the straps with their mouth, and if they get bored or uncomfortable they might decide to take care of it. At work I've lost so many harnesses from dogs chewing them off, that I just automatically assume that every dog will chew it off if left unattended.