this may not be a popular idea, but since you live in a house with 5 people who come and go through the day(as well as all their friends I would assume)... I think it may be the best one.
1) if you have your OWN room... not one that you share. Lock him in the room while your gone.
2) if you don't have your own room, then I would suggest getting one of those cat cages. They're like giant ferret cages... with multi-levels. Deck it out really nice with all kinds of toys, hidey-things, beds, etc... when you leave put him in there.
There is no way to control what goes on when your not home, and he's your cat so you really shouldn't expect other people to have to worry about letting him out... and then spending hours looking for him if he does slip past them.
If it was my cat, I wouldn't like the idea of locking him up either. However, I like the idea of coming home to my cat missing or dead even less.