Maybe just tell him if anyone calls or texts him and asks to come over, to say "I have to check with Allison and see if she feels up to it, I'll let you know."
Thankfully I have ZERO issue with telling people "I really enjoyed our visit, but it's time for me to feed the baby/nap/rest, so you need to leave in a few minutes." But my plan is telling friends who want to visit "I'm not feeling great, but why don't you stop by for an hour just to meet the baby/visit."
I figure you sort of get a free pass to be a little crazy/bitchy right after giving birth.
ETA: Oh...if you haven't...CALL your hospital and ask about visiting policy. My hospital's website says the exact same thing as far as visiting (no one except Dad or Coach in the delivery room, no visitors in Labor and Delivery, and visitors age 14+ in maternity). Yeah...nope. At the hospital tour they told us that you can have anyone/as many people as you want in L&D, visiting hours in maternity are pretty much open-ended, and they'll only kick out children if they appear sick or are misbehaving (being loud, running around, etc.)