Funny, I am 33 and have two baby teeth that never fell out so I have caps on them that make them the right size. Apparently doesn't happen often, but I did not have the adult teeth there to push them out. I never got wisdom teeth either. Small favors.
However, deciduous baby teeth in a puppy can really offset the bite and create problems with how the bite comes together. The teeth may not meet properly which will cause the opposing teeth to come down (or up) farther than they should (over time) exposing the roots. As said they may get food trapped which may causes abscesses if it goes unnoticed. I am assuming that the adult teeth have come in?
I would definitely have them removed when he is sedated for his neuter. If he is already altered, then I would have it done anyway. It will probably only take a very few minutes to do. You don't want to encourage dental problems, little dogs have enough of them without any excuse.