You might want to change the words you use to call him. I'd wait till he starts coming toward you on his own accord and then softly and sweetly use a new word he may not have heard before. Maybe the words you use are words that were used on him before and in not such a nice way. Or...maybe he's just plain submissive anyhow. The wagging in his case is likely an appeasement gesture. Give him a little treat when he comes. Make sure to never use punishment based training techniques on this guy. If you need any specifics on positive reinforcement type training methods, feel free to ask. Clicker training would be really effective with him. (if he's not afraid of the clicker. lol...but there are alternatives to that)
Will he engage in a tug game? That's a fun game and can build confidence. If he doesn't pull, just take it in baby steps. I think as you work with him more...teaching him some fun obedience tricks, he'll get more bonded to you, more trusting and more confident. Some other fun things to teach are "give," "take," "leave it" "fetch." Or any other thing you think he would like learning. Using a lot of capturing of behavior helps a lot with shy dogs....like when you're teaching him "down," unless he already knows that one? Just wait for the behavior to occur on it's own, mark it with the clicker or "yesssss" and treat. As he starts offering the behavior more and more,
then add in a cue...not right off in the beginning of a new behavior.
Make all lessons a fun game, gentle, never stern and use lots of teensy treats or his favorite toy or game for rewards. Start rewarding him for eye contact. He may be intimidated if you stare at him or if other people do. You can counter condition him to enjoying eye contact if you tie a high value treat to it. Just wait and see if he'll look at you. When he does, give a gentle cue, like, "watch" and immediately give him a treat/praise. (be sure your praise or any play is done in such a way that it is not over whelming. Is he afraid of other people or things out in his environment? Work with him in his comfort zone and gradually desensatize him to other things and people.
Agility training helps a lot with shy dogs to build confidence. You could set up a few make-shift jumps in your yard or google how to make weave poles or a few other pieces of equipment. Make sure they're sturdy and strong and if you do any boards for him to walk on, start with them right on the ground, not raised...again...baby steps. I don't know how he'd fare at a class. If you think he'd feel comfortable, that might be fun. You know your dog best.
Give him time. He may also never be super confident. This might be his temperament or he may have not had a good beginning. Do you know his history at all?
I guess what I'd do is not do too much in an overt way for now...kind of let him warm up to you more on his terms when you can. He sounds like a sweet dog, but building confidence would be good for him.
Here's something else you might like.
Book Review : Help for Your Shy Dog