Oh nonsense.....he's trying to scare you! My ex said that crap to me. He'd have to prove first that you are unfit, that wont happen. Tell him go ahead and quit his job. He's bluffing. He knows he'd have to live off his mom if he does.
When my ex told me he was going to take my baby, I said ok, but just so you know, I'm dragging every piece of dirt I know about you and your family and in the end, you still wont get her but your name will be trash!
Also, when I left my husband, I was pregnant (he cleared the kitchen table on me), final straw. But because I was pregnant I wasnt allowed to divorce because the lawyer said pregnant people are to emotional and basically unstable in the eyes of the court. He told me dont to let my husband take our child. If I let her go for him for a day, he wouldnt legally have to bring her back. So I had to refuse him to take her until after I had my baby and could go get temporary custody. I dont know how all that works when your not married.