Fear aggressive Pit Bulls

I would actually be pleased if the OP came on saying "my dog growls when you grab his collar" because at least he's warning you before he resorts to biting.

Though since he goes straight to biting, I wonder if he's been punished in the past for growling.

This isn't always the case. Chloe, our HA dog, almost never growls before biting and she has never been punished for ggrowling. Even as a pup, her only warning was hard eyes and a tight mouth, then she'd bite. Some dogs just don't growl.