Wow I guess I really don't say it very much!
With my SO, we say it before hanging up on the phone, but we don't really talk on the phone very much. Otherwise, it is reserved for pretty special moments. Sometimes there will be long periods where neither of us say it, but I kind of like that because when it is said, it really has meaning. We definitely don't throw it around casually.
With my mom I usually say it before hanging up on the phone, and very occasionally in person. My dad and I don't say it much, for whatever reason, though we have a great relationship and are very open with each other.
I say I love you to my sister a lot, probably the most really. Every phone convo and email, message etc, and sometimes in person.
Pretty much never with friends, except occasionally in messages, and recently I busted it out with a really good friend after we had a really intense conversation on the phone, after having a rough few months, and she was crying and we were pretty emotional, so it just seemed like the right time to say it.