I have an architecture degree, and I originally graduated with it in 2006. This January I quit my job and I'm now back in school in pre-med, basically taking all of the math and science that I didn't have from my first degree. I'm 27 right now, which at my university means I'm the age of most of my TAs or older, and well older than the general population.
It was tough, being out of school for so long. It took me a while to get into the swing of things and back into an academic routine. It's also stressful for me because I basically can't afford to get less than an A in any class, so I'm beating the crap out of myself. I'm not working; I quit my job entirely. But I do volunteer several times a week at a local animal shelter working with dogs with behavioral issues, and at a hospital downtown in the ER.
The thing I think that hits home the most is when I was working, I got used to being able to come home and shut off my brain essentially. Even if I was working overtime, which I did a lot, I could drive home, turn on the computer/tv and veg out. Now, if I have school til 6:15 and I don't get home til 7:00, I have homework to do on top of all the rest of my regular household responsibilities. I can't sit down and space out. I forgot how annoying it is to not have any real time for myself because all of it is dedicated to school.
It's definitely worth it if that's something you really want to do, but the older you get and the more responsibilities you have, and the more you get used to having free time to go along with your paycheque, the tougher it can be to readjust. I love being a student, I'm excited about applying to med school next spring, and I'm so glad I took the chance on myself and quit the dead end job, there are just some nights where all I want to do is scream at the top of my lungs and burn all of my books in a fire.
Good luck in your decision, it's a tough one but I think if you do head back you will find it very rewarding.