Grass will grow. It will be okay. I tore out parts of 6" deep asphalt pad and dug up tons of gravel and road pack that was underneath it. The grass is growing quite happily.
Asphalt pad.
Pad cut and removed, most of the road pack and gravel removed.
Happy grass.
Another angle.
Clearing out just rock beds will be MUCH easier, but will take plenty of physical labor and preferably a sifter to make it go faster.
You have a few options; if there's somewhere on the driveway or the front to put down a plastic sheet, you can clear it up and wheelbarrow it up there and dump it. Then post on craigslist that you have free rock for anybody who wants to come scoop it and load it.
The other option is to post about it and have people come clear it out of your yard themselves, but that's a bit skeevy and I wouldn't do it. There's a guy on our local craigslist who is getting rid of river rock from flower beds around his house and he is inviting people to come dig it up away from the house, and I wouldn't even go dig it up... just a little creepy to show up and start digging up rock away from somebody's house. =P
Another option would be to relocate the beds and landscape your front yard instead - that way you don't really lose the "so pretty!!" of having all the plants and stuff, but you move it to the front of the house where it boosts curb appeal up there. And then you can leave the backyard itself to being a yard. Honestly it will all depend on who buys the house after you, but anybody with dogs or kids will probably appreciate a nice backyard more than pretty landscaping with very little grass.
With the road pack, we dumped it on the driveway in front of the garage and I posted on craigslist that it was free if anybody wanted to come haul it off. One guy came and took a truckload, then came back the next day with a buddy and they loaded up the rest and away they went. He was expanding his driveway in the country and needed it. The large chunks of asphalt we actually were able to just wheelbarrow over to my neighbor's, because they were pouring a new concrete driveway and actually needed large fill to help level it out. Any rocks or chunky stuff we dug up after the one guy took all the road pack I'd dug up also went over there. So I got rid of the whole thing for free, which would have cost me several hundred bucks to haul off, and two people also got stuff for free when it would have cost them a few hundred apiece to get the stuff too. Pretty sweet for everybody involved.
In some places the city will actually help you take down sweet gum trees - I don't know if they're still doing it, but my city was doing it here not that long ago. The pods were getting into the sewer system and causing trouble and damage, so they were paying for people to cut them down. =P
We cut down a lot of limbs off my trees to let more light come through - there were also a lot of dead limbs that needed to come down, and while we were at it I cut them away from where my dog walk sits. The city came by last week and cut the limbs away from the powerlines, so that has brightened it up some too. My front yard has a bigger issue with shade under the big tree but I haven't really addressed it much since that's only an aesthetic annoyance rather than THIS MUD. THE DOGS. ARGH. I cut some stuff down after a big storm knocked some things down to start but I really want to remove more. A lot of it we were able to get just with a pole saw but you can also hire a tree guy to come out. I'll probably have one come out next spring to take down the remainder of what I want down. See what winter kills off first.
Anyway, no doubt it can be done, it just depends on how much work you want to put into it. And, of course, you don't need to do it all at once. You can remove just one bed at a time.