Kids having freedom is fine. When I was younger, me and the neighborhood kids were always out and about playing but we never set camp in someone else's yard. We played in OUR yards. We would cut through (unfenced) yards but that was about it. I can't even imagine climbing over someone's fence.
Plus the idea of kids that young being unsupervised is unsettling to me. My nephews are 6 and 5 years old... they're smart kids but at the end of the day they're still only 6 and 5, and not immune to danger. I know a little girl (who was about that age) who was playing with some neighborhood kids one night and she ended up wandering onto the freeway that their neighborhood is adjacent to. No, that story does not have a happen ending. And the parents had no idea where their kids were at the time. Why not??!