My reasoning for trying when we did was...
Averages are only OK if you are the average or below it!!
I wanted kids before I was 35. If we struggled to get pregnant the NHS won't see you before you've been trying for 12 months. So that's a year. Then add on how long it can take to work out what's going on... You could be looking at 2, 3, maybe more years to conceive.
There IS NEVER A RIGHT TIME. Whose to say to you won't be in a worse off place in a year or 2?!
Your fertility tends to decline as you age, I'd been diagnosed with a cyst....
Ultimately it took us 10 months of sort of trying.
My advice would be if you want it, things are ok/you'd survive (no major devastating issues) then do it!!