I have this issue also. When I found out I was pregnant it was a total shock to me, because I never really know when I'm ovulating. Plus I had just done treatment for ovarian cyst (3 mos of Birth Control) so double shocker. I just go with the OB's estimate of age based on size. I figure he does this for a living and knows what he's doing. I am right on time with all developmental milestones so far and AS confirmed the age OB decided.
I would say don't stress yourself about the stuff you can't control. Stay healthy, listen to your body. It will tell you when you are in labor!! Maybe be prepared a little early if you are worried.