I too took the 3 hour and passed, but at 35 weeks my baby was estimated at 7 lbs 7 oz. He never mentioned checking glucose again. My doctor just said it didn't really matter since I am scheduled to have a c-section anyway. My weight gain has been acceptable, though not great. I thought my doctor was overly cautious, but it would appear that yours is more so!
I'd be annoyed too. That test is awful.