I have a very similar story to yours. My first lap was with an OB. The lap was solely for diagnosing the endo, since that's the only way to definitively diagnose it. He wasn't qualified to remove the endo, and didn't have access to the equipment anyway, so it was diagnosed, staged, and then I was closed up. He also did an HSG while I was under, which was very nice of him :)
I was then put on 6 months of Lupron, which sent me into a pseudo-menopause and was supposed to "quiet" the endo lesions down. After that, we were told to try on our own again. It would be ideal to get pregnant within the first 4 months. Well...that didn't happen.
So we went to an RE, who did another lap on me to remove the endo. I was only stage I-II, but I had most of it on my colon and bladder :/ He also removed a fibroid that we found on ultrasound.
Plan after the endo removal was instead of us trying on our own, to move straight to IUI to increase our odds. We did 3 back to back IUIs with no luck. I was on Clomid, and my cycles were not monitored, so...whether that made a difference or not, I don't really know.
After we ran out of money, we tried on our own really hard b/c I knew our greatest chances of getting prego after the endo removal surgery was a year, but...no luck :(
We are moving on to IVF next month, and I am hoping the endo will not complicate that. I asked my RE (different from the one who did the endo surgery) whether or not that decreases our odds of success, and he said no. My endo symptoms are very, very mild compared to some of the stories I've heard, so at least I have that going for me...
So it's hard to say whether you should go straight to IVF or get "cleaned out" first. I think removing the endo is a good idea, but you may want to consider IUI or IVF right afterwards to increase your odds.
Good luck in your decision!!