I gave up on regular pants and have been wearing maternity leggings since like week 13, and definitely have a bump now at 21 weeks. I lost weight at first and have only gained 5 pounds total. I had a bit of a tummy before getting pregnant too.
Don't worry about being "too big" and definitely don't compare yourself to anyone else! Everyone is different. Your Birth Month Board probably has a Bump Day when everyone posts pictures of their tummies. If you look at those you'll see there is a HUGE variation in "normal". Some women get big right away, and some don't. It's all good!
Also, take a look at this website: http://www.msichicago.org/whats-here/exhibits/you/the-exhibit/your-beginning/make-room-for-baby/interactive/
You'll see that even at 16 weeks your baby is taking up a lot of room. Your bump is definitely not just weight gain at this point!