It all depends on what you put into it- I gained 65 lbs with DD, and had lost it all before her 1st birthday but my hips were still a bit wider than they had been in the past making my pre-pregnancy size 4's a bit tighter than they used to be. I started focusing on a long-term healthy goal, switching up workout routines to focus on gaining muscle tone and worked in runs to help with the 'jiggle' (my saddle bags react best to hitting the pavement) and I ended up in better shape than I was pre-pregnancy. It felt like it took forever to lose the initial baby weight, but I was focused on it for the wrong reasons. And I also pretty much ate whatever and didn't work out much when I was pregnant, so I'm hoping I will handle this time around differently (26 weeks in and up 20 lbs).
A lot of it also depends on your body type- if you're the type who doesn't have to watch what you eat or work out to be small, then you'll probably be a little 'softer' afterwards but the weight will go away. If you have to work to maintain your weight, then you'll have to continue to work at it but that won't be an issue because you're already used to working at it in the first place.
A word of caution though- our bodies have amazing memory, so if you have been overweight in the past, your body will "remember" what that's like and feel comfortable drifting towards it. When I was 20 I weighed 205 lbs, I lost 70 lbs spread out over two years- and had been around the same weight for almost two years prior to getting pregnant, but my body still easily went back to that weight. As I mentioned, I wasn't the best during my first pregnancy but still, some people gain 40-45 when they aren't watching everything but my 40-45 included an extra 20-25...
I promise, if you've lost it before (and your'e dedicated) you can lose it again. Don't stress too much over it, make sure you're LO is healthy and focus on that. Enjoy your pregnancy and the amazing bonding time!