BAH is a with/without deal. You either get it or you don't. So your baby will not impact his BAH assuming you are already married.
COLA, on the other hand, does change with # of dependants (up to 5). But you will only recieve COLA if you are living OCONUS (in a country with a higher cost of living or HI) or CONUS in AK.
Housing on base is often decided by family size. For example, 2 kids usually gets you a 3br, 3 kids will get you a 4br (if available). But that definitely depends on the availability. One kid will pretty much change nothing in that arena.
Congrats on your pregnancy!