Our situations are similar. DD was an IVF pregnancy and then DS was a happy surprise (they are 16 months apart).
I think we are done. I am honestly 50/50 but DH is more adamant that we are complete. It has more to do with his job (military - he is gone a lot) and our financial goals than the experience of 2u2.
Just remember, baby and toddlerhood is a short season. Yes, it is really hard when you are knee deep in it, but it is one small portion of parenting and the family dynamic. I try to focus on long term goals when weighing my feelings on more kids - everything from college tuition to how I envision our thanksgiving table in 20 years. Not my emotional reaction to the tantrum my 2 year old had yesterday.