Studies have shown that severe anxiety and stress during pregnancy can affect the baby in a negative way (pre-term labor, low birth weight, etc). I don't say that so that you feel worse, but to encourage you to reach out for help.
Do you have a psychiatrist? If not, please try to get one. Primary care physicians mean well, but they are not specialized and sometimes don't give the best advice when it comes to anxiety and depression (especially during pregnancy). My own PCP told me to "stop worrying so much" and said that medication was not an option during pregnancy. He refused to give me my medication, so I had no choice but to wean off.
I only got help when I hit rock bottom, became suicidal and had to admit myself to an inpatient psychiatric program at the hospital (I was 5 weeks pregnant at the time). I was under the care of a wonderful psychiatrist who said that women who have mental illness before pregnancy will probably experience an increase in symptoms during pregnancy. For many of us (myself included) going off medication is a very bad idea.
If you find a psychiatrist, I guarantee they will work with you to find a safe medication to take during pregnancy. They can speak without about the risks (if any) to the baby if that's something that is worrying you. I personally take 10 mg of lexapro, and I felt so much better when I started taking it again.
I also go to therapy, and it helps a little, but therapy alone wouldn't work for me.