This is what I have learned. Going to court is actually the NICEST thing to do, not the other way around. It may not seem like it at first but once people have guidelines to follow, relationships get easier and (usually) things get way better.
You might qualify for legal aid so you could check that. Also even if you have to come up the money for a lawyer, its one of the best gifts you could give to yourself and your child.
Do not go to court without a lawyer, it might work sometimes but in your case, your ex sounds very manipulative and it would be too risky.
GL