It's going to be different for everyone. It could be painful or not, you could become engorged or not, you could get a clogged duct or not... it just depends. I would just listen to your body and any new aches or pains you get.
If it's been four days already, it shouldn't be too much longer before you're in the clear.
GL!