My DD is extremely sensitive to caffeine. I was only drink a little bit...like a cup of tea type thing. I had no idea what was wrong with her. She started only sleeping 20 minutes at a time and she became really fussy. She was owl-eyed all the time. I was looking up her symptoms...and suddenly i read up about caffeine. And sure enough, I stop drinking and eating anything with caffeine and she was back to her normal self.
But as pp said, i think it depends on the baby.