Something similar happened with us though DH does not have MFI issues and my issues are tubal. We had 12 mature eggs, 10 fertilized with ICSI. All ten were looking pretty good on day 3, six of those looked great. And then by day 5 only one looked "okay" and two more were barely okay. We transferred all 3. I was devastated. None made it to freeze.
Our RE said we were probably looking at a sperm quality issue because its the egg that is in charge the first three days. On day 3 the sperm "wakes up" so typically when things are fine up until day 3 he said its usually the sperm to blame.
We ended up doing a DNA fragmentation test on DH's sperm. His regular SA's have always been great but apparently you can have high levels of DNA fragmentation and have a normal SA so we went ahead and did the test after our transfer. We just got the results back and they were excellent. I haven't discussed them with our RE just yet but I'm perplexed to say the least.
My only conclusion is that sometimes there is just no explanation. I'm curious to hear what your doctor has to say as well as my own. He was convinced it was a sperm issue, and who knows maybe it was, maybe it was just a bad day. I don't really know how that works.
I wish you the best!