I agree. If it doesn't take a lot of time out of his day, then I don't see the problem. Everyone is entitled to their own "personal" time. The addition of a new baby shouldn't mean he has to give up something like this.
UNLESS... you feel he's doing it as an escape from you and the baby. Or if you feel the mother is getting in between you two as a couple. Which it sounds like it is not.. he's just giving her a ride. If he was spending every Saturday or Sunday all day at her house or something, then I'd put my foot down. This doesn't seem so bad.
Are you jealous of the time he spends with her? It seems pretty minimal, really. Is this a battle worth fighting?