I say go for it. I doubt anyone will think it's weird, unless your first son goes by the names "Thomas Edward" to everyone. If it's just a middle name on a paper it will probably not be given a second thought.
I have a slightly similar situation where my husband wants the middle name of our male baby to be Charles, because that's been a tradition in his family for a few generations. I told him that if we end up having three boys though, we'll have three babies with Charles as their middle name. I just don't think it's fair to give one boy a special and meaningful name and not the others. Plus it's just a middle name, so who's going to really care that much? I think the same goes for your situation. The name means something to you, so name your baby Edward. :)