In general, I do mind. But not with vowels (like Alyssa A*es) and the L in your example works fine too. I love Hazel, even if that's not the name you're considering.
I agree it's worst with harder letters, but the last name for our kid starts with an F and it doesn't work either (think Ralph Ford - it's terrible, IMHO).
I also don't like alliteration, so most names starting with an F are out too, and because DH has a stammer, all names with K/C (pronounced K) and T are out too. Luckily, I tend not to like those.