I'd put it on his ankle so that it doesn't make it into his mouth often. Plus, at the hospital they put the baby bracelet on his ankle, so I think that would probably be a good spot.
Our hospital gave out stickers that stick onto a carseat with name, address, birthdate, allergies/medical info, and contact info. I guess it is so that, in the case of an emergency, they have more info if you can't provide it. That might be an option. Ours is a bright orange sticker- certainly visible. (We haven't put it on yet, so I'm not sure where it would go.) That way if something happened to his ankle in an accident, it would be on his carseat as well. I don't think people would check a carseat for a bracelet, you know?
And honestly, you'll probably forget the diaper bag or switch bags, so I don't think that would be a good place for one either.
If it were me, I'd leave it on all the time so that 1) he's used to it and 2) it wasn't forgotten. My brain is fuzzy all the time now, and I'm sure I'd leave without it despite my best attempts.