I think my dad could have written this post! 
On my mother's side I am first generation. My mother has been very forthright with her cultures and traditions, and when my grandparents and aunt (or other family members) would visit, they would speak in their native tongue, which my dad did not speak. However, my grandparents knew English (they taught themselves so they could make a better life here) so they did speak English more often than not - so maybe it's not exactly the same.
The point is, as a child brought up in a similar situation, I will say I am more familiar with my mom's culture and traditions, and even identify myself as that culture. BUT I know A LOT about my dad and his family and feel a bond with that side as well.
I agree with PPs who said to try and learn some of the language - it'll be beneficial in the long run and maybe it will show your DH's family you are making strides and respect their culture so hopefully, they will in return, respect yours as well.
Good luck! I hope it works out the way it did for my family for yours!