When I hear one of these words and can't distinguish it by context
All human languages, Russian being no exception, have a high level of redundancy. Many languages don't have the concept of plural as a grammatical feature at all; in others, it's not mandatory. In the vast majority of cases, this distinction can either be deduced from the context or just isn't really important.
In rare cases, when you absolutely can't distinguish these words from the context, and this distinction does matter, you'll just have to clarify: Простите, как вы сказали? "Музей" или "музеи"?
or when I have to speak these two words differently
In regular, lax speech these words can indeed be confused. When I was in middle school, literally no one in my class, me included, knew that the name of the Jedi movie was Звёздные войны ("Star Wars"). Everyone thought it was Звёздные воины ("Star Warriors").
However, when you pronounce these two words clearly, different sounds are used: /j/ for музей and /i/ for музеи. If you articulate the final sound, you'll have to pronounce the final /i/ as a separate syllable. This way, it will be as clear as the difference between English "joy" and "Joey".