It seems that your confusion stems from an inadequate translation.
Они живут у меня дома. means 'They are staying with me' or 'They are living at my place'. The emphasis is on me and my household. They are a part of my household.
Они живут в моём доме. means 'They live in the same building as I do'. The emphasis is on the building. We do not know whether they are part of my household or not.
The preposition y has the following meanings in Russian:
- near, next to (около, рядом с ◆ Жить у моря./ to live near the sea);
- from (от ◆ Узнать у друзей./ to learn (to get to know about something) from friends)
- possession (обозначает владельца ◆ У меня есть./ I have...[I am the owner of...] )
- indication of a part of a whole (обозначает часть целого ◆ Ножка у стула./a leg of a chair)
The general meaning of the preposition в is 'inside'. For other meanings please consult your grammar book.
These prepositions are not interchangeable and convey different relationships with an object. However, translations to other languages may not reflect these differences for one reason or another. Often times it is related to collocations used in a translation target language.