I am not sure about the exact rules. But maybe with example it can be more understandable.
For instance, if you want to say "A person who wants an apple", it can be
Человек, который хочет яблоко.
or
Человек, который хочет яблока.
First one is accusative case, second - genitive. They both mean that a person wants to have an apple. But there is a slight difference in meaning. In first case, it means that there is one full apple that a person wants. It might even be a specific apple. But it is a one. In second case, it means the person wants an apple as a fruit, but not specifically one and maybe not even full one. Maybe just a slice of it.
In your case, (I am not sure if there is a grammatical explanation for that) since it is indirectly implies that the man wants ONE family as a whole, it is usually said
Мужчина, который хочет семью.
But if you say
Мужчина, который хочет семьи.
it means that the man wants just some kind of family, in general. Both cases can be used. Just with a slight tilt of meaning :)
Regarding the надежных отношений.
In this case, there is no ONE specific relationship, but it is more of relationships in general, that's why it is in genitive. It also worth to mention that in Russian in contrast to English people use the word relationship in plural form in this situation.
But it is more my logical understanding of it.