Why is the word сосед declined as соседей instead of соседов since the stem ending is a consonant?
It's just an exception. In Zalizniak's Grammatical Dictionary this word is marked with the △ sign:
- сосе́д мо 1a △ мн. сосе́д|и, -ей, -ям
This word's stem gets softened in the plural: соседи, соседям, соседях (rather than *соседы, *соседам, *соседах) so the declension follows the soft pattern (as if the nominative singular were *соседь, similar to медведь, лебедь, жёлудь).
As to exactly why and how this exception came into existence, I have no information. At least соседи is consistent throughout the plural. There are cases when softness is lost in just one form, genitive plural: башни - башен (not *башень) but деревни - деревень.