Does anyone know of any nouns like путь that are declined in a mixed masc./fem. way? What I mean is путь follows a typical feminine pattern for the dat./gen./prep. cases (пути), but its instrumental case follows the masculine one (путём).
В украинском языке существительное «путь» женского рода и склоняется так же, как и все подобные существительные, твор. падеж «пу́ттю», что сответствовало бы рускому «*путью».– Yellow SkyJul 19, 2015 at 10:18
That's a well-known school question. These nouns are called "разносклоняемые". There are eleven neuter gender nouns ending in "-мя" and one masculine "путь" which you mentioned above.
The full list is бремя, время, вымя, знамя, имя, пламя, племя, семя, стремя, темя, голомя and, of course, путь.
BTW. That's not a "mixed masc./fem. way" as declining Russian nouns depends upon their type (I, II, III) not (generally) upon their gender. Here they behave like both II and III types.
As @user4419802 mentioned above, Russian declension system is not that much about masculine and feminine as it is about etymology of the word, or, in particular, the ending of its stem in Proto-Slavic.
Путь < *pont-ĭ-s (along with гость < *gost-ĭ-s, ночь < *nokt-ĭ-s and several other nouns) had the stem ending in -ĭ.
In Proto-Slavic those words could be both masculine and feminine, but were declined in the same way, regardless of gender.
However, on some stages of language development (different stages for different dialects of Proto-Slavic, which had later become different Slavic languages), gender started to matter, and masculine nouns accepted paradigm of words with stem in -ŏ (which were only masculine and neuter). Those were words like конь < *kŏnj-ŏ-s, нож < *nŏzj-ŏ-s.
In Old Church Slavonic masculine words in -ĭ (гость-like) differed from feminine words only in inst. sg. (гостьмъ / нощью) and nom. pl. (гостiе / нощи). Other cases were the same: гости / нощи (sg. loc. and dat.), гость / нощь (sg. acc.) etc. Similar situation was in Old Russian.
In Modern Russian, masculine words originally in -ĭ (гость-like) completely changed their inflection by analogy with declensions of words with stems originally in -ŏ (конь-like).
However, since nothing left there to tell the gender of a word in -ь, the gender of such words became very mobile.
Words тополь and шинель are masculine and feminine, accordingly, in Modern Russian, but it's the other way around in Modern Belorussian. Pushkin used feminine лебедь less than 200 years ago in his poetry. As @YellowSky mentioned above, путь is feminine in Modern Ukrainian. And even native Russian speakers sometimes confuse the gender of words like тюль, шампунь, лосось and similar.
So путь apparently had swerved into feminine and back to masculine on its way from Proto-Slavic to Russian, but the constructs like к пути and на пути were probably considered idiomatic and had not been changed.
Now it's time to actually answer your question.
Words in -ĭ were the most common examples of mixed-gender declension, but not the only ones.
Око and ухо had stems in -s (чудо-like, чудес- in oblique cases) but accepted declension pattern of -ĭ in dual. So while dual is not productive in Russian anymore, we can still find it fossilized in the adverb воочию (compare ночью).
Дитя originally had stem in -nt (телёнок-like, телят- in plural cases), but accepted declension pattern of -ĭ in plural. We can use it only in nom. and acc. in singular, but in all cases in plural, including -ьми in pl. inst., which is otherwise used only with feminine nouns (with two else exceptions below). This is the only neuter noun which does not end in -о, -е or -мя.
Зверь and люди are the male nouns (originally in -ĭ) which can use -ьми in pl. inst. Люди does not have a singular form in Modern Russian, however, it's etymologically masculine. This is something people like Dostoevsky had to know by heart before the orthographic reform of 1918, so that "Poor Folk" would have been correctly named Бедные люди (and not бедныя as it would if люди were feminine).
Сын on one hand, and дочь and мать on the other hand, had stems in -ŭ (short u) and -r, accordingly. Words of those declensions had an extra syllable in oblique cases. Thus we see those words have the similar declensions in oblique plural cases: сыновей, дочерей, матерей. Other masculine words with the same declension pattern were most probably modeled after сын: кумовья, шурья (plural of кум and шурин < *шурь).