Why is it not Божьей Матери ?
Божьей is the feminine genitive of божий
These are two types of spelling of the same word.
Just like for example сиянье - сияние, ученье - учение, and the rest of verbal nouns, all of which are current and legitimate today.
In this particular case Божия is an archaic form which is kept due to being part of a proper name.
Another example is вражья - вражия
The form pertains also to plural
"Письма к разным лицам о разных предметах веры и жизни" by Святитель Феофан Затворник (1815-1894)
Прп. Феодор Студит, "Творения", translation from 1871-2
as well as to Instrumental of мысль - мыслию
"Четыре слова о молитве", Слово II by Св. Феофан Затворник, Nov 22, 1864
basically Instrumental of quite a few words belonging to the 3d declension group ending with -ь 1
Христос воскресе из мертвых, смертию смерть поправ Easter lity
В свободное от должности время большею частию сижу дома Dostoevsky,"The Landlady" (1846—1847)
and multiple case declensions of the 2nd declension group neuters ending with -e
The word статья back in the 19th century would be spelled as статия (nowadays this is the normative spelling in particular in Bulgarian)
"Кирилло-Белозерский монастырь и его устройство до второй четверти XVII века" Том 1, by Н. Никольский, 1897
This might originate from the Church Slavonic or be a result of its influence as a liturgical language of Russian Orthodox Church.
Similar alternation would occur in the pair судья - судия, of which the second form is typical to Church Slavonic.
1 In fact such form of Instrumental ending (along with the modern form) for the said declension group is attested in textbooks on Old Russian. In particular in "ДРЕВНЕРУССКИЙ ЯЗЫК. Учебно-методическое пособие", Appendix 3, table "Существительные с основами на ĭ". So the suggestion of Church Slavonic influence may not apply to this case or even at all.
The church was named long before "Божьей" became a grammatic norm.
More importantly, Russian Orthodox Church uses Church Slavonic for many things, including official naming. This particular brand of Slavonic is very archaic and preserves letters, words and grammatic constructs that fell into disuse centuries ago. Don't bother trying to learn it, simply keep in mind that whenever something relating Church is considered, something weird might pop up.