Your understanding of the difference in the verbs' meaning is absolutely spot on.
However pас-/отпугивать мне is not from me, it means to my displeasure and inconvenience. A difficult one to render idiomatically in English.
It's a colloquial expression of displeasure or indignation with someone else's actions when they interfere with or disturb one's comfort, plans etc. For example
Что ты мне тут расселся? - What's up with you sitting around here?
Вы мне тут ходите туда-сюда, а я убирай (a typical rant from a grumpy cleaner)
Ты мне оставь/брось эти выходки! - You'd better stop your tricks!
отпугивать can be used with preposition от + Genitive, but not so much распугивать. The latter doesn't go well together with indirect object. One it can be used successfully with in my opinion is that which denotes a place and is governed by preposition из/с(о) + Genitive.