Since there are many possible diminutive endings, how do Russians know which one to use for a word that has not yet been so modified? For instance, if the English word "blob" (i.e. a formless, viscous drop or daub) is borrowed as блоб and a Russian wants to indicate a smaller or repulsive version of it, how would she or he decide what to say or write - блобик, блобочка, блобец, or something else?
There's no formal rule. But if you borrow this word as 2nd declension male gender, then, I believe, you should conform to other nouns of such type. Say, "мяч"->"мячик", "конь"->"коник" etc. So the most natural form sems to be "блоб"->"блобик".
Note that the alteration of consonants works for the borrowed words too, e.g. "блог"->"бложик".
I do not know how to construct a repulsive version using diminutive. It would be rather augmentative that would be repulsive (блобище).
The natural diminutive here is блобик. It is rather sympathetic than repulsive. Блобочка is of different gender, it is a diminutive of a would-be word "блоба" which does not exist. Блобец is somewhat ironic but also underlines unvanishing and/or problematic existence of the object. It is something a thing, you know.
In such a case, every person would use the form s/he likes. As for me, I would use блóбка. As the word is used more and more with the course of time, one or two diminutives would remain, the rest of them would get stylistically marked, that is, acquire some connotations, like derisive, humiliating, not that little as one would expect, etc. It is not uncommon for Russian nouns to have more than one diminutive forms, e. g. from книга such diminutives are possible: книжка, книжечка, книженция, книжонка, книжоночка, even книжулька and книжулечка. The diminutive suffixes don't have any specific meaning, they are context-oriented, e. g. in some contexts, книжечка can mean exactly what книга means, in other contexts it can mean 'small book', in other 'good book', in other 'bad book', and also as many other variations of the idea of a book as the number of contexts possible.
how do Russians know which one to use
They just do. :)
The thing is: every native speaker will have heard and used hundreds and thousands of diminuted words in context so will instinctively know which suffix tends to be used to convey which shade of meaning. And here's the gotcha: every diminutive suffix conveys a certain attitude of the speaker to the person/object called by a diminuted form of the basic word.
You may find the following somewhat informative:
Diminutive form in Russian is created by adding suffices, not endings. Of cause there are rules, and here are a couple of links:
I’d encourage you not to give into the “only the native speaker can blah blah blah” argument. My opinion is that people often use it inflate their self-worth, and not as often to point out minute peculiarities that you failed to see. I’ve stopped doing that (the self-worth part, I mean), and now my message is this: basic grammar is not beyond your grasp.