The Difference between "могла" and "смогла"
"Она не могла сдержать слёз" implies that the person couldn't hold back their tears. "Oна не смогла сдержать слёз" means the same thing - in both cases, this person had actually been crying; there's nothing figurative about it.
Understanding the actual difference between the usage of imperfective "мочь" and its perfective counterpart "смочь" in these phrases is quite tricky.
When we say "она не могла сдержать слёз", it means that at that exact point in time, she failed to hold back her tears. When we say "она не смогла сдержать слёз", it also means that she failed to hold back her tears, but we're referring to that moment in the past.
The difference is very, very subtle - one can say "глядя на себя в зеркало, я не могла сдержать слёз" and "глядя на себя в зеркало, я не смогла сдержать слёз". It all comes down to the narrator's perspective - whether the action is described as something that happened simultaneously to the act of looking in the mirror or as something we're recalling. So both phrases are valid.
However, there are cases when we cannot choose "смочь" over "мочь". If the action is repetitive, then it's kind of imperfective by design, as in this sentence:
Каждый раз, как она смотрела на себя в зеркало, она не могла сдержать слёз.
Here, we cannot say "она не смогла" since that would imply a one-time, completed action.
As for the implication that the person didn't necessarily actually cry, well, that may very well be, but I say it's more a matter of actual usage like, you know, when someone says, "I was literally dying there" - that doesn't mean that they were literally dying there, but then that has nothing to do with grammar, gerunds or whatever.
I'd say that imperfectiveness adds a shade of possibility, but most likely there was confusion between "я не могла сдержать слёз" and "я могла сдержать слёзы" - the latter being indeed ambiguous without any context (they potentially either managed to not cry or failed in not crying).
The Difference between "cдержать" and "сдерживать"
"Cдержать" and "сдерживать" is yet another perfective/imperfective pair, but in this context they can be used interchangeably.
To understand the general difference, consider the following sentences:
Постарайся сдерживать свой гнев.
and
Постарайся [сейчас] сдержать свой гнев.
In the first sentence, the advice "try to control your anger" is general, like one should be able to control their emotions under any circumstances.
The second sentence is about some specific situation that happened, and the advice is given in the context of that specific situation, as if saying "try not to be mad now/anymore".