The question in the title and the question you wrote are not quite the same. The question in the title sounds like it is asking "what case is used after mathematical objects denoted by letters like G, H, etc.", while the question you wrote is asking "what case follows a counting variable"? [EDIT: The question in the title was later changed to match the question being asked.] A mathematical notation
for an object (not counting something) is often considered neuter, but things following it may have case endings based on the gender for the Russian name of the object. For example, an equation x = y has the sign = pronounced as равен, равна, or равно depending on if we consider x to be masculine, feminine, or neuter. Often x is neuter, but if the equation A = B involves linear operators then = may be pronounced as равен (masculine) because оператор is masculine. If an equation f = g involves functions then = may be pronounced as равна since функция is feminine.
You thought "such that" might be который. It is такой что, but there's more to say about this than just the direct translation. Specifically, in ordinary speech the words такой and что are often not next to each other. This is best understood by examples. I give three below, putting in parentheses the pronunciation of the mathematical notation. (If you know how to pronounce the alphabet in French then the Russian pronunciation of letters in math will make sense.)
1) Let x_n be a sequence such that = Пусть x_n (икс эн) — такая последовательность, что
2) Let f(x,y) be a function of two variables such that = Пусть f(x,y) (эф от икс игрек) — такая функция двух переменных, что
3) A generator of a group G is an element g such that = Образующий группы G (Жэ) — такой элемент g (жэ), что
These literally say "Let x_n be such a sequence that", "Let f(x,y) be such a function of two variables that", and "A generator of the group G is such an element g that", where "such" modifies the object of interest (a sequence, a function, or an element), and all of these sound clumsy in English.
The abbreviation of такой что on the blackboard is ",т.ч." (note the comma), which is like the use of s.t. for such that in English. However, because такой and что are often separated from each other in speech, it is awkward to replace them with т.ч. on a blackboard if writing resembles speech. To make т.ч. more natural when writing and speaking at the blackboard, the word такой is moved in speech right before что even though, I am told, the result usually sounds stylistically awkward to non-scientists. The new versions of the above examples are:
1) Пусть x_n — последовательность такая, что
2) Пусть f(x,y) — функция двух переменных такая, что
3) Образующий группы G — элемент g такой, что
The ending on такой depends on what is it modifying. Here are some examples to illustrate this, where the same English sentence is translated in more than one possible way.
1) the dimensions of the spaces such that = размерности пространств такие, что
(when "such that" refers to dimensions)
2) the dimensions of the spaces such that = размерности пространств таких, что
(when "such that" refers to the spaces)
3) sequence of points in a metric space such that = последовательность точек в метрическом пространстве такая, что (when "such that" refers to the sequence)
4) sequence of points in a metric space such that = последовательность точек в метрическом пространстве таких, что (when "such that" refers to the points)
5) sequence of points in a metric space such that = последовательность точек в метрическом пространстве таком, что (when "such that" refers to the space)