While there are not direct equivalents to geek, and nerd, we can find some terms that can match them closely, depending on context.
According to OED definition, geek is “an unfashionable or socially inept person”, with the sub-sense of “a knowledgeable and obsessive enthusiast” (emphasis mine). If we stress the obsessiveness of that enthusiasm, then the translations чо́кнутый (cracked 2., batty 1, also тронутый (touched)) and поме́шанный (insane; literally: messed (up)) often given by on-line dictionaries are suitable. Moreover, they line up with original etymology of geek (“related to Dutch gek ‘mad, silly’”). But these words do not necessarily imply “social ineptitude”, although they are usually used with disapproval to describe someone who is way too much into something or someone.
Nerd, who is defined by OED as “a foolish or contemptible person who lacks social skills or is boringly studious”, is easy to match by the word бота́ник (literally: botanist, shortened as ботан or бата́н). Why this discipline of biology acquired such dubious distinction is unclear, but there is a version that looks more like a folk etymology (as I have not seen a serious source on the subject), but it actually makes sense.
In 1825 classic of Russian literature, Griboyedov’s satire Woe from Wit (Горе от ума), there is a passage in Act III, where the Duchess talks about her nephew who is so involved in studying chemistry and botany that he is not interested in women or a career:
Нет, в Петербурге институт
Пе-да-го-гический, так, кажется, зовут:
Там упражняются в расколах и в безверьи,
Профессоры!! — У них учился наш родня,
И вышел! Хоть сейчас в аптеку, в подмастерьи.
От женщин бегает, и даже от меня!
Чинов не хочет знать! Он химик, он ботаник,
Князь Фёдор, мой племянник.
The word очкарик (one who wears glasses), while a good alternative, obviously limits its application to people whose sight had suffered from all that studying. On the contrary, why some dictionaries list зануда (bore, pest) as the first choice translation for nerd is beyond my comprehension. The word умник, while originally may have had a positive sense of someone who is laudably intelligent, now became to mean to address someone (usually in a form of exclamation) who is “too clever” and knows too much for their own good.
On a related topic, as a warning, while the word ботаник came to be strongly associated with nerdiness, the informal meaning of the word химик (chemist) and related slang verb химичить (“do chemistry”) are usually used to describe someone doing some inappropriately dangerous tinkering, messing things up.