In Proto-Slavic, this preposition was a merger of two earlier Proto-Indo-European words: one meaning "around" (> Latin ambi- > "ambiguous, ambivalent", > Greek amphi- > "amphibia, amphitheater"), another one meaning "on" (> Greek epi- > "epilogue, epithet", > Latin ob- "observe, oblige"). It did have the final b.
Due to the law of the rising sonority, that final b became impossible before less sonorous consonants (fricatives and affricates).
There were three ways to deal with it:
- Insert a prosthetic vowel after the b
- Drop the b
- Drop the consonant after the b.
Dropping the consonant after the b would change the root, so it was only used when the consonant in question was either another b or v: обить, обонять, оболочка etc. It was not always used: in e.g. овеять the b was dropped; in обобрать the prosthetic was used.
The other two options (ob > obъ and ob > o) were widely used, quite interchangeably. There were probably some finer phonological reasons for choosing one or another option, but they would come with lots of exceptions.
Shortly after that, another process, the fall of the reduced, was added in the mix. The vowel ъ, being a reduced, had either to fall (disappear) or to morph into the full vowel о.
Reduced vowels fell in so called weak positions: before a syllable with a full vowel or a vowel under stress. In strong positions (either under stress or before a syllable with a weak reduced) they morphed into the full vowels.
This meant that the prefix объ- would change into обо- if it was in a strong position, meaning that the next syllable had a weak reduced. In modern Russian, such roots have a so-called "fleeting vowel" in them, which disappears ("flees") as the word changes its form: оборви / обрывать, обопри / опереть, обдери / ободрать etc.
When used as a preposition, in modern Russian, обо is used before single-syllable words with historical чьт, вьс, мьн: обо всём, обо всех, обо что, обо мне. In other cases, о should be used before consonants. But this is quite an artificial rule, and old texts are full of phrases like обо ржи, обо Льве, обо льготе etc. Note that prefixes and prepositions are very closely related in Slavic languages and until quite recently (early XIX century) grammarians didn't make any distinction between them.
In some cases, after the prosthetic ъ had fallen, the consonant clusters like бб, бв, бп appeared again. Some of them were further simplified (опасть, опека, обежать etc.), some left as is.
Russian has a lot of doublets like обвязать / обязать, обниму / обойму, обволочь / облечь etc., where one of the words is native Russian, and another one is a Church Slavonic borrowing. These doublets show the different evolution of this prefix in different Slavic languages, when the roots were the same, but phonological conditions were different.
The fleeting о is a thing in a lot of Russian prefixes and prepositions: c-/со-, ис-/изо-, над-/надо- etc, for the same reason. In these prepositions, however, the historic ъ was not a prosthetic vowel but rather an integral part of the preposition, so it always stayed in its place right until the fall of the reduced.
