What is the logic behind these adverbs? I'm trying to get a more literal understating of why they are in the accusative feminine and why the preposition "в" or "за" is used.
It's highly likely that most of these were formed after this had already grammaticalised as a way to form adverbs, so they don't really have a literal reading. The original adverbs formed this way (maybe even a single one) probably did make sense.
Maybe there was a feminine noun that got lost. Or maybe this isn't feminine singular accusative at all, but obsolete dual locative. Dual body parts are certainly likely candidates for omitted nouns, though what particular ones were originally referred to, and with what adjectives, is hard to tell.
Note that вовсю also belongs on this list, while вправду doesn't really.
The adverbs you listed follow actually two different patterns:
в + adjective ( впустую)
в + noun ( вправду)
The first one has perhaps developed in some way from the second, but I can't say how.
As for the second pattern, I can shed some light on it:
в + smth (accusative) has the meaning for the sake of smth, in order to do smth, in order to form (make up) smth up
Such constructions are rarely used in today's language. Most of them has transformed into adverbs, but some still keep its original state:
во исполнение - in order to implement, in order to form an implementation
в продолжение - in order to continue, in order to form a continuation
во избежание - in order to avoid
Thus, вправду means in order to make truth. Я и вправду опоздал на поезд - I have really missed the train (the fact that I have missed the train makes the sentence true.)
Adverbs are unchangeable parts of speech. The word вправду doesn't exist in russian. I guess you meant взаправду. There is no any logic in using specific prefixes. Because the words впустую, вплотную can not be used with the prefix за like the word зачастую can not be used with the prefix в. You just need to remember its meaning.