The key words, I am guessing, are через день, but I don't understand what the phrase is supposed to connote in this context.
Example: я перепрыгнул через камень - I jumped over a stone.
In such phrases "через" conveys an idea of some 'obstacle' to be skipped.
So, "хожу через [один] день" is conveying that after you visited the place, the next day would be an obstacle to skip over.
Like those mentioned above "через два дня" and "через три дня" conveying that after each visit next two or three days are again considered a nuisance to be skipped over.
I am not a native English speaker, but maybe you can translate your phrase as "No, I am attending the pool skipping next (after visits) day(s)".
So why is an abstract verb of motion appropriate in that context? How does it represent a repeated action or a motion without a specified direction?
Because you are not talking of any specific visit. You are not asking "how was you pool session on the last Tuesday going?", you are not suggesting "When you depart for the pool, take the umbrella, skies are clouded", you do not mean any specific visit in itself. What you are querying about is the general pattern of behavior, a schedule of all visits without stressing any specific one. Here "ходить" is rather "to attend" than "to go" or "to reach" or "to move".
By the way, within the example with a "focused" verb "Я пойду через день" by @shabunc the meaning of "через" also can be conveyed via skipping: "I will first skip the tomorrow day and then I will be going the day after skipped tomorrow".
Also, there is a word "чересполосица" :-D