@Quassnoi has explained the meaning, but I would highlight the main reason why the endings are different: cases.
The preposition "в" can be used with different cases depending on the meaning. In your case, the two phrases have different meanings, and thus use different cases:
не в себе
this is meaning 1 according to the link above:
[with prepositional] (location) in, at, on
It refers to a location, means that something is (or isn't) somewhere. "Out of one's mind", as @Quassnoi puts it. Therefore, the prepositional case is used. Compare "он не в театре", "вода в кастрюле", etc.
как не в себя
this is meaning 2.1 according to the link above:
[with accusative] (direction) to, into
It refers to the direction of some movement, that something is moving or being moved (into) somewhere. "As if not into oneself", as @Quassnoi puts it. Therefore, the accusative case is used. Compare "он идет не в театр", "наливать воду в кастрюлю", etc.
Compare also the phrase "выйти из себя" ("get out of oneself", "get out of one's mind"). While the allusion is the same as in "не в себе", this phrase refers to the direction of movement, not to a static location, and therefore uses the accusative case.