You're asking about an example that is somewhat atypical as far as "that"-dropping in Russian is concerned, so even though you've already picked a best answer I want to warn you against generalising from this example. The emotional tone is only here with знать, which does not normally drop the conjunction. I don't think он любит Россию is even a subordinate clause here, rather than я знаю being a parenthetical, unattached statement, which is even movable (он любит Россию, я знаю works, but you can't say *Он любит Россию, он сказал).
Now with сказать, говорить and (по)думать, the situation resembles English much more closely. Он сказал, он любит Россию and Я думал, он любит Россию are a highly colloquial, but entirely legitimate and very frequently used alternative to он сказал, что and я подумал, что. Outside of these two aspectual pairs of verbs, though, что is generally mandatory — again, if we're talking about sentences with true subordinate clauses, which the Russian version of your example isn't.