I've noticed in more than an occasion that the short form of adjective is used in places where I would use the normal (long) form typically for example:
I was listening to a song in Russian is which the singer says: "Моя душа полна тобой" and my question is why he used 'полна' and not 'полная'?
so, what are the rules to when to use a short or long form of adjectives?
Basically, you use it only after "is". Of course in Russian there is no "to be" in the present tense, but none the less you can only use short forms (in modern russian) in this position. You can use the long form in this position too but it's meaning can be different.
Моя душа (is) полна тобой
Short forms describe things that are true right now, in the present moment. In contrast if we use a long form we need to put some time marker сейчас, сегодня и тд to show that it is not a permanent characteristic of the person.
он весёлый—he is a happy person.
он весел—he is happy (now/ looks happy)
он сейчас весёлый—he is happy (now/ looks happy).
Without some time reference implied or otherwise the long forms show permanent characteristics. Also remember that adjectives only describe appearances, and adverbs describe true emotions.
он веселый, хотя ему не весело. He looks, seеms happy, but really he isn't.
It can be hard to generalize because the nuances vary depending on a given adjective; it's very context sensitive. A few generalizations:
As VCH250 mentions, when an adjective can express either a temporary or a more permanent state, the short form is used for the temporary state (think "short time, short adjective"):
Вчера она была больна. -She was ill yesterday.
Он больной. -He's sickly/an invalid/has some long-term health issue.
Short forms are often used when there's some limitation in the context; the corresponding long form would indicate a more general, inherent quality. This touches on your example Моя душа полна тобой - my soul is full with/because of you.
Она права. -She's right (about this particular thing).
Наше дело правое. -Our cause is just (an inherent quality).Он ей не был симпатичен. -He wasn't attractive to her (maybe others did find him attractive, but she didn't)
Он очень симпатичный. -He's very nice/attractive.Они не способны выполнять свои обязанности. -They aren't capable of fulfilling their obligations. (maybe they can do other things, but not this)
Я его давно знаю, он очень способный. -I've know him a long time, he's very capable (in general).
Short forms are often used with a preposition for phrases best learned as vocabulary:
готов к "ready for," похож на "similar to," знаком с "acquainted with," согласен с "in agreement with," etc.
Short forms can be used to describe an action:
Пить вредно. -Drinking is bad for you.
If none of the above applies to a given context, then the short form, as others have mentioned, often sounds more formal, archaic, or bookish.
Это хороший вопрос. -That's a good question. (normal, conversational tone)
Вопрос хорош. -The question is a good one. (heard in a radio interview with an academically oriented expert)
Learners should avoid using short forms before a noun, since they're only used in the predicate (the English translation will have 'was,' 'is/are,' or 'will be'). Russians can play with word order - in Евгений Онегин Pushkin wrote "Но вреден север для меня." - but note that it means not 'the harmful north,' rather 'the north is harmful for me (doesn't agree with me).'
