In compound nouns like
- самолёт-истреби́тель
- самолёт-разве́дчик
- самолёт-невиди́мка
- ве́ктор-строка́
- ве́ктор-столбе́ц
- спортсме́н-лы́жник
- спортсме́н-плове́ц
- садово́д-люби́тель
the first noun gives a more general description of the word and the second one gives a more specific description. First let's consider several other examples and then we'll discuss the words вектор-строка, вектор-столбец, and радиус-вектор. “Самолёт-невидимка” is an aircraft (this is a more general description) and more specifically, it's a stealth aircraft (this is a more specific description); it wouldn't make much sense to describe it as a stealth object (“невидимка”), which is more specifically a stealth aircraft. Similarly, “садовод-любитель” is a gardener — more specifically, an amateur gardener; again, it wouldn't make much sense to say that a “садовод-любитель” is an amateur, more specifically, an amateur gardener.
Similarly, “вектор-строка” is a vector — specifically a vector written as a row; “вектор-столбец” is a vector written as a column. It would make much less sense to say that “вектор-строка” is a row that forms a vector (there is no separate mathematical notion of row).
Now “радиус-вектор” is the position (radius) of a point, which is written as a vector. The word вектор specifies that радиус-вектор doesn't just give the distance to the position but also the direction.
Perhaps one could argue that “радиус-вектор” should be called “вектор-радиус” because it's a vector that (more specifically) gives the position of an object. I think the main reason “радиус-вектор” is used is the following: the term “радиус-вектор” is mostly used in physics, not in linear algebra. The word вектор just specifies the type of the variable: \vec r is “радиус-вектор”, whereas r is радиус (скаляр/scalar). If “радиус-вектор” was a concept studied in linear algebra and we had different kinds of vectors, then we would have terms like: вектор-радиус, вектор-ускорение, вектор-функция.
Summary To check what word order to use, you can apply the following test. Write A-B if you can say “он/она — A, а точнее A-B”:
Это самолёт, а точнее самолёт-истребитель. not Это истребитель, а точнее самолёт-истребитель.
Пусть v — произвольный вектор, а точнее вектор-столбец. not Пусть v — произвольный столбец, а точнее вектор-столбец.
Пусть \vec r — радиус (положение) частицы, а точнее её радиус-вектор. not Пусть \vec r — вектор частицы, а точнее её радиус-вектор.