"Вашей" is definitely wrong, it should be "вашу":
Нет, я не видел вашу машину.
The direct object of the sentence, "машину", is in the Accusative case, that is why its attribute should also be in the Accusative ("вашу"), because the possessive pronoun "ваша" aggrees in case with the noun it modifies.
Note, that in Russian, if the predicate verb is negative, the direct object can be in the Genitive case, not in the Accusative as usually. So, if the task were to insert "ваша" into such a sentence as "Нет, я не видел ____ машины" in which the direct object is in the Genitive case, then "вашей" would fit it well, since "вашей" is the Genitive case of "ваша".