Well, it's just like that. You have to remember this.
The long story: here "брата" is actually a relict of dual number, not singular. Some time ago it must be kind of "два старша брата", yet adjectives didn't keep any trace of dual form until nowadays, so for 2..54 it is now like this: "plural adjective" + "singular noun".