This strange structure is not random, it comes from history. In Old Russian (and other Slavic languages), there was a dual number - a special grammatical form used specifically for two items.
Over time, the dual number disappeared, and its forms partially merged with plural forms.
As a result, nouns after 2–4 kept old singular-like forms, but adjectives switched to plural forms.
What we see today is a grammatical fossil – a hybrid of old and new systems.