Balance was never the primary reason for level limits, though as balance became a buzzword in RPGs TSR starting pushing this angle more and more. But when the original D&D was shown to favor magic-users too much, the solution was not to cap magic-users' levels, but to give the other classes more hit points and better attacks.I agree that, for B/X and BE part of BECMI, Dwarves and Elves as 'you cap out 2-4 levels early, but get XYZ in exchange' works fine if you accept the central conceit of TSR-D&D balance (more powerful at some points in the gaming career, but weaker in others). Halflings being most constrained was never logical, AFAIC.
The real reason was verisimilitude. Gygax preferred his fantasy with a strong humanocentric viewpoint. Hobbits never rise to the great heights of characters like Conan or Fahfrd. The highest they rise is how Merry and Pippin end up in The Lord of the Rings, and that very believably maps to Hero level. Lo, in the original D&D rules, hobbits are limited to level 4 as fighters.
Viewed with this history in mind, halflings going up to level 8 as Sheriff is pretty high level, and the elves' and dwarves' limits of 10 and 12 are hardly limits at all in a game that rarely went beyond 14th level.