This isn't really a problem, from what little information you've already gathered (I'm treading very carefully around spoilers, here).Why this doesn't make as much sense as it should: Tokyo is a huge city, has since the Edo period, and presumably still is in the Future, yet every single crystal point has been conveniently within a single small district of the city.
The big crystal building in the middle of Crystal Tokyo is just the Palace... and it isn't that big. Moreover, the defensive Crystal points are very near it, from what we can see in the scene where they activate.
From Rubeus's map, we see there are two sets of Crystal points : 4 in the very center, and 5 at more outlying regions. But considering the scale of the Palace on that map, we can deduce that the map only depicts the very center of the City (where the Crystal points are, which is all that matters).
Now, I'm sure you can conjure up reasons for why Crystal Tokyo's palace would be set in the very middle of Azabu-Juuban, especially considering this episode's "reveal" about Chibi-Usa's mother...
On the other hand, your "the Crystal points can't be repaired yet" theory does a good job of explaining why they never hit the same point twice. Which, considering the baddies have already hit more than half of them, makes the situation a lot more desperate for our heroes than it appears at first...