Why do NCAA tournament brackets lack symmetry?

Very important stuff here!

Take a look at a portion of a normal NCAA tournament bracket. This is one region, cut off at the sweet 16, with all the favorites filled in:

Let me highlight the first round games that lead to the sweet 16 teams, assuming all the favorites win:

Isn't that literally the worst thing you've ever seen? No symmetry. I can see why you put the 2 seed at the bottom, so that the 1 and 2 approach each other from opposite ends. I can see why you do the same for the 4 seed.

But why leave the 3 seed in no man's land? This looks much better, with the 6-11 and 3-14 games swapped:

This doesn't change any of the games played, it just makes for a more elegant and symmetric bracket.

Am I missing something here? If not, does anyone know how we wound up in this horrible situation? It's hard to Google for information, because I just find articles about ranking teams and placing teams in the bracket, not about the actual structure of the bracket itself!