I would like to see you division breakdown. The Colts get 4 games against 2 of the 5 worst teams in the league. Like you said, the Patriots have played in a crap division for years.
In college the rivalries still exist and those teams play each other only once a year. I don't think getting rid of divisions hurts rivalries at all. It may actually add new ones.
The West: Seattle, Oakland, SF, SD
The Southwest: Arizona, Dallas, Houston, New Orleans
The Midwest: Denver, KC, StL, Tennessee
The Southeast: Atlanta, Jacksonville, Tampa Bay, Miami
The North: Minnesota, GB, Chicago, Detroit
The East: Pittsburgh, Baltimore, Washington, Philadelphia
The Northeast: Buffalo, NYJ, NYG, NE
The Mideast: Indianapolis, Cincinnati, Cleveland, Carolina
Don't know if it really matters which is NFC and which is AFC.