Jeff Fischer keeps making comments about being in the competition committee about "well, if you open up the replay system to THIS type of play, what about THAT type of play. Where do you draw the line?"
I think the answer is simple. You don't draw any line at all. If the refs throw a flag, you should be able to challenge it. If the refs DON'T throw a flag, you should be able to challenge it. Any time the refs do or don't blow the whistle, you should be able to challenge it (e.g. was the forward momentum stopped when the player fumbled).
I also think they should increase the number of challenges, maybe 3 per half, where you get another one for every successful challenge. I don't think there should be any limitations on when you can challenge (e.g. 2 mins left in the half). I think the NFL should have to provide a public written statement for every challenge ruling. I think that any ref who is found to have missed more than one call in a game should be fined (sauce for the goose, sauce for the gander, right?). I think any ref who is found to have missed more than one call in favor or a particular team should be suspended without pay for a minimum of 2 weeks and investigated.
It would water the game down too much. My opinion is this...you have your two challenges. You can challenge specific judgement calls such as targeting/defenseless player rules, roughing the passer, pass interference, and intentional grounding(can challenge an intentional grounding no call as well...but the rest of the rules, you cannot).
Unfortunately, you'd have to omit procedural penalties such as false start, offsides, etc. and non-specific judgement calls like holding, block in the back, illegal contact, etc.
Some will ask, why is holding omitted? Because on any given play, you're going to find holding somewhere. The refs will sometimes make a mistake on the number of the player being called. You shouldn't be able to get it overturned because they called out the wrong number. And unfortunately, they're going to be able to find a holding call that would justify the call standing if they really want to.
Whereas with roughing the passer or targeting, there's only one player that could have had the penalty committed against them(only one QB or one WR getting blown up on that play). So you can't claim, "Oh I made a mistake...I didn't mean holding on #77, I meant #67".
Ironically, you can already challenge penalties. Too many men on the field is a reviewable call IIRC.