Sorry, Alex, but the referees are not to blame here. While the call was correct, it really didn't affect the game. Here's what really changed the game: The Giants had three fumbles during the game, and the Patriots recovered zero of them. During one of those fumbles that the Patriots would have recovered, it was nullified because the Patriots defense had too many men on the field, which was quickly verified on replay with 12 guys counted by the NBC graphics. I thought it was the biggest mistake of the game even though it was just the first quarter as the Giants scored their first touchdown two plays later to make it 9-0.
The bottom line is that the Patriots committed more severe mistakes at the most critical moments of the game than the Giants did. You have to make your opponent pay for making errors, and the Patriots simply didn't do it. They failed to convert when it mattered, and those dropped passes in the fourth quarter cost them dearly. The Giants did a better job making the Patriots pay for their mistakes and making more critical plays on offense, defense, and special teams. That's why they won Super Bowl XLVI.
At least your team even made it there and competed well, Alex. Thirty other teams wished they could have been in your position, and your team has won three Super Bowls in the past decade. That's still quite an achievement.