Ivanov showed 16 yellow cards, which equaled the competition record, and sent off an unprecedented four players - one more than the previous high - in a foul tempered match between Portugal and the Netherlands on Sunday.
His performance was roundly criticized and even led to Sepp Blatter, the boss of world governing body FIFA, suggesting that he should have received a yellow card himself.
But along with Graham Poll, an English referee who booked a Croatian player three times before finally producing the red card when the Balkan country played Australia, Ivanov will now be returning home after presiding over three matches.
However, German Markus Merk, who courted controversy after becoming involved in a spat with Australia's Harry Kewell in a group match, is one of the referees on the FIFA list for the remaining quarterfinals and later stages of the tournament.
