The "explosive charges" were found in front of the treasury and the department of public finances in Bastia commune, according to the BFMTV.
"I firmly condemn this foiled attack on a public finance centre in Bastia … An attack on a tax centre is an attack on the Republic," Budget Minister Gerald Darmanin tweeted.
READ MORE: Over 60% of French People Support Macron's Idea to Create European Army — Poll
A bomb squad was reportedly called in when the first bomb was discovered in the morning. A second explosive device was found hours later, prompting an evacuation.