The Iraqi security forces retreated from Ramadi, some 70 miles west of Baghdad, last Sunday after days of heavy fighting with IS militants.
"They vastly outnumbered the opposing force, and yet they failed to fight, they withdrew from the site, and that says to me, and I think to most of us, that we have an issue with the will of the Iraqis to fight ISIL and defend themselves," Carter was quoted as saying by the CNN.
The defense secretary added that it was "very concerning" that the local forces showed little willingness to counter ISIL militants, who currently control large parts of Iraq and neighboring Syria.
Earlier this week, US President Barack Obama said that the fall of Ramadi was a "tactical setback," but it did not mean that the United States was losing a war against the ISIL.
The US-led coalition has been carrying out airstrikes targeting ISIL positions in Syria and Iraq since August 2014.
