According to prosecutors, 39-year-old Siraj Wahhaj, who had an AR-15 rifle and four pistols when police officers encountered him, was training some of the children, aged one to 15, to commit school shootings.
Police said the compound included "no food, clean water, leaking propane gas, filthy conditions, hazardous wood and broken glass, no hygiene or medical care."
A foster parent of one of the 11 children held at the compound told police that Wahhaj "had trained the child in the use of an assault rifle in preparation for future school shootings."
"These children were hungry, they were thirsty, they were filthy," Taos County Sheriff Jerry Hogrefe told reporters.
The children are now in the care of the New Mexico Children, Youth and Families Department.
READ MORE: Hundreds of Children Abducted, Killed by Islamists in Iraq Over Four Years
According to US media reports, Siraj Wahhajj's father, Imam Siriaj Wahhaj, a prominent Muslim leader in Brooklyn, New York, testified at the trial of Omar Abdel-Rahman, who was later convicted of the 1993 bombing of the World Trade Center.
In 1991, he became the first Muslim to lead an opening prayer before the US House of Representatives in Washington.
