Pierre Morel said the talks had "encountered procedural difficulties," without elaborating further, and that all parties concerned had decided to suspend the meeting.
The talks in Geneva were to focus on security arrangements for Georgia's rebel republics Abkhazia and South Ossetia, as well as on Georgian and Western concerns over the Russian military presence in the region.
Russia and Georgia fought a five-day war after Georgia attacked South Ossetia on August 8. Moscow repelled the attack and recognized South Ossetia and Abkhazia, where many residents have Russian citizenship, as independent states two weeks after the operation "to force Georgia to peace" was concluded.
The Geneva talks were designed to follow up on the cease-fire deal brokered by French President Nicolas Sarkozy.
Johan Verbeke, the UN secretary general's special representative for Georgia, warned against overdramatizing events at the beginning of talks, saying that all the parties to the conflict had come to Geneva and set out their cases. "The process is on track," he said.
Russian Deputy Foreign Minister Grigory Karasin, who leads the Russian delegation, promised to comment at a news conference later on Wednesday.
Georgia's president, Mikheil Saakashvili, accused Russia of walking out of the talks, Reuters reported.
Abkhazia's foreign minister said the republic "was not satisfied with the status" it was offered at the talks. Sergei Shamba said Georgia refused to treat Abkhazia as an equal partner, and the meetings were not attended by Georgia.
"There were two separate meetings," Shamba said.
South Ossetia's acting prime minister, Boris Chochiyev, also said the republic's and Russian representatives did not meet with the Georgians in Geneva.
"What can one talk about with them [the Georgians]? No, we did not have any discussions," Chochiyev said.
The participation of Abkhazia and South Ossetia in the talks was a last-minute decision, taken after repeated demands by Russia that representatives of the republics be invited to attend.
The meetings in Geneva are hosted by the European Union, the United Nations and the Organization for security and Cooperation in Europe (OSCE).
The United States, which has backed its ally Georgia throughout the crisis, is represented by a delegation led by Assistant Secretary of State Daniel Fried
