Die Linke had been tipped to form a left-wing coalition with the SPD in Saarland, but the parties needed to win another two of the 51 seats in the state parliament in order to get the required majority.
The CDU, which has governed Saarland in coalition with the SPD for the past five years, received 24 seats. The SPD took 17, Die Linke seven and Alternative for Germany three.
Co-chairwoman of Die Linke Katja Kipping accused the Social Democrats of being too hazy in their presentation of an alternative to CDU government.
Announcements about a possible "Red-Red" coalition of the SPD and Die Linke were "very vague and undefined," Kipping said.
"This contributed to the fueling of anxiety, but it didn't contribute to enthusiasm for a just change, a possible political change," she told the German broadcaster ZDF.
Die Linke co-chairman Bernd Riexinger also blamed the party for not giving specific election promises, Der Spiegel reported.
"This has shown that the hype about Schulz didn't translate into election results in Saarland. You can't win with general statements and slogans," he said.
While there has been speculation that the SPD might form a coalition with Die Linke and the Greens after Germany's federal elections in September, the party hasn't made any concrete promises.
For the last four years, the SPD has governed Germany as the junior partner in coalition with the center-right CDU/CSU bloc.
Activitists such as the Young Socialists have called on Schulz to exclude the possibility of another coalition with the conservatives, since "socially just politics aren't possible with the CDU and CSU."
Rather than enjoying the expected bounce after the election of Martin Schulz as party leader, the Social Democratic party (SPD) took 29.6 percent of the vote, down one percent in comparison with elections in 2012.
However, Martin Schulz believes the result is based on regional factors, such as the popularity of CDU Minister-President Annegret Kramp-Karrenbauer, which won't translate to national elections to the Bundestag in September.
"Congratulations must go to the Minister President, Mrs. Kramp-Karrenbauer, who I believe has won in Saarland thanks to the Kramp-Karrenbauer effect," he said.
"That (the success of Die Linke) is why I believe this election result – as well as the result of Die Linke – can't allow any conclusions about the federation to be made," Schulz added.
For her part, Saarland Minister-President Annegret Kramp-Karrenbauer (CDU) said the result shows "the CDU can win votes when it mobilizes."
The CDU increased its share of the vote by 5.5 percent, to 40.7 percent. In fourth place was Alternative for Germany with 6.2 percent of the vote, followed by the Greens with 4 percent and the liberal Free Democratic Party FDP with 3.3 percent.
Alternative for Germany leader Frauke Petry said she was disappointed with the result.
"It must be said that a majority of the Saarlanders have chosen the grand coalition, which in our view promises only a standstill," she said.
