Federer, desperate to recapture his best form, went into the match boasting an overwhelming 6-1 record against the flamboyant Frenchman but although he showed flashes of his inspiration, he went down 6-4, 6-7 (5/7), 6-3.
It follows a fourth-round loss at the US Open and a second-round exit at Wimbledon to players he would have brushed aside at his peak, results that have contributed to a slide to number seven in the rankings.
The 17-time Grand Slam champion described the third-round defeat to Monfils as "a tough loss" but said the margin between success and failure was small.
Federer still believes he can compete at the sharp end of the men's game, saying he can turn his season around as he bids to qualify for next month's season-ending World Tour Finals in London.
"I can still finish strong. I believe that," he said. "There's not much time left. But if I do qualify for London, that gives me an extra shot there. I usually play well indoors. So I hope this year's going to be one of those years again."
"It's just important not to worry too much, to be honest. It's important to keep on doing what I'm doing," added the 32-year-old.
"Obviously I might get tougher draws as we move along with my ranking not being in the top four anymore. But that's OK. I don't really care that much about the rankings if I'm not world number one."
Monfils said despite his slide down the rankings, Federer still inspired fear in opponents. "When he's on the court, he's still Roger," said Monfils. "He can rip it from anywhere. He can do stuff that not many players can do."
"Maybe it helps a little bit because you know he has had a couple of losses," he added. "So it's less fear, but still, you know, you're not going, 'OK, I play Federer, it's an easy draw'. No, definitely not."
Playing his first tournament since the US Open, Federer started in the worst possible way, limply losing his first service game to immediately hand the initiative to Monfils, who went on to take the opener.
Monfils, clad in bright orange, put a straightforward forehand long when he had a chance to engineer two match points in the second set tie-break, allowing Federer back into the set.
But with both players committed to attack, Monfils grabbed the initiative in the third set, breaking to take a 3-1 lead and keeping his nerve to see out the match despite vocal support from a partisan crowd for Federer.
On a day of shocks in Shanghai, third seed David Ferrer lost 6-4, 6-3 to Germany's Florian Mayer and Tomas Berdych, seeded fourth, saw his campaign ended by Spain's Nicolas Almagro in three sets.
But top seed Novak Djokovic had few problems in disposing of Italy's Fabio Fognini 6-3, 6-3 and Rafael Nadal, chasing a record sixth Masters title in a single season, saw off Carlos Berlocq of Argentina 6-1, 7-6 (7/5)
Djokovic, who plays Federer's conqueror Monfils in the quarter-finals, said of the Frenchman: "I'm sure he was very motivated to beat Roger today. He succeeded in that. I don't think he's going to lack this kind of motivation for our match tomorrow."
Seventh seed Jo-Wilfried Tsonga brushed aside Kei Nishikori 7-6 (7/5), 6-0 and Juan Martin del Potro progressed with a walkover after German veteran Tommy Haas withdrew with a back injury.
Switzerland's eighth seed Stanislas Wawrinka beat Canada's Milos Raonic, the 10th seed, 7-6 (7/2), 6-4.