Frist-half goals from Thulani Serero and Danny Hoessen had the hosts well in command at the break.
However, Barca were back in the game minutes after the restart as Joel Veltman was sent off for bringing down Neymar inside the area and Xavi converted the resulting spot-kick.
Barca then pressed for an equaliser, but failed to breakdown a stubborn Ajax backline as they slipped to a first defeat in any competition this season.
AC Milan's 3-0 win away to Celtic in the other game in Group H means the Dutch side remain third, a point behind Milan, and will need to win in Italy when the two sides meet in two weeks' time to progress.
Despite injuries to a series of key players, including Lionel Messi, and having already qualified for the knockout stages, Barca boss Gerardo Martino still named a strong side.
However, the Catalans were forced back from the off by a young Ajax side full of energy and quality on the ball.
Jose Manuel Pinto got down well to his left to save Viktor Fischer's effort after 11 minutes, but there was nothing the stand-in 'keeper could do to prevent Serero opening the scoring eight minutes later as he turned home Ricardo Van Rhijn's low cross.
Barca should have been level six minutes later when Neymar was released through on goal by Pedro, but the Brazilian dragged his shot well off target.
Lasse Schone then forced Pinto into another smart save, but the Barca defence was breached for a second time four minutes before half-time.
Pinto again did well to parry Fischer's first effort, but Hoessen reacted quicker than the statuesque Gerard Pique and Carles Puyol to double the Dutch champions' advantage.
Barca were handed a lifeline just two minutes after the restart when Veltman brought down Neymar just inside the area and was sent off for denying a clear goalscoring opportunity.
Xavi took responsibility from the spot in Messi's absence and sent Jasper Cillessen the wrong way. Neymar should have levelled seven minutes later when he was played in by a wonderful pass from Xavi, but Cillessen just got enough on his lofted effort to turn it behind.
The scarcity of options on offer to Martino given Barca's current injury crisis was demonstrated as he introduced youngsters Patric, Sergi Roberto and Adama Traore from the bench as they searched for an equaliser.
But they failed to create another clear chance in front of goal as Martino tasted defeat for the first time in 21 games since taking charge in July.