Goals from Cristiano Ronaldo, Kaka and a Jose Callejon double in the Santiago Bernabeu sealed the Spanish champions' victory, but top spot in the group went to Borussia Dortmund who beat Manchester City 1-0.
Derk Boerrigter got the consolation for the Dutch who gave a good account of themselves and their reward is a place in the Europa League after finishing third ahead of Manchester City, who came a miserable last.
Afterwards Madrid coach Jose Mourinho was full of praise for the performance of ex-Tottenham player Luka Modric, who gave two wonderful assists for his side's first goals of the evening.
"Modric has already performed well for us in a number of games, he is a great player who can play at the highest level, nobody can deny he is going to be an important signing for the future of this club," he said.
Ajax coach Ronald de Boer was not as enthused with his side's performance but looked forward to playing the Europa League.
"I'm disappointed, Madrid have some fantastic players but we had eight or so playing below their normal level, I told them at half-time and they improved," he said.
"Overall we can be proud to come third in this group and we have a good chance in the next competition, we can reach the final."
Mourinho had earlier taken advantage of having already qualified to rest a number of first team players, mixing youth with experience, a tactic for which he has recently come under fire in the Spanish press for rarely doing.
Reserve goalkeeper Adan started and a Champions League debut was handed to full-back Nacho Fernandez.
Mourinho also introduced 17-year-old Jose Rodriguez and Alvaro Morata, a year his elder, late on while the experienced Ricardo Carvalho started his first Champions League game of the season.
Fabio Coentrao limping off early, with what looked like a hamstring injury, was the only negative on the night for Mourinho.
With a Europa League place still to fight for de Boer named the same side that began Saturday's encouraging 3-1 win over PSV Eindhoven.
However, the weakened Madrid side did not appear to be worried as they dominated possession and it was not long before the chances began arriving thick and fast.
Karim Benzema had the ball in the net as early as the seventh minute but the French striker was ruled offside.
By the time Cristiano Ronaldo hit Madrid's first of the night on 13 minutes, a post had already come between Fabio Coentrao and the opening goal.
Modric won the ball in his own half before finding Benzema with a long raking pass that allowed the Frenchman to check back and square the ball into the path of Ronaldo for an easy finish.
Kenneth Vermeer in the Ajax goal had to make good saves from Benzema and Ronaldo before Callejon doubled Real's lead on 28 minutes.
Modric was again involved, turning under pressure on halfway before digging out a fantastic pass for Callejon to race onto and finish with aplomb.
A Sami Khedira shot on 39 minutes had to be dealt with by Vermeer, and it was his performance that kept Madrid's lead down to two at the break.
For Ajax, 18-year-old Dane Viktor Fischer was lively all evening on the left wing and he brought the first save of the night from Adan just before the interval.
Kaka hit Madrid's third, exquisitely curling a left-footed shot home off the post from just outside the area just after the break.
Ajax then enjoyed a good period of pressure and much needed relief.
Fischer went close with a curling shot before Derk Boerrigter pounced on a punched clearance from Adan to grab a consolation for the Dutch on the hour mark.
Boerrigter forced a save from Adan with a free-kick five minutes later and Ajax went on to play bright attacking football for much of the second half.
Mourinho shuffled his pack and introduced youth and Morata responded with a great run on the right and cross for Callejon to head home his second and his side's fourth with two minutes remaining.