Forty-one of the 47-member council backed a resolution urging an investigation by the Commission of Inquiry on Syria, set up by the council last year to gather evidence on alleged rights abuses there.
Russia, China and Cuba voted against the resolution which they said was "unbalanced." Ecuador and Uganda abstained while the Philippines was absent.
The resolution "requests the Commission of Inquiry to urgently conduct a comprehensive, independent and unfettered special inquiry, consistent with international standards, into the events in Houla, and if possible to publicly identify those who appear responsible for these atrocities."
It also asks that the probe "preserve the evidence of crimes for possible future criminal prosecutions or a future justice process."
The resolution was voted through at a special session of the council, the fourth to be held on Syria.
UN High Commissioner for Human Rights Navi Pillay said in a statement to the meeting that the Houla massacre could be a "crime against humanity" and repeated a call for events in Syria to be referred to the International Criminal Court in the Hague.
Last week's massacre sparked outrage across the world and prompted many Western nations to expel their Syrian diplomats in protest.