Raju was reacting to Bharatiya Janata Party (BJP) leader Mukhtar Abbas Naqvi's comment that a "love affairs ministry" should be created for Tharoor.
Gujarat Chief Minister Narendra Modi, at an election rally in Himachal Pradesh two days ago, described Sunanda Pushkar as a "Rs.50 crore girlfriend" of Tharoor.
Tharoor then retorted, "My wife is worth a lot more than your imaginary 50 crores. She is priceless. But you need to be able to love someone to understand that."
Wednesday morning, Naqvi took the fight forward.
"For an international love guru like him (Tharoor), a ministry of love affairs should be created," Naqvi commented.
HRD Minister Raju called the comment unfair.
"It is very unfair to comment on a person who is as experienced as Shashi Tharoor. He has vast experience, he has worked in the UN (United Nations)," Raju said.
Former under-secretary general of the United Nations, Tharoor was appointed minister of state for human resource development in Sunday's reshuffle.
Raju added, "We shall use that experience in our ministry."
Tharoor had to quit the government in 2010 over the IPL Kochi cricket controversy involving his then friend Sunanda Pushkar.
'Rock On 2' trailer launch: Farhan, Shraddha enthrall audience with live gig
Flashback: When bikini babes set a new world record in Australia
Photos: 'Rock On 2' stars Farhan, Shraddha on 'Yaaron Ki Baraat'
Photos: Ajay Devgn launches 'Shivaay' comics at Mumbai Comic Con
Photos: WWE star Sheamus and John Abraham clash in Mumbai