Raju was reacting to Bharatiya Janata Party (BJP) leader Mukhtar Abbas Naqvi's comment that a "love affairs ministry" should be created for Tharoor.
Gujarat Chief Minister Narendra Modi, at an election rally in Himachal Pradesh two days ago, described Sunanda Pushkar as a "Rs.50 crore girlfriend" of Tharoor.
Tharoor then retorted, "My wife is worth a lot more than your imaginary 50 crores. She is priceless. But you need to be able to love someone to understand that."
Wednesday morning, Naqvi took the fight forward.
"For an international love guru like him (Tharoor), a ministry of love affairs should be created," Naqvi commented.
HRD Minister Raju called the comment unfair.
"It is very unfair to comment on a person who is as experienced as Shashi Tharoor. He has vast experience, he has worked in the UN (United Nations)," Raju said.
Former under-secretary general of the United Nations, Tharoor was appointed minister of state for human resource development in Sunday's reshuffle.
Raju added, "We shall use that experience in our ministry."
Tharoor had to quit the government in 2010 over the IPL Kochi cricket controversy involving his then friend Sunanda Pushkar.
Photos: Akshay Kumar, Taapsee Pannu promote 'Naam Shabana'
Flashback: When sexy models sizzled in bikinis at Tokyo fashion show
Photos: These Indian cities top the list in online purchase of sex products
In pictures: Indian-origin artistes in Hollywood
Photos: Varun Dhawan and Alia Bhatt at studio in Bandra