As Rudd won 57 votes to 45 in a leadership ballot of the party's lawmakers, the change has come ahead of a general election due in September, which polls suggest Labor Party is set to lose, BBC reported.
The Wednesday's result was the latest twist in a long and bitter rivalry between the two politicians. But it could be the last as Gillard, the country's first woman prime minister, has decided to retire from politics.
"I will not re-contest the federal electorate... at the forthcoming election," said Gillard.
"What I am absolutely confident of is it will be easier for the next woman and the woman after that and the woman after that, and I'm proud of that," she added.
Gillard, has been sacked by her party just months before the next election and replaced by the man she ousted three years ago, The Guardian reported.
Rudd, in his victory speech, dedicated himself to winning the election in September.
"In 2007 the Australian people elected me to be their prime minister. That is a task that I resume today with humility, with honour and with an important sense of energy and purpose."
Paying tribute to Gillard, Rudd described her as "a woman of extraordinary intelligence, great strength, great energy".
"All of you across the nation would recognise those formidable attributes," he said. "Julia has achieved much under the difficult circumstances of minority government, helped by a dedicated set of ministers."
Rudd defeating Gillard in a ballot of MPs to become Labor party leader is only the second time a sitting Australian prime minister has been removed from their first term in office by their party; Rudd was the first.