Swamy said a company allegedly floated by Sonia Gandhi and Rahul Gandhi had done a "sham and bogus" deal to "grab the Rs.1,600 crore worth Herald House" and other properties of the group and demanded a CBI probe.
The Congress was dismissive of the allegations and referred to Swamy as an "ajube" (strange character).
A much more strong denial came from Rahul Gandhi which said the allegations were "baseless and defamatory" and that legal action would be taken for the "scandalous abuse".
"The allegations made by you are utterly false, entirely baseless and defamatory," said a letter to Swamy. "We are committed to pursuing legal action."
Swamy told the media that the two Gandhis floated a company called Young Indian in which each had 38 percent stake.
Young Indian, he said, acquired Associated Journals Pvt Ltd (AJPL) and became owner of the National Herald and Quami Awaz and all high value real estate property in Delhi and Uttar Pradesh belonging to the group.
"Rahul Gandhi was already a shareholder in APJ Private Ltd in 2008 but failed to disclose this in his affidavit filed for the Lok Sabha elections in 2009," Swamy said.
Swamy said AJPL obtained unsecured zero interest loan of over Rs.90 crore from the All India Congress Committee (AICC).
He termed the transaction illegal, saying a political party can't give loans for commercial purposes.
He said in February 2011 AJPL was sold by transfer of shares to Young Indian. Swamy said Rahul Gandhi later transferred his 262,411 shares in AJPL to his sister Priyanka Gandhi Vadra.
The Janata Party leader said Young Indian filed statements with the Registrar of Companies disclosing that the shareholders meeting was held at 10, Janpath, a government house allocated to Sonia Gandhi.
"This is in violation of law since government accomodation cannot be used for commercial purposes and business," he said.
"The deal was to grab Herald House and other properties of the National Herald and Quami Awaz in Delhi and UP."
Swamy demanded an investigation by the Serious Fraud Investigation Office or the Central Bureau of Investigation.
He also urged the Election Commission to look into the illegality of loan given by the Congress to a private company.
The Congress poured scorn on Swamy.
"In every country and every society there are some strange characters," party general secretary Janardan Dwivedi said.
"The name you are taking (Swamy), he probably is one of them. He can say anything, anytime."