"My reservations had no impact and the decision was not mine. My voice went unheard," Terzi told the Italian parliament, reported Xinhua.
"I waited to hand in my resignations here in this parliament to express publicly my position that I cannot be part of this government anymore," he said.
The "honour of Italy, of its armed forces and diplomacy" was a fundamental value, Terzi said, adding that he was supportive with the two marines and their families.
The minister's resignation came following strong criticism of the government's handling of the case of two Italian marines accused of killing the fishermen off India's southern coast in February last year.
Massimiliano Latorre and Salvatore Girone were sent back to India March 22, days after the caretaker cabinet of Prime Minister Mario Monti said they would not be sent back to India after being allowed to come home to vote in the national polls.
The U-turn was justified by a statement from the government saying it had decided to maintain the commitment that the marines should return to India by March 22 after obtaining a written assurance about "the protection of their fundamental rights".
Italy's decision not to return the couple had angered the Indian authorities who called the situation unacceptable and ruled that Italy's Ambassador to New Delhi Daniele Mancini could not leave India without permission.
The diplomatic row between the two countries, which traditionally have good relations, started in February 2012 as Latorre and Girone were charged with shooting dead the fishermen mistaken as pirates from a merchant vessel.
While Italy says the shooting took place in international waters, India insists the marines should be tried on its soil, where the accident happened.
Latorre and Girone, posted aboard oil tanker MV Enrica Lexie on security duty, Feb 15, 2012, opened fire at a fishing boat off the coast of Kerala, suspecting that the boat carried pirates. Two fishermen -- Ajesh Binki and Gelastine -- were killed in the firing.