Police in Sao Paulo said the workers at the Arena Corinthians stadium were crushed to death as they scrambled to meet a December 31 set by football's governing body FIFA to complete work at the venue.
A third worker was rushed to hospital with injuries after the accident, which brought a shocked response from FIFA president Sepp Blatter.
"I'm deeply saddened by the tragic death of workers @Corinthians arena today," Blatter said on Twitter. "Our heartfealt condolences are with the families."
FIFA, which has faced recent scrutiny over worker conditions at under-construction venues for the 2022 World Cup in Qatar, later said safety standards for workers were "a top priority."
Brazilian media said the accident happened when a crane lifting a 500-ton lattice structure to the top of the stands toppled over, bringing down three metal structures.
Firefighters, ambulances and a police helicopter were at the scene, which was cordoned off. The company overseeing the construction work immediately halted activities at the site and said it would investigate.
The stadium is one of 12 World Cup venues planned for the tournament, which has Brazil has been struggling to overhaul its sagging infrastructure in time for the finals.
In addition to the opening of the Cup, the arena will host five other games, inclding the semi-final, while the final will be played in Rio's iconic Maracana stadium next July 13. The reported $11 billion cost of staging the event has aroused public protests.
One site worker, Jose Mario da Silva, 48, told the G1 news portal that the accident happened as most workers were having a lunch break.
"I had walked underneath this structure to go and have lunch. if it had happened at a different moment, many more people could have died -- I could have died myself. I just heard a commotion in the distance."
Da Silva added that the accident happened as "a crane was lifting the last and heaviest piece of the stadium roof." FIFA's secretary general Jerome Valcke expressed his shock and sadness in a tweet.
"Extremely shocked by the news from Sao Paulo," Valcke said. "Our thoughts are with the families of the victims of this accident. We are currently awaiting further details from the authorities, who are investigating this tragic accident."
A FIFA statement added: "The safety of workers is the top priority for FIFA, the LOC and the federal government. We know the safety of all workers has always been paramount for all the construction companies contracted to build the 12 FIFA World Cup stadiums."
FIFA set a strict December 31 deadline for completing their 12 host venues amid fears that Brazil will struggle to be totally ready.
Earlier this month, engineering firm Odebrecht, which is in charge of the project, said the Corinthians arena was 94 percent complete, with the roof's steel structure already mounted and the pitch laid.
About 1,500 employees are currently working around the clock to ensure the 68,000-seat stadium is finished on time. Wednesday's accident is not the first in this huge nation of more than 200 million people, however.
Work was temporarily suspended in the southern city of Curitiba after a labor court ruled that there were various safety breaches and a "serious risk of workers being buried, run over and of collision," as well as being hit by construction material.
Two earlier tragedies struck preparations for the tournament -- in June 2012 when one constuction worker died in a fall at the new stadium in Brasilia. In March, another worker fell to his death at the Amazonia Arena in Manaus.