"The Jordanian Olympic Committee has decided to pull out the three athletes. They will arrive in Amman later today," a committee official told AFP on condition of anonymity.
The three men, released Wednesday on combined bail and sureties of £5,500 ($8,700, 6,970 euros) each, "will still attend their trial," he said.
The official added that "other members of the Jordanian team will continue to take part in the games."
Another committee official told AFP: "We reject any misconduct and we prefer that they do not take part."
The commitee's decision comes a day after it stressed that it "will not tolerate any misconduct. It has full confidence in British judiciary."
Two wheelchair-using power-lifters and their trainer appeared at Coleraine Magistrates' Court on Wednesday facing charges including sexual assault and voyeurism.
A senior Jordanian government official has denied reports that Jordan's King Abdullah II had personally intervened in the case, although he said the monarch was "concerned" by the allegations.
"If proven, these actions are condemned by Jordan, which fully backs the court. But at the same time, the kingdom supports Jordanian citizens inside the country and abroad," the official told AFP on Wednesday.
The men were arrested on Monday in the town of Antrim, northwest of Belfast, after three women lodged complaints.
Power lifter Omar Sami Qaradhi, 31, faces three charges of sexual assault, two against a child, and one of voyeurism after allegedly entering a women's changing room at the team's training centre at the Antrim Forum leisure centre.
Police said he was identified by a 14-year-old girl who claimed that she posed for photos with him before he groped her between the legs in Antrim town centre.
The same day, a girl aged 16 said she was walking along the river path beside the team's training base at Antrim Forum with a friend when their path was blocked by the three accused.
Police alleged that the girl was pushed by one of the men towards Omar, who placed his arm around her waist and tightened his grip, but then she ran away.
The three have to make weekly contact with the Jordanian embassy in London before their next appearance in court on October 18.