Police officials said negligent driving was the cause of the accident Wednesday, as the driver, who jumped off the bus just before the bus plunged into the river, was on cellphone.
The 42-seater private bus carrying over 70 passengers was going to Ani town from Kullu when it fell into the river near Jhari, some 20 km from here.
The accident spot, known for white water rafting, is 250 km from the state capital Shimla.
Rescue workers said rafters rescued 17 passengers trapped in the wreckage of the bus.
Eyewitnesses said the administration had a tough time in extricating the victims from the bus, although villagers and private rafters started the rescue operations before the authorities could reach the spot.
Meanwhile, the government has ordered a magisterial probe into the accident.
Kullu Deputy Commissioner Sharab Negi told IANS that search operation is on along the banks of the river as some of the passengers might have swept away.
Scarcity and low frequency of passenger buses in Himachal Pradesh often leads to overcrowding of transport vehicles.
Spotted: Virat Kohli, Anushka Sharma and other celebs at Mumbai airport
Photos: Salman Khan's 'special friend' Iulia Vantur at a song launch
Photos: Hrithik Roshan, Anushka Sharma, other celebs dazzle at event
In Pictures: Sex scandals involving Indian politicians
Photos: Abhishek Bachchan and Farah Khan at Boman Irani's birthday bash