Police officials said negligent driving was the cause of the accident Wednesday, as the driver, who jumped off the bus just before the bus plunged into the river, was on cellphone.
The 42-seater private bus carrying over 70 passengers was going to Ani town from Kullu when it fell into the river near Jhari, some 20 km from here.
The accident spot, known for white water rafting, is 250 km from the state capital Shimla.
Rescue workers said rafters rescued 17 passengers trapped in the wreckage of the bus.
Eyewitnesses said the administration had a tough time in extricating the victims from the bus, although villagers and private rafters started the rescue operations before the authorities could reach the spot.
Meanwhile, the government has ordered a magisterial probe into the accident.
Kullu Deputy Commissioner Sharab Negi told IANS that search operation is on along the banks of the river as some of the passengers might have swept away.
Scarcity and low frequency of passenger buses in Himachal Pradesh often leads to overcrowding of transport vehicles.