This is the second major accident in the state in less than a week.
Deputy Superintendent of Police Khajana Ram told IANS that all bodies have been recovered, most of them badly mutilated, from the wreckage of the bus.
The bus was on its way from Ucha Tikkar to Renuka-ji, a place famous for a Hindu shrine.
The accident took place near Jabrog village, about 70 km from district headquarter Nahan and 150 km from the state capital, in Sangrah subdivision.
Ram said 19 people died on the spot. Among the dead 18 were males.
Deputy Commissioner Vikas Labroo said the government has ordered a magisterial probe into the accident and has announced a relief of Rs.20,000 to the next of kin of the deceased.
The cause of the accident is yet to be ascertained, he added.
Most of the victims were locals and belonged to closely-knit families.
Official sources said bad condition of the road owing to recent heavy rains was mainly responsible for the accident.
Witnesses said the administration had a tough time in extricating the victims from the wreckage of the bus. Locals started the rescue operations well before the authorities could reach the spot.
Governor Urmila Singh and Chief Minister Virbhadra Singh have expressed grief over the deaths in the accident.
Sirmaur is one of remotest places in the state. Buses plying there have poor frequency, and the scarcity of means of transport often leads of overcrowding of vehicles.
This is the second major accident in the state in less than a week. Ten people, including four women, died when a private vehicle fell into a gorge in Lahaul-Spiti district Sep 21.
The victims, from Kishtwar area in Jammu and Kashmir, were on a pilgrimage.
Police records say over 800 people die every year in Himachal Pradesh in road accidents.
Police have identified 556 accident-prone spots, including 210 on national highways.