At 276-3 the Black Caps were moving serenely towards the close in Chittagong but when the bails were flicked they had been reduced to 280-5. Centurion Kane Williamson was the first to go, departing for 114, before captain Brendon McCullum was dismissed with what proved to be the final ball of the day.
It was a messy end to the day for the tourists, although before that Williamson had batted brilliantly for his fourth Test hundred.
The Yorkshire batsman needed 175 balls to reach the milestone, but had to depart before the end when he was pinned leg before by Shakib Al Hasan.
McCullum quickly followed him, playing across the line to Abdur Razzak (two for 100) and leaving his teammates with a bit of work to do today.
Earlier in the day Williamson and Peter Fulton had shared a 126-run stand for the second wicket to set New Zealand up.
Brief scores: New Zealand 280-5 (Kane Williamson 114, Peter Fulton 73; Abdur Razzak 2-100) vs Bangladesh
Photos: Swara Bhaskar, Sonali Kulkarni at Jagran Film Festival awards
Pics: Shahid Kapoor's transformation from chocolate boy to hottie
Photos: Kareena Kapoor Khan and Aamir Khan at Mumbai airport
Photos: SRK, wife Gauri, Aishwarya Rai, Shweta Bachchan at Vogue Awards
Photos: Harbhajan Singh-Geeta Basra's TV outing with daughter Hinaya