The 22-year-old off-spinner had Corey Anderson leg-before, Bradley-John Watling caught behind and Doug Bracewell caught in the slips off successive balls at the Z A Chowdhury Stadium.
Gazi’s hat-trick, which gave him his sixth wicket in the innings, saw the Black Caps slump from 260-4 to 260-7 in their second knock after lunch on the fifth and final day.
The tourists declared the innings closed at 287-7, leaving Bangladesh a target of 256 runs in a minimum of 45 overs to win the Test and take the lead in the two-match series.
The hosts showed no appetite to chase the stiff target and ended play on 173-3 in the 49th over, with Shakib Al Hasan unbeaten on 50 and newcomer Mominul Haque on 22. Gazi, who hit an unbeaten 101 batting at No 8 on Saturday, joined Alok Kapali as only the second Bangladesh bowler to claim a Test hat-trick.