A 76th-minute goal from James Maddison looked to have secured the three points for Brendan Rodgers' Leicester until a dramatic finale with the Dutch substitute turning the game around.
Bergwijn bagged an equaliser in the fifth minute of stoppage time after Hojberg floated a ball to Matt Doherty in the box and the Dutchman pounced to fire home the loose ball.
He then astonishingly grabbed the winner after Youri Tielemans gave the ball away and Harry Kane played a superb through ball to the Dutchman who rounded Kasper Schmeichel and slotted into the unguarded goal.
"It is fantastic. The fans had a fantastic day. I scored two goals. I am happy. The manager told me to score goals, make difficulties for the defenders and stay close to Harry," Bergwijn told the BBC.
"For me he is an important player with a special type of characteristics. No one else in the team has these characteristics," Spurs coach Antonio Conte said of Bergwijn."He is good technically, he beats the man, he is a striker and in good physical condition. It is an important choice for me -- I can make him in the starting XI or bring him on in the game."
"To concede the equaliser was disappointing to happen so late, but the winner was a very poor goal to concede. It is hard to not have taken anything from the game," Leicester coach Brendan Rodgers told BT Sport. "That was a great opportunity tonight, but you have to see that through.
"I don't think Tottenham won the game, we presented them the three points.
"It's bitterly disappointing but we have to get back on the training pitch and again for the weekend."
Kane went close twice in the early stages, including a header against the bar in the 19th minute, as Conte's side dominated.
The visitors got their eventual reward when they drew level in the 38th minute with a clinical finish from Kane, who beat the offside trap, skipped inside Caglar Soyuncu and drilled a low shot in off the far post.
Kane then wasted a glorious chance, firing high over the bar after being put through on goal by Lucas Moura.
It was a tighter affair after the break but Leicester got in front in the 76th minute when substitute Harvey Barnes found Maddison whose low shot struck the outstretched leg of Japhet Tanganga and flew in via the post.
That looked to be enough until substitute Bergwijn levelled before sparking wild celebrations with the winner in the seventh minute of stoppage time.
"What a rollercoaster of emotions that was," said Kane.
"Overall we deserved the victory. We were the better side. Disappointed to go 2-1 down but we were creating the chances. We never gave up and Stevey came on and made the difference. To get two goals like that -- these are games you will never forget in a career."