Mitrovic came off the bench to score the winner with a header after the Portugal defense failed to clear a cross into the area.
Ronaldo looked upset and complained with his teammates after the late goal that could keep him from making his fifth straight World Cup appearance.
Portugal will have to face the playoffs with the other nine group runners-up and two group winners from the UEFA Nations League. The 12 countries will play next March in three four-team brackets, with the seeded teams getting a home game in the bracket semifinals.
“It was terrible match for us,” Portugal midfielder Bernardo Silva said. “It’s hard to explain. We should have done a lot better. We have to apologize to our fans.”
The loss ended Portugal’s winning run in World Cup qualifiers at home dating back to 2013.
A draw would have been enough for the hosts, with Portugal trying to qualify for the World Cup for the sixth straight time.
Serbia, which had never beaten Portugal, finished first in Group A with 20 points, three more than second-place Portugal.
“We were the better team today in every aspect of the game,” said Mitrovic, whose penalty shootout miss against Scotland a year ago took Serbia out of the playoffs for the 2020 European Championship. “I think we deserved to win the game and to be in Qatar.”
Renato Sanches put Portugal ahead with a right-footed shot from close range only two minutes into the match at the Stadium of Light after Silva stole possession from a defender outside the area. Video review confirmed the goal despite Serbia claiming a foul by Silva.
Dusan Vlahovic nearly equalized for Serbia 10 minutes later with a shot that hit the post, and Dusan Tadic didn’t miss in the 33rd with a strike from outside the area that took a deflection from a defender and fooled goalkeeper Rui Patrício.
It was a well-placed cross by Tadic from the right side that allowed Mitrovic to score the winner with a header that bounced off the ground before going in past Patrício by the near post.
“It was a good cross from Tadic,” Mitrovic said. “We know each other very well. He knows where I am going to be, I know where he’ll put the ball.”
Portugal and Serbia entered the final round level on points after seven matches, but Portugal had the advantage of being able to qualify with a draw because of a better goal difference.
Serbia had rallied from two goals down to earn a 2-2 draw with Portugal in March.
Portugal has played in every World Cup since 2002 after failing to make it to the tournament from 1990-1998. It was eliminated by Uruguay in the round of 16 of the World Cup in Russia in 2018.