The protesters were concerned about the amount of pollution the mine would create. Serbia is already one of the most polluted countries in Europe and would have to spend billions to reach targets set by the EU for potential admission to the bloc.
Rio Tinto bought up land in the Loznica region after discovering lithium deposits in 2004. The company had planned to invest $2.4 billion (€2.12 billion) in the project. Lithium is a key component in modern technologies, such as electric cars.
"Everything about the Jadar project is finished," Brnabic said on Thursday, putting an end to the plan.
But activists vowed to continue their protests until the government bans lithium — as well as borate — mining altogether.
"Just one more step! We're close," protest leader Savo Manjlovic tweeted after the prime minister's announcement.
Government looks to upcoming elections
The government's change of heart comes ahead of elections in April. The ruling Serbian Progressive Party (SNS) had initially supported the development of mines in Serbia, but the party has seen its support decline.