Alexandre de Moraes fined the parties in Bolsonaro's coalition 22.9 million reais ($4.27 million) for "bad faith" litigation.
"The complete bad faith of the plaintiff's bizarre and illicit request ... was proven, both by the refusal to add to the initial petition and the total absence of any evidence of irregularities and the existence of a totally fraudulent narrative of the facts," de Moraes wrote in his decision.
He added that Bolsonaro's challenge the election results appeared to be a ploy to encourage more anti-democratic protests in the country.
The challenge came three weeks after left-wing political heavyweight Lula da Silva won the election over Bolsonaro by less than two percentage points — the narrowest margin seen since Brazil's return to democracy in 1985.
What was the basis of Bolsonaro's election challenge?
The outgoing president filed a 33-page formal appeal with the country's election authority, blaming the lost election on software bugs and demanding that votes cast on the majority of the nation's electronic machines be annulled.
The complaint characterized the bug as "irreparable non-compliance due to malfunction" and called into question the authenticity of the results.
The Liberal Party claimed that 280,000 of voting machines lacked individual identification numbers in internal logs. The complaint did not specify what this meant for the results.
However, lectoral experts told the Associated Press that the alleged software bugs would not have affected the reliability of the voting machines since they are other ways to identify each unit.
The electoral court also said in its ruling on Wednesday that "when one of these mechanisms stops working, others replace it, without affecting its traceability."
Lula set for inauguration
Brazil's election authority has already formally acknowledged da Silva as the victor. Many of the president's political allies have also accepted the election results.
"De Moraes' message to the political establishment is: the game is over," said Mauricio Santoro, a political science professor at the State University of Rio de Janeiro, told AP.
"Questioning the result of the elections is not fair play, and people and institutions who do that will be punished harshly."
Da Silva is set to take office on January 1, 2023.
mk, zc/rt (Reuters, dpa, AP)