Authorities from Colombia's disaster response unit said on Friday at least four bodies had been found, while the rest of the group of workers remained missing.
Workers already retrieved a number of corpses on Thursday, further dashing hopes of relatives waiting for news.
Recovered remains will be taken to the nearby city of Pereira for identification.
The unlicensed gold mine in the central Colombian town of Riosucio, around 220 kilometers (136 miles) northwest of the capital Bogota, collapsed on Wednesday.
Officials believe the accident was caused when a power failure meant pumps used to draw water from the nearby Cauca River stopped working.
Dozens of workers scrambled to escape the mine 17 meters (55 feet) underground as floodwaters rushed in.
One man told The Associated Press news agency most of the men couldn't swim and were drowned.
Around 80 emergency workers are working at the site to recover victims, including many from the Colombian Red Cross and the armed forces.
The mine was in the middle of the formalization process, which would mean illegal miners could obtain licenses to work through the government.
It's estimated there are around 3,000 other semi-illegal mining projects also seeking government authorization.
In a statement from the country's National Mining Agency said the miners did not have contracts with the company.
Local media is reporting one of the gold mine's owners said the process had been started two years ago, but as yet authorities had not inspected the site.
Colombia is one of the world's largest producers of gold, an industry which is largely fuelled by illegal mines.
Despite the high accident rate, entrenched poverty has driven many locals to work in the dangerous digs, with the price of gold skyrocketing from less than $400 (350 euros) an ounce to almost $1,200 (1,050 euros).
an/kms (AFP, AP ,Reuters)