Sunday's meeting came the day before EU foreign policy chief Federica Mogherini was due to present a plan to the UN about military operations against smugglers responsible.
Ministers from France, Germany Poland and Spain discussed how any UN-led plan - which could involved the destruction of boats used by traffickers - would work.
EU defense ministers agreed on sharing of intelligence about smugglers to prevent the flow of desperate migrants via Libya, but said a UN mandate would be necessary if effective action were to be taken.
"First we must exchange intelligence to locate the smugglers and their means of transport," French Defense Minister Jean-Yves Le Drian said after the meeting, which took place in the French city of Lorient."Then, we need coercive action and this cannot take place without a UN mandate which is being discussed."
Ban criticizes plan
The EU wants the UN Security Council to endorse the plan. However, UN Secretary-General Ban Ki-moon has criticized the plan,saying the destruction of boats could affect the livelihoods of Libyan fishermen.
Brussels is currently concentrating on halting unflagged vessels and sharing of information on smugglers gleaned by radar, aerial surveillance, and wiretaps. The aim would be to identify points of departure and set up a multinational team of planning officers.
Uncertainty looms over quotas
Italy and other southern states have been demanding help from other EU nations, with pressure for action intensifying last month when 800 migrants drowned as a ship sunk off the coast of Libya.
It is expected that the European Commission will on Wednesday propose a "mandatory migrant quota system" that would share responsibility for resettlement of migrants during times of crisis. However, it is far from certain that all 28 countries will agree, with Britain and Hungary particularly opposed to any such quootas.
European leaders who in April met in reponse to the tragedy agreed to step up rescue resources, without reaching any firm deal on the number of migrants who would be allowed to resettle within the bloc, or where they would be sent.
More than 5,000 refugees have died in the past 18 months as boats operated by smugglers capsized at sea between Libya and Italy.
Smugglers have adopted a variety of tactics, including using unpiloted larger vessels, as well as rubber lifeboats.
rc/bw (AFP, AP, dpa, Reuters)