One official is quoted as saying 63 migrants have disappeared after Sunday’s incident east of Tripoli, while 41 people have been rescued.
About 170 migrants have gone missing in the Mediterranean Sea since Friday.
Smugglers are exploiting migrants desperate to get to the EU before it tightens its borders, the International Organization for Migration (IOM) says.
“There is an alarming increase in deaths at sea off Libya’s coast,” IOM’s Libya Chief of Mission Othman Belbeisi said.
More than 1,000 people have died while sailing from Libya to Europe so far this year.
- Smugglers ‘shot escaping migrants’ in Libya
- Who is responsible for helping migrants at sea?
- Is Italy taking in thousands of migrants?
- Migrant crisis: Just a mess or cynical politics?
What does Libya say about Sunday’s disaster?
The boat capsized off Garabouilli, about 50km (31 miles) east of the capital, Tripoli.
Citing eyewitnesses, Gen Ayoub Kacem, a spokesman for the Libyan Navy, told the AFP news agency that 63 people were missing.
He said 41 would-be migrants, who were wearing life jackets, were saved.
How is Europe tackling the problem?
Last week, EU leaders clinched a new deal on migrants after lengthy talks in Brussels.
They agreed controlled centres would be set up by EU states on a voluntary basis and migrants whose claims were rejected would be “returned”.
Refugees could be resettled in EU states which agreed to take them.
The deal follows weeks of diplomatic wrangling over migrant rescue ships, and which country should take them in.