Three people have died and a fourth is fighting for her life after a mass drowning at a treacherous unpatrolled beach on Victoria’s Phillip Island, south-east of Melbourne.

Distressing scenes unfolded near Forrest Caves beach, where a large group had been enjoying a day out on Wednesday afternoon.

Emergency services responded to reports of four people struggling in the water about 3.30pm.

Three women in their 20s and a 40-year-old man were pulled unresponsive from the water at the exposed beach known for horrendous rips.

First responders administered CPR on all four, but three of them died at the scene.

The sole survivor, a woman in her 20s, was flown to hospital, and was fighting for her life on Thursday morning.

Ambulance Victoria said she was taken to The Alfred hospital in a critical condition.

First responders in the carpark near Forrest Caves beach. Photo: 9NEWS screenshot

The Age reports that more than a dozen distraught family members and friends were huddled at the beach car park on Wednesday night as covered bodies were loaded into vans.

A middle-aged man in a white singlet was inconsolable, repeatedly yelling, “I should have stopped them” and cried out “I’m sorry, I’m sorry” while others tried to comfort him.

The family and friends of the victims were very upset when they arrived on the scene, a witness told the Seven Network.

“There seemed to be a group of about 10 people who would have come here for just a really nice day, and obviously it’s ended in tragedy,” Alex Tzatzimakis said.

“[They were] crying and comforting each other.”

The beach is on the ocean side of Phillip Island. Photo: 9NEWS screenshot

Phillip Island is a popular holiday destination about 90 minutes from Melbourne, famous for its penguins and the MotoGP.

The Visit Phillip Island tourism body said the caves were formed by erosion of the cliffs and were accessible only at low tide. At high tide, teh waves are promoted as “for experienced surfers only”.

Police will prepare a report for the coroner and are not treating the deaths as suspicious.

-with AAP