Opposition Leader Peter Dutton has doubled down on his demands for an apology after federal Labor ministers lashed him as a “paedophile protector”.

Dutton on Friday would not rule out legal action, even as Prime Minister Anthony Albanese refused to apologise on behalf of his ministers.

“I’ve got thick skin, I give as good as I can take. But I think that the Labor Party and the Prime Minister crossed the line this week,” Dutton told the Nine Network’s Today show.

“I haven’t got an apology yet. I would have thought, you know, if [Albanese is] man enough he’ll apologise. But I’m not expecting one, so that’s up to him.”

It follows allegations by Home Affairs Minister Clare O’Neil and Sports Minister Anika Wells that Dutton had voted in Parliament to protect paedophiles rather than children. The accusation also relates to a claim by the government that Dutton did not use discretion to deport a convicted pedophile while immigration minister, instead allowing him to reapply for a visa.

Asked if he would consider legal action, Dutton did not rule it out.

“I would wait to see whether there’s an apology forthcoming and then make a decision after that. But I think, in the end, what it is, is a distraction away from what’s been a debacle for this government,” he said.

Albanese was also grilled on the issue on Friday, telling Melbourne’s 3AW radio that Dutton “had a record”.

“He moved a motion yesterday in the parliament saying essentially that we wanted a range of … [people] including child sex offenders, rapists, everyone, that we wanted them out on the streets,” he said.

“I stand by the fact that my ministers have been doing everything they can to protect community safety. And Peter Dutton from day one has been focused on just politics.

“The fact is that Peter Dutton has made a decision consciously as he used to as a minister [to] stand up and accuse us consistently of not being concerned about community safety.”

The unsavoury spat revolves around debate on the fallout from the High Court’s ruling last month that indefinite immigration detention is unlawful, overturning a two-decade-old precedent that the justices found had given politicians powers that should have been reserved for the judiciary.

This prompted the release of about 140 detainees as the government rushed through legislation in an effort to contain and trace them.

Some newly freed immigrants were convicted of serious offences such as child abuse and murder. Others had faced charges for traffic offences.

All had served their entire sentences before being placed in unlawful immigration detention.

On Friday, Dutton claimed the government’s response would undo the work of Operation Sovereign Borders, a border protection campaign set up in 2013 after the Coalition won an election on a “stop the boats” platform.

It implemented a zero-tolerance approach to refugees and asylum seekers who had travelled thousands of kilometres on cramped and often dangerous fishing boats to reach Australian shores.

“This whole debacle is likely to unwind Operation Sovereign Borders because they’ve admitted to the people smugglers that they don’t have control of the system,” Dutton told Today.

“This is a huge, huge scandal.”

But Education Minister Jason Clare said Australia’s border policy remained consistent.

“If you try and get on a rickety boat and come to Australia, you’ll get turned back, or you’ll get sent to a country like Nauru, or you’ll get sent back to your country of origin,” he told the Seven Network’s Sunrise.

“It’s important to point out here: That was the policy under the former government, it’s the policy under us.”

Following the court decision, the government passed laws that would subject the newly released immigrants to daily curfews, require them to wear ankle tracking bracelets, and increase capacity to bring prosecutions if they breach visa conditions.

Politicians have also allocated $255 million to enforce these changes.

Legal experts and refugee advocates maintain the laws are an over-reach that penalise non-citizens, given that Australian citizens who’ve served jail sentences for the same crimes are generally released into the community without any restrictions.

According to Australian Border Force deputy commissioner Tim Fitzgerald, of the 145 immigrants released from indefinite detention, 142 required ankle monitors, and 140 have now been fitted.

The ABF’s October update on Operation Sovereign Borders revealed no new unauthorised maritime arrivals in that month, and 14 who had previously reached Australia were resettled in a third country.

-with AAP