Immigration Minister Peter Dutton has apologised for joking about the plight of Pacific island nations facing rising seas from climate change.

Mr Dutton was overheard making a quip about “water lapping at your door” in Port Moresby to Prime Minister Tony Abbott on Friday, comments that offended Pacific island leaders.

“I made a mistake, I apologise to anyone who has taken offence. It was a light hearted discussion with the PM and I did not mean any offence to anyone,” Mr Dutton told Sky News.

• Bishop, Dutton gaffes caught
• PM denies axing ministers

Deputy Labor leader Tanya Plibersek said the minister’s apology didn’t go far enough.

Mr Dutton had managed to insult millions of people in one go and he should have apologised to Pacific leaders and indigenous people in Cape York, she told ABC TV on Sunday.

“The problem is that they (the government) don’t care that these Pacific Island nations are facing an existential threat,” she said.

Mr Dutton had also quipping that a late-starting meeting was running to “Cape York time”.

Mr Abbott replied: “We had a bit of that up in Port Moresby”, with Mr Dutton replying: “Time doesn’t mean anything when you’re about to have water lapping at your door.”

The president of Kiribati has called the joke “vulgar” and “quite unbecoming of leadership”, while Cape York traditional owner Gerhardt Pearson labelled Mr Dutton’s views “soft bigotry” that harked back to the mission days of the past century.

Despite the criticism, Mr Abbott said Mr Dutton should be remembered for something else he did this week.

“He masterminded the plan to bring 12,000 needy people to this country,” Mr Abbott said in Canberra.

Later on Saturday in Perth, where he was campaigning ahead of the Canning by-election next Saturday, Mr Abbott was again questioned about what he has called a “lame joke”.

He reiterated his comments about Mr Dutton’s work involving the refugees and, when asked if he thought it was funny, said “it is not about me”.

“We made a decision that reflects Australia at its best and then we have a subsequent Twitter storm which, if I may say so, reflects Australia at its worst,” he told reporters.

Mr Dutton has declined to comment further on the conversation, picked up on a boom mike that was eventually spotted by Social Services Minister Scott Morrison.