Controversial anti-Voice campaigner Gary Johns has been condemned after declaring that Aboriginal people wanting a Voice to Parliament should “learn English”.

“If you want a Voice, learn English. That’s your Voice,” the former Keating government minister told an audience of conservatives Sunday at the CPAC conference.

The CPAC conference, which has Warren Mundine as its chairman, was centred on criticism of the upcoming Voice referendum.

Mr Mundine and Jacinta Price, leaders of the Fair Australia campaign from conservative political group Advance, were the headline speakers on the opening morning of the two-day conference, while other speakers included former prime minister Tony Abbott and former Liberal MP Bronwyn Bishop.

On the second day of the conference, CPAC Australia defended the comedian Rodney Marks, who referred to traditional owners as “violent black men” and called the 18th century Indigenous leader Bennelong a “woman-basher”.

In his speech on the second day of the conference, Mr Johns said: “If you’re not trying to get those people either out of that remote community or out of the stupor in which they live, or give them the tools to allow them adapt to life in the modern world, the world we inherited, then you’re doing wrong.

“Being practical is not the answer. You have to do practical things in the name of integration.”

Mr Johns then claimed to quote Ms Price’s father.

“As Dave Price, Jacinta’s dad, has said to me often enough: ‘If you want a voice, learn English. That’s your voice,’” he said.

Prime Minister Anthony Albanese on Monday said he was “concerned” by Mr Johns’ comments and his leading role in the no campaign against the voice.

NSW Liberal MP Matt Kean said Mr Johns was treating No campaign colleagues Warren Mundine and Jacinta Nampijinpa Price with “complete disrespect” and “cowardice”.

Greens Senator Sarah Hanson-Young condemned the comments as a “racist rant”.

“There can be no fence sitters in this vote. Either you think this nasty rubbish is OK or you don’t,” she said.

Mr Johns was formerly the federal Member for the Queensland seat of Petrie and served as Assistant Minister for Industrial Relations, Special Minister of State and Vice-President of the Executive Council in the Keating government during the 1990s.

He has since served as a senior fellow at right-wing think tank the Institute of Public Affairs, as a newspaper columnist and as commissioner of the Australian Charities and Not-for-profits Commission during the Turnbull government.