Opposition Leader Peter Dutton has hit out at sporting “elites” as more of the country’s big organisations come down in favour of the Indigenous Voice to parliament.

The AFL, Rugby Australia and Commonwealth Games Australia all announced on Thursday that they had decided to back the “yes” vote on the Voice.

They joined other sporting organisations, including the NRL, Football Australia, the Australian Olympic Committee and Tennis Australia, that had already said they supported a “yes” vote in this year’s referendum.

The growing support among the country’s sporting codes in a stark contrast to the federal opposition.

The Nationals are opposed to the Voice while, Mr Dutton declared last month that the Liberal Party also formally opposes it.

On Friday, Mr Dutton said it was “counter-productive” for sporting codes to advocate a position.

“Most of their fans are really scratching their head as to why the elites within sport, particularly the elites involved in the administration of the game, are taking a position into the Voice,” he said.

“Those who are watching a football match are happy to hear the arguments for and against and have the detail and understand what it is that they are being asked to vote for at the referendum later this year.

“They will make their own minds up, they don’t need to be told by the CEO of the organisation, they don’t need to be told by some multimillionaire living in a capital city in a mansion how they should be voting when they are struggling out in the suburbs.”

Mr Dutton cited research that showed Australians were feeling a “level of frustration” about the looming referendum – although he offered no further details about whose research it was.

“They’re not being trusted with the advice, the detail, to make a decision on their vote,” he said.

“Never in the country’s history has a prime minister said we’ll vote on the Saturday and we will work out on the Monday what you are voting for.”

• Noel Pearson launches ‘bedwetter’ spray at Voice campaigner

Mr Dutton reiterated his support for constitutional recognition for Indigenous people.

“I support a local and regional voice in legislation so that we can have Indigenous people, particularly those who are at the coal face, having a tangible say in which we can improve situations in Indigenous communities or in remote areas more generally,” he said.

“But enshrining the Voice in the constitution, as the Prime Minister is proposing and not explaining how it will work, I don’t think people understand the ramifications.

“I do think a lot of Australians are starting to work out all of this chat about the Prime Minister, and Labor seeing political advantage in the wedge against the Coalition, might be part of the Prime Minister’s interest in this topic.”

Former prime minister Tony Abbott also lashed out at Rugby Australia’s decision, accusing the organisation of succumbing to “moral blackmail”.

Tweet from @HonTonyAbbott

Prime Minister Anthony Albanese, however, backed the sports groups.

“Great to see the AFL and Rugby Australia come out in support of a Voice to Parliament. They’re joining Football Australia and the NRL. Recognising Aboriginal and Torres Strait Islander people in our Constitution will be a unifying moment for our nation,” he tweeted.

On Thursday, the AFL said it had decided to back a “yes” vote on the voice after many discussions and consultation with clubs, staff, players and its own Aboriginal and Torres Strait Islander advisory council.

“Aboriginal and Torres Strait Islander players have long made a stellar contribution on the footy field,” the statement said.

“Their off-field contributions are equally valued for the impact they have made on our game and our community.

“While we encourage everyone to seek the information they need to form their own views on the referendum, the AFL proudly supports the recognition of Aboriginal and Torres Strait Islander peoples in the Australian Constitution through the voice to parliament.”

Rugby Australia ran full-page newspaper advertisements on Thursday telling supporters that it was time for a “level playing field” for First Nations people.

Commonwealth Games Australia also announced on Thursday it supports the ‘yes’ campaign.

Announcing its decision earlier in May, the NRL said it had come from extensive consultation.

“True change comes through listening, learning and taking action and we encourage everyone in the rugby league community to get informed by the facts, and use their voice, so that we can move forward together,” it said.

-with AAP