Former Labor Government minister and power broker Graham Richardson has challenged Treasurer Joe Hockey to “put up or shut up over” claims he operated a Swiss bank account.

Mr Richardson, now a political commentator, invited Mr Hockey to appear with him on his show on Sky News next Wednesday.

“You turn up, you present your evidence and I’ll have a go back. Why don’t we do that – put up or shut up. That’s a pretty old Australian phrase,” he said.

• Joe Hockey forces sale of $39m Sydney home
• Ladies’ lunch: Abbott’s big win for women
• PM gives ground on defence pay

During question time on Wednesday, Mr Hockey said he had been able to sign an agreement with the Swiss government to ensure the Australian Taxation Office could locate Australians with Swiss bank accounts.

Mr Hockey speaking about Swiss bank accounts earlier on Wednesday in Question Time. Photo: AAP

“I bet there are a few people in the Labor Party shuddering at that thought,” he said.

“There is a former minister for communications who has a television program who might have had a Swiss bank account.

“But I will leave that to another day.”

Mr Richardson proceeded to pay out on Mr Hockey’s performance as treasurer.

“When it comes to getting criticism of you, I don’t have to go far,” he said.

“You are regarded amongst your own show as a joke and you ought to know it.”

Mr Richardson said most of the government’s problems related to the “rotten” 2014 budget.

“Now what are you doing? You are in full retreat. You are backing off everywhere, all over the shop you are going backwards. Why? Because you got it wrong in the first place,” he said.