The prime minister is rejecting allegations he has not taken a firm stance on union boss John Setka, who has waged war on one of the nation's biggest sporting leagues.

The Victorian CFMEU secretary is demanding the AFL sack its head of umpiring, Stephen McBurney, warning his staff will not cooperate with the league to deliver construction projects on time and on budget.

Mr Setka's targeting of Mr McBurney flows from his former role as the head of the now-abolished building watchdog, the Australian Building and Construction Commission (ABCC).

The threats could result in cost blowouts and delays on major developments, such as the proposed Hobart football stadium and the Adelaide Crows' new headquarters.

The AFL is backing Mr McBurney, and the prime minister was asked whether Mr Setka should end his campaign against the league.

"Yes … it's just common sense," Anthony Albanese said in Adelaide.

He added he had never even heard of Mr McBurney, and was not sure what his role was, until the CFMEU boss had weighed in this week.

"[Staffing is] a matter for the AFL, frankly," Mr Albanese said.

On Wednesday, when asked about Mr Setka's comments, the prime minister said the union boss was no longer a member of the Labor party.

"I removed John Setka from the Labor Party many years ago as one of my first acts as leader," he said.

Shadow Employment Minister Michaelia Cash said Mr Albanese had failed to condemn Mr Setka's actions, and described his comments as "weak and insipid".

"What has booting Mr Setka out of the Australian Labor Party got to do with Mr Setka's actions today?" Senator Cash asked.

"[Labor] still accept their money, $4.3 million in the lead-up to the last election.

"They still delivered Mr Setka's number one demand, the abolition of the Australian Building and Construction Commission — one of the very first acts when they came to government."

On Wednesday, Mr Setka conceded it could be fair to describe his position as a result of a "grudge" against Mr McBurney, but said it was not a threat to the AFL.

"I don't want to make any threats. I'm only talking about exposing people," he said.

"People like McBurney can't attack workers and their conditions and be so undemocratic they make the North Korea look like a democracy, and then just ride off into the sunset like nothing's ever happened."

"The consequences will be that we are going to point him out to everyone, we're going to let everyone know who he is, what he did, what he'd done to workers."

The CFMEU state secretary insisted his members would still work on AFL projects.

"We're not going to stop or put bans on stadiums being built," Mr Setka said.

"Our members sometimes work Sundays, they work rostered days off, they work all sorts of hours, time that they could spend with their families, to deliver these projects ahead of schedule and under budget.

"And if the AFL are going to employ people like McBurney — well, why would our members want to go out of their way in facilitate that? They'll just work as normal, and that's it, and if it runs behind, that's bad luck."

Employment and Workplace Relations Minister Tony Burke cautioned Mr Setka against doing just that.

"The laws about unlawful industrial action have not changed, they have not changed, they are very specific and have been in Australia for a long time now about the limitations of industrial action being assigned to specific bargaining periods, not because you don't like someone who's running umpires," he said.

"Anyone making any threat to Australian sport, it's not the way to win over the Australian people.

"I find the whole thing very odd."

The AFL is standing by Mr McBurney.

"Steve McBurney umpired 401 games, including four AFL grand finals, he has been a long-time mentor to umpires at every level and has done an outstanding job since returning to the AFL to take up the role of Head of Officiating," the league said in a statement.

"All projects the AFL contributes to are designed to provide better training venues for AFL and AFLW players, gender-friendly facilities and change rooms at community football clubs and to create a more welcoming footy environment for families who make up many of the 1.2 million club members and the many more millions of fans who attend games

"We are hopeful any intended action does not impact players, supporters or the wider community who benefit from upgrades to local clubrooms and our stadiums."