Unionists have walked out on federal Labor's deputy leader Richard Marles during the Victorian party's annual conference.

It comes after construction union boss John Setka lashed out at Labor leader Anthony Albanese for not attending the meeting in Melbourne.

Mr Setka on Saturday accused the federal opposition leader of turning his back on working class Australians and tweeted he should "grow some balls".

Hours later, some delegates including from the Construction, Forestry, Maritime, Mining and Energy Union walked out on Mr Albanese's deputy as he prepared to address the event.