Dozens of shipping containers were sent tumbling when carriages from a 1.7 kilometre-long train went off the tracks at Inverleigh early on Monday.
Sixteen of the carriages derailed, blocking the rail corridor between Adelaide and Melbourne.
No one was injured and the cause is under investigation.
About 30 workers started to remove the debris on Tuesday morning and initial assessments showed one kilometre of track damaged.
The Australian Rail Track Corporation said it was difficult to say when the line would reopen.
Heavy machinery has been delivered to the site and recovery work is expected to increase in coming days.
On Monday officials indicated it could take up to a week for operations to return to normal.
Heavy rain across the area didn't excuse what had happened, according to Rail, Tram and Bus Union National Secretary Mark Diamond.
"Rail workers are increasing concerned that Australia's rail safety regime is no longer fit for purpose and not enough is being done to ensure that tracks are safe," Mr Diamond said in a statement.
"While the cause of yesterday's derailment will be the subject of an investigation, rail workers are already aware of the failings in the rail safety system and they want to see action."
Mr Diamond said safety laws introduced 10 years ago need to be reviewed and the regulator needed greater powers to hold operators to account.
He said workers who report safety concerns cannot currently follow up to see whether they have been addressed.
"This undermines rail workers' trust in the system, as they do not know whether anything has been done about the safety issue," he added.
The Australian Transport Safety Bureau has taken control of the site.
Investigators spent much of Monday and Tuesday collecting evidence for further analysis.
13°C