The opposition leader said he had a constructive second discussion with the referendum working group but the prime minister was still withholding key details and putting in a "half-hearted effort" to explain the change.
"I think the voice is not going to get up, it's not going to be successful," he told reporters on Friday.
Indigenous Australians Minister Linda Burney is pleading with Mr Dutton to "stay engaged" after revealing he asked her to leave the meeting.
"I'm very sure the momentum that's gathering behind the voice and the decency of the Australian public will see a successful referendum," Ms Burney said.
Labor frontbencher Jason Clare said Mr Dutton's commentary was not helpful and implored him to "not make the same mistake again" after apologising for boycotting the national apology to the Stolen Generations.
"This week Peter Dutton apologised for turning his back on Aboriginal Australians," Mr Clare said.
"Peter, you don't want to be looking back in 15 years time and apologising for doing the same thing."
The government is also turning to the United States on its pathway to reconciliation and efforts to boost Indigenous consultation, with Ms Burney meeting with US interior secretary Deb Haaland on Friday.
Both are the first Indigenous women to hold cabinet positions in their respective nations.
The minister and secretary discussed how governments could advance Indigenous self-determination and the role of First Nations people in managing national parks and water resources.
Ms Haaland's department oversees the US government's "nation-to-nation relationships" and treaty obligations with almost 600 sovereign tribes.
"We don't do anything without consulting with tribes first," Ms Haaland said.
"Whatever issue affects tribes, it's our job to make sure that tribal consultation comes first so we know exactly what they'd like for us to do."
She said the impacts of climate change, habitat loss and dying coral reefs could have been mitigated if early colonists valued the stewardship and environmental wisdom tribes cultivated over thousands of years.
Ms Burney is mulling over the words to be put to the public at a referendum to outline who the voice will consult and on what.
But she is waiting for the working group's recommendation before determining whether the proposed constitutional amendment should include the term "executive government".
Concerns have been raised about the referendum wording opening up possible High Court challenges as well as the removal of some words potentially impacting the scope and power of the voice.
Ms Burney said the decision would be informed by the group's recommendations, noting it was important any changes were informed by Australia's Indigenous communities.
"The importance of the working group and the engagement group ensures that First Nations people are at the decision-making table when we make decisions around questions and amendments," she said.
Legislation will be introduced in March and examined by a Senate committee.
A number of Australian land councils have thrown their support behind the voice, citing a long-term need for their constituents to be heard.
Northern Land Council chief executive Joe Martin-Jard, who heads one of the oldest representative Indigenous organisations in Australia, said a voice was needed to start a conversation about a treaty with First Nations people.
19°C