The opposition leader has bound his frontbench to the No case and while he supports constitutional recognition of Indigenous Australians he says the proposed voice will change Australia's democracy.
Mr Dutton has advanced an alternative model to create local and regional voices through legislation, saying that would more effectively advocate for Aboriginal communities.
"The Liberal Party model will limit the local and regional bodies to issues specific to improving lives and outcomes locally. It has no business in defence, RBA deliberations, energy and environment policy," Mr Dutton told The Weekend Australian.
He said his model would sit outside the Constitution and be better than giving a national body free rein to make representations on any issue affecting Aboriginal or Torres Strait Islanders.
Mr Dutton said the government's proposed voice would "require thousands of public servants and billions of dollars to run a new arm of the government to facilitate the consultation."
Former prime minister Tony Abbott, meanwhile, has called on Prime Minister Anthony Albanese to halt the referendum process and hold a constitutional convention.
He told The Weekend Australian the voice was a "Trojan horse" and a "power grab" that would create special political entities based on ancestry, akin to "Indigenous separatism."
"This assumption that Indigenous people are 'different' and need to be treated differently – this separatist mindset – is at the heart of the problem," he said.
Mr Albanese on Friday praised former Indigenous Australians minister Ken Wyatt as taking a principled stand in quitting the Liberal Party over its position on the voice.
"(Mr Wyatt) worked very hard for such a long period of time … for years (former prime minister) John Howard said we needed to recognise Aboriginal people in our constitution and it hasn't happened," Mr Albanese told Sydney radio station 2GB.
"Peter Dutton has underestimated how many Liberal-National Party voters will vote 'yes'."
The Guardian reported on Friday that three leading Liberal moderates – Simon Birmingham, Paul Fletcher and Marise Payne – spoke out in a shadow cabinet meeting on Wednesday against the party's plan to oppose the voice.
The trio reportedly argued in favour of a free say for all MPs but were voted down and will be bound to the party's position as members of the shadow cabinet.
Senator Birmingham refused to say on Friday whether he would actively campaign against the voice.
He said his approach would be "one of respecting the Australian people as they go about making their decision".
He added that former Liberal treasurer Peter Costello campaigned for a republic while then prime minister John Howard campaigned against it and a similar split could be replicated with the voice.
