Scott Morrison insists the Commonwealth government has only ever supported mandates that relate to aged care workers, disability workers and those who are working in high risk situations in the health system.
All other mandates have been imposed unilaterally by state governments.
Mr Pitt agrees and says while everyone is entitled to protest as their democratic right, in this case it was misdirected.
"These are the decisions of the premiers, it's not the federal government that is making these decisions, these are state based decisions," he told Sky News' Sunday Agenda program.
"They vary all over the country, There is so many decisions being taken, there is a lot of confusion around it."
But Deputy Opposition Leader in the Senate Kristina Keneally said it was the prime minister who set up the national cabinet and was prepared to take all the credit when the premiers took their measures.
"As soon as a group of protesters walks into town critical of some of those decisions, he points the fingers at the premiers, says it's not my fault, it's theirs," Senator Keneally told ABC's Insiders program.
"He should stand up and condemn the violent extremists who are part of that protest. And he should make clear that has no place in Australian democracy."
Meanwhile, NSW has reported 6686 new COVID-19 cases and 22 virus-related deaths, while in Victoria there were 7223 infections and 18 deaths.
In the ACT there were 458 cases, no deaths.
Like the ACT, which has ended QR code check-ins apart from a few exceptions, Victorian Premier Daniel Andrews said this is under review, as well as other rules.
"We hope to make some announcements quite soon," the premier told reporters on Sunday.
"Now that we've hit the peak of Omicron, now those cases are coming down, we do have some room to make some refinements and adjustments."
