The move is the latest in a string of personnel actions targeting employees who worked on probes looking into President Donald Trump or his allies.
One of the fired agents, Jeremy Desor, has been targeted in recent days on social media after Senate Judiciary Committee Chairman Charles Grassley, a Republican, publicly released more than 1000 pages of subpoenas from Smith's investigation, code-named Arctic Frost, into attempts to keep Trump in power after he lost the 2020 election to his Democratic rival, Joe Biden.
The subpoenas did not redact any of the names of the FBI or Justice Department employees who were involved.
Another fired agent, Jamie Garman, was initially placed on administrative leave several weeks ago, shortly after Grassley released other records showing that Smith had sought limited "tolling data" from the mobile phones of eight Republican senators and one House of Representatives member in the days before and after the January 6, 2021, attack on the US Capitol.
The other two terminated agents were Blaire Toleman, who most recently worked in the bureau's Chicago office, and David Geist, who previously served as an assistant special agent in charge in the Washington Field office during the time of the investigation, three of the sources said.
Toleman previously led an elite public corruption squad that was shut down by the bureau earlier this year.
Reuters could not immediately reach the four agents for comment. A spokesman for the FBI declined to comment.
"The public has a right to know how the government's spending their hard-earned tax dollars, and if agents were engaged in wrongdoing they ought to be held accountable," Grassley said in a statement to Reuters. "Transparency brings accountability."
Criminal investigations often include reviews of phones' tolling data, which contains basic details such as the duration and general location of calls, but does not reveal the contents of the communications.
Senators have since accused the FBI, without evidence, of "spying" on them, prompting Smith's lawyers to respond with a letter seeking to "correct inaccurate assertions" being made by politicians whose tolling records were subpoenaed.
Grassley and other Trump allies have alleged the FBI's probe improperly targeted a wide range of Republicans, though Trump was the only one who was charged federally in the 2020 election case.
Since January, dozens of FBI agents, prosecutors and support personnel who worked on Smith's investigation or handled cases investigating individuals involved in the January 6 attack have been fired from the Justice Department.
