Rubio's convoy drove through the central Roman boulevard leading to the Vatican under tight security, arriving on Thursday morning for the first visit between the Pope and a Trump cabinet official in almost a year.
The closed-door meeting between Leo and Rubio, who also serves as Trump's national security adviser, is expected to last half an hour.
Rubio will meet afterwards with the Vatican's top diplomat, Italian Cardinal Pietro Parolin.
Leo, the first US pope, drew Trump's ire after becoming a firm critic of the US-Israeli war on Iran and the Trump administration's hardline anti-immigration policies.
The president has kept up an unprecedented series of public attacks on the Pope in recent weeks, drawing a backlash from Christian leaders across the political spectrum.
On Monday, Trump falsely suggested the Pope believed it was OK for Iran to obtain nuclear weapons and said Leo was "endangering a lot of Catholics" by opposing the war.
Leo told journalists after the latest attack that he was spreading the Christian message of peace.
The Pope also firmly rejected the idea that he supported nuclear weapons, which the Catholic Church teaches are immoral.
"The mission of the Church is to preach the Gospel, to preach peace," the Pope said.
"The church has spoken out for years against all nuclear arms, on that there is no doubt."
Leo, who on Friday marks his first year leading the 1.4-billion-member church, has grown more outspoken on the world stage in recent weeks.
During a four-nation African tour in April he forcefully decried the direction of global leadership and said the world was "being ravaged by a handful of tyrants", in comments he later said were not aimed directly at Trump.
Rubio is Catholic, as is Vice-President JD Vance.
The two met Leo a year ago after attending the Pope's inaugural mass.
Rubio said at a White House briefing on Tuesday that he expected to discuss Cuba and concerns over religious freedom around the world with Leo.
He arrived in Rome on Thursday morning without any press accompanying him on his plane, which is unusual for a US secretary of state.
The US ambassador to the Holy See, Brian Burch, told journalists earlier on Tuesday that the conversation between the pope and cabinet official was likely to be "frank".
Rubio is visiting Rome for two days, and is due to meet Italian Prime Minister Giorgia Meloni, who has defended the Pope from Trump, on Friday.
