The Kurdish-led Syrian Democratic Forces said several of its fighters have been killed and more than a dozen others wounded.
The clashes came as SDF chief commander Mazloum Abdi is said to be in Damascus to discuss a ceasefire deal reached on Sunday that ended days of deadly fighting during which government forces captured wide areas of northeast Syria from the SDF.
The SDF, the main US-backed force that fought IS in Syria, controls more than a dozen prisons in the northeast where about 9000 IS members have been held for years without trial.
Many of the detained extremists are believed to have carried out atrocities in Syria and Iraq after IS declared a caliphate in June 2014 over large parts of the two countries.
The army said in a statement that some of the Shaddadi Prison detainees in the town of Shaddadeh were able to flee amid the chaos and a curfew has been imposed because of the breakout, calling for information on those who escaped as search operations continue.
The army and the SDF traded accusations over the release of the detainees, with the group confirming in a statement it lost control over the prison, which is about 50km from the border with Iraq.
The Kurdish-led force also said nine of its members have been killed and 20 others wounded in fighting around another prison, al-Aqtan, northeast of the northern city of Raqqa.
An Associated Press reporter saw a US convoy entering the prison area, apparently to mediate between the two sides.
The United States has good relations with both.
After days of fighting with government forces, the autonomy-minded SDF, agreed on Sunday to withdraw from two Arab-majority provinces they had controlled for years, including oil fields.
Syrian government troops tightened their grip on Monday across a swathe of northern and eastern territory abruptly abandoned on Sunday by the SDF in a dramatic shift that consolidated President Ahmed al-Sharaa's rule.
The Syrian government had warned the SDF earlier on Monday not to use "cases of terrorism for political blackmail," saying it is ready to implement international law regarding the detainees.
"The government warns the SDF's command not to facilitate the fleeing of Daesh detainees or opening prisons as a revenge measure or for political pressure," read a government statement carried on state media.
The government used the term Daesh, an Arabic acronym for the Islamic State group.
IS was defeated in Iraq in 2017 and in Syria two years later but the group's sleeper cells still carry out deadly attacks in both countries.
with Reuters