The wounded were evacuated and receiving medical attention, while the Cuban patrol commander was also wounded, Cuba's Interior Ministry said in a statement, adding the matter was under investigation to clarify exactly what happened.
The highly unusual incident on Wednesday, local time, took place amid tension between Cuba and the United States, which has blocked virtually all oil shipments to the island, increasing pressure on the Communist-run government.
American forces captured Venezuelan President Nicolas Maduro in Caracas on January 3, removing a key Cuban ally from power.
The speedboat came within 1.9 kilometres of a channel on Falcones Cay, on the north coast of Cuba about 200 kilometres east of Havana, when it was approached by five members of a Cuban border patrol unit, Cuba said.
The speedboat then opened fire, wounding the commander of the Cuban vessel, the statement said.
None of the dead or wounded aboard the encroaching vessel were identified, but Cuba said it was registered in Florida with the number FL7726SH.
The US State Department did not immediately respond to a Reuters request for comment.
"Faced with the current challenges, Cuba reaffirms its commitment to protecting its territorial waters, based on the principle that national defence is a fundamental pillar for the Cuban state in safeguarding its sovereignty and stability in the region," the Cuban statement said.
