They are attractive targets for terrorist recruiting, and their continued presence threatens to drown Lebanon's identity, he said.
A permanent solution to the refugee crises throughout the Middle East requires lasting peace and the repatriation of refugees, not resettlement to third countries, he added.
Rai spoke June 27 at Catholic Near East Welfare Association headquarters in New York while in the United States for a pastoral visit.
"A political solution to the conflicts ought to be the top priority, and a just, global and permanent peace should be established as soon as possible," the Lebanese cardinal said.
"We would ask nations to help refugees where they are; but it's not enough to help, they should also stop wars, because every day we are at war, we're creating new refugees," Rai said. "We must find a just, global and lasting peace for refugees, repatriate them and help them rebuild their lives and businesses.
"The longer we delay the solutions, the more open we become to terrorism, because terrorists recruit among the refugees," Rai said. The refugees need money and the terrorists pay them, he explained.
The conflict between Sunni and Shia Muslims is the largest conflict in the region, with Saudi Arabia representing the Sunni faction and Iran the Shia, Rai said. The United States could play an important role mediating peace between the two countries, he said.
In Syria, reconciliation will only be accomplished when the foreign factions that entered the conflict reconcile their own differences, practice justice and recognize the humanity of those affected.
"The Israeli-Palestinian conflict is at the origin of the Middle Eastern problems" and could be solved through "the establishment of a Palestinian state alongside an Israeli state, the return of Palestinian refugees, and the withdrawal of Israeli troops from the occupied territories of Palestine, Syria and Lebanon," the patriarch said.
"You cannot really come to agreement or establish peace without justice," he said.
"The separation between religion and state for both Judaism and Islam is one of the basic conditions for a permanent political solution in the region," he said. When a country has an official religion, "you are in deep trouble because you are automatically excluding citizens who do not confess the religion of the state," he said.
Rai said, "Despite all the difficulties plaguing the Lebanese system," it could be a model for the workable separation of Church and state." It is the only country in the Middle East without either a state religion or requirement that the head of state must be a Muslim. Under an agreement forged in 1943, Christians and Muslims in Lebanon "live together on an equal basis" and share leadership of the government, he said.
There are 18 distinct religious groups in the country and all are recognized in the parliament. Under the agreement, the president must be a Maronite Christian, the prime minister a Sunni Muslim and the speaker of the parliament a Shia Muslim. Nonetheless, the Lebanese presidency has been vacant since May 2014.
Rai said Islamic nations cause great harm to Islam by acting as "silent observers, simply looking and doing nothing about the so-called Islamic State and all terrorist organizations." Sunnis and Shias should condemn attacks on Christian and ethnic and religious minorities, including the violation of their churches, homes and possessions, he said.
Christians and Muslims have lived together in the Middle East for 1,400 years. In recent years, most of the Christians in Iraq and Syria have been killed or displaced by international military, ethnic and racial conflicts in which they took no sides, but paid the heaviest price, Rai said. Despite the circumstances, Christians remain attached to their lands, and committed to Gospel values, Christian witness and co-operation with their compatriots, he said.
"The absence of Christians from the Middle East, or the weakening of their cultural influence, will certainly impoverish both Christians and Muslims, as it harms the culture of dialogue and co-existence," the patriarch said.
Rai said the international community bears significant responsibility for the conflict in Syria because some members give monetary, military political or personnel support to terrorist organizations who take the conflicts to neighbouring countries and Western nations, and make terrorism a global problem.
Rai is the leader of the Lebanon-based Maronite Catholic Church, the largest of six Eastern Catholic patriarchal churches. There are more than three million members worldwide, approximately 85,000 of whom live in the United States.
Lebanon is the size of the state of Connecticut. The country's four million inhabitants are host to 1.5 million Syrian refugees and 500,000 Palestinian refugees. The Palestinians began to arrive more than 30 years ago for what was anticipated to be a temporary stay.
The presence of so many refugees is destabilizing, according to Archbishop Paul Sayah, vicar general of the patriarchate. Suicide bombings in a Christian village in northeast Lebanon June 27 killed at least five people and are an example of what could happen more frequently if refugees are not repatriated, he said.
"I don't know how Lebanon is surviving. It's a miracle it's still functioning. The refugees are living in misery. No human should be subjected to such misery," he said.
"The problem is we are making refugees of the Lebanese," Rai said. Lebanese are leaving the country because of economic stress. He said Syrians will work for half the wages, their shops will sell goods for half the profit and they don't pay taxes. "Our cultural identity is threatened by the presence of the refugees," he said.
Rai said Catholic aid is crucial to the people of Lebanon. Citing Catholic Near East Welfare Association, Catholic Relief Services and Aid to the Church in Need, Rai said, "If they don't do it, it's not getting done. The vast majority of people being helped are non-Christian and the vast majority of the helpers are Christian."