Stefano Carosi tidies the glasses lining his coffee bar in Castel Gandolfo, a lakeside town southeast of Rome, awaiting his absent clientele.

“Of course it’s affected business,” he said, reflecting on Pope Francis’ decision to stay away from the town. Bar Carosi has stood in Castel Gandolfo’s main square since 1870, in the shadow of the Apostolic Palace, which popes have used as their summer residence for centuries.

That all changed with the papacy of Pope Francis, who for his third summer as pontiff has chosen to remain at the Holy See. His decision has been interpreted as part of the Pope’s broader wish to create a “poor church” and push through his ambitious reform agenda, rather than while away his time at the summer palace.

But his plans are having a damaging impact on Castel Gandolfo, which has thrived on its ties to the Vatican since the 17th century. “When John Paul II was here, he delivered the Angelus; the eyes of the world were on Castel Gandolfo,” Carosi remembers. “There were Swiss Guards outside at the open gate.”

When John Paul II died in 2005, his successor, Benedict XVI, continued the visits to the town and, after resigning in 2013, was flown by helicopter to the Apostolic Palace to stay for a few weeks. Benedict most recently visited Castel Gandolfo earlier this month (July), although the palace gates remained closed.

While the heat of late July often deters tourists from visiting Italian towns, the silence that hangs over Castel Gandolfo is, according to the locals, principally the result of Pope Francis’ absence.

Laura Fortini, working in an empty gelato shop on the town’s main strip, hopes the Pope will deliver Mass in Castel Gandolfo on the Aug. 15 national holiday. “Of course he’s busy, he has other priorities,” she said. “But on Ferragosto (Aug. 15) there’s been a Mass for centuries; he could come for that.”

Both Fortini and Carosi are cautiously optimistic that the town’s fortunes could change with the Holy Year of Mercy, due to open on Dec. 8, which was announced by Francis earlier this year. Although they note the fall in visitor numbers, others are more optimistic.

One restaurant worker, standing beside photos of past pontiffs, said a few tours passed by earlier in the morning, and noted that it’s normal for there to be fewer tourists when there are no papal audiences. If the Pope is in Rome, they’d rather go to Rome, she said.

But she said tourists still come to see the Villa Barberini; since last spring tourists have been able to visit the villa and its gardens, owned by the Vatican, although the palace itself remains private.

With its views over Lake Albano, narrow streets and a host of restaurants, Castel Gandolfo also has the potential to draw in the non-Catholic crowd. Elia Cagnoli, working in an empty souvenir shop, said the town is quieter but people still visit: “There are fewer pilgrims, more tourists.”

But the reliance on Catholic faithful is evident from Cagnoli’s shop, where a wealth of pope paraphernalia is on sale. The handful of other tourist shops have almost identical wares on sale, while there is also a shop dedicated solely to religious gifts.

The general air is one of respect for the papal household, but frustration at Francis’ choice to forget those at Castel Gandolfo.

“You can’t change tradition just like that,” said Carosi.