Boris Johnson’s hopes of sealing a Brexit deal before an EU summit on Thursday have been dealt a blow after Finland’s prime minister said he believed time had run out.

As British and EU officials negotiated late into Monday evening, Antti Rinne, whose country holds the EU’s rolling presidency, appeared to call time on the prime minister’s race to find an agreement.

“I think there is no time in a practical or legal way to find an agreement before the EU council meeting. We need more time,” Rinne told a news conference after talks with the incoming president of the European council, Charles Michel, in Helsinki.

The issue of customs checks on the island of Ireland and the details of a mechanism for providing democratic consent to arrangements for avoiding a hard border have dogged negotiations in Brussels.

On Sunday night, the EU’s chief negotiator, Michel Barnier, said there was still “a lot of work to do” after slow progress over the weekend.

EU sources suggested there was more cause for hope after talks between the negotiating teams on Monday, in a sign that the UK government might be preparing to make yet another concession. “We are cautiously optimistic,” said one diplomatic source.

But such is the complexity of the customs arrangements being discussed and the sensitivity of the issue of consent that EU officials have long doubted a legal text could be ready for leaders to approve when they meet at the summit. The EU has insisted the heads of state and government will not negotiate during their two-day meeting in Brussels.

In Luxembourg on Monday, Ireland’s deputy prime minister, Simon Coveney, appeared to concede that the talks could eat into next week, raising the chances of a Brexit extension being required.

He said: “I think it’s too early to say, even though we’re only a few days away from the summit. It’s too early to say whether it’s possible to get a breakthrough this week or whether it will move into next week.

“What they’re attempting to do here is to write a legal text for an international treaty – that is, a withdrawal agreement. That means it’s got to be watertight, it’s got to stand up to full scrutiny and legal challenge potentially, and what they’re trying to do is complicated.”

The EU has rejected the UK’s proposal of a dual customs system at Northern Ireland’s ports and airport that would involve tracking the destination of all goods entering from Great Britain and applying differential treatment depending on their final destination.

Barnier has pushed the UK to accept a model closer to a Northern Ireland-only backstop with Northern Ireland staying in the EU’s single market for goods. It would not be part of the EU’s customs territory but the bloc’s full customs code would be enforced in the Irish Sea, a move that risks provoking the Democratic Unionist party.

Coveney said: “They’re trying to put in place an arrangement that is Northern Ireland-specific, that protects UK interests and EU interests, in a way that’s quite complex. So we need to give negotiating teams space to do that. I think there is goodwill and a political determination to get this done. It is being left very late in the day but we shouldn’t write off this summit as a potential opportunity for agreement.”

Should Johnson fail to find an agreement by Saturday he will be forced by the Benn act to request an extension of the UK’s membership of the EU.

The issue of an extension was raised for the first time in months in a meeting between Barnier and ambassadors representing the EU27 on Sunday. Sources said there appeared to be two options: a short “technical extension” if a deal was on the cusp of being negotiated by the end of October or a longer prolongation of the UK’s membership to accommodate a general election or second referendum.