To put this result into perspective, Arsène Wenger had never previously lost an FA Cup third-round tie in his 21 years as the longest-serving manager in the country. He has won this trophy seven times, the same as Chelsea and Liverpool throughout their entire histories, and plainly felt the gap to the Championship was substantial enough to risk putting out a weakened side. Nottingham Forest, with their songs about the European Cups and stands named after Brian Clough and Peter Taylor, will not appreciate it being called a giantkilling – but it still counts as a significant shock and a jarring experience for Wenger.

Along the way, there was a reminder here why the City Ground, with the volume turned up and the floodlights reflecting on the River Trent, is still one of the great places to watch football. They have not had much to celebrate here in the 19 years since dropping out of the Premier League but there were olés after Ben Brereton’s penalty had made it 3-1 and more jubilant scenes when Kieran Dowell, from another spot-kick, soothed the crowd’s nerves, just as it had seemed as through a goalkeeping mistake might set up Arsenal for a late feat of escapology.

Dowell’s penalty was laced with controversy, with Arsenal’s players protesting that he had slipped at the vital moment and inadvertently struck the ball twice. Yet it would have been a cop-out for Wenger to lay the blame on the officials and, for once, it was nice to hear him choose a different option. “Nottingham Forest were better than us,” he said. “They looked sharper than us, they had more chances and they deserved to win.”

The only disappointment for the Championship’s 14th-placed team came in the form of Joe Worrall’s late red card for a second bookable offence but it was rare to hear Wenger being so magnanimous and Gary Brazil, Forest’s caretaker manager, made the same point. “There wasn’t a point when I didn’t think we were going to win,” he said.

For Arsenal, the indignities were considerable on a day when Wenger had to watch from the directors’ enclosure because of the three-match touchline ban he has received for his previous criticisms of referees. This could conceivably have been Theo Walcott’s final match in Arsenal’s colours and, if so, it was a wretched way to go out. Forest do not even have a permanent manager in place after Mark Warburton’s sacking, the glory here going to the club’s academy director, and there will have to be questions about Wenger’s decision to gamble with such a youthful, experimental side.

Perhaps Wenger was emboldened by the fact that when these sides met in the EFL Cup last season he did the same and watched his team stroll to a 4-0 win. That, however, was against a team that ended up avoiding relegation to League One only by goal difference. It was the last game Fawaz al-Hasawi attended as Forest’s owner before leaving the club – in the words of the new chairman, Nicholas Randall – in “intensive care”.

Forest evidently still have a few problems bearing in mind Warburton was dismissed after back-to-back defeats against relegation-threatened Sunderland and managerless Sheffield Wednesday, leaving them with more defeats than any other club in the Championship bar Birmingham City in bottom place. Yet it quickly became apparent here that Forest, featuring five of Brazil’s academy graduates, had no intentions of hoisting a white flag for a second time.

Brereton, in particular, made it clear why, at the age of 18, he is so highly regarded by his club, having been restored to a central striking role rather than playing out of position on the wing – as he was, bewilderingly, under Warburton.

“That’s the Ben I know,” Brazil said. “He’s a nightmare for centre-halves.” Per Mertesacker, a player 15 years his senior, can testify to that while Dowell, on loan from Everton, also showed in flashes why he is one to keep an eye on for the future. Dowell has a left foot that will rekindle memories of Kevin Sheedy for Evertonians of a certain generation and it was his free-kick, struck with a brilliant mix of power and precision, that led to Lichaj heading in the opening goal.

When Mertesacker made it 1-1 three minutes later it would not have been a surprise to see Arsenal take charge. Forest’s defence has been their weakness all season and they were unable to clear their lines after Rob Holding had headed Walcott’s cross against the post, Mertesacker turning in the loose ball.

Unfortunately for Arsenal, this was an afternoon when their own defensive shortcomings were exposed. Lichaj’s second goal was spectacular but it also came from a weak attempt to clear the ball by Holding, whereas Alex Iwobi and Mohamed Elneny, the two closest players to Forest’s scorer, surely ought to have closed him down more quickly. Lichaj had time to chest the ball down, let fly with his right foot and fire in a screamer. They show a pre-match film here these days to relive some of the club’s famous goals, drawing heavily on the European Cup days under Clough, and this one was struck well enough to be worthy of inclusion.

Two-one behind at half-time, Arsenal had plenty of time to save themselves but Brereton’s penalty extended Forest’s lead after Holding impeded Matty Cash, one of several outstanding performers in Forest colours, and the home team were looking comfortable until that moment in the 79th minute when their goalkeeper, Jordan Smith, who otherwise had a splendid game, allowed the ball to slip out of his grasp and Danny Welbeck slid in Arsenal’s second goal.

A comeback would have flattered the holders and Dowell’s penalty, after a terrible pass by Mertesacker, a lovely run by Armand Traoré and a risky challenge from Mathieu Debuchy, left Wenger to reflect on what it was like to watch his side go out of the Cup without being allowed in the dugout. “Bad,” he said. “I hated it.”