Sam Allardyce's future at West Ham United is likely to come under further scrutiny after his gamble on a youthful team selection backfired miserably against a rampant Nottingham Forest, who dumped them out of the FA Cup in some style.

With injury and suspension problems to contend with, Allardyce admitted to having one eye on West Ham's Capital One Cup semi-final first leg against Manchester City on Wednesday. It prompted him to field nine changes, giving debuts to Seb Lletget, Danny Whitehead and Callum Driver, with George Moncur making a maiden start.

"I don't think I had a choice," said Allardyce. "I make the decisions for the benefit of everybody at the football club. We go to Manchester City on Wednesday night. Manchester City played yesterday, we had to play today. Everybody changes their team in the Capital One Cup early on in the season but doesn't make a fuss about it. Many, many managers have changed the team in all divisions, but my changes had to be severe, I didn't want it to be but it had to be."

The Championship side's rout began with Djamel Abdoun's cheeky Panenka penalty, which was followed by Jamie Paterson's second-half hat-trick and Andy Reid's late finish. Their reward is a home tie against Ipswich Town or Preston North End.

West Ham's youngsters can hardly be blamed for the humbling result, but Allardyce may not be as lucky. Speculation over his future, however, should not detract from the heroics of Paterson, who won his side's spot-kick and then scored three fine goals of his own.

The opener came after 12 minutes when Paterson beat Moncur with some neat footwork before being tripped by the midfielder. Martin Atkinson pointed to the spot and Abdoun dinked the ball down the middle with a supreme confidence not usually associated with a player who had never previously scored for the club.

West Ham's most experienced players were stationed on either wing, with the captain for the day, Stewart Downing, left and Matt Jarvis right. The team's response to the goal was to look for the pair with every attack but the closest they came was through Ravel Morrison's 20-yard free-kick in the 23rd minute, which would have crept under the crossbar had Karl Darlow not made a fine save.

But Forest continued to press and West Ham's goalkeeper, Adrian, was tested first by Jamaal Lascelles' long-range shot and then by Greg Halford's downward header.

Forest started the second half at a fierce pace and Allardyce responded by throwing on more youth, handing debuts to Reece Burke and Blair Turgott and also sending on the callow Matthias Fanimo.

The Premier League side might have equalised before the hour mark had it not been for Darlow, who saved low from Lletget while unsighted and stood firm to Morrison's near-post drive.

That was as good as it got, though, with the game ended as a contest on 65 minutes when Halford held the ball up well before teeing up Paterson, who opened his body and found the bottom corner.

Paterson's second came just six minutes later. Henri Lansbury squared the ball for Paterson, who stroked home confidently with a first-time effort. By now, Forest's dominance was complete and with 11 minutes to go Paterson completed his treble when he raced into the area, shuffled into position as defenders stood off and deposited the ball neatly into the bottom corner.

There was one final insult when Reid lashed in Abdoun's lay-off in injury time, leaving West Ham thoroughly humbled and with plenty to ponder.

By contrast Forest's manager, Billy Davies, was in irrepressible form – announcing a new deal for the inspirational Reid, stating his intention to keep hold of the defender Jack Hobbs, on loan from Hull, and even challenging a journalist to a race.

Of his team's buoyant showing, he said: "I think someone was waiting to be hit for four or five here. Our play has been very good and we created a host of chances. I'm delighted with the way we played. I have to say we know Sam Allardyce has bigger fish to fry … we know he has injuries like ourselves. But before the game we were concerned, because in Downing and Jarvis, [Modibo] Maïga and Morrison they had £20m of talent in the front four."

For Paterson, whose goals were his first since his move from Walsall last summer, it was a day to remember. "I certainly won't forget that game. It's a good result for us, we've been doing well in the league and we took it into a cup game," said Paterson, who helped Forest extend their unbeaten run to eight matches.

This is an agency report because the Guardian remains banned from the City Ground on the orders of Nottingham Forest's owners and management