Exclusive Details

Daniel Tosh will pay dearly for insinuating Rob Dyrdek diddles 12-year-old boys -- in fact, sources close to the "Fantasy Factory" star tell TMZ, Dyrdek actually wants to FIGHT Tosh over a recent on-air pedophile jab.

In case you missed the sketch on "Tosh.0" -- the comedian dresses up like Dyrdek at a "Childish Dream Factory for Young Boys Only" ... filled with shirtless, 12-year-olds on Razor scooters ... and even plies his young guests with wine and drugs.

Play video content

But Rob -- who also hosts an extremely similar clip show on MTV called "Ridiculousness" -- isn't laughing ... responding to the sketch on Twitter, writing, "The idea of having beef with @danieltosh is truly embarrassing ... for real ... imagine that 'clip show host' beef ... but now that beef is very real."

Dyrdek said he wanted to settle things "the old-fashioned way" -- and sources close to MTV reality star insist he isn't joking ... Dyrdek seriously wants to throw down with Tosh. He is NOT happy.