One of this year’s hottest trades is betting on stock volatility. It is also one of the riskiest.

Traders big and small have sought to profit from the market’s wild swings. They are doing so through derivatives and exchange-traded products that tend to profit when markets are shaky. Some of the investments promise heightened exposure to volatility and often gain in value when stocks fall and turbulence rises.

It...