The federal government is ramping up pressure on a leading digital-currency company to turn over a vast amount of its customer records as part of a tax-evasion probe, an effort that is raising privacy alarms.

The Internal Revenue Service is focusing on Coinbase Inc., a popular exchange and “wallet” service for bitcoin, which has surged in value over the past year as a kind of digital alternative to gold.

Some...