A Boston-based bitcoin startup became the first firm to receive a license, allowing it to offer digital-currency services in New York.

The New York Department of Financial Services issued it first “BitLicense” to Circle Internet Financial Ltd., a firm founded two years ago by Jeremy Allaire and backed by Goldman Sachs Group Inc.

The goal of...