A new bank is suing the Federal Reserve Bank of New York, saying it is unfairly preventing the firm from pursuing a novel business strategy.

TNB USA Inc.—run by a former top New York Fed staffer—said its primary business activity will be to enable large institutional money-market investors to earn higher interest rates from the Federal Reserve than they could otherwise, according a complaint filed in federal court Friday. Such investors include pension funds, companies and other entities managing large sums of money.

But...