Bank of New York Mellon Corp. was unable to process client payment instructions sent over the Swift network for several hours earlier this week, a rare outage that caused some payments to fail and casts a spotlight on operational risks at one of Wall Street’s largest custody and clearing banks.

The incident comes as the bank has drawn scrutiny from supervisors at the Federal Reserve for its large market share as a payment and settlements provider.

The...