HONG KONG—A panel of foreign experts recruited to ensure the objectivity of an investigation into allegations of excessive force by Hong Kong police during recent protests said it was stepping down after their concerns about the probe went unresolved, a step likely to fuel criticism of the review.

The panel said the body tasked with the probe lacks the powers and investigative capabilities “necessary…to begin to meet the standards citizens of Hong Kong would likely require of a police watchdog operating in a society that values...