The Education Department has adopted a broader definition of anti-Semitism that will allow it to more aggressively pursue civil-rights investigations into anti-Israel student organizations, according to a letter sent by a senior department official.

Kenneth Marcus, the department’s new head of civil-rights enforcement, laid out the shift in a letter reopening a civil rights complaint against Rutgers University last month. He said the Obama administration, which had closed the matter, ignored important evidence suggesting the school had created a hostile environment for Jewish students.

The move injects the Trump administration into a contentious debate playing out on college campuses over whether and when student activism against the Israeli government becomes a form of anti-Semitism.

Mr. Marcus—who founded the Louis E. Brandeis Center for Civil and Human Rights, which fights anti-Semitism—has been a leading voice arguing that forceful anti-Israel protests can amount to discrimination against Jewish students.

That position has raised concerns among some administration allies, who worry that challenging such activism is tantamount to prosecuting free speech. The administration has positioned itself as a champion of campus free speech, especially regarding conservative speakers targeted by liberal activists. Attorney >General Jeff Sessions said U.S. universities were becoming “a shelter for fragile egos” by squelching the speech of some.

Using opposition to Israel as a “definition to decide whether or not someone’s speech on a college campus amounts to discriminatory harassment is unacceptable,” said Joe Cohn, legislative and policy director at the Foundation for Individual Rights in Education, a nonpartisan group. “The definition includes obvious examples of core political speech, like criticisms of Israel,” he added.

Mr. Marcus’s letter signals a broader shift in the government’s approach to religious discrimination at schools and colleges. The Education Department does not have authority to challenge religious discrimination.

In a July interview with the Wall Street Journal, he said he recognized the agency’s limited authority, but “we’ll look to see what we can do to make sure that we are fully enforcing the law.”

Mr. Marcus contends that given their common background and descent, Jews—like Muslims—make up an ethnic group as well as a religion—and the department can pursue ethnic discrimination.

The original discrimination complaint against Rutgers, filed in 2011, concerned an incident in which a pro-Palestinian student group charged admission for an event only for Jewish students or others who backed the Israeli government.

The event, entitled “Never Again for Anyone,” featured several speakers including Holocaust survivors and a Palestinian activist. Palestine Legal, an organization backing the student organizers, said the students had initially requested donations from attendees, but began charging mandatory fees after a Jewish students’ group and local synagogues urged protests.

But one of the organizers, according to an OCR report, sent an email telling students manning the door of the event to admit supporters for free--resulting, critics said, in charging admission only to backers of the Israeli government, mostly Jews.

Palestine Legal also said pro-Israel protesters physically assaulted attendees at the event, calling them “towel heads” and “suicide bombers.”

In the letter to the Zionist Organization of America, the group that filed the complaint, Mr. Marcus said he was reopening the investigation not as a matter of religious discrimination, but as a case of possible ethnic bias.

He also said the Education Department had adopted a definition of anti-Semitic acts, in use by the State Department and subject to much debate, that includes “denying the Jewish people their right to self-determination, e.g., by claiming that the existence of a State of Israel is a racist endeavor.”

In effect, some activists said, the government is labeling the Palestinian cause anti-Semitic.

“Marcus is sending a clear signal that attacking free speech for Palestinian rights is at the top of his agenda at OCR,” said Dima Khalidi, director of Palestine Legal, referring to the Education Department’s Office of Civil Rights.

Liz Hill, a department spokeswoman, said the decision to reopen the case was driven by neglect on the part of the Obama administration, not a shift in policy.

“Secretary [Betsy] DeVos has made clear that OCR will look at the specific facts of each case and make determinations accordingly,” Ms. Hill said. “The facts in the Rutgers case, many of which were disregarded by the previous administration, are troubling.”

A spokeswoman for Rutgers, Dory Devlin, said in a statement that the university has yet to receive notification that the investigation has reopened“There is no place for anti-Semitism or any form of religious intolerance at Rutgers,” she said.

Mr. Marcus noted in his letter that opinions about Israel are not subject to civil rights law. But he added, “Discrimination on the basis of actual or perceived shared ancestry or ethnic characteristics — which may include discrimination against Jewish or Muslim students — is.”

Under his leadership at the Brandeis Center, Mr. Marcus urged universities to discipline students involved in the Boycott, Divest, Sanction movement against Israel, or BDS, and said the federal government should cut off funds to Middle East studies and other academic programs perceived to have an anti-Israel slant.