This week Angelina Jolie expressed her hope that a summit hosted by the UK government in June will lead to increased prevention of sexual violence in armed conflict. Sexual violence is one of the many horrific facets of violent conflict and any steps taken to prevent it and to support the survivors are extremely important. However, there is a risk that narrowly focusing on sexual violence alone may not only fall short of the mark but also have perverse effects.

Over the past few years, the topic of sexual violence, especially in the context of violent conflict and war, has increasingly made it on to the international political agenda, thanks to the efforts of non-governmental organisations and international agencies, but also concerned national governments, including the UK. Rather than continuing to be seen as an "inevitable" consequence of armed conflict, sexual violence is increasingly being treated as a war crime and a crime against humanity. A range of UN security council resolutions have been passed to address the issue and a plethora of projects by different organisations, including my own, are working on the issue in conflict-affected regions. This is all, of course, a good thing – but it is often not enough.

Currently much of the focus is almost exclusively on sexual violence in the context of armed conflict against women and girls, who tend to form the largest group of victims. While this is an issue that deserves all the attention it is at long last receiving, other issues risk being overlooked. For instance, sexual violence is usually not the only kind of violence the victims are exposed to, and it does not only occur in the context of violent conflict. In addition to women and girls; men, boys and sexual minorities are also, though rarely acknowledged, direct and indirect victims of sexual violence. While talking about and dealing with sexual violence against women and girls is already a major taboo, that against men, boys and sexual minorities is even more so. Nonetheless, it is a common feature of violent conflict, from Central America to the Caucasus, from Congo to Cambodia. Again, it is not only a phenomenon limited to violent conflict. In spite of under-reporting, 9% of reported sexual violence in the UK and 15% of rapes in the US are against men and boys. Sexual harassment, exploitation and violence to the point of murder against lesbians, gays, bisexuals, transgender and intersex persons is depressingly common across the globe.

In addition to the narrow focus on one type of violence against one group of victims, the responses by outside actors tend to be mostly of a medical, psychosocial and legal nature. While this is extremely important, addressing sexual violence effectively requires more than short-term interventions by outside experts. Sexual violence needs to be seen in the context of the larger political, cultural, economic and social framework of gender inequality in which it occurs: it is not something that happens in a vacuum and the driving factors are not the same in all conflicts. Whereas, for example, in the concentration camps of the Bosnian war sexual violence was used systematically against women and men, in other contexts, such as in the Democratic Republic of Congo, it may be more due to a loss of military control and individual frustration rather than a planned strategy. Dealing with the issue requires a willingness to listen to what local understandings of gender and violence are, instead of assuming that these are already understood in the same way by all involved. As difficult as it will be, there is also a need to better understand driving factors of sexual violence, including group and individual motivations of perpetrators.

Finally, as also outlined in the recent book Sexual Violence as a Weapon of War? by Maria Eriksson Baaz and Maria Stern, and also confirmed by our field research, the exclusive focus on conflict-related sexual violence against women and girls means funding is often available for supporting only these victims, while others in need are turned away or expected to pay for medical services. This includes male and transgender victims of sexual violence; victims of sexual violence not directly perpetrated by conflict parties; as well as women and girls with gynaecological problems not caused by sexual violence. Given that the only way of accessing necessary medical services may be to claim being a victim of conflict-related sexual violence, this understandably can lead to misreporting to save one's life.

While the international interest and focus on conflict-related sexual violence is immensely important, it needs to be seen both in a broader context of gendered inequality and in the specific context of where it is occurring. And we should also not forget that sexual violence and gendered inequalities do not only need to be tackled in places such as Afghanistan, eastern DRC, South Sudan or Syria but also in our own western societies. Every day.

• The headline on this piece was amended on 3 April 2014 in order to clarify the writer's argument. It originally said: "Women are not the only victims of sexual violence". This has now been changed.