Larger groups often have a greater ability to solve cognitive tasks compared to smaller ones or lone individuals. This is well established in social insects, navigating flocks of birds, and in groups of prey collectively vigilant for predators. Research in social insects has convincingly shown that improved cognitive performance can arise from self-organised local interactions between individuals that integrates their contributions, often referred to as swarm intelligence. This emergent collective intelligence has gained in popularity and been directly applied to groups of other animals, including fish. Despite being a likely mechanism at least partially explaining group performance in vertebrates, I argue here that other possible explanations are rarely ruled out in empirical studies. Hence, evidence for self-organised collective (or ‘swarm’) intelligence in fish is not as strong as it would first appear. These other explanations, the ‘pool-of-competence’ and the greater cognitive ability of individuals when in larger groups, are also reviewed. Also discussed is why improved group performance in general may be less often observed in animals such as shoaling fish compared to social insects. This review intends to highlight the difficulties in exploring collective intelligence in animal groups, ideally leading to further empirical work to illuminate these issues.