We improve on all the above points in a new r‐package trackdem, which provides a method to obtain estimates of the number, individual body sizes as well as behavioural metrics from recordings of multiple organisms simultaneously. We implemented machine learning to filter out noise or to distinguish single species in multi‐species systems, based on visual characteristics. Here, we briefly describe the methodology and evaluate its performance on both simulated and live populations. We show that applying trackdem is flexible and straightforward, and that it provides accurate and reproducible results in a wide range of study systems without the need for expensive equipment.

A promising alternative to manual counting is automated image or video analysis; however, to date, these tools have only been used occasionally in ecological studies (Dell et al., 2014 ; Gerum, Richter, Fabry, & Zitterbart, 2016 ; Pennekamp, Schtickzelle, & Owen, 2015 ). Studies that do use video analysis often either (1) lack the generality and flexibility needed for easy application across systems or set‐ups; (2) are not freely available or platform‐independent; (3) are cumbersome to install or use; (4) are not accurate enough to base demographic studies on, or (5) fail to provide measures of accuracy and robustness (e.g. Auclerc, Libourel, Salmon, Bels, & Ponge, 2010 ; Bánszegi, Kosztolányi, Bakonyi, Szabo, & Dombos, 2014 ; Færøvig, Andersen, & Hessen, 2002 ; Hooper et al., 2006 ; Mallard, Bourlot, & Tully, 2013 ; Noldus, Spink, & Tegelenbosch, 2002 ; Pennekamp et al., 2015 ; Pérez‐Escudero, Vicente‐Page, Hinz, Arganda, & De Polavieja, 2014 ). These drawbacks need to be addressed, before ecology can harness the full power and potential of automated image and video analysis.

Many ecological studies rely on accurate measurements of the number, size distribution or behaviour of individuals in populations. Commonly, such metrics are obtained by manual counting and measuring, which can be time consuming and labour intensive, thereby limiting the scope of the studies. Moreover, manual measurements may be subjective, and are often not reproducible, as no records or materials are preserved. Improved measurement accuracy and data management are especially important when the reproducibility of science is being challenged (Collins & Tabak, 2014 ; Errington et al., 2014 ; Ioannidis, 2005 ) and scientists are ever increasing the scale and scope of data and research questions (Bolker et al., 2013 ; Petrovskii & Petrovskaya, 2012 ).

2 A BRIEF OVERVIEW OF THE METHODOLOGY OF trackdem

To apply trackdem to live populations, the first step is to record a short movie. Since trackdem is based on movement detection, it is crucial that particles of interest move. In most systems, a video of approximately 4 s will suffice, and the best results are obtained with a stable (contrasting) background and strong (background) illumination (Pennekamp & Schtickzelle, 2013). The implemented methodology consists of four steps: (1) image sequence creation, (2) particle identification, (3) particles tracking and (4) machine learning for detection optimization and noise filtering. We explain the main aspects of each step below, while detailed information can be found in Appendix S1. All functions are combined in an open‐source software package for “particle tracking and demography,” which is freely available as the r‐package trackdem on CRAN (Bruijning, Visser, Hallmann, & Jongejans, 2017; http://github.com/marjoleinbruijning/trackdem; https://CRAN.R-project.org/package=trackdem). An overview of the most relevant functions, for version 0.3.1, is given in Figure 1, and a tutorial can be found in Appendix S2.

Figure 1 Open in figure viewer PowerPoint Schematic overview of a typical workflow, including most relevant functions and output provided in r ‐package trackdem . After initial setup, the function runBatch() may automate the analysis of large amounts of video material

Since all functions run in r, installation is straightforward, and full transparency and user modification of all functions is enabled. A native r implementation is both convenient and flexible: users have access to the full range of statistical packages in r and users can skip steps depending on preference. Moreover, we optimized code to run fast and refactored key functions in C++ (sensu Visser et al., 2015), using Rcpp and the plugin RcppArmadillo (Eddelbuettel & Sanderson, 2014), showing that a native r solution is not necessarily slow, as suggested in Pennekamp and Schtickzelle (2013).

2.1 Creating and loading an image sequence in r The package uses Python code and the open‐source software libav (libav.org) to automatically convert recorded movies of study organisms into image sequences. These sequences are then stored in r as an array A with four dimensions (X and Y grid of the pixels, colour layers and time), making use of r‐package png (Urbanek, 2013).

2.2 Particle identification The particle identification starts with the detection of a “still background,” containing all motionless objects. As trackdem is developed to work with short movies, we use static background models, which are not continuously updated, and require a stable background where only focal particles move. Whenever backgrounds change, in for example longer videos or more complex field conditions, more advanced dynamic background modelling can be implemented (e.g. Barnich & Van Droogenbroeck, 2011; Bouwmans, 2014; Appendix S2.3). Three different methods for background detection are implemented natively in trackdem (Appendix S1.3). Movement is subsequently detected by subtracting each original frame from the still background, where large differences in colour values indicate movement. Here, pixels are defined to be background (0) or part of a moving particle (1) depending on a movement threshold T: its value is either automatically determined by minimizing within‐class variance in colour values or provided by the user (Appendix S1.3). Finally, individual particles are identified and labelled by merging all eight adjacent particle‐pixels, using connected component labelling (Chang, Chen, & Lu, 2004; package SDMTools by VanDerWal, Falconi, Januchowski, Shoo, & Storlie, 2014). If the minimum and maximum sizes of the study organism (in pixels) are known, a first filter can now also be applied to remove noise such as small air bubbles or large moving debris.

2.3 Tracking of particles We implemented a tracking algorithm similar to the one described in Sbalzarini and Koumoutsakos (2005) to identify and reconstruct the movement trajectories of all individual particles over time. In short, particles between frames are assumed to belong to the same individual if they are similar in size, speed, direction and location. Similarity is measured by a cost function, which describes the costs to link a particle i in frame k to a particle j in frame k + 1, and which is minimized across particles. In some instances, particles may disappear (e.g. due to overlap), or conversely appear (e.g. due to noise), which may result in unequal number of particles between frames. Whenever this occurs, the start or end of an individual trajectory is generated (more details in Appendix S1.4). This first step results in tracked segments where particles are linked between consecutive frames. When all existing particles are detected and there is no noise, this gives a complete and correct tracking record. However, in real life applications, this may not be the case, as particles are sometimes overlapping, not detected in some frames, or confounded with noise. Accuracy is then further improved by merging incomplete track segments. The same cost function is used to calculate the cost of linking the end of segment i in frame k to the start of a segment j in frame k + r. In an iterative process ranging r between 1 and maximum R, incomplete track segments across frames are linked. Particle tracking then provides a sequence of locations and sizes of identified particles, which can be used to calculate statistics of interest. Note that trackdem assumes that splitting and merging of particles (as explicitly incorporated in Jaqaman et al., 2008) does not occur, as such processes, akin to cell division, do not generally occur in ecological systems of short time spans of typical video material (seconds to minutes).

2.4 Noise reduction and detection optimization A particle tracking algorithm will overestimate actual numbers whenever movies contain identifiable moving particles which are not of interest (noise). Noise may include false positives, such as moving debris, or non‐focal particles, such as other species in the case of multi‐species systems. When such noise has not been filtered out in previous steps by thresholding and size filtering, it may be removed by implementing supervised machine learning. In trackdem, a set of functions allow users to interactively select true and false positives (using the package shiny; Chang, Cheng, Allaire, Xie, & McPherson, 2017), and then train an artificial neural network (using the package neuralnet; Fritsch & Guenther, 2016). Neural networks are algorithms that are used, here, to distinguish between objects of interest and noise, based on visual characteristics (e.g. shape or colour), and are described in detail in various textbooks (e.g. Bishop, 2006; Fielding, 1999). In trackdem, a neural network is optimized through cross‐validation, and the final trained neural network is then applied to the identified particles. After application of the neural network, noise is removed, and particle tracking is reapplied, now with increased precision of estimates (see Section 2.1; Figure 2c). Figure 2 Open in figure viewer PowerPoint Application of trackdem to simulated movies. In (a), the estimated number of particles is plotted against the simulated number of particles. Different colours show different particle sizes relative to the image resolution, from small (dark grey) to large (light grey) particle sizes. In (b), we show bias as a function of total particle area, simulating 100 particles per image sequence. In (c), the accuracy of trackdem was tested for different numbers of noise particles. Black points indicate estimates without machine learning, and red points with machine learning. Lines show moving averages based on generalized additive models. In (d), a Q–Q plot shows estimated body size distributions vs. simulated body size distributions. Ten independent image sequences were simulated, with 100 particles each having a randomly drawn size. In (e) and (f), simulated trajectories are compared with estimated trajectories, for 10 simulated particles