Working in science is fulfilling, but can also be frustrating at times. As early-career researchers (ECRs), we get frustrated when we try to download an interesting paper, but hit a paywall; when we plan to use an existing paradigm, but cannot exactly reproduce it as the methods information is incomplete and the materials and code are unavailable; when we want to try out a new analysis on a data set, but cannot get access to it; and when our paper gets rejected, because the results are not statistically significant and are thus deemed unpublishable. Incidents like these made us reflect on how science should be done and why we want to be scientists. In this paper, we will share our thoughts and answers on these questions that we found within the practices and philosophy of Open Science.

Since October 2018, we are working together in the Control of Impulsive Action (Ctrl-ImpAct) lab, led by Frederick Verbruggen at Ghent University (https://fredvbrug.github.io). Our principal investigator (PI) Frederick Verbruggen has been practicing Open Science for some years now, which is why interest and/or experience in Open Science is considered a prerequisite for joining the lab. As a lab, we are all convinced that we should base our research routine on the principles of Open Science, “the practice of science in such a way that others can collaborate and contribute, where research data, lab notes and other research processes are freely available, under terms that enable reuse, redistribution and reproduction of the research and its underlying data and methods” (Foster, n.d.).

Open Science applies to all research disciplines. Here, we will address the topic from the perspective of a cognitive psychology and neuroscience lab. More specifically, the present paper is our lab’s mission statement regarding Open Science. The paper consists of three parts. First, we as ECRs would like to contribute to the discussion on Open Science by outlining our personal motivation for Open Science; second, we will describe how we incorporate Open Science practices into our scientific workflow; and third, we will discuss potential solutions for general problems that could arise when adopting Open Science practices. This mission statement is our lab’s public commitment to Open Science. We hope it will also inspire and help other labs and/or researchers to base their research routine on the principles of Open Science, by providing a concrete example of how they can incorporate Open Science practices into their own scientific workflow.

We are aware that many paths lead to a common goal, and that the Open Science practices we adopt may not necessarily suit everyone. Furthermore, with the emergence of new ideas and technologies, good research practices of today may be deemed inappropriate tomorrow, and better ways of conducting research may appear. Our mission statement thus reflects what seems most useful to us at the present time, and we are aware that for other researchers or in other times, ideas about what good research practices entail may change. We will therefore continuously reflect on how we do research and how we can further improve our workflow. The specific guidelines that we develop in the lab will be shared on GitHub (see links in the Appendix), and we invite other researchers to participate by for instance providing feedback and sharing experiences. With this paper, we thus also wish to reach out to other researchers and start an ongoing exchange on how we want to do research as a community.

Our personal motivation to follow Open Science To start thinking about why we as ECRs embrace Open Science, we first need to rethink what science is, why we as ECRs want to pursue an academic career, and what we consider as the output of scientific research. For many, the output of one’s research may be the publication that summarizes the findings. However, as John Claerbout, an earth scientist at Stanford, put it, “an article about a computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data, that produced the result.” (as quoted in Donoho, 2010, p. 385). In psychology research, one may similarly argue that journal articles are just part of, but not the whole scholarship. The whole scholarship lies not only in the motivating research question and theoretical background (which is usually part of journal articles), but also in study materials and experimental scripts, collected data, and data analysis scripts used to produce the results. In other words, all products generated in a research project should be considered research output. This view dovetails well with what distinguishes scientific findings from mere opinions and hearsay. In many fields, including our own, science is based on empirical testing and the reproducibility of empirical findings, rather than mere trust between individual scientists (Klein et al., 2018). When a finding is shared, the scientific community needs to carefully assess and critically evaluate the accuracy of the finding, independently replicating the results multiple times, before it becomes accepted as part of scientific knowledge. However, when the study materials, data, experimental and analysis scripts supporting the reported finding are not available, it becomes difficult or sometimes even impossible for scientists to verify it. Such unverifiable findings cannot be considered as scientific findings. Open Science, in our opinion, is thus just what science entails, whereas siloed science is not science. In addition to our strong conviction that Open Science is just science done right (Imming & Tennant, 2018), following the principles of Open Science also confers multiple benefits (compared to science practiced in a closed manner). Transparency is at the heart of Open Science, and we believe that transparent workflows increase the credibility of the research. When products of all stages of research (i.e., study plans, raw data, experimental and analysis scripts, study materials, publications) are available, researchers enable their colleagues and anyone who is interested to completely retrace their work and build on it. The possibility to examine all steps that precede a publication will likely increase the credibility of the findings, because all relevant information is available for close scrutiny, not only the summarized version in form of publication. Such rich information would enhance the understanding of each other’s work and inspire new ideas (Colavizza, Hrynaszkiewicz, Staden, Whitaker, & McGillivray, 2019). Moreover, by making study materials and experimental scripts open, researchers can more easily replicate others’ work. As argued above, we consider replicability and thus replication of empirical findings an essential part of science. By increasing the reputation of replication studies and making replication studies easier to conduct, Open Science would motivate more researchers to invest time and resources into such investigations, which may eventually increase the overall credibility of the psychological literature. From a pragmatic point of view, we are convinced that Open Science helps avoid effort duplication and allows ECRs like us to get a smoother start into new research projects. Having access to materials of previous experiments can provide valuable opportunities for us to improve our research skills, especially at the beginning of new projects. Programming new experiments by using functioning scripts saves time and could reveal important design choices that we would miss otherwise. Especially when programming skills are still low, improving these skills is easier when starting with working examples than from scratch. This may especially be the case for ECRs who cannot fall back on a collection of scripts and study materials yet. In addition, having access to data and data analysis scripts of the studies we want to build on gives us the chance to reanalyze the data or try new analyses before building our own experiments on them (Leonelli, 2018). Apart from what in our opinion are good scientific practices, we would like to emphasize that Open Science offers researchers much more. Open Science can foster creativity. In today’s publication culture, we notice that many researchers – including ourselves – seem to have a strong focus on positive findings (i.e., statistically significant findings) when designing new studies. For example, some researchers seem to make tidy but only small variations to familiar questions and established paradigms, instead of addressing new open questions and designing new paradigms. Thus, these researchers seem to follow an approach that is more likely to show positive results. This mindset is understandable, given a culture where a publication is seen as the sole research output and where positive results are deemed more publishable than null results. In our opinion, this mindset is restricting and limits researchers’ thinking in a way that hinders the advance of science. Open Science has the potential to liberate researchers from this restrictive mindset in at least two ways. First, when one thinks of all products from a scientific project as research output, the fixation on the final results and the publication will necessarily decrease. Carefully created study materials, novel experimental tasks, and sophisticated data analysis methods are all valid and useful output from one’s research, regardless of whether the final result is positive or not. By thinking of and evaluating research output in such a broad sense, researchers will not need to forsake creativity for positive findings. Doing research may also become more fulfilling and rewarding, as each line of code one writes, and each data set one collects may become small building blocks that eventually make up the edifice of solid science. Second, certain Open Science practices will help researchers to adopt a creative mindset when thinking about new studies (Allen & Mehler, 2019; Frankenhuis & Nettle, 2018). This is mainly accomplished by pre-registering studies, as pre-registration allows for evaluating study quality instead of the significance of the findings (Nosek, Ebersole, DeHaven, & Mellor, 2018). Researchers can pre-register studies by themselves on platforms such as the Open Science Framework (OSF; osf.io) or submit their study plans to journals as registered reports (Chambers, 2013; Hardwicke & Ioannidis, 2018). In the latter case, a study plan is peer-reviewed before the study is conducted. If the study plan is approved and the study is well conducted, it will be published irrespective of whether the findings are positive or null. Especially for us as ECRs, registered reports offer a chance to integrate reviewer comments in the study plan before data collection and this could help us design better studies and conduct higher quality research. All in all, Open Science practices such as pre-registrations empower researchers to follow their curiosity irrespective of the outcome of the research, by changing the main criterion for publication from merely positive findings towards high quality research. This, in turn, may help researchers daring to leave familiar paths and start exploring unfamiliar terrain, starting new collaborations and fostering novel and creative approaches to important theoretical questions. Lastly, we consider doing science in an open manner an ethical obligation. As researchers working at a public university, with research funded by taxpayer’s money, we feel it is imperative to make our research open to the public. Although as we argued above, all products from a research project should ideally be made open, Open Access publication is of special relevance here, as the published article has potentially the largest audience. Furthermore, having access to an article is the first step towards discovering the corresponding study materials, scripts, and data. As employees of Ghent University, we have access to the majority of journals in our field. However, this is not the case for other – and especially non-Western – universities and research institutes, and also not for the general public who is not affiliated with universities and research institutes. One major motivation for us to work in science is to contribute to collective human knowledge, and we would like our contribution to be accessible for as many people as possible. When publications are freely available, they are more likely to be read and cited (Piwowar & Vision, 2013; Wang, Liu, Mao, & Fang, 2015). In that sense we believe that Open Access publications could increase the diversity of the scientific community, inspire new research, improve the scientific progress and help to spend public money more thoughtfully (McKiernan et al., 2016). Despite our conviction that Open Science confers multiple benefits, we do acknowledge that this endeavor may have potential downsides. We will discuss some of them in more detail throughout and especially in the last part of the paper. Here we want to mention two potential risks that researchers, especially ECRs, should be aware of when following Open Science (Allen & Mehler, 2019). First, integrating Open Science practices into the research routine can be time-consuming at the beginning. The time spent writing pre-registrations, anonymizing raw data files and sharing experimental and data analysis scripts means that less time is available for other tasks. Second, Open Access publications could mean publications in journals with lower impact factors, so that Open Science practitioners could appear less competitive on their CVs. At first glance, this may sound like bad news, but it should not scare researchers away from considering the move to Open Science. In the following, we will describe the Open Science practices in our lab in more detail and argue why we think the benefits of Open Science outweigh these potential downsides.

The Open Science workflow in our lab Open Science is an umbrella term that encompasses multiple aspects of a research process. In our mission statement, we will focus on four aspects of Open Science that are particularly relevant for our lab: Open Methodology, Open Data, Open Source, and Open Access. As can be seen in Figure 1, these Open Science practices are embedded in our ‘co-pilot’ system, in which a lab member is involved in another lab member’s project from the very beginning until the end to check each step in the whole research process. In the following, we will first outline the ‘co-pilot’ system, and then describe how we incorporate the Open Science practices into our scientific workflow as a lab.

Co-Pilot system Being human, making mistakes is almost inevitable. To reduce potential errors and increase the reliability of scientific knowledge, the scientific community has implemented several quality control mechanisms, such as the collaboration of multiple authors on a paper to double-check each other’s work, and the expert peer-review procedure that papers need to go through before publication. However, it seems that not every step in the research process has received similar levels of scrutiny. For instance, a survey with psychologists revealed that double-checking each other’s data analysis and reported results was uncommon (Veldkamp, Nuijten, Dominguez-Alvarez, van Assen, & Wicherts, 2014). Accordingly, reporting errors in statistical results in psychology papers are relatively prevalent (Bakker & Wicherts, 2011; Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2016). To reduce such errors, a quality control system that checks each step in the research process is needed. In working environments where human errors are common and may have huge consequences, systems have been implemented to help reduce errors (Reason, 2000). One such system can be found in aviation, namely the co-piloting system, where the co-pilot double-checks every step of the pilot. The co-piloting system seems to significantly reduce the risk of human errors leading to airplane crashes (Wiegmann & Shappell, 2003). Wicherts (2011) argued that scientists should learn from how aviation deals with human errors. He proposed the ‘co-pilot’ system as a way to reduce errors in statistical analysis, in which at least two researchers conduct statistical analyses independently to increase the chance of veracity of reported results. We believe that such a co-pilot system should not be limited to data analysis, but should include other steps in a research process as well. That is why our lab implements a similar ‘co-pilot’ system. For each study, there is a lead researcher (or ‘pilot’) and a ‘co-pilot’, who is involved in the entire workflow. The ‘co-pilot’ researcher not only double-checks the data analysis scripts of the ‘pilot’ researcher, but is also involved in other tasks, from the conception of a research idea until the publication of the manuscript. We are using the co-pilot system since February/March 2019. We implemented the co-pilot system via GitHub, as GitHub allows for version control and provides integration with OSF that we use for pre-registering experiments and sharing data, scripts, and study materials (see below). At the beginning of a new project of a lab member (the pilot researcher), another lab member will be assigned as his/her co-pilot. The pilot researcher creates a private GitHub repository for the project and invites the co-pilot as a collaborator. In this way, the co-pilot has full access to all materials used in the project. Pre-registrations, experimental scripts, anonymized data files, and data analysis scripts are shared on GitHub between the pilot researcher and the co-pilot. The co-pilot then checks the work of the pilot researcher at each step. For instance, the co-pilot scrutinizes the experiment documentation (or pre-registration) to ensure that the research question, hypothesis, and data analysis plan are accurate and as intended. The co-pilot also double-checks the experimental scripts by running the experiment on him or herself and by working through the code. In the case of data analysis, the co-pilot starts with the anonymized raw data files, and checks the analysis code step by step. The co-pilot can also analyze the data independently or try out alternative data analysis techniques to see if the results obtained by the pilot researcher are robust (Steegen, Tuerlinckx, Gelman, & Vanpaemel, 2016). In each instance, the co-pilot provides feedback via GitHub, and the pilot researcher incorporates the comments from the co-pilot. A checklist for all the steps involved in the co-pilot system and more detailed guidelines can be found in the Appendix. The co-pilot system will not only increase the quality of our work by weeding out potential errors, it may also make our work more impactful by allowing other researchers to build on it more easily. For the co-pilot system to work well, the pilot researcher needs to provide thorough documentation for experimental scripts, raw data, and data analysis scripts. By making these materials open, we are thus creating and sharing sustainable knowledge, which allows our future selves and other researchers to easily reuse the data, adapt the experimental scripts, and reproduce the analyses (Hardwicke et al., 2018; Leonelli, 2018; Simmons, Nelson, & Simonsohn, 2011; Stodden, Seiler, & Ma, 2018). Our work will be useful not only in terms of publication, but also in all other research outputs that we generate throughout the project. We are aware that running such a co-pilot system is costly in time. After all, each of us needs to not only work on our own projects, but also be co-pilots for others. Moreover, writing comprehensive documentation is also time-consuming. This investment in time may well slow down the progress in our lab. One way to address this problem is to reward co-pilots with meaningful co-authorships. Contributions of each co-author will thus be clearly and transparently reported using the CRediT taxonomy (Brand, Allen, Altman, Hlava, & Scott, 2015), such that the contributions of each author, including that of the co-pilot, will be clear. Furthermore, writing documentations may be time-consuming in the short-term, but in the long-term it may save us time, as we will not need to retrospectively prepare data files or scripts when other researchers request them. By making these materials open, other researchers can reuse them without our intervention (although, of course, we would love to hear from other researchers who are using our data, experimental and data analysis scripts or study materials). Furthermore, by focusing on quality rather than quantity (The Slow Science Academy, 2010), we may eventually save time for both our lab and the whole scientific community by reducing errors in our findings. So far, the co-pilot system is working well in our lab. We try to distribute the workload evenly and discuss regularly whether the workload is acceptable. It has not been a problem yet to assign co-pilots, since we all have similar skills that are required to check each other’s work. Moreover, the close collaboration among lab members enables us to learn from each other and thus to improve and align our skill sets. However, this situation may change, for example when some of us move on to specific and/or more advanced analysis techniques or methodologies. This situation has in fact already occurred. When none of us has the required skills to qualify as co-pilot, we ask researchers with the required expertise outside the lab to collaborate and act as co-pilots. One project may thus require multiple co-pilots, each focusing on a specific aspect of the project. In our view, science is an inherently collaborative enterprise. Implementing the co-pilot system fosters close collaboration among lab members, which is the way we would like to work. Furthermore, sharing one’s finding with the whole scientific community is exciting, but can also be nerve-racking for ECRs, as one can never be 100% certain that no error is made along this complicated and at times daunting process. Having someone closely check each step of your work helps alleviate this feeling of uncertainty. Being involved in each other’s projects also allows us to learn Open Science practices from each other, which will be important assets in our career. In the following sections, we will describe the specific Open Science practices in more detail.

Open Access Open Access is another key principle of Open Science. Open Access literature is “digital, online, free of charge, and free of most copyright and licensing restrictions” (Suber, 2012, p. 4). We strongly aim for Open Access publications, as we believe that knowledge and in that sense education should be freely available for everyone. Therefore, we will strive for Diamond or Fair Gold Open Access. Diamond Open Access is “a publishing model that provides free access to peer-reviewed journal articles without charging […] [article processing charges] (APCs). Peer review and editing is done by volunteers” (Open Access Library, 2018a). On the other hand, Gold Open Access is “a publishing model that ensures free access to peer-reviewed articles through […] APCs” (Open Access Library, 2018b). Accordingly, we aim for publishing our work in fair/nonprofit Open Access journals that provide immediate Open Access to the articles and that require no APCs (Diamond) or low APCs (Gold). On our lab homepage, we will summarize the Open Access status of our publications (see https://fredvbrug.github.io/publications.html). Importantly, Diamond and Fair Gold Open Access will allow us to keep the copyright of our papers. This policy is also in line with the recommendation for Open Access publications of the ERC (European Research Council), the main funder of our lab, and of cOAlition S, a group of national research funding organizations (cOAlition S, 2019). We will publish via Green Open Access (which is also supported by the ERC) only in case that Diamond or Gold Open Access is not possible or when there are no suitable alternatives for the specific studies (but our first choice will always be Diamond or Fair Gold Open Access). Green Open Access is also known as self-archiving, meaning that papers (usually post-prints) are published in an open repository after having been published in closed access journals for a specific embargo period. In such cases, preferences will be given to journals with no embargo periods. Pre-prints should also be mentioned, as they seem to complement traditional publication routes. Pre-prints are manuscripts that are shared via repositories (e.g., psyarxiv.com) without having gone through peer-review. One should keep in mind, however, that not all journals accept manuscripts that have been previously made available as pre-prints (on sherpa.ac.uk/romeo, one can check journal policies concerning pre-prints). As a lab, we may consider pre-prints as well. Since we pre-register our study plans (see above) and plan to publish via Diamond or Gold Open Access, making our manuscripts available before submitting them to journals seems logical to us.

General problems when practicing Open Science and adequate solutions There are, of course, some obstacles on the road towards the adoption of Open Science practices. In the following, we will address a few general problems and provide solutions from our lab’s perspective. First, advocating and implementing Open Science could invite criticism from researchers who may find errors in our work. If other researchers find errors in our work, we will transparently communicate the corrections. Although we believe that committing errors is human, being criticized by other researchers can still be difficult to handle. This may especially be the case for us as ECRs, since we may feel insecure about our qualification as researchers in such situations. Facing criticism together with all lab members, including our PI, gives us social support that makes it easier to take criticism. This in turn gives us the ability to use the criticism to our advantage and build on it. Taking criticism and building on it is a personal skill that can be acquired. Fostering an environment where it is safe to expose ourselves to criticism and where failure is an accepted part of work is thus crucial in this respect. As a reviewer pointed out, many PIs or senior researchers do believe in Open Science, but may not recommend it to ECRs, because of possible drawbacks like lower productivity and publications with lower impact factors. This may be a legitimate concern, as researchers are often still being assessed according to traditional standards. However, we feel the incentive structure is quickly changing, as Open Science practices are increasingly being rewarded (Gernsbacher, 2018; Kidwell et al., 2016). PIs are in a more powerful situation than ECRs to implement Open Science on a broader level. If PIs who believe in Open Science actively fought for sustainable changes towards Open Science, adopting Open Science practices will no longer put ECRs at a disadvantage (for instance, for our current positions, experience with or interest in Open Science is considered a prerequisite). Second, we think Open Science is not an all-or-none commitment. For researchers who remain skeptical of certain aspects of Open Science, they can still incorporate other Open Science practices into their workflow. For instance, ECRs may be concerned that submitting their work to Open Access journals may lead to lower impact factors and jeopardize their careers (see below). In such cases, they may still try to publish in traditional closed high-profile journals, but make data and materials open, and share their manuscript via Green Open Access. We believe as more researchers adopt Open Science practices, the incentive structure for Open Science will be in place sooner. As already mentioned, another issue concerns the benefits and drawbacks of Open Access publications. Because Diamond and Gold Open Access are our desired publication routes, we have to accept that we will not always attempt to publish in closed journals with high impact factors. Accordingly, this may result in more publications with lower impact factors compared to other ECRs. This could harm our careers, because the impact factors of one’s publications are still a major factor in the competition for research positions. However, we are convinced that Open Science will lead to higher research quality. Additionally, the citation advantage associated with Open Access publications could increase the impact of our work and our chances of finding a permanent position (McKiernan et al., 2016; Piwowar & Vision, 2013; Wang et al., 2015). Moreover, in these times, the academic job market is very competitive. Therefore, we think it is important to have additional skills to stand out. In our view, experience with Open Science may become an essential skill for the job market (Working Group on Rewards under Open Science, 2017). Furthermore, many concrete skills that we will acquire when adopting Open Science practices, such as the use of version control systems, programming in popular open-source software such as Python, JavaScript, and R, will also benefit our careers. As with any investment, the potential benefits are greater the closer to the ground floor you get in. We believe this is an excellent time to gain Open Science experience. Having experience with Open Science is not only relevant from a practical perspective, that is, we know how to implement Open Science into our workflow; rather, and more importantly, having experience with Open Science says a lot about our understanding of how we want to work as scientists (Gewin, 2016). Doing Open Science as a whole lab will increase our visibility as Open Science practitioners and in turn raise the probability that we will be identified as such.

Conclusion To conclude, Open Science changes the way we do research. For us and probably many other researchers, doing science is more than a job. The principles of Open Science allow us to honor everything we love about science and to protect the quality of our work, so that we can be proud of it. We believe that Open Science brings researchers together so that they can revive what science is about at its core: striving to discover regularities about the world and helping to improve the way we live, sharing work with colleagues, as well as building up humanity’s knowledge (Grubaugh, 2017). In practice, we are aware that Open Science still has certain costs that can be daunting, especially to ECRs. A major concern is that the implementation of Open Science practices takes time at the beginning and may slow down the workflow, especially the publication process. However, we are convinced that it is time well spent. For us, the principles of Open Science not only represent good research practices, so that the quality of research will be improved; rather, the principles of Open Science liberate researchers and empower them to use a creative mindset when thinking about their work and designing new studies. In our lab, the time investment for Open Science is rewarded with meaningful co-authorships as part of the co-pilot system. Finally, we also hope to encourage ECRs to seek out PIs who actively support Open Science practices instead of compromising on their ideals (Allen & Mehler, 2019; Poldrack, 2019). Doing Open Science is easier when all lab members – and especially PIs – have an Open Science state of mind.