European Journal of Work and Organizational Psychology
Group and Organization Management
International Journal of Selection and Assessment
Journal of Business and Psychology
Journal of Managerial Psychology
Journal of Personnel Psychology
Journal of Vocational Behavior
Organizational Research Methods
Work, Aging and Retirement
Joint Announcement Letter:
We all strive to produce useful, robust, and impactful research. Recently, however, how we (and really all sciences) conduct our research has drawn critique. Unfortunately, these critiques appear quite well-founded as studies have demonstrated severe replication challenges (Open Science Collaboration, 2015), suppression of non-significant results in published research (publication bias, Banks, Kepes, & McDaniel, 2015; Schmidt & Hunter, 2015), post-hoc fabrication of hypotheses (HARKing, Bosco, Aguinis, Field, Pierce, & Dalton, in press; Kerr, 1998), and other forms of questionable research practices (Banks et al., in press; John, Loewenstein, & Prelec, 2012; O’Boyle, Banks, & Gonzalez-Mule, in press), most notably P-Hacking which is a “tireless” drive to seek out significant results in one’s database so as to compose the desired “story” often driven by the conscious or unconscious belief that only a package of “clean” significant results will ultimately yield publication (Gerber & Malhotra, 2008a; Gerber & Malhotra, 2008b; Masicampo & Lalande, 2012). As editors, we believe we have a duty to seek out ways to address these issues by examining the publication “system” and how it can be altered so that it does not indirectly reinforce counterproductive research practices. To that end, we will be launching a special trail initiative — a hybrid registered reports submission option. This new protocol is NOT replacing the existing protocols, but rather expands the options available to authors and simply serves as an alternative “pathway to publication.
In this alternative path, for example, authors submit the introduction, methods, measurement information, and analysis plan of a completed study (no results or discussion in the first round). This abbreviated paper then undergoes peer review and is evaluated on the merits, rigor, and quality of the project rather than what was actually found. Following review, the article will then be either rejected, receive an R&R, or accepted in principle for publication. Following an in-principle acceptance (IPA), the authors will then produce a more traditionally formatted manuscript that includes Results and Discussion sections (Stage 2). It is important to note that the Stage 2 manuscript can still include ancillary ad hoc analyses as appropriate as an honest accounting is the ideal, they are just noted as such. Assuming that the stage 2 submission is consistent with what was committed to in stage one, and a sensible interpretation and discussion of the findings, the manuscript will be published regardless of the results. The approach is hybrid in that it combines features of traditional submissions (data have already been collected) and of registered reports of research proposals (reviewers are unaware of study outcomes). We hope that this new format combines “the best of both worlds” and encourages researchers who may be hesitant trying submission formats more radically different from traditional protocols.
Our goal is to encourage authors to propose conceptually sound, interesting, and methodologically rigorous research. We welcome the results from sound research no matter if they support proposed hypotheses, yield ‘null’ results, or replicate (or fail to replicate) previous work. Simply speaking, well conceived, designed, and conducted research should form the corpus of knowledge. We believe this special initiative provides an opportunity to do just that.
Overall, the hybrid approach emphasizes seeking truth, whatever that truth may be. We also all recognize that this path won’t solve all of our problems as a science. In fact, no single solution can address all concerns. We see this initiative as a meaningful and quite intuitive approach with no significant downsides. It might just be very helpful to our collective efforts to advance science meaningfully. As editors, we are willing to experiment with solutions.
Some Key Notes
Although all the aforementioned journals are participating in this trial initiative, we are doing it independently. As a result, processes and procedures may vary – perhaps quite considerably – among us. However, the overall aim will be to improve the integrity and quality of research. Authors are encouraged to consult each journal’s website for more details. The editors, however, will be conferring regularly on the initiative, its success and opportunities for improvement.
Participating journals will communicate the specifics of their approach sometime in 2016, this will also be the time they will accept manuscripts in this alternative format.
The hybrid registered reports approach is appropriate for inductive, deductive, mixed methods, and papers involving multiple studies. In the case of the latter type, the author is encouraged to contact the editor to decide on the best approach for submission. There are multiple options as the hybrid registered reports approach is highly flexible. Overall, an initial conversation with the editor can clarify an approach that makes sense for the project in question – ultimately it is not useful to assume a one size fits all approach given the eclectic nature of research.
John Antonakis, Leadership Quarterly (incoming editor)
Kevin Daniels, European Journal of Work and Organizational Psychology
Nadya A. Fouad, Journal of Vocational Behavior
William Gardner, Group and Organization Management
James LeBreton, Organizational Research Methods
Bernd Marcus, Journal of Personnel Psychology
Steven G. Rogelberg, Journal of Business and Psychology
Lois E. Tetrick, Journal of Managerial Psychology
Chockalingam Viswesvaran, International Journal of Selection and Assessment
Mo Wang, Work, Aging and Retirement
Banks, G. C., Kepes, S., & McDaniel, M. A. (2015). Publication bias: Understand the myths concerning threats to the advancement of science. In C. E. Lance, & R. J. Vandenberg (Eds.), More Statistical and methodological myths and urban legends (pp. 36-64). New York, NY: Routledge.
Banks, G. C., O’Boyle Jr., E., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. (in press). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management.
Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (in press). HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology.
Gerber, A., & Malhotra, N. (2008a). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3,: 313-326.
Gerber, A. S., & Malhotra, N. (2008b). Publication bias in empirical sociological research do arbitrary significance levels distort published results? Sociological Methods & Research, 37, 3-30.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532.
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2: 196-217.
Masicampo, E. J., & Lalande, D. R. (2012). A peculiar prevalence of p values just below. 05. The Quarterly Journal of Experimental Psychology and Aging, 65, 2271–2279.
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. in press. The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Newbury Park, CA: Sage.