Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Nate Breznau; Eike Mark Rinke; Alexander Wuttke; Hung H.V. Nguyen; Muna Adem; Jule Adriaans; Amalia Alvarez-Benjumea; Henrik K. Andersen; Daniel Auer; Flavio Azevedo; Oke Bahnsen; Dave Balzer; Gerrit Bauer; Paul C. Bauer; Markus Baumann; Sharon Baute; Verena Benoit; Julian Bernauer; Carl Berning; Anna Berthold; Felix S. Bethke; Thomas Biegert; Katharina Blinzler; Johannes N. Blumenberg; Licia Bobzien; Andrea Bohman; Thijs Bol; Amie Bostic; Zuzanna Brzozowska; Katharina Burgdorf; Kaspar Burger; Kathrin B. Busch; Juan Carlos-Castillo; Nathan Chan; Pablo Christmann; Roxanne Connelly; Christian S. Czymara; Elena Damian; Alejandro Ecker; Achim Edelmann; Maureen A. Eger; Simon Ellerbrock; Anna Forke; Andrea Forster; Chris Gaasendam; Konstantin Gavras; Vernon Gayle; Theresa Gessler; Timo Gnambs; Amélie Godefroidt; Max Grömping; Martin Groß; Stefan Gruber; Tobias Gummer; Jan Paul Heisig; Andreas Hadjar; Sebastian Hellmeier; Stefanie Heyne; Magdalena Hirsch; Mikael Hjerm; Oshrat Hochman; Andreas Hövermann; Sophia Hunger; Christian Hunkler; Nora Huth; Zsófia S. Ignácz; Laura Jacobs; Jannes Jacobsen; Bastian Jaeger; Sebastian Jungkunz; Nils Jungmann; Mathias Kauff; Manuel Kleinert; Julia Klinger; Jan Philipp Kolb; Marta Kołczyńska; John Kuk; Katharina Kunißen; Dafina Kurti Sinatra; Alexander Langenkamp; Philipp M. Lersch; Lea Maria Löbel; Philipp Lutscher; Matthias Mader; Joan E. Madia; Natalia Malancu; Luis Maldonado; Helge Marahrens; Nicole Martin; Paul Martinez; Jochen Mayerl; Oscar J. Mayorga; Patricia McManus; Kyle McWagner; Cecil Meeusen; Daniel Meierrieks; Jonathan Mellon; Friedolin Merhout; Samuel Merk; Daniel Meyer; Leticia Micheli; Jonathan Mijs; Cristóbal Moya; Marcel Neunhoeffer; Daniel Nüst; Olav Nygård; Fabian Ochsenfeld; Gunnar Otte; Anna O. Pechenkina; Christopher Prosser; Louis Raes; Kevin Ralston; Miguel R. Ramos; Arne Roets; Jonathan Rogers; Guido Ropers; Robin Samuel; Gregor Sand; Ariela Schachter; Merlin Schaeffer; David Schieferdecker; Elmar Schlueter; Regine Schmidt; Katja M. Schmidt; Alexander Schmidt-Catran; Claudia Schmiedeberg; Jürgen Schneider; Martijn Schoonvelde; Julia Schulte-Cloos; Sandy Schumann; Reinhard Schunck; Jürgen Schupp; Julian Seuring; Henning Silber; Willem Sleegers; Nico Sonntag; Alexander Staudt; Nadia Steiber; Nils Steiner; Sebastian Sternberg; Dieter Stiers; Dragana Stojmenovska; Nora Storz; Erich Striessnig; Anne Kathrin Stroppe; Janna Teltemann; Andrey Tibajev; Brian Tung; Giacomo Vagni; Jasper Van Assche; Meta van der Linden; Jolanda van der Noll; Arno Van Hootegem; Stefan Vogtenhuber; Bogdan Voicu; Fieke Wagemans; Nadja Wehl; Hannah Werner; Brenton M. Wiernik; Fabian Winter; Christof Wolf; Yuki Yamada; Nan Zhang; Conrad Ziller; Stefan Zins; Tomasz Zółtak

doi:10.1073/pnas.2203150119

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Hung H.V. Nguyen, Muna Adem, Jule Adriaans, Amalia Alvarez-Benjumea, Henrik K. Andersen, Daniel Auer, Flavio Azevedo, Oke Bahnsen, Dave Balzer, Gerrit Bauer, Paul C. Bauer, Markus Baumann, Sharon Baute, Verena Benoit, Julian Bernauer, Carl Berning, Anna BertholdFelix S. Bethke, Thomas Biegert, Katharina Blinzler, Johannes N. Blumenberg, Licia Bobzien, Andrea Bohman, Thijs Bol, Amie Bostic, Zuzanna Brzozowska, Katharina Burgdorf, Kaspar Burger, Kathrin B. Busch, Juan Carlos-Castillo, Nathan Chan, Pablo Christmann, Roxanne Connelly, Christian S. Czymara, Elena Damian, Alejandro Ecker, Achim Edelmann, Maureen A. Eger, Simon Ellerbrock, Anna Forke, Andrea Forster, Chris Gaasendam, Konstantin Gavras, Vernon Gayle, Theresa Gessler, Timo Gnambs, Amélie Godefroidt, Max Grömping, Martin Groß, Stefan Gruber, Tobias Gummer, Jan Paul Heisig, Andreas Hadjar, Sebastian Hellmeier, Stefanie Heyne, Magdalena Hirsch, Mikael Hjerm, Oshrat Hochman, Andreas Hövermann, Sophia Hunger, Christian Hunkler, Nora Huth, Zsófia S. Ignácz, Laura Jacobs, Jannes Jacobsen, Bastian Jaeger, Sebastian Jungkunz, Nils Jungmann, Mathias Kauff, Manuel Kleinert, Julia Klinger, Jan Philipp Kolb, Marta Kołczyńska, John Kuk, Katharina Kunißen, Dafina Kurti Sinatra, Alexander Langenkamp, Philipp M. Lersch, Lea Maria Löbel, Philipp Lutscher, Matthias Mader, Joan E. Madia, Natalia Malancu, Luis Maldonado, Helge Marahrens, Nicole Martin, Paul Martinez, Jochen Mayerl, Oscar J. Mayorga, Patricia McManus, Kyle McWagner, Cecil Meeusen, Daniel Meierrieks, Jonathan Mellon, Friedolin Merhout, Samuel Merk, Daniel Meyer, Leticia Micheli, Jonathan Mijs, Cristóbal Moya, Marcel Neunhoeffer, Daniel Nüst, Olav Nygård, Fabian Ochsenfeld, Gunnar Otte, Anna O. Pechenkina, Christopher Prosser, Louis Raes, Kevin Ralston, Miguel R. Ramos, Arne Roets, Jonathan Rogers, Guido Ropers, Robin Samuel, Gregor Sand, Ariela Schachter, Merlin Schaeffer, David Schieferdecker, Elmar Schlueter, Regine Schmidt, Katja M. Schmidt, Alexander Schmidt-Catran, Claudia Schmiedeberg, Jürgen Schneider, Martijn Schoonvelde, Julia Schulte-Cloos, Sandy Schumann, Reinhard Schunck, Jürgen Schupp, Julian Seuring, Henning Silber, Willem Sleegers, Nico Sonntag, Alexander Staudt, Nadia Steiber, Nils Steiner, Sebastian Sternberg, Dieter Stiers, Dragana Stojmenovska, Nora Storz, Erich Striessnig, Anne Kathrin Stroppe, Janna Teltemann, Andrey Tibajev, Brian Tung, Giacomo Vagni, Jasper Van Assche, Meta van der Linden, Jolanda van der Noll, Arno Van Hootegem, Stefan Vogtenhuber, Bogdan Voicu, Fieke Wagemans, Nadja Wehl, Hannah Werner, Brenton M. Wiernik, Fabian Winter, Christof Wolf, Yuki Yamada, Nan Zhang, Conrad Ziller, Stefan Zins, Tomasz Zółtak

Political Science and International Relations

Research output: Contribution to journal › Article › peer-review

Original language	English
Article number	e2203150119
Journal	Proceedings of the National Academy of Sciences of the United States of America
Volume	119
Issue number	44
DOIs	https://doi.org/10.1073/pnas.2203150119
State	Published - Nov 1 2022

ASJC Scopus Subject Areas

General

Keywords

analytical flexibility
immigration
many analysts
metascience
policy preferences
researcher degrees of freedom

Access to Document

10.1073/pnas.2203150119

Cite this

Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H. V., Adem, M., Adriaans, J., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., Balzer, D., Bauer, G., Bauer, P. C., Baumann, M., Baute, S., Benoit, V., Bernauer, J., Berning, C., ... Zółtak, T. (2022). Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences of the United States of America, 119(44), Article e2203150119. https://doi.org/10.1073/pnas.2203150119

Breznau, N, Rinke, EM, Wuttke, A, Nguyen, HHV, Adem, M, Adriaans, J, Alvarez-Benjumea, A, Andersen, HK, Auer, D, Azevedo, F, Bahnsen, O, Balzer, D, Bauer, G, Bauer, PC, Baumann, M, Baute, S, Benoit, V, Bernauer, J, Berning, C, Berthold, A, Bethke, FS, Biegert, T, Blinzler, K, Blumenberg, JN, Bobzien, L, Bohman, A, Bol, T, Bostic, A, Brzozowska, Z, Burgdorf, K, Burger, K, Busch, KB, Carlos-Castillo, J, Chan, N, Christmann, P, Connelly, R, Czymara, CS, Damian, E, Ecker, A, Edelmann, A, Eger, MA, Ellerbrock, S, Forke, A, Forster, A, Gaasendam, C, Gavras, K, Gayle, V, Gessler, T, Gnambs, T, Godefroidt, A, Grömping, M, Groß, M, Gruber, S, Gummer, T, Heisig, JP, Hadjar, A, Hellmeier, S, Heyne, S, Hirsch, M, Hjerm, M, Hochman, O, Hövermann, A, Hunger, S, Hunkler, C, Huth, N, Ignácz, ZS, Jacobs, L, Jacobsen, J, Jaeger, B, Jungkunz, S, Jungmann, N, Kauff, M, Kleinert, M, Klinger, J, Kolb, JP, Kołczyńska, M, Kuk, J, Kunißen, K, Sinatra, DK, Langenkamp, A, Lersch, PM, Löbel, LM, Lutscher, P, Mader, M, Madia, JE, Malancu, N, Maldonado, L, Marahrens, H, Martin, N, Martinez, P, Mayerl, J, Mayorga, OJ, McManus, P, McWagner, K, Meeusen, C, Meierrieks, D, Mellon, J, Merhout, F, Merk, S, Meyer, D, Micheli, L, Mijs, J, Moya, C, Neunhoeffer, M, Nüst, D, Nygård, O, Ochsenfeld, F, Otte, G, Pechenkina, AO, Prosser, C, Raes, L, Ralston, K, Ramos, MR, Roets, A, Rogers, J, Ropers, G, Samuel, R, Sand, G, Schachter, A, Schaeffer, M, Schieferdecker, D, Schlueter, E, Schmidt, R, Schmidt, KM, Schmidt-Catran, A, Schmiedeberg, C, Schneider, J, Schoonvelde, M, Schulte-Cloos, J, Schumann, S, Schunck, R, Schupp, J, Seuring, J, Silber, H, Sleegers, W, Sonntag, N, Staudt, A, Steiber, N, Steiner, N, Sternberg, S, Stiers, D, Stojmenovska, D, Storz, N, Striessnig, E, Stroppe, AK, Teltemann, J, Tibajev, A, Tung, B, Vagni, G, Van Assche, J, van der Linden, M, van der Noll, J, Van Hootegem, A, Vogtenhuber, S, Voicu, B, Wagemans, F, Wehl, N, Werner, H, Wiernik, BM, Winter, F, Wolf, C, Yamada, Y, Zhang, N, Ziller, C, Zins, S & Zółtak, T 2022, 'Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty', Proceedings of the National Academy of Sciences of the United States of America, vol. 119, no. 44, e2203150119. https://doi.org/10.1073/pnas.2203150119

@article{78e9c999ad95492d8879e11a0d0637e3,

title = "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty",

keywords = "analytical flexibility, immigration, many analysts, metascience, policy preferences, researcher degrees of freedom",

author = "Nate Breznau and Rinke, {Eike Mark} and Alexander Wuttke and Nguyen, {Hung H.V.} and Muna Adem and Jule Adriaans and Amalia Alvarez-Benjumea and Andersen, {Henrik K.} and Daniel Auer and Flavio Azevedo and Oke Bahnsen and Dave Balzer and Gerrit Bauer and Bauer, {Paul C.} and Markus Baumann and Sharon Baute and Verena Benoit and Julian Bernauer and Carl Berning and Anna Berthold and Bethke, {Felix S.} and Thomas Biegert and Katharina Blinzler and Blumenberg, {Johannes N.} and Licia Bobzien and Andrea Bohman and Thijs Bol and Amie Bostic and Zuzanna Brzozowska and Katharina Burgdorf and Kaspar Burger and Busch, {Kathrin B.} and Juan Carlos-Castillo and Nathan Chan and Pablo Christmann and Roxanne Connelly and Czymara, {Christian S.} and Elena Damian and Alejandro Ecker and Achim Edelmann and Eger, {Maureen A.} and Simon Ellerbrock and Anna Forke and Andrea Forster and Chris Gaasendam and Konstantin Gavras and Vernon Gayle and Theresa Gessler and Timo Gnambs and Am{\'e}lie Godefroidt and Max Gr{\"o}mping and Martin Gro{\ss} and Stefan Gruber and Tobias Gummer and Heisig, {Jan Paul} and Andreas Hadjar and Sebastian Hellmeier and Stefanie Heyne and Magdalena Hirsch and Mikael Hjerm and Oshrat Hochman and Andreas H{\"o}vermann and Sophia Hunger and Christian Hunkler and Nora Huth and Ign{\'a}cz, {Zs{\'o}fia S.} and Laura Jacobs and Jannes Jacobsen and Bastian Jaeger and Sebastian Jungkunz and Nils Jungmann and Mathias Kauff and Manuel Kleinert and Julia Klinger and Kolb, {Jan Philipp} and Marta Ko{\l}czy{\'n}ska and John Kuk and Katharina Kuni{\ss}en and Sinatra, {Dafina Kurti} and Alexander Langenkamp and Lersch, {Philipp M.} and L{\"o}bel, {Lea Maria} and Philipp Lutscher and Matthias Mader and Madia, {Joan E.} and Natalia Malancu and Luis Maldonado and Helge Marahrens and Nicole Martin and Paul Martinez and Jochen Mayerl and Mayorga, {Oscar J.} and Patricia McManus and Kyle McWagner and Cecil Meeusen and Daniel Meierrieks and Jonathan Mellon and Friedolin Merhout and Samuel Merk and Daniel Meyer and Leticia Micheli and Jonathan Mijs and Crist{\'o}bal Moya and Marcel Neunhoeffer and Daniel N{\"u}st and Olav Nyg{\aa}rd and Fabian Ochsenfeld and Gunnar Otte and Pechenkina, {Anna O.} and Christopher Prosser and Louis Raes and Kevin Ralston and Ramos, {Miguel R.} and Arne Roets and Jonathan Rogers and Guido Ropers and Robin Samuel and Gregor Sand and Ariela Schachter and Merlin Schaeffer and David Schieferdecker and Elmar Schlueter and Regine Schmidt and Schmidt, {Katja M.} and Alexander Schmidt-Catran and Claudia Schmiedeberg and J{\"u}rgen Schneider and Martijn Schoonvelde and Julia Schulte-Cloos and Sandy Schumann and Reinhard Schunck and J{\"u}rgen Schupp and Julian Seuring and Henning Silber and Willem Sleegers and Nico Sonntag and Alexander Staudt and Nadia Steiber and Nils Steiner and Sebastian Sternberg and Dieter Stiers and Dragana Stojmenovska and Nora Storz and Erich Striessnig and Stroppe, {Anne Kathrin} and Janna Teltemann and Andrey Tibajev and Brian Tung and Giacomo Vagni and {Van Assche}, Jasper and {van der Linden}, Meta and {van der Noll}, Jolanda and {Van Hootegem}, Arno and Stefan Vogtenhuber and Bogdan Voicu and Fieke Wagemans and Nadja Wehl and Hannah Werner and Wiernik, {Brenton M.} and Fabian Winter and Christof Wolf and Yuki Yamada and Nan Zhang and Conrad Ziller and Stefan Zins and Tomasz Z{\'o}{\l}tak",

note = "Publisher Copyright: Copyright {\textcopyright} 2022 the Author(s). Published by PNAS.",

year = "2022",

month = nov,

day = "1",

doi = "10.1073/pnas.2203150119",

language = "English",

volume = "119",

journal = "Proceedings of the National Academy of Sciences of the United States of America",

issn = "0027-8424",

publisher = "National Academy of Sciences",

number = "44",

}

TY - JOUR

T1 - Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

AU - Breznau, Nate

AU - Rinke, Eike Mark

AU - Wuttke, Alexander

AU - Nguyen, Hung H.V.

AU - Adem, Muna

AU - Adriaans, Jule

AU - Alvarez-Benjumea, Amalia

AU - Andersen, Henrik K.

AU - Auer, Daniel

AU - Azevedo, Flavio

AU - Bahnsen, Oke

AU - Balzer, Dave

AU - Bauer, Gerrit

AU - Bauer, Paul C.

AU - Baumann, Markus

AU - Baute, Sharon

AU - Benoit, Verena

AU - Bernauer, Julian

AU - Berning, Carl

AU - Berthold, Anna

AU - Bethke, Felix S.

AU - Biegert, Thomas

AU - Blinzler, Katharina

AU - Blumenberg, Johannes N.

AU - Bobzien, Licia

AU - Bohman, Andrea

AU - Bol, Thijs

AU - Bostic, Amie

AU - Brzozowska, Zuzanna

AU - Burgdorf, Katharina

AU - Burger, Kaspar

AU - Busch, Kathrin B.

AU - Carlos-Castillo, Juan

AU - Chan, Nathan

AU - Christmann, Pablo

AU - Connelly, Roxanne

AU - Czymara, Christian S.

AU - Damian, Elena

AU - Ecker, Alejandro

AU - Edelmann, Achim

AU - Eger, Maureen A.

AU - Ellerbrock, Simon

AU - Forke, Anna

AU - Forster, Andrea

AU - Gaasendam, Chris

AU - Gavras, Konstantin

AU - Gayle, Vernon

AU - Gessler, Theresa

AU - Gnambs, Timo

AU - Godefroidt, Amélie

AU - Grömping, Max

AU - Groß, Martin

AU - Gruber, Stefan

AU - Gummer, Tobias

AU - Heisig, Jan Paul

AU - Hadjar, Andreas

AU - Hellmeier, Sebastian

AU - Heyne, Stefanie

AU - Hirsch, Magdalena

AU - Hjerm, Mikael

AU - Hochman, Oshrat

AU - Hövermann, Andreas

AU - Hunger, Sophia

AU - Hunkler, Christian

AU - Huth, Nora

AU - Ignácz, Zsófia S.

AU - Jacobs, Laura

AU - Jacobsen, Jannes

AU - Jaeger, Bastian

AU - Jungkunz, Sebastian

AU - Jungmann, Nils

AU - Kauff, Mathias

AU - Kleinert, Manuel

AU - Klinger, Julia

AU - Kolb, Jan Philipp

AU - Kołczyńska, Marta

AU - Kuk, John

AU - Kunißen, Katharina

AU - Sinatra, Dafina Kurti

AU - Langenkamp, Alexander

AU - Lersch, Philipp M.

AU - Löbel, Lea Maria

AU - Lutscher, Philipp

AU - Mader, Matthias

AU - Madia, Joan E.

AU - Malancu, Natalia

AU - Maldonado, Luis

AU - Marahrens, Helge

AU - Martin, Nicole

AU - Martinez, Paul

AU - Mayerl, Jochen

AU - Mayorga, Oscar J.

AU - McManus, Patricia

AU - McWagner, Kyle

AU - Meeusen, Cecil

AU - Meierrieks, Daniel

AU - Mellon, Jonathan

AU - Merhout, Friedolin

AU - Merk, Samuel

AU - Meyer, Daniel

AU - Micheli, Leticia

AU - Mijs, Jonathan

AU - Moya, Cristóbal

AU - Neunhoeffer, Marcel

AU - Nüst, Daniel

AU - Nygård, Olav

AU - Ochsenfeld, Fabian

AU - Otte, Gunnar

AU - Pechenkina, Anna O.

AU - Prosser, Christopher

AU - Raes, Louis

AU - Ralston, Kevin

AU - Ramos, Miguel R.

AU - Roets, Arne

AU - Rogers, Jonathan

AU - Ropers, Guido

AU - Samuel, Robin

AU - Sand, Gregor

AU - Schachter, Ariela

AU - Schaeffer, Merlin

AU - Schieferdecker, David

AU - Schlueter, Elmar

AU - Schmidt, Regine

AU - Schmidt, Katja M.

AU - Schmidt-Catran, Alexander

AU - Schmiedeberg, Claudia

AU - Schneider, Jürgen

AU - Schoonvelde, Martijn

AU - Schulte-Cloos, Julia

AU - Schumann, Sandy

AU - Schunck, Reinhard

AU - Schupp, Jürgen

AU - Seuring, Julian

AU - Silber, Henning

AU - Sleegers, Willem

AU - Sonntag, Nico

AU - Staudt, Alexander

AU - Steiber, Nadia

AU - Steiner, Nils

AU - Sternberg, Sebastian

AU - Stiers, Dieter

AU - Stojmenovska, Dragana

AU - Storz, Nora

AU - Striessnig, Erich

AU - Stroppe, Anne Kathrin

AU - Teltemann, Janna

AU - Tibajev, Andrey

AU - Tung, Brian

AU - Vagni, Giacomo

AU - Van Assche, Jasper

AU - van der Linden, Meta

AU - van der Noll, Jolanda

AU - Van Hootegem, Arno

AU - Vogtenhuber, Stefan

AU - Voicu, Bogdan

AU - Wagemans, Fieke

AU - Wehl, Nadja

AU - Werner, Hannah

AU - Wiernik, Brenton M.

AU - Winter, Fabian

AU - Wolf, Christof

AU - Yamada, Yuki

AU - Zhang, Nan

AU - Ziller, Conrad

AU - Zins, Stefan

AU - Zółtak, Tomasz

PY - 2022/11/1

Y1 - 2022/11/1

KW - analytical flexibility

KW - immigration

KW - many analysts

KW - metascience

KW - policy preferences

KW - researcher degrees of freedom

UR - http://www.scopus.com/inward/record.url?scp=85141004640&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85141004640&partnerID=8YFLogxK

U2 - 10.1073/pnas.2203150119

DO - 10.1073/pnas.2203150119

M3 - Article

C2 - 36306328

AN - SCOPUS:85141004640

SN - 0027-8424

VL - 119

JO - Proceedings of the National Academy of Sciences of the United States of America

JF - Proceedings of the National Academy of Sciences of the United States of America

IS - 44

M1 - e2203150119

ER -

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

ASJC Scopus Subject Areas

Keywords

Access to Document

Other files and links

Cite this