Open science principles such as computational reproducibility are useful for establishing credible scientific claims. However, individual researchers cannot trivially replicate an entire study, let alone dozens of them. Faced with this problem, how does an individual researcher know whether an article is sufficiently trustworthy to be cited, built upon, or included in a research synthesis? This talk discuses the emerging field of Forensic Meta-Science, particularly in light of Cochrane’s recent endorsement of the INSPECT-SR Trustworthiness Assessment checklist. I provide an overview of the existing methods and the ones being developed in our research group, and the principles that we rely on (e.g., “think stupider”). These methods can be organised along two axes: the strength of verdict they license (coherence, plausibility, possibility) and the class of argument they rest on (interpretation, repetition, recalculation, granularity, bounds, and reconstruction). I close with a discussion of opportunities and challenges facing this nascent field, including the need for greater formalisation and validation of methods, and questions of usability and scalability, and feasible ways to reorganise incentive structures to reward this work. 

Ian is Senior Lecturer in Digitalisation of Psychology at the University of Bern, Switzerland. His research focuses on quality assessment in science, forensic meta-science, and measurement/schmeasurement practices. He co-leads ERЯOR: A bug bounty program for science (https://error.reviews).