How systematic are systematic literature searches?

Abstract

Systematic literature reviews and meta-analyses have gained importance due to the increasing rates of academic publications and the need to evaluate and integrate findings. The systematic literature search using databases and platforms is the essential first step in the process of conducting research syntheses. We investigated the replicability of literature searches conducted within two databases (ERIC, PsycINFO) and two search platforms (ProQuest, Web of Science). Fifteen researchers from different institutions performed identical searches guided by a manual. We found that ERIC searches were replicable, but the type of quotation marks used in the search string mattered. Despite PsycINFO being frequently reported in published reviews and meta-analyses, none of the researchers had direct access, but used alternative platforms (EBSCOhost or Ovid) to access PsycINFO. Using a specific platform (either EBSCOhost, Ovid, or ProQuest) to indirectly search the databases, PsycINFO and/or ERIC provided identical search results across institutions. However, the indirect search via platforms mostly delivered different numbers of hits compared to the direct search in the databases. Additionally, the searches via platforms identified publications that partially differed from those found by direct searches in the databases. Web of Science searches showed a low degree of replicability; results differed between institutions with regard to the number of hits and in the composition of the identified publications. To increase replicability, we suggest expanding reporting guidelines like PRISMA-S to require researchers to publish their full initial search hits.