Assessment of quality of reporting meta-analysis and systematic review of pharmacotherapy of COVID-19 using preferred reporting items for systematic reviews and meta-analyses 2020 statement

INTRODUCTION

The global impact of the COVID-19 pandemic has catalysed an unprecedented volume of research focused on identifying effective treatment options, leading to an increase in clinical trials and subsequent meta-analyses (MAs). These MAs, which combine data from randomised controlled trials (RCTs), play a vital role in generating evidence to guide clinical practice. However, the reliability and utility of these MAs depend on transparent and comprehensive reporting, a critical aspect addressed by the Preferred Reporting Items for Systematic Reviews and MAs (PRISMA) statement.^[1]

As researchers continue to grapple with the complexities of COVID-19, the years 2022–2023 are expected to be marked by fruitful research aimed at improving our understanding of effective treatment modalities.^[2] The new study seeks to assess the quality of meta-analysis reports published during this period using the PRISMA statement as a rigorous evaluation framework.^[3]

The abundance of clinical trials and subsequent MAs presents an opportunity to gain valuable insights into treatment options for COVID-19. However, the validity and generalisability of these findings largely depend on the clarity and completeness of the reports. Inadequate reporting can cause bias, hinder reproducibility and impede the translation of research findings into applicable clinical guidelines.^[4]

As the dynamic landscape of the pandemic unfolds, it is imperative to critically evaluate the quality of meta-analysis reports. MAs conducted between 2022 and 2023 summarise the latest developments in research into the treatment of COVID-19, making this period an ideal focus for evaluating reporting practices. Such assessment is essential to recognise patterns, identify potential biases and facilitate improvements in reporting standards.^[5]

MAs have a substantial impact, shaping clinical decisions and informing public health strategies. Robust reporting not only strengthens the credibility of MAs but also makes the translation of research into practical knowledge more efficient. Through the lens of the PRISMA statement, this study aims to identify specific aspects of reporting that can benefit from standardisation and provide a basis for greater transparency and reliability.^[6]

Careful assessment of the quality of reporting is critical to advancing evidence-based practice for the treatment of COVID-19. Transparent reporting not only instils confidence in the synthesised evidence but also facilitates accurate interpretation and comparison of results across studies. In the context of a rapidly evolving pandemic, where timely and informed clinical decisions are paramount, the reliability of synthesised evidence is of the utmost importance.^[6]

MATERIALS AND METHODS

This evaluation focused on examining the reporting standards of MAs based on RCTs, specifically concerning COVID-19 pharmacotherapy. The PRISMA 2020 guidelines served as the framework for assessing whether the selected studies presented their methodology and procedures with sufficient transparency. This evaluation is specifically confined to meta-analysis with a systematic review. Furthermore, it exclusively considers studies employing RCT designs. The assessment entails scrutinising the quality of reporting MAs through PRISMA scoring and rigorous statistical analysis.

Data extraction

The initial query returned 197 studies, from which 166 were excluded following eligibility assessment, leaving 31 for full review. Fourteen MAs met all criteria and were included for final evaluation. Four-phase flow diagram summarises this process [Figure 1]. The 27-item checklist of PRISMA 2020 was determined for each meta-analysis.

Close

Figure 1:

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) 2020 flow diagram. RCT: Randomized Controlled Trails, SR: Systematic Reviews.

Export to PPT

RESULTS

DISCUSSION

Considering the pyramid of evidence-based medicine, systematic reviews (SRs) and meta-analyses (MAs) are at the top and provide high-level information and evidence about healthcare interventions. They synthesise findings from multiple studies, offering comprehensive insights into clinical interventions and informing evidence-based guidelines. MAs, in particular, aggregate effect sizes across studies, yielding more precise estimates through weighted data analysis.^[20]

The guidance tool called the PRISMA statement was published in 2009.^[20] The 2020 revision comprises 27 main items and a structured four-phase flow diagram to enhance transparency and reproducibility. To the best of our knowledge, this study is the first to assess the quality of reporting of SRs and MAs of pharmacotherapy of COVID-19 using PRISMA 2020 statement where apart from assign scoring to the main items in checklist we also assign scoring to the sub-items and compare the both scores to see that assign the scoring to the sub-items have play any role for the assessment of quality of reporting of SRs and MAs.

The median compliance of this assessment was 17 out of 26 items (65.38%) or 20 out of 41 items (48.78%). In the 26-item analysis comparison with Gundogan et al., it was 83% in Cochrane and 73% in non-Cochrane-based articles. Also, in Tounakaki et al.’s study, the mean score of PRISMA items was 23.2/27 (86.1%), and in Peters et al.’s journal, the mean item reported in 5 journals was 54.4% and the CDSR journal reported 100%.^[20-22] In comparison to the our 42-item score analysis, a study done by Park et al., which identified 24 items reported in <80% articles.^[23] It is notably higher compared to previous studies in other fields. This result may reflect an overall improvement in the adherence to the PRISMA statement by authors over time. In addition, it suggests that the mandatory requirement by journal editors and reviewers for PRISMA statement compliance as a condition for manuscript acceptance has contributed to enhanced reporting quality.^[20]

However, compliance was suboptimal in several areas, such as registration information (57.14%), protocol accessibility (42.85%), and amendment details (57.14%), all this included in protocol and registration which scored 52.38% in our finding which is similar to study done by Gundogan et al. reported 53% and higher than compared to study done by Park et al. (0%) and Tounakaki et al. (41.60%).^[20,21,23] Certainty assessment in our study reported 57.14% which is lower than the study done by Gundogan et al. (63%) but higher than the study done by Park et al. (9%).^[21,23] The insufficient reporting of protocol and registration details is consistent with findings from previous studies in other fields.^[22,23] In contrast, bias assessment in both methodology (78.57%), which is lower than Gundogan et al. (96%) but higher than that of the study done by Park et al. (65%) and Tounakaki et al. (55.50%).^[20,21,23] Bias results reporting (85.71%), which is similar to the study done by Gundogan et al. (88%) but higher than the study done by Park et al. (65%) and Tounakaki et al. (55.50%), as well as robustness evaluation (71.42%), which is higher than the study done by Park et al. (28%), showed higher compliance compared to other investigations.^[20,21,23] The use of structured bias assessment tools is essential for improving MA reliability and clinical applicability. In evidence-based medicine, the ability to evaluate methodological quality is vital for translating findings into practice.

Correlation analysis between PRISMA score % with various parameters of our study shows positive correlation between the number of authors and PRISMA score % (r = 0.53, P < 0.05) in comparison with the study done by Tounakaki et al., which reported there is no correlation, but there is a positive correlation with JIFs in Tounakaki et al.’s study (r = 0.792, P = 0.002), but not significant in our study. Regression analysis of PRISMA score % and number of authors in both 26 items and 41 items shows there is a significant relation in 41 items (F = 5.167, P < 0.05) but no significant relation in 26 items (F = 4.684, P > 0.05).^[20]

The final primary outcome examined the association between high PRISMA scores (≥24/26 and ≥36/41) and low PRISMA scores (≤23/26 and ≤35/41) in MAs with various combination of variables such as the number of studies, number of authors, number of patients, year of publication and JIFs (P = 0.8 for 26 and 41 item). Similarly, multivariate analysis showed no meaningful relationship between the type of bias assessment tool and PRISMA compliance (P = 0.93 for 26 items; P = 0.87 for 41 items).

Analysis by PRISMA endorsement status also showed no significant correlation with overall reporting quality (P = 0.41 for 26 items; P = 0.97 for 41 items). However, one notable finding was that a combination of JIFs and PRISMA score (26-item) correlated significantly with endorsement status (P = 0.02), suggesting that higher-impact journals may enforce stronger compliance.

This study also conducted a separate assessment of PRISMA for Abstract items. Abstract reporting showed an average compliance rate of 65.48%. In line with these findings, a study done by Hopewell et al. identified several inadequately reported items in conference abstracts, including key aspects such as participant information, adverse effects, study strengths and limitations and funding sources.^[24] This highlights ongoing gaps in abstract reporting that may influence the clarity and transparency of research findings.

The inclusion of MAs which analysed only RCTs is one of the key strengths of this study, but along with it, the COVID-19 pandemic presented challenges such as limited time and available data, which may have impacted the quality of reporting in some cases.

A major limitation of this study is that the high compliance (100%) in 20 PRISMA items could suggest some scoring leniency. Nevertheless, the scoring approach remained stringent and aligned with previous studies when evaluating items related to risk of bias assessment. Another limitation is that the exclusion of unpublished or preprint MAs may introduce publication bias.

CONCLUSION

The compliance is good for the 26-item score analysis as compared to the 41 items. The protocol and registration item have suboptimal compliance. There is a moderate positive association between the score of PRISMA items of individual meta-analyses and the number of authors. We recommend that journals mandate the inclusion of the PRISMA checklist during the submission process to promote consistency and transparency in systematic review reporting. Furthermore, the number of authors appears to be a significant factor influencing reporting quality. To elevate the standard of published evidence syntheses, stricter adherence to the PRISMA 2020 guidelines is essential, with particular emphasis on clearly documenting risk of bias procedures and outcome assessments.