We appreciate your interest in our publication and the time taken to read and thoughtfully critique our work [1]. We believe in constructive criticism and cherish the opportunity to both address concerns raised by Drs. Ali and Tariq and further improve our work.
On the first point raised by Drs. Ali and Tariq concerning Prospective Register of Systematic Reviews (PROSPERO) registration: we routinely register planned reviews on PROSPERO and are fully aware of its intended purpose of transparency and reducing study duplication [2-5]. However, for countries other than the United Kingdom, the waiting time prior to the pandemic was at least 3 months, and in our experience, even longer. Due to current world affairs, PROSPERO is not actively reviewing submissions and is uploading them “as-is” prior to initial screening if unrelated to coronavirus disease 2019, thus its capacity as a quality-control filter is currently limited. Additionally, in our experience, we have found that full compliance with registration does not prevent other authors, from performing the same or similar studies [6-8].
Due to the aforementioned factors, we have decided not to pursue registration when we anticipate a study will take us less than a month from inception to submission, as we estimate publication would take an approximately similar time to the registration certification and upload. This defeats its use in preventing study duplication and our methods are transparent enough to allow and encourage this kind of discussion. Such is the case of this manuscript, which as detailed in methods, was initiated in November and submitted in December. Additionally, in this study’s specific case, we had already performed a small version of the analysis for in-house outcomes grand rounds session, which, per PROSPERO rules, would have made it ineligible for registration [9].
A recent study in vaccine research found that registration was one of the least complied factors within guidelines, finding only 26% of compliance [10]. Similar studies on ophthalmology reported again that registration was the least complied guideline at 9%, 19.4% in maxillo-craniofacial surgery, and 14% in neuroendovascular reviews [11-13]. Prior iterations of similar studies have also been successfully performed without registration [14,15]. Thus, given the time implication, that it does not actively prevent other studies from being performed, and the current relaxed standards, we believe registration shouldn’t be a barrier to performing and publishing reviews and currently reported compliance seems to reflect similar sentiments.
Regarding the second point mentioned on heterogeneity, we discuss the matter in the closing statements of the discussion section; we believe heterogeneity comes from unaccounted variations in concurrent treatment and staging from patients across different studies as well as from differences in indications for surgical intervention, which is difficult to control for and we even encouraged future dedicated studies. However, and again as we state various times in the manuscript, this is a difficult population to study and this should not preclude current attempts of analyzing available evidence.
We typically perform stepwise addition analysis of studies on I2 % to assess which studies induce heterogeneity and might impact results and determine if further stricter reanalysis or subgrouping is needed as recommended by Cochrane and have done so successfully in the past [3]. In this paper’s case, we found that heterogeneity changed effect size but not significance or direction in any analyzed variable’s result except for operative time, in which we opted for a more conservative conclusion. We typically perform sensitivity analysis to a degree in all of our work; however, the most accurate way to explore this heterogeneity is a meta-regression. In either case, Review Manager ver. 5.3 (RevMan; Cochrane, London, UK) does not feature dedicated functions for these processes [16,17].
Thirdly, on publication bias, other than a funnel plot, Review Manager ver. 5.3 (Cochrane) does not offer additional tools for publication bias such as Egger’s test. While we routinely assess funnel plots for symmetry, we find it to be too arbitrary to give any credible weight and attempt to correlate our findings with prior published literature. Funnel plots alone have been found to have limited use and studies have found researchers’ limited ability to adequately interpret funnel plots [18,19].
Lastly, we find McGrath’s method to be interesting and will undoubtedly consider it for future studies. However, it is important to notice that while the publication of McGrath et al. [20]~ is around 2 years, it has only been cited approximately 40 times compared to the publication of Wan et al. [21] (2,765), suggesting it’s yet to be more widely adopted and therefore we believe it to be an unfair assumption to expect it to be the default method for data estimation. Additionally, we specifically stated in our limitations section that data estimation was a limitation of our study but will look to incorporate this possible more accurate data estimation tool in future studies.
Once again, we appreciate the feedback and hope to have provided adequate responses to Drs. Ali and Tariq’s concerns.
No potential conflict of interest relevant to this article was reported.
ReferencesHinojosa-GonzalezDERoblesgil-MedranoAVillarreal-EspinosaJBMinimally invasive versus open surgery for spinal metastasis: a systematic review and meta-analysis2021Sep1[Epub]. https://doi.org/10.31616/asj.2020.0637Hinojosa-GonzalezDESalgado-GarzaGTorres-MartinezMEndovascular treatment of transplant renal artery stenosis: a systematic review and meta-analysis2021Aug16[Epub]. https://doi.org/10.1177/15266028211038593Hinojosa-GonzalezDEBuenoLCRoblesgil-MedranoALaparoscopic vs open portoenterostomy in biliary atresia: a systematic review and meta-analysis2021Jul16[Epub]. https://doi.org/10.1007/s00383-021-04964-5Hinojosa-GonzálezDESalgado-GarzaGPortoenterostomy for biliary atresia: laparsocopic versus open approach: PROSPERO 2020 CRD42020204438 [Internet]LondonNational Institute for Health Research2020[cited 2021 Sep 20]. Available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020204438Hinojosa-GonzalezDEPercutaneous angioplasty alone vs percutaneous angioplasty and stenting in the treatment of transplant renal artery stenosis: a systematic review and meta-analysis: PROSPERO 2020 CRD42020171352 [Internet]LondonNational Institute for Health Research2020[cited 2021 Sep 20]. Available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020171352BertoloRGaristoJBovePMottrieARoccoBEAU Robotic Urology Section (ERUS) Working Group on SciencePerioperative outcomes between single-port and “multi-port” robotic assisted radical prostatectomy: where do we stand?202115513843LiKYuXYangXPerioperative and oncologic outcomes of single-port vs multiport robot-assisted radical prostatectomy: a meta-analysis2021Jul21[Epub]. https://doi.org/10.1089/end.2021.0210Hinojosa-GonzalezDETorres-MartinezMSingleport vs multi-port robotic prostatectomy: a systematic review and meta-analysis: PROSPERO 2021 CRD42021228744 [Internet]LondonNational Institute for Health Research2020[cited 2021 Sep 20]. Available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021228744NdzeVNJacaAWiysongeCSReporting quality of systematic reviews of interventions aimed at improving vaccination coverage: compliance with PRISMA guidelines201915283643LuVMAlviMAGoyalAKerezoudisPBydonMThe potential of minimally invasive surgery to treat metastatic spinal disease versus open surgery: a systematic review and meta-analysis2018112e85968LeeSYSagooHFarwanaRWhitehurstKFowlerAAghaRCompliance of systematic reviews in ophthalmology with the PRISMA statement201717178AkhigbeTZolnourianABultersDCompliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature201739458PidgeonTEWellsteadGSagooHJafreeDJFowlerAJAghaRAAn assessment of the compliance of systematic review articles published in craniofacial surgery with the PRISMA statement guidelines: a systematic review201644152230HughesHCarthyAMSheridanGADonnellJMDoyleFButlerJThoracolumbar burst fractures: a systematic review and meta-analysis comparing posterior-only instrumentation versus combined anterior-posterior instrumentation202146E8409National Institute for Health ResearchYorkUniversity of York2019BakerWLWhiteCMCappelleriJCKlugerJColemanCIHealth Outcomes, Policy, and Economics (HOPE) Collaborative GroupUnderstanding heterogeneity in meta-analysis: the role of meta-regression200963142634HigginsJPThomasJChandlerJCochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021) [Internet]LondonCochrane2021[cited 2021 Sep 20]. Available from: www.training.cochrane.org/handbookSterneJASuttonAJIoannidisJPRecommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomized controlled trials2011343d4002TerrinNSchmidCHLauJIn an empirical evaluation of the funnel plot, researchers could not visually identify publication bias200558894901McGrathSZhaoXSteeleRThombsBDBenedettiADEPRESsion Screening Data (DEPRESSD) CollaborationEstimating the sample mean and standard deviation from commonly reported quantiles in meta-analysis2020Jan30[Epub]. https://doi.org/10.1177/0962280219889080WanXWangWLiuJTongTEstimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range201414135