| Statistical Component | Manuscript Approach | Harrell’s Critique |
|---|---|---|
| Primary Endpoint (OS) | Log-rank test, Kaplan-Meier estimates, stratified Cox model (HR 0.59, p=0.0096). Post-hoc multivariable Cox model adjusted for prognostic factors. | Relies on marginal effects (minimal stratification), which Harrell criticizes for poor generalizability and lack of patient-specific relevance. Post-hoc covariate adjustment is insufficient; pre-specified adjusted models preferred. Censoring risks informative bias. |
| Secondary Endpoints (PFS, CRR) | PFS: Log-rank test, stratified Cox model (HR 0.53, p=0.0002). CRR: Cochran-Mantel-Haenszel test (58.5% vs 25.3%, p<0.0001). Post-hoc EFS analysis. | Marginal estimates for PFS and CRR obscure heterogeneity. Harrell would advocate for covariate-adjusted models and unit-level risk differences for CRR. Post-hoc EFS analysis risks bias. |
| Subgroup Analyses | Unstratified Cox models for prespecified subgroups (e.g., age, IPI score), presented in a forest plot. | Unstratified subgroup analyses risk spurious findings. Harrell prefers a single covariate-adjusted model with interaction terms to explore heterogeneity. |
| Sensitivity Analyses | Censored COVID-19 deaths and treatment discontinuations; addressed stratification discrepancies. | Censoring COVID-19 deaths assumes non-informative censoring, which Harrell would question. Competing risk models needed. Post-hoc nature reduces robustness. |
| Statistical Power | Sample size (270 patients, 138 events) for 80% power (HR 0.60). Interim analysis at 70% events (p≤0.0174). | Power calculation is reasonable, but covariate adjustment could increase power. Harrell would scrutinize continued data collection post-interim analysis. |
| Model Assumptions | Cox proportional-hazards model; assumption not tested. SAS (version 9.4) used for analysis. | Failure to test proportional hazards assumption is a flaw. Harrell would recommend R’s rms package for flexible, transparent modeling. |
| Reporting | HRs, 95% CIs, p-values, Kaplan-Meier curves, forest plots. Emphasis on statistical significance and clinical implications. | Overemphasis on p-values; Harrell prefers effect sizes and confidence intervals. Marginal HRs lack clinical relevance for individuals; predictive models or nomograms needed. |
26 Case Studies for Critique
26.1 STARGLO Study
Overview of STARGLO Trial Statistical Analyses The STARGLO trial is a global phase 3, open-label, randomized controlled trial comparing glofitamab plus gemcitabine and oxaliplatin (Glofit-GemOx) versus rituximab plus GemOx (R-GemOx) in patients with relapsed or refractory diffuse large B-cell lymphoma (DLBCL). The primary endpoint was overall survival (OS), with key secondary endpoints including progression-free survival (PFS), complete response rate (CRR), and duration of complete response. The study employed intention-to-treat (ITT) analyses, stratified Cox proportional-hazards models, Kaplan-Meier estimates, and Cochran-Mantel-Haenszel tests, with post-hoc multivariable adjustments and sensitivity analyses to address potential biases (e.g., COVID-19-related deaths). Key Statistical Components and Harrell’s Critique 1. Primary Endpoint: Overall Survival (OS) Manuscript Approach: OS was defined as the time from randomization to death from any cause, analyzed using a two-sided log-rank test and Kaplan-Meier estimates.
A stratified Cox proportional-hazards model was used to estimate the hazard ratio (HR) with 95% confidence intervals (CIs), adjusting for stratification factors (number of previous lines of therapy [1 vs ≥2] and outcome of last systemic therapy [relapsed vs refractory]).
The primary analysis (March 29, 2023) showed a significant OS benefit for Glofit-GemOx (HR 0.59, 95% CI 0.40–0.88, p=0.0096), meeting the O’Brien-Fleming boundary (p≤0.0174).
Sensitivity analyses censored COVID-19-related deaths and treatment discontinuations, and a post-hoc multivariable Cox model adjusted for prognostic factors (details in appendix).
Harrell’s Critique: Marginal vs. Conditional Effects: The primary OS analysis relies on a stratified Cox model, which Harrell would classify as a marginal effect estimate because it only adjusts for stratification factors and not for a broader set of prognostic covariates (e.g., age, ECOG performance status, Ann Arbor stage, IPI score). Harrell argues that marginal effects, such as those from minimally adjusted models, are less generalizable because they assume the trial sample is a random sample from the target population, which is rarely true in RCTs due to inclusion/exclusion criteria and selection biases. The STARGLO trial’s reliance on marginal HRs risks overgeneralization, especially given the heterogeneous patient population (e.g., 42% aged ≥70 years, 61% refractory to last therapy).
Covariate Adjustment: Harrell strongly advocates for covariate-adjusted models to improve power and precision by accounting for outcome heterogeneity. The post-hoc multivariable Cox model (appendix 1, p. 10) is a step in this direction, but its late inclusion (post-hoc rather than pre-specified) limits its interpretability. Harrell would argue that a pre-specified, covariate-adjusted model should have been the primary analysis to reduce bias and enhance clinical relevance for individual patients.
Transportability: The trial’s OS estimates may not be transportable to other populations due to the specific eligibility criteria (e.g., transplant-ineligible patients, exclusion of double/triple hit lymphomas). Harrell emphasizes that marginal effects are particularly sensitive to such sampling assumptions, making the reported HR less applicable to broader DLBCL populations.
Censoring Assumptions: The sensitivity analyses for COVID-19-related deaths involve censoring, which Harrell would scrutinize for potential informative censoring. If COVID-19 deaths were related to treatment exposure or disease severity, censoring could bias the OS estimates, a concern Harrell often raises about survival analyses that assume non-informative censoring.
- Secondary Endpoints: PFS, CRR, and Duration of Response Manuscript Approach: PFS was defined as the time from randomization to disease progression or death, assessed by an independent review committee (IRC) using a log-rank test, Kaplan-Meier estimates, and a stratified Cox model (HR 0.53, 95% CI 0.37–0.74, p=0.0002).
CRR was compared using the Cochran-Mantel-Haenszel test, with Clopper-Pearson 95% CIs (Glofit-GemOx: 58.5%, R-GemOx: 25.3%, p<0.0001).
Duration of complete response was analyzed similarly to PFS, without stratification.
A post-hoc event-free survival (EFS) analysis addressed censoring imbalances, defined as time to progression, death, or new anti-lymphoma therapy.
Harrell’s Critique: Binary Outcomes (CRR): Harrell discourages reliance on one-number summaries like marginal odds ratios or response rates for binary outcomes (e.g., CRR), as they obscure heterogeneity across patients. The Cochran-Mantel-Haenszel test provides a marginal estimate of the treatment effect on CRR, which Harrell would criticize for lacking patient-specific context. He would recommend reporting the distribution of unit-level risk differences (e.g., using the marginaleffects R package) to capture variability in treatment response across subgroups (e.g., by IPI score or refractory status).
PFS and EFS: The PFS and EFS analyses, while stratified, suffer from the same marginal effect limitations as OS. Harrell would advocate for covariate-adjusted models to account for prognostic factors like tumor burden or ECOG status, which could explain heterogeneity in progression risk. The post-hoc EFS analysis is a pragmatic response to censoring imbalances, but Harrell would question why such adjustments were not pre-specified, as post-hoc analyses risk data-driven bias.
Nonlinear Models: For binary (CRR) and time-to-event (PFS, duration of response) outcomes, Harrell notes that marginal effects in nonlinear models (e.g., logistic or Cox models) are harder to interpret because coefficients do not directly translate to population-averaged effects. He would suggest using techniques like double-difference contrasts to explore interactions (e.g., treatment effect by refractory status), which the manuscript does not report.
- Subgroup and Sensitivity Analyses Manuscript Approach: Subgroup analyses for OS were conducted using unstratified Cox models across prespecified subgroups (e.g., age, sex, IPI score, refractory status), presented in a forest plot.
Sensitivity analyses addressed stratification discrepancies, COVID-19 impacts, and treatment discontinuations due to drug supply issues.
Post-hoc multivariable Cox models for OS and PFS adjusted for prognostic factors (appendix 1, p. 10).
Harrell’s Critique: Subgroup Analyses: Harrell is skeptical of subgroup analyses that rely on unadjusted or minimally adjusted models, as they risk spurious findings due to multiple testing and lack of covariate adjustment. The unstratified Cox models for subgroups are particularly problematic, as they ignore the trial’s stratification factors, potentially introducing bias. Harrell would recommend a single, covariate-adjusted model with interaction terms to test for treatment effect heterogeneity, rather than separate subgroup analyses.
Sensitivity Analyses: While the sensitivity analyses for COVID-19 and stratification discrepancies are commendable, Harrell would question their robustness. For example, censoring COVID-19 deaths assumes these events are unrelated to treatment or disease, which may not hold. Harrell would advocate for competing risk models (e.g., Fine-Gray models) to handle such events more rigorously.
Post-Hoc Adjustments: The post-hoc multivariable models are aligned with Harrell’s preference for covariate adjustment, but their post-hoc nature undermines their credibility. Harrell emphasizes pre-specified models to avoid data dredging, and the manuscript’s reliance on post-hoc adjustments suggests a reactive approach to unanticipated issues (e.g., censoring imbalances).
- Statistical Power and Sample Size Manuscript Approach: The sample size (270 patients, 138 events) was calculated to provide 80% power to detect a 7.3-month difference in median OS (HR 0.60), with an interim analysis at 70% events (97 events) using an O’Brien-Fleming boundary.
The primary analysis was based on 101 events, with recalculated p-value thresholds (p≤0.0174 for OS, p≤0.03244 for secondary endpoints).
Harrell’s Critique: Power Calculations: Harrell would approve of the rigorous sample size calculation but caution that power estimates based on marginal effects (e.g., HR 0.60) may be overly optimistic if covariate heterogeneity is not accounted for. Adjusting for prognostic covariates in the primary analysis could have increased power, potentially requiring a smaller sample size.
Interim Analysis: The use of an O’Brien-Fleming boundary is standard, but Harrell would scrutinize the decision to continue the study after meeting the primary endpoint, as this risks dilution of the treatment effect if additional events are less favorable. He would advocate for clear stopping rules and transparency about the IDMC’s rationale.
- Model Assumptions and Robustness Manuscript Approach: The Cox proportional-hazards assumption was not explicitly tested in the manuscript.
SAS (version 9.4) was used for all analyses, with no mention of alternative software or packages (e.g., R’s rms package, which Harrell developed).
Harrell’s Critique: Proportional Hazards: Harrell emphasizes the importance of testing model assumptions, such as proportional hazards in Cox models. The manuscript’s silence on this suggests a potential oversight, as non-proportional hazards could bias the HR estimates. Harrell would recommend diagnostic plots (e.g., Schoenfeld residuals) or flexible parametric models (e.g., restricted cubic splines) to assess and address violations.
Software and Transparency: Harrell, a proponent of R and the rms package, would criticize the exclusive use of SAS for its lack of transparency and limited support for advanced modeling techniques (e.g., spline-based adjustments for nonlinear effects). He would advocate for open-source tools and detailed reporting of model specifications to enhance reproducibility.
Robustness: Harrell’s claim that covariate-adjusted models are often more robust than marginal models (despite misspecification concerns) applies here. The trial’s marginal analyses are vulnerable to biases from unmodeled heterogeneity, whereas a covariate-adjusted approach could have mitigated these risks.
- Reporting and Interpretation Manuscript Approach: Results are presented with HRs, 95% CIs, and p-values, with Kaplan-Meier curves and forest plots for visualization.
The manuscript emphasizes statistical significance (e.g., p=0.0096 for OS) and clinical implications (e.g., Glofit-GemOx as a second-line option).
Harrell’s Critique: Overemphasis on P-Values: Harrell warns against over-relying on p-values, as they can exaggerate findings in the presence of heterogeneity. The manuscript’s focus on p<0.05 thresholds (even with alpha adjustment) risks oversimplification. Harrell would advocate for reporting effect sizes (e.g., HRs) with confidence intervals and graphical displays of effect distributions (e.g., Harrell plots).
Clinical Relevance: Harrell emphasizes that marginal effects are less clinically relevant for individual patients. The reported HR of 0.59 for OS is a population-averaged estimate, but clinicians need patient-specific estimates (e.g., adjusted for IPI score or age). Harrell would recommend predictive models or nomograms to translate trial results into individualized predictions.
Visualization: While Kaplan-Meier curves and forest plots are included, Harrell would suggest additional visualizations, such as partial effect plots or nomograms, to better communicate covariate-adjusted effects and their uncertainty.
Table: Statistical Analyses in STARGLO Trial with Harrell’s Critique Below is a table summarizing the key statistical components of the STARGLO trial, evaluated through Frank Harrell’s perspective, formatted as requested using R code to generate a markdown table. R
Overall Assessment The STARGLO trial’s statistical analyses align with standard clinical trial practices but fall short of Frank Harrell’s standards for robust, patient-centered statistical modeling. The reliance on marginal effect estimates (e.g., stratified Cox models, Cochran-Mantel-Haenszel tests) limits the generalizability and clinical relevance of the findings, as Harrell warns. The post-hoc multivariable adjustments and sensitivity analyses are steps toward addressing heterogeneity and bias, but their reactive nature undermines their credibility, a point Harrell frequently critiques. The failure to test model assumptions (e.g., proportional hazards) and the emphasis on p-values further diverge from Harrell’s principles of transparent, assumption-checked, and effect-focused reporting. Recommendations Inspired by Harrell Pre-Specified Covariate Adjustment: Use a pre-specified, covariate-adjusted Cox model for OS and PFS, including prognostic factors like age, IPI score, and ECOG status, to improve power and precision.
Unit-Level Effects for Binary Outcomes: Report distributions of unit-level risk differences for CRR, using tools like the marginaleffects R package, to capture heterogeneity.
Interaction Terms for Subgroups: Replace separate subgroup analyses with a single model including treatment-covariate interactions to rigorously assess effect heterogeneity.
Competing Risk Models: Use Fine-Gray or other competing risk models for sensitivity analyses involving COVID-19 deaths to account for informative censoring.
Assumption Testing: Test proportional hazards and report diagnostics (e.g., Schoenfeld residuals) to ensure model validity.
Enhanced Visualization: Include Harrell plots, partial effect plots, or nomograms to communicate covariate-adjusted effects and their uncertainty.
Open-Source Tools: Use R’s rms package for flexible, reproducible modeling, aligning with Harrell’s emphasis on transparency.
Conclusion The STARGLO trial provides valuable evidence for Glofit-GemOx in DLBCL but employs statistical methods that Harrell would critique for their reliance on marginal effects, limited covariate adjustment, and post-hoc analyses. By adopting Harrell’s principles—covariate-adjusted models, patient-specific estimates, and robust assumption testing—the trial could enhance the interpretability, generalizability, and clinical relevance of its findings. The provided table encapsulates these critiques, offering a clear framework for understanding the manuscript’s statistical strengths and weaknesses through Harrell’s lens.
26.2 AQUILLA Study
Attaching package: 'dplyr'The following objects are masked from 'package:stats':
filter, lagThe following objects are masked from 'package:base':
intersect, setdiff, setequal, union[1] "MAIN CRITICAL ANALYSIS"| AQUILA Trial: Frank Harrell’s Critical Assessment | |||||
| Methodological Strengths, Concerns, and Missing Elements | |||||
| Assessment Domain | Specific Metric | Daratumumab | Control/Context | Statistical Result | Harrell's Assessment |
|---|---|---|---|---|---|
| Design Quality | Primary endpoint | Progression-free survival | Time-to-event analysis | Appropriate choice | GOOD: Clinically meaningful endpoint |
| Design Quality | Sample size | 194 patients | 196 patients | 85% power achieved | GOOD: Adequate statistical power |
| Design Quality | Risk stratification | Binary high/low risk | Threshold-based | Arbitrary cutoffs | CONCERN: Should use continuous modeling |
| Design Quality | Multiple testing | Hierarchical approach | Partial adjustment | Some control | CONCERN: Incomplete adjustment |
| Baseline | Age median | 63.0 years | 64.5 years | Well balanced | GOOD: Appropriate population |
| Baseline | High-risk patients | 79.5% (≥3 factors) | 79.6% (≥3 factors) | Balanced | CONCERN: Only 40.5% by Mayo 2018 |
| Baseline | Plasma cells % | 20% median | 20% median | 10-60% range | CONCERN: Wide range, continuous better |
| Efficacy | Events (progression/death) | 67 (34.5%) | 99 (50.5%) | HR 0.49 (0.36-0.67) | GOOD: Clinically meaningful effect |
| Efficacy | 5-year PFS | 63.1% | 40.8% | 22% absolute benefit | GOOD: Substantial benefit |
| Efficacy | Number needed to treat | 4.5 patients | over 5 years | NNT = 4.5 | GOOD: Clinically relevant |
| Methods | Primary analysis | Stratified log-rank | Cox regression | HR with 95% CI | GOOD: Appropriate methods |
| Methods | Proportional hazards | Not tested | Assumption unclear | May vary over time | CONCERN: Should test assumption |
| Methods | Treatment interactions | Limited subgroups | Descriptive only | No formal tests | MAJOR GAP: Missing heterogeneity |
| Methods | Competing risks | Death events | Not modeled | Standard survival | CONCERN: Should model competing risks |
| Clinical Impact | Absolute benefit | 22% fewer progress | at 5 years | Meaningful for many | GOOD: Clinically relevant magnitude |
| Clinical Impact | Patient selection | All high-risk treated | Binary approach | Crude stratification | MAJOR GAP: Need precision tools |
| Clinical Impact | Individual prediction | Population estimates | No personalization | One-size-fits-all | MAJOR GAP: Need risk calculators |
| Safety | Grade 3-4 events | 40.4% | 30.1% | 10% increase | CONCERN: Moderate toxicity increase |
| Safety | Serious infections | 16.1% | 4.6% | 11.5% increase | CONCERN: Clinically significant |
| Safety | Treatment deaths | 1.0% | 2.0% | Lower with treatment | GOOD: No excess mortality |
[1] "\nHARRELL'S RECOMMENDATIONS"| Frank Harrell’s Recommendations for Future Research | |||
| From Population Effects to Precision Medicine | |||
| Priority Level | Current AQUILA Limitation | Harrell's Recommended Approach | Expected Clinical Benefit |
|---|---|---|---|
| HIGH | Binary risk classification | Continuous risk modeling with splines | Better patient selection precision |
| HIGH | No treatment heterogeneity analysis | Formal interaction testing | Identify who benefits most |
| HIGH | Population-level decisions | Individual risk calculators | Personalized treatment decisions |
| MEDIUM | Proportional hazards assumption | Test and model time-varying effects | More accurate risk estimates |
| MEDIUM | Competing risk ignored | Fine-Gray competing risk models | Realistic outcome probabilities |
| MEDIUM | Multiple comparisons | Fewer, pre-specified endpoints | Reduced false discovery risk |
| LOW | Limited communication tools | Nomograms and natural frequencies | Better informed consent |
| LOW | Binary treatment decision | Threshold-based recommendations | Nuanced clinical guidance |
[1] "\nR CODE EXAMPLES"| R Code for Harrell’s Recommended Analyses | ||
| Method | Code_Example | Application_to_AQUILA |
|---|---|---|
| Continuous Risk | library(rms) cph(Surv(time, event) ~ rcs(plasma_cells,3) + rcs(m_protein,3)) | Replace binary high/low risk with smooth curves |
| Treatment Interaction | fit <- cph(Surv(time, event) ~ treatment * rcs(baseline_risk,3)) anova(fit) | Test if treatment effect varies by baseline risk |
| Competing Risk | library(cmprsk) crr(time, event_type, cov1=treatment) | Model death without progression as competing event |
| Validation | validate(fit, method=‘boot’, B=1000) calibrate(fit, cmethod=‘KM’, B=1000) | Assess model performance and calibration |
| Nomogram | nom <- nomogram(fit, fun=plogis, funlabel=‘5-yr PFS Probability’) plot(nom) | Create bedside prediction tool for clinicians |
============================================================ FRANK HARRELL'S BOTTOM LINE ASSESSMENT:============================================================ STRENGTHS:• Well-conducted randomized trial with appropriate methods• Clinically meaningful primary endpoint and effect size• Adequate sample size and statistical power• Clear demonstration of efficacyMAJOR OPPORTUNITIES:• Move from binary to continuous risk prediction• Develop tools for individualized treatment decisions• Model treatment effect heterogeneity• Create bedside clinical prediction toolsHARRELL'S QUOTE:'This trial definitively answers whether daratumumab works.Now we need to answer who it works best for and developthe statistical tools to personalize these decisions.'