Impact of the COVID-19 Pandemic on the Collection and Analysis of Efficacy in Ongoing Clinical Trials
The concept of efficacy can cover a very wide range of endpoints and domains, depending on the therapeutic area and the type of study drug under investigation. It may cover visit-based assessments, e.g. laboratory assessments, RECIST scans, PRO assessments or respiratory assessments, or may also incorporate events, e.g. deaths, relapses, or progression. In this blog we will discuss key concepts related to the impact of COVID-19 specifically on programming. Impact on statistical inference will be considered in some parts of this article, but in-depth analysis of this is not within the scope of this article. This article is intended for trials which were ongoing during the COVID-19 pandemic, and not for trials specifically testing vaccines or treatments for COVID-19.
We summarise firstly the key points to consider from the FDA and the EMA on the impact of COVID-19 on efficacy in trials which are ongoing during the pandemic.
FDA Guidance on COVID-19 Impact on Efficacy: Key Points 
- With respect to efficacy assessments, the FDA recommends consultation with the appropriate FDA review division regarding protocol modifications for the collection of efficacy endpoints, such as use of virtual assessments, delays in assessments, and alternative collection of research-specific specimens, if feasible.
- For individual instances where efficacy endpoints are not collected, the reasons for failing to obtain the efficacy assessment should be documented (e.g. identifying the specific limitation imposed by COVID-19 leading to the inability to perform the protocol-specified assessment).
- If the results of laboratory tests or imaging assessments are the basis for formal hypothesis testing, including primary or secondary efficacy endpoints, sponsors should consult with the relevant FDA review division. For example, disparities in laboratory measurements or imaging protocols will introduce increased variability and thus can affect type I and type II error rates.
In addition, the FDA has recently issued guidance on statistical considerations for proposed changes to trial conduct that may impact the analysis and interpretation of primary or key secondary endpoints in the trial . The FDA recommends that sponsors consult with the relevant FDA review division when considering protocol changes and changes to the statistical analysis plan that may impact the analysis and interpretation of these endpoints.
EMA Guidance on COVID-19 Impact on Efficacy: Key Points 
The following is a brief summary of points from the EMA guidance that may impact on efficacy assessments and analysis.
- Impact on recruitment, data collection, analysis, and interpretation of results for each trial will need a thorough case-by-case assessment.
- Risk assessment of the impact of COVID-19 potentially affecting trial participants directly and affecting clinical trial conduct on trial integrity and interpretability is recommended.
- Sponsors are advised to pre-plan how systematic deviations resulting from the measures and individual decisions related to the COVID-19 pandemic are captured.
- Data collection should preferably not stop and should continue as long as possible. However, potential risks for study participants when undergoing study-specific procedures take priority in decisions taken by study participants and health institutes.
- Potential follow-up considerations based on the risk assessment may include the following:
- proposals to deal with any identified potential sources of bias comprising identification of newly emerging intercurrent events or missing values, or other unforeseeable required changes to trial elements;
- the need to adjust the trial sample size;
- recommendations from a trial participant’s safety perspective on how to stop, pause or restart the trial;
- recommendations of additional measures when completing the trial after the pandemic (e.g. validation of outcomes that were measured differently).
Discussion on Impact of COVID-19 on Efficacy
An estimand is a precise description of the treatment effect reflecting the clinical question posed by a given clinical trial objective . There are five attributes used in the construction of estimands; namely the treatment condition, the population of interest, the patient-level endpoint (variable), the handling of intercurrent events, and the population-level summary measure.
Intercurrent events are events occurring after treatment initiation that affect either the interpretation or the existence of the measurements associated with the clinical question of interest. Examples may include use of rescue medication, discontinuation of treatment, and death.
COVID-19 may introduce additional intercurrent events, and the definition of estimands may need to be revised, or additional estimands may be required to be defined. Therefore, a study may require additional sensitivity analyses. It is very important to perform a full risk assessment for each study and identify a list of efficacy endpoints that may be affected by COVID-19, and therefore may need sensitivity analysis. Subsequently, a list of variables should be identified in order for additional information to be collected on the eCRF.
The Pharmaceutical Industry COVID-19 Biostatistics Working Group (Meyer et al., 2020)  recommends a forward-looking assessment to anticipate effects which may occur, and that risk assessment should continue and be updated regularly throughout the trial. It is highly recommended to reference this paper for more detail on risk assessment and estimands.
Examples of potential disruptions to efficacy assessments due to COVID-19 which may be defined as intercurrent events could include:
- missed visits/assessments
- remote/virtual visits
- alternative methods of assessment (e.g. local assessments instead of central, RECIST scans being done at different locations)
- missed study drug dosing
- changes in study drug dosing
- diagnosis of COVID-19
Handling of intercurrent events during the COVID-19 pandemic may need to be adjusted due to a different expected frequency of the event, for example interruptions to the study drug may have been rarely expected and therefore not previously have had a significant impact on efficacy; however, during the pandemic there have been study-/region-wide drug supply problems and therefore interruptions to the study drug may be more frequent and have a much greater impact.
It is important to ensure that all variables related to these events are collected on the eCRF and mapped appropriately into SDTM/ADaM to enable sensitivity analysis to take place. For example, the visit-based information for remote/alternative assessments should be collected in the same visit structure as the visit-based efficacy endpoint in order for it to be merged to the corresponding analysis dataset and flagged appropriately.
There may also be problems in ability to test the existing primary endpoint, for example hospitals may be unable to conduct certain assessments such as progression-free survival scans or FEV1 assessments. The sample size or the primary estimand may need to be adjusted. Sponsors should ensure enough data is collected to answer the original scientific question posed by the trial.
Sensitivity analysis may need to be conducted by region, by time period (e.g. pre-pandemic vs during pandemic) or by assessment method as COVID-19 may have a high impact on results, e.g. quality of life assessments. Effects such as these may introduce confounding in establishing the efficacy of a drug. In this case it is important to clearly define the regions and the time periods, e.g. what date is considered as the cut-off for the start of the pandemic? This cut-off may differ between country and region. Virtual or remote assessments may cause additional variability in data, and it is recommended  that if the conduct/method of an assessment has not already been tested that a pilot testing of a small number of patients occurs before applying this to the entire trial population.
It is recommended that the target population for the efficacy analysis remains as originally planned . However, sensitivity analysis may be required on additional populations, e.g. pre- and post-pandemic randomised patients, or subsets of the full analysis set. As mentioned previously, care should be taken over these definitions to be very specific about any cut-offs used and these should be clearly defined in the protocol and/or SAP.
For studies where a significant number of deaths would have been expected pre-pandemic, or where assessment of death is required for key endpoints (e.g. overall survival), sponsors need to ensure the reason for death is very accurately captured on the eCRF so that sensitivity analysis can be conducted differentiating expected deaths and additional deaths. There need to be detailed discussions on clear definitions of whether a death is associated with COVID-19 and the specific reasons available for death in the creation of the eCRF fields and eCRF completion guidelines. If a patient has a positive diagnosis of COVID-19, the death may be solely due to COVID-19, or due to existing medical conditions exacerbated by COVID-19. The patient may also have suspected COVID-19, but there was no ability to test for a confirmed diagnosis. The patient may have died due to problems in an overwhelmed health service but not actually had COVID-19 themselves.
Information on missed visits can be mapped into the custom Visit Events (VE) domain . The VE domain is recommended as SV can only be used to map visits that actually occurred. VE can map missed visits and reasons for the missed visit (FDA recommended ). The full list of visit names and numbers should be mapped to VE.VISIT and VE.VISITNUM. The numbers and naming should be consistent with the SV domain such that VISIT and VISITNUM values are the same. VE.VEOCCUR should be marked Y/N if the visit occurred or not, and if not, then the reason should be documented in VE.VEREASOC.
Regarding missing assessments, CDISC  recommends the following: If a visit was missed completely, there is no need to represent separately the fact that some assessments scheduled for that visit were missed. If some of the data scheduled for a visit was collected, then the assessments expected but missed at that visit generally should be represented in the relevant domain in with --STAT = "NOT DONE" and the reason for being missing in --REASND.
Remote/virtual visits and alternative assessments
Information on remote/alternative visits can be mapped into the custom Visit Events (VE) domain  (similarly to missed visits), with the mode of contact being specified in the non-standard variable (NSV) VECNTMOD.
If specific assessments require a different type of assessment, it is suggested to record this in the relevant domain in the standard variable --METHOD.
Interruption or discontinuation to study treatment
Last dosing record could represent whether the stoppage of study treatment is permanently discontinued or temporary by populating either EXRSDISC (for permanent discontinuation) or NSV EXRSINT (for temporary discontinuation) . In case of permanent discontinuation, then DSTERM could indicate if discontinuation is COVID-19 related or not using NSV DSEPRELI.
CDISC guidance indicates that temporary interruptions or adjustments to study treatment can be captured by introducing collection of the following NSV variables:
EXRSINT EXEPADJI EXEPINTI EXEPDSCI .
Infection/Diagnosis of COVID-19
If information about COVID-19 infection/symptoms is collected in studies that are ongoing, then CDISC recommends the information be mapped into AE (using the NSV AEEPRELI = Y) .
Diagnosis date, i.e. specific identification of virus from a collected sample will be mapped into the MB domain.
Note: historical infections/diagnosis of COVID-19 should be mapped into MH.
Sponsors need to ensure the reason for death is accurately captured on the eCRF. If death is part of a key endpoint of the trial (e.g. overall survival), then reasons and dates of death should be mapped to CE as standard. If deaths are not an endpoint of interest, then the information should be mapped to the DS, AE and DD (Death Domain) as standard.
ADaMs may need to be more complex than originally planned due to the additional information required for analysis. This may be present in the form of additional subject-level population flags, additional record-level flags, or additional derived parameters.
Additional population flags can be added to ADSL, in case sensitivity analysis needs to be performed. CDISC guidance already allows for bespoke population flags in the form of ----FL, using Y/N values for each patient. These ADSL flags can then be merged onto each ADaM.
Derived parameters may need to be added for efficacy, e.g. in ADTTE there may be additional parameters required where we censor at date of diagnosis of COVID-19.
The visit windowing in AVISIT/AVISITN should remain the same as planned, but analysis flags can be added to the datasets to give the opportunity to perform sensitivity analysis.
Record-level flags for inclusion in analysis can also be added in each ADaM in the form of CDISC-specified population record-level flags (e.g. FASRFL or PPROTRFL) or bespoke population record-level flags (----RFL). If record-level flags are not specific to a population definition, general flags such as ANL01FL can be added.
Tables, Figures and Listings
The key elements leading to TFL production related to trials ongoing during the pandemic are:
- risk assessment
- implementing any required protocol or SAP amendments, in particular adjusting any estimands
- collecting of any additional information on the eCRF in an accurate and clearly defined way
- mapping additional information accurately to SDTM
- using the SDTM information to derive additional populations, parameters or record-level flags in ADaM.
The additional information in ADaM can then be used for selection into the relevant efficacy TFLs including any sensitivity analysis.
The impact of COVID-19 on efficacy assessments can be very varied depending on the therapeutic area and stage of the trial. It is essential that a risk assessment and detailed discussions take place to ensure a clear plan to accurately collect any additional information required, and that this information be mapped into SDTM in a usable way. ADaM datasets may need additional population flags, record-level flags and parameters to be derived. These should all be detailed clearly in documentation so that the appropriate statistical analysis on efficacy can be conducted to answer the original questions posed by the trial.
 FDA (2020a), Guidance on Conduct of Clinical Trials of Medical Products During COVID-19 Public Health Emergency, Guidance for Industry, Investigators, and Institutional Review Boards. Updated on May 14, 2020.
 FDA (2020b), Statistical Considerations for Clinical Trials During the COVID-19 Public Health Emergency Guidance for Industry. June, 2020.
 EMA/CHMP (26 June 2020), Points to Consider on Implications of Coronavirus Disease (COVID-19) on Methodological Aspects of Ongoing Clinical Trials, EMA/158330/2020 Rev. 1.
 ICH (2019), Addendum on Estimands and Sensitivity Analysis in Clinical Trials to The Guideline on Statistical Principles for Clinical Trials: https://database.ich.org/sites/default/files/E9-R1_Step4_Guideline_2019_1203.pdf
 Statistical Issues and Recommendations for Clinical Trials Conducted During the COVID-19 Pandemic (2020): R. Daniel Meyer, Bohdana Ratitch, Marcel Wolbers, Olga Marchenko, Hui Quan, Daniel Li, Chrissie Fletcher, Xin Li, David Wright, Yue Shentu, Stefan Englert, Wei Shen, Jyotirmoy Dey, Thomas Liu, Ming Zhou, Norman Bohidar, Peng-Liang Zhao, Michael Hale.
 CDISC Guidance for Ongoing Studies Disrupted by COVID-19 Pandemic Version 1.0, 2020-04-21.
Posted by Kayley Phillpott on
6 July 2020 at 12:00 AM
Working Groups News