This article has been prepared by Miranda Davies MFPM 1,2, Samantha Lane 1,2 and Saad Shakir FFPM 1,2
1Drug Safety Research Unit, United Kingdom, 2University of Portsmouth, United Kingdom
It is provided for information and does not constitute advice or represent official FPM views or policy.
How to cite:
Davies, M et al (2020), ‘Principles of benefit-risk assessment: A focus on some practical applications’, Journal of the Faculty of Pharmaceutical Medicine, 10 November 2020. Available at: https://www.fpm.org.uk/journals/principles-of-benefit-risk-assessment-a-focus-on-some-practical-applications/ (Accessed: <date>).
Assessing benefit versus risk is integral to decision making throughout the life cycle of any drug/product. Over recent decades the method of conducting benefit-risk (BR) assessments has evolved from a non-structured and often mainly qualitative process to one that has become increasingly systematic and more transparent . In 1998, the seminal report by the Council for International Organizations of Medical Sciences (CIOMS) was produced, entitled “Benefit–risk balance for marketed drugs: evaluating safety signals” . The authors acknowledged that there was no structured and harmonized approach to BR assessment at that time. The report also included reference to how benefits and risks could be scored (or ranked) based on the properties of each outcome.
In the early 2000s two papers generated additional interest in quantitative BR methods using weighted outcomes [3, 4]; these were followed by numerous initiatives to explore BR methodologies further. In 2006 the Pharmaceutical Research and Manufacturers of America (PhRMA) developed a structured framework: the Benefit Risk Action Team (BRAT) Framework . A few years later, the European Medicines Agency (EMA) began a three-year “Benefit-risk methodology project” with the aim of identifying decision-making models to improve consistency and transparency. In the 2012 report from this project, “Work package 4 report: Benefit-risk tools and processes,” it was suggested that application of the PrOACT-URL model (a descriptive framework) may be sufficient. PrOACT-URL is a generic decision-making guide with eight steps: Problems, Objectives, Alternatives, Consequences, Trade-offs, Uncertainty, Risk attitudes, and Linked decisions. The framework itself is generic and can be applied to any decision-making problem, but the EMA’s Benefit-Risk Project adapted PrOACT-URL to potential use for decision-making in medicines  . Alternatively the report suggested that development of a multi criteria decision analysis (MCDA) model may be useful for difficult or contentious cases, i.e. when the BR balance is marginal or in the case of many conflicting attributes [7, 8].
In 2009 the FDA began an initiative to develop a structured method for BR assessments, and concluded that a qualitative descriptive approach could be sufficient while acknowledging that quantification of certain components of the assessment was important to support decision-making . The PROTECT project (2009-2015) funded by the Innovative Medicines Initiative (IMI), was a public-private partnership between the European Union and the European Federation of Pharmaceutical Industries and Associations (EFPIA) and aimed to address limitations of current methods used in pharmacovigilance and pharmacoepidemiology and to significantly strengthen the monitoring of benefit-risk of medicines marketed in Europe . One of the BR methods used in the PROTECT project and applied to various case studies was the BRAT framework . This is a descriptive framework, based on six main steps which are shown below . In this framework the individual benefits and risks are not integrated per se but are assessed separately and clearly, so that a comparison can be more readily achieved. This was a conscious proposal to avoid synthesising data into complex statistical models .
In response to the ongoing pandemic and the urgent need to identify effective and safe treatments for COVID-19 disease, the Drug Safety Research Unit (DSRU, UK) conducted four BR assessments on potential COVID-19 therapies (remdesivir, lopinavir/ritonavir, hydroxychloroquine, convalescent plasma) [12-15]. All of these BR assessments were conducted in the context of treatment of patients with severe COVID-19 disease (i.e. amongst those patients who required hospitalisation). The underlying approach was based on the BRAT framework, however where sufficient data was available further quantitative analysis was conducted.
As with similar descriptive frameworks, the key anticipated benefits and risks associated with each drug were identified by clinicians with pharmacovigilance expertise, based on literature search findings (including clinical trial protocols for relevant study endpoints, especially relevant for identifying key anticipated benefits of treatment). Key outcomes refer to those outcomes likely to impact the BR assessment, which are then visually presented in the form of a value tree, in which both the benefits and risks are ranked (separately) based on levels of seriousness.
BRAT Six-Step Process 
|Define decision context||• Define drug, dose, formulation, indication, patient population, comparator(s), time horizon for outcomes, perspective of the decision makers (regulator, sponsor, pateint, or physician).|
|Identify outcomes||•Select all important outcomes and create the initial value tree. •Define a preliminary set of outcome measures/endpoints for each. •Document rationale for outcomes included/excluded.|
|Identify data sources||•Determine and document all data sources (e.g. clinical trials). •Extract all relevant data for the data source table, including detailed references and any annotations, to help the subsequent interpretations create summary measures.|
|Customise framework||•Modify the value tree on the basis of further review of the data and clinical expertise. •Refine the outcome measures/endpoints. May include tuning of outcomes not considered relevant to a particular benefit-risk assessment or that vary in relevance by stakeholder group.|
|Assess outcome importance||•Apply or assess any ranking or weighting of outcome importance to decision makers or other stakeholders.|
|Display & interpret key benefit-risk metrics||•Summarise source data in tabular and graphical displays to aid review and interpretation. •Challenge summary metrics, review source data, and identify and fill any information gaps. •Interpret summary information.|
Where data was available for each risk and benefit from controlled studies, this was summarised and differences in risk calculated between the drug of study and the comparator group (placebo, standard of care, other treatment). As mentioned above, this framework also lends itself (where enough data is available) to perform some additional “add-on” quantitative analysis which may be helpful where there are multiple outcomes of interest involved or the difference is considered marginal. To facilitate this additional quantitative analysis, both benefits and risks are ranked on a single continuum and preferential weights are applied for each outcome. This forms the basis of many quantitative analyses, i.e. the combination of weighted benefits and risks to produce an overall metric. As time goes on and more study results are made available for the ongoing COVID-19 potential treatment options, it is possible to calculate the weighted net clinical benefit (wNCB). In this approach, benefits have a positive contribution to the wNCB and risks have a negative contribution; the overall wNCB is positive (benefit outweighs the risk) where wNCB is >0 .
Points to Consider:
There are some points to consider throughout the application of both descriptive and quantitative approaches involving weighted outcomes, some of which are described below. Firstly, how should studies with relevant data be screened to ensure that only those which are sufficiently scientifically robust in both their design and conduct are included in the assessment? During the COVID-19 pandemic, there have been many published studies in which interpretation of study findings is potentially limited by several factors, including the lack of a control group, small sample sizes, confounding by co-administration of alternative anti-viral treatments and other important baseline risk factors that were not accounted for in the analysis. These limitations are often apparent in observational studies, with their associated risk of bias and residual confounding. One option would be to apply a set of predefined criteria, such as the “GRADE” criteria which are a systematic and transparent approach to grading the quality of evidence [17, 18]. Use of such criteria standardises decisions made on the quality of studies contributing to the BR assessment. Whichever system is used to assess quality of individual studies, the objective is to identify studies which may lead to a less reliable overall assessment. Accordingly, and integral to conducting BR assessments on various COVID-19 treatment options, is the need to assess the quality of individual studies through a critical appraisal and identification and description of limitations. Conclusions regarding the reliability of individual study findings should be considered in the overall BR evaluation.
Furthermore, it may not always be straight forward to rank outcomes and apply associated weightings. For example, how is a very serious but rare outcome ranked? The individual nature of the event including the prognosis and associated mortality should be considered alongside the incidence of the event when making clinical judgements about rankings. Ideally this should be done independently by at least two clinicians. This raises the question as to how patient perspectives can be incorporated into this process. Over recent years there have been multiple initiatives to explore patient perspectives in decision making processes and there has been an acknowledgement that the trade-offs made by patients may differ from those made by clinical experts [19, 20]. Despite these initiatives, patient perspectives are not routinely incorporated into BR assessments. Who should be consulted, and how and when should this data be collected? The EMA supported a study to evaluate methods for including patient perspectives and preferences in decision making. The “VALUE Study” (“Value and Utilities among European Patients”) was conducted among a population of patients with multiple sclerosis, to assess the use of the Measuring Attractiveness by a Categorical Based Evaluation Technique (MACBETH) to measure patient preferences for treatment outcomes , and the question of how to achieve more structured patient involvement in this process was included in the EMA road map to 2015 [20, 22].
Finally, how frequently should BR assessments be conducted? For approved medicines there are well defined regulatory milestones during which BR is formally assessed, with the assumption that overall BR is being continually monitored. At other times however, such as during the COVID-19 pandemic, treatments are given to patients based on very little or no evidence either in the absence of findings from more robust randomised studies or because these results take time to accrue. In a fast-changing environment such as this, how often should systematic BR assessments be reassessed to incorporate new or changing evidence as this becomes available? The acknowledgment that BR is based on a living body of evidence and as such is a dynamic evaluation that will change over time is a fundamental principle of BR assessment. Essentially, updated BR assessments should be produced for those drugs under ongoing study as more data becomes available, facilitating the addition of supportive quantitative analyses. The increasing transparency and structure seen in assessing BR allows for a clearer understanding of this process by all parties and likely facilitates more meaningful comparison between various treatment options.
- Juhaeri J. Benefit-risk evaluation: the past, present and future. Ther Adv Drug Saf. 2019;10:2042098619871180. doi:10.1177/2042098619871180.
- (CIOMS) CfIOoMS. Benefit–risk balance for marketed drugs: evaluating safety signals. 1998.
- Holden WL, Juhaeri J, Dai W. Benefit-risk analysis: a proposal using quantitative methods. Pharmacoepidemiol Drug Saf. 2003;12(7):611-6. doi:10.1002/pds.887.
- Holden WL, Juhaeri J, Dai W. Benefit-risk analysis: examples using quantitative methods. Pharmacoepidemiol Drug Saf. 2003;12(8):693-7. doi:10.1002/pds.794.
- Coplan PM, Noel RA, Levitan BS, Ferguson J, Mussen F. Development of a framework for enhancing the transparency, reproducibility and communication of the benefit-risk balance of medicines. Clin Pharmacol Ther. 2011;89(2):312-5. doi:10.1038/clpt.2010.291.
- PROTECT. Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium. PrOACT-URL Appraisal. .
- EMA. Benefit-risk methodology project – Revision 1. 2012.
- Tervonen T, Naci H, van Valkenhoef G, Ades AE, Angelis A, Hillege HL et al. Applying Multiple Criteria Decision Analysis to Comparative Benefit-Risk Assessment: Choosing among Statins in Primary Prevention. Med Decis Making. 2015;35(7):859-71. doi:10.1177/0272989X15587005.
- Administration FaD. Structured Approach to Benefit-Risk Assessment in Drug Regulatory Decision-Making 2013.
- PROTECT. The Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium.
- Levitan BS, Andrews EB, Gilsenan A, Ferguson J, Noel RA, Coplan PM et al. Application of the BRAT framework to case studies: observations and insights. Clin Pharmacol Ther. 2011;89(2):217-24. doi:10.1038/clpt.2010.280.
- Davies M, Osborne V, Lane S, Roy D, Dhanda S, Evans A et al. Remdesivir in Treatment of COVID-19: A Systematic Benefit-Risk Assessment. Drug Saf. 2020;43(7):645-56. doi:10.1007/s40264-020-00952-1.
- Osborne V, Davies M, Lane S, Evans A, Denyer J, Dhanda S et al. Lopinavir-Ritonavir in the Treatment of COVID-19: A Dynamic Systematic Benefit-Risk Assessment. Drug Saf. 2020;43(8):809-21. doi:10.1007/s40264-020-00966-9.
- Osborne V DM, Dhanda S, Roy D, Lane S, Evans A, , Shakir SAW. Systematic benefit-risk assessment for the use of chloroquine or hydroxychloroquine as a treatment for COVID-19: Establishing a dynamic framework for rapid decision-making. medRxiv pre-print platform. 2020.
- Davies M LS, Evans A, Denyer J, Dhanda S, Roy D, Osborne V, Shakir SAW. Convalescent Plasma in treatment of COVID-19: A review of evidence for a living systematic benefit-risk assessment. medRxiv pre-print platform. 2020.
- Osborne V, Davies M, Roy D, Tescione F, Shakir SAW. Systematic benefit-risk assessment for buprenorphine implant: a semiquantitative method to support risk management. BMJ Evid Based Med. 2020. doi:10.1136/bmjebm-2019-111295.
- Schunemann HJ, Mustafa RA, Brozek J, Steingart KR, Leeflang M, Murad MH et al. GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy. J Clin Epidemiol. 2020;122:129-41. doi:10.1016/j.jclinepi.2019.12.020.
- Schunemann HJ, Mustafa RA, Brozek J, Steingart KR, Leeflang M, Murad MH et al. GRADE guidelines: 21 part 2. Test accuracy: inconsistency, imprecision, publication bias, and other domains for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables. J Clin Epidemiol. 2020;122:142-52. doi:10.1016/j.jclinepi.2019.12.021.
- Muhlbacher AC, Juhnke C. Patient preferences versus physicians’ judgement: does it make a difference in healthcare decision making? Appl Health Econ Health Policy. 2013;11(3):163-80. doi:10.1007/s40258-013-0023-3.
- Muhlbacher AC, Juhnke C, Beyer AR, Garner S. Patient-Focused Benefit-Risk Analysis to Inform Regulatory Decisions: The European Union Perspective. Value Health. 2016;19(6):734-40. doi:10.1016/j.jval.2016.04.006.
- Methodologie EMAW-BR. Evaluating methods to capture stakeholder preferences: MACBETH: A Non-numerical Method for Eliciting Preferences. .
- EMA. Road map to 2015: The European Medicines Agency’s contribution to science, medicines and health. Adopted by the Agency’s Management Board on 16 December 2010.