The science of combining apples and oranges: Joint EFSA/EBTC scientific colloquium on evidence integration in risk assessment
Lisbon, Portugal, 25-26 October 2017
Leading scientists took part in a joint EFSA/EBTC scientific colloquium to explore future directions for evidence integration in human risk assessment of chemicals. The 82 participants, from 19 countries, brought a diversity of experience, methodological approaches and practical expertise to the lively discussions.
EFSA’s Didier Verloo co-chaired the meeting. He said: “We were lucky to have here some major game-changers and leading minds from the relatively small but hugely impactful world of risk assessment methodologies.
“We focused on key themes in our practice of evidence integration – from our use of various applications to adapting different methods and frameworks. The deepening pool of approaches is allowing assessors to better use all of the available evidence in an assessment quantitatively and qualitatively, taking account of bias, overall uncertainty and sensitivity, for example.”
Co-Chair Katya Tsaioun, of EBTC at the Johns Hopkins Bloomberg School of Public Health, added: “Recognising where we are now helped to set the scene and to map out themes in our future directions. We underlined the importance of standardised terminology and concepts, learning and testing, training and the need for practice. But the assessment community needs to communicate and share its approaches, refining them along the way.”
The opening and closing plenary sessions were webstreamed live. A recording is available below:
The keynote speakers were Donald Rubin and Stijn Vansteelandt (causal inference), Julian Higgins and Sofia Dias (evidence synthesis, meta-analysis), Kris Thayer and Holger Schunemann (evidence synthesis, GRADE and GRADE-based methods), and Marc Aerts, Wout Slob and Matthew Wheeler (dose-response modelling).
The 82 participants came from 15 European countries, Canada, Qatar, Tunisia and the USA. They included EFSA staff and external experts from EFSA’s panels and working groups, 6 EBTC members, and representatives from 16 national authorities and 23 universities/research institutes. Representatives of international organisations, NGOs and private sector organisations also took part.
Dr Verloo summed up: “There is a real desire not only for guidance to make evidence integration transparent and effective, but also for real-life examples to help us implement them in practice.”
Dr Tsaioun said: “We need to be open-minded, flexible and iterative so we can learn, try, do and improve as we go forward.”
The event was the 23rd in the EFSA scientific colloquium series and the first to be jointly organised with EBTC at Johns Hopkins Bloomberg School of Public Health.
Evidence-based scientific assessments involve applying structured and standardised approaches to minimise bias and maximise impartiality and transparency in the process for collecting, evaluating and combining evidence relevant to well-formulated research questions, according to pre-defined protocols. These approaches are well-established for healthcare intervention questions and their value has been extensively acknowledged also in the field of chemical risk assessment, where their application continues to be explored.
In evidence-based scientific assessments, evidence synthesis is the step that occurs after appraising the validity of the individual studies selected for the assessment. In evaluations of the efficacy of therapeutic interventions, this is usually done through meta-analysis, which encompasses statistical methods for combining data from similar, readily-comparable studies.
In hazard identification and characterisation for chemical risk assessment, the underlying evidence bases are diverse and not readily comparable. Unlike in medicine, in this research field heterogeneity of evidence stems from not only varying degrees of validity and precision of studies and diverse data types (e.g. individual Vs aggregated), but also from dissimilar designs/settings/models, endpoints, routes of exposure, or evidence streams (human observational studies, experimental animal studies, in vitro and computational models data). As such, a process for combining evidence not only within- but also across evidence streams is needed. This process is defined as ‘evidence integration’ and is particularly relevant for assessing effects caused by exposure to a chemical substance (hazard identification), and for deriving health-based guidance values through dose-response modelling (hazard characterisation).
Evidence integration is a recognised challenge in evidence-based risk assessment and to be conducted soundly, it has to draw on valid approaches to collecting and evaluating evidence in prior steps in the risk assessment, and the integration methodology itself needs to be valid. Different methods for integrating evidence exist, ranging from approaches based on expert judgement, through structured qualitative methods, to complex quantitative methods.
Objectives of the Colloquium
EFSA and EBTC have organised a colloquium, guided by experts in the field, with the goal of developing a multi-stakeholder understanding of the best practices for evidence integration in human risk assessment of chemicals, with a specific focus on hazard identification and on combining multiple studies and endpoints for dose-response modelling in hazard characterisation.
We will discuss current practice, challenges, recent developments and innovative approaches to integrating evidence within and across evidence streams and to combining multiple studies and endpoints. Even if the case-studies and examples discussed throughout the colloquium will be specific for the field of chemicals, we aim at addressing these methodological aspects from a broad, cross-cutting perspective, relevant to other research contexts (e.g. dietary reference values).
Relevant EFSA projects
EFSA PROMETHEUS (PROmoting METHods for Evidence Use in Scientific Assessments) project: https://www.efsa.europa.eu/en/methodology/evidence
EFSA (European Food Safety Authority) Scientific Committee, 2016. Guidance on Uncertainty in EFSA Scientific Assessment - Draft version for internal testing. Available at https://www.efsa.europa.eu/en/topics/topic/uncertainty
EFSA Guidance on the use of the weight of evidence approach in scientific assessments: https://www.efsa.europa.eu/it/efsajournal/pub/4971
EFSA Guidance on the assessment of the biological relevance of data in scientific assessments: https://www.efsa.europa.eu/en/efsajournal/pub/4970
Structure of the meeting
Colloquia are not consensus meetings: their objective is to convene leading scientists from Europe and beyond. Rather than offering a series of lectures, this Colloquium will provide ample opportunity for an exchange of expert opinions and scientific debate. Following the opening plenary session with introductory key note speeches, the meeting will be structured to enable participants to reach conclusions and make recommendations in small groups, focusing the discussions on four specific topics.
The four discussion groups (DGs) will focus on the following themes:
- DG 1 – Qualitative methods for integrating evidence within- and across evidence streams for hazard identification
- DG 2 – Bias-adjusted meta-analysis
- DG 3 – Quantitative approaches to combining evidence across evidence streams for hazard identification
- DG 4 – Using multiple endpoints and multiple studies for dose-response modelling: quantitative approaches
The outcome of the discussion groups will be presented and discussed in a final plenary session to formulate conclusions of the Colloquium and, as appropriate, recommendations. The outcomes of the Colloquium will be summarised in an overall report after the meeting.