In February 2019, the NIHR Greater Manchester Patient Safety Translational Research Centre (NIHR Greater Manchester PSTRC) and the Information Commissioner’s Office (ICO) co-funded two Citizens’ Juries, the first in Coventry and the second in Manchester.
What is a Citizens’ Jury?
A Citizens' Jury is a method of deliberation developed by the Jefferson Center, a US-based charity, where a small group of people (between 12 and 24), representative of the demographics of a given area, comes together to deliberate on a specific issue. Our set of two juries focus on the use of Artificial Intelligence (AI) to make decisions, across a number of different situations.
Decision making and explainable Artificial Intelligence
AI is being used to train computers how to do many things, for example, making medical diagnoses. Scientific studies have shown that AI can diagnose skin cancer from skin images as reliably as dermatologists. But how do we know whether we can trust a diagnosis? If we are given it by a human doctor, we can ask for an explanation. However, the most advanced AI systems are very complex and therefore they may not be able to explain to humans how they reach their decisions. We could require that these systems are always able to give an explanation that people can understand, but doing so may come at the expense of less accurate decisions. So, there is a tradeoff between accuracy and ‘explainability’ of AI systems.
The set of two Citizens’ Juries consider this tradeoff in four different scenarios:
- Healthcare: diagnosis of acute stroke
- Healthcare: finding matches between kidney transplant donors and recipients
- Criminal Justice: deciding which offenders should be referred to a rehabilitation programme
- Recruitment: screening job applications and making shortlisting decisions
The following video was filmed at the citizens’ jury held at the Bridgewater Hall in Manchester.
Jury design documentation
|No.||Name||Brief description||Reviewed by Oversight Panel?||File for download|
|A1||AI Citizens' jury specification||A specification of the design for the citizens' juries, including the jury questions and juror selection criteria.||Yes||A1 AI citizens jury specification|
|A2||Brief for expert witnesses||A brief provided to all five expert witnesses to guide their presentations to the juries, including specific brief for each named witness.||Yes||A2 Brief for expert witnesses|
|A3||Citizens jury oversight panel brief||A brief to the three members of the oversight panel (Soren Holm, Reema Patel, David Leslie) describing their role to monitor bias.||Yes||A3 Citizens jury oversight panel brief|
|A4||Oversight panel signed questionnaires||The set of forms comleted and signed by the three members of the oversight panel with their assessments and statements on bias.||N/A||A4 Oversight panel signed questionnaires|
|A5||Jury recruitment questionnaire||A copy of the online form completed by people applying to be Coventry jurors (an equivalent form was used for Manchester)||Yes||A5 Jury recruitment questionnaire|
|A6||Start of jury questionnaire||The questionnaire that all jurors completed at the start of day 1 of the jury process||Yes||A6 Start of jury questionnaire|
|A7||End of jury questionnaire||The questionnaire that all jurors completed at the end of day 5 of the jury process||Yes||A7 End of jury questionnaire|
|A8||Daily bias questionnaire||A form used to capture feedback from the jurors, particularly about potential bias, at the end of days 1, 2,3 and 4.||No||A8 Jefferson Center bias questionnaire|
|No||Name||Brief description||Reviewed by Oversight Panel?||File for download|
|B1||Jurors ring binder contents||A folder of materials produced by Citizens Juries c.i.c. and the Jefferson Center, printed out and provided in a ring binder to each jury member. It had 16 sections including a table of contents (section 0), and all the slides from the expert witnesses. Jefferson Center copyright material has been redacted from this downloadable version.||All witness slides were reviewed||B1 Jurors ring binder contents|
|B2||Scenario handouts||Paper handouts provided to jury members during the course of the five-day event, namely the four scenarios with associated questions that the juries considered.||Yes||B2 Scenario handouts|
|No||Name||Brief description||File for download|
|C1||Coventry jurors report||The report of the jury conclusions produced by Kyle Bozentko of the Jefferson Center on day 5 of the Coventry jury with 18 jurors||C1 Coventry Jurors report|
|C2||Manchester jurors report||The report of the jury conclusions produced by Kyle Bozentko of the Jefferson Center on day 5 of the Manchester jury with 18 jurors||C2 Manchester Jurors report|
|C3||AI citizens' juries management summary||A two-page report summarising the findings of the two citizens' juries||C3 AI citizens' juries management summary|
|C4||AI citizens' juries report||The report of the design and findings of the two citizens' juries (with functioning hyperlinks to report sections)||C4 AI citizens' juries report|
|C5||Juries word cloud||A word cloud generated automatically using the answers given by the 36 jurors from the two juries in the end-of-jury questionnaire to the question "overall, what was it like participating in the citizens' jury over the five days? Please say 3 things in 3 words to sum up your experience."||C5 Juries word cloud|
|C6||Coventry start-of-jury questionnaire results||A summary of the answers given for the questionnaire completed by the Coventry jurors at the start of day 1||C6 Coventry start-of-jury questionnaire results|
|C7||Manchester start-of-jury questionnaire results||A summary of the answers given for the questionnaire completed by the Manchester jurors at the start of day 1||C7 Manchester start-of-jury questionnaire results|
|C8||Coventry scenario question results||A summary of the answers given by the Coventry jury for the four scenarios on days 3 and 4||C8 Coventry scenario question results|
|C9||Manchester scenario question results||A summary of the answers given by the Manchester jury for the four scenarios on days 3 and 4||C9 Manchester scenario question results|
|C10||Coventry general question results||A summary of the answers given by the Coventry jury for the general questions considered on day 5||C10 Coventry general question results|
|C11||Manchester general question results||A summary of the answers given by the Manchester jury for the general questions considered on day 5||C11 Manchester general question results|
|C12||Coventry daily bias questionnaire results||A zip file of 4 spreadsheets of the results from the daily bias questionnaire completed by jurors at the end of days 1, 2, 3 and 4 of the Coventry jury (on day 5 the jury completed the end-of-jury questionnaire instead)||C12 Coventry daily bias questionnaire results (zip file)|
|C13||Manchester daily bias questionnaire results||A zip file of 4 spreadsheets of the results from the daily bias questionnaire completed by jurors at the end of days 1, 2, 3 and 4 of the Manchester jury (on day 5 the jury completed the end-of-jury questionnaire instead)||C13 Manchester daily bias questionnaire results (zip file)|
|C14||Coventry end-of-jury questionnaire results||A spreadsheet of the results from the end-of-jury questionnaire completed by Coventry jurors at the end of day 5||C14 Coventry end-of-jury questionnaire results|
|C15||Manchester end-of-jury questionnaire results||A spreadsheet of the results from the end-of-jury questionnaire completed by Manchester jurors at the end of day 5||C15 Manchester end-of-jury questionnaire results|
- Other than where specified, the main author of the documentation above was Dr Malcolm Oswald, Director of Citizens Juries c.i.c.. Reviewers varied depending on the naure of the document.
- The Oversight Panel brief was to review the main jury design documentation, but not the jury outputs.
We have published a number of blog posts about these Citizens' Juries:
- Should Artificial Intelligence give reasons for decisions even if it affects accuracy – Citizens’ Juries deliberate (27 February 19)
- Citizens' Juries: Using public opinion on Artificial Intelligence to inform policy (14 February 19)
- If a computer gives you a diagnosis, should it also give you an explanation? (15 November 2018)