Progress and achievements
Evaluating procedures or ways of working is only part of the picture. It is also important to assess whether people were able to engage with the issues and with each other; what each contributed and the decisions they influenced; what they achieved, and/or whether this was what they set out to achieve 
Patient or clinician involvement in setting research priorities might change the patients or clinicians themselves: what they learn about research and each other. It might influence the research priorities or the costs of identifying them, and ultimately the commissioned research or the costs of conducting it.
The success of a Priority Setting Partnership rests in large part on enabling participants to debate fairly amongst themselves. They thus need a situation that helps them to
- engage with the issues
- share a common language, using expressions in the same way
- speak without being suppressed or excluded
- have equal opportunities to introduce new ideas
- accept no force other than strong argument supported by convincing evidence
- understand that their own views come from a particular perspective
- listen to each other
Evaluation is therefore highly relevant to assess this and a questionnaire can capture some of the issues by asking for example:
- Were there terms or phrases that participants did not understand?
- Did participants feel free to speak, and did they feel heard?
- What did participants learn from each other?
However, questionnaires tend to miss how positions, values and decisions are shared during an encounter 
. If funds allow, an experienced independent observer may be more sensitive to the dramatic turns of some debates and provide invaluable feedback on these. Input and influence
: Neither having a seat at the table nor responding to a request for information guarantee being able to influence decisions. Whether this is achieved may be seen from observing people’s interactions, inspecting and comparing records of contributions and of decisions, or by asking contributors or those alongside them whether influence was exerted. Influence may be immediate, by commanding attention during a discussion, or delayed, by influencing what appears on a list of priorities and ultimately when one of these priorities is translated into funded research.
Influence at all these stages was observed in an evaluation of public involvement in the NIHR Health Technology Assessment Programme. To find out more click here
NIHR Health Technology Assessment Programme
Taken from Oliver S, Armes D, Gyte G. Public involvement influences a national research agenda. The Patient: patient centred outcomes research. (In press).
"Routine management records of the HTA programme were examined for public influence of research topics. The nature and influence of contributions from the public were compared with those of other experts. Structured observations of Advisory panel meetings investigated how discussion and decisions related to patient and public perspectives and how panel members responded to public input to the programme. Semi-structured interviews gathered perceptions of staff and Advisory Panel members.
"[This work found that] public involvement has influenced decisions about research commissioned by the HTA programme with only relatively minor changes to the procedures and resources for managing the programme. This results in outcomes research that incorporates patients' and publics' preferences and values being freely available for evidence-informed health services."
Priority Setting Partnerships' records of uncertainties can be analysed to find out who makes original suggestions, whether these suggestions attract support from others during discussions or rounds of voting, and whose suggestions appear in the top 10 priorities. An independent observer can record who initiates each discussion and who contributes to it. An independent interviewer can ask subsequently how participants see their role, contributions and influence and how they see that of others.Achievements
: examples of questions for finding out what has been achieved include 
- Do participants have greater insight into priority setting and each others' priorities?
- Did priorities change or did participants have more confidence in priorities, thereby making clear that time was well spent and shared decisions not merely 'rubber stamping'?
- Were decisions made with appropriate use of evidence, consistent reasoning, with priority setting integrated into wider systems and decision-making aligned with agreed goals and procedures?
- Were participants satisfied with the process and willing to continue participating and accept decisions even if they did not agree with all the outcomes?
Whether or not these outcomes have been achieved can be addressed in interviews or by analysing routine records or other documents.Partnership goals
: assessing the extent to which a partnership has reached the goals set by its own participants is vital. Successful partnerships need to be clear about what they are trying to achieve. If partners develop and agree clearly defined objectives at the outset, whether or when these objectives are met can be a central part of an evaluation.
Commentary on the goals and achievements of the JLA Asthma Priority Setting Partnership can be viewed here
Extract from Petit-Zeman S. Society Guardian, Monday 21 May 2007:
"In August 2004, in the imposing marbled halls of London's Royal Society of Medicine (RSM), doctors, patients and carers met to discuss the current state of play in asthma research and treatment. By the end of the day there was collective agreement that these traditionally separate bedfellows might do well to forge a new relationship: working together to agree research priorities... It has taken almost three years to move the [JLA] asthma partnership to where it is now: an agreed list of ten research priorities that matter most to both sides."Click here to read the full article.
An independent reflective report of the Asthma Priority Setting
Partnership can be downloaded here
Renn, O., Webler, T. and Wiedemann, P. (1995) Fairness and Competence in Citizen Participation: Evaluating models for environmental discourse. Kluwer Academic: London.
Harvey M. Drama, Talk, and Emotion: Omitted Aspects of Public Participation Science, Technology & Human Values March 2009 34: 139-161
Sibbald SL, Singer PA, Upshur R and Martin DK. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Services Research 2009, 9:43 doi:10.1186/1472-6963-9-43.