Progress and achievements

Evaluating procedures or ways of working is only part of the picture. It is also important to assess whether people were able to engage with the issues and with each other; what each contributed and the decisions they influenced; what they achieved, and/or whether this was what they set out to achieve ^[1].

Patient or clinician involvement in setting research priorities might change the patients or clinicians themselves: what they learn about research and each other. It might influence the research priorities or the costs of identifying them, and ultimately the commissioned research or the costs of conducting it.

The success of a Priority Setting Partnership rests in large part on enabling participants to debate fairly amongst themselves. They thus need a situation that helps them to:

• engage with the issues
• share a common language, using expressions in the same way
• speak without being suppressed or excluded
• have equal opportunities to introduce new ideas
• accept no force other than strong argument supported by convincing evidence
  
• understand that their own views come from a particular perspective
• listen to each other

Evaluation is therefore highly relevant to assess this and a questionnaire can capture some of the issues by asking for example:

• Were there terms or phrases that participants did not understand?
• Did participants feel free to speak, and did they feel heard?
• What did participants learn from each other?  

However, questionnaires tend to miss how positions, values and decisions are shared during an encounter ^[2]. If funds allow, an experienced independent observer may be more sensitive to the dramatic turns of some debates and provide invaluable feedback on these.

Input and influence: Neither having a seat at the table nor responding to a request for information guarantee being able to influence decisions. Whether this is achieved may be seen from observing people’s interactions, inspecting and comparing records of contributions and of decisions, or by asking contributors or those alongside them whether influence was exerted. Influence may be immediate, by commanding attention during a discussion, or delayed, by influencing what appears on a list of priorities and ultimately when one of these priorities is translated into funded research.


Priority Setting Partnerships' records of uncertainties can be analysed to find out who makes original suggestions, whether these suggestions attract support from others during discussions or rounds of voting, and whose suggestions appear in the top 10 priorities. An independent observer can record who initiates each discussion and who contributes to it. An independent interviewer can ask subsequently how participants see their role, contributions and influence and how they see that of others.

Achievements: examples of questions for finding out what has been achieved include ^[3]:

• Do participants have greater insight into priority setting and each others' priorities?
• Did priorities change or did participants have more confidence in priorities, thereby making clear that time was well spent and shared decisions not merely 'rubber stamping'?
• Were decisions made with appropriate use of evidence, consistent reasoning, with priority setting integrated into wider systems and decision-making aligned with agreed goals and procedures?
• Were participants satisfied with the process and willing to continue participating and accept decisions even if they did not agree with all the outcomes?

Whether or not these outcomes have been achieved can be addressed in interviews or by analysing routine records or other documents.

Partnership goals: assessing the extent to which a partnership has reached the goals set by its own participants is vital. Successful partnerships need to be clear about what they are trying to achieve. If partners develop and agree clearly defined objectives at the outset, whether or when these objectives are met can be a central part of an evaluation.

---

¹ Renn, O., Webler, T. and Wiedemann, P. (1995) Fairness and Competence in Citizen Participation: Evaluating models for environmental discourse. Kluwer Academic: London.

² Harvey M. Drama, Talk, and Emotion: Omitted Aspects of Public Participation Science, Technology & Human Values March 2009 34: 139-161

³ Sibbald SL, Singer PA, Upshur R and Martin DK. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Services Research 2009, 9:43 doi:10.1186/1472-6963-9-43.