Toolbox on Open Research Agenda Setting

This toolbox has been developed by the Arqus European Alliance and it is the result of a collaborative effort to include stakeholders in research by co-creating research agendas. The goal is to make Open Research Agenda Setting (ORAS) knowledge and tools accesible, lower barriers for researchers, and promote connections and exchanges in research. The toolbox contains seven participatory methodologies and hands-on advice on the organisation of ORAS workshop.

Organizing the workshop

A stakeholder engagement workshop is designed to generate concrete outcomes and conclusions. In this case, these workshop outcomes are meant to inform and influence research and innovation agenda setting. While there are many different ways to organize and run a successful stakeholder engagement workshop, there are some common steps and deliberations that need to be made as the workshop is planned and organized. This chapter reviews the key steps involved in planning, organizing, running and evaluating a fruitful workshop. This starts from clearly defining the workshop purpose and topic to identifying and recruiting relevant stakeholders, designing the workshop agenda, running the workshop and making sure that the proper documentation and workshop evaluation is in place. It ends with some special considerations for organizing online workshops.

Planning the workshop


The purpose of engament and topic of the workshop

A stakeholder engagement workshop needs a clear purpose and topic in order to be successful. An important question to address early on in the planning process is: What is the desired outcome of the workshop? Examples of outcomes include a prioritized list of research questions, a map of people and resources affected by the research topic, or descriptive narratives of how a topic could potentially develop in the future (scenarios). Having a firm idea of the desired outcome will guide decisions about what types of stakeholders to engage and which workshop activities are most appropriate. It is also important to consider which level of stakeholder engagement (see ch. 2) the workshop is aiming for. A workshop with the aim to inform or consult stakeholders will likely be quite different from a workshop which aims to engage stakeholders as partners in designing and codeciding on the research agenda. Define the workshop purpose and topic at an early stage, and use that purpose to guide the rest of the planning process, including the recruitment of stakeholders, the choice of workshop methods and the arrangements for documentation.

Identifying and recruiting stakeholders

Stakeholder recruitment often takes more time and effort than anticipated, and it’s important to start early and allot adequate resources and administrative capacity to the recruitment process. The aim of the workshop will help define the scope of what kind of stakeholders to invite. Common aspects to consider include stakeholders’ education level, expertise in the topic, gender and age. The narrowness or broadness of the recruitment process will depend on the specific goals of the workshop; some workshop topics and tools function best with a homogenous group, while others thrive on groups that are diverse in terms of their level of relevant knowledge and background. Identifying intermediaries and using existing networks can be one strategy to facilitate recruitment. Depending on the topic, NGOs, professional networks and volunteer groups can be a good place to start. Stakeholder mapping can also help identify relevant participants and ensure appropriate breadth and depth in stakeholder background.

One important aspect of stakeholder recruitment is to clearly communicate workshop goals so that potential participants can see the value in participating. Stating workshop goals when recruiting can help stakeholders see how their expertise or background can contribute to the workshop. In addition, making this connection may give some stakeholders more confidence in participating, as they can more clearly see the value they can provide to the larger group.

Workshop organizers should be clear about what can be “promised” to stakeholders as output of the workshops. Some stakeholders may be satisfied with opportunities to have their voices heard or to expand their networks via the workshops. Other stakeholders may have high expectations for setting research agendas including, perhaps, unrealistic expectations about the speed and certainty with which research agendas are generated and funded. In these cases, workshop hosts should be careful to manage stakeholder expectations and focus on the more immediate and certain benefits such as expanding networks on a given topic. In addition, all stakeholders benefit from understanding how the workshop fits within the larger research agenda process as well as clear communication of workshop goals.

The composition of the stakeholder group is an important consideration. A group of stakeholders with diverse backgrounds and perspectives can lead to rich discussions, but if the workshop topic is controversial, it may require careful facilitation to avoid contentious arguments. Conversely, a group of similar stakeholders may allow for exploration of a single perspective in depth, though the discussion may illuminate fewer facets of the issue. Remember that stakeholders will also have perceptions of each other, which may affect their participation. For example, a “regular” person may be reluctant to share their views on stem cell research if placed in a group with stem cell researchers. Some organizers prefer to recruit stakeholders step-wise. After recruiting several participants, they consider what skills or backgrounds will complement these initial stakeholders when recruiting more.

Keep in mind that different types of stakeholders may have different abilities to participate. For example, pensioners may be happy to participate in a three-hour afternoon workshop, while a school teacher, though equally interested in participating, may have very limited time to participate. Similarly, scheduling a workshop during the workday may be easier for some stakeholders (if their employers support their participation), while an evening workshop may be easier for participants with less flexible jobs. The workshop should be scheduled in such a way that it’s most convenient for the target stakeholder groups.

Some workshops are one step in a longer stakeholder engagement process while others are a one-time event intended to bring people together for a single exchange of ideas. Long-term engagement requires a more careful consideration of how stakeholder time is used, how intermediate results are collected and communicated, and how stakeholder enthusiasm and engagement can be maintained. One-time events may require more flexibility in facilitating (as the group will not be previously known to the facilitator or to each other). Both approaches are seen as relevant, depending on the goals of the stakeholder engagement process.

Designing the workshop

The workshop methods should be aligned with the purpose of the workshop and the types of stakeholders that are involved. It is useful here to consider which outputs these methods will generate, and how these outputs can subsequently be used to inform/decide on the research agenda.

A good workshop plan keeps its purpose in mind and gives room to different types of stakeholders and personalities to contribute to the discussion. Often a mix of different types of activities (e.g. group work and plenary presentations/discussion) is useful. Switching between different types of activities also helps to create a good “pulse” throughout the workshop and keeps participants engaged and energized. Including an “ice breaker” exercise in the beginning often sets the stage for creative and constructive discussions later on.

When creating the detailed workshop plan, it is important to allocate realistic amounts of time to each step or activity. It is easy to be overly ambitious with regards to the number of different activities and exercises that can be included in a given amount of time. Remember that group discussions and sharing of results almost always take more time than anticipated, as do transitions between activities. In addition, when time is limited, stronger facilitation skills are needed to keep the process on track. When in doubt, allocate more time.

The workshop plan should also keep the participants’ energy in mind. Maintaining a good energy level throughout the workshop is key to a successful outcome. Many workshop exercises can be both challenging and intensive and can demand a lot of energy from the participants. Therefore, make sure to mix intensive group exercises with ones that demand less energy from the participants. When planning the schedule, also remember that participants need breaks. In addition to giving the participants “time off”, breaks are often valuable and productive time for building social relationships, networking, etc.

Especially if using a new tool or working with new facilitators, having a “dress rehearsal” can help identify parts of the workshop plan that are unclear or require extra resources. In addition, it can help determine how much time to allocate to each section of the workshop. Finally, make sure to have a Plan B ready in case something does not work as planned. This could be simple workshop exercises that participants can do in case the technology fails or a plan for which workshop activities to skip if time becomes short.

Sample workshop time plan (3 hours)

Time (minutes) Activity
15 Welcome from workshop convenor and introduction to workshop aims
15 Icebreaker activity
20 Presentation on topic
10 Break
40 Small group work activity
10 Small group summarizes their findings to present
10 Break
20 Representative from each small group presents their results in plenum
30 Facilitated plenary activity/discussion of small group work
10 Facilitator summarizes workshop results, informs on next steps, and thanks participants

Plan for documentation

When planning the workshop activities, always keep in mind what output this activity will generate and how these results will be recorded. Recording and documenting workshop output can be achieved in many different ways:

  • Workshop participants can be asked to take notes (e.g. on flipchart papers) during group discussions and exercises. These notes can then be used for the workshop report. However, keep in mind that these types of notes will often be fragmented, contextual and can be hard to decipher by the facilitators.
  • Workshop participants can be asked to present the key insights and conclusions from their discussions, which can be recorded by notetakers. These presentations often provide the highlights from the workshop exercises and can serve to contextualize and supplement the participants’ own notes.
  • A notetaker can be allocated to each of the workshop groups (see the following section).
  • Filming or audio recording can provide a high level of detail, but keep in mind that it may also inhibit participation. Any filming or recording will likely require an additional consent form from participants.

Defining the workshop roles

Some activities require or benefit from having a team of facilitators to divide up responsibilities. Example roles include:

  • Facilitator: facilitates discussion among participants, ensuring that everyone is heard and keeping the group and discussion focused on the relevant task.
  • Notetaker/recorder: records what is said and collects data during the workshop.
  • Timekeeper: ensures the group stays within the agreed upon times in the agenda.
  • Workshop convenor: welcomes participants to the workshop and introduces them to the workshop goals. This can be someone who is trusted/respected by participants but is not involved with the actual running of the workshop.

Arranging the workshop venue

The venue can have a significant impact on the comfort of stakeholders and the success of the workshop. Especially in cases of contentious topics, it’s important that the venue be “neutral” (e.g., not in the conference room of a business that advocates for one side of an issue). Visit the workshop venue in advance to know where materials are located, how to connect to internet/projector, and how to access technical support if needed. If the venue doesn’t offer coffee and snacks, then workshop organizers may need to find a way to provide this.

It is important to have a plan in place for how the workshop venue will be arranged. Make sure that the room is large enough and with proper ventilation for the group size that is convening. Room(s) should be set up in a way that’s conducive to the workshop activities. If participants will be engaging with each other, make sure they will be able to easily make eye contact with and hear each other. Keep in mind that having many small groups working in one large room can make it difficult for participants to hear each other and stay focused. In such cases, moving groups into smaller breakout rooms may be a better option.

The role of the facilitator

Researchers are often used to engaging in processes as “experts” in their field. However, when they engage as facilitators, their role is quite different. The facilitator’s role is to guide the group through the workshop exercises, and to ensure that the workshop activities lead to interesting and relevant output. As such, the facilitator is also a “learner” that needs to be open to listen and learn more about the workshop topic throughout the workshop. Conversely, the workshop facilitator does not need to be an expert in all the details of the topics that the workshop covers, but he/she should know enough to be able to guide the workshop activities in a fruitful way.

The facilitator’s role is to stay neutral and balanced. The facilitator works on behalf of the group to ensure everyone is heard and treated fairly, and he/she should not advocate for a specific position or side with one part of an argument.

During the workshop


Introducing the workshop

The workshop introduction sets the stage for the workshop and makes sure that all workshop participants know enough about both the workshop topic and the practical details to be able to engage constructively throughout the workshop. Common themes to cover in the workshop introduction are:

  • Welcome everyone
  • Introduce the workshop topic, purpose and expected outcomes
  • Present the workshop agenda at the start of the workshop so participants know what to expect, when they’ll have breaks, etc.
  • Provide information on how this workshop fits within the larger research agenda setting process.
  • Introduce the facilitation/convenor team and the workshop participants
  • Provide information about where to find the emergency exits, bathrooms, and coffee.

Facilitating the workshop activities

As the workshop participants move into the workshop exercises, the role of the facilitator is to make sure that everyone knows what to do, answer any questions, and support the group discussions:

  • Make sure to introduce each workshop activity properly. It is helpful to make an oral presentation that walks the participants through the exercise and to provide written instructions to each group.
  • Give the participants some minutes to get started.
  • Check in with each group to make sure that everyone knows what to do, and to answer any questions.
  • Respecting the agenda is a powerful way to show respect to participants. When feeling pressed for time, it can be tempting to cut breaks, but this can result in participants losing focus and motivation.
  • Be ready to improvise, but always keep the workshop purpose and results in mind. It might be a better idea to cut a workshop activity in order to provide more time for an important discussion, as long as the workshop as a whole still generates the output (and possibly decisions) that is needed.

Concluding the workshop

At the conclusion of the workshop, it is useful to bring everyone together again. Common topics to cover at the end of the workshop are:

  • Ask the participants to share their key insights and conclusions from the workshop.
  • Leave a few minutes at the end of the workshop for debriefing and reflecting. This allows participants to share any final thoughts and give feedback on the workshop process and results. This evaluation helps in understanding the workshop from the participants’ perspective and gives ideas for how to improve. There are many simple frameworks/activities for gathering feedback from participants. One simple activity (easily found with an internet search) is called “Start, Stop, Continue.”
  • Inform the participants about the next steps, e.g., how the workshop outputs will be used, when they can expect to hear from researchers/organizers, and how they can get in touch if needed.
  • Thank all participants for their time and contributions.

After the workshop


If the result of the workshop is an unfinished product, then it’s a courtesy to send the final product to participants when it is finished. It’s important to evaluate the workshop soon after its conclusion, while details are still fresh in everyone’s mind. Important topics for convenors/facilitators are:

  • How effective was each workshop activity? What could be done to make them more effective?
  • Were there any barriers to participation for any stakeholders? What could be done to overcome these barriers?
  • Was there an appropriate amount of time allocated to each activity? Was there an appropriate number and length of breaks?
  • Were there any dips in energy/engagement from stakeholders? If so, how could those be avoided in the future?
  • Did the workshop produce the desired outputs? Why or why not?
  • Lastly, it can be a courtesy to send a thank you to participants, along with any products or results from the workshop.

Special considerations for online workshop


Running workshops online requires adjustments to facilitation and activities. It can be more difficult to build rapport with participants and participants often experience a higher threshold for participating. Some specific challenges, and suggestions for overcoming them, are presented below.

ChallengeSuggestions
Familiarity with technology can introduce a power dynamicProvide training before the workshop; local assistants when needed
Limited ability to “read the room”Icebreakers increase comfort; frequent check in with participants
Remote participation can feel awkwardUse more structured techniques (to elicit ideas, etc.) when needed
Distractions (as with any online meeting)Provide frequent short breaks for checking
email, etc. Build breaks into the schedule
and honour them

Tools for Open Research Agenda Setting

Stakeholder mapping

mmmmmmmmmmmmmmmmm

Stakeholder mapping is a tool with two objectives: first, to bring researchers and stakeholders willing to engage in an ORAS process together by mapping the ecosystem linked to their field of scientific expertise and/or interestsecond, to question the researcher’s position and facilitate collaborations between academic and non-academic worlds. The main goal is to lay the groundwork for identifying stakeholders likely to be involved or add value to ORAS and (in some cases) to the research projects that follow. In addition, the tool aims to understand the relationships among stakeholders and their capacity and means to engage. This type of comprehensive analysis will aid researchers in understanding the stakeholder base and developing an effective engagement strategy. This workshop tool can also be used to develop sensitivity to these types of opening practices within the scientific community itself.

This workshop should be considered a first step and is designed to be complementary to other tools being used in ORAS. We strongly encourage organisers to use it internally to identify potential stakeholders and intermediaries.

World café

mmmmmmmmmmmmmmmmm

The World Café Method is a consultation tool that allows for an informal, personable setting and open discussion around pre-formulated questions. It can serve for brainstorming, feedbacking and enriching research agendas. The World Café method can be used to foster collaborative dialogue between different groups of stakeholders (policymakers, citizens, activists, researchers etc.), but can also be used with just one stakeholder group. It is an engaging method meant to enable open exchange and connection. It can be conducted as a half-day or a full-day workshop and be carried out with small as well as very large groups.

The caravan

mmmmmmmmmmmmmmmmm

The core concept of the caravan is to meet stakeholders directly “at home”. By travelling for a certain period of time and stopping for a few hours (between 2 to 4) in different places to meet various stakeholders and policy makers, the objective is to progressively enrich and deepen content and questions already developed. This incremental process allows stakeholders to reflect and build on ideas presented by others in the previous stops of the caravan.

Instead of a passive workshop waiting for potential stakeholders likely to be involved in the ORAS process, the caravan is an active opportunity to go directly to them thanks to a mobile unit.

Brainstorming

mmmmmmmmmmmmmmmmm

Brainstorming method is a way of generating ideas individually or in a group, that allows participants to come up with a wide range of options for solving a question or problem in a short period of time. It is based on principles of refraining from idea criticism and evaluation, focusing on quantity of ideas rather than their quality, without fear of expressing unconventional ideas, and building on each other’s ideas in order to make them better.

Systems mapping

mmmmmmmmmmmmmmmmm

In a group model building workshop, participants engage in a series of activities that help them develop a shared understanding of a particular problem or trend and create maps of

  • the causal relationships in the system that give rise to these problems and trends,
  • the connections and feedbacks underlying these problems and trends, and
  • the concerns of stakeholders in the system The system map provides an overview of how these problems and trends cut across disciplines and other boundaries and can aid in problem definition.

Real-time Delphi

mmmmmmmmmmmmmmmmm

Real Time Delphi is an advanced online form of a conventional paper-and-pencil-Delphi study. At the heart of either Delphi variant is a quantitative and qualitative questionnaire based on a specific issue/topic that is sent to experts in a given field. Therefore, this tool is especially dedicated to stakeholders with deeper knowledge in the research topic. Thus, the Real Time Delphi method is not a workshop, but a particular form of iterative survey. Using specific software, the Real Time Delphi method provides a structured group communication platform that allows the participants to evaluate and change their answers in real-time. This online process usually lasts 2-8 weeks, depending on the scope of the survey. The main purpose of a Real-Time  Delphi is forecasting and scenario development, but it can also be used for idea aggregation, consensus-building or decision-making.

Four quadrant scenarios: exploring alternative futures

mmmmmmmmmmmmmmmmm

In this workshop, participants will use a simple, well-established qualitative scenario method to map and plan for key future uncertainties. Using the four quadrant-scenario method – also known as “Shell scenarios” – participants will identify important and uncertain future trends and then create four different scenarios that explore how the future might play out depending on these trends. They will then use these scenarios to “backcast” relevant themes for research and innovation agenda setting in the present.

Newsletter

Don’t miss any news, calls or opportunities from Arqus.
Subscribe now! The future begins today.

    Subscribe


    You can change your mind at any time by clicking the unsubscribe link in the footer of any email you receive from us.

    We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

    Controller: University of Granada (as Coordinator of the Arqus European University Alliance). Legal basis: Arqus is entitled to process your data under the provisions of Article 6.1. (a) of the GDPR: “the data subject has given consent to the processing of his or her personal data for one or more specific purposes”. Purpose: to manage your subscription to mailing lists and to periodically send you the requested information by electronic means. Recipients: Mailchimp. Rights: access, object, rectification, erasure, restriction of processing, data portability. Additional information here.

    Select our profile:

    Apart from the general information, what specific news would you like to receive from Arqus?

    Controller: University of Granada (as Coordinator of the Arqus European University Alliance). Legal basis: Arqus is entitled to process your data under the provisions of Article 6.1. (a) of the GDPR: “the data subject has given consent to the processing of his or her personal data for one or more specific purposes”. Purpose: to manage your subscription to mailing lists and to periodically send you the requested information by electronic means. Recipients: Mailchimp. Rights: access, object, rectification, erasure, restriction of processing, data portability. Additional information here.