The purpose of this report is to provide details on the development of an Augmented Reality (AR) tool which integrates digital storytelling for participatory research knowledge translation on climate-related sexual health experiences among youth (aged 16-25 years) in climate-affected regions in Kenya. AR is a technology field that involves the overlay of computer-generated virtual images on the real world, in such a way that the virtual content is aligned with real-world objects, and can be viewed and interacted with in real-time.1 In contrast to Virtual Reality (VR), AR users remain engaged at all times in real-time in the real world.2 The growing accessibility of AR on mobile devices has significantly expanded the use of AR across diverse sectors including education,3 engineering,1 entertainment,1 marketing,4 medicine,4 architecture,4 environmental sustainability,5 and sexual health.6 In education, AR has been shown to enhance learning, particularly with reading and for visual or kinesthetics learners as it facilitates greater knowledge retention than learning with 2D computer interfaces.4 In medicine, AR has been used in a variety of ways to aid patient education and health literacy.7 A reproductive system AR was used to enhance sexual health education.6 When combined with scenario-based learning, AR has been shown to support Spanish nursing students’ “attitudes and environmental awareness toward climate change sustainability,”8(p1) including understanding linkages between climate change, resources, and health.8 AR has been used to raise awareness on climate issues and sustainable behaviours,9 for environmental education with youth,10 and for experiential learning in Africa,11 while immersive storytelling AR has helped promote pro-environmental behaviour change.12 The use of AR for knowledge translation—improving the research and delivery of research findings13—is understudied. This is particularly important regarding research on climate impacts among youth.

CASE DESCRIPTION

Description of the Augmented Reality Tool Development

This paper details an AR project which consists of seven unique image markers linking seven corresponding digital storytelling videos made by Kenyan youth in 2023. This AR tool presents initial research study findings in the form of a table with icons, and digital storytelling (DS) videos. Research Ethics Board approval was obtained from the University of Toronto (#27312) and AMREF-ESRC Kenya (#ESRC P1052-2021) and informed consent was obtained from participants. The videos amplify Kenyan youth’s voices through stories, drawings, narration, and sound effects, are context-specific and culturally relevant, and may therefore be relatable to, and effective in engaging youth in the Kenyan and potentially other African contexts.

Head-mounted or handheld displays are required by AR applications to achieve the overlay of virtual and real-world imagery.14 Some AR technology involves the use of an image marker which is scanned by a device to trigger the superimposed experience. Marker-less AR has sometimes been used where the interface interacts instead with a person’s eyes, lips, face, or hands.15 We chose to use markers because of the ease of use, both for us as AR designers and for AR end users. Figure 1 presents an image marker from the Kenya AR project combining a photo and a custom-made icon representing a climate change-related event (drought). The original language of one DS video is Swahili while the remaining six are in English. Each DS video includes English captions and brightly coloured icons signifying one or more sexual health vulnerabilities (e.g., gender-based violence, early marriage, or transactional sex) that the video’s story highlights. All image markers are contained within a single, multiple-page PDF document which can be used digitally online or downloaded and printed. The first page of the PDF (see Figure 2) orients audiences to the study and the AR experience and includes: a text-based research summary; images from the DS video workshop; a summary of key findings presented in a table using icons and text; step-by-step instructions for downloading and using a free AR mobile app called Overly; and acknowledgements, funders, and partners.

Figure 1
Figure 1.Sample image marker from the Kenya AR project combining photo and custom-made icon representing a climate change-related event (drought).
Figure 2
Figure 2.The first page of the Kenya AR project PDF.

Developing this AR tool for knowledge translation (KT) involved seven main steps: 1) establishing knowledge users, goals, and key messages; 2) determining the technology; 3) conceptualizing the AR experience; 4) creating the image markers; 5) creating the AR content; 6) creating the AR experience; and 7) designing a poster and PDF to help motivate and guide users to interact with the AR tool. Further details are described below.

1. Establishing knowledge users, knowledge use goals and key messages

The first step in our knowledge translation planning framework was to establish knowledge user audiences, goals related to these audiences, and key messages linked to these goals.16 When beginning the AR methodology we had the research content available (videos, photos, text, initial findings, and research methodology) from a participatory arts-based research project.

We identified our target audiences as: decision-makers, policymakers, researchers, non-governmental organizations (NGOs), civil society, and youth in Kenya and internationally. We also identified potential ways of presenting the AR: meetings with high-level decision-makers (e.g., government officials); presentations at international conferences and universities; and community-based workshops with youth and community health workers. Once knowledge user audiences were established, we identified our key goals in relation to these audiences: 1) to share initial findings; 2) to share the study’s DS video methodology; and 3) to share DS videos made by youth in the study. An additional goal for when the AR would be used in community dialogues in Kenya was to engage viewers in priority ranking the potential solutions youth had suggested. How we aimed to achieve this last goal will be detailed in subsequent sections below. We then identified three key messages linked to these audiences and goals, as outlined below in Figure 3.

Figure 3
Figure 3.Three key messages linked to audiences and goals.

2. Determining the technology

The next step was to identify and explore a user-friendly AR technology application that could help us to achieve our goals, in terms of its capabilities, limitations, accessibility, and affordability. We identified the Overly app as a potential tool for this project, with an ‘AR Creator’ tool that is easy to use without prior AR experience. The Overly AR Creator facilitates marker-based AR creation. An AR marker is “an image or an object that can be recognized by an AR-enabled mobile app and is used to trigger augmented reality features.”17 We were drawn to the accessibility of marker-based AR, where we can create AR with minimal technical skills, and where our end-users only need smartphones, Wi-Fi, and the free Overly app to engage with the AR experience.17 The next step involved exploring the Overly app by reviewing the Overly website and blog with tutorials covering all the app’s available functions while testing these out on our own devices. The Overly app is a cloud-based Augmented Reality software made by Overlyapp with no specific versions.

We identified several features of the app that were most applicable to our project goals, and the nature and subject matter of the audiovisual materials we would be working with. These features include: a) an AR image gallery; b) a video AR layer; and c) custom-made AR buttons. An AR image gallery allows the AR experience to link to a slide show with up to 12 images (e.g., photos, artwork, slides with text), with interactive arrows on screen allowing end users to scroll forwards and backwards between images. A video AR layer enables end users to experience a static image (referred to as an ‘image marker’ when scanned to trigger an AR experience) transforming into a video with audio. Custom-made AR buttons consist of one or more buttons which appear in an end user’s mobile device upon scanning a designated image marker, where end users can click on this button to link to an online resource such as a website or article.

The next step was to test out the technology. To do so, we used content from a different research project primarily aimed at knowledge mobilization. Testing and piloting the technology gave us first-hand experience with the process of developing image markers, programming an image gallery and video elements into augmented reality experiences, and facilitating a public engagement experience using these AR materials.

3. Conceptualizing the AR experience

To develop a concept for the AR experience in terms of what it would look like, how users would interact with it, and how the technology would bring value to the information shared, we considered project goals, technological opportunities and limitations, existing audiovisual materials, and the ways end users would retrieve AR content.

For clarity of communication, we decided to use a single image marker per video layer. Amongst the videos created by youth, numerous were related to drought and floods, requiring us to decide which videos should be represented in the AR experience, why, and how. To do this, we developed a list of each video in terms of its climate-related factor, resource insecurity, and sexual health vulnerability/ies represented. We then selected videos that reflected these content areas, while at the same time representing a balanced cross-section in terms of demographics (e.g., videos by young women or young men).

Part of conceptualizing the AR experience also involved designing the size and position of the AR content (e.g., images in an image gallery, or the video in an augmented reality video layer) that will appear within the frame of the mobile device after an end-user scans the image marker. We designed this so that videos triggered by the image marker would fill the viewer’s frame to a size exactly matching that of the image marker.

The audiovisual materials generated through the research process (DS videos created by youth, photo documentation of the DS video workshop process) informed our concept for the AR experience. This inspired the idea to create an image marker dedicated to the video methodology, which could link to an image gallery of the video workshop process for end-users to scroll through. The original 21 videos generated in the study each contain a set of youth-generated ideas for solutions, and this underpinned the idea of utilizing the custom-made AR buttons. In terms of the way(s) end users would retrieve AR content, we established that we wanted to engage users through our institution’s website and have a printable PDF poster available for download which could be used in community workshops, stakeholder meetings, and conference presentations.

4. Creating image markers

After conceptualizing the AR experience, the next step was to design and create the image markers which Overly creators describe as ‘fundamental’13 to the AR experience. When preparing image markers, there are technical specifications to ensure a successful end-user experience–full details of which can be found on the Overly app website. One important guideline we followed in developing our image markers was to ensure that each image was “rich in detail and contrast, including unique patterns… with irregular shapes…that look different from all angles”17 so that they will be recognized by the Overly app as the specific image marker that will trigger our desired AR content. It is also important that each image marker’s design is distinct so as not to mistake one for another. Furthermore, our aim was for each image marker to be visually appealing to draw in end-user engagement, and to clearly communicate the topic that it related to. We also needed to consider the size of image markers, which may vary depending on whether image markers will be presented to end-users digitally (e.g., website for PDF) or in print, and where we planned to do both online and print presentations. The size of image markers directly determines the size of the AR content that audiences can trigger using the Overly app.

5. Creating the AR content

Preparing the AR experience required preparatory video editing and graphic design work. We subtitled the DS videos, partly to ensure a global audience would be able to understand the narration, and partly to keep a consistent look between those narrated in English and those in Swahili. We also embedded further graphic elements into the videos to help communicate additional information to end users. First, we added text at the start of each video with the video’s title and demographic information about the video creator (e.g., by young mothers). Secondly, for end users to be able to identify which videos corresponded to particular resource insecurity/ies or sexual health vulnerability/ies, we embedded icons (see Figure 4) representing various forms of resource insecurity and HIV vulnerabilities directly into the videos using video editing software. We intentionally edited these in such a way that the icons remain on screen throughout the entire duration of videos so the information they convey could remain alongside the unfolding stories.

Figure 4
Figure 4.Video still with embedded icons representing resource insecurities (housing and water) and HIV vulnerabilities (GBV and early marriage), video title, creators’ demographic, and English subtitles.

To maximize the AR experience’s potential impact while honouring the integrity of the research data (videos), we conducted basic video editing to maximize production value. For example, the audio volume was increased while background noise was reduced. Preparing photo documentation of the video methodology for becoming an AR image gallery involved some graphic design work. Photos were selected based on their level of visual engagement and storytelling in terms of communicating important aspects of the video-making workshop process (e.g., storyboarding, drawing, photographing, recording, editing, and screening). Photos were organized in groups, and then organized chronologically, based on the steps of the video-making methodology. A series of title cards were designed, each containing an icon and text, to demarcate a particular stage in the video-making methodology. These were treated like images in the image gallery and interspersed between the photos. It was important to choose a dark background for these title cards because the interactive element of the AR image gallery has white-coloured interactive arrows which need to be visible to end users. Once inside the image gallery, a custom AR button appears as a colourful rectangular-shaped table in the bottom-left of the frame of the first image which appears after scanning the image marker (see Figure 5). This custom AR button was designed with large text that reads ‘CLICK HERE’. Once viewers click on this custom AR button, it links to a downloadable/printable 7-page PDF document, designed specifically for this AR experience. This document serves as a tool for priority ranking youth-generated solutions, generated from the study, with the aim of addressing the various challenges raised in the youth-produced DS videos.

Figure 5
Figure 5.Custom AR button in lower left-hand corner of first image in image gallery AR layer.

6. Creating the AR experience

Creating the AR experience using the Overly AR Creator was user-friendly. The steps included: a) creating a workspace; b) creating a project; c) creating an AR experience within that project; and d) selecting which element you want to include in the AR experience (e.g., image gallery, multiple video layer); e) uploading content; f) resizing/repositioning content; and g) publishing the experience. Once we uploaded our content, we used tools to resize and reposition the element within the frame that end users will see. There was the option to publish and preview our AR experience, and then to unpublish and edit the AR experience, repeating this process until we were satisfied.

7. Motivating people to interact with the AR tool

The final step was designing a graphic tool we could use to let end users know that our AR content is available and how to interact with it, with the aim of motivating them to interact with our AR. Our AR content functions in conjunction with a poster offering high-level additional contextual information, therefore, we designed a PDF that incorporates our image markers with the poster (see Figure 2 above), providing an introduction to the project and instructions for accessing the AR experience.

There are several details we included in our instructions for end users to retrieve AR content, including where/how to download the Overly app, scanning the whole picture, and scanning in the same orientation (either portrait or landscape) as the image marker. See Figures 6, 7 and 8 as an illustration of how users retrieve video content from an image marker. For our project, end users also need to know to click on the arrows to scroll forwards and backwards through the image gallery. We included text with these instructions on our poster. When sharing the poster for download on our institution’s website, we also need to indicate that the poster–because of the image markers it contains–needs to be printed on hard material, which can withstand environmental conditions, and matte so as not to reflect the available light.13

Figure 6
Figure 6.User perspective of holding smartphone over ‘Excess Winds’ image marker (step 1 of AR experience) displayed digitally on laptop.
Figure 7
Figure 7.User perspective of holding smartphone while Overly app retrieves the AR content (video layer) via the image marker (step 2 of AR experience).
Figure 8
Figure 8.User perspective of viewing AR video layer filling frame of image marker on device, and option to view video full screen (step 3 of AR experience).

CONCLUSIONS

This context-specific, accessible, scalable participatory research knowledge translation project meaningfully incorporates videos made by youth in Kenya illustrating complex pathways between climate-related factors and sexual health vulnerabilities. Youth engagement and health promotion professionals can use the AR tool for public health and climate change education and youth empowerment through community screenings, workshops, and community dialogues with youth and community health workers, meetings with high-level decision-makers and policymakers, presentations at international conferences and universities, and subsequent research following a similar methodology. Youth can also use the AR tool in peer education on topics related to sexual health and climate change.

LIMITATIONS

The AR tool requires developers to have access to internet, smartphones, and a paid account with the Overly app (approximately $3 USD per AR experience per month with 80% educational discount), and skills in audio-visual content creation, while end users are required to have access to the internet and smartphones. This may limit the tool from being used in some low-resource settings. As the tool presents stories, images, characters, and languages (subtitled) from Kenya, using the tool in other cultural contexts may require additional explanations. Furthermore, the initial AR tool was developed prior to its use with policymakers and could benefit from potential iterations after feedback from these stakeholders.

RECOMMENDATIONS

Practitioners or policymakers can leverage the results – showing complex pathways linking Kenyan youth’s experiences of climate change with three HIV vulnerabilities (gender-based violence, early marriage, and transactional sex) – by inviting youth in their context to respond to this AR tool for comparative studies. Future mixed-methods research could explore: a) among stakeholders, what was learned through using the AR tool about youth and climate change, and if these learnings have generated new insights to inform practice, policy, or research; and b) among youth with lived experiences of climate change, if the AR tool is associated with hope, engagement, self-efficacy, and/or empowerment.


Acknowledgements

We are grateful to Gatis Zvejnieks who provided technical support with the Overly AR app and consultation in writing about the technical aspects of the app, and to Ali Khaleghi who provided conceptual support with exploring initial AR possibilities.

Disclaimer

The views expressed in the submission are our own and not an official position of the institution or funder.

Ethics statement

Ethics approvals were obtained from The University of Toronto Research Ethics Board (Protocol Number: #27312) and AMREF-ESRC Kenya (#ESRC P1052-2021). Informed consent was obtained from all participants involved in the study.

Funding

This study is funded by the Social Sciences and Humanities Research Council of Canada Partnership Development Grant (2021-2023) (Logie, Grant File No. 880-2020-080) and New Frontiers Research Fund (2022-2023) (Logie, Grant No. NFRFR-2021-00192).

Authorship contributions

Conceptualization and study design: SVB, CHL, AH. Funding acquisition: CHL, JK, HE, LG, LT. Data collection: SVB, JK, CG, MC, MM, BO, MO. Investigation, methodology, and formal analysis: CHL, SVB, AH, JK, HE. Writing: original draft preparation: SVB. Writing: review and editing: SVB, CHL, JK, CG, MC, MW, HE, BO, MO, AH, LG, LT. Guarantor of the study: SVB.

Disclosure of interest

The authors have no conflicts of interest to declare.

Correspondence to:

Sarah Van Borek, Ph.D, Postdoctoral Fellow
Factor-Inwentash Faculty of Social Work, University of Toronto
246 Bloor Street W, Room 504, Toronto, Ontario, M5S 1V4
Canada
[email protected]