From Data Stories to Dialogues: A Randomised Controlled Trial of Generative AI Agents and Data Storytelling in Enhancing Data Visualisation Comprehension

Over recent decades, data storytelling has emerged within data visualisation to bridge the gap between raw data and meaningful insights. Conventional visualisations, such as bar charts, line graphs, and scatter plots, present structured data visually, enabling independent data exploration and interpretation(Tufte,2001;Roberts et al.,2018;Hicks,2009).However, these methods may not fully convey deeper narratives, especially to those with limited data analysis expertise(Bravo and Maier,2020;Martinez-Maldonado et al.,2020;Börner et al.,2016;Maltese et al.,2015).Data storytelling converts raw numbers into cohesive narratives, making data understandable and actionable. This approach uses storytelling to embed facts, insights, emotions, and intentions into engaging narratives, thus giving complex data representations more meaning and actionability(Blythe,2017;Fog et al.,2005;Dimond et al.,2013).Data storytelling’s growing adoption across industries like sports analytics, education, psychology, and social work underscores its effectiveness in communicating data-driven insights(Fu and Stasko,2022;Zhi et al.,2019;Martinez-Maldonado et al.,2020;Wang et al.,2019;Lan et al.,2022;Wilkerson et al.,2021;Shan et al.,2022).Despite its significance, the formal conceptualisation of data storytelling has gained academic attention only recently.

Several frameworks and principles have been established to guide effective data storytelling.Segel and Heer (2010)integrated narrative elements with graphics to convey specific messages.Kosara and MacKinlay (2013)clarified that conventional visualisation aids data analysis, while data storytelling presents derived insights. Despite many identified patterns, the literature converges on five foundational principles for creating ’data stories’(Echeverria et al.,2017;Zdanovic et al.,2022a;Ryan,2016;Knaflic,2015):(i) Identifying an explicit goalinvolves understanding the audience’s tech savviness and subject matter expertise to ensure relevance and accessibility(Bach et al.,2018;Segel and Heer,2010;Lee et al.,2015;Dykes,2015).(ii) Removing redundant elementsdeclutters by eliminating unnecessary labels and markers to focus on critical information(Knaflic,2015;Feigenbaum and Alamalhodaei,2020;Ryan,2018).(iii) Utilising storytelling elements judiciouslymeans using annotations and cues sparingly for clarity(Kong et al.,2019;Kalyuga,2009).(iv) Capturing interestinvolves minimising irrelevant visuals while emphasising crucial components, using colours or bold elements to highlight key data points(Knaflic,2015;Feigenbaum and Alamalhodaei,2020).(v) Calling for actionrequires a coherent visualisation that guides users towards the intended action through explicit titles and clear messages(Bach et al.,2018;Ojo and Heravi,2018).These principles ensure data stories are visually appealing and effective in delivering actionable insights.

Many authors suggest data stories convey key insights more effectively than conventional visualisations.Segel and Heer (2010)noted that combining narratives with graphics offers a rich medium for presenting analysis results.Gershon and Page (2001)argued that storytelling enhances intuitive and effective communication.Daradkeh (2021)added that data storytelling can improve decision-making and business outcomes.Knaflic (2015)andKrum (2013)highlighted that such visualisations leverage human perceptual skills for recognising visual patterns, based on pre-attentive processing(Hicks,2009).Researchers likeRyan (2016),Zhang (2018),Zhang and Lugmayr (2019),andZhang et al.(2022)emphasised data storytelling’s potential for efficient communication by focusing on essential data, reducing complexity, and shortening comprehension time. Yet, empirical evidence supporting the benefits of data stories is limited and only recently emerging. For example,Pozdniakov et al.(2023)conducted an eye-tracking study with 23 higher education teachers, showing that data stories can attract attention and facilitate exploration, especially for those with low visualisation literacy. Additionally,Shao et al.(2024)conducted a comparative analysis with 103 participants, finding that data stories enhanced effectiveness and efficiency in information retrieval and insights comprehension, regardless of individuals’ visualisation literacy. However, differences between data stories and conventional visualisations in fostering empathy(Boy et al.,2015,2017;Morais et al.,2021;Liem et al.,2020;Zdanovic et al.,2022a)or supporting memorisation(Zdanovic et al.,2022a)have not been significant.

While the effectiveness of data stories in aiding user comprehension of data visualisations is increasingly recognised, concerns about the adaptiveness and scalability of this approach remain(Fernandez-Nieto et al.,2024;Li et al.,2024b).Specifically,Li et al.(2024b)conducted a systematic review of existing human-AI collaboration tools for creating data stories and concluded that further research and development are necessary to enhance the level of agency and automation in these tools. They also recommended integrating and leveraging the power of emerging large-scale AI systems, such as GenAI technologies.

The latest developments in GenAI have introduced new avenues for improving users’ comprehension of data visualisations(Chen et al.,2024;Yan et al.,2024d;Li et al.,2024b;Ye et al.,2024;Gupta et al.,2024;Yan et al.,2024b).Large language models (LLMs) like GPT and Llama are now capable of generating text-based content in response to user prompts, making them invaluable for summarising complex datasets and visualisations, providing explanatory narratives, and crafting data stories with minimal human oversight(Ma et al.,2023;Yan et al.,2024d;Chung et al.,2022;Dang et al.,2023).In parallel, image-generating diffusion models such as Midjourney and DALL-E have proven effective in creating visual representations from textual descriptions, aiding the intuitive understanding of data through visual means(Dibia,2023;Wu et al.,2021;Liu et al.,2022).The advent of multimodal models like GPT-4o further enhances GenAI’s ability to interpret and synthesise both textual and visual information, opening up new possibilities for more comprehensive explanations of data visualisations(Achiam et al.,2023;Chen et al.,2024;Yan et al.,2024d;Ye et al.,2024).These advancements in foundational models point towards a future where conveying insights to users through data visualisations could become more interactive and diverse, thereby enriching the methods through which data-driven narratives can be communicated(Ye et al.,2024;Wu et al.,2021;Li et al.,2024b).

The capability of multimodal GenAI to comprehend both natural and visual languages at a high-quality, zero-shot level is crucial for advancing users’ comprehension of data visualisations through interactive dialogues. Although conversational GenAI has been increasingly utilised in various sectors, including customer service, user engagement, and learning assistance(Okonkwo and Ade-Ibijola,2021;Muller et al.,2024;Ma et al.,2023;Kim et al.,2024;Hou et al.,2024),their application in supporting users’ comprehension of data visualisations has been relatively limited(Li et al.,2024b).This limitation is primarily due to the complexities involved in achieving high-quality, zero-shot understanding of visual language(Lee et al.,2023).Additionally, generating mere descriptions of visualisations could fall short of delivering meaningful data insightsZdanovic et al.(2022a); Ryan (2016); Knaflic (2015).To truly benefit users, narratives and explanations of data visualisations must be contextually anchored in the specific data scenario. Such contextual grounding is essential for aiding decision-making processes and prompting deeper insights(Milesi and Martinez-Maldonado,2024;Borges et al.,2022;Burns et al.,2020).Thus, developing comprehension tools for data visualisation necessitates using multimodal GenAI models that can effectively interpret both natural and visual languages.

The advancement of supporting infrastructures could further enhance the ability of GenAI to convey precise and contextually relevant insights for data visualisations. GenAI models have faced criticism for producing well-articulated but inaccurate content, known as hallucination(Ji et al.,2023;Leiser et al.,2024),which can mislead users. One effective method to mitigate this issue is RAG, which confines content generation to contextually relevant material(Shuster et al.,2021).For instance, relevant information can be converted into vector embeddings, which are representations that map objects like words or entities into a continuous space, capturing semantic or relational meanings(Mikolov et al.,2013).During interactions with GenAI models, this information can be dynamically retrieved through semantic search(Li et al.,2024a),typically by calculating the cosine similarity between stored embeddings and those generated from user queries. This approach has proven effective in reducing hallucinations and enhancing the domain-specific accuracy of generated outputs(Siriwardhana et al.,2023).Therefore, integrating GenAI with RAG could offer on-demand, real-time support to help users understand data visualisations by providing contextualised explanations to address any potential confusion(Yan et al.,2024d).

Furthermore, the development of GenAI agents—autonomous, adaptive entities that operate independently to achieve predefined goals without constant user input—holds promise for enhancing communication between users and data visualisations. These agents can offer a more engaging and structured experience by proactively guiding users through scaffolding techniques(Wu et al.,2023;Park et al.,2023b;Oertel et al.,2020).Scaffolding, an established educational strategy, aligns well with data storytelling’s purpose(Segel and Heer,2010;Zdanovic et al.,2022a;Ryan,2016;Knaflic,2015).Techniques often involve breaking down complex information into smaller parts, asking guiding questions, and providing feedback to help users make sense of data. For instance, similar to how an educator might ask,” What trends do you notice in this graph?” or” Why do you think this data point is an outlier?”, proactive GenAI agents could prompt users to focus on specific parts of data visualisations and offer step-by-step guidance(Gibbons,2002).This strategy effectively supports individuals in achieving a deeper understanding and mastery of a subject by breaking down complex information and offering guidance as they build their knowledge and skills(Gibbons,2002;Van de Pol et al.,2010;Kim et al.,2018).Consequently, this approach may enhance user comprehension more effectively than passive GenAI agents that merely respond to queries. Although this method seems promising and innovative system frameworks are being developed(Yan et al.,2024d;Ma et al.,2023),empirical evidence remains limited on whether such agents improve the effectiveness and efficiency of users’ comprehension of data visualisations compared to data stories.

Effectively communicating data visualisations can be challenging, even when the visualisation and its accompanying tools, such as data stories and GenAI agents, are well-designed(Donohoe and Costello,2020).Literacy in data visualisation is essential for understanding how these different methods can improve user comprehension.Visualisation literacyrefers to the ability to accurately interpret data visualisations, efficiently extract insights, and draw informed conclusions(Boy et al.,2014;Firat et al.,2022).Research on the interaction between visualisation literacy and data storytelling is both limited and mixed. For example,Pozdniakov et al.(2023)found that teachers with lower visualisation literacy benefit the most from data storytelling enhancements, as it reduces their cognitive load. In contrast,Shao et al.(2024)found that data stories help users of all data literacy levels equally well in extracting insights. These mixed findings necessitate further research on the effects of visualisation literacy on different methods of enhancing users’ comprehension of data visualisations.

This study aims to fill the gaps in the literature identified above by empirically evaluating the impact ofthree intervention conditions,– data stories, passive GenAI agents (those that can reactively respond to questions), and proactive GenAI agents (those that can proactively guide users to make sense of data visualisations by asking questions or providing suggestions)– on users’ effectiveness and efficiency in extracting insights from data visualisations. Specifically, we sought to quantitatively assess how each condition enhances accuracy and reduces the time required for insights extraction from data visualisation acrossthree intervention phases:pre-intervention, during the intervention, and post-intervention (i.e., after the intervention is removed). Including a post-intervention phase is crucial for understanding whether these supportive approaches function merely as temporary tools or also facilitate the learning of key comprehension skills in data visualisation during the process(Segel and Heer,2010;Asamoah,2022;Bahtaji,2020).This motivates the first research question:

• •

  RQ1:To what extent do participants’effectivenessandefficiencyin comprehending data visualisationschangefrom pre- to post-intervention phases across the three intervention conditions?

We also aimed to compare the effectiveness and efficiency of the three interventions to identify the most effective method for sustaining comprehension improvements. This between-subject comparison is critical for understanding the relative efficacy of novel methods, such as GenAI agents, compared to state-of-the-art approaches like data storytelling for enhancing data visualisation comprehension(Li et al.,2024b).This leads to the second research question:

• •

  RQ2:To what extent do participants’effectivenessandefficiencyin comprehending data visualisationsdifferbetween the three intervention conditions in pre-intervention, the intervention, and post-intervention phases?

Additionally, considering the important role that visualisation literacy plays in data visualisation comprehension(Firat et al.,2022;Pozdniakov et al.,2023;Shao et al.,2024),we aimed to examine the effects of visualisation literacy on the three intervention conditions through the third research question:

• •

  RQ3:To what extent dovisualisation literacyimpact participants’ effectiveness and efficiency in comprehending data visualisations in each of the three intervention conditions?

Finally, in addition to the three prior quantitative investigations, we also aimed to gain a deeper understanding of potential differences through qualitative explorations. This provides in-depth insights to support researchers and practitioners in choosing between different supportive methods of data visualisation by specifying their unique benefits. This led to our final research question:

• •

  RQ4:What are participants’experiencesandperceptionsof the intervention’s effectiveness in helping them understand data visualisations for each condition?

The current study utilises a multimodal dataset to examine teamwork in healthcare simulations. The simulated learning environment (see Figure1), featuring four patient beds equipped with medical devices like oxygen masks and vital signs monitors, used advanced patient manikins that could simulate different heart rates and pulses, controlled by teaching staff via a tablet. The main goal was to assess and enhance students’ teamwork, communication, and prioritisation skills in handling clinical emergencies.

The dataset included three primary types of physical and physiological data: positioning, audio, and heart rate. Students’ x-y positioning data were collected using the Pozyx creator toolkit¹¹1https://www.pozyx.io/creator,audio data were captured using wireless headsets with unidirectional microphones for multi-channel audio recording, and heart rate data were recorded using Empatica E4 wristbands.

This dataset was selected because its visualisations have already been validated and utilised by healthcare educators and students for reflective purposes [Anonymised]. As depicted in Figure2,three different visualisations with increasing complexity were used. The first visualisation is a bar chart illustrating four prioritisation strategies demonstrated by students during the simulation. This type of visualisation is typically effective for comparing different categories(Saket et al.,2018).For example, the bar chart allows for a straightforward interpretation of the proportion of time each team spent on various prioritisation behaviours.

The second visualisation is a social network, or sociogram, mapping the communication behaviours among students and other actors in the simulation. Social network analysis is commonly used in collaborative learning and computer-supported cooperative work studies(De Laat et al.,2007;Dado and Bodemer,2017).This visualisation can help understand interaction dynamics within the team, identifying who communicated most frequently and the direction of these communications. It can also highlight interactions with the patient, the doctor, and the relative, providing insights into the roles and engagement levels of each participant.

The third visualisation is a ward map that displays students’ physical positions, verbal communication duration, and peak heart rate locations. Inspired by advanced techniques in sports analytics, such as CourtVision(Goldsberry,2012),this visualisation combines heatmaps(Gu,2022)with specific healthcare-relevant information. The heatmap component illustrates the frequency and distribution of verbal communications, with saturated colours indicating areas of high verbal activity. Additionally, the map shows the spatial distribution of students around the simulation space, providing a comprehensive view of their physical and behavioural engagement. Furthermore, the map includes the location and value of each student’s peak heart rate, indicating areas of highest physiological arousal. These visualisations collectively offer insights into prioritisation, teamwork, and communication strategies, which are challenging to comprehend without advanced visualisation literacy [Anonymised].

To investigate the impact of passive and proactive GenAI agents on participants’ ability to comprehend data visualisations, we utilised an open-sourced prototype of [Anonymised]²²2Hidden for double-blinded review.This prototype integrates multimodal GenAI (e.g., OpenAI GPT-4o³³3https://openai.com/index/hello-gpt-4o/), RAG frameworks (e.g., LangChain), and agentic system designs (e.g., Autogen(Wu et al.,2023)) to ensure the accuracy and contextual relevance of information provided by the GenAI agents. Multimodal GenAI processes both graphical and textual data, tailoring AI-generated responses specifically to the visualisations participants inquire about. RAG incorporation ensures GenAI agents access relevant and accurate knowledge for context-specific prompts(Lewis et al.,2020;Gao et al.,2023).Agentic systems(Park et al.,2023b)enable GenAI agents to address irrelevant responses generated by incomplete user prompts(White et al.,2023).As shown in Figure3,the design of the passive and proactive GenAI agents includes five main components: i) agent characteristics, ii) prompt integration, iii) knowledge database, iv) response generation, and v) exemplar behaviours.

To reliably create data visualisations enhanced with data storytelling elements, we first needed to clearly identify these storytelling techniques. We established design actions, based on the framework byZdanovic et al.(2022a),to transform the original visualisations, as presented in Figure2,into data stories (examples presented below in Figure5by adding or removing visual design elements. This process follows three main phases: i) exploring the data, ii) creating the visualisation, and iii) telling the story. While conventional visualisations typically involve only the first two phases, data stories incorporate additional storytelling elements added in the third phase(Zdanovic et al.,2022a;Echeverria et al.,2017;Ryan,2016;Knaflic,2015).These include the following main data storytelling elements:

Using the right visualisation technique:Selecting the appropriate visualisation based on the storytelling goal is critical(Knaflic,2015).This could involve switching chart types (e.g., from pie to line charts) to better illustrate trends(Tufte,2001;Few,2004).Our design process did not result in changing the chart types, as we aimed to maintain the increasing complexity from the bar chart, through the social network visualisation, to the custom-made ward map. This approach allowed us to preserve the progression of complexity across the visualisations.

Eliminating clutter:Simplifying visuals by removing unnecessary elements highlights key points and enhances readability, adhering to principles of minimising the data-ink ratio(Tufte,2001;Heer and Shneiderman,2012).

Directing attention:Emphasising important data points or trends through techniques like bold lines or contrasting colours to draw attention to key insights(Knaflic,2015;Ware,2019;Few,2004).

Adding annotations:Providing contextual or explanatory information directly on the visualisation clarifies important insights(Knaflic,2015;Hullman et al.,2013).

Adding an explanatory title:Crafting titles that summarise key findings or insights guides the viewer toward the intended interpretation(Knaflic,2015;Pozdniakov et al.,2023;Kong et al.,2018).

The process of creating visualisations enhanced with data storytelling elements was conducted collaboratively. Initially, one researcher designed a first draft of the data stories followingZdanovic et al.(2022a)’s framework. Then, a second researcher with deep expertise in data storytelling reviewed and improved the designs through discussion. The refined visualisations were further reviewed by two additional researchers, leading to a final design agreed upon by the team. This iterative process, and followingZdanovic et al.(2022a)’s framework for infusing visualisations with data storytelling elements, ensured that the visual elements were thoughtfully developed and aligned with the research objectives.

This randomised controlled study was designed following gold standards for effectiveness research(Hariton and Locascio,2018).We conducted a mixed-design (3x3) experiment, using both within-subject and between-subject comparisons to evaluate the effectiveness of data storytelling, passive GenAI agents, and proactive GenAI agents in enhancing insight extraction from data visualisations. The study was conducted using Qualtrics⁶⁶6https://www.qualtrics.com/,an online platform for designing and conducting experiments, integrated with a custom-developed website. Participants were recruited from Prolific⁷⁷7https://www.prolific.co/,compensated £8, and the study was structured to be completed within one hour. Integration between Qualtrics and Prolific streamlined participant flow, redirecting them from Prolific to the Qualtrics survey and verifying data before confirming participation, enhancing reliability. Ethics approval was obtained from [Anonymised] University (Project Number: Anonymised). Informed consent was secured from each participant.

Participants were randomly assigned to one of three intervention conditions: passive GenAI agents, proactive GenAI agents, or data storytelling. Passive GenAI agents responded to user queries without initiating further dialogue or guidance, while proactive GenAI agents guided users with scaffolding questions. The data storytelling condition presented visualisations within a narrative format, highlighting key insights and trends. Each participant interacted with their assigned condition during the intervention phase to support insight extraction from visualisations. The effectiveness of each method was evaluated through pre-, during, and post-intervention assessments.

The study followed a structured procedure to ensure consistency across conditions, comprising five main parts: i) the participants received an introduction to the study and completed demographic and background questions on Qualtrics; ii) they answered 12 items from the mini-Visualisation Literacy Assessment Test (mini-VLAT)(Pandey and Ottley,2023)to measure their visualisation literacy; iii) the participants were directed to our custom platform to complete three analytical writing tasks, each followed by six evaluation questions to assess their understanding of the visualisations; and iv) finally, the participants completed a questionnaire to provide feedback on their experience with the assigned intervention method. Figure6illustrates the experiment design and procedure. Details are elaborated below.

We conducted a priori power analysis for a mixed-design study with three between-subject conditions and three within-subject repeated measures using G*Power(Faul et al.,2009).The analysis indicated that a minimum sample size of 108 participants (36 per condition) was needed to detect a medium effect size (0.25) with 80% power at a 0.05 significance level. To account for an anticipated attrition rate of 15% to 20% due to potential data issues, particularly given the one-hour duration of the online study, we aimed to recruit 150 participants. Participants were required to be fluent in English and use a laptop or desktop to ensure consistent visualisation displays. We specifically recruited individuals with healthcare, medical, or nursing backgrounds to control for familiarity with the healthcare simulation context of the visualisations.

Before the main study, a pilot study with 27 participants (9 per condition) was conducted to refine the Qualtrics survey for clarity and ease of use. We also tested the custom-developed website to ensure timely responses from both passive and proactive GenAI agents. Data from the pilot study were excluded from the final analysis due to subsequent improvements in the study flow, such as mandatory reading time for each task instruction and improved clarity of several context description sentences.

For the main study, participants had to complete all three writing tasks, essential for internalising their understanding of data visualisations. Incomplete responses, including failure to complete any writing task or the entire survey, were deemed invalid. Out of the 141 responses received, 24 were invalid due to incomplete writing tasks, resulting in a final sample of 117 valid responses. Specifically, 36 participants completed the data storytelling condition, 41 completed the passive GenAI agents condition, and 40 completed the proactive GenAI agents condition.

The participants came from six different regions, with the majority coming from North America/Central America (51), Europe (28), and Africa (25). The remainder were from Australia (5), South America (4), and other regions (4). They identified as female (68), male (48), and non-binary (1). The age distribution was predominantly within the 25-34 years old range (47), followed by 18-24 years old (41), 35-44 years old (16), 45-54 years old (8), 55-64 years old (3), and 65+ years old (2). Most participants attained a Bachelor’s degree (55) or high school or equivalent education (28), while others held a Master’s degree (15), vocational training or diploma (11), Doctorate (4), or other qualifications (4). Regarding experience with data analysis, most participants were at an intermediate level (50), with a moderate level of experience. Others were beginners (36), advanced users (17), had no experience (12), or were experts (2). In terms of familiarity with GenAI tools, most participants were at an intermediate level (63), having used the tools on several occasions. Others were beginners (34), advanced users (15), experts (4), or had no prior experience (1).

The analysis aimed to address four research questions (RQs) regarding the impacts of different interventions on enhancing participants’ effectiveness and efficiency in extracting insights from data visualisations. We first calculated a set of metrics related to the RQs using the responses and times for six evaluation questions for each task (AppendixB). These metrics were calculated for both within- and between-subject analyses, including the three intervention phases, specifically, the pre-intervention baseline (Pre), the intervention phase, and the post-intervention phase (Post), as well as for the three intervention conditions, namely data storytelling (DS), passive GenAI agents (GAI), or proactive GenAI agents (DSAI). In particular, the following metrics were calculated:

• •

  Correct Score:To investigate the effectiveness of insights extraction from data visualisations (RQ1-2), we computed the correct score for the six evaluation questions for each condition by summing the total number of questions that participants answered correctly (each score ranges from 0 to 6), resulting in three correct scores for each participant (Pre_score, Intervention_score, and Post_score). The correction-for-guessing formula(Thorndike et al.,1991)was applied to account for guessing in these multiple-choice questions (with four options).

• •

  Success Time:To investigate participants’ efficiency in data insights extraction (RQ1-2), we summed the total time they spent on answering the six evaluation questions for each condition in seconds and then divided by the corresponding correct score for that given condition, resulting in three success time metrics for each participant (Pre_time, Intervention_time, and Post_time).

In addition to the above two metrics, visualisation literacy (mini-VLAT) was calculated separately for each participant to address RQ3. This calculation was done by summing the number of correct responses. Specifically, visualisation literacy ranges from 0 to 12. For both literacy scores, the correction-for-guessing formula(Thorndike et al.,1991)was applied to account for guessing behaviours. All calculations and analyses were conducted in Python using packages including NumPy, SciPy, and Statsmodels. The following sections elaborate on the analysis for each RQ.

To ensure the comparability of the three intervention conditions, we conducted a preliminary analysis of participants’ visualisation literacy (mini-VLAT), data analysis expertise, and GenAI expertise.

For visualisation literacy (mini-VLAT), the Kruskal-Wallis test did not reveal significant differences between the three conditions: DS (Median = 6.0, IQR = 4.0), GAI (Median = 6.67, IQR = 5.33), and DSAI (Median = 6.67, IQR = 4.0). The test results were$H(2)=0.49,n=117,p=0.784$,indicating that the groups were comparable in terms of visualisation literacy.

For data expertise, the Kruskal-Wallis test also did not show significant differences between the conditions: DS (Median = 3.0, IQR = 1.0), GAI (Median = 3.0, IQR = 1.0), and DSAI (Median = 3.0, IQR = 1.0). The test results were$H(2)=1.02,n=117,p=0.601$,confirming that the groups were comparable in terms of data analysis expertise.

For GenAI expertise, the Kruskal-Wallis test again did not reveal significant differences between the conditions: DS (Median = 3.0, IQR = 0.25), GAI (Median = 3.0, IQR = 1.0), and DSAI (Median = 3.0, IQR = 1.0). The test results were$H(2)=1.42,n=117,p=0.491$,indicating that the groups were comparable in terms of GenAI expertise.

These non-significant findings in visualisation literacy, data expertise, and GenAI expertise metrics are critical for comparing the three intervention conditions. Boxplots were drawn to visualise the sample distribution for each condition, further confirming the similarity of the groups (Figure8). These findings ensure that any differences observed in the subsequent analyses can be attributed to the interventions rather than pre-existing disparities among the participants.

Figure9illustrates the within-subject changes across the three phases for each intervention condition. For the DS condition (N=36), the Friedman test revealed a significant difference in correct scores across the three phases,$\chi^{2}(2)=30.05,p<.001,W=0.35$(medium effect(Pereira et al.,2015)). Post-hoc Wilcoxon signed-rank tests indicated that the correct scores significantly improved from the pre-intervention (Median = 3.0, IQR = 2.0) to the intervention phase (Median = 5.0, IQR = 1.0),$W=46,p<.001,r=0.91$(large effect(Kerby,2014)), and from the pre-intervention to the post-intervention phase (Median = 5.0, IQR = 2.0),$W=52,p<.001,r=0.88$(large effect). However, the change from the intervention phase to the post-intervention phase was not significant. In terms of success time for the DS condition, the Friedman test also showed significant differences,$\chi^{2}(2)=33.17,p<.001,W=0.46$(medium effect). Wilcoxon signed-rank tests revealed that the success time significantly decreased from the pre-intervention (Median = 75.72 seconds, IQR = 62.53 seconds) to the intervention phase (Median = 39.61 seconds, IQR = 25.96 seconds),$W=76,p<.001,r=0.89$(large effect), and from the pre-intervention to the post-intervention phase (Median = 35.58 seconds, IQR = 21.27 seconds),$W=22,p<.001,r=0.97$(large effect). The change from the intervention to the post-intervention phase was not significant. These results suggest that the DS intervention significantly improved the participants’ ability to extract correct insights from data visualisations and reduced the time taken to do so. The improvements from the pre-intervention to the intervention phase and from the pre-intervention to the post-intervention phase were significant, indicating the effectiveness of the DS condition. However, there was no significant difference between the intervention and the post-intervention phase, suggesting that the gains in performance were maintained post-intervention.

For the GAI condition (N=41), the Friedman test indicated significant differences in correct scores,$\chi^{2}(2)=29.56,p<.001,W=0.29$(small effect). Post-hoc tests showed significant improvements from the pre-intervention (Median = 3.0, IQR = 1.0) to the intervention phase (Median = 4.0, IQR = 1.0),$W=71,p<.001,r=0.87$(large effect), and from the pre-intervention to the post-intervention phase (Median = 5.0, IQR = 2.0),$W=104,p<.001,r=0.81$(large effect). The change from the intervention to the post-intervention phase was not significant. The success time for the GAI condition also showed significant differences,$\chi^{2}(2)=40.59,p<.001,W=0.49$(medium effect). Wilcoxon signed-rank tests indicated significant reductions from the pre-intervention (Median = 72.12 seconds, IQR = 77.27 seconds) to the intervention phase (Median = 46.34 seconds, IQR = 42.88 seconds),$W=57,p<.001,r=0.93$(large effect), and from the pre-intervention to the post-intervention phase (Median = 36.8 seconds, IQR = 19.74 seconds),$W=86,p<.001,r=0.90$(large effect). The change from the intervention to the post-intervention phase was not significant. These findings indicate that the GAI intervention was effective in significantly enhancing participants’ correct scores and reducing their success times from the baseline to both the intervention and the post-intervention phase. However, the lack of significant differences between the intervention and the post-intervention phase suggests that the improvements were sustained post-intervention, similar to the DS condition.

For the DSAI condition (N=40), the Friedman test revealed significant differences in correct scores,$\chi^{2}(2)=46.48,p<.001,W=0.50$(large effect). Post-hoc Wilcoxon signed-rank tests indicated significant improvements from the pre-intervention (Median = 3.0, IQR = 2.0) to the intervention phase (Median = 5.0, IQR = 1.0),$W=32,p<.001,r=0.94$(large effect), and from the pre-intervention to the post-intervention phase (Median = 6.0, IQR = 1.0),$W=19,p<.001,r=0.97$(large effect). Additionally, there was a significant improvement from the intervention to the post-intervention phase,$W=209,p=.019,r=0.52$(large effect). Significant differences were also found in the success time for the DSAI condition,$\chi^{2}(2)=43.65,p<.001,W=0.55$(large effect). Wilcoxon signed-rank tests indicated significant reductions from the pre-intervention (Median = 98.28 seconds, IQR = 76.31 seconds) to the intervention phase (Median = 41.94 seconds, IQR = 19.77 seconds),$W=20,p<.001,r=0.98$(large effect), and from the pre-intervention to the post-intervention phase (Median = 35.75 seconds, IQR = 16.06 seconds),$W=42,p<.001,r=0.95$(large effect). The change from the intervention to the post-intervention phase was not significant. These results demonstrate significant improvements in both correct scores and success times from the baseline to the intervention and the post-intervention phase. Unlike the other conditions, the DSAI intervention also showed significant improvement from the intervention to the post-intervention phase, indicating a continued positive improvement post-intervention.

Figure10illustrates the between-subject changes across the three conditions for each intervention phase. The Kruskal-Wallis test did not reveal significant differences between the three conditions for the correct scores in both the pre-intervention and the intervention phase. However, for the post-intervention correct scores, the Kruskal-Wallis test revealed significant differences,$H(2)=10.62,n=117,p=.005,\eta^{2}=0.08$(medium effect). Pairwise comparisons using the Mann-Whitney U test indicated significant differences between the DSAI and DS conditions ($U=967.5,p=.007,r=0.30$;medium effect), and between the DSAI and GAI conditions ($U=1119.0,p=.003,r=0.31$;medium effect). Specifically, the median correct score for DSAI was 6.0 (IQR = 1.0), which was significantly higher than the median scores for DS (Median = 5.0, IQR = 2.0) and GAI (Median = 5.0, IQR = 2.0). For the success times, the Kruskal-Wallis tests revealed no significant differences between the intervention conditions across all three phases. These between-subject results suggest that while all three interventions (DS, GAI, and DSAI) were effective in improving correct scores and reducing success times within subjects (from RQ1 findings), the DSAI condition was particularly effective in achieving higher correct scores in the post-intervention phase compared to the DS and GAI conditions. This indicates that the DSAI intervention may be more effective in sustaining performance improvements post-intervention. There were no significant differences in success times between the conditions, suggesting that all interventions were similarly effective in reducing the time taken to extract insights from the visualisations.

As data visualisation gains widespread adoption across various domains to convey data insights(Sun et al.,2023;Park et al.,2023a;Yang et al.,2023;Schroeder et al.,2020),it becomes crucial to identify practical and scalable methods to assist the general public in understanding these visualisations. Many individuals struggle with interpreting visualisations due to low levels of visualisation literacy(Maltese et al.,2015;Börner et al.,2016;Donohoe and Costello,2020).This comparative study offers empirical evidence on three distinct approaches—data storytelling, passive GenAI agent, and proactive GenAI agent—to improve the effectiveness and efficiency of comprehending data visualisations.

Regarding the first research question, all three approaches significantly enhanced both the effectiveness (measured by correct scores) and efficiency (measured by average success time) of extracting insights from data visualisations. These improvements were maintained post-intervention for data stories and the passive GenAI agent, while the proactive GenAI agent condition showed continued enhancement even after the intervention was removed. These findings provide empirical evidence supporting prior assumptions that data storytelling enhances participants’ ability to accurately and efficiently comprehend insights conveyed through data stories more than conventional data visualisations(Ryan,2016;Zhang,2018;Zhang and Lugmayr,2019;Zhang et al.,2022;Kosara and MacKinlay,2013;Gershon and Page,2001).Such findings also provided empirical evidence to supportDykes (2015)’s foundational definition of data storytelling. Likewise, the findings also contribute empirical evidence supporting the use of GenAI agents as an alternative to data storytelling for facilitating accurate and efficient comprehension of data visualisations. This aligns with recent recommendations to leverage advanced AI systems for supporting the understanding of data visualisations(Li et al.,2024b;Fernandez-Nieto et al.,2024;Yan et al.,2024d).Furthermore, the sustained and continued improvements following the removal of interventions indicate that these supportive methods do more than serve as temporary tools; they facilitate the learning of key comprehension skills in data visualisation, extending their benefits beyond mere utility to education(Segel and Heer,2010;Asamoah,2022;Bahtaji,2020).

In terms of the second research question, we found that the proactive GenAI agent infused with scaffolding techniques significantly outperformed both data stories and the passive GenAI agent in the effectiveness of extracting insights from data visualisations after the intervention was removed. This finding supports the effectiveness of scaffolding techniques for mastering a subject(Gibbons,2002;Van de Pol et al.,2010;Kim et al.,2018),which in this study pertains to users’ ability to comprehend data visualisation. As these techniques are also frequently used in the creation of data stories(Segel and Heer,2010;Zdanovic et al.,2022a;Ryan,2016;Knaflic,2015),the proactive agent can be viewed as a combination of data storytelling and the passive GenAI agent. It actively conveys data stories by prompting users with targeted questions instead of passively responding to user prompts. In this sense, the current finding also provides empirical evidence to support the additive effects of combining data storytelling with GenAI technologies to achieve better comprehension and learning skills in the interpretation of data visualisations.

For the third research question, the absence of interactional effects between intervention conditions and visualisation literacy on participants’ effectiveness in understanding data visualisations is consistent withShao et al.’s findings(Shao et al.,2024).This indicates that all three interventions are equally beneficial for users, regardless of their level of visualisation literacy. Contrary to the common belief supported by previous studies(Ma et al.,2012;Zhang et al.,2022;Figueiras,2014;Pozdniakov et al.,2023),our results did not statistically confirm that any supportive methods, including data storytelling, are more advantageous for individuals with lower visualisation literacy. Furthermore, the significant results demonstrating the proactive GenAI agent’s ability to enhance participants’ effectiveness in comprehending data visualisations after the intervention was removed, while accounting for visualisation literacy, further validate its benefits for users across all levels of visualisation literacy.

Regarding the last research question, qualitative insights revealed that participants appreciated the enhanced efficiency, clear directions, practical examples, and multiple perspectives offered by data stories. This finding supports the proposed benefits of data storytelling, where integrating narrative elements could improve the communicative power of data visualisations(Schulz et al.,2013;Dykes,2015).However, such benefits seem to be less applicable in supporting participants to integrate insights from multiple visual aspects and contextualising interpretations, emphasising the need for integrative data storytelling techniques that support users in making connections between multiple heterogeneous facets(Chen et al.,2018;Martinez-Maldonado et al.,2020).On the other hand, the passive GenAI agent was noted for providing valuable contextual insights, facilitating textual interpretation, and improving task efficiency, and the proactive GenAI agent was praised for integrating multiple insights for a holistic understanding and offering structured guidance and feedback. These findings suggest that these GenAI agents hold considerable promise in complementing data stories, especially in scenarios where users need to synthesise complex data insights from multiple sources(Streit et al.,2011;Chen et al.,2018;Martinez-Maldonado et al.,2020).

This study presents several critical implications for the field of HCI and data visualisation research. The demonstrated comparative effectiveness of GenAI agents suggests that leveraging advanced AI technologies can provide alternative and complementing methods to data storytelling in support users in interpreting complex data visualisations(Fernandez-Nieto et al.,2024;Li et al.,2024b;Yan et al.,2024d).Specifically, the proactive agent’s superior performance, especially after the intervention was removed, underscores the potential of scaffolding techniques in facilitating and cultivating essential comprehension skills of data visualisations(Gibbons,2002;Kim et al.,2018).Researchers should explore how these agents can be further developed to provide adaptive, context-aware support that evolves with the user’s growing comprehension skills(Asamoah,2022;Bahtaji,2020).

The positive findings on GenAI agents by no means invalidate the value of data storytelling. Instead, they further substantiate the foundational principles of creating effective data stories(Segel and Heer,2010;Zdanovic et al.,2022a;Ryan,2016;Knaflic,2015),as these design principles closely align with established scaffolding techniques in the field of education and training(Van de Pol et al.,2010;Kim et al.,2018).However, the current evidence is limited to enhanced effectiveness and efficiency in comprehending insights from data visualisations and does not necessarily translate to improved memory recall(Zdanovic et al.,2022a),user engagement(Boy et al.,2015;Zhao et al.,2019),or fostering empathy towards the subject matter(Boy et al.,2017;Morais et al.,2021;Liem et al.,2020).Consequently, this study contributes to and advocates for a broader research initiative that promotes the investigation of human-AI collaboration in designing and developing tools to better support users in comprehending insights from data visualisations and developing their comprehension skills during the process. This is becoming increasingly essential for work across various domains(Sun et al.,2023;Park et al.,2023a;Yang et al.,2023;Schroeder et al.,2020).

The absence of significant interactional effects between visualisation literacy and the intervention conditions is consistent withShao et al.’s findings(Shao et al.,2024).This indicates that visualisation literacy might not be as crucial as previously believed in determining the effectiveness of supportive methods(Pozdniakov et al.,2023).Nevertheless, visualisation literacy remains an important factor affecting the effectiveness of comprehending data visualisations once the support is removed. This discovery paves the way for more in-depth investigations into the complex relationship between supportive methods and visualisation literacy, promoting a more comprehensive understanding of how visualisation literacy aids in users’ comprehension of data visualisation. Additionally, these results suggest the need for further research into other individual differences, such as cognitive load or domain-specific knowledge, that might impact the effectiveness of these supportive methods.

This study provides actionable insights for enhancing data visualisation comprehension through practical methods. Incorporating data storytelling and GenAI agents into data visualisations can significantly improve users’ ability to extract and understand insights, making it a valuable strategy for data analysts, educators, and communication professionals(Dykes,2015;Martinez-Maldonado et al.,2020;Wang et al.,2019;Shan et al.,2022).The demonstrated improvements in both effectiveness and efficiency of these methods are critical in decision-making contexts, ensuring timely and accurate comprehension of insights communicated via data visualisations(Daradkeh,2021;Schroeder et al.,2020).Moreover, the sustained improvements after removing these supportive methods highlight their educational role in developing data visualisation comprehension skills, making them suitable for the general public, who often struggle with low levels of visualisation literacy(Maltese et al.,2015;Börner et al.,2016;Donohoe and Costello,2020).These methods are effective regardless of users’ visualisation literacy levels, further validating their appropriateness for diverse audiences.

Given the findings on sustained improvements in comprehension skills, practitioners should consider these supportive methods as educational tools rather than temporary aids. Training programs and workshops could integrate these tools to deepen understanding of data visualisation principles and techniques. Notably, the proactive GenAI agent demonstrated prolonged improvements from intervention to post-intervention, unlike data stories and passive GenAI agents, which only showed sustained improvements after removing the intervention. This highlights the unique advantage of the proactive GenAI agent in enhancing both performance and learning(Soderstrom and Bjork,2015).The guided experiences provided by the proactive GenAI agent make it an effective method for onboarding users to complex data visualisations, particularly when integrating insights from multiple data sources and visual aspects is required(Streit et al.,2011;Chen et al.,2018;Martinez-Maldonado et al.,2020).This learning process is valuable for in-service training across various industries, such as education, healthcare, business, sports analytics, and social work, where data visualisation is increasingly used to communicate essential insights and support decision-making(Fu and Stasko,2022;Zhi et al.,2019;Martinez-Maldonado et al.,2020;Wang et al.,2019;Lan et al.,2022;Wilkerson et al.,2021;Shan et al.,2022).

Despite the benefits of GenAI technologies in supporting data visualisation comprehension, challenges remain, especially in settings that prioritise data privacy and security(Yan et al.,2024c;Zohny et al.,2023;Yan et al.,2024a).Establishing knowledge databases for storing contextual information and transmitting such information to third-party GenAI models could pose privacy and security concerns. Therefore, integrating GenAI technologies should be carefully considered, with strict adherence to regulatory guidelines.

While this study offers valuable insights, it also has several limitations that should be addressed in future research. Firstly, the study was conducted in a controlled experimental setting, which may not fully capture the complexities of real-world data visualisation tasks. Future studies should consider field experiments or longitudinal studies to examine the long-term effects of these interventions in naturalistic settings. Secondly, the participant pool was limited by the recruitment platform and available resources, resulting in an overrepresentation of participants from North America and Europe. Future research should strive to include a more diverse participant group. Thirdly, the study highlighted the potential of proactive GenAI agents but did not delve into specific design features and interaction mechanisms contributing to their effectiveness. Future research should explore the design and implementation of these agents in greater detail, optimising scaffolding techniques and feedback mechanisms to support users’ comprehension of data visualisations. Additionally, the generalisability of the findings may be limited by the specific sample and context used in this study. Future research should replicate the study with diverse populations and in different domains to validate and extend the findings. Lastly, we did not analyse the analytical writing content due to potential subjectivity in coding and grading responses. This decision was influenced by the limited insights such analyses would contribute to our study’s focus on understanding effectiveness and efficiency. In future work, we plan to use automatic essay grading and learning analytics advancements to uncover process-level insights from participants’ interactions with the supportive methods and their impacts on the writing process(Ramesh and Sanampudi,2022;Kovanovic et al.,2017).

This study provides empirical evidence on the effectiveness of data storytelling, passive GenAI agents, and proactive GenAI agents in enhancing users’ comprehension of data visualisations. The findings demonstrate that all three approaches significantly improve the effectiveness and efficiency of extracting insights, with the proactive GenAI agent showing the most promise in sustaining and further enhancing comprehension skills. These results have important implications for both research and practice, highlighting the need for more integrative and adaptive approaches to data visualisation support. Future research should continue to explore the potential of GenAI technologies and scaffolding techniques to develop more user-centred data visualisation tools that can cater to the diverse needs of users across different contexts and levels of expertise.