Academic researchers put a ton of time, effort, and deep thought into their work. And then, to get it published, they need to write it all up in a story that makes sense to people who weren’t as involved in the research as they were.

Most papers end up needing a few figures - and most academics aren’t trained in making great figures that add to the story they are telling. They’re just dropped into it and try to do the best they can with what they know.

I wrote this article to help give academics a concrete example that will help them get their figures from “Is another table ok here?” to “I’m proud of how this supports my work.”

Stay tuned for Part 2 - Visual Best Practices for Academic Papers.

The Sufficient Context Dataset

For this example, I’m using my work on the paper for Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. Unless otherwise noted, all images seen here are credited to that paper.

The data the authors wanted to visualize started as this table:

Map the Data Model

For this figure, we needed to encode 4 variables:

That’s 72 pieces of data across 4 axes. It’s not a ton of data, but it is more than a brain can hold and compare at once.

There are almost infinite options for ways we could lay this out and encode the data, and that can be overwhelming - so here’s my process for limiting the urge to try everything and deliberately settling for a local maximum. We’re not going for the perfect figure. We’re going for a figure that does the job.

Get clear on the purpose of the figure

When a figure sits inside the structure of your paper, it must add to - not distract from - the story you are telling about your research.

It is not your reader’s job to infer your goals or insights. It’s your job to create a figure that helps your reader understand the story you’re telling and what conclusions you came to.

Thus, ask yourself:

What idea do I want my readers to come away with after looking at this figure?

Think about this in your case. You might want them to understand the difference between two test methods. Or you might want to highlight unexpected outliers.

Whatever it is, work backward from there. Ask “what would a figure that did X exceptionally well look like?”

Wait! I don’t know what I want my figure to communicate!

If you don’t know what you want your reader to come away with yet, that’s ok!

Play with the data and ask it what it’s telling you. If you find yourself wanting to make a different point than you originally expected, that’s ok… think about restructuring the story as a whole.

Engage with your data, and let it help shape the story you are telling. You just may find a place you never expected to go.

Have a conversation with your data

Whether you know what you want to show, or you don’t, the next step is exploration. I personally like to explore with the “no bad ideas” mindset. Sometimes a great idea can come from combining a “good” idea with a “mediocre” idea.

Data Conversations in the age of AI

It’s really easy to go right to AI with your dataset and ask it to make a few options. I caution against using AI first.

Instead, think about your data model and the different ways you could map it. Take 5-10 minutes and sketch out the idea for eight - YES, EIGHT - different chart types and data mappings you think would be useful.

Before AI, I used to mock up the more promising figures myself with python or figma. Now, I find that AI can get early options in front of me for evaluation very quickly, so I use that instead.

If you want to use AI for the mockup step, I recommend giving it:

This will get you real charts showing the shape of your data that are inspired by your knowledge, help guide the AI to brainstorm in a direction that will be more useful for you, and it will also get you more engaged in the conversation.

Evaluate Different Graph Types

The AI will give you a critique, but that’s not a substitute for your own thinking. Try to think deeply about each output and where your eyes are drawn before reading the AI assessment. You’re the human in the loop. You’ll use a combination of your judgement and the AI’s assessment to pick a figure that works well for your argument.

From the first round of explorations on this paper, I digitized my 3 favorite ideas with the real data, and then critiqued each one with the paper authors.

Option 1 | Slope Graph

At first glance, this seems pretty interesting - this graph shows a direct comparison of the model performance when it had sufficient vs. insufficient context. In theory, that’s what this paper is trying to do - show how models behave differently when they don’t quite have the information to answer a query.

It’s an interesting graph, but it doesn’t quite align with the story the authors were telling around their findings.

The eye is first drawn to the downward slopes in the top row - from Sufficient to Insufficient context. That’s expected, and obvious. The next thing we notice is the uptick in BOTH hallucinations and abstained answers. That’s also somewhat expected, and somewhat obvious.

The point the authors want the reader to come away with is that the rate of hallucinations is usually higher than abstains in both cases, and this layout makes it difficult to visually compare the rate of hallucinations vs. abstains.

Option 2 | Stacked Bar Graph

This one makes a ton of sense logically. The bars are all percentages that add to 100, which means we get an apples to apples comparison of the results.

This graph makes it easy to see the differences within a single model in all scenarios, but it is difficult to visually compare the sizes of abstained vs hallucinated results within a single test case, which is the point the authors want to make.

Additionally, the easiest thing to notice in this viz is the big differences in Gemma 27B’s performance vs the rest of the models. Comparing models isn’t the point for this paper - but it is nice to know that Gemma is an outlier.

Option 3 | Grouped Bar Charts - the winner!

This is grouped by output first and model second, with the test set mapped to the texture.

Pulling apart the stacked bars and grouping them by model response gives us an initial insight that works better with the story the authors are telling.

Here, the easiest thing to notice is all the bars on the left for “correct” and how models hallucinate more than they abstain in both cases, but this effect is exaggerated when the models have enough context. That’s great - it’s an insight the authors want to communicate.

The other thing that’s really easy to see here is the difference in correct answers when models don’t have enough context. That’s exactly the other insight the authors thought was important.

This chart made me decide to move forward with splitting sufficient and insufficient context into separate rows, but I moved on to some more variations within this structure before settling on a final design.

Play with Variations on your Chart

The next step is exploring some other data mappings and organization within this framework.

A second set of explorations can help you find something that communicates your point better, or feel confident in your original choice.

Variation 1 | Group by Test Set

In this option we’ve grouped by test set first and secondarily by outcome, and used texture to indicate the model itself.

This still makes the point that models generally hallucinate more than they abstain, but it’s a bit harder to see the pattern than in the viz above which was split by outcome first.

This also feels like a holistic comparison between test sets rather than models. The paper isn’t explicitly making either of those points, but authors were more interested in letting readers secondarily analyze the models holistically than the test sets, since models have more staying power than test sets.

Variation 2 | Group by Model - winner of round 2!

Splitting the grouping changes the pros and cons again. This figure is grouped first by model, second by outcome, and the test set is mapped back onto the texture.

The model grouping makes it easy to see how much each model abstains vs. hallucinates, and encourages the reader to examine the pattern across each model to understand the trend holistically.

The Published Figure

I worked with the authors to decide which option to go with - the Group by Model and Group by Output were the top contenders - and we went with group by model so that our readers could easily evaluate each model against itself in both scenarios.

I also made sure some dataviz best practices were applied to the figure and paper as a whole before giving it the green light to add to the paper. More on that in Part 2 - Visual Best Practices for Academic Figures.

References:

[1] Sufficient Context: A New Lens on Retrieval Augmented Generation Systems

[2] Lucia Ferro - Illustrator

Alyshia Olsen has spent her career in HCI helping communicate complex concepts via visual and experience design, including at Tableau, Microsoft, Plotly, and Intel. She runs an independent consulting practice called Forma Libera.

✍ This case study was written and refined by hand, with help from AI in finding and fixing typos and grammatical errors❣️