Case Study: Analyzing Procreate App Reviews With Painboard
Overview
Procreate is my all-time favorite drawing app. It provides powerful features that are able to unleash the creativity in everyone. But what can be done to make it even better? Let's turn to its user reviews for answers.
On App Store, Procreate has over 39K reviews. These reviews are no doubt a gold mine in the eyes of the product team. The catch, however, is that it's too much data to go through manually. While some reviews are short and easy to process, many of them span a few paragraphs and some like mini novels. Furthermore, people often write mixed reviews, and it's not uncommon to see a 5-star review with a few complaints ("This is app is great but..."). So how can we make sense of all these data?
The manual way
I downloaded the most recent 750 reviews and loaded them into a spreadsheet. The usual way of analyzing this data would be reading through the reviews one by one, with another spreadsheet open to collect any points of interest. Along the process, you'd get a list of common themes with the number of users who mentioned them.
This is obviously laborious work. I don't know about you, but it took me a few minutes on average to process a review. For 4 mintues each, that's 750 * 4 = 3000 minutes, or 50 hours total. If you hire someone to do it, even at $30/hr, it'd cost you $1500.
And that's just for 750 reviews. Imagine if you have 39K reviews to go through, and don't forget that new ones keep coming in every day!
The Painboard way
I fed the same 750 reviews into Painboard. A few minutes later, I got some results:
As you can see, Painboard has scanned through the reviews, gleaned interesting themes, and organized them in this clean, glanceable interface. Immediately I am able to tell some of the most complained things about the app:
- Brush and Pencile Tool Issues
- Photo Import and Reference Issues
- Color Tool and Palette Issues
Also the app seems to have quite bit of a learning curve. Adding some high-quality tutorials would be a good idea.
There are other themes and columns that aren't shown in the screenshot above. But the idea is simple: a theme is a topic that users frequently mentioned in their reviews. Themes are grouped in relevant columns and sorted by the number of users who mentioned them.
Drilling into details
Now image that you want to take a deep dive into a theme. For example, you want to understand what the exact issues about "Brush and Pencile Tool" are. You can expand the theme to see the original quotes in their context:
You can get a detailed report that includes an executive summary and a weekly trend chart. This works for every theme and column on the board.
You can also see the distribution of the reviews across versions. This is useful to see if a theme is a recent issue or a long-standing one. Similarly, you can see the distribution across different countries, which is useful to see if a theme is a regional issue.
Comparing across dimensions
Sometimes you might want to focus on reviews from a specific time period. For example, you want to see how the app has been doing in the last quarter. You can filter the reviews by time. You can also see a comparison between the selected time frame and the previous period. The numbers in red or green indicate the change in the percentage of users who mentioned the theme.
You'd be able to quickly see what's going on with a quick glance. Take a look at the screenshot below and try to answer: how has Procreate been doing in the areas of "Brushes", "Photo import" and "Color tool", in terms of user reviews in Q4 2023?
Time is not the only dimension you can filter by. You can also filter themes by version, and compare with another version (or all versions). The best part? Filters are customizable. You can enable it for any field you want as long as it exists in the metadata. For example, you can add a "Country" filter.
Humans in the control
Text analysis is by nature subjective. Different teams, different people or even the same person at different times might have different interpretations of the same text. Painboard is not meant to replace human analysts. Instead, it's meant to be a tool that helps them do their job better and faster.
This user quote really illustrates the point:
AI does the lion's share of the grunt work and then a human comes back and curates the result. That's really powerful!
Painboard analyzes all the reviews, and collects an initial set of themes. Then it's up to you to decide which themes to keep, which ones to merge, and which ones to discard. You can also create new columns and move themes around.
For example, in the "User interface" column, there are two themes that are very similar: "Complexity and User Interface Issues" and "Confusion and Intuitiveness of the App". You can merge them into one theme, and rename it to "Complexity and Intuitiveness of the App".
Remember, on this merged theme, you have the same ability to view the stats, the original quotes, the reports, and filtering an comparison.
Conclusion
Alright. This is the end of our journey through the analysis of Procreate's user reviews. It has highlighted Painboard's strengths in its glanceable interface and extensive customizability. We've seen how these features enable a swift identification of common themes, detailed analysis, and cross-dimensional comparisons.
It's important to note that Painboard supports many data sources in addition to App Store, such as Google Play, Intercom, Zendesk, Jira, Gong and more. You could either put them on separate boards, or merge them into a single board for comparison across different channels. Painboard is your one-stop destination for all your customer feedback analysis needs.
Do you want to analyze your own app reviews or other qualitative data with Painboard?