My proprietary survey of business owners (run in 2022)

My proprietary survey of people in design professions (run in 2023)

The World Happiness Report data set from 2015

The World Happiness Report data set from 2019

AI can do your quant analysis, but don’t get in over your head

If you're not experienced in stats, avoid relying on AI for analyses you can't verify. Some AI models are decent at math, but you should always check AI’s work, especially for complex tasks. It’s not anywhere close to perfect, according to performance benchmarking tests. GPT-4o has the best accuracy score for math problems at 76%, versus top expert humans’ accuracy around 90%. Hallucinations still happen, though, and you never know when they might.

AI is fast but can miss nuances and reliability issues
AI is great for quick analysis and visualizations but often misses subtle insights. For example, it might take survey responses at face value and present calculations as fact, without considering if participants are misrepresenting themselves, like showing overconfidence. While this isn’t always crucial for useful insights, it’s worth considering depending on your data and goals.

Example: In one of my surveys, participants seemed to overestimate their knowledge of product development, with many rating their confidence a 5 on a 1-5 scale, saying that they were certain they were developing a product customers would value. At first glance, it seemed they had no issue knowing what to build for customers. However, AI didn’t mention possible respondent bias or overconfidence in its analysis.

Additionally, none of the models flagged unreliable data, like biased survey questions, until I specifically asked for a bias assessment.

AI can adapt quickly and save time, but check that it adapts correctly

AI may attempt an analysis even when the data isn’t formatted correctly. For example, Julius AI tried to create an Odds Ratio using data that wasn’t suitable, initially using an inaccurate proxy. After I provided feedback on how to work with specific relevant parts of the data, it improved and generated an accurate odds ratio. Always double-check AI’s data manipulation before trusting its final analysis output.

AI can make odd data categorizations
AI sometimes creates unusual groupings. In one test with World Happiness Report data, Julius AI - using ChatGPT-4o - grouped the U.S., Canada, Australia, and New Zealand into a region called "North America and Oceania." (ChatGPT and the other models tested in their own environments did not). This error was easy to catch by skimming the Python code, but more complex mistakes could be harder to spot.

Example: I used two data sets from the World Happiness Report to analyze “generosity” across regions. The data didn’t include a Region column, so AI fetched a Python library to match countries in the data set to geographic regions. However, it combined Australia, New Zealand, the U.S., and Canada into a region called “North America and Oceania” without explanation. Firstly, I requested that it focus on North America and Europe for the analysis (not Oceania). Second, that’s not a commonly used geographic region…

Visualizations can be useless unless you tell AI which parts of the data to use

When I let AI decide which parts of my data to visualize, it frequently generated graphs that weren’t helpful. Example: One model created a confidence interval analysis for my survey, but it included free-text responses from the “Other” answer option alongside multiple-choice answers on the X-axis. The result was a messy, confusing graph that I couldn’t use as-is.

When I got clear on the type of graph I wanted first, and specified the exact columns and responses to use from my data, AI consistently delivered what I needed much faster and very accurately. Tip: Imagine what kind of visualization and output you’re looking for, and back into how to ask AI to create it. It sounds obvious, but I’ve seen too many people jump into AI for analysis and let it do the thinking for them.

AI can prepare and format data much faster for diverse analyses

AI is proactive in some helpful ways, such as automatically renaming mismatched column headers when merging datasets. This saved me a lot of time and ensured consistency in the analysis across different reports. It’s something that would have taken me significantly longer as a non-expert at Python.

A tool that offers more transparent step-by-step analysis can be better for less experienced users

Many of us struggle to understand how AI reaches analysis conclusions. I found Julius to be more transparent than ChatGPT for showing it’s “thinking” process. It explains its process step-by-step, including showing the Python libraries and code used, allowing users to review and edit steps to improve accuracy and relevance as an analysis runs. It runs on the major models (ChatGPT, Claude, Gemini, Cohere), but offers its way of interfacing with the process.

Your tool choice may subtly influence your analysis approach

Different tools guide your analysis in different ways, possibly without you noticing. Julius AI prompts you to pick from common analysis types immediately, which can limit broader exploration. In contrast, ChatGPT starts with general suggestions like “identify top trends” or “compare data sets,” starting in a more open space. This helped me uncover interesting insights in the World Happiness Report data using ChatGPT that Julius AI didn’t catch, because Julius steered me toward specific methods that weren’t always as useful.

The AI model equals human emotion recognition ability.
It’s emotion-recognition accuracy of 68.9% doesn’t sound like much, but it matches human ability.

Machine Learning models and humans both more effectively recognize negative affective states (emotions).

The theory about why is that negative emotional states and expressions are more intense, and therefore potentially more straightforward to recognize.

My interpretation: we can likely use AI to recognize friction and problems in a user’s experience - such as in product tests - about as well as a human can. 👀

The biggest performance difference was on the correctness level of analysis (and humans win).

Senior Data Analysts received a correctness rating of 0.98 out of 1 (near perfect), while GPT-4 scored 0.82. While 0.82/1 is still a high score, there’s a noteworthy difference between the scores. Humans still win.

GPT-4 performed on par with junior and intern-level analysts.
In tasks like creating visualizations and extracting data (see “Chart Type” and “Aesthetics” in the table above), GPT-4 performed almost as well as entry-level analysts.

Human analysts still outperform GPT-4 on nuanced insights.
Senior analysts provided deeper, more context-rich interpretations of the data. GPT-4 sometimes lacked the ability to interpret complex patterns or offer cautious, assumption-based insights, instead making overly confident predictions.