WORKFLOW UPGRADES

🖼️ Miro & Figjam AI tools in Beta

No more manually copy-pasting user quotes to sticky notes? 👀 I was hopeful. Miro and Figjam both have AI tools that seemed to help with synthesis. My take: Miro saves more time by automating sticky notes, but Figjam summarizes the content better.

Watch the video for details on how they each work, what works best, and my optimal workflow so far.

〰️

📚 A Team Prompt Library for repeatability

You know that standardizing processes lead to reliability and repeatability. Creating a prompt library for your team matters when various team members don’t have the same prompting skills - and want to get reliable output regularly. Here’s a starting point for your own prompt library.

To the prompt library template »

PROMPTING PLUS

🧑‍💻 Do you really need to learn how to prompt? A perspective.

AI expert Ethan Mollick hypothesizes that prompting skills will become unnecessary as LLMs (large language models) shift to understanding your intent and handling the prompting for you. But how long from now that happens is unknown.
For the moment, the people who can prompt really well do get a lot more out of pre-trained models like ChatGPT, Gemini, and Claude.

Listen to Ezra Klein interview Ethan Mollick on Apple or Spotify.

〰️

🔌 Plugins for your ChatGPT sidebar

If you haven’t yet tried ChatGPT plugins (for paid accounts), these are a few to add to your sidebar. Why these? You don’t need expert prompting skills for any of them, and they’ve made a huge time-saving impact for me.

• Prompt Wizard - let this plugin figure out the prompt you need for the outcome you want

• Diagrams - save time creating flowcharts, journeys, mindmaps and other diagrams to visualize your thoughts

• Scispace - speed up desk research and get summarized scientific studies with accurate sources instead of ChatGPT’s hallucinations

DATA PRIVACY

🙅 Settings in ChatGPT to know

Taking the steps below will limit ChatGPT’s ability to train its models on your data.

1. Find Chat History & Training in Settings > Data Controls and turn it off.

2. Ask OpenAI to stop using your account for training.

3. Use Playground in ChatGPT - they say that the Playground is not used for training data.

〰️

Quick reminder: When using ChatGPT and other pre-trained models…

• Never upload or input information that competitors or non-employees at your company shouldn’t know. This includes:

  • PII about your customers/users

  • Proprietary information about your company

  • Anything you think you should not talk openly about regarding users, your employer’s or client’s business, products, future developments…

AI STUDIES

🧑‍🔬 How reliably do LLMs summarize documents?

A study published this month showed that LLM models aren’t yet reliable summarizers of long-form text content.

The study looked at how well LLM models (1) accurately summarized recently released fiction books that they had not been trained on, and (2) fact-checked LLM-generated claims about the books.

They tested GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, Mistral, Claude-3-Opus. Claude-3-Opus was by far the most faithful to the text.

Why should we care?

A lot of qualitative research output is long-form text that’s analyzed similarly. In both books and our research output, there are text documents that can be broken down to capture stories with contexts, events, characters, relationships, and themes.

These study findings are applicable to customer research -

1. Accuracy / Faithfulness in interpreting content

We may want to do more quality work, faster and cheaper. But only Claude-3-OPUS is even close to humans’ accuracy in interpreting documents where “indirect reasoning” is required. And faithfulness is just at 90%.

2. Fact-checking claims

At best, LLMs had a 47.5% accuracy rating for fact-checking their own work in this study (and therefore won’t fact-check ours well, either).

3. Omissions of key information

Key information was frequently left out of the summaries. A noteworthy 33-65% of summaries failed to capture some key events across LLMs. Of the five omission categories, there were most omissions of events and attributes, and least on average of relationship information.

If this were customer research analysis, we’d likely miss events that make a big difference in how we understand customer journeys and prioritize product decisions.

4. More focus on book endings

We, too, have a recency bias, remembering more clearly what we heard or observed toward the end of a user session. LLMs prioritized information that came later in books, and might therefore prioritize observations from transcripts and notes it analyzes later in a session or a collection of documents.

Can we rely on LLMs for summarizing? My take: We shouldn’t remove ourselves from analysis and synthesis just yet.

Plus…

• An AI Glossary of terms to get familiar with

• CompliantChatGPT is a HIPAA-compliant ChatGPT that anonymizes information for you. I haven’t tested this personally yet.

• Figure out if the EU AI Act applies to AI systems you’re using at work

WHAT’S COMING NEXT?

Here’s what’s coming in the next few editions -

• How well can AI run customer sessions for us? Testing AI moderators.

• Where is AI counteracting human biases, and where does it add too much data bias to be worth it?

• So many AI tools for note-taking 🤯 Is there a winner?

• Comparing LLMs to find the best one for specific research tasks

• …and more!

See you next time!

-Caitlin

P.S. Stoked about this project and want to help it grow? My goal is to grow enough support to release two or more editions for you every month, instead of one.

I’d be extra grateful if you refer friends and colleagues below 🙏