How ChatGPT makes your Experience Management survey more intelligent
In less than a year, ChatGPT is all of a sudden everywhere. New business models are rapidly proliferating, but you don’t need to search far at all to see how Large Language Models can support existing business solutions and processes. In this blog post, Larissa Haas, Senior Data Scientist at sovanta, shows how integrating ChatGPT into Qualtrics can automate tedious follow-up surveys.
Use Case: Employee Technology Experience Surveys
Companies are increasingly interested in the extent to which employees utilize their software tools and whether such tools meet employee expectations. These assessments seek to capture what’s referred to as Employee Technology Experience (ETX). They correlate analytical data, such as login frequency or other usage patterns, with surveys that collect employee perceptions about using the technology.
Systems like Qualtrics XM greatly simplify the work of obtaining and processing the necessary experiential and operational data. Until recently, however, tasks like fully analyzing open text survey responses or picking a direction for additional analysis have resisted automation. ChatGPT and other Large Language Models can take these tasks over, saving time, money, and valuable human analytical capacity. How? Let’s take a look, sketching out a use-case where ChatGPT takes answers from a completed Qualtrics survey and generates follow-on questions for the next one.
ChatGPT automatically asks your follow-up question
Imagine a typical ETX project. As a transformation effort runs its course, the project team wants to monitor if their technological transformation makes good progress. At a critical touchpoint a survey window opens and closes to collect feedback and when this happened, the project team finds patterns within responses using various digital tools. Then they formulate additional questions for another user questionnaire, everything with the goal in mind to get the right answers to shepherd the project along.
By integrating ChatGPT into Qualtrics, this entire process gets a lot easier. How? We replace the manual analysis and formulation of follow-up questions with an intermediate step hosted on Cloud Foundry which sends carefully designed prompts to ChatGPT (or any other custom Large Language Model). ChatGPT generates brand new questions, which we use to automate the creation of a survey via the Qualtrics Survey API.
Advantages of ChatGPT over other text analysis tools
Before we go any further, a clarification. Don’t platforms like Qualtrics already include machine learning approaches that summarize text, cluster text into topics, or extract keywords? They do, but the advantage of ChatGPT is its greatly reduced need for structure. The previous generations of text analysis tools do not allow us to extract insights on topics that we have previously defined, since to do this requires that we make assumptions about what we will find in the data.
For a clustering algorithm, we need to know the number of clusters beforehand. For a classification algorithm, we need to define the topics we want to classify. For a text summary, we might lose data if the words have a high ambiguity. ChatGPT understands the “meaning” behind the words thanks to its very large language model. It understands that we want the “most important” topics, not the top three, and it knows that it should formulate questions out of them. This step of text generation makes it very easy for us to use within this use case, especially in comparison to existing alternatives.
Let’s review the technical setup
How did we realize this? The following paragraphs show you what we did technically to perform this use case, and as you will see, it was not as hard as it sounds.
Qualtrics Survey Setup and Intercept
The first step to start this journey is to have a Qualtrics survey at hand. In my case I implemented a classical ETX Survey with four short questions:
- How much do you like using [TOOL NAME] for your daily work? 5-point-scale
- How do you rate [TOOL NAME] in terms of…
- … how effective you feel while working with it. (0-100)
- … how much you enjoy working with it. (0-100)
- … how much time it saves you. (0-100)
- How frequent do you use [TOOL NAME]?
- What do you else think about [TOOL NAME]?
This last question was the most important in our approach, though the others are also handy to filter and analyze the results later. In the free text question, we often find the most valuable insights. It’s a tall order to receive open-ended feedback, so if we do receive answers, they should be taken seriously and we should derive actions from those key insights.
The survey was then added as a website intercept directly within the homepage of our tool. The advantage of this approach is that employees can provide their feedback directly from within the tool they are working. With Qualtrics you can configure different versions, for example showing the survey when certain triggers are met, or only showing it to a specific fraction of users.
Controller on SAP BTP Cloud Foundry
Now that you have started collecting results, you can think about your middleware. In our case we used a Python FastAPI service deployed on the SAP BTP Cloud Foundry to handle the next steps. This Python service does the following:
- Receives answers from Qualtrics API (see API documentation)
- Sends responses to ChatGPT with prompts to extract most frequent topics from the text questions
- Turns those topics into new questions
- Creates a new survey within Qualtrics via API with the newly generated questions
We’ve created an additional dashboard to perform and show these steps, but this is not strictly necessary. It is also possible to configure a cronjob that runs this every week or at the end of every month. With this you get constant feedback about your tool and the evolution.
Quick Tip: Just be careful to not overload your employees with surveys and questions.
The last step is now to send out the new survey, wait for responses, and act on the feedback received.
This is just the beginning …
If you liked the idea, there are also other ideas for extending and transforming this use case such as:
- weekly summaries of open text results received via email,
- general summary of top features and flop features
- automatically generated frequently asked questions to publish on the FAQ site of your tool.
I am sure there are tons of other ideas about how to include ChatGPT into your daily workflow to make your applications a little bit more “intelligent”. I would be very happy to share ideas with you. Feel free to contact me.
