In our member services operations, we ask members to rate their interactions with us on member satisfaction (MSAT, general satisfaction with Oscar) and agent satisfaction (ASAT, their satisfaction with the care guide who helped them). How good is GPT-4 at estimating those two ratings from member services transcripts?

To test this, we pulled transcripts for each combination of MSAT and ASAT, which both go from 1 (bad) to 5 (great).

The prompt: “ You are to evaluate the quality of a conversation between a member of a health insurance company and the insurer's customer service agent. You will have transcript of the conversation. After the call ended, members are asked to evaluate how satisfied they were with the agent on the call (agent satisfaction, or ASAT), and they are asked how satisfied they were with their general experience with the health insurer (member satisfaction, or MSAT). Members can rate both satisfaction scores, ASAT and MSAT, on a scale of 1 to 5. A rating of 1 means the member is extremely dissatisified, and a rating of 5 means the member is fully satisfied with the agent (ASAT) or insurer (MSAT). Your task is to guess, from reading the transcript, how the member would likely have rated ASAT and MSAT. State your guess for ASAT and MSAT, and describe for each why you guessed the way you did. “

This is zero-shot learning, and the results aren’t particularly impressive. Because MSAT and ASAT can differ, we need to collapse them into one metric on the x-axis, so here we average them:

Here we just look at those calls where MSAT and ASAT are the same:

MSAT seems easier to measure than ASAT, it has slightly more variation in the chart above. But generally, GPT stays way too close to the middle of the range, and gets both bad and good calls wrong.

Now let’s try to move this more towards extracting meaning and drivers of satisfaction.

The prompt: “

A health insurer records and transcribes all conversations between the insurer's customer service agents and the insurer's members. After each conversation, the member is asked to respond to a survey that evaluates their satisfaction as a member. The member rates how satisfied they were with the agent on the call (agent satisfaction, or ASAT), and they rate how satisfied they are with their general experience with the health insurer (member satisfaction, or MSAT). Members can rate both satisfaction scores, ASAT and MSAT, on a scale of 1 to 5. A rating of 1 means the member is very dissatisfied with the agent (ASAT) or insurer (MSAT), and a rating of 5 means the member is very satisfied with the agent (ASAT) or insurer (MSAT). Below, you will see the transcript of such a conversation, and how the member rated MSAT and ASAT after the conversation. You are the most competent supervisor of the health insurer's customer service agents and have deep empathy for members dealing with the complicated U.S. healthcare system. You are the best at helping your customer service agents understand in detail how they can do a better job to lift the satisfaction of their members. For that reason, you are to identify the reasons why the customer in this transcribed conversation rated ASAT and MSAT the way they did, so you can help other customer service agents do the best imaginable job in the future and make their members happy. For ASAT, list up to 3 reasons why the conversation made the member rate the ASAT in the way they did. Describe each reason in sufficient detail for future education of customer service agents. Include the evidence or clue from the call transcript that made you pick the reason. If the ASAT is bad, find reasons why it is so bad; if it is good, find reasons why it is good. If it is balanced, find both. For each reason, estimate the impact that this reason alone would have had on the member's ASAT rating: -- for very negative, - for negative, +- for neutral, + for positive, ++ for very positive. For MSAT, list up to 3 reasons why the member rated his general satisfaction with the insurer in the way they did. Describe each reason in sufficient detail for future education of customer service agents. Number each reason by 1 to 3 (or fewer). Include the evidence or clue from the call transcript that made you pick the reason. If the MSAT is bad, find reasons why it is so bad; if it is good, find reasons why it is good. If it is balanced, find both. For each reason, estimate the impact that this reason alone would have had on the member's MSAT rating: -- for very negative, - for negative, +- for neutral, + for positive, ++ for very positive. Output your data in the following format of the example below:

ASAT Driver 1 Description = The agent was patient and polite.

ASAT Driver 1 Impact = ++

ASAT Driver 1 Evidence = When the member got upset that the insurer would not cover the cost of the surgery, the agent very calmly described how the member can appeal the decision by filing a case. The member evidently calmed down after the agent’s thoughtful discussion.

ASAT Driver 2 Description = The agent solved the member’s problem without further delay.

ASAT Driver 2 Impact = ++

ASAT Driver 2 Evidence = The agent was able to authorize the member’s drug reimbursement on the call. He did not want to make the member call a different phone number. Instead, he went to great lengths to solve the issue directly.

ASAT Driver 3 Description = The agent lacked confidence in some of the information he gave.

ASAT Driver 3 Impact = -

ASAT Driver 3 Evidence = When the member asked about whether podiatry is included in his plan benefits, the agent stammered and was not sure that would be the case. Even after consulting documents, he did not give a clear answer.