Self-Consistency in Prompt Engineering

Proposed by researchers, self-consistency aims to "replace the naive greedy decoding method used in chain-of-thought (CoT) prompting." The core idea is to sample multiple different reasoning paths via few-shot CoT and select the most consistent answer from generated results. This helps improve CoT prompting performance on tasks involving arithmetic and commonsense reasoning.

Arithmetic Reasoning Example

Prompt:

Plaintext

When I was 6 years old, my sister was half my age. Now I am 70 years old, how old is my sister?

Incorrect Output:

Plaintext

35

Improvement via Self-Consistency

Using few-shot exemplars (referenced from Table 17):

Prompt Set:

Plaintext

`Q: There are 15 trees in the forest. Forest workers will plant trees in the forest today. After completion, there will be 21 trees. How many trees did the forest workers plant today?
A: We started with 15 trees. Later, there were 21 trees. The difference must be the number of trees planted. Therefore, they must have planted 21 - 15 = 6 trees. The answer is 6.

Q: There are 3 cars in the parking lot, and 2 more cars come. How many cars are in the parking lot?
A: There were originally 3 cars in the parking lot. 2 more cars arrived. Now there are 3 + 2 = 5 cars. The answer is 5.

Q: Leah has 32 chocolates, and her sister has 42 chocolates. If they eat 35 chocolates, how many chocolates do they have left in total?
A: Leah has 32 chocolates, and her sister has 42 chocolates. This means there were initially 32 + 42 = 74 chocolates. 35 chocolates were eaten. Therefore, they have 74 - 35 = 39 chocolates left in total. The answer is 39.

Q: Jason has 20 lollipops. He gives some lollipops to Denny. Now Jason only has 12 lollipops. How many lollipops did Jason give to Denny?
A: Jason had 20 lollipops. Since he now has only 12, he must have given the rest to Denny. The number of lollipops he gave to Denny is 20 - 12 = 8. The answer is 8.

Q: Shawn has five toys. At Christmas, he received two toys from his parents. How many toys does he have now?
A: He had 5 toys. He received 2 toys from his mother, so after that he had 5 + 2 = 7 toys. Then he received 2 toys from his father, so in total he has 7 + 2 = 9 toys. The answer is 9.

Q: There are 9 computers in the server room. From Monday to Thursday, 5 computers are installed each day. How many computers are in the server room now?
A: There are 4 days from Monday to Thursday. 5 computers were added each day, meaning a total of 4 * 5 = 20 computers were added. There were initially 9 computers, so now there are 9 + 20 = 29 computers. The answer is 29.

Q: Michael has 58 golf balls. On Tuesday, he lost 23 golf balls. On Wednesday, he lost 2 more. How many golf balls does he have left at the end of Wednesday?
A: Michael initially had 58 balls. On Tuesday, he lost 23, so he had 58 - 23 = 35 balls left. On Wednesday, he lost 2 more, so now he has 35 - 2 = 33 balls. The answer is 33.

Q: Olivia has $23. She bought five bagels at $3 each. How much money does she have left?
A: She bought 5 bagels at $3 each, meaning she spent 15 dollars. She has 8 dollars left.

Q: When I was 6 years old, my sister was half my age. Now I am 70 years old, how old is my sister?
A:`

Sampled Outputs:

• Output 1:

  Plaintext

  When I was 6 years old, my sister was half my age, which is 3 years old. Now that I am 70 years old, she is 70 - 3 = 67 years old. The answer is 67.

• Output 2:

  Plaintext

  When the narrator was 6 years old, his sister was half his age, i.e., 3 years old. Now that the narrator is 70 years old, his sister should be 70 - 3 = 67 years old. The answer is 67.

• Output 3:

  Plaintext

  When I was 6 years old, my sister was half my age, which is 3 years old. Now that I am 70 years old, she is 70 / 2 = 35 years old. The answer is 35.

By analyzing the majority-consistent result, the final answer is 67.