Skip to main content
Category: Effects
Type: Judgment bias
Origin: Daniel Kahneman and Amos Tversky, Psychological Review (1973)
Also known as: unwarranted confidence from a good fit
Quick Answer — The Illusion of Validity is confidence in a fallible prediction that comes from how well the outcome fits the evidence. Daniel Kahneman and Amos Tversky named it in 1973. In their grade study, people were more sure when two aptitude tests agreed. That is the setting where a combined prediction is weaker, if each test is equally valid on its own.

​
What is Illusion of Validity?

The Illusion of Validity is unwarranted confidence in a fallible prediction. The confidence comes from a good fit between the chosen outcome and the evidence, rather than from a record of how often that kind of prediction is right.
People are prone to experience much confidence in highly fallible judgments, a phenomenon that may be termed the illusion of validity.
That sentence is from Kahneman and Tversky, Psychological Review, 1973, page 249. The fit can be a coherent story, a consistent set of scores, or an extreme value that resembles the evidence. Representativeness Heuristic is the shortcut that picks the outcome which resembles the evidence. This page is the confidence that shortcut leaves behind. Overconfidence Effect later splits confidence into a score, a rank, and a range. Those quiz points stay on that page. Two introductory grades of B feel like a surer student than an A and a C, even when the average is the same B. The matching pair is one kind of evidence said twice. The Illusion of Validity is what happens when that repetition is read as a stronger prediction.

​
Illusion of Validity in 3 Depths

  • Beginner: A tidy story is a feeling of fit. Before you trust two matching grades, a smooth interview, or a folder that says the same thing in every note, write whether the clues repeat one fact or add a new one.
  • Practitioner: Put a known accuracy, or the share of people who succeed anyway, beside the feeling before you act. If you only follow the people you picked, their success rate is the record of that group.
  • Advanced: When each clue is equally valid, agreement among the clues can raise confidence and lower the combined correlation. A later success can also be the help you then gave the people you picked. A remembered gain after a structured army interview is a different report from the vivid exercise that felt certain.

​
Origin

The name was coined beside an obstacle course and printed in a journal almost twenty years later. The counted demonstration is a grade prediction. The course itself has no printed correlation. Daniel Kahneman’s Nobel biographical essay says he coined “the illusion of validity” while assessing officer candidates for the Israel Defense Forces. He was drafted in 1954 as a second lieutenant. After a year as a platoon leader he moved to the Psychology branch. One duty used methods from the British Army of the Second World War. Eight candidates, with rank markings removed and only numbers to identify them, lifted a telephone pole and had to take it past an obstacle such as a 2.5-meter wall. The pole was not to touch the ground or the wall, and no candidate was to touch the wall. If any of that happened, they declared it and started again. Two observers watched. The exercise often took half an hour or more. Every month or so, a statistics day brought feedback from the officer-training school. The essay says the ability to predict performance at the school was negligible. The next day a new group produced the same clear sense of who was a leader. Kahneman writes that the statistical information and the experience of insight did not connect. He calls it the first cognitive illusion he discovered. The essay says the term entered the technical literature almost twenty years later, in Kahneman and Tversky, 1973. It prints no correlation for the pole-and-wall ratings. The paper that prints the name is “On the psychology of prediction,” Psychological Review, volume 80, pages 237–251, July 1973. The byline lists the Hebrew University of Jerusalem and the Oregon Research Institute. The journal records receipt on 8 November 1972. The thesis of that paper is prediction by representativeness. The section “Confidence and the illusion of validity” argues that confidence tracks how representative the chosen outcome is. Consistency of the input is a major part of that fit. The same paper, in an earlier section, reports a field observation from the Officer Selection Board of the Israel Army. Experienced officers normally scored candidates on a 7-point scale after several days of testing and observation. For the study they also predicted, for each candidate who got through that assessment, the final grade in officer training. In over 200 cases, judged by a substantial number of different officers, the distribution of predicted grades was virtually identical to the distribution of actual grades. The exception was that predicted failures were less frequent than actual failures. The two highest categories matched in frequency. The judges knew of research putting their predictive validity on the order of .20 to .40. The predictions were still as extreme as the grades. That range is the research they knew. The paper does not compute a correlation on those 200 predictions. Regression to the Mean is the page for an extreme measure being followed by a result closer to the middle. The 1974 article in Science lists the illusion of validity as one way representativeness departs from probability. The representativeness page already treats that list as a map back to the 1972 and 1973 papers, rather than as six new trials inside the 1974 article.

​
Key Points

Four checks keep the Illusion of Validity tied to the fit, the correlation, and the people you never see. The numbers stay inside the passage that printed them.
1

A tidy profile feels more predictable

Kahneman and Tversky’s picture is an overall B. Two introductory courses of B support that average with more confidence than an A and a C. The matching grades are consistent, so the predicted score looks representative. They cite Paul Slovic, 1966, for a drop in confidence when the input is inconsistent. This page does not carry Slovic’s figures. They also say the intuition is incompatible with the usual linear model, in which expected accuracy does not depend on scatter inside the profile. The A and the C are a less tidy profile in the example. The model they cite does not treat that scatter as a lower expected accuracy.
2

Clues that move together weaken the combination

Consistent profiles are common when the clues are highly correlated. Given clues of stated validity, the multiple correlation with the outcome falls as the correlation among the clues rises. Confidence moves the other way. The grade study found more confidence on the correlated pair, across the whole range of predicted scores, t = 4.80, df = 129, p < .001. The paper does not print a mean confidence. It does not score the predictions against later grades. The lower validity is the model’s reading of the instructions.
3

Count the margin over the base rate

Hillel J. Einhorn and Robin M. Hogarth, in 1978, ask why experience fails to shake this confidence. When a judgment is used to choose an action, the later record is mostly the people you accepted. Their illustration, which is not a hiring audit, sets an 80 percent hit rate among the accepted next to a 70 percent base rate. The comparison they ask for is the margin. In the extreme illustration, ability and the effect of the action are both zero and the base rate is .75, so the hit rate among the accepted is still .75. Base Rate Neglect is the page for setting a single case apart from the share in the group.
4

A known moderate validity can leave the forecast extreme

On the officer board, the judges knew a validity on the order of .20 to .40, and the spread of predicted grades still matched the spread of actual grades, with too few predicted failures. The paper says confidence in this kind of prediction is a J-shaped function of the predicted level. It cites Johnson, 1972, and the authors’ own studies. People are very sure of outstanding success, and surer of utter failure than of a mediocre result. When the forecast is not pulled toward the middle, the largest misses sit at those extremes. No curve from Johnson is printed here.

​
Applications

Use the fit, the base rate, and the follow-up when a tidy file is about to be treated as a validity. Leave each printed figure inside its own study.

Label each clue as a repeat or a new fact

In a class report, a job folder, or a family school file, list what each note measures. Two similar courses of B are the paper’s picture of a consistent profile. Add one clue from a different kind of work, something that can disagree, before the agreement itself raises your confidence. Scatter inside one profile is the A-and-C picture. The linear model they cite does not lower expected accuracy for that scatter.

Write the base rate beside the hit rate

If you admit, hire, or fund only the people who looked promising, write the share who succeed in the wider pool before you read the successes. Einhorn and Hogarth’s illustration is 80 percent against a 70 percent base rate. The margin is the comparison. It is an illustration in their argument. If you have no record of the people you turned away, the correlation is still missing.

Set the forecast inside a known accuracy

When a team already has a validity for this kind of rating, put that number beside the vivid candidate before you match the story’s extreme. The officer-board judges knew a range on the order of .20 to .40, and the two highest predicted grades still matched the actual frequencies. That range belongs to the research they knew. A new interview needs its own record. Regression to the Mean is the pull toward the middle.

Separate the help from the judgment

A training seat, a grant, or a manager’s extra time changes the later result for the people you picked. Einhorn and Hogarth use research grants as an illustration and write that they know of no data on that case. The success of funded work mixes the selection with the resources. Read the later success as a mixed record. A validity coefficient would have to separate the two.

​
Case Study

The counted demonstration sits in the section that names the Illusion of Validity. Kahneman and Tversky asked people to predict grade point average from two pairs of aptitude tests. One pair, creative thinking and symbolic ability, was described as highly correlated. The other pair, mental flexibility and systematic reasoning, was described as uncorrelated. The scores people saw fit those descriptions. For half the subjects, the labels on the correlated pair and the uncorrelated pair were reversed. Everyone was told that all four tests had been found equally successful in predicting college performance. The paper’s default pool, unless a study says otherwise, was paid volunteers recruited through a student paper at the University of Oregon, in group settings. This study does not name another pool. It does not print a sample size beside the degrees of freedom. Under that instruction, the uncorrelated pair is the stronger combined predictor. Each test is equally valid, and tests that do not move together add information the other test did not already carry. The correlated pair repeats itself. The authors’ multivariate model says the multiple correlation falls as the correlation among the inputs rises. Confidence went the other way. Subjects were more confident when they predicted from the correlated tests, across the whole range of predicted scores, t = 4.80, df = 129, p < .001.
That is, they were more confident in a context of inferior predictive validity.
The study reports no mean confidence and no later grade. “Inferior validity” is what the model says about the instructions. It is not a hit rate computed on these predictions. The names of the tests were assigned, and reversed for half the subjects, so the result follows the correlation label. It does not follow a special property of creative thinking or of systematic reasoning. A t of 4.80 in this exercise stays inside the exercise. The officer board’s .20 to .40 is a different passage.

​
Boundaries and Failure Modes

The Illusion of Validity is the wrong summary when three army reports are merged into one coefficient, when an illustration is treated as an audit, and when a neighbor’s statistic is carried across. The first boundary is the army, which is three reports. The biographical essay’s pole-and-wall exercise had eight candidates at a time, two observers, and feedback described as negligible. No coefficient is printed. The 1973 paper’s officer board is another passage. It covers over 200 cases, a 7-point scale, and several days of observation. The judges knew a validity on the order of .20 to .40. The grade predictions stayed as extreme as the outcomes. The essay’s later assignment, which he dates to 1955, was an interview for combat-unit recruits, just after Paul Meehl’s 1954 book. The essay says that book showed clinical prediction to be consistently inferior to actuarial prediction. The interviewers were a small cadre, mostly young women just out of strong high schools, chosen for psychometric scores and an interest in psychology. They had been forming a general impression and then a global rating. Kahneman introduced six ratings from questions about civilian life. He remembers the dimensions as including masculine pride and sense of obligation. After they objected to losing that global judgment, he let them add one at the end. His recollection of the first validity data, against ratings of later performance, is correlations close to .30, set against about .10 for the previous methods. The global evaluation at the end was the most predictive of the ratings they made. He writes that trying to be reliable had made them valid. Those two recollected correlations belong to that interview. They are separate from the pole-and-wall feedback and from the board’s .20 to .40. The essay says the interview remained in use, with little modification, for many decades. It does not print a sample size, and it does not treat the result as a difference between women and men. The second boundary is the 1978 paper, Psychological Review, volume 85, number 5, pages 395–416. Einhorn was at the Graduate School of Business, University of Chicago. Hogarth was at the London Graduate School of Business Studies and at INSEAD in Fontainebleau. The 80 percent, the 70 percent, and the base rate of .75 are illustrations in the argument. They cite Taylor and Russell, 1939, for tables in which a low correlation can still yield a high hit rate among the accepted when few people are selected. No cell from those tables is copied here. The grant story is marked as an illustration with no data. Their learning model says that once the confidence is acquired, negative feedback on a partial schedule should be slow to remove it. That is a prediction of the model. The paper does not print an extinction curve. They also note that personnel officers are often penalized for a bad hire and rarely asked about the people they turned away. That structure is not a counted rate of missed candidates. The third boundary is export. The engineer-file medians, Dick’s sketch, and the Tom W. ranks stay on Representativeness Heuristic. The result that 73 of 108 graduate students built a symmetric interval around an IQ of 140, and the flight-school question, stay with regression. The quiz points 0.79, minus 0.22, 0.48, and minus 1.36, and the 73.1 percent hit rate, stay on Overconfidence Effect. Langer’s cents, asking prices, and office shares stay on Illusion of Control, which is a personal chance set above the odds on a chance task. Illusory Superiority is the page for placing oneself above others on an easy trait. Einhorn and Hogarth cite Oskamp’s rise in confidence, and those percentages stay on the overconfidence page. Each of these figures has its own study.

​
Common Misconceptions

Three shortcuts travel with the name. Agreement is read as extra validity. A high hit rate among the chosen is read as a correlation. One army memory is asked to hold three different numbers.
Agreement raises confidence. In the grade study the correlated pair produced more confidence, t = 4.80, df = 129, p < .001, under instructions that made that pair the weaker combined predictor. Two B’s, against an A and a C, are a second picture: a tidier profile feels surer, while the linear model they cite does not change expected accuracy for scatter inside the profile. The study never scored a later grade point average. The t stays inside the confidence comparison.
Eighty percent is Einhorn and Hogarth’s illustration of a hit rate among accepted cases. Set next to a 70 percent base rate, the margin is 10 points. In their extreme illustration, ability is zero, the action changes nothing, and the base rate is .75, so the hit rate they describe is .75. Without the people you rejected, the percent correct describes the group you decided to see. Taylor and Russell’s tables are cited for the same structure. No cell from those tables is printed here.
Negligible is the essay’s word for the pole-and-wall feedback, and the essay prints no coefficient there. The order of .20 to .40 is the research the officer-board judges already knew, in a passage about more than 200 grade predictions. Close to .30, against about .10, is a recollection of the later structured interview, where the free global rating at the end was the most predictive of the ratings. The three reports describe three procedures. None of them is the t of 4.80.
The Illusion of Validity sits next to resemblance, a too-sure score, and a missed base rate. It also sits next to a forecast that stays too extreme, a steered chance, and a record that shows only the people still in view. The links are neighbors. Each one keeps its own evidence.

Representativeness Heuristic

Picking the outcome that resembles the evidence. The 1973 paper traces this confidence to that shortcut. The engineer medians and Tom W. stay on that page.

Overconfidence Effect

Confidence ahead of a score, a rank, or a range. The Illusion of Validity is confidence from a fit. The 2008 quiz points stay on that page.

Base Rate Neglect

Judging a case apart from the share in the group. Einhorn and Hogarth ask for the hit rate minus that share. Their 80 and 70 are not the cab problem.

Regression to the Mean

An extreme noisy measure is followed by a result closer to the middle. The officer board predicted grades as extreme as the outcomes. The flight-school story stays on the regression page.

Illusion of Control

An expectancy that one’s own chance sits above the odds, once a skill cue is placed on a chance task. A tidy interview is a different miss from a chosen lottery ticket.

Survivorship Bias

Learning from the successes that remain visible. Here the missing people are often the ones a judge turned away, and the accepted group can look accurate because the base rate was already high.

​
One-Line Takeaway

Before you trust a tidy story, write whether the clues repeat one fact, and write the base rate beside the people you chose to see.