mean+standard_deviation (25% level, 50% level/median, 75% level) [n = number of data points]
[…]
Big 5 (O): 60.6 + 25.7 (41, 65, 84) [n = 453]
Big 5 (C): 35.2 + 27.5 (10, 30, 58) [n = 453]
Big 5 (E): 30.3 + 26.7 (7, 22, 48) [n = 454]
Big 5 (A): 41 + 28.3 (17, 38, 63) [n = 453]
Big 5 (N): 36.6 + 29 (11, 27, 60) [n = 449]

So, it looks like rats are more Open than average, and less Conscientious, Extraverted, Agreeable, and Neurotic than average, but even then, the differences aren’t that large. If we convert the median percentiles to z-scores, we get:

So, it doesn’t seem like rats are that different from the general population in personality. Except, I don’t buy that. Rats are weird. They talk about weird things, do weird things, and take weird ideas seriously. So, for a trait like Openness, I’d expect rats to be much higher than just 0.39 SD above the population mean. There must be something going on.

The problem is the online personality test’s norms. The 2012 survey instructed respondents to take the OutOfService BFI-44 test¹ and enter the percentiles it gave them. If the test’s norms come from people who take personality tests online rather than the general population, those percentiles could make rats look more ordinary than they are. To check, we’d want to compare LessWrong’s raw scores with those of a more representative sample. Unfortunately, the survey only collected percentiles. Foretunately, VincentYu found the means and SDs the test used which allows me to recover most of the trait scores in the public survey data.²

For my representative sample, I’ll use Bogg & Vo (2014). They administered the BFI-44 to a weighted, probability-based U.S. sample (N = 1,015). So, how do LessWrong’s raw scores compare with the test’s assumed means and the U.S. sample’s means? Here they are:

As you can see, the online test assumed higher average Openness and lower average Conscientiousness and Agreeableness than the U.S. sample, so it made LessWrong’s differences on those traits look smaller. And while the online test made rats look less Neurotic than average, their mean is almost exactly the U.S. mean.

So, using the U.S. means and SDs, the median LessWrong score on each trait comes out to roughly:³

• 88th percentile (+1.2 SD) on Openness
• 10th percentile (−1.3 SD) on Conscientiousness
• 27th percentile (−0.6 SD) on Extraversion
• 28th percentile (−0.6 SD) on Agreeableness
• 50th percentile (0 SD) on Neuroticism

The revised scores now match my intuitive sense of LessWrong users. LessWrong covers ideas outside the norm, and the Openness score now reflects that. Additionally, many users reported suffering from akrasia, which is also now reflected in the revised Conscientiousness score.

What lesson should we draw from this? Perhaps, when an online test gives you a percentile, ask: compared with whom? If the comparison group is unusual (which it usually is), that number will give you a misleading picture of where you stand in relation to the the general population.

Here’s what each task involves:

The seven tasks report different kinds of item scores. Here is how LiveBench’s published scores relate to the responses I use in this analysis:

I don’t like how Paraphrase and Story Generation are currently graded. Their published scores average instruction-level accuracy with an all-or-nothing prompt-level component, so missing just one instruction costs more than half the grade. I therefore use instruction-level accuracy alone.

For Plot Unscrambling, I logit-transform the score using this count-adjusted formula:

where $D$ is the edit distance and $n$ is the number of sentences.

Determining the Number of Factors

Naturally, I used parallel analysis to determine the number of factors. It yielded 19, which is a lot (see the plot).

However, some item pairs have no models in common, and others have only a few, making their correlations unavailable or imprecise. So I modeled the correlation matrix Bayesianly and repeated parallel analysis across posterior draws to see how much the recommended factor count varies. Each of the first 12 factors exceeds the chance threshold in at least 95% of posterior draws, and the 90% interval for the number retained is 12–13.

Even 12–13 factors is a lot for seven tasks. I would have expected something closer to seven, so let’s look at the tasks individually to see where the extra dimensions might be coming from.

Select a task to see its Bayesian parallel analysis. The badge counts leading factors above the chance threshold in at least 95% of posterior draws.

LCB Generation2 factors

Coding Completion2 factors

Connections3 factors

Plot Unscrambling2 factors

Typos7 factors

Paraphrase2 factors

Story Generation2 factors

Although the Bayesian parallel analyses support more than one factor for every task, the first dimension dominates in six of them. The ratio of the first two (posterior-median) eigenvalues ranges from 3.1 to 10.6 for those tasks, compared with just 1.5 for Typos. The within-task item-correlation matrices offer another way to see this:

Items are ordered by median task-factor loading. Grey means unavailable, not zero. Scroll to compare tasks; select a matrix for full size.

Most of the matrices suggest a clear positive manifold. Typos is the exception: it still looks quite ugly after the items are sorted by loading. That makes me want to check whether the Typos items themselves are sound. Auditing every item would take too long, but CTT/IRT measures can flag suspicious ones for us to focus on…

Item Flags

In my post about flagging suspicious questions in AI benchmarks, I discussed flags based on item discrimination and distractor behavior. None of the items here is multiple choice, so there are no distractors to examine. That leaves item discrimination, which I measure using the corrected item–task score correlation: the correlation between an item and its task score calculated without that item.

Red: below 0 · Yellow: 0–0.2 · Green: above 0.2 · Grey: undefined. Choose a task, then select its histogram for full size.

Typos LCB Generation Coding Completion Connections Plot Unscrambling Paraphrase Story Generation

All 494 items are counted. Percentages use each task's total, including undefined correlations, and are rounded to whole numbers; rows may not sum to 100%.

The flags uncovered two Typos items with valid alternative answers, three LCB Generation items with grading problems, and three Story Generation items with prompt or checker problems. The item-by-item audit gives the archived-answer counts and rescoring checks. I could inspect only a subset of items and models, so other problems may remain.

Factor Structure

Time to look at the factor structure. I’ll exclude the items I found problems with: 832610e9 and 0becbf34 from Typos; “Wrong Answer,” “Takahashi Quest,” and “Bad Juice” from LCB Generation; and 0d828b10, 6c5eb0ac, and 230fffb5 from Story Generation. Other problematic items may remain because I couldn’t inspect them. As noted above, every task except Typos is strongly unidimensional. A separate factor analysis of Typos produced factors that were hard to interpret, so for simplicity I’ll model each task, including Typos, with a single factor.

The item loadings in the correlated-task model look like this:

Fourteen items have 90% loading intervals that include zero. That doesn’t mean they’re flawed, but it does make them worth a closer look. I’ve put my notes in a collapsible section so they don’t interrupt the main discussion.

The task-factor correlations look like this:

There’s a clear positive manifold, and parallel analysis of the task-factor correlation matrix supports a single factor. This would imply a hierarchical model in which a higher-order factor explains the correlation among tasks. However, since LiveBench groups tasks into categories, we might instead add Coding, Language, and Instruction Following domain factors. Those domains could correlate freely or load on an even higher-order factor.

Unfortunately, each of these models fits worse than the correlated-task model (see the initial comparison).

The domain estimates help explain why. In the correlated-domains model, the domain correlations are quite high:

Even so, they can’t account for some of the task-factor correlations, as we’ll see below. In the grand-factor model, all three domain loadings are very close to 1; the lowest loading is 0.9993. That leaves little domain-specific variance, so modeling the categories doesn’t seem to add much.

That brings us back to the hierarchical model. It also fits worse than the correlated-task model, but I find it more plausible a priori. The positive manifold is what I’d expect from LLMs, and parallel analysis of the task-factor correlations supports one common factor. Its poorer fit suggests that the general factor alone misses some relationships between tasks. We can allow for those relationships by adding correlated residuals, so that selected task factors can correlate more than the general factor predicts.

To see which links might be worth including, I fit an exploratory hierarchical model with positive-only shrinkage priors on all 21 task-residual correlations:

The largest estimated residual correlations are Plot Unscrambling–Typos (+.52), LCB Generation–Coding Completion (+.29), and Connections–Story Generation (+.27). But Coding Completion’s loading on the general factor is almost one in this exploratory fit, leaving virtually no task-specific variance. Its +.29 residual correlation therefore adds only about +.002 to the implied correlation between the two tasks. I’ll include Plot–Typos and Connections–Story, but not LCB–Coding. Because the exploratory prior rules out negative residual correlations, an interval above zero is not, by itself, a reason to include a link.

In the final hierarchical model, only those two residual correlations are estimated, with priors that allow either sign. All other residual correlations are fixed at zero. The posterior estimates are:

Comparing the new model with the previous ones:

The two residual correlations improve the hierarchical model’s fit.

The loadings of the seven tasks on the general factor in this fit are:

We can also ask how much of each task’s total-score variance is attributable to the general factor, its task-specific factor, or item-specific variation. This decomposition is for an equal-weighted sum of underlying item responses, not the observed mixed-format LiveBench score:

Correlations with the ECI

This compares Epoch’s ECI with posterior-mean factor scores from the selected hierarchical fit.² The general factor correlates strongly with ECI. The task factors do too, though much of that correlation appears to come from their shared general component. Once that component is removed, only the Plot Unscrambling and Paraphrase residuals have 90% intervals entirely above zero.

Blue: full task factor · Orange: task residual after removing the general factor · General factor at left. Bars show 90% bootstrap intervals; counts include direct task responses only.

Takeaways

Once again, benchmark item flags proved useful for finding problems. The task factors display a positive manifold, as expected. What’s more notable is the lack of clear domain factors beyond the general factor. Human cognitive ability is well modeled by g, but not perfectly: someone may be better at spatial tasks, and someone else better at verbal tasks, than their levels of g would predict. They could have the same g, yet if you needed to navigate an unfamiliar city or write an essay, you might have a clear choice between them.

That distinction is much less apparent for the AI models and tasks tested here. LiveBench divides its tasks into categories, but I find little evidence that these categories capture distinct abilities. It doesn’t seem especially useful to say “use Model A for Coding and Model B for Language” when performance across those domains is so closely tied to general performance. Individual tasks can still differ: Plot Unscrambling and Typos, for example, are more closely related than the general factor alone predicts. But LCB Generation and Coding Completion do not show much extra association, despite both involving coding. The distinctions worth paying attention to seem to lie with particular tasks, not the broad category labels.

Appendix

Classical Parallel Analysis

Flagged-item audit

Looking at the flagged items, along with items that have constant scores across models:

Typos

• For the item with ID prefix 832610e9, the scoring key says “algebraical” should be changed to “algebraic”. However, “algebraical” is a valid word listed in the Oxford English Dictionary. Of the 77 archived model answers I can check, 45 were scored wrong, and 19 of those retained “algebraical”. Among these models, crediting those otherwise-valid answers raises the item–Typos correlation from +.07 to +.52.
• For the item with ID prefix 0becbf34, “behavour” can be corrected to either the US “behavior” or the UK “behaviour”, but only the US spelling was accepted. Of the 77 archived answers I can check, 74 were scored wrong, and 3 of those used the UK spelling. Among these models, crediting those answers raises the item–Typos correlation from +.19 to +.35.
• The other six Typos items I was able to check showed no issues.
• I can’t audit 12 flagged Typos items. The public release has their scores but neither their prompts and keys nor archived answers.

LCB Generation

• The “Wrong Answer” item is simple: given $A$ and $B$, the program should print any digit from 0 to 9 except $A+B$. For the input 2 5, both 0 and 2 are valid, but the stored test expects one specific output, such as 2. A valid program that prints 0 therefore fails. Despite the item’s simplicity, all 159 judged models received zero. Among the 76 with archived answers, 34 have at least one well-formatted program that prints a digit other than the sum. The current item–task correlation is undefined; crediting these 34 apparently valid archived programs results in a correlation of +.56.
• “Takahashi Quest” has another fixed-output grading issue. All 178 judged models scored zero, making the current item–task correlation undefined. Among models with archived answers, crediting the 34 apparently valid programs results in a correlation of +.24.
• “Bad Juice” is an interactive problem, but it was not evaluated with an interactive judge. A correct program first prints how it will distribute the bottles among the minimum number of friends, reads the judge’s reply about which friends became sick, and then prints the spoiled bottle. LiveBench instead supplies one static input string, 3 1\n, and expects one fixed output transcript. It doesn’t provide replies tailored to each program’s printed groups. All 159 judged models scored zero. Under my provisional re-scoring of the archived answers, the item–task correlation becomes +.31, but I still cannot determine how many programs would pass a real interactive judge.
• The other three LCB Generation items I was able to check showed no issues.

Coding Completion

• The two flagged items had correct answer keys and scoring.

Connections

• The two flagged items had correct answer keys and scoring.

Plot Unscrambling (No flagged items)

Paraphrase

• The three flagged items had correct answer keys and scoring.

Story Generation

• For the items with ID prefixes 0d828b10 and 6c5eb0ac, the prompt is somewhat contradictory. Models were given a news story and told to “Please generate a story based on the sentences provided. Answer with one of the following options: (‘My answer is yes.’, ‘My answer is no.’, ‘My answer is maybe.’)”. A typical prompt instead adds instructions such as “Entire output should be wrapped in JSON format. You can use markdown ticks such as ```.”, which modify the story’s format or content. Nonetheless, 91/97 models received full credit for 0d828b10, and 93/97 received full credit for 6c5eb0ac. This concerns me: among 77 archived answers per item, 34 and 31, respectively, consisted solely of a permitted phrase. Many models passed the mechanical check, but their scores did not reflect the story-writing request.
• For the item with ID prefix 230fffb5, models were instructed to generate a story with fewer than 241 words and a P.P.S postscript at the end. The checker counted \w+ tokens, which can split hyphenated expressions and P.P.S into multiple “words” even when whitespace-based counting would count each as one. Of the 77 models with archived answers, 32 did not receive full credit, and 11 of those were affected by this word-count issue. The postscript check had a separate problem: three models put P.P.S at the beginning of their answers but still received credit for that instruction; two received full item credit. Among models with archived answers, counting words by whitespace and requiring P.P.S at the end raises the item–task correlation from +.08 to +.29.
• The other flagged item had no scoring issue.

I can inspect archived answers for only a subset of items and models, so I cannot determine the full scope of these problems. Nonetheless, I’ve tried my best doing what I can do.

Initial Model Comparison

Loadings vs Difficulties

These plots place each item’s task-factor loading against its estimated difficulty in the selected hierarchical fit.

Choose a task, then select its plot for full size and 90% intervals. Difficulty uses a task-specific response scale, so compare horizontal positions only within a task.

LCB Generation Coding Completion Connections Plot Unscrambling Typos Paraphrase Story Generation

Task Information Curves

The seven task-factor information curves are overlaid on the same axes, so their heights are directly comparable.

Select a task or its curve to highlight it. Hover, focus, or tap a model marker for its name, median task score, 90% interval, and information at that score. The full-size plots also show individual item curves.

Loading interactive model markers…

This is wrong. People dislike the idea of AI-generated content. But when you don’t tell them it’s AI-generated, people love AI slop. Sure, maybe you and your friends hate AI-generated content¹. However, you² and your circle are quite unrepresentative.

In studies where participants aren’t told whether something was made by a human or an AI, people generally rate the AI-generated version as about as good as or better than the human-generated version. This has been found for poetry, jokes, short stories, and visual art.

The poetry results are particularly illustrative. Participants couldn’t reliably distinguish AI-generated poems from poems by famous poets, and rated the AI poems higher in overall quality. Every one of the five AI poems was rated more highly than every one of the five human poems. In another study, ChatGPT-3.5’s jokes were rated as funnier than jokes written by ordinary people, while its satirical headlines were rated about as funny as headlines from The Onion. For short stories, researchers found no significant difference in human-rated creativity between human- and AI-written stories. And in one visual-art experiment, participants significantly preferred DALL·E 2 images to human-made artworks.

What seems to hurt AI-generated content isn’t the quality of the content itself, but people knowing or believing that it’s AI-generated. In the poetry study, the exact same poems received lower ratings when participants were told they were AI-generated rather than human-written. A 2024 study of visual art found something similar: participants generally preferred the AI-generated images on their actual merits, but rated images less favorably the more strongly they believed those images had been generated by AI.

And some of these studies are from up to two years ago, ancient by AI standards. We should expect current models to perform much better.

Solved hyperstimuli

A framework for thinking about “taste” that I think is apt is laid out in “High Culture and Hyperstimulus”: what separates low culture from high culture is not that low culture consists of hyperstimuli, since the same is true of high culture, but that low culture consists of solved hyperstimuli, while high culture consists of unsolved hyperstimuli.

A Playboy centerfold is an obvious example of tasteless low culture, seemingly because it presents the hyperstimulus of a naked body. But Renaissance nudes also contain naked bodies and are considered high culture. Why are alcoholics considered distasteful while sommeliers are considered sophisticated? Why is K-pop disposable mass culture while symphonies are high culture? Each of these cases involves roughly the same kind of underlying reward, so what makes one vulgar while the other is respectable?

The answer, on this framework, is whether the hyperstimulus is solved, whether we know how to produce it reliably on demand.

The alcoholic is indiscriminating in the alcohol he consumes, provided it gets him drunk, while the sommelier focuses on subtle flavors such as the notes that (supposedly) tell you where the grapes grew and under what conditions. It’s harder to mass-produce these subtle characteristics, or so the sommeliers say, than to make some kind of alcohol that can get any fool drunk. Anyone can go on the internet and look at naked bodies, but to view a Renaissance nude in a museum, you have to pretend to know something about various historical artists and movements. K-pop is famous for having a solved formula that allows producers to manufacture predictable hits.

This also explains the barber pole of culture. A particular form of art starts out as ineffable, with no one quite sure what the secret is behind it and only a blessed few able to pull it off. But as people figure out the formula, it becomes solved and, inevitably, low culture.

You can see how this poses a problem for AI-generated content: it is the epitome of low culture. Any kind of art it can produce has, almost by definition, been solved so completely that a machine can churn out effectively infinite content on demand.

Perhaps if the user is heavily involved in the creation process, then that particular piece of art might not be so low culture. But I expect that, in practice, we’ll always be a little suspicious that the “artist” was not as involved as they claim, like the person who says they wrote most of their 10,000-word essay when in reality they wrote three bullet points and asked Claude to expand on their thoughts.

Revealed taste

I also think some of this is a mixture of preference falsification and people being bad at knowing their own preferences.

The high-status thing to say in these circles is that, of course, while AI may be better than most creators in a technical sense, it’s missing that spark that is the reason we consume the work of the top 1% of creators. Or that we value human provenance³ so much that we’d actually prefer to consume genuinely kinda terrible human-made content over better AI-generated content.

I think this is mostly posturing and self-deception.

When people know they’re looking at AI-generated work, they may dislike what it represents, dislike the production process, dislike the social implications, or simply believe that liking it reflects badly on their taste. But when you strip away the label and ask them to judge the thing itself, the evidence so far suggests that their preferences lean the opposite way.

Taken at face value, this is true. The study used a scale called the Moral Foundations Questionnaire 2 (MFQ-2), which says it measures moral foundations, and cognitive ability was negatively correlated with all of its sub-scales, which happen to be labelled “Care”, “Equality”, “Proportionality”, “Loyalty”, “Authority”, and “Purity”.

But looking at just the names psychologists give to the traits they measure is a good way to be misled about reality. Looking at how they define those traits can also be a good way to be misled. If you really want to understand what a psychological scale is measuring, you should look at how it actually measures it: the questions people are being asked.

For example, with the MFQ-2, names like “Equality” and “Loyalty” make you think the sub-scales are measuring broad traits like preferences for equal treatment and rights, or how much one values attachment and obligations to one’s friends, family, and other groups. But if you look at the items, you’ll see that the Equality sub-scale largely measures support for equalizing incomes and resources (e.g., “I believe it would be ideal if everyone in society wound up with roughly the same amount of money”), while the Loyalty sub-scale largely measures patriotism and national identity (e.g., “I think children should be taught to be loyal to their country”).

So, after looking at the items, we realize “people with higher cognitive ability have weaker moral foundations” is rather misleading. It isn’t that people with higher cognitive ability are less morally egalitarian in some broad sense, or that they’re particularly disloyal to their family and friends. What the study actually establishes is quite different: people with higher cognitive ability show less support equalizing income and resources, and have less patriotism and national identity.

The Systemizing Quotient (SQ) gives another example of the same problem. The SQ is supposed to measure a trait called “systemizing”, defined as “the drive to analyze or construct systems”, and men do score higher on the SQ than women do. If you stop there, the natural conclusion is that men have a substantially stronger drive to analyze and construct systems.

But when we actually look at the items, we find that, while many do seem related to thinking analytically, they’re unusually focused on male-typical interests like cars, computers, sports, stocks, technology, etc. When we instead look at factors made up of more gender-neutral content, such as nature, language, or attention to detail, the gender difference decreases dramatically. So at least part of the famous sex difference in “systemizing” is really a sex difference in the particular kinds of systems the questionnaire asks about.

Looking at the items themselves gets us much closer to understanding what a scale measures, but we can go another step. Not every item contributes equally to a scale or factor. We should also look at the item loadings: how strongly each item is related to the latent construct the scale is supposed to be measuring.

For example, taking a look at Tailcalled’s Targeted Personality Test, one might assume that a factor named “Charisma” measures how charismatic the respondent actually is. And, to be fair, there are some items dealing with relatively concrete social behaviors (e.g., “In conversations, I jump straight to the point rather than doing smalltalk and other irrelevant things”). But the highest-loading items are much more directly about how charismatic or socially skilled the respondent perceives themself to be.

This matters if, for example, one wants to look at correlations between Charisma and other variables like self-esteem. If the Charisma score is heavily determined by questions asking people whether they think they’re charismatic or socially competent, then a correlation between Charisma and self-esteem is partly a correlation between two kinds of positive self-evaluation. That’s a rather different result from showing that people who are actually perceived by others as charismatic have higher self-esteem.

There are related issues whenever the method of measurement introduces something important into the construct. If you’re reading a study (or Substack post) that’s looking at the relationship between attractiveness and other variables, but attractiveness is measured not through external raters but through self-reports, then a substantial component of the “attractiveness” variable is going to be how attractive people think they are. So the people who score as most “attractive” won’t necessarily just be the best-looking people; they’ll also tend to be the people with the highest opinions of their own appearance.

There was a recent analysis that found that people who read horoscopes every day report personalities more similar to what we’d expect from stereotypes about their Zodiac sign, while the effect among people who never read horoscopes was approximately zero. But remember how we usually measure personality: through self-reports. One possibility is that there’s a sort of placebo affect where astrology changes people’s actual personalities if people believe in astrology. Another, much more likely possibility is that people who believe in astrology incorporate the stereotypes associated with their Zodiac sign into their self-perception and therefore into how they answer personality questionnaires. If we want to investigate the effect of reading horoscopes on someone’s actual personality, we’d have to look at other-ratings or behavioral measures rather than solely self-ratings.

The broader lesson is this: look at the items. The name of a scale is just the researcher’s summary of what they think the scale is measuring (sometimes not even that!), and researchers are often wrong. The items, on the other hand, are the essence of the scale. So, always look at the items.

AI benchmarks have a problem: lots of the questions are scored incorrectly. Sometimes the answer key selects the wrong answer as correct, sometimes the question is literally impossible and there is no correct answer, and sometimes there’s more than one correct answer. Obviously, this makes measuring AI progress harder. We might think models are plateauing when really they’ve hit the benchmark’s ceiling and many of the remaining items are incorrectly scored, ambiguous, or otherwise flawed.

For example, in September 2026, GPT-5.6-Sol scored a 47.3% mean@4 on the Physics section of Humanity’s Last Exam. However, when researchers audited questions models repeatedly got wrong, they discovered that many of the questions were incorrect, ambiguous, or otherwise flawed. In many cases, the problem was with the question, not the model. After removing or repairing the flawed questions, GPT-5.6-Sol’s measured mean@4 increased from 47.3% to 78.7%.

How are we to avoid problems like these? Well, the basic idea behind benchmarks is to measure how capable AI models are and how much they know. As it turns out, there’s a field already dedicated to figuring out how to measure abilities and knowledge accurately in humans: psychometrics.

Psychometrics is especially pertinent because cognitive and achievement tests can be very important: they can determine what college one goes to, whether one is able to enter a profession they’ve spent years and hundreds of thousands of dollars preparing for, or inform education policies affecting millions of students. In short, these tests have an enormous effect on the lives of millions.

As such, it’s very important that these tests have correct items, measure the intended trait accurately, and are free of bias. Items undergo rigorous screening before they appear on an exam. I’m going to go through a few of the flags¹ that psychometricians look at when deciding whether an item is suspicious and in need of a closer look: item discrimination, item difficulty, and distractor behavior. I’ll be looking at MMLU-Pro². For each flag, I’ll look³ at some of the most extreme items it identifies, so we can see whether they really are bad items.

Discrimination

An item’s discrimination is the degree to which the item is able to distinguish between high-ability and low-ability respondents. There are a number of different measures we could use: the $a_i$ parameter from our fitted IRF, the point-biserial correlation between item correctness and total score, or the discrimination index, which is the difference in the proportion answering correctly between the top 27% and the bottom 27%.

Any of these being negative for an item is a bad sign: it means that less capable respondents are more likely to answer that specific item correctly than more capable respondents. That should make us wonder whether the indicated correct answer is actually correct, the question is ambiguous, or something else has gone wrong with the item.

But even positive, near-zero discriminations are suspect. For example, the SAT flags pretest items for review when their item-total score correlations are below 0.20, while the ACT generally expects items to have item-total correlations of at least 0.20. PISA flags items when their IRT discrimination parameter is below 0.1, while the MBE says that, for item selection, the discrimination index should at least be positive and preferably greater than 0.20. Such low discriminations indicate that item correctness barely changes with ability, which is still a problem. Is an item really measuring ability if more capable respondents are barely more likely to answer it correctly? At the very least, something about the item warrants investigation. And even if nothing is technically wrong with it, there’s not much point in including an item that contributes almost nothing to distinguishing between more and less capable respondents. We’re just wasting time (and tokens).

When looking at discrimination in the IRT fit, instead of using a logistic model, I’ll be using a probit model and reporting the standardized loading. A common rule of thumb treats loadings with magnitudes below 0.3 as weak, so I’ll use loadings below 0.3 as a flag for review. Negative loadings are even more suspect: they mean that respondents become less likely to answer the item correctly as their ability increases.

It looks like it doesn’t much matter which measure we use, since they’re all very correlated with each other. They also all agree that ~35% of the items should be flagged for review, indicated by the red and yellow, with ~15% having discriminations of the wrong sign, indicated by the red, and another ~20% merely having very weak discriminations, indicated by the yellow.

Now, let’s look at some of the items with the worst discriminations. We’ll use the item-total score correlation, since it has the highest average correlation with the other measures:

4 out of the 5 items we looked at would not pass review. This suggests that using the discrimination flag would be quite helpful for identifying defective or anomalous items.

Difficulty

An item’s difficulty is, uh, how difficult it is. One way to measure it is to look at what proportion of respondents answer the item correctly. Another way is to look at the $b_i$ parameter of the fitted IRF, where $b_i$ indicates the ability level at which a respondent has a 50% chance of answering the item correctly.

In typical high-stakes assessments, test-makers want items to be neither too difficult nor too easy. The SAT flags items for review when the proportion answering correctly exceeds 0.90 or falls below 0.20. The ACT flags items for review when the proportion answering correctly falls outside the range 0.100–0.899 and, additionally, tries to ensure a balanced distribution of items across defined difficulty bands. PISA flags items where $|b_i| > 5$.

While it makes sense for benchmark creators to want to exclude items that are too easy, the case for excluding very difficult items is trickier. If we’re interested in measuring progress in AI, we may actually want to retain some easy items so that the benchmark remains informative for weaker models. Likewise, if you ensure that none of your items are too difficult, then your benchmark will quickly saturate and become obsolete. But the problem with items that no model can answer correctly is that, a lot of the time, the reason no model can answer them correctly is that the item is faulty, as in the case of Humanity’s Last Exam mentioned above.

Extreme difficulties also pose problems for estimating other item parameters, such as discrimination. It’s difficult to determine how well an item discriminates when nearly every model gives the same response, either because almost all models answer it correctly or almost all answer it incorrectly, regardless of ability. That said, let’s look at the most difficult items, as measured by the percentage of models answering correctly, and see what happens:

1 out of the 5 items we looked at would not pass review. In fact, the fact that so many models aren’t able to answer some of these items correctly confuses me a bit. Interestingly, 47 of the 102 (46%) most difficult items had H⁴ as the scored key, compared to 1,100 of 11,836 (9%) items overall. Regardless, the difficulty flag seems less useful than the discrimination flag for identifying defective items.

Distractor Behavior

Distractors are the incorrect answer choices in multiple-choice questions. In the same way that we expect the probability of selecting the correct option to increase with capability, we expect the probability of selecting an incorrect option to decrease with capability. If, instead, the probability of selecting a particular distractor increases with capability, we have to wonder what’s going on. Perhaps the indicated correct answer is not actually correct, perhaps there’s more than one defensible answer, perhaps the question itself is ambiguous, or perhaps something else has gone wrong. This is why the SAT flags items for review when “a distractor has a correlation with the total score greater than 0.05.”

So, I’ll take a look at the items with the highest distractor-total score correlations. However, because I’m taking the highest distractor-total correlation for each item and I’m also looking at 11,832 items, some correlations may appear quite high purely by chance. So, I use p-values⁵ and adjust them for the false discovery rate using the Benjamini-Hochberg procedure.

So, lots of items were flagged: more than half. This is another indicator of potential problems. Items with concerning distractor behavior also tend to have concerning discriminations.

Let’s look at the worst offenders:

3 out of the 5 items we looked at would not pass review. So, the distractor flag also seems helpful for identifying defective or anomalous items.

Takeaways

As you can see, all of the above criteria are flags for further review. They’re not automatic exclusion criteria that mean an item should immediately be eliminated. It’s also the case that the discrimination and distractor flags seem much more informative than the difficulty flags, which makes sense: they more directly indicate that something may be wrong with the item itself.

This is also much faster than going through every item one by one or randomly selecting items to inspect. And it’s better than simply looking at the items that many models got wrong, since, as we saw, difficulty by itself isn’t nearly as informative as discrimination or distractor behavior.

Psychometric flags can’t automatically tell us which benchmarks are faulty. But they can tell us where to look. Instead of waiting until models appear to hit a mysterious performance ceiling and only then auditing hundreds or thousands of items, we can use the same kinds of statistical diagnostics that psychometricians already use to identify suspicious items for closer review. If we’re going to use benchmarks to measure increasingly capable AI systems, the current state of affairs is abysmal, and we need to start holding benchmarks to higher standards.

Factor Analysis Results

Parallel analysis suggested nine factors. Expand each factor below for its interpretation and loadings.

Ruggedness

This factor indicates a preference for roughness and danger. The items dealing with bodily preferences (‘Smooth Skin’ (negative loading), ‘Scars’, ‘Rough Skin’, ‘Sweat’) indicate a preference for a sort of ruggedness: a weathered, physically tough appearance rather than a smooth or polished one. The items dealing with personality (‘Dangerous’, ‘Cocky Attitude’, ‘Adventurous’) indicate the behavioral archetype of the “bad boy”: someone daring, cocky, and somewhat risk-seeking. Other items, such as ‘Guns’, ‘Motorcycles’, and ‘Clunky Old Cars’, fit into the same general aesthetic of roughness, danger, and disregard for polish. Taken together, the factor seems to capture a preference for a rough, dangerous form of masculinity, encompassing both rugged appearance and roguish behavior.

Factor Loadings

Trait	Loading
Scars	0.64
Smooth Skin	-0.63
Rough Skin	0.61
Dangerous	0.54
Sweat	0.49
Pretty Faces	-0.36
Clunky Old Cars	0.36
Motorcycles	0.35
Guns	0.34
Cocky Attitude	0.34
Adventurous	0.30
Rugged Looks	0.24

Companionability

This factor represents a preference for a warm, lively, enjoyable personality. This is obvious from the items that deal directly with personality, such as ‘Enthusiastic’, ‘Silly’, ‘Humorous’, and ‘Adventurous’, but we can learn more by looking at the other items. This is a relatively embodied personality, as evidenced by items such as ‘A Good Dancer’, ‘A Good Cook’, and ‘The Life of the Party’: someone who expresses their personality through doing things rather than merely possessing abstract personality traits. There’s also an emphasis on the man being “safe”, as you can see from items such as ‘Kind’, ‘Sympathetic’, and ‘Good with Children’, as well as the negative loading of ‘Unfaithful’. Taken together, this seems to describe someone energetic, fun, warm, and socially engaging.

Factor Loadings

Trait	Loading
Enthusiastic	0.54
A Good Dancer	0.53
Kind	0.52
Silly	0.45
Sympathetic	0.40
A Good Cook	0.40
The Life of the Party	0.38
Humorous	0.36
Good with Children	0.36
Adventurous	0.35
Unfaithful	-0.28

Affluence

Honestly, there’s not much to say here: this factor is obvious and only has a measly three items. It represents a preference for rich men. ‘Money’ and ‘Wealthy’ are obvious. ‘Fast Cars’ loads positively, but weakly, probably because fast cars are a visible symbol of wealth. Notably, this factor isn’t about a preference for high-status men per se. Education, intelligence, suits, dominance, and other potential indicators of status don’t load on this factor. It really does seem to be about money specifically.

Factor Loadings

Trait	Loading
Money	0.64
Wealthy	0.60
Fast Cars	0.29

Boyishness

This factor indicates a preference for youthful prettiness. The age component is obvious: ‘Young’ and ‘Teenagers’ load positively. There’s also a facial and bodily component: ‘Pretty in the Face’, ‘Pretty Faces’, and ‘Slender’ all load positively. Evidently, hair is not seen as youthful, as shown by the negative loadings of ‘Chest Hair’, ‘Facial Hair’, and ‘Receding Hairlines’.

Factor Loadings

Trait	Loading
Pretty in the Face	0.63
Chest Hair	-0.61
Receding Hairlines	-0.53
Facial Hair	-0.48
Pretty Faces	0.48
Young	0.46
Slender	0.44
Short Necks	-0.37
Teenagers	0.33

Intellect

This factor indicates a preference for intelligence in all its forms: education, brilliance, maturity, independent thought, and wit. This is obvious when we look at item pairs such as ‘Educated’ (positive loading) vs. ‘Uneducated’ (negative loading), or ‘Brilliant’ (positive loading) vs. ‘Not so bright’ (negative loading). Beyond book smarts, there’s also a focus on verbal intelligence, as indicated by items such as ‘Witty’ and ‘Simple Spoken’ (negative loading). Interestingly, ‘Humorous’ does not load on this factor, despite the loading of ‘Witty’, suggesting that the relevant distinction is verbal cleverness rather than simply being funny.

Factor Loadings

Trait	Loading
Uneducated	-0.71
Educated	0.71
Not so bright	-0.60
Brilliant	0.48
Witty	0.39
Free Thinking	0.39
Simple Spoken	-0.38
Mature	0.38

Pigmentation

This is a factor that I was not expecting, yet it’s fairly straightforward: it represents a preference for darker coloration over lighter coloration. ‘Dark Skin’, ‘Dark Eyes’, and ‘Black Hair’ all load positively, while ‘Fair Skin’, ‘Light Hair’, and ‘Light Eyes’ load negatively. In other words, all the items dealing with light/dark coloration are present and have exactly the loading signs you would expect. The only slight oddity is that ‘Black Hair’ loads considerably more weakly than the skin, hair-lightness, and eye-color items. I’m not sure why. Other than that, this seems to be an unusually clean preference for darker pigmentation across skin, hair, and eyes.

Factor Loadings

Trait	Loading
Fair Skin	-0.60
Dark Skin	0.53
Light Hair	-0.49
Dark Eyes	0.41
Light Eyes	-0.37
Black Hair	0.22

Adiposity

This is another straightforward factor: it represents a preference for fat men, as evidenced by the positive loadings of ‘Comfortably Overweight’ and ‘Heavyset’ and the negative loading of ‘Slender’. Importantly, items such as ‘Athletic’ and ‘Bulging Muscles’ don’t load on this factor, suggesting that this isn’t simply a preference for larger or more physically substantial men in general. Rather, it seems specifically to capture a preference for greater body fat and a heavier build.

Factor Loadings

Trait	Loading
Comfortably Overweight	0.58
Heavyset	0.58
Slender	-0.26

Bohemianism

This factor indicates a preference for an artistic, expressive, unconventional presentation, or a sort of bohemian type in that it is both artistic and socially unconventional. The artistic side of the factor can be seen in items such as ‘Artistic’ and ‘Musically Talented’ while ‘Long Hair’ fits naturally with the same alternative aesthetic. At the other end are items associated with more conventional forms of masculinity and presentation: ‘Men in Uniform’, ‘Jocks’, ‘Suits and Ties’, ‘Cologne’, and ‘Guns’ all load negatively. Taken together, the factor seems to contrast an artistic, expressive, alternative masculinity with a more conventional, institutional, and stereotypically masculine presentation.

Factor Loadings

Trait	Loading
Men in Uniform	-0.54
Long Hair	0.50
Guns	-0.47
Jocks	-0.45
Artistic	0.42
Musically Talented	0.33
Cologne	-0.32
Cocky Attitude	-0.32
Suits and Ties	-0.31

Virility

This factor indicates a preference for sexually dimorphic masculinity. Most of the items relate to the physical side of things: broad rather than narrow shoulders, broad rather than weak jaws, deep voices, greater musculature, tall rather than short stature, and larger rather than smaller genitals. The items relating to personality point in the same direction: ‘Dominant’ loads positively while ‘Shy’ loads negatively, suggesting a preference for a stereotypically masculine personality alongside the stereotypically masculine body.

Factor Loadings

Trait	Loading
Narrow Shoulders	-0.69
Broad Shoulders	0.63
Broad Jaws	0.61
Deep Voices	0.58
Weak Chins	-0.55
Short	-0.55
Bulging Muscles	0.53
Small Genitals	-0.49
Dominant	0.48
Shy	-0.47
Virile	0.46
Extremely Tall	0.39
Very Large Genitals	0.35
Rugged Looks	0.31

Comparison with Apple Pie

Apple Pie’s factors map most closely onto the factors found here as follows:

An(other) Aside About Acquiescence

At this point, I should mention acquiescence bias. Acquiescence bias is the tendency to agree with statements in a questionnaire regardless of what those statements actually assert. For example, in this dataset, there are traits that are natural opposites: Tall vs. Short, Fair skin vs. Dark skin, Young vs Old. A respondent high in acquiescence would say that they strongly prefer both tall men and short men, young men and old men, and so on. Here’s a visualization:

Now, it’s perfectly fine for someone to prefer many traits, even if they are ‘opposites’, but this doesn’t help us determine the factors, that is, the substantive sources of covariation among preferences. In fact, acquiescence will obscure those factors, because a general tendency to endorse items will make all indicators more positively correlated with one another, regardless of their content.

For this reason, I’ve modeled an acquiescence factor that affects all items equally, allowing this general endorsement tendency to be separated from the substantive preference factors. This seems to have been warranted, seeing as the common loading on the acquiescence factor was about 0.20 (90% posterior HDI: [0.18, 0.21]).

Factor Correlation Matrix

The correlations between preference factors make intuitive sense. Virility is a preference for a stereotypically masculine man, and so naturally has a negative correlation with Bohemianism, which is a preference for an unconventional man. Preferences for youthful prettiness (Boyishness) are negatively correlated with preferences for roughness and danger (Ruggedness), as well as with the aforementioned stereotypical manliness (Virility). The negative correlation between preferences for Affluence and Bohemianism makes sense if one keeps in mind the stereotype of the starving artist. As for positive correlations, the one between preferences for Ruggedness and Virility makes sense, since both are, in their own way, preferences for a particular kind of masculinity.

Takeaways

To be honest, I’m not sure what to put here. I think I’ve said everything that needs to be said. I have no further opinions. Thanks for reading.

Thanks to Apple Pie for sharing the data!

Appendix

Correlation Fishing

Let’s take a look at some of the correlations¹ between preferences and other traits. First, preferences and personality.

There isn’t much going on here, so let’s move on to the correlations between preferences and politics.

The political items used were:

• Hierarchy: It is important for society to have a hierarchy.
• Religion: Religion is very important in my life.
• Euthanasia: People suffering from incurable diseases should have the right to be put painlessly to death.
• Equality: We would have fewer problems if we treated people more equally.
• Free speech: Free speech is important, and should be protected even if some people’s feelings are hurt.
• Compulsory schooling: Teenagers should be legally required to go to school.

Responses to the hierarchy item were positively correlated with the Virility factor and negatively correlated with the Bohemianism factor (as well as with the Companionability and Pigmentation factors). It seems like women who think societal hierarchies are important have stronger preferences for conventionally masculine men, along with stronger preferences for fairer-skinned men.

Responses to the equality item were positively correlated with the Bohemianism factor (and the Pigmentation factor) and negatively correlated with the Affluence factor. All in all, these correlations seem to be broadly the reverse of the pattern for the hierarchy item.

Other correlations were weaker, though one possibly interesting finding is the positive correlation between endorsing euthanasia and preference for Intellect.

Finally, social desirability:

It seems that the most socially desirable preference is for a man who would make a good lead in a romantic comedy?

Factor Model of Personality

I modelled the HEXACO items as resulting from 8 orthogonal factors: the 6 HEXACO factors, an acquiescence factor that all items loaded on equally, and a social desirability factor.

HEXACO item loadings

Honesty-Humility

• Honest, Honorable: 0.20 (90% HDI: [0.09, 0.29])
• Amoral, Carefree: -0.31 (90% HDI: [-0.42, -0.20])

Emotionality

• Sentimental, Soft: 0.42 (90% HDI: [0.33, 0.51])
• Rugged, Unemotional: -0.44 (90% HDI: [-0.53, -0.34])

Extraversion

• Active, Talkative: 0.62 (90% HDI: [0.55, 0.69])
• Introverted, Withdrawn: -0.63 (90% HDI: [-0.70, -0.56])

Agreeableness

• Easygoing, Calm: 0.49 (90% HDI: [0.39, 0.58])
• Tense, Hot-Tempered: -0.49 (90% HDI: [-0.59, -0.40])

Conscientiousness

• Organized, Thorough: 0.40 (90% HDI: [0.30, 0.49])
• Disorganized, Careless: -0.37 (90% HDI: [-0.46, -0.27])

Openness

• Interested in Art, Deep: 0.44 (90% HDI: [0.34, 0.54])
• Down-to-Earth, Unimaginative: -0.44 (90% HDI: [-0.56, -0.35])

Acquiescence: one shared loading of 0.035 (90% HDI: [0.013, 0.059]).

Social desirability item loadings

• Honest, Honorable: 0.55 (90% HDI: [0.47, 0.64])
• Disorganized, Careless: -0.48 (90% HDI: [-0.57, -0.39])
• Organized, Thorough: 0.43 (90% HDI: [0.34, 0.52])
• Active, Talkative: 0.20 (90% HDI: [0.11, 0.28])
• Amoral, Carefree: -0.19 (90% HDI: [-0.31, -0.08])
• Introverted, Withdrawn: -0.16 (90% HDI: [-0.24, -0.07])
• Sentimental, Soft: 0.16 (90% HDI: [0.05, 0.25])
• Easygoing, Calm: 0.15 (90% HDI: [0.05, 0.25])
• Tense, Hot-Tempered: -0.12 (90% HDI: [-0.22, -0.03])
• Interested in Art, Deep: 0.11 (90% HDI: [0.03, 0.21])
• Rugged, Unemotional: -0.08 (90% HDI: [-0.16, -0.01])
• Down-to-Earth, Unimaginative: -0.06 (90% HDI: [-0.14, -0.002])

The loadings on the HEXACO factors are moderate, though some, such as Honesty-Humility, are lower than I’d like. Overall, though, the model looks decent, and there doesn’t seem to be much acquiescence occurring.

Factor Model of Politics

Apple Pie chose the six politics items to represent the three factors of political views that they found in their earlier research: Conservatism, Tough-mindedness, and Libertarianism. Apple Pie says the factors are orthogonal, but I thought I’d test that assumption anyway by allowing the factors to correlate. I also included an orthogonal acquiescence factor.

Political item loadings

Conservatism

• It is important for society to have a hierarchy: 0.28 (90% HDI: [0.17, 0.39])
• Religion is very important in my life: 0.29 (90% HDI: [0.18, 0.40])

Tough-mindedness

• People suffering from incurable diseases should have the right to be put painlessly to death: 0.11 (90% HDI: [0.03, 0.21])
• We would have fewer problems if we treated people more equally: -0.11 (90% HDI: [-0.20, -0.01])

Libertarianism

• Free speech is important, and should be protected even if some people's feelings are hurt: 0.23 (90% HDI: [0.12, 0.36])
• Teenagers should be legally required to go to school: -0.21 (90% HDI: [-0.31, -0.10])

Acquiescence: one shared loading of 0.053 (90% HDI: [0.015, 0.089]).

Factor correlations

Conservatism / Tough-mindedness

0.001 (90% HDI: [-0.123, 0.130])

Conservatism / Libertarianism

0.015 (90% HDI: [-0.101, 0.132])

Tough-mindedness / Libertarianism

0.015 (90% HDI: [-0.100, 0.130])

Unfortunately, it’s not a good model: the factor loadings are very weak. Perhaps this is because there are substantial cross-loadings; for example, we might expect the euthanasia item to also load on Libertarianism. I’d also expect the hierarchy and equality items to be more negatively correlated than would be implied by the factor correlations and loadings alone. Regardless, modeling the factors seems like a bad idea here, so I look at correlations using the raw items instead.

“You called yourself luck,” Akua said, “but that is a lie, Intercessor. You are not a blind roll of the dice. You take sides.”

“I’ve helped both sides of the Game,” the Intercessor dismissed, “I-“

“You help Good,” Akua said. “When you have the choice, that is the truth of you.”

— ErraticErrata, A Practical Guide to Evil (Book 7: Chapter 68: Hallow; Hollow)

Now, the title of this post shouldn’t be a surprise to anyone, but ever since the Trump administration began promoting “Americanism, not Effective Altruism” and thrust EA even further into the political spotlight, I’ve seen EA-aligned people claiming that EA isn’t inherently left-wing or right-wing. I’ll get into the “inherently” part later, but right now, I want to argue that, in practice, EA is pretty obviously left-wing¹.

Political Leanings of Effective Altruists

An obvious place to look is the political leanings of effective altruists themselves. Looking at the (semi-)yearly EA Surveys, it has never been the case that more than 5% of effective altruists identify as either right or center-right. Meanwhile, the proportion of effective altruists identifying as left, excluding for the moment center-left, varies between 25% and 40%. Including the center-left brings that figure to between 70% and 85%. So left-leaning effective altruists outnumber right-leaning effective altruists by, at minimum, about 14 to 1. That’s very lopsided².

Political Leanings of EA Cause Areas

What about the cause areas EA prioritizes? What partisan leanings do those issues have? The major causes EAs seem to care about are AI risk, global poverty and health, biosecurity, and animal welfare. Looking at Rethink Priorities’ Pulse Wave 2, Democrats give higher importance ratings than Republicans to every cause area surveyed, and also show higher support for donating to every cause area. The largest gaps are for climate change, civil rights, global health and development (GHD), and pandemic preparedness³. The gaps are smallest for AI risk, cancer research, and nuclear weapons.

The partisan leaning of AI risk is currently… fluid, so I’ll set it aside for now.

Looking at other sources, a 2025 Pew Research poll shows a 14-point partisan gap on whether the US should provide foreign aid in the form of medicine and medical supplies to developing countries, and a 34-point gap on foreign aid aimed at supporting economic development in developing countries. In a 2018 Pew poll, 32% of Democrats, compared to 12% of Republicans, said “helping improve living standards in developing nations” should be a top foreign-policy priority.

On pandemic preparedness, a 2025 YouGov poll found that 71% of Democrats thought the US needed to do more to prepare for future pandemics, compared to 31% of Republicans. In a 2024 Pew Research poll, 63% of Democrats versus 41% of Republicans said “reducing the spread of infectious diseases” should be a top priority in US long-range foreign policy.

A 2025 Faunalytics survey found that Democrats, on average, rated standard animal-agriculture practices as more unacceptable than Republicans did. A 2023 ASPCA/Ipsos national survey found that Democrats consistently showed higher support for humane-treatment policies.

So, all in all, looking at the partisan skew of major EA cause areas, Democrats consistently show more support for them.

Political Donations and Candidacies

Another way to get at the political leanings of a group is to look at where its political donations end up. Looking at some of the most prominent EA-associated donors:

• In 2020, Dustin Moskovitz and Cari Tuna donated ~\$50 million to Democratic candidates and liberal groups, and \$0 to Republican candidates and conservative groups.
• In 2024, Dustin Moskovitz donated ~\$50 million to Democratic candidates and liberal groups, and \$0 to Republican candidates and conservative groups.
• In 2022, Protect Our Future, an SBF-funded PAC organized around pandemic preparedness, spent \$21 million in independent expenditures, exclusively in Democratic House primaries.
• In 2022, SBF publicly disclosed ~\$37 million in contributions to Democratic candidates and only $320,000 in contributions to Republicans. However, he also later said that he had secretly been the source of approximately \$37 million, and potentially as much as \$80 million, in contributions to Republicans.

What about actual EAs running for office? In 2022, Carrick Flynn, whom The New Yorker described as the first explicitly EA-affiliated congressional candidate, ran as a Democrat.

Admittedly, this evidence is weaker, since I’ve focused on a handful of major donors and the one explicitly EA-affiliated congressional candidate. But, given everything else we’ve seen, it is at least suggestive of the same overall pattern.

Is EA Inherently Left-Wing?

So, all in all, it seems to me that EA is, in practice, very left-leaning. But is it inherently left-wing?

I’m not a fan of theories that try to reduce the two American political sides to a single simple principle⁴. They seem to overlook the fact that, in the American system, there aren’t two inherent political sides. There are two major parties, for historical and structural reasons, but those parties are better thought of as coalitions of various groups and interests. Additionally, the partisan alignment of particular issues can vary for somewhat contingent reasons⁵.

So I wouldn’t say EA is inherently anything. But, given the political environment we actually live in, it does seem likely that EA was always going to end up left-leaning. Focusing heavily on the well-being of foreigners and animals is already politically left-leaning in the contemporary US. Pandemic preparedness seems like a more unfortunate case of contingent polarization.

None of this is meant as a criticism of EA. It doesn’t seem to me that EA has been deliberately trying to become overtly partisan. The partisan leanings of its cause areas and of the people who make up the movement are what they are. What, if anything, can be done about that? I don’t know.

But I do think it’s important not to deceive ourselves about the situation, especially as EA is thrust further into the political spotlight. If EA looks overwhelmingly left-leaning from the outside, that matters in an environment where people are increasingly suspicious that institutions associated with the political out-group are really vehicles for the out-group’s interests. Whatever EA does in response, it seems better to start from an accurate picture of how the movement is actually positioned.

To understand mogging, we must first grasp that it is a transfer of status. That is, most forms of mogging attempt to raise or lower the status of individuals via game-like structures, with defined roles and a structurally predictable script. There is always a mogger, a moggee, and, crucially, an audience (which can coincide with the moggee, by design or accident). The moggee may or may not be present. So, there are at least three roles in a mog, of which the role of audience may be played by a group. This gives us three basic forms of mogging.

Autist (Two-Person) Mogging

Two-person mogging is Autist mogging. If you attempt a mog with just one other person present, and you are only capable of gaining status if the other person loses status (as is necessary, since status is zero-sum), you get a terminally stupid situation that only Autists will attempt to enact. Note that two-person moggings with an absent moggee are really three-role moggings, so they don’t count.

In a two-person situation, you either get non-adversarial self-mogs (which reinforce existing status), or an adversarial mog. Since social proof works by majority vote, two-person adversarial mogs cannot work unless the moggee laughs at himself, accepting a loss of status. If the moggee fights back, with no neutral audience to determine status changes, you get a pointless game of one-upmanship. This is why the mog is autist. It boils down to he-said-she-said. Without an audience, there is no mogging, and there is no meaningful transfer of AMOG status.

Sociopath (One-Person) Mogging

One-person mogging is Sociopath mogging, and is psychologically more complex. It can only happen when the mogger and audience are the same person (which replaces social proof with individual judgment), and everybody else present is a moggee, often unaware that they are being mogged. Sociopath mogging often possesses push-button cruelty, in the sense of a cat-and-mouse “pushing buttons” exercise in viciousness for private pleasure, and is “objective” mogging in that sense, since the moggee’s loss of status is reflective of an “objective” fact about the world.

But mogging gets really interesting when there are more than two active participants. That gets you to Normie mogging, an engine of social status transfers.

Normie (Group) Mogging

In Normie mogging, as with Autist mogging, the status transfer comes via the judgment of somebody other than the mogger (audience != mogger != moggee). But unlike the two-person stalemate that is the norm in Autist mogging, Normie mogging usually creates clear outcomes because democratic social proof can work. The smallest meaningful Normie group is three people (including some special cases where the moggee is absent, and both mogger and audience accept the proposed status transfer, providing a 2/3 social-proof majority).

In general, the transfer of status depends entirely on the reactions of the audience. What breaks the status stalemate in groups of three is that meaningful status movements can occur. Alphas can become Betas, and Betas can become Alphas. Due to additive effects, bigger groups are even better: if three people, who collectively are all Betas, are present, then we can add up all the micro-shifts in status and get a macro-shift in status.

How does mogging create status transfers? Consider a simple three-person situation. A attempts a mog of B. Without C present, you’d get Autist dynamics. But if C is present and deems the mog successful, he bonds with the mogger and transfers status to A, in the sense that B (and C, to an extent) are now envious of A. If he frowns or otherwise indicates that the mog was in bad taste, he bonds with the moggee and transfers status in the other direction (A is now envious of B). And crucially, if he does not react, no status is transferred.

Laugh/frown votes are a powerful weapon for the passive members of any situational group. In the most extreme situation – the smallest possible group of three people – there is enormous power wielded by just one person.

Apologies to Venkat

However, they appear to have chosen five factors a priori, based on their review of the literature, rather than using the data to determine how many factors to retain. So, I decided to replicate their analysis using parallel analysis to determine the number of factors empirically.

I also broadened the analysis in two ways. Instead of restricting the sample to 60 MSAs, I used every contemporary metropolitan area with at least 1,000 members of the focal racial group and at least 1,000 non-Hispanic White-alone residents¹. And instead of examining only 1980, I repeated the analysis for 1990, 2000, 2010, and 2020 to see how stable the findings are over four decades. The national-sample result is remarkably consistent: parallel analysis retains three factors in every census year. The fixed panel corresponding to the original 60 metropolitan identities produces a more conservative result, as discussed below.

Massey and Denton’s five dimensions describe the different ways in which residential segregation might occur:

• Evenness: How evenly distributed is a group across residential areas?
• Exposure: How much potential contact do members of one group have with members of another group?
• Concentration: How much physical space does a group occupy?
• Centralization: How concentrated is a group in the city center?
• Clustering: To what degree does a group disproportionately live in contiguous areas?

How many factors are there?

The national sample consistently yields three factors, not five.

I’ve shown the findings for when the local-environment radius for the spatial measures is set at 2km, but the same result happens when we use 0.5, 1, or 4 km (see the Appendix).

Does the result depend on the sample?

As a robustness check, I repeated the analysis using a fixed panel corresponding to Massey and Denton’s original 60 metropolitan identities, following the same metro names while using each census year’s contemporary boundaries. The number of retained factors is the same at every spatial radius within a given sample and year:

The fixed panel has a much smaller weighted effective sample size (between 69 and 92 observations, compared with 230 to 472 in the national sample) so parallel analysis is more conservative. Still, we can see that the data does not support five factors.

What do the factors measure?

The three factors are Evenness, Isolation, and Concentration. Expand each factor below for its interpretation and loadings.

Why are there fewer than five factors?

So, are there really only three dimensions of segregation? The national sample says three, while the fixed panel generally says two. But neither supports five. I’d argue that there are three relevant dimensions, even if the smaller fixed panel does not always distinguish all three empirically.

A problem that immediately jumped out at me when reading about the five dimensions was that the Centralization dimension makes an assumption not about how a group is distributed, but about where the group is distributed. This assumption makes me uneasy. I don’t like my mathematical measures to bake in such messy, empirical assumptions. Now, it does make sense that, given the fact that minority groups tend to be located near the center of urban areas, we’d expect centralization to be correlated with concentration. But I’d prefer to use concentration measures and then, once we determine that certain groups tend to end up concentrated, ask where those groups are concentrated. If that turns out to be the city center, then great.

What about the two dimensions that don’t show up in my factor analysis, Exposure and Clustering? Well, conceptually, clustering seems related to evenness: if a group is all clustered together, then it isn’t evenly distributed. As for exposure, that seems related to isolation. If a group is isolated, that is, its members are particularly likely to live near one another, then they’re not very exposed to members of other groups.

So, it doesn’t actually seem like we needed five dimensions in the first place.

Which measures should we use?

Which measures should be used for each factor? For the Evenness factor, there are seven indices with consistently near-perfect loadings, so really, we could choose any of them and we’d be fine. But the dissimilarity index ($D$) seems to be the most popular, so we’ll go with that.

For the Isolation factor, the three highest-loading indices are, well, isolation indices. Of these three, distance-decay isolation and spatial isolation seem the most conceptually appropriate: we should expect neighboring tracts to matter when determining isolation. Of these, distance-decay isolation consistently loads higher, so let’s go with that.

For the Concentration factor, I’d prefer to avoid any measures that explicitly refer to or rely on the idea of a city center, which leaves delta ($\text{DEL}$) as our best option.

So, if you want to measure segregation, I would distinguish three dimensions: evenness, isolation, and concentration. If you’re interested in evenness, use the dissimilarity index. If you’re interested in isolation, use distance-decay isolation. If you’re interested in concentration, use the delta index.

Appendix

Measures of Segregation

The analysis uses 22 nonredundant indices: 18 from the original Massey-Denton battery and four later extensions. Expand any measure below for its interpretation and formula.

Population and area notation:

• $x_i$ and $y_i$ are the focal-group and non-Hispanic-White populations of tract $i$.
• $t_i=x_i+y_i$ is the tract’s two-group population.
• $X$, $Y$, and $T$ are the corresponding metropolitan totals.
• $p_i=x_i/t_i$ is the focal-group share of tract $i$, and $P=X/T$ is the focal-group share of the metropolitan population.
• $a_i$ is tract land area (in square miles).

For the original proximity measures:

• $d_{ij}$ is the distance in miles between the representative points of tracts $i$ and $j$.
• $z_{ij}=e^{-d_{ij}}$ is the distance-decay weight.
• Within a tract, self-distance is approximated as $d_{ii}=\sqrt{0.6\,a_i}$. This is to avoid treating every resident of a tract as if they’re in the exact same place.

For the added local-environment measures:

• $r_{ij}$ is the distance in kilometers between tracts $i$ and $j$.
• $h$ is the local-environment radius.

Nearby tracts are weighted using the biweight kernel:

The resulting focal-group share of tract $i$’s local environment is:

Indices of Segregation

Parallel Analysis for Other Distances

Factor Loadings at Other Distances

The factors are matched and sign-aligned to each radius’s 1980 solution. As in the primary 2-km analysis, each table includes only indices whose mean absolute loading on that factor across the five censuses is at least .40, ordered by that mean.

0.5-km radius

1-km radius

4-km radius

AI’s moving pretty fast these days. How fast? Really fast. You want a quantitative answer? That’s what benchmarks are for. Just have the AI models answer questions and complete tasks, then score the results. You can use the scores to compare models and chart AI progress. But benchmarks have a problem. They saturate too fast. AI progress is so fast that benchmarks can become useless within a couple of years, sometimes much less.

So, you might think to solve this problem by ‘linking’ benchmarks together. If you know how much more difficult one benchmark is than another, you can sort of combine them into a mega-benchmark that works even as model capabilities saturate easier benchmarks. This is how Epoch’s ECI works.

It borrows from item response theory, specifically the 2PL model. The 2PL model is so called because it uses the logistic function and two parameters, the benchmark discrimination ($\alpha_b$) and difficulty ($D_b$), in the following way (for Epoch’s ECI):

where:

where:

• $\mu_{mb}$ is the predicted performance of a model, $m$, on a benchmark $b$.
• $\alpha_b$ is the benchmark’s discrimination. It tells us how good the benchmark is at ‘discriminating’ between high- and low-capability models. On a benchmark with high discrimination, small differences in capability produce large differences in performance, while on a benchmark with low discrimination, performance changes more gradually with capability.
• $D_b$ is the benchmark’s difficulty. It tells us, well, how difficult the benchmark is.
• $C_m$ is the model’s capability. Models with high capabilities will be able to do well even on difficult benchmarks, while those with low capabilities will struggle with even easy benchmarks.

The ECI assumes these parameters are constant across time, but that’s not a safe assumption to make. Whenever we’re comparing a latent construct across groups, we need to ensure that our measurement instrument is measuring the same construct in the same way. In psychometrics, this is called measurement invariance. Otherwise, we could have biased items that make one group artificially score higher than another, giving us misleading results.

For example, vocabulary is known to be very g-loaded, and so one might decide to use a vocabulary test as a proxy for cognitive ability, like WORDSUM in the General Social Survey. However, there are words that men are more likely to know than women and vice versa¹.

If we have a test that consists of many of these words, it’ll be biased: one gender will have a higher chance of getting items correct than the other, even holding cognitive ability constant.

Going back to AI models, we might decide to group models by when they were released. In this case, for example, a benchmark might become popular, so much so that labs start explicitly optimizing for performance on it. We might then see models’ performance on that benchmark increase very rapidly, much faster than their general capability. In that case, making use of the benchmark without accounting for this shift would cause us to overestimate AI progress. And so, we must check whether the parameters are actually static across time.

For this analysis, I use the public ECI data downloaded on September 15, 2026, containing 2,745 reported scores from 264 models across 58 benchmarks. I measure time using each model’s release month. Thus, a change over time means that models released in different months have different expected performance on a benchmark after accounting for their estimated general capability.

First, what happens when we allow the benchmark discriminations to vary depending on model release date?

There are two benchmarks that display a unique, peculiar U-shape: VPCT and GSM8K. I’m not quite sure why this is. Other than those two, most benchmarks have either relatively static discriminations or decreasing discriminations over time, the most prominent of which are ARC-AGI-2, DeepSWE, GeoBench, and GDPval. There aren’t actually any benchmarks I can confidently say have had increasing discriminations, rather than the weird U-shape. This might be because model capabilities eventually progress past the point at which benchmarks have peak discriminative ability and into a region where they become increasingly poor at discriminating between models, because they don’t have enough items of the requisite difficulty.

What happens when we allow a benchmark’s difficulty to vary depending on model release date?

Benchmarks with increasing difficulty include Winogrande, Fiction.LiveBench, DeepResearch Bench, and ARC AI2, which seem to share a focus on language tasks. Benchmarks with decreasing difficulty include DeepSWE, GSM8K, MATH Level 5, and OSWorld. DeepSWE is the benchmark that’s exhibited the biggest decrease in difficulty, so much so that it’s an outlier. It’s also focused on long-horizon software engineering, which seems to be a popular focus of labs as of late, a fact which I don’t think is a coincidence. The other fast-decreasing benchmarks are focused on math and software ability, which have also been a focus of labs recently.

Now, I don’t think it makes sense to think of benchmarks as literally getting more or less difficult. Ideally, they should be the same difficulty across time. So I think it’s best to interpret increasing benchmark “difficulty” as models improving on that benchmark more slowly than they’re improving in general capability, and decreasing benchmark “difficulty” as models improving on that benchmark faster than they’re improving in general capability. Under that interpretation, the pattern makes sense. We see models increasing their math and coding capabilities faster than their general ability, while improving at language tasks more slowly than their general ability. This makes sense. Labs are focusing heavily on math and coding through post-training regimes such as RLVR, while comparatively less attention is being paid to language tasks.

Is there a correlation between average discrimination changes and average difficulty changes?

If we ignore the outliers that are DeepSWE and VPCT, our mean estimate is that the correlation is exactly… 0. So, it doesn’t seem like there’s anything interesting there.

So, should we throw out the ECI because it uses static parameters and try something new? No. It turns out that predicted model capabilities taking changing parameters into account correlate at 0.995 with Epoch’s predicted model capabilities, making them nearly identical. The same holds for benchmark difficulties, which correlate at 0.98, though at the extremes my estimates suggest that the hardest benchmarks aren’t quite as hard as Epoch estimates and the easiest benchmarks aren’t quite as easy. Benchmark discriminations are less correlated, at only 0.85, but there’s no discernible systematic pattern to the differences.

Still, it does mean we need to be careful when treating AI capability as a unitary construct, as the ECI does. Which capabilities are most relevant changes over time, partly because labs shift their focus. It used to be reading and understanding language, but as models became proficient at that, attention shifted toward coding and mathematics. A chart that represents AI progress with a single capability score is therefore somewhat misleading: what counts as capability changes over time. Progress in currently relevant capabilities, such as mathematics and coding, may be underestimated by combining them with benchmarks that emphasize domains in which progress is now slower, such as general language understanding. Those benchmarks pull the construct the ECI is measuring towards slower-moving domains.

In conclusion, the ECI remains a (very) useful summary of average benchmark performance, but it should not be mistaken for a measure of a single, unchanging construct. We should take care to think about which capabilities we think are useful to measure.

Appendix

The Real Treasure Was The Models We Made Along The Way

I didn’t begin with the final two-stage model. I started by reproducing Epoch’s estimator as literally as possible, then changed one assumption at a time, to ensure I wasn’t making any silly mistakes. Expand the steps below to follow the progression from the public ECI implementation to the two-stage model used in this post. (You could always just skip to the final model, but I think it’s easier to go step-by-step.)

The code, frozen data, and compact results for all six models are available in the accompanying GitHub repository.

1. Bayesianizing the ECI

The core of Epoch’s model is, as described above, the logistic function (source) (line 242 of the ECI implementation):

\[\mu_{mb} = \sigma(\alpha_b[C_m-D_b])\]

where:

\[\sigma(x) = \frac{1}{1+e^{-x}}\]

Free parameter vector

There are $M$ models and $B$ benchmarks. Epoch’s implementation estimates:

\[\phi = (C_1, \dots, C_M,D_1,\dots,D_B,\alpha_1,\dots,\alpha_{B-1})\]

There is one benchmark discrimination excluded from the free parameter vector: Winogrande’s discrimination, which is fixed at 1. The number of free parameters is therefore:

\[K = M + 2B - 1\]

The raw parameter bounds are (lines 257-266 of the ECI implementation):

\[-10 \leq C_m \leq 10\] \[-10 \leq D_b \leq 10\] \[0.1 \leq \alpha_b \leq 10\]

Observed scores are clipped to $[0.001,0.999]$ before fitting (lines 196-197 of the ECI implementation).

Implemented residual vector

For every observed score, the code (line 243 of the ECI implementation) supplies SciPy with its residual:

\[r_{mb}(\phi) = \mu_{mb}(\phi) - s_{mb}\]

It also appends a residual used for regularization (lines 245-246 of the ECI implementation):

\[r_\text{reg}(\phi) = \sqrt{\lambda \frac{1}{K} \sum_{k=1}^{K}\phi_k^2}\]

where the default regularization strength, $\lambda$ is set to 0.1 (line 138 of the ECI implementation).

Scipy’s least_squares minimizes one-half of the sum of squared residuals, so Epoch’s actual implemented objective is:

\[\begin{aligned} \mathcal{L}(\phi) &= \frac{1}{2}\left(\left(\sum_{(m,b) \in \mathcal{O}} r_{mb}(\phi)^2\right) + r_\text{reg}(\phi)^2\right) \\ &= \frac{1}{2}\left(\left(\sum_{(m,b) \in \mathcal{O}} [\mu_{mb}(\phi) - s_{mb}]^2\right) + \lambda\frac{1}{K}\sum_{k=1}^{K}\phi_k^2\right) \\ &= \frac{1}{2}\sum_{(m,b) \in \mathcal{O}}[\mu_{mb}(\phi) - s_{mb}]^2 + \frac{1}{2}\frac{\lambda}{K}\sum_{k=1}^{K}\phi_k^2 \end{aligned}\]

Public scale

After fitting, Epoch converts the raw capabilities to the official ECI scale using an affine transformation while setting Claude 3.5 Sonnet at 130 and GPT-5 at 150.

Exact MAP-equivalent model

A model’s performance on a benchmark is modeled as:

\[s_{mb}\mid\phi \sim \mathcal N\left(\mu_{mb}(\phi),\sigma_\varepsilon^2\right).\]

where $\sigma_\varepsilon$ is a fixed residual standard deviation.

Ignoring constants, the negative log-likelihood is:

\[-\log p(s\mid\phi) = \frac{1}{2\sigma_\varepsilon^2} \sum_{(m,b)\in\mathcal O} \left[s_{mb}-\mu_{mb}(\phi)\right]^2.\]

We place independent Gaussian priors with a common standard deviation $\tau$ on the free parameters, subject to the same bounds used by Epoch:

\[C_m\sim\operatorname{TruncatedNormal}(0,\tau^2;-10,10),\] \[D_b\sim\operatorname{TruncatedNormal}(0,\tau^2;-10,10),\]

and:

\[\alpha_b \sim \operatorname{TruncatedNormal}(0,\tau^2;0.1,10)\]

for each non-Winogrande benchmark. The Winogrande slope remains fixed:

\[\alpha_{\text{Winogrande}}=1.\]

Away from the bounds, the negative log-prior is:

\[-\log p(\phi) = \frac{1}{2\tau^2} \sum_{k=1}^{K}\phi_k^2 +\text{constant}\]

So, the negative log-posterior is

\[-\log p(s\mid\phi) - \log p(\phi) = \frac{1}{2\sigma_\varepsilon^2} \sum_{(m,b)\in\mathcal O} \left[s_{mb}-\mu_{mb}(\phi)\right]^2 + \frac{1}{2\tau^2} \sum_{k=1}^{K}\phi_k^2\]

If we multiply the negative log-posterior by $\sigma_\epsilon^2$, we get:

\[\frac{1}{2} \sum_{(m,b)\in\mathcal O} \left[s_{mb}-\mu_{mb}(\phi)\right]^2 + \frac{1}{2} \frac{\sigma_\varepsilon^2}{\tau^2} \sum_{k=1}^{K}\phi_k^2\]

This matches Epoch’s objective when:

\[\boxed{ \frac{\lambda}{K} = \frac{\sigma_\varepsilon^2}{\tau^2} }\]

or equivalently:

\[\boxed{ \tau = \sigma_\varepsilon\sqrt{\frac{K}{\lambda}} }.\]

It’s important to note that there’s no unique pair $(\sigma_\epsilon, \tau)$ implied by the objective. Only the ratio is implied. We could set $\sigma_\epsilon = 1$, but, since the scores lie in $[0.001, 0.999]$, a residual standard deviation of 1 would be absurdly large on the observed scale.

What’s more, this also affects $\tau$. For example, if $K = 379$ and $\lambda = 0.1$, then setting $\epsilon_e = 1$ implies

\[\tau = \sqrt{379/0.1} \approx 61.6\]

This is nearly flat over the capability and difficulty bounds $[-10, 10]$ and the discrimination bounds $[0.1, 10]$. As such, the penalty isn’t really doing much.

Unfortunately, I first ended up doing taking the ‘convenient’ approach of setting $\sigma_\epsilon = 1$, and so while the MAP estimates match exactly, the posterior means are all over the place.

2. Learning the Residual Scale

The first model set $\sigma_\varepsilon$ to 1 because that was convenient. However, a residual standard deviation of 1 is absurdly large for scores bounded between 0 and 1 (well, technically between 0.001 and 0.999). Given that only the ratio $\frac{\sigma_\varepsilon^2}{\tau^2}=\frac{\lambda}{K}$ matters, it might be better to let the model fit the residual standard deviation instead.

So, the second model I fit assigns

\[\log\sigma_\varepsilon\sim\mathcal N(\log 0.1,1)\]

and deterministically sets

\[\tau=\sigma_\varepsilon\sqrt{\frac{K}{\lambda}}.\]

For any value of $\sigma_\varepsilon$, the model has the same variance-to-penalty ratio as Epoch, but now it can learn the appropriate residual variance from the data. Everything else remains unchanged: Winogrande’s discrimination is fixed at 1, capabilities and difficulties remain bounded to $[-10,10]$, and the other discriminations remain bounded to $[0.1,10]$. As you can see, the MAP estimates still match Epoch’s exactly, while the posterior mean estimates are much more reasonable, though there’s a bend in the curve starting below an ECI of ~120 such that the Bayesian estimates are higher than Epoch’s estimates.

3. Unfixing Winogrande

The next model estimates Winogrande’s discrimination along with every other benchmark discrimination. There are now

\[K=M+2B\]

free ECI parameters, and every discrimination receives the same bounded Gaussian prior:

\[\alpha_b\sim\operatorname{TruncatedNormal}(0,\tau^2;0.1,10).\]

Removing the Winogrande anchor creates a scale identifiability issue. For any $a>0$,

\[C_m'=aC_m,\qquad D_b'=aD_b,\qquad \alpha_b'=\frac{\alpha_b}{a}\]

produces exactly the same predictions. The likelihood is also unchanged when the same constant is added to every capability and difficulty. The bounded proper priors make the posterior proper and select a particular origin and unit, but I still don’t like the model.

Nevertheless, it’s still a useful intermediate model. If we’re to model changes in discriminations, we can’t also fix the discrimination of one of the benchmarks. The model produces MAP estimates, that no longer exactly match those of Epoch, but are still extremely close. The posterior mean estimates have also gotten closer to Epoch’s estimates.

4. Replacing the Anchor and Bounds with Symmetric Identification

Our next model removes the hard bounds and resolves the identifiability issue without privileging a particular model or benchmark.

Capabilities and difficulties are combined into one vector and assigned a zero-sum Gaussian prior:

\[(C_1,\ldots,C_M,D_1,\ldots,D_B) \sim\operatorname{ZeroSumNormal}(\tau),\]

which enforces

\[\sum_m C_m+\sum_bD_b=0.\]

This identifies the origin of the latent scale. Discriminations are modeled on a logarithmic scale:

\[s_\alpha\sim\operatorname{HalfNormal}(0.5),\] \[(\log\alpha_1,\ldots,\log\alpha_B) \sim\operatorname{ZeroSumNormal}(s_\alpha).\]

Therefore,

\[\sum_b\log\alpha_b=0\]

and the geometric mean of the discriminations is one:

\[\left(\prod_b\alpha_b\right)^{1/B}=1.\]

This identifies the multiplicative scale. This static model becomes the foundation for the parameter-varying models. Also, our posterior mean estimates are finally lining up with Epoch’s estimates, which is nice.

5. Allowing Discriminations to Change Over Time

The first parameter-varying model keeps capabilities and difficulties static but allows benchmark discriminations to depend on model-release month. Let $t_{0b}$ be benchmark $b$’s earliest observed month. Its initial log discrimination receives the prior described above:

\[s_\alpha\sim\operatorname{HalfNormal}(0.5),\] \[(a_{1,0},\ldots,a_{B,0}) \sim\operatorname{ZeroSumNormal}(s_\alpha).\]

Temporal change follows a stationary RBF Gaussian process. All benchmarks share the same length scale,

\[\ell_\alpha\sim\operatorname{Uniform}(2,36),\]

which is measured in months. The covariance function is given by

\[K_{\alpha,tt'} = \exp\left[-\frac{(t-t')^2}{2\ell_\alpha^2}\right]+10^{-4}I.\]

Each benchmark has its own Gaussian process trajectory $f_b$:

\[f_b\sim\mathcal N(\mathbf 0,K_\alpha),\] \[\kappa_b\sim\operatorname{HalfNormal}(0.5).\]

A benchmark’s monthly discrimination is given by

\[\log\alpha_{b,t} = a_{b,0}+\kappa_b\left(f_{b,t}-f_{b,t_{0b}}\right).\]

The likelihood for observed scores is

\[s_{mb} \sim \mathcal N\left( \sigma\left[\alpha_{b,t_m}(C_m-D_b)\right], \sigma_\varepsilon^2 \right).\]

When a single discrimination is needed for a model (such as in the following graphs), I use the geometric mean across a benchmark’s supported months. This is the first stage of the final analysis.

6. The Final Two-Stage Model

The analysis in this post uses a two-stage model. Stage 1 was the model described I just described above. For Stage 2, rather than passing only Stage 1’s posterior means, I use discrimination draws

\[\boldsymbol\alpha^{(q)} = \{\alpha^{(q)}_{b,t}:b=1,\ldots,B;\ t=1,\ldots,T\}.\]

This allows us to use and model the posterior distribution of Stage 1’s results, rather than treating the posterior mean as the only possible set of discriminations.

Each draw is normalized such that its geometric mean of the discriminations is equal to 1. If $\mathcal T_b$ is benchmark $b$’s supported calendar range, define the average log discrimination, $g^{(q)}$ as

\[g^{(q)} = \frac{1}{B} \sum_b \left[ \frac{1}{|\mathcal T_b|} \sum_{t\in\mathcal T_b} \log\alpha^{(q)}_{b,t} \right].\]

The draw we end up using is

\[\widetilde\alpha^{(q)}_{b,t} = \exp\left[\log\alpha^{(q)}_{b,t}-g^{(q)}\right].\]

This normalization does not alter relative differences between benchmarks or temporal changes within a benchmark; it only fixes the otherwise arbitrary unit of the latent scale.

For every propagated draw, I fit a model in which the difficulties can vary across time. The set of discriminations is fixed within that fit, while capabilities and benchmark difficulties are re-estimated. Difficulty deviations follow another shared-length-scale RBF process:

\[\ell_D\sim\operatorname{Uniform}(2,36),\] \[h_b\sim\mathcal N(\mathbf 0,K_D),\] \[\omega_b\sim\operatorname{HalfNormal}(0.5).\]

Let the raw temporal-difficulty be

\[r_{b,t}=\omega_bh_{b,t}.\]

The raw temporal-difficulties could absorb both benchmark averages and a movement shared by all benchmarks in a calendar month. To prevent that, we impose the constraint

\[\sum_{t\in\mathcal T_b}\delta_{b,t}=0\]

for every benchmark. Thus, the deviations average to zero across each benchmark’s supported months and do not change its overall difficulty. We also impose

\[\sum_{b:(b,t)\text{ supported}}\delta_{b,t}=0\]

for every month. Thus, within each month, the deviations average to zero across the benchmarks used in that month. So deviations measure how benchmark difficulties change relative to one another. Monthly difficulty is then

\[D_{b,t}=\bar D_b+\delta_{b,t},\]

and the conditional likelihood is

\[s_{mb} \sim \mathcal N\left( \sigma\left[ \widetilde\alpha^{(q)}_{b,t_m} (C_m-D_{b,t_m}) \right], \sigma_\varepsilon^2 \right).\]

I repeat this fit for complete surfaces drawn from Stage 1 and pool the conditional posteriors:

\[p_{\mathrm{mod}}(\Theta_D\mid s) \approx \frac{1}{Q} \sum_{q=1}^{Q} p_2\left( \Theta_D\mid s,\widetilde{\boldsymbol\alpha}^{(q)} \right).\]

This results in our final estimates, which, when we ignore changes in discriminations and difficulties over time, match Epoch’s estimates pretty well.