An AI originality verdict should come with evidence you can challenge

The judge is part of the experiment: read its labels first

Ai2 published an October 27 overview of Old Ideas, Novel Problems, a study of how language models judge the novelty of research ideas. The paper was first submitted on October 1 and revised on October 26. Those are distinct dates: the new overview is not the paper’s first appearance.

The researchers report that seemingly small changes in evaluation setup can substantially change judgments. Across the tested models, retrieval and greater reasoning effort did not provide a consistent remedy. The authors also released evaluation data and code. These are findings from a specific study, not proof that every model is incapable of useful research comparison.

The ethical concern is straightforward. If a fluent verdict helps decide which idea receives attention, a brittle judging process can quietly become a gatekeeper. Someone may abandon a worthwhile project because an assistant calls it obvious, or overstate originality because another assistant calls it unprecedented.

Our interpretation is that novelty tools should produce challengeable comparisons before they produce final-sounding labels. The useful artifact is an account of relevant prior work, the proposed difference and what remains uncertain. A numerical score or an enthusiastic paragraph cannot replace those pieces.

We have not reproduced the study or tested the judges. The exercise in this article is a suggested way to inspect novelty judgments in a small learning setting. It is not a new benchmark, a validated research-review method or evidence that a particular current model is reliable.

The study’s benchmark uses two settings with different evidential strength. The human-only setting draws higher- and lower-novelty ideas from ICLR 2026 submissions, using explicit reviewer statements as well as selection criteria. The mixed setting pairs the higher-novelty human ideas with simply generated ideas treated as lower novelty through weak labels.

That difference matters. A weak label is an assumption used for evaluation, not an individually verified fact about every generated idea. Treating the mixed setting as proof that all generated ideas lack originality would go beyond what the benchmark establishes.

The paper reports that changing the framing about reviewer novelty judgments could alter verdicts on more than half of identical idea pairs in a tested configuration. Its core lesson concerns sensitivity to evaluation design. A headline-sized performance number without the prompt, data setting and comparison method can hide the conditions that produced it.

The released repository documents pointwise and pairwise evaluations, different idea formats and controlled changes. It also describes external API requirements for reproducing the work. Access to code does not make a full replication costless or guarantee that later model versions reproduce the same results.

Before using any novelty score, ask what the labels mean and which population the examples represent. Research selected from a conference’s extremes is useful for a controlled question. It is not automatically representative of a student’s early proposal, an interdisciplinary community project or an idea whose relevant literature is difficult to retrieve.

Source: Ai2: Can we trust LLMs to judge whether a research idea is novel? ↗ · Sternlicht and colleagues: Old Ideas, Novel Problems, revised October 26 ↗ · Ai2: Novelty-eval code, benchmark settings and reproduction requirements ↗

Break originality into a concrete comparison

Consider a fictional proposal: use audio from household appliances to detect when a filter needs cleaning. An AI judge might call it novel because it combines sound and maintenance. Another might call it old because acoustic monitoring already exists. Both replies can sound reasonable while leaving the actual comparison unfinished.

Write the proposed contribution in a few precise parts. What signal is measured? What decision is made? For whom, under which conditions and with what evaluation? This does not establish originality. It makes the claim specific enough that a reader can compare it with something else.

Now describe the closest prior work using the same parts. Perhaps an earlier project used vibration rather than audio, detected bearing wear rather than filter condition or worked only in an industrial environment. Those distinctions may matter, but difference alone is not significance. You still need to explain what the change enables or tests.

Separate a new application from a new mechanism and a new evaluation. A project may contribute through one of these without inventing all three. If your description promises a new algorithm when the real contribution is a careful field study, the novelty question has been misframed before the judge begins.

The aim is a claim someone can dispute intelligently. Instead of declaring that nobody has done this, describe the closest verified work you found, the difference you propose and the search boundary. That wording is more limited, but it also gives the next reviewer somewhere concrete to start.

Keep retrieval and judgment in separate lanes

An assistant can help search for related work and can help compare texts. Those are distinct tasks with different failure modes. A missing paper is a retrieval problem. Misreading a retrieved paper’s method is an interpretation problem. A poor novelty verdict can involve either or both.

For a small review, keep a source list containing the works actually inspected. Include titles, direct links and the specific passages or sections relevant to the comparison. Do not promote a plausible citation into evidence until you have opened the real source and checked that it says what the comparison needs.

Ask the system to identify the most relevant overlap before discussing differences. This counters the temptation to celebrate a proposal by listing only what sounds new. A fair comparison should survive seeing the strongest nearby example, not just several weakly related titles.

Then mark which claims rely on abstracts and which rely on fuller reading. An abstract may be enough to identify a candidate comparison, but not enough to settle a technical distinction. If access is missing, preserve that limitation rather than filling it with a likely account of the paper.

Retrieval volume is not the same as coverage. Twenty similar papers from one vocabulary or venue can miss a relevant research tradition using different terms. Write down alternative terms and fields to investigate. The assistant’s failure to retrieve a match is evidence about the search you ran, not proof that no match exists.

Try a wording-stability exercise

A manageable exercise begins with two short idea descriptions you are allowed to use. Choose a familiar topic where you can verify at least a few relevant sources. Avoid real grant, admissions or hiring decisions. The purpose is to expose instability without letting a classroom experiment affect someone’s opportunities.

Create a plain description of each idea, using comparable detail and tone. Then make a second version that changes presentation while preserving the substantive proposal. Remove flattering adjectives from one, simplify jargon in another or change sentence order. Keep a written record of exactly what changed.

Ask a chosen model to compare the plain versions, using a fixed instruction and evidence packet. Save the complete result. Repeat with the presentation variants and record whether the verdict, reasoning or cited evidence changes. This is a proposed sensitivity check, not a reproduction of Ai2’s experiments.

You can also reverse the order of the two ideas in a separate comparison. Treat that as its own controlled change, rather than changing order, wording, sources and model simultaneously. If you alter everything at once, a different answer gives you little information about why it changed.

Do not select only the output that agrees with your preference. Keep the uncomfortable results. A change of verdict is not automatically an error, but a verdict that flips while the evidence and substantive claims remain constant deserves investigation before you rely on it.

Ask what the explanation actually supports

After each comparison, read the explanation without the final label. Can you identify the prior work, the claimed overlap and the proposed difference? Could another person inspect the same sources and reconstruct the argument? If not, the explanation may be too vague to evaluate.

Watch for conclusions that outrun the evidence. A system may say an idea is unique because none of the three supplied papers uses the exact phrase in its title. That is a narrow observation turned into a broad originality claim. A more defensible conclusion would name the inspected set and the unresolved search.

Also watch for a change in the question. The assistant may prefer the more feasible idea, the more socially valuable idea or the better-written abstract. Those can be meaningful judgments, but they do not settle which contribution is more original. Ask for them separately if they matter.

An explanation can contain correct observations and still produce a weak verdict. For example, it may accurately identify that one idea uses a different sensor, then assume that any hardware change creates a significant contribution. Challenge the missing step: why does that difference matter for the research question?

Do not assume a longer reasoning passage is more reliable. More text can add useful distinctions or additional unsupported claims. Evaluate the evidence-bearing steps, not the amount of explanation. A brief comparison with a verifiable limitation may be more useful than a confident essay with no clear source boundary.

Give the person being judged a route to respond

If a novelty judgment affects someone else’s work, the process should leave room for correction. A person may know a relevant source the system missed, identify a mistaken description or explain that the proposal’s contribution was framed inaccurately. Design that response route before treating scores as decisions.

For a student group, that might mean sharing the comparison sheet and inviting corrections before choosing a project. For a research team, it might mean having the author identify the nearest prior work and clarify the proposed contribution. These are suggested practices, not a claim that one appeal mechanism fits every institution.

Keep the original judgment and the correction distinguishable. Quietly editing the assistant’s answer into a better argument erases evidence about how the process failed. A small record of the initial claim, the challenge and the revised conclusion makes the improvement understandable.

Be careful with confidential proposals. Getting another AI opinion is not a reason to upload someone’s unpublished idea to an unauthorized service. The right comparison environment depends on the material and the organization’s rules. A public, fictional exercise is often enough to learn the method.

This is where the ethics moves beyond model accuracy. An evaluation process distributes attention and credibility. People should be able to understand the evidence used about their work, identify missing context and know who remains responsible for the actual decision.

Keep value broader than novelty and finish with an evidence note

A project can be valuable without being the first of its kind. Replication, translation, accessible tooling and careful implementation in a different setting may deserve support for reasons other than a dramatic novelty claim. A ranking that rewards only apparent surprise can hide those reasons.

In the appliance example, a study that fails to invent a new algorithm might still establish whether an existing method works under a neglected set of practical constraints. The contribution would need to be described and evaluated honestly. Its value should not depend on persuading an assistant to use a stronger adjective.

Separate your decision criteria before reviewing proposals. You might care about usefulness, feasibility, learning value, originality and the quality of the planned evidence. Writing them separately lets a modestly original but well-designed project compete on the merits that actually matter to your group.

Do not average several AI novelty scores and assume you have solved the problem. Multiple judges can share missing sources or react to the same persuasive framing. Agreement is an observation about the judges. It becomes more informative when paired with independently checked evidence and a clear question.

Likewise, disagreement does not always mean the project should stop. It can show where the contribution needs clarification or where the prior-work search is incomplete. Treat a split verdict as a reason to inspect the comparison, rather than a contest in which the most enthusiastic model wins.

At the end of the exercise, write a short note with four elements: the proposed contribution, the closest verified comparison, the meaningful difference and the biggest unresolved question. Keep the model’s rating as a secondary observation if you retain it at all.

For the fictional appliance project, the note might say that earlier work already uses acoustic monitoring, while the proposal investigates a different maintenance target under household conditions. It would then identify the actual prior work and explain what evidence is still needed. That is an illustrative structure, not a literature finding about appliance research.

Include the wording-stability result. If presentation changes altered the verdict, say so plainly. If the verdict remained stable in your small exercise, do not generalize to all proposals. The result only describes the material, model and setup you used.

Choose a next action tied to the unresolved question. Read a candidate paper in full, ask a domain expert about a distinction or narrow the proposal’s claim. Another round of unsupported ratings is usually less useful than resolving one identifiable evidence gap.

Ai2’s study makes the evaluator itself an object worth inspecting. A novelty tool can assist research without becoming an unchallengeable authority. The practical standard is an answer that exposes its comparisons and limitations clearly enough for someone else to improve it.