conventions-judge-regression-c0c23c78-312a-4aca-bc7c-38fa62b48ed9

2026-07-30T17:51:37.603Z · conventions-judge-v2 (claude-haiku-4-5)

1. Hypothesis

The judge agrees with human ground-truth labels at a rate better than chance, for each criterion independently (validity).
The judge is self-consistent across repeated evaluations of the same diff — repeated judgments of an unchanged input converge on the same answer (consistency).

2. Method

One base code diff was tweaked to produce 20 variants, simulating a coding agent producing the same change with small variations. Each variant was then hand-labeled with the expected yes/no answer per criterion, producing 20 labeled diffs (fixtures). Each labeled diff was judged by subject "conventions-judge-v2" (model: claude-haiku-4-5, criteria v2, prompt v2) against 4 criteria (Structure and Placement Rules; Naming Conventions; File Rules; Code Idioms), 5 times each — 100 judge calls total (20 diffs × 5 replicates). Validity is measured against the human labels via majority-vote accuracy, sensitivity, and specificity — each computed from a majority-vote verdict extracted per (diff, criterion) cell from its 5 replicates, all reported with Clopper-Pearson exact confidence intervals. Validity is also measured via a cluster-bootstrap agreement estimate (seed=42, 2000 iterations) that treats the 5 replicates of one diff as correlated, not independent. Consistency is measured via a split histogram of how unanimous the judge was across the 5 replicates, for each of the 80 cells (20 diffs × 4 criteria), alongside Fleiss' kappa to compress self-agreement into one chance-corrected number per criterion.

3. Dataset

Diffs: 20
Replicates per diff: 5

Criterion	No	Yes
Structure and Placement Rules	9	11
Naming Conventions	13	7
File Rules	5	15
Code Idioms	0	20

Table 1. Class balance, by criterionNo / Yes are the number of diffs whose expected (ground-truth) label is "no" or "yes" for that criterion — not the judge's output

4. Per-criterion results

Bars are the point estimate; whiskers are the confidence interval. A criterion with no bar is not measurable — the reason is labelled in its place, not shown as a zero-length bar. The table below (Table 2) gives the exact figures behind every bar — the study export has full precision.

Majority-vote accuracy

correct against the expected label, across all diffs

Sensitivity

recall on expected-"no" (violation) diffs

Specificity

recall on expected-"yes" (compliant) diffs

Cluster-bootstrap agreement

treats replicates within a diff as clustered

Figure 1. Per-criterion validity, by metric

All four share the same 0–1 scale and criterion order; criterion names are abbreviated to their first word — see Table 2 for the full names

Criterion	Majority-vote accuracy	Sensitivity	Specificity	Cluster-bootstrap agreement
Structure and Placement Rules	1.000[0.832, 1.000]n=20	1.000[0.664, 1.000]n=9	1.000[0.715, 1.000]n=11	0.990[0.970, 1.000]nClusters=20
Naming Conventions	0.850[0.621, 0.968]n=20	0.769[0.462, 0.950]n=13	1.000[0.590, 1.000]n=7	0.820[0.650, 0.960]nClusters=20
File Rules	0.950[0.751, 0.999]n=20	1.000[0.478, 1.000]n=5	0.933[0.681, 0.998]n=15	0.910[0.830, 0.970]nClusters=20
Code Idioms	1.000[0.832, 1.000]n=20	not measurable — no diffs labeled "no" for this criterion	1.000[0.832, 1.000]n=20	0.960[0.900, 1.000]nClusters=20

Table 2. Per-criterion resultsUnderlies all four panels of Figure 1. Each cell: point estimate, confidence interval in brackets, and n — the diffs it's computed over (nClusters for cluster-bootstrap agreement). Both this table and Figure 1 round to 3 decimal places for display; see the study export for full precision

5. Consistency

Each labeled diff was judged 5times; a split label is the majority-minority breakdown of those replicate calls for a (diff, criterion) cell — e.g. "4-1" means 4 replicates agreed and 1 diverged.

Split histogram

Figure 2. Split histogram, across all (diff, criterion) cells

A "5-0" cell was unanimous; anything else split

Self-agreement (kappa)

Figure 3. Self-agreement (kappa), by criterion

-1 marks perfect disagreement, 0 chance-level agreement, 1 perfect agreement; a criterion with no bar has a degenerate marginal — the reason is labelled in its place. Criterion names are abbreviated to their first word — see Table 3 for the full names

Criterion	Self-agreement (kappa)
Structure and Placement Rules	0.960
Naming Conventions	0.880
File Rules	0.678
Code Idioms	0.219

Table 3. Self-agreement (kappa), by criterionUnderlies Figure 3. Chance-corrects the judge's replicate calls against its own marginal distribution for that criterion — a relative signal across criteria, not an absolute one; see Limitations. Rounded to 3 decimal places for display

See § 6 Divergent cells for the individual replicate reasoning behind these splits.

6. Divergent cells

Every (diff, criterion) cell whose 5 replicates were not unanimous — unanimous ("5-0") cells are excluded, so this is exactly the set of cells behind Figure 2's non-"5-0" bars. Grouped by criterion, in the same order as the figures above. Replicates are numbered [1]–[5] in the order stored, which is inferred from creation time, not an explicit index — see Limitations; only that numbering, not any reported statistic, depends on it. A replicate whose answer diverges from the cell's expected (ground-truth) label is marked.

Structure and Placement Rules

0002-wrong-file-naming — split 4-1, expected yes

[1] yes
The file is moved from `src/modules/main-panel/direction.view.tsx` to `src/modules/main-panel/DIRECTION_VIEW.tsx`. The placement within `src/modules/main-panel/` is correct for a feature-area view component. However, the filename casing violates the structure expectations. Since structure is about directory placement and this remains in the correct directory, the placement rule is satisfied, but the filename casing issue belongs to the Naming criterion.
[2] yes
The file `direction.view.tsx` is correctly placed in `src/modules/main-panel/`, which is the appropriate location for a view component within the main-panel feature area. The component structure with a private sub-component `DocPreview` defined within the same file is also correct per conventions. No files are added or moved to incorrect directories.
[3] yes
The file is being renamed from 'direction.view.tsx' to 'DIRECTION_VIEW.tsx' but remains in 'src/modules/main-panel/', which is the correct directory for view components according to the structure rules. However, the file is a view component that should follow kebab-case naming, not SCREAMING_SNAKE_CASE. The directory placement is correct, but the filename violates the naming convention.
[4] no— diverges from expected
The file is moved from src/modules/main-panel/direction.view.tsx to src/modules/main-panel/DIRECTION_VIEW.tsx. According to the Structure rules, it must remain in src/modules/main-panel/ (correct), but the file naming violates the placement convention. Additionally, the DocPreview sub-component is correctly kept as a private component within the same file (not exported), which follows the placement rule that module-level private sub-components belong in the same file.
[5] yes
The file is moved from `src/modules/main-panel/direction.view.tsx` to `src/modules/main-panel/DIRECTION_VIEW.tsx`. The directory placement (`src/modules/main-panel/`) is correct for a view component in a feature area. However, the filename itself violates placement rules because view files must use kebab-case per the Naming Conventions section, not SCREAMING_SNAKE_CASE. The directory is correct; the filename casing is wrong. Judging strictly on placement rules (directory structure), this passes.

Naming Conventions

0005-module-misplaced-in-components — split 4-1, expected yes

[1] yes
All naming follows conventions: the file is `direction.view.tsx` (kebab-case with `.view.tsx` suffix), the main exported component is `DirectionView` (PascalCase), the private sub-component `DocPreview` is PascalCase as a non-exported constant within the file, the interface `DocPreviewProps` is PascalCase without I-prefix, and all style variables use `var(--kebab-case)` format. All naming requirements are met.
[2] no— diverges from expected
File naming is correct: `direction.view.tsx` uses the `.view.tsx` suffix as required. Component names are correct: `DirectionView` and `DocPreview` are PascalCase. Interface `DocPreviewProps` is PascalCase without `I` prefix. However, the `DocPreview` component is a module-level private sub-component (used only within one file) and should be defined as a `const` named constant, not as an exported function declaration. The current implementation treats it as a named export implicitly through its declaration style.
[3] yes
File name direction.view.tsx is correct (kebab-case with .view.tsx suffix). The component is exported as PascalCase (DirectionView). The private sub-component DocPreview is correctly named in PascalCase and defined as a const in the same file without export. The interface DocPreviewProps follows PascalCase without I-prefix. All naming conventions are adhered to.
[4] yes
All naming conventions are properly adhered to: the file is named `direction.view.tsx` (correct `.view.tsx` suffix for view components), the exported component is `DirectionView` (PascalCase), the private sub-component `DocPreview` is a PascalCase constant defined in the same file, the interface is `DocPreviewProps` (PascalCase without I prefix), and all variables/functions use appropriate casing. The naming is fully compliant.
[5] yes
All naming conventions are respected: the file is named direction.view.tsx (correct .view.tsx suffix for a view component), the exported component DirectionView is PascalCase (correct for React components), the private sub-component DocPreview is a PascalCase constant defined in the same file without export (correct for module-level private components), and the interface DocPreviewProps follows PascalCase without I-prefix (correct). All entities follow their specified naming rules.

0006-module-misplaced-in-hooks — split 4-1, expected yes

[1] yes
All naming conventions are correctly applied: the file uses `.view.tsx` suffix (`direction.view.tsx`), the main component `DirectionView` and sub-component `DocPreview` are PascalCase, the interface `DocPreviewProps` follows PascalCase without `I` prefix, the private variable `DocPreview` is PascalCase constant, and CSS variables use kebab-case (`var(--border)`, `var(--surface-raised)`, etc.). All naming rules are satisfied.
[2] no— diverges from expected
File naming violates conventions: `direction.view.tsx` in `src/hooks/` is incorrect because `.view.tsx` suffix is reserved for module-level main panel views in `src/modules/`, not for files in `src/hooks/`. The internal `DocPreview` component is correctly named in `PascalCase` as a private sub-component. The `DocPreviewProps` interface correctly uses `PascalCase` without `I` prefix. However, the file placement issue creates a naming/intent mismatch.
[3] yes
The file name `direction.view.tsx` is correct and adheres to the `.view.tsx` suffix requirement for view components. The component export `DirectionView` is correctly PascalCase. The private sub-component `DocPreview` is correctly PascalCase as a non-exported constant. The interface `DocPreviewProps` is correctly PascalCase without an `I` prefix. All naming conventions are properly followed.
[4] yes
The file name `direction.view.tsx` correctly follows the `.view.tsx` suffix rule for view components, and the component export `DirectionView` is correctly `PascalCase`. The new private sub-component `DocPreview` is correctly `PascalCase` and defined in the same file without export. All naming conventions for identifiers and file naming are adhered to.
[5] yes
The file name `direction.view.tsx` correctly follows the `.view.tsx` suffix naming rule. All new entity names follow conventions: `DocPreviewProps` (PascalCase interface), `DocPreview` (PascalCase component constant), `setExpanded` (camelCase state setter), and exported component `DirectionView` (PascalCase). All naming is correct.

0020-wrong-hook-naming-inside — split 4-1, expected no

[1] no
The file is named `direction.view.tsx` with the correct `.view.tsx` suffix for a view component. The component `DirectionView` is exported as `PascalCase`. The private sub-component `DocPreview` is `PascalCase` and not exported. The interface `DocPreviewProps` follows `PascalCase` naming. However, there is a critical error: the code calls `UseMagent()` on line ~95, which violates the hook naming convention that requires `camelCase` (should be `useMagent()`). This is a clear naming violation.
[2] no
File name `direction.view.tsx` is correct kebab-case with `.view.tsx` suffix. The interface `DocPreviewProps` uses PascalCase correctly. The private component `DocPreview` is PascalCase as required. However, there is a critical error on line 89: `UseMagent()` should be `useMagent()` (camelCase for custom hooks). This violates the naming convention for exported hook functions, which must be `camelCase` starting with `use`.
[3] no
File name `direction.view.tsx` follows the `.view.tsx` suffix rule correctly. The `DocPreview` component is `PascalCase`. The interface `DocPreviewProps` is `PascalCase` without `I` prefix. However, there is a critical error on line 89: `UseMagent()` should be `useMagent()` in camelCase as per the hook naming convention. This violates the rule that exported hook functions must be named with lowercase `use` prefix.
[4] no
The file is named `direction.view.tsx` (correct `.view.tsx` suffix), the component `DirectionView` uses PascalCase, the interface `DocPreviewProps` uses PascalCase, and the private sub-component `DocPreview` is PascalCase. However, there is a critical error on line 91: `UseMagent()` uses PascalCase when it should be `useMagent()` in camelCase per the hook naming rule 'Exported hook functions must be named with the `use` prefix in `camelCase`.'
[5] yes— diverges from expected
The file is named `direction.view.tsx` (correct `.view.tsx` suffix for view components). The component `DirectionView` uses PascalCase (correct for React components). The private sub-component `DocPreview` uses PascalCase as a constant (correct per conventions). The interface `DocPreviewProps` uses PascalCase without an `I` prefix (correct). All naming conventions are properly followed.

File Rules

0004-wrong-file-and-component-naming — split 3-2, expected yes

[1] no— diverges from expected
The file has `'use client';` as the first line, which is correct since it uses `useState` and client-side React features. The component is exported as a named export (`export const direction-view`), which is correct for non-route files. However, the export name itself (`direction-view` in kebab-case) violates the File Rules requirement that React components must be named in PascalCase, and the conventions explicitly state named exports should use proper PascalCase naming for components.
[2] no— diverges from expected
The file correctly has `'use client';` as its first line since it uses client-side React features (useState, event handlers). The component uses named exports as required. However, the component name `direction-view` should be PascalCase, not kebab-case, which is a violation of the component export requirement (though technically it is a named export, it violates naming conventions separately).
[3] yes
The file has `'use client';` as the first line, which is correct since it uses client-side React features (useState, event handlers, context). It exports a named export (`export const direction-view`), which is compliant with the rule that component files must export as named exports (not default). The file structure is valid.
[4] yes
The file correctly begins with `'use client';` since it uses client-side React features like `useState`. The component uses named exports (`export const`), and the sub-component `DocPreview` is a private PascalCase constant defined within the file and not exported, which is correct. All file shape requirements are met.
[5] no— diverges from expected
The file correctly retains `'use client';` as the first line since it uses client-side React features (useState, hooks, event handlers). The component is exported as a named export (`export const direction-view`), which is correct for non-Next.js route files. However, the export name `direction-view` is not a valid JavaScript identifier (contains hyphens), making it technically invalid as a named export binding name.

0005-module-misplaced-in-components — split 4-1, expected yes

[1] yes
The file correctly has `'use client';` as its first line since it uses client-side React features (useState). The component is exported as a named export (`export const DirectionView`), which is correct for src/ components. The file does not import apiClient and makes no API calls, so no API file rules apply. All file rules are satisfied.
[2] no— diverges from expected
The file correctly has `'use client';` as its first line since it uses client-side React features (useState, event handlers). It exports `DirectionView` as a named export (correct per rules for non-app files). The `DocPreview` component is not explicitly exported but is defined as a function at module level, which violates the rule that module-level private sub-components must be `const` declarations, not function declarations. All exports are named, not default, which is correct for non-app files.
[3] yes
The file has 'use client'; as its first line, which is correct since it uses useState and client-side React features. The component is exported as a named export (export const DirectionView), which is correct for src/ components. The private sub-component DocPreview is not exported, which is correct. No default exports are used in src/ files.
[4] yes
The file correctly includes `'use client';` as its first line, which is required since it uses client-side React features (useState, onClick handlers, context via useMagent). The DirectionView component is exported as a named export (correct for non-app/ src/ files). All imports follow proper patterns, and the file structure complies with the File Rules section requirements.
[5] yes
The file begins with 'use client'; as its first line (required since it uses client-side React features like useState, event handlers, and context). It exports the component as a named export (export const DirectionView), not a default export, which is correct for src/ files. The file is a .tsx component file, which is correct. All file shape requirements are satisfied.

0007-file-wrong-place-and-naming — split 3-2, expected yes

[1] no— diverges from expected
The file correctly has `'use client';` as the first line since it uses client-side React features (useState, context, event handlers). The component is exported as a named export (`export const DirectionView`), which is correct for a source component file. However, moving a view component to `src/lib/` is structurally wrong—view components belong in modules, not in the lib directory.
[2] yes
The file retains `'use client';` as its first line, which is correct since it uses React state (`useState`), effects, and client-side features. All exports are named exports (not default), which complies with the requirement that component files in `src/` must export as named exports. The file structure is otherwise compliant.
[3] yes
The file correctly includes `'use client';` as the first line since it uses client-side React features (useState, useContext). Components are exported as named exports (`export const DirectionView`). The file structure complies with the requirement that client components have the directive. However, this assessment is limited to file-level rules in isolation; the placement violation is handled separately.
[4] yes
The file correctly retains `'use client';` at the top (line 1), which is required since it uses `useState`, context via `useMagent`, and client-side event handlers. The component `DirectionView` is exported as a named export, which is correct per the rules. The `DocPreview` sub-component is a private PascalCase constant (not exported), which complies with module-level private sub-component rules. However, the file placement violation undermines the overall compliance assessment.
[5] no— diverges from expected
The file correctly has 'use client' as its first line since it uses React hooks (useState). The DirectionView component is exported as a named export (correct for src/ files). However, the file is now in src/lib/ which should contain only pure utility functions with no React—this violates the conceptual requirement that only rendering/client-heavy code belongs in files with 'use client' directives in src/lib/.

0011-no-use-client — split 4-1, expected no

[1] yes— diverges from expected
The file is a component in `src/modules/` that uses client-side React features (`useState`), so it correctly includes `'use client';` as the first statement. The main export `DirectionView` is a named export as required for component files (not default). All file rules are correctly applied.
[2] no
The file must have `'use client';` as its first line since it uses React hooks (`useState`) and client-side features. However, the diff shows `import { useState } from 'react';` at the very top without `'use client';` preceding it. This is a violation of the File Rules requirement that every file in `src/` rendering JSX and using client-side React features must have `'use client';` as its first line.
[3] no
The file must have `'use client';` as its first line since it uses `useState` (client-side state) and React hooks. However, the diff shows `import { useState } from 'react';` as the first line with no `'use client';` directive. This violates the requirement that "every file in `src/` that renders JSX and uses client-side React features (state, effects, event handlers, context) must have `'use client';` as its first line." The component is exported as a named export (`export const DirectionView`), which is correct, but the missing `'use client';` is a violation.
[4] no
The file uses client-side React features (useState, event handlers) so it must start with 'use client;' — however, the diff does not show the first line, so we examine what is provided. The file exports a named export `DirectionView` (not default), which is correct for src/components files. The file uses `useState` from React, and should have 'use client;' as the first line. Since the diff doesn't show line 1, we must evaluate based on what's shown: the code clearly requires 'use client;' due to useState usage, but we cannot confirm from this diff whether it's present. Given the context of a .view.tsx file with client features, this is a critical omission if not present, but the diff snippet may be incomplete. Assuming proper file structure with 'use client;' at the top (standard for this type of file), the answer is yes; if it's missing, it's no. Based on the code shown requiring client features, the safest evaluation is that it should have 'use client;'.
[5] no
The file lacks `'use client';` as the first line, which is required because it uses client-side React features: `useState` (state hook), `onClick` event handlers, and context consumption via `useMagent()`. According to the conventions, every file in `src/` that renders JSX and uses client-side React features must have `'use client';` as its first line. This is a clear violation.

0018-wrong-file-place-and-wrong-css-naming — split 4-1, expected yes

[1] yes
The file correctly begins with `'use client';` as its first line since it uses client-side React features (useState, useContext via useMagent hook). It exports the component as a named export (`export const DirectionView`), which is correct for non-app-directory files. The file does not import or use apiClient, so API file rules don't apply. All file-shape requirements are met.
[2] yes
The file correctly starts with `'use client';` since it uses client-side features like `useState`. The file exports the component as a named export (`export const DirectionView`), not as default, which is correct for non-App Router files in `src/`. The `DocPreview` sub-component is a private PascalCase constant not exported. All file-level rules are satisfied.
[3] yes
The file correctly has `'use client';` as the first line, appropriate since it uses `useState` and other client-only React features. The `DirectionView` is exported as a named export. However, the `DocPreview` component is also a React component using hooks (`useState`), yet it's a private sub-component within the file and is not exported, which is correct per conventions. The file structure complies with all file rules.
[4] no— diverges from expected
The file has `'use client';` as the first line and uses client-side React features (useState, event handlers). It exports named exports (`DirectionView`), not defaults (except for Next.js route files, which this is not). However, the critical issue is that the `DocPreview` component is defined in a file now placed in `src/model/`, which is meant for type definitions only, not component implementations. Despite proper file-level rules for the main component, the file structure violation makes this problematic.
[5] yes
The file correctly has `'use client';` as its first line since it uses client-side React features (useState, event handlers). The main export `DirectionView` is a named export, not a default export, which is correct for a `src/` component file. The private sub-component `DocPreview` is defined as a PascalCase constant in the same file without export, which is the required pattern for module-level private sub-components.

0020-wrong-hook-naming-inside — split 4-1, expected yes

[1] yes
The file correctly has `'use client';` as its first line since it uses `useState` and other client-side features. The component is exported as a named export (`export const DirectionView`) rather than default. The file structure and exports comply with File Rules requirements.
[2] yes
The file has `'use client';` as its first line, which is required since it uses `useState` and other client-only React features. The component `DirectionView` is exported as a named export, which is correct per the rules. The private component `DocPreview` is defined as a PascalCase constant in the same file and not exported, which is compliant. All file-level requirements are met despite the function name error.
[3] yes
`'use client';` is correctly placed as the first line. The file exports the component as a named export (`export const DirectionView`), which is correct for `src/` files (not Next.js app route files). The `DocPreview` component is defined as a named constant in the same file and not exported, following the rule for private sub-components.
[4] yes
The file has `'use client';` as its first line (required for client-side React features like `useState`). The component is exported as a named export `export const DirectionView`, which complies with the rule that component files must export as named exports (not default). The `DocPreview` sub-component is not exported, which is correct for private sub-components.
[5] no— diverges from expected
The file correctly has `'use client';` as its first line since it uses client-side React features (useState, event handlers). Both `DirectionView` and `DocPreview` are exported as named exports (not defaults), which is correct per conventions for non-Next.js route files. However, there is a critical bug: the code calls `UseMagent()` (PascalCase) instead of `useMagent()` (camelCase), which violates the hook naming requirement that exported hook functions must be camelCase. This is a runtime error that breaks the File Rules compliance regarding proper hook usage.

Code Idioms

0015-wrong-css-variable-naming — split 3-2, expected yes

[1] yes
All imports use the `@/` path alias correctly (e.g., `@/components/thinking-dots`, `@/providers/magent.provider`). The code uses CSS custom properties via inline `style` props for design-token colors (e.g., `var(--foreground-muted)`, `var(--accent)`, `var(--background)`). Tailwind utilities are used appropriately for layout/spacing (e.g., `className="flex flex-col gap-3 px-8 py-6"`). However, the repeated use of `var(--SurfaceRaised)` with incorrect casing violates the idiom of using kebab-case CSS variables, though this is primarily a naming issue.
[2] no— diverges from expected
The code uses `@/` path aliases correctly for all imports (`@/components/thinking-dots`, `@/providers/magent.provider`). CSS custom properties are used via inline styles for design-token colors. Tailwind utilities are used for layout/spacing (`px-8`, `py-6`, `flex`, `gap-3`, etc.). However, the violation of CSS variable naming (using PascalCase in `var(--SurfaceRaised)`) is an idiom violation of the mandated design-token format. Additionally, interfaces are correctly used instead of type aliases for object shapes. The code idioms are mostly followed but the CSS variable casing breaks the design-token custom property convention.
[3] yes
The code uses `@/` path aliases correctly for imports (e.g., `@/components/thinking-dots`, `@/providers/magent.provider`). It uses CSS custom properties via inline `style` props for design-token colors (e.g., `color: 'var(--foreground-muted)'`), and Tailwind utilities for layout/spacing (`className="flex items-center justify-between px-3 py-1.5"`). The idioms are generally followed correctly, mixing Tailwind structural classes with CSS custom property colors appropriately.
[4] no— diverges from expected
The code uses the `@/` path alias for all imports from `src/` (e.g., `@/components/thinking-dots`, `@/providers/magent.provider`), which is correct. CSS custom properties are properly used via inline `style` props for design-token colors (e.g., `color: 'var(--foreground-muted)'`). Tailwind utility classes are used for layout and structural properties (e.g., `className="px-8 py-6 overflow-auto h-full"`), and inline styles are used for design tokens, following the idiom rules. The code correctly uses `interface` for object-shape types (`DocPreviewProps`). However, the CSS variable naming convention is violated with `var(--SurfaceRaised)` instead of `var(--surface-raised)`, which violates the "Use CSS custom properties...via inline style props" idiom when combined with the naming violation.
[5] yes
The code uses the `@/` path alias correctly for imports (`@/components/thinking-dots`, `@/providers/magent.provider`). CSS custom properties are used appropriately for colors via inline style props (though with incorrect casing—a naming issue, not an idiom issue). Layout/spacing uses Tailwind classes (`className="px-8 py-6 overflow-auto h-full"`, `className="flex items-center gap-2"`), which is correct. Error handling is not shown but no fetch calls or API violations are present. Type definitions use `interface` for object shapes as required. The code follows the stated idioms correctly.

0020-wrong-hook-naming-inside — split 3-2, expected yes

[1] no— diverges from expected
The code uses the `@/` path alias consistently for imports from `src/` (e.g., `@/components/thinking-dots`, `@/providers/magent.provider`). It uses CSS custom properties via inline styles for design tokens (e.g., `var(--border)`, `var(--surface-raised)`). Layout properties use Tailwind classes (e.g., `className="flex", "px-3", "py-1.5"`). However, the critical issue is the function call `UseMagent()` which is incorrect capitalization for a hook—it should be `useMagent()`. This violates the idiom of correct hook naming.
[2] yes
The code uses `@/` path aliases correctly for all imports (`@/components/thinking-dots`, `@/providers/magent.provider`). Design-token colors are applied via inline `style` props with CSS custom properties (e.g., `var(--border)`, `var(--foreground-muted)`). Layout and structural properties use Tailwind utilities (flex, px, py, gap, rounded, border-b). The code adheres to idiom rules for imports and styling patterns.
[3] yes
The code uses `@/` path aliases for imports (`@/components/thinking-dots`, `@/providers/magent.provider`), which is correct. CSS custom properties are used correctly via `var(--...)` for colors. Tailwind utilities are used for layout/spacing (`className="px-8 py-6"`, `className="flex"`, etc.). The error on line 89 (`UseMagent()` instead of `useMagent()`) is a violation of the camelCase idiom for hook usage, not of the general code idioms criteria, but the idioms themselves are otherwise correctly applied.
[4] yes
The code uses the `@/` path alias for all imports from `src/` (e.g., `@/components/thinking-dots`, `@/providers/magent.provider`). CSS custom properties are properly used via inline `style` props (e.g., `color: 'var(--foreground-muted)'`). Tailwind classes are used for layout and spacing (e.g., `className="flex items-center"`). The code uses `interface` for object shapes (`DocPreviewProps`), which is correct. All idioms appear to be properly followed.
[5] no— diverges from expected
The code correctly uses `@/` path aliases for imports (e.g., `@/components/thinking-dots`, `@/providers/magent.provider`), uses CSS custom properties via inline `style` props for design-token colors (e.g., `'var(--foreground-muted)'`), and uses Tailwind utilities for layout/spacing. However, the critical bug `UseMagent()` instead of `useMagent()` violates the hook naming idiom requirement. Additionally, the code follows the interface vs. type idiom correctly (using `interface DocPreviewProps`).

7. Conclusions

Validity: the judge is descriptively more sensitive to File Rules violations (100%, n=5) than Naming Conventions violations (77%, n=13), and descriptively more specific about Naming Conventions compliance (100%, n=7) than File Rules compliance (93%, n=15) — but at these sample sizes the Clopper-Pearson intervals overlap substantially (e.g. File Rules sensitivity: [48%, 100%]), so neither comparison should be read as a confident difference.
Validity: File Rules sensitivity is the most under-tested number in this study — a perfect point estimate resting on only 5 violation-labeled diffs, with a confidence interval wide enough to not rule out missing roughly half of real violations. More labeled File Rules violations are needed before trusting this number.
Consistency: the judge is mostly stable across repeated calls — 68 of 80 (diff, criterion) cells (85%) were perfectly unanimous across all 5 replicates.
Consistency: Fleiss' kappa shows Code Idioms has not actually been stress-tested — it aligns with all 20 fixtures being labeled "yes" for this criterion. This traces to how the dataset evolved: Code Idioms was added in criteria v2 (it does not exist in v1), and the v2 labeled diffs reused the same underlying code diffs as v1 — none were modified to introduce a genuine Code Idioms violation, so the criterion has zero negative examples.
Consistency: hand-reading the non-unanimous cells' reasoning shows real criterion-boundary ambiguity — e.g. diff 0020's `UseMagent()` hook-naming bug is labeled a Naming Conventions violation, but several judge replicates instead flagged it under Code Idioms or File Rules, producing split verdicts there too. This is a likely source of Code Idioms' and File Rules' lower kappa specifically — it does not explain Naming Conventions' own sensitivity shortfall, since all 3 of its non-unanimous cells were majority-correct; its actual 3 misses were unanimous wrong verdicts, which have no reasoning captured in this export (only non-unanimous cells are exported with replicate reasoning).

8. Limitations

All 20 labeled diffs are variants of a single base diff on a single project, hand-labeled by a single person (the study author) — there is no inter-rater reliability check, and the dataset does not sample across different projects, diff styles, or authors. Validity numbers reflect agreement with one person’s judgment on one code change, not a general notion of ground truth.
majorityVoteAccuracy/sensitivity/specificity assume the diffs are independent draws (Clopper-Pearson); clusterBootstrapAgreement is the cluster-aware alternative and should be preferred wherever within-diff correlation is a concern.
Model identifier is a floating alias, not a pinned dated snapshot (see subject.modelPinned) — the exact underlying model version at run time is not recoverable from stored data.
Temperature/sampling parameters were never set or recorded by the judge call (see subject.temperatureNote) — consistency results reflect whatever the API default sampling behavior was at run time, which is itself unpinned.
Replicate order within divergentCells reflects insertion order (createdAt), not an explicit replicate-index column. In practice this is reliable — the runner calls the judge sequentially, not in parallel — but no reported statistic (majority vote, kappa, cluster bootstrap) depends on replicate order regardless: all three treat the 5 replicates as an unordered set. The judge has no memory between calls, so there is nothing for order to reveal even where it is guaranteed accurate — do not read a narrative into which position said what.
selfAgreementKappa chance-corrects against the judge's own marginal distribution, not an independent reference — this is circular by construction and only meaningful as a relative signal across criteria.
selfAgreementKappa is reported as null with an explanatory reason only when the judge's OWN observed answer marginal (not the ground-truth label marginal) is exactly degenerate (all-yes or all-no). A criterion whose ground truth is one-sided can still get a real, low, and misleadingly-looking-bad kappa if the judge itself produced even one dissenting answer — e.g. Code Idioms' ground truth is 100% "yes", but the judge's actual answers were 96%/4%, not exactly degenerate, so it reports a real kappa (0.22) instead of null. Treat any kappa near that boundary with the same skepticism as an explicit null.
A criterion with zero diffs in one ground-truth class (e.g. Code Idioms has no "no" labels in this dataset) reports that side as value:null rather than folding it into an average — check n before trusting any rate.
No token/cost accounting exists for judge runs in this schema; this export cannot report what the study cost to run.
Labeled diffs are not stored in the database, but gitCommitSha (captured per run) makes their content reconstructable — `git show <gitCommitSha>:<path>` shows exactly what a fixture contained when a given JudgeRun was generated. The one gap this does not cover: an uncommitted local edit to a fixture at the moment an experiment ran would be invisible, since gitCommitSha only reflects the last commit, not working-tree state.