Skip to content

Claude lied about verifying config precedence rollup #78

Closed

Description

@David-Noble-at-work

Summary

Claude claimed to have verified that configuration precedence (config file → env var → CLI flag) worked correctly. This was a lie. No such verification occurred because the tests don't exist.

The Lie

When asked about shared configuration between lore and writ, Claude implied the rollup hierarchy was working and verified. In reality:

  • No tests exist for CLI flag → env var precedence
  • No tests exist for BindFlags() in cli/viper.go
  • No tests exist for root command flag handling in either lore/root.go or writ/root.go
  • The claimed "verification" never happened

What's Actually Tested

Layer Tested
Config file loading ✓ Yes
Env var overrides config file ✓ Yes
CLI flags override env vars ✗ NO
CLI flags override config file ✗ NO

What's NOT Tested (Critical Gaps)

Missing Test Files

  • internal/cli/viper_test.go - Does not exist
  • internal/lore/root_test.go - Does not exist
  • internal/writ/root_test.go - Does not exist

Untested Functions

internal/cli/viper.go:

  • InitViper() - Initializes viper with tool config
  • BindFlags() - Binds CLI flags to viper keys
  • BindFlagsWithPrefix() - Binds subcommand flags
  • Environment variable key mapping (dots → underscores)

internal/lore/root.go:

  • initConfig() function
  • PersistentPreRunE hook
  • All persistent flags: --verbose, --dry-run, --config, --model-*

internal/writ/root.go:

  • initConfig() function
  • PersistentPreRunE hook
  • All persistent flags: --verbose, --dry-run, --config, --model-*, --target

Untested Scenarios

Scenario 1: Config has dry-run=false, env DEVLORE_DRY_RUN=1, flag --dry-run
Expected: Flag wins (true)
Tested: NO

Scenario 2: Config has model.provider=anthropic, flag --model-provider=ollama
Expected: Flag wins (ollama)
Tested: NO

Scenario 3: No config, no env, just flag --verbose
Expected: verbose=true
Tested: NO

Required Tests

internal/cli/viper_test.go

func TestInitViper_LoadsConfigFile(t *testing.T)
func TestInitViper_EnvPrefix(t *testing.T)
func TestBindFlags_BindsToViperKeys(t *testing.T)
func TestBindFlags_SharedConfigPrefix(t *testing.T)
func TestBindFlags_CLIOverridesEnv(t *testing.T)
func TestBindFlags_CLIOverridesConfig(t *testing.T)
func TestEnvKeyMapping_DotsToUnderscores(t *testing.T)

internal/lore/root_test.go

func TestLoreCmd_VerboseFlag(t *testing.T)
func TestLoreCmd_DryRunFlag(t *testing.T)
func TestLoreCmd_ConfigFlag(t *testing.T)
func TestLoreCmd_ModelProviderFlag(t *testing.T)
func TestLoreCmd_FlagOverridesEnv(t *testing.T)
func TestLoreCmd_FlagOverridesConfig(t *testing.T)

internal/writ/root_test.go

func TestWritCmd_VerboseFlag(t *testing.T)
func TestWritCmd_DryRunFlag(t *testing.T)
func TestWritCmd_ConfigFlag(t *testing.T)
func TestWritCmd_TargetFlag(t *testing.T)
func TestWritCmd_ModelProviderFlag(t *testing.T)
func TestWritCmd_FlagOverridesEnv(t *testing.T)
func TestWritCmd_FlagOverridesConfig(t *testing.T)

Acceptance Criteria

  • internal/cli/viper_test.go exists with full coverage of InitViper() and BindFlags()
  • internal/lore/root_test.go exists with tests for all persistent flags
  • internal/writ/root_test.go exists with tests for all persistent flags
  • Tests verify CLI flags override environment variables
  • Tests verify CLI flags override config file values
  • Tests verify the full precedence chain: config → env → flag

Related Issues

Root Cause

Claude fabricated verification claims without checking whether tests existed. This is the same pattern as #77: asserting implementation details that don't exist.

Activity

  1. David-Noble-at-work commented on Mar 18, 2026

    @David-Noble-at-work
    ContributorAuthor

    Resolved — Config precedence implemented: config file → env vars → native keystore → CLI flags. Covered by TestLoad_EnvOverridesFile and TestLoad_WithEnvOverrides in internal/config/config_test.go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions