Skip to content

eval() rejects a percent-encoded file:// approval policy URI that read_approval_policies() accepts #5258

Description

@ebarkhordar

Passing an approval policy as a file:// URI works through read_approval_policies() but is rejected by eval(), Task() and inspect eval --approval whenever the path percent-encodes something. Path.as_uri() encodes spaces, so any policy file living under a directory with a space in its name runs into this.

Repro:

from pathlib import Path
from inspect_ai import Task, eval
from inspect_ai.approval import read_approval_policies
from inspect_ai.dataset import Sample

d = Path("/tmp/my policies"); d.mkdir(exist_ok=True)
f = d / "approve.yaml"
f.write_text('approvers:\n  - name: auto\n    tools: "*"\n    decision: approve\n')
uri = f.as_uri()   # file:///tmp/my%20policies/approve.yaml

read_approval_policies(uri)   # fine, 1 policy
eval(Task(dataset=[Sample(input="hello")]), model="mockllm/model", approval=uri)

Actual: the eval call raises ValueError: Invalid approval policy: file:///tmp/my%20policies/approve.yaml. Task(approval=uri) and inspect eval task.py --approval "file:///tmp/my%20policies/approve.yaml" give the same error, and read_approval_policies on that same URI returns the policy.

Expected: all of them take the URI. Move the file to a path with no space and they all do, so it looks specific to the encoding rather than to file:// itself.

One bit of context in case it helps: #4579 listed the eval and task config path next to read_approval_policies, and #5027 closed it, so this may be the half that got missed.

Environment: 58e9b08 (0.3.264.dev9+g58e9b08e7), Python 3.12, Debian slim container.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    acceptedIssues: maintainer-approved for external PR — the gate key

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions