Skip to content

core/get-site-info: LLM should filter response to match user's specific question #109

Description

@pluginslab

Problem

When a user asks for a specific piece of site info, the LLM returns all fields instead of filtering to the relevant one.

  • "core/get-site-info siteurl" → returns site name, tagline, URL, version, language, admin email
  • "what is my address URL" → same full dump

The user expects just the URL.

Reported by @wpbullet in #79.

What works

Root cause

The LLM receives the full tool result and should summarize it contextually based on the user's question. With the 1.7B model's limited reasoning capacity and tight 4096-token context, it tends to dump everything rather than filtering to what was asked.

Expected behavior

The LLM should answer contextually:

  • "what is my site URL?" → "Your site URL is https://example.com"
  • "what is the site name?" → "Your site name is 'My Blog'"
  • "tell me about my site" → full info dump (appropriate here)

Possible approaches

  1. LLM summarization (preferred): This should work naturally with better models or when using external providers. The current interpretResult already provides a concise string — but the LLM still receives the raw JSON and chooses to echo all of it. May improve with prompt tuning or model upgrades.

  2. JS-side pre-filtering: The execute function could detect keywords in the user message (e.g., "url", "name", "version") and strip the result down to just the matching field before the LLM sees it.

  3. Split into sub-abilities: Separate core/get-site-url, core/get-site-name, etc. — but this adds tool bloat for the context window.

Approach 1 is the right long-term solution. Approach 2 is a pragmatic workaround for the 1.7B model.

Refs

Refs #79 (selection part fixed in PR #102)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions