Problem
When a user asks for a specific piece of site info, the LLM returns all fields instead of filtering to the relevant one.
- "core/get-site-info siteurl" → returns site name, tagline, URL, version, language, admin email
- "what is my address URL" → same full dump
The user expects just the URL.
Reported by @wpbullet in #79.
What works
Root cause
The LLM receives the full tool result and should summarize it contextually based on the user's question. With the 1.7B model's limited reasoning capacity and tight 4096-token context, it tends to dump everything rather than filtering to what was asked.
Expected behavior
The LLM should answer contextually:
- "what is my site URL?" → "Your site URL is https://example.com"
- "what is the site name?" → "Your site name is 'My Blog'"
- "tell me about my site" → full info dump (appropriate here)
Possible approaches
-
LLM summarization (preferred): This should work naturally with better models or when using external providers. The current interpretResult already provides a concise string — but the LLM still receives the raw JSON and chooses to echo all of it. May improve with prompt tuning or model upgrades.
-
JS-side pre-filtering: The execute function could detect keywords in the user message (e.g., "url", "name", "version") and strip the result down to just the matching field before the LLM sees it.
-
Split into sub-abilities: Separate core/get-site-url, core/get-site-name, etc. — but this adds tool bloat for the context window.
Approach 1 is the right long-term solution. Approach 2 is a pragmatic workaround for the 1.7B model.
Refs
Refs #79 (selection part fixed in PR #102)
Problem
When a user asks for a specific piece of site info, the LLM returns all fields instead of filtering to the relevant one.
The user expects just the URL.
Reported by @wpbullet in #79.
What works
core/get-site-info(verified in PR fix: wpbullet ability selection regressions + 23 new test cases #102)Root cause
The LLM receives the full tool result and should summarize it contextually based on the user's question. With the 1.7B model's limited reasoning capacity and tight 4096-token context, it tends to dump everything rather than filtering to what was asked.
Expected behavior
The LLM should answer contextually:
Possible approaches
LLM summarization (preferred): This should work naturally with better models or when using external providers. The current
interpretResultalready provides a concise string — but the LLM still receives the raw JSON and chooses to echo all of it. May improve with prompt tuning or model upgrades.JS-side pre-filtering: The
executefunction could detect keywords in the user message (e.g., "url", "name", "version") and strip the result down to just the matching field before the LLM sees it.Split into sub-abilities: Separate
core/get-site-url,core/get-site-name, etc. — but this adds tool bloat for the context window.Approach 1 is the right long-term solution. Approach 2 is a pragmatic workaround for the 1.7B model.
Refs
Refs #79 (selection part fixed in PR #102)