Vector Database: Semantic ability selection + NLP test framework #3

Goal

Implement client-side vector database for semantic ability selection, with automated testing framework for NLP input validation.

Part 1: Vector Database Implementation

Architecture

• Embeddings model: all-MiniLM-L6-v2 via transformers.js (~80MB)
• Vector store: hnswlib-wasm OR simple cosine similarity (if only 12 abilities)
• Flow:
  1. User input → embedding
  2. Similarity search → top 3-4 relevant abilities
  3. Pass only those to ReAct loop (not all 12)

Embedding Data

For each ability, embed:

• Name + description
• Example trigger phrases
• Keywords
• Expected outcomes

Benefits

• Reduced context: Small model sees 3-4 abilities instead of 12
• Better accuracy: More relevant tool suggestions
• Privacy-first: All embeddings computed client-side
• Semantic matching: "site is slow" matches cache-flush + db-optimize, not error-log-read

Part 2: NLP Testing Framework

Goal

Unit-test-style validation for vector DB quality. Essential for hackathon team iteration.

Test Structure

const nlpTests = [
  {
    input: "my site is slow",
    expectedAbilities: ["cache-flush", "db-optimize", "site-health"],
    notExpected: ["error-log-read", "plugin-activate"]
  },
  {
    input: "500 error on homepage",
    expectedAbilities: ["error-log-read", "plugin-list", "site-health"],
    notExpected: ["cache-flush", "revision-cleanup"]
  },
  {
    input: "need to clean up database",
    expectedAbilities: ["db-optimize", "revision-cleanup", "transient-flush"],
    notExpected: ["plugin-activate", "rewrite-flush"]
  }
];

Test Runner

• Run similarity search for each test input
• Verify top-K results match expected abilities
• Flag mismatches
• Output: Pass/Fail with similarity scores
• CI-friendly (can run in GitHub Actions)

Iteration Workflow (for hackathon team)

1. Tester adds new NLP input example
2. Runs test suite
3. If fails, adjusts ability embeddings (description, keywords)
4. Re-runs tests
5. Repeat until all pass

Test Output Example

✅ "my site is slow" → cache-flush (0.89), db-optimize (0.85), site-health (0.78)
❌ "500 error" → Expected error-log-read (0.92) ✅, plugin-list (0.65) ✅, but got cache-flush (0.71) ❌

Implementation Files

src/
├── services/
│   ├── embeddings.js          # transformers.js wrapper
│   ├── vector-store.js         # similarity search
│   └── ability-matcher.js      # top-K retrieval
└── __tests__/
    ├── nlp-ability-matching.test.js  # main test suite
    └── fixtures/
        └── test-cases.json           # test inputs + expectations

Success Criteria

• Embeddings model loads in browser
• Similarity search returns top-K abilities
• Test framework validates NLP → ability mappings
• At least 20 test cases covering common SRE scenarios
• >90% test pass rate before hackathon

Priority

High - critical for vector DB quality validation

Deadline

Target: March 8 (1 week before CloudFest for team iteration)

Resources

• transformers.js
• hnswlib-wasm
• Audio explanation: /tmp/vector-db-explanation.mp3 (sent to Marcel on Feb 6)