Skip to content

fix: Agent and sync bugs - #259

Merged
alex289 merged 1 commit into
mainfrom
fix/agent_not_reporting
Jul 22, 2026
Merged

alex289 merged 1 commit into
mainfrom
fix/agent_not_reporting

Conversation

@alex289

@alex289 alex289 commented Jul 22, 2026

Copy link
Copy Markdown
Member

Type of change

  • 🐛 Bug fix
  • 🚀 New feature
  • ❓ Other (please specify)

Description

Fixes applications getting permanently stuck on "Syncing"/"Unknown" status and history filling up with spurious "Failed" (hub restarted before the operation result was received) entries, caused by duplicate webhook deliveries (e.g. GitHub retrying a package event for GHCR pushes).

  • Image pull webhook: the agent silently dropped its response to the hub whenever an image pull found nothing new to pull — even for explicitly triggered requests (webhook/manual/GitHub Actions), not just background polling ticks. When GitHub redelivers a webhook, the second delivery finds the image already pulled by the first and never reports back. The corresponding application_event then sits in running forever (only cleared on the next hub restart, which is where the "hub restarted..." failures come from), and the application's sync_status never gets reset out of Syncing.
  • Commit-sync / deployment webhooks: the same class of duplicate-delivery problem existed here too, plus a more serious gap — there was no locking at all preventing two overlapping deploys for the same application from running concurrently on the agent (docker compose up racing against itself for the same project directory).

Notes for reviewers

  • No commit/deploy correctness change is required beyond serialization — the deploy path already always reports a result back to the hub (unlike the image-poll path before this fix), so it doesn't get permanently "stuck"; it was purely a concurrency/race risk.
  • The repository-level sync lock is released once the sync call returns (i.e. once jobs are enqueued), not once the deploy actually finishes. In practice this is fine because the agent-side lock added here serializes the actual work — a residual reordering (rather than corruption) risk remains for genuinely back-to-back distinct triggers, which is an acceptable tradeoff over the added complexity of holding the lock across async queue processing.

@alex289 alex289 self-assigned this Jul 22, 2026
@alex289 alex289 added the bug Something isn't working label Jul 22, 2026
@alex289
alex289 requested a review from timokoessler July 22, 2026 16:18

@timokoessler timokoessler left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. Do we still need additional deduplication for incoming webhooks in a short timeframe?

Comment thread backend/internal/agent/websocket.go
@alex289

alex289 commented Jul 22, 2026

Copy link
Copy Markdown
Member Author

Cool. Do we still need additional deduplication for incoming webhooks in a short timeframe?

This should not be a problem I think. Even if both jobs end up in the queue the new agent lock serializes the docker compoes executions. So in worst case it will just be delayed

@alex289
alex289 merged commit 1f6ea97 into main Jul 22, 2026
10 checks passed
@alex289
alex289 deleted the fix/agent_not_reporting branch July 22, 2026 18:46
@orcacd-bot orcacd-bot Bot mentioned this pull request Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent backend bug Something isn't working hub

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants