Crawly

Crawly is a Flask-based issue detection and autofix dashboard. It scans repositories, creates issues, generates code fixes, and can connect to GitHub or GitLab workflows.

• Scans repositories for code issues
• Creates and tracks issues in the dashboard
• Generates fixes with the configured AI provider
• Supports GitHub OAuth and optional GitHub App bot identity
• Supports GitLab repositories and webhook-driven processing
• Runs background work with Celery or the built-in scheduler paths

• Python 3.11+ recommended
• Conda environment named crawly or another Python environment with the project dependencies installed
• PostgreSQL
• Redis for Celery/background tasks
• Ollama if you use the default local AI provider
• GitHub OAuth app credentials
• Optional: GitHub App credentials for bot-created issues and PRs
• Optional: GitLab OAuth/app credentials if you use GitLab

1. Create and activate the Python environment.

conda activate crawly

2. Install dependencies.

python -m pip install -r requirements.txt

3. Copy .env.example to .env and fill in the values you need.

4. Make sure PostgreSQL, Redis, and Ollama are running if you use those features.

Start with .env.example. The template contains placeholders only and no credentials.

Common values you usually need to set in .env:

• GITHUB_CLIENT_ID
• GITHUB_CLIENT_SECRET
• FLASK_SECRET_KEY
• DATABASE_URL
• REDIS_URL
• AI_PROVIDER
• OLLAMA_API_URL and OLLAMA_MODEL if you use Ollama

Optional values:

• GITHUB_APP_ID
• GITHUB_APP_PRIVATE_KEY_PATH
• GITHUB_APP_INSTALLATION_ID
• GITLAB_CLIENT_ID
• GITLAB_CLIENT_SECRET
• GITLAB_URL
• GITLAB_WEBHOOK_SECRET

Start the dashboard:

conda activate crawly
python run.py

The app defaults to:

• Dashboard: http://127.0.0.1:5000

The startup script will:

• Load environment variables from .env
• Check Ollama if AI_PROVIDER=ollama
• Start background schedulers
• Launch the Flask-SocketIO dashboard

If you use Celery, start the worker in a second terminal:

conda activate crawly
python run_worker.py

On Windows, the worker uses the solo pool so it can run without extra multiprocessing setup.

1. Log in through GitHub OAuth.
2. Add one or more repositories in the dashboard.
3. Run a crawl or let the scheduled tasks process repositories.
4. Open the Issues page to review generated issues.
5. Use the issue actions to:
   • Create Fix
   • Generate Fix
   • Close Issue
   • Delete Issue

The Issues page shows created issues from your connected repositories. Depending on the issue metadata, the page can show:

• Create Fix to request a code-fix record and generate the fix
• Generate Fix to run the on-demand fix flow when file and line context exists
• Close to close the issue on the remote platform
• Delete to remove the issue on the remote platform

1. Go to GitHub Developer Settings.
2. Create an OAuth app.
3. Set the callback/redirect URL used by your deployment.
4. Place the client ID and secret in .env.

If you want Crawly to act as a bot for issue and PR actions:

1. Create a GitHub App.
2. Generate a private key.
3. Install the app on the target repositories.
4. Set these values in .env:
   • GITHUB_APP_ID
   • GITHUB_APP_PRIVATE_KEY_PATH
   • GITHUB_APP_INSTALLATION_ID

If these are blank, Crawly falls back to user OAuth behavior.

For GitLab repositories, set the GitLab OAuth/webhook values in .env and make sure the repositories are linked in the dashboard.

The default AI provider is Ollama.

Example local settings:

AI_PROVIDER=ollama
OLLAMA_API_URL=http://127.0.0.1:11434
OLLAMA_MODEL=llama3.2
AI_MODEL=llama3.2

• If the dashboard does not start, verify .env values and database connectivity.
• If Create Fix does nothing, check the server logs and confirm the repository has file and line context for the issue.
• If Ollama is unavailable, start it with ollama serve or switch AI_PROVIDER.
• If background tasks are not running, confirm Redis and run_worker.py are active.

• run.py is the dashboard entry point.
• run_worker.py is the Celery worker entry point.
• .env.example intentionally contains no secrets so it is safe to commit.