Repository navigation
Harden Rack::Attack and pagination against aggressive web crawlers - #4772
Conversation
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
@cesy what do you reckon with this? Production logs are all useless crawlers galore (semrush and clones), and a fair few end up on abuse lists. |
- Configure Allow2Ban in Rack::Attack to ban IPs requesting >500 pages per day for 1 week (7 days). - Add honeypot route /dont-crawl-me disallowed in robots.txt and configure Fail2Ban in Rack::Attack to ban IPs visiting it for 7 days upon 1 hit. - Update PhotosController#index to raise ActiveRecord::RecordNotFound when page parameter is out of bounds, returning 404 Not Found to crawlers instead of 200 OK. - Add request and controller specs for Rack::Attack rules and pagination 404 responses. Co-authored-by: CloCkWeRX <[email protected]>
Remove error handling from index_everything method.
19219b3 to
2c0521e
Compare
|
LGTM... i can see bots killing off the app (just like all the other apps on the internet these days, damn robots) |
|
Good idea, thank you |
3 participants
This change hardens the Rack::Attack configuration against abusive web crawlers:
Allow2Baninconfig/initializers/rack_attack.rbto ban IPs that request more than 500 pages in a 24-hour period for 1 week (7 days)./dont-crawl-meinpublic/robots.txtandconfig/robots.txt, created the GET route inconfig/routes.rb, and configuredFail2BaninRack::Attackto ban any IP visiting/dont-crawl-mefor 7 days upon 1 hit.PhotosController#indexto raiseActiveRecord::RecordNotFoundwhen@photos.out_of_bounds?(e.g. requestingpage=105when fewer pages exist), returning HTTP 404 Not Found to stop crawlers from endlessly probing invalid page numbers.spec/requests/rack_attack_spec.rband updatedspec/controllers/photos_controller_spec.rbto verify honeypot banning, rate limit banning, and 404 pagination responses.PR created automatically by Jules for task 2674394950484400196 started by @CloCkWeRX