This shows you the differences between two versions of the page.
Next revisionBoth sides next revision | |||
fuss:security [2018/08/31 15:20] – created office | fuss:security [2018/08/31 16:17] – [Blocking SemrushBot] office | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Blocking SemrushBot ====== | ====== Blocking SemrushBot ====== | ||
- | SemrushBot is an annoying web crawler that has proven to completely disregard the robots policies as well as hammering webservers hard by recursively following all the links on a website without delay. | + | SemrushBot is an annoying web crawler that has proven to completely disregard the robots policies as well as hammering webservers hard by recursively following all the links on a website without delay and outright ignoring any repeating '' |
+ | |||
+ | {{: | ||
+ | ===== IP Layer ===== | ||
On the IP layer: | On the IP layer: | ||
Line 8: | Line 11: | ||
</ | </ | ||
- | Which is a great solution to get rid of this pest without even hitting the application layer! | + | Which is an awful solution to get rid of this pest without even hitting the application layer! |
+ | |||
+ | ===== Apache2 ===== | ||
If are okay with your frontend being hammered by this total garbage, then the '' | If are okay with your frontend being hammered by this total garbage, then the '' | ||
Line 32: | Line 37: | ||
which is a bad solution because '' | which is a bad solution because '' | ||
+ | |||
+ | ===== Varnish ===== | ||
Perhaps blocking with Varnish may be a good compromise between having your Apache2 hammered and blocking the string '' | Perhaps blocking with Varnish may be a good compromise between having your Apache2 hammered and blocking the string '' | ||
Line 44: | Line 51: | ||
} | } | ||
+ | </ | ||
+ | |||
+ | An even better method would be to use fail2ban to block '' | ||
+ | |||
+ | ===== Varnish and Fail2Ban ===== | ||
+ | |||
+ | For Varnish, copy ''/ | ||
+ | < | ||
+ | badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/ | ||
+ | </ | ||
+ | |||
+ | then correct the '' | ||
+ | < | ||
+ | failregex = ^< | ||
+ | </ | ||
+ | |||
+ | |||
+ | and finally add the following to the jail configuration: | ||
+ | < | ||
+ | [varnish-badbots] | ||
+ | enabled | ||
+ | port = http,https | ||
+ | filter | ||
+ | logpath | ||
+ | maxretry = 1 | ||
+ | </ | ||
+ | |||
+ | and restart '' | ||
+ | |||
+ | To check that the bots are being banned, tail ''/ | ||
+ | < | ||
+ | fail2ban.jail[18168]: | ||
+ | </ | ||
+ | indicating that the '' | ||
+ | |||
+ | Hopefully followed by lines similar to: | ||
+ | < | ||
+ | NOTICE [varnish-badbots] Ban 46.229.168.68 | ||
</ | </ | ||