This shows you the differences between two versions of the page.
Next revision | Previous revisionNext revisionBoth sides next revision | ||
fuss:security [2018/08/31 15:20] – created office | fuss:security [2022/04/19 08:28] – external edit 127.0.0.1 | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Blocking SemrushBot ====== | ====== Blocking SemrushBot ====== | ||
- | SemrushBot is an annoying web crawler that has proven to completely disregard the robots policies as well as hammering webservers hard by recursively following all the links on a website without delay. | + | SemrushBot is an annoying web crawler that has proven to completely disregard the robots policies as well as hammering webservers hard by recursively following all the links on a website without delay and outright ignoring any repeating '' |
+ | |||
+ | {{: | ||
+ | ===== IP Layer ===== | ||
On the IP layer: | On the IP layer: | ||
Line 8: | Line 11: | ||
</ | </ | ||
- | Which is a great solution to get rid of this pest without even hitting the application layer! | + | Which is an awful solution to get rid of this pest without even hitting the application layer! |
+ | |||
+ | ===== Apache2 ===== | ||
If are okay with your frontend being hammered by this total garbage, then the '' | If are okay with your frontend being hammered by this total garbage, then the '' | ||
Line 32: | Line 37: | ||
which is a bad solution because '' | which is a bad solution because '' | ||
+ | |||
+ | ===== Varnish ===== | ||
Perhaps blocking with Varnish may be a good compromise between having your Apache2 hammered and blocking the string '' | Perhaps blocking with Varnish may be a good compromise between having your Apache2 hammered and blocking the string '' | ||
Line 44: | Line 51: | ||
} | } | ||
+ | </ | ||
+ | |||
+ | An even better method would be to use fail2ban to block '' | ||
+ | |||
+ | ===== Varnish and Fail2Ban ===== | ||
+ | |||
+ | For Varnish, copy ''/ | ||
+ | < | ||
+ | badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/ | ||
+ | </ | ||
+ | |||
+ | then correct the '' | ||
+ | < | ||
+ | failregex = ^< | ||
+ | </ | ||
+ | |||
+ | |||
+ | and finally add the following to the jail configuration: | ||
+ | < | ||
+ | [varnish-badbots] | ||
+ | enabled | ||
+ | port = http,https | ||
+ | filter | ||
+ | logpath | ||
+ | maxretry = 1 | ||
+ | </ | ||
+ | |||
+ | and restart '' | ||
+ | |||
+ | To check that the bots are being banned, tail ''/ | ||
+ | < | ||
+ | fail2ban.jail[18168]: | ||
+ | </ | ||
+ | indicating that the '' | ||
+ | |||
+ | Hopefully followed by lines similar to: | ||
+ | < | ||
+ | NOTICE [varnish-badbots] Ban 46.229.168.68 | ||
+ | </ | ||
+ | |||
+ | ====== Intercept SSL 2.0 / SSL 3.0 Using SSLsniff ====== | ||
+ | |||
+ | Redirect SMTPs to custom port for SSLsniff: | ||
+ | <code bash> | ||
+ | iptables -t nat -A PREROUTING -p tcp --destination-port 995 -j REDIRECT --to-ports 4995 | ||
+ | sslsniff -a -c / | ||
+ | </ | ||
+ | |||
+ | Example Session: | ||
+ | < | ||
+ | 1385227016 INFO sslsniff : Added OCSP URL: ocsp.ipsca.com | ||
+ | 1385227016 INFO sslsniff : Certificate Ready: * | ||
+ | sslsniff 0.8 by Moxie Marlinspike running... | ||
+ | 1385227031 DEBUG sslsniff : Read from Server (mail.net.hu) : | ||
+ | +OK POP3 PROXY server ready < | ||
+ | |||
+ | 1385227032 DEBUG sslsniff : Read from Client (mail.net.hu) : | ||
+ | USER harry | ||
+ | |||
+ | 1385227032 DEBUG sslsniff : Read from Server (mail.net.hu) : | ||
+ | +OK Password required | ||
+ | |||
+ | 1385227032 DEBUG sslsniff : Read from Client (mail.net.hu) : | ||
+ | PASS secretpassword | ||
</ | </ | ||