Differences

This shows you the differences between two versions of the page.

--- fuss:apache [2017/02/22 18:30] – external edit 127.0.0.1
+++ fuss:apache [2025/05/22 15:11] (current) – [Creating Ban Lists for Apache] office
@@ Line 360: / Line 360: @@
 &\approx& 71
 \end{eqnarray*}
+====== Implementing a Global and Consistent Directory Index Style ======
+The following setup implements a global and consistent directory index style that can be used for multiple virtual hosts just by adding an include to any ''Directory'' stanza within a virtual host configuration.
+<ditaa>
++ /
+|
++---+ /etc/apache2/
+|         +
+|         |
+|         +---+ conf-available/
+|         |           +
+|         |           |
+|         |           +---+ fancyindex.conf
+|         |
+|         +---+ includes/
+|         |         +
+|         |         |
+|         |         +---+ fancyindex.conf
+|         |
+|         +---+ sites-available/
+|                    +
+|                    |
+|                    +---+ vhost.conf
+|
++---+ /var/www/
+         +
+         |
+         +---+ .fancyindex.css
+</ditaa>
+where:
+  * ''/etc/apache2/conf-available/fancyindex.conf'' contains:
+<code>
+Alias "/fancyindex.css" "/var/www/.fancyindex.css
+</code>
+  * ''/etc/apache2/includes/fancyindex.conf'' contains:
+<code>
+IndexOptions +Charset=UTF-8
+IndexOptions +TrackModified
+IndexOptions +Charset=UTF-8
+IndexOptions +FoldersFirst
+IndexOptions +NameWidth=*
+IndexOptions +FancyIndexing
+IndexOptions +HTMLTable
+IndexOptions +SuppressDescription
+IndexIgnore favicon.ico
+IndexIgnore auth*
+IndexIgnore include*
+IndexIgnore css*
+IndexIgnore share*
+IndexIgnore upload*
+IndexIgnore incoming*
+IndexStyleSheet "/fancyindex.css"
+Options +Indexes
+</code>
+  * ''/etc/apache2/sites-available/vhost.conf'' is a virtual host configuration where the indexing can be turned on within any ''Directory'' stanza just by including ''/etc/apache2/includes/fancyindex.conf'', ie enabling directory indexing for the directory named ''path'' relative to the root of the virtual host:
+<code>
+    <Directory /path>
+        include "includes/fancyindex.conf"
+    </Directory>
+</code>
+  * ''/var/www/.fancyindex.css'' contains the following:
+<code>
+* {
+    font-family: monospace;
+}
+</code>
+and is responsible for setting the CSS for the rendered index page.
+<WRAP info>
+''IndexStyleSheet'' takes an URL relative to the virtual host document root such that the aliasing performed within ''/etc/apache2/conf-available/fancyindex.conf'' redirects the requests to the default virtual host where the ''/var/www/.fancyindex.css'' file is placed. It ain't pretty, but it works; at least short of changing ''IndexStyleSheet'' itself!
+</WRAP>
+Perhaps a good reason for preferring this setup to using ''AllowOverride'' and ''.htaccess'' files is that the style of the directory index is generated by Apache itself rather than the website such that one can avoid mixing data with code. Furthermore, given multiple virtual hosts with the same owner, a consistent style may be preferred and with a centralized way of batch-changing all directory indexes.
+====== Globally Enable brotli Compression ======
+On Debian, to enable brotli compression, the Apache module must first be installed by issuing the command:
+<code bash>
+apt-get install brotli
+</code>
+and then the Apache module must be enabled by issuing the command:
+<code bash>
+a2enmod brotli
+</code>
+Lastly, create the file ''/etc/apache2/conf-available/brotli.conf'' with the following contents:
+<code>
+<IfModule mod_brotli.c>
+    AddOutputFilterByType BROTLI_COMPRESS text/html text/plain text/xml text/css text/javascript application/javascript
+</IfModule>
+</code>
+The purpose of this file is to instruct the brotli Apache module to compress certain resources, namely textual content as well as javascript with the brotli compressor.
+Finally, enable the configuration file by issuing the command:
+<code bash>
+a2enconf brotli
+</code>
+and reload the Apache configuration:
+<code bash>
+systemctl apache2 reload
+</code>
+The brotli compressor should now be enabled and will globally compress content as it is transferred from the Apache server.
+Note that an alternative configuration is to add the contents of the file ''/etc/apache2/conf-available/brotli.conf'' to a virtual host definition in order to only compress some websites.
+====== Rate Limiting an IP Network with mod_QoS ======
+The process of rate-limiting an IP network with Apache mod_qos involves matching an IP address block using the Apache2 ''SetEnvIfExpr'' expression:
+<code>
+SetEnvIfExpr "-R '47.128.0.0/16'" Amazon=yes
+</code>
+where:
+  * ''"-R '47.128.0.0/16'"'' is a network block match and,
+  * ''Amazon=yes'' is a variable ''Amazon'' being set to ''yes''
+followed by using mod_qos to rate-limit:
+<code>
+QS_EventLimitCount Amazon 10 60
+</code>
+where:
+  * ''10'' represents the number of connections allowed per ''60'' seconds.
+====== Creating Ban Lists for Apache ======
+People conventionally block IP addresses on the networking layer instead of the application layer with Apache and for good reasons due to IP being a property of networking and not something that would be specific to Apache. However, given cloud services Apache might end up downstream with a real IP address passed via an HTTP header such that the banning is only possible after the HTTP session is decapsulated by a reverse proxy or an HTTP server like Apache.
+In any case, it makes somewhat sense to block IP addresses at he application layer in that context. Previously we have written scripts that would pull databases, such as [[/fuss/iptables#peerblock_level_1|the PeerBlock Level 1]] or [[/fuss/iptables#block_attacks_with_emerging_threats|the "emerging threats" lists]] and then generated ipsets to be used with a firewall such that the same scripts could be modified slightly to generate Apache blocklists.
+<code bash>
+#!/usr/bin/env bash
+###########################################################################
+##  Copyright (C) Wizardry and Steamworks 2025 - License: GNU GPLv3      ##
+###########################################################################
+# This script is used to generate an universal blocklsit for Apache in    #
+# order to block various IP addresses or networks in CIDR notation by     #
+# pulling lists from various sources on the Internet.                     #
+###########################################################################
+APACHE_CONFIGURATION_FILE=/etc/apache2/conf-available/bans.conf
+PEERBLOCK=$(curl -s -L "http://list.iblocklist.com/?list=bt_level1&amp;fileformat=p2p&amp;archiveformat=gz" -o - | \
+    gunzip | \
+    cut -d: -f2 | \
+    grep -E "^[-0-9.]+$" | \
+    awk '{print $1}' | \
+    xargs ipcalc -rn | grep -v deaggregate)
+AWS=$(curl -s https://ip-ranges.amazonaws.com/ip-ranges.json -o - | \
+    jq '.prefixes[] | .ip_prefix' | \
+    grep -P -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\/[0-9]{0,2}')
+# preamble
+cat >"$APACHE_CONFIGURATION_FILE" <<EOF
+<Location />
+    # Amazon AWS Blocking
+    <RequireAll>
+        Require all granted
+EOF
+# body
+printf "        # PEERBLOCK\n" >> "$APACHE_CONFIGURATION_FILE"
+while IFS=$'\n' read PEER; do
+    printf "        Require not ip $PEER\n" >>"$APACHE_CONFIGURATION_FILE"
+done <<< "$PEERBLOCK"
+printf "        # AMAZON\n" >> "$APACHE_CONFIGURATION_FILE"
+while IFS=$'\n' read PEER; do
+    printf "        Require not ip $PEER\n" >>"$APACHE_CONFIGURATION_FILE"
+done <<< "$AWS"
+# postamble
+cat >>"$APACHE_CONFIGURATION_FILE" <<EOF
+    </RequireAll>
+</Location>
+EOF
+</code>
+The script above is a variant of the ipset-generating scripts that is meant to create a single file that is placed within the ''config-available'' folder on a Debian-based distribution. The file can then be made active by creating a symlink within the ''config-enabled'' folder in order to block requests from the file generated by the script.
+On the other hand, the script itself is placed within the daily crontab in order to regenerate the lists on a daily basis.
+For the curious, the script above blocks Amazon AWS, because it has become very cheap hosting for harvesters and various other indexers (more than likely feeding "A.I." Eliza machines materials) such that blocking these indexers seems fairly beneficial. This is more or less the same as the Google mail system being used by spammers due to being sometimes outright whitelisted by system and E-Mail system administrators.
+====== Max Open Files ======
+One error that might appear out of the blue when dealing with Apache is the "max open files" error, perhaps followed by an Apache error ''AH00529'' that is picked up from the errors log file when accessing a website. It is a tough error to track down because regardless of the ''ulimit'' settings that can be made, it just seems that Apache runs out of file descriptors for no reason and even unlikely given limits that can be raised.
+Furthermore, checking the limits per process for Apache processes by issuing ''cat /proc/APACHE_PID/limits'' where ''APACHE_PID'' is the PID of a running Apache process, seems to reveal some magic number ''8192'' for the "Max open files" limit that cannot be traced to anything.
+The issue stems from ''/usr/sbin/apachectl'', a script that is meant to manipulate Apache that contains the following lines:
+<code bash>
+ULIMIT_MAX_FILES="${APACHE_ULIMIT_MAX_FILES:-ulimit -n 8192}"
+if [ "x$ULIMIT_MAX_FILES" != "x" ] && [ `id -u` -eq 0 ] ; then
+    if ! $ULIMIT_MAX_FILES ; then
+</code>
+These lines make it such that the command defined by the variable ''APACHE_ULIMIT_MAX_FILES'' is ran and if the variable ''APACHE_ULIMIT_MAX_FILES'' is empty, then the command ''ulimit -n 8192'' is taken as the default.
+The issue is mostly encountered under containerization, such as running Apache under Docker using a Debian image, when multiple operating system layers are involved, such that the OS is not able to set variables properly. In other to resolve the issue, simply set the ''APACHE_ULIMIT_MAX_FILES'' global environment variable to some command that will set the ''ulimit'' properly. For example:
+<code bash>
+APACHE_ULIMIT_MAX_FILES='ulimit -S -n `ulimit -H -n'
+</code>
+or set a limit, as in:
+<code bash>
+APACHE_ULIMIT_MAX_FILES='ulimit -n 65536'
+</code>
+Note that the variable ''APACHE_ULIMIT_MAX_FILES'' has to contain a command to set the limit, not the actual number of maximum files that can be opened.