This tutorial is about adding spam filtering capabilities to postfix using the following packages:

• ClamAV
• AMaViS
• SpamAssassin

with additional leads to:

• postscreen, a feature available in Postfix versions 2.8 and higher,

Other enhancements can be found on their separate pages:

• Domain Keys Identified Mail (DKIM)
• Sender Policy Framework (SPF)
• Domain-based Message Authentication, Reporting and Conformance (DMARC)

The tutorial assumes that you have a Debian distribution up to date and that you have a working postfix install.

To begin, install amavis, clamav and spamassassin:

aptitude install clamav clamav-daemon amavisd-new spamassassin 

additionally, install the following packages for better spam detection:

aptitude install libnet-dns-perl libmail-spf-perl pyzor razor

In order to better scan inside compressed archives, install the following packages:

aptitude install arj bzip2 cabextract cpio file gzip lhasa nomarch pax rar unrar-free unzip p7zip-full zip zoo poppler-utils fuzzyocr tesseract-ocr-all

To activate spam and antivirus detection edit /etc/amavis/conf.d/15-content_filter_mode and enable the lines:

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);

and:

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

by uncommenting them.

If this is a front-end mail server, consider editing /etc/amavis/conf.d/20-debian_defaults and setting:

$final_banned_destiny     = D_BOUNCE;

to

$final_banned_destiny     = D_REJECT;

in order to avoid e-mails such as: Considered UNSOLICITED BULK EMAIL, apparently from you being generated once an address is banned.

Having installed poppler-utils, fuzzyocr and tesseract-ocr-all, amavis can be configured to scan inside email attachments looking for spam content. Edit /etc/spamassassin/FuzzyOcr.cf and enable the line by removing the hash sign in front (#):

focr_bin_helper pdfinfo, pdftops, pstopnm

in order to enable scanning within PDF files.

Optionally, fuzzyocr can hash images to speed up the spam recognition process. In order to enable hashing, set:

focr_enable_image_hashing 2

Note that fuzzyocr depends on gifinter to scan GIF image files and that gifinter has been deprecated in favour of imagemagick and the convert tool. To work around the issue, install xmlto required for generating documentation with:

aptitude install xmlto

and then download the giflib 5.0.0 source package. Unpack the package and issue:

./configure

followed by:

make

to compile the tools.

Then, copy the gifinter utility to /usr/local/bin:

cp util/gifinter /usr/local/bin

This will place the gifinter tool under the default /usr/local/ prefix that will supersede the distribution giflib-tools package with a default prefix of /usr/bin.

In order to check the setup, issue:

for i in /usr/share/doc/fuzzyocr/examples/*; do spamassassin --debug < $i 2&> fuzzyocr-check.txt; done

to perform all the fuzzyocr example testing. The command will log some output to a file named fuzzyocr-check.txt in the local directory that can then be checked for errors.

Edit /etc/postfix/main.cf and add the lines:

content_filter = smtp-amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings

after that, edit /etc/postfix/master.cf and add the lines:

smtp-amavis     unix    -       -       -       -       2       smtp
   -o smtp_data_done_timeout=1200
   -o smtp_send_xforward_command=yes
   -o disable_dns_lookups=yes
   -o max_use=20

127.0.0.1:10025 inet    n       -       -       -       -       smtpd
   -o content_filter=
   -o local_recipient_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_delay_reject=no
   -o smtpd_client_restrictions=permit_mynetworks,reject
   -o smtpd_helo_restrictions=
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o smtpd_data_restrictions=reject_unauth_pipelining
   -o smtpd_end_of_data_restrictions=
   -o mynetworks=127.0.0.0/8
   -o smtpd_error_sleep_time=0
   -o smtpd_soft_error_limit=1001
   -o smtpd_hard_error_limit=1000
   -o smtpd_client_connection_count_limit=0
   -o smtpd_client_connection_rate_limit=0
   -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

and add the following two lines immediately below the pickup transport service:

   -o content_filter=
   -o receive_override_options=no_header_body_checks

which should prevent report messages on spam to be classified as spam.

SpamAssassin (and amavis) can be trained to better recognise spam as well as use some community databases such as Razor and PyRazon in order to better recognise spam.

Amavis uses the SpamAssassin libraries such that configuring or starting SpamAssassin is not necessary. Instead we can enable razor and pyrazor as well as train amavis to detect spam.

To enable the use of razor and pyrazor issue:

su - amavis -s /bin/bash
razor-admin -create
razor-admin -register
pyzor discover

To train SpamAssassin, change directory to /usr/src/ and download some spam archives and unpack them in /usr/src. Then issue:

chown -R amavis:amavis spam

where spam is the unpacked directory.

You should end up with directories such as spam or spam_2. To train, you issue:

su amavis -c 'sa-learn --spam spam'

where spam is the unpacked directory.

Additionally, from the same URL, download some ham archives, unpack them, change ownership to amavis and then issue:

su amavis -c 'sa-learn --ham ham'

where ham is the unpacked directory.

Finally, run as root:

sa-update -D

to update the rules.

In order to test the setup issue:

# spamassassin -D -t < /usr/share/doc/spamassassin/examples/sample-spam.txt 2>&1 | tee sa.out

which will redirect the output to sa.out.

Edit /etc/default/spamassassin and set:

CRON=1

which will enable periodic updates of the spam database.

If you get the error on Debian:

wget: MIRRORED.BY: Permission denied
error: unable to refresh mirrors file for channel updates.spamassassin.org, using old file
wget: 1732432.tar.gz: Permission denied
wget: 1732432.tar.gz: Permission denied
wget: 1732432.tar.gz: Permission denied
wget: 1732432.tar.gz: Permission denied
channel: could not find working mirror, channel failed
sa-update failed for unknown reasons

whilst running the cron-job /etc/cron.daily/spamassassin, then execute the following command in order to fix permissions:

chown -R debian-spamd:debian-spamd /var/lib/spamassassin

Some SpamAssassin perl modules may not be installed and although SpamAssassin may work some checks may not be performed. To check which perl modules have not been installed that could be used by SpamAssassin, issue:

spamassassin -D --lint 2>&1 | grep -i failed

For example, Debian would need the extra perl modules installed:

aptitude install libgeo-ip-perl libencode-detect-perl libnet-patricia-perl libdbi-perl

and some of them may be lacking, for example the old module Digest::SHA1 which may be installed using CPAN by issuing as root:

aptitude install cpanminus
cpanm Digest::SHA1

AMaViS can be configured to bounce spam e-mails but that may not be desired if the user would like to check if the mails are indeed spam. This can be changed in /etc/amavis/conf.d/20-debian_defaults by setting:

$final_spam_destiny = D_PASS;

You can also set the spam threshold by modifying:

$sa_tag_level_deflt  = -999;

which sets the score very low.

Additionally, by editing /etc/amavis/conf.d/50-user and setting:

$hdrfrom_notify_sender = "postmaster\@domain.tld";

where domain.tld is a domain name, reports can be generated and sent to that e-mail.

Having set:

$final_spam_destiny = D_PASS;

the effect is that e-mails will be tagged with the X-Spam-Flag header to indicate that they are most likely spam and then sent to a mail delivery agent.

One can configure a mail delivery agent such as Dovecot to move the e-mails to the junk folder. In order to do this and assuming that you have configured sieve, a new sieve can be added by editing /etc/dovecot/conf.d/90-sieve.conf and enabling and configuring one of the sieve_after directives:

sieve_after = /etc/dovecot/sieve-after

After that create the directory /etc/dovecot/sieve-after and create a file, say spam.sieve containing the lines:

require "fileinto";
if exists "X-Spam-Flag" {
    if header :contains "X-Spam-Flag" "NO" {
    } else {
        fileinto "Junk";
        stop;
    }
}

where:

• Junk is the folder that you want Dovecot to send the spam to. With mbox configurations, this is usually INBOX.Junk.

The script needs to be compiled with sievec:

sievec /etc/dovecot/sieve-after/spam.sieve

The final step is to restart amavisd:

/etc/init.d/amavisd restart

and dovecot:

/etc/init.d/dovecot restart

In order to check that the spam is filed accordingly, the swaks command line tool can be installed that is able to deliver a mail (in this case, the body will contain an example spam signature from SpamAssassin):

swaks --to me@server.tld --server 127.0.0.1 --data /usr/share/doc/spamassassin/examples/sample-spam.txt

where:

• me@server.tld should be a valid e-mail on the server you are working that you can check
• 127.0.0.1 is the hostname or IP address of the server
• /usr/share/doc/spamassassin/examples/sample-spam.txt is a sample spam file provided by the SpamAssassin package.

Now that you have setup Amavis and Spamassassin, as well as bootstrapped the Bayes filter using the default training packages, and additionally redirect spam to user Junk folders, you can tell you users to move all messages that are spam to the Junk folder and automate the SPAM and HAM training.

The provided script should be executed as a cronjob on a daily or weekly basis (on Debian, you can create it at /etc/cron.daily/amavis-spamassassin-autolearn). The script will run through all the user mailboxes and look for Junk folder. When a Junk folder is found, the script invokes sa-learn in order to train the Bayes filter.

amavis-spamassassin-autolearn

#!/bin/sh
###########################################################################
##  Copyright (C) Wizardry and Steamworks 2017 - License: GNU GPLv3      ##
###########################################################################
# A script for Amavis/Spamassassin to train spam from "Junk" directories  #
# based on the spam filtering article from Wizardry and Steamworks at:    #
#   * http://grimore.org/networking/postfix/spam_filtering                #
# using the bayesian filter.                                              #
#                                                                         #
# Requirements:                                                           #
#   * Amavis                                                              #
#   * Spamassassin                                                        #
#                                                                         #
# Installation:                                                           #
#   - Ensure that Amavis/Spamassassin is set-up following the tutorial.   #                                  
#   - Make sure that Amavis/Spamassassin has been bootstraped.            #
#   - Run this script as a cronjob on a daily basis.                      #
#   - Instruct your users to drop spam e-mail in the Junk folder.         #
#                                                                         #
###########################################################################
 
###########################################################################
#                             CONFIGURATION                               #
###########################################################################
 
# Set this to the folder designated to hold spam E-Mails.
JUNK_FOLDER="Junk"
 
# Set this to the Amavis/Spamassassin database path.
SPAMASSASSIN_DATABASE="/var/lib/amavis/.spamassassin"
 
# The type of mailbox - assumed sdbox using dovecot.
MAILBOX_TYPE="sdbox"
 
# The path to user home directories.
HOME_DIRECTORIES="/home"
 
###########################################################################
#                                INTERNALS                                #
###########################################################################
 
# Train spam from junk folder.
for i in `find $HOME_DIRECTORIES/*/$MAILBOX_TYPE/ -type d -name "$JUNK_FOLDER"`; do
    HOME_FOLDER=`echo $i | awk -F"/" '{ print $3 }'`
    sa-learn --use-ignores \
        --dbpath "$SPAMASSASSIN_DATABASE" \
        -p "/home/$HOME_FOLDER/.spamassassin/user_prefs" \
        --spam "$i" > /dev/null 2>&1
done
 
# Sync databases
sa-learn --dbpath "$SPAMASSASSIN_DATABASE" --sync > /dev/null 2>&1
 
# This has to be done in case this script runs before bootstrapping.
chown -R amavis:amavis "$SPAMASSASSIN_DATABASE"
 
# Update spam assassin.
sa-update -D

Issue as root:

adduser amavis clamav
adduser clamav amavis

and then edit /etc/clamav/clamd.conf to uncomment and enable the line:

AllowSupplementaryGroups true

There are many signature files provided by various companies on the Internet and the clamav-unofficial-sigs project is an effort to automate downloading those signature files and making them usable by ClamAV.

To add the unofficial signatures to ClamAV, first clone the git repository somewhere on your server:

git clone https://github.com/extremeshok/clamav-unofficial-sigs

and then some files will have to be moved:

• Copy clamav-unofficial-sigs.sh to /usr/local/bin/
• Set 755 permissions on /usr/local/bin/clamav-unofficial-sigs.sh by issuing: chmod 755 /usr/local/bin/clamav-unofficial-sigs.sh.
• Make the directory /etc/clamav-unofficial-sigs/ by issuing mkdir -p /etc/clamav-unofficial-sigs/.
• Copy the contents of config/ into /etc/clamav-unofficial-sigs/
• Copy the contents of cron.d/ into /etc/cron.d/
• Copy the contents of logrotate.d/ into /etc/logrotate.d/
• Copy clamav-unofficial-sigs.8 into /usr/share/man/man8/
• Make the directory /var/log/clamav-unofficial-sigs/ by issuing mkdir -p /var/log/clamav-unofficial-sigs.
• Change directory to /etc/clamav-unofficial-sigs/ and create a link from the appropriate distribution configuration file os.your-distro.conf to os.conf, where your-distro is your distribution. For example, for Debian 8, you would issue: ln -sf os.debian8.conf os.conf inside the /etc/clamav-unofficial-sigs/ directory.
• Set your user config options in the configs /etc/clamav-unofficial-sigs/user.conf - you will have to uncomment user_configuration_complete="yes".

Instead of waiting for cron, you can start the update process now by issuing the command:

/usr/local/bin/clamav-unofficial-sigs.sh

which should download the unofficial sigs.

To test that the signatures are now loaded, issue:

clamscan --debug 2>&1 /dev/null | grep "loaded"

and you are set.

Depending on your environment, some "virus" signatures have to be ignored - for instance, the unofficial signature package contains a YARA rule that flags any office document with macros as a virus-infected file - regardless whether the macro is malicious or not.

After the configuration, the configured services have to be restarted, so we issue:

service clamav-daemon restart
service amavis restart
service postfix restart

Postscreen introduces some very small delays that are acceptable for properly configured clients but will throw off mass-mailers that do not wait for the entire conversation to take place before sending commands.

First, add to /etc/postfix/main.cf the following lines:

# Postcreen configuration
postscreen_access_list = permit_mynetworks
postscreen_dnsbl_threshold = 2
postscreen_dnsbl_sites = zen.spamhaus.org*2,
        bl.spamcop.net*1,
        b.barracudacentral.org*1
postscreen_greet_banner = Welcome!
postscreen_greet_action = enforce

postscreen_pipelining_enable = yes
postscreen_pipelining_action = enforce

postscreen_non_smtp_command_enable = yes
postscreen_non_smtp_command_action = enforce

postscreen_bare_newline_enable = yes
postscreen_bare_newline_action = enforce

where:

• postscreen_greet_banner means that the banner Welcome! will be sent to the connecting client and then 5 seconds later the remaining information will be sent. Mass-mailers will not wait for this time to elapse.
• postscreen_pipelining_enable this makes postfix tell the client that it needs to send one command at a time since postscreen does not support ESMTP.
• postscreen_non_smtp_command_enable some mass-mailers such as spam bots use proxies to send spam which means that some HTTP methods (such as CONNECT, GET, POST) may leak.
• postscreen_bare_newline_enable enforces using proper carriage-return and line-feed since a lot of mass-mailers just use line-feed.

After that, /etc/postfix/master.cf should be changed to use postscreen. Find the line:

smtp      inet  n       -       -       -       -       smtpd

and replace it with:

smtp      inet  n       -       -       -       1       postscreen
smtpd     pass  -       -       -       -       -       smtpd
tlsproxy  unix  -       -       n       -       0       tlsproxy
dnsblog   unix  -       -       -       -       0       dnsblog

and restart postfix.

However, be aware that postscreen can introduce delays up to 20m or more!

On Debian, there may be issues with clamav not being able to start. The common pattern in /var/log/mail.log is somewhere along the lines of:

Jul 23 21:16:06 mumu amavis[1188]: (01188-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory
Jul 23 21:16:06 mumu amavis[1188]: (01188-01) (!)ClamAV-clamd: All attempts (1) failed connecting to /var/run/clamav/clamd.ctl, retrying (2)
Jul 23 21:16:12 mumu amavis[1188]: (01188-01) (!)connect to /var/run/clamav/clamd.ctl failed, attempt #1: Can't connect to a UNIX socket /var/run/clamav/clamd.ctl: No such file or directory
Jul 23 21:16:12 mumu amavis[1188]: (01188-01) (!)ClamAV-clamd av-scanner FAILED: run_av error: Too many retries to talk to /var/run/clamav/clamd.ctl (All attempts (1) failed connecting to /var/run/clamav/clamd.ctl) at (eval 98) line 613.\n
Jul 23 21:16:12 mumu amavis[1188]: (01188-01) (!)WARN: all primary virus scanners failed, considering backups

If you run the command:

clamd -c /etc/clamav/clamd.conf

You may notice the following error:

ERROR: Parse error at line 11: Unknown option AllowSupplementaryGroups
ERROR: Can't open/parse the config file /etc/clamav/clamd.conf

So to work around the issue, open up /etc/clamav/clamd.conf and comment out the AllowSupplementaryGroups line.