virus scanning using QHPSI

A large fraction of today’s emails is infected by a virus or a worm. It is necessary to recognize those malicious emails as soon as possible already in the DATA phase of the SMTP conversation and to reject them.
When you use IndiMail, it is ultimately qmail-queue which is responsible for queueing your messages. qmail-queue stores the message component of queued mails (captured duing DATA phase of the SMTP conversation) under the mess subdirectory.
Files under the mess subdirectory are named after their i-node number. Let us look at a typical log sequence for a message received on the local system.

@400000004b9da2f03b424bb4 new msg 660188
@400000004b9da2f03b426324 info msg 660188: bytes 2794 from fogcreek_xxx@response.whatcounts.com qp 3223 uid 555
@400000004b9da2f03b42c0e4 starting delivery 6: msg 660188 to local mailstore@indimail.org
@400000004b9da2f03b42dc3c status: local 1/10 remote 0/20
@400000004b9da2f106a1e234 delivery 6: success: did_1+0+0/
@400000004b9da2f1091e676c status: local 0/10 remote 0/20
@400000004b9da2f1091fa3d4 end msg 660188

The above lines indicates that qmail has received a new message, and its queue ID is 660188. What this means is that is qmail-queue has created a file named /var/indimail/queue/mess/NN/660188. The i-node number of the file is 660188. This is the queue file that contains the message. The queue ID is guaranteed to be unique as long as the message remains in the queue (you can't have two files with the same i-node in a filesystem).
To perform virus scanning, it would be trivial to do virus scanning on the mess file above in qmail-queue itself. That is exactly what IndiMail does by using a feature called Qmail High Performance Virus Scanner (QHPSI). QHPSI was conceptualized by Erwin Hoffman. You can read here for more details.
IndiMail takes QHPSI forward by adding the ability to add plugins.
The QHPSI extension for qmail-queue allows to call an arbitary virus scanner directly, scanning the incoming data-stream on STDIN. Alternatively, it allows plugins to be loaded from the /var/indimail/plugins directory. This directory can be changed by defining PLUGINDIR environment variable. QHPSI can be advised to pass multiple arguments to the virus scanner for customization. To run external scanner or load scanner plugins, qmail-queue calls qhpsi, a program setuid to qscand. By default, qhpsi looks for the symbol virusscan to invoke the scanner. The symbol can be changed by setting the environment variable QUEUE_PLUGIN to the desired symbol.
Today’s virus scanner -- in particluar Clam AV -- work in resource efficient client/server mode (clamd/clamdscan) and include the feature to detect virii/worms in the base64 encoded data stream. Thus, there is no necessity to call additional programs (like reformime or ripmime) except for the virus scanner itself.
To enable virus scanning in IndiMail during the SMTP data phase, you can implement either of the two methods below

1. Using tcprules

Define QHPSI in tcp.smtp and rebuild tcp.smtp.cdb using tcprules.
:allow,QHPSI=’/usr/bin/clamdscan %s --quiet --no-summary’

2. Using envdir for SMTP service under supervise(8)

Define QHPSI in SMTP service's variable directory

# echo /usr/bin/clamdscan %s --quiet --no-summary > /service/qmail-smtpd.25/variables/QHPSI

If you have installed IndiMail using RPM available here or here, QHPSI is enabled by default by defining it in the qmail-smtpd.25 variables directory. If you have clamd, clamav already installed on your server, the rpm installation also installs two services under supervise.

• freshclam - service to update the clamd virus databases
• clamd - service to run the clamd scanner
  

You may need to disable clamd, freshclam startup by your system boot process and enable the startup under indimail. Do have the clamd, freshclam service started up by indimail, remove the down file. i.e.

# rm /service/freshclam/down /service/clamd/down

# /usr/bin/svc -u /service/clamd /service/freshclam

% tail -f /var/log/indimail/freshclam/current

@400000004b9da034170f6394 cdiff_apply: Parsed 17 lines and executed 17 commands
@400000004b9da03417103e54 Retrieving http://database.clamav.net/daily-10574.cdiff

@400000004b9da0342261b83c Trying to download http://database.clamav.net/daily-10574.cdiff (IP: 130.59.10.36)
Downloading daily-10574.cdiff [100%]g daily-10574.cdiff [ 13%]
@400000004b9da03509c39c64 cdiff_apply: Parsed 436 lines and executed 436 commands
@400000004b9da03510c3485c daily.cld updated (version: 10574, sigs: 24611, f-level: 44, builder: ccordes)
@400000004b9da03510c4d2e4 bytecode.cvd version from DNS: 2
@400000004b9da03510c4de9c bytecode.cvd is up to date (version: 2, sigs: 2, f-level: 44, builder: nervous)
@400000004b9da03510c82e44 Database updated (729340 signatures) from database.clamav.net (IP: 130.59.10.36)

% cat /var/log/indimail/clamd/current

@400000004b9da0260d6c1a94 Limits: Global size limit set to 104857600 bytes.
@400000004b9da0260d6c264c Limits: File size limit set to 26214400 bytes.
@400000004b9da0260d6c3204 Limits: Recursion level limit set to 16.
@400000004b9da0260d6c3dbc Limits: Files limit set to 10000.
@400000004b9da0260d6c4974 Archive support enabled.
@400000004b9da0260d6c5144 Algorithmic detection enabled.
@400000004b9da0260d6c5cfc Portable Executable support enabled.
@400000004b9da0260d6c68b4 ELF support enabled.
@400000004b9da0260d6c7084 Detection of broken executables enabled.
@400000004b9da0260e7abfbc Mail files support enabled.
@400000004b9da0260e7acb74 OLE2 support enabled.
@400000004b9da0260e7ad344 PDF support enabled.
@400000004b9da0260e7adefc HTML support enabled.
@400000004b9da0260e7ae6cc Self checking every 600 seconds.
@400000004b9da2a3116a177c No stats for Database check - forcing reload
@400000004b9da2a3206deb04 Reading databases from /var/indimail/share/clamd
@400000004b9da2a70489facc Database correctly reloaded (728651 signatures)
@400000004b9da2a7061e372c /var/indimail/queue/queue2/mess/16/660188: OK

Once you have QHPSI enabled, qmail-queue will add the header X-QHPSI in the mail. You will have the following header

X-QHPSI: clean

in case the email is clean and the following header if a virus is found

X-QHPSI: virus found

The default configuration of IndiMail will allow these emails to be delivered to the inbox. This is because some sites have have legislations like SOX, etc to enforce archiving of all emails that come into the system. In case you want to reject the email at SMTP you can do the following

# echo 1 > /service/qmail.smtpd/variables/REJECTVIRUS

# svc -d /service/qmail-smtpd.25

# svc -u /service/qmail-smtpd.25

One can also create a vfilter to deliver such email to the quarantine folder

/usr/bin/vcfilter -i -t virusFilter -c 0 -k "virus found" -f Quarantine -b 0 -h 28 prefilt@$1

If you implement different method, than explained above, let me know.

Authenticated SMTP tutorial

IndiMail supports three AUTH methods. LOGIN, PLAIN and CRAM-MD5. Most email clients like thunderbird, outlook, outlook express, evolution support these methods. These methods are provided using checkpassword compatible modules vchkpass(8) and pam-checkpwd(8)

To understand how these methods work is to use telnet and the base64 encoding/decoding utility /usr/bin/base64

For illustration purpose, let's say we have a user 'postmaster@example.com' with the password 'pass'

1. AUTH LOGIN
% echo -n postmaster@example.com | /usr/bin/base64 -i
cG9zdG1hc3RlckBleGFtcGxlLmNvbQ==

% echo -n pass | /usr/bin/base64 -i
cGFzcw==

% telent 0 smtp
220 Laptop (NO UCE) ESMTP IndiMail 1.28 21 Jun 2003 22:35:24 +0530
auth login
334 VXNlcm5hbWU6
cG9zdG1hc3RlckBleGFtcGxlLmNvbQ==
334 UGFzc3dvcmQ6
cGFzcw==
235 ok, go ahead (#2.0.0)

2. AUTH PLAIN

% printf "\0postmaster@example.com\0pass" | /usr/bin/base64
AHBvc3RtYXN0ZXJAZXhhbXBsZS5jb20AcGFzcw==

% telnet 0 smtp
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 Laptop (NO UCE) ESMTP IndiMail 1.28 21 Jun 2003 23:08:33 +0530
auth plain AHBvc3RtYXN0ZXJAZXhhbXBsZS5jb20AcGFzcw==
235 ok, go ahead (#2.0.0)
3. AUTH CRAM-MD5

The CRAM-MD5 is a challenge-response method where the password is not sent over the network. It is expected that the password is stored in the clear in IndiMail's backend database MySQL.

% sudo /usr/bin/vpasswd postmaster@example.com -e pass

Next step is to write a script named cram-md5

% cat > cram-md5 <<>"
sys.exit(1)
str=cram_md5_response(sys.argv[1], sys.argv[2], sys.argv[3]);
print "%s" %str
EOF

% sudo chmod +x ./cram-md5

Now when you do (see below) auth cram-md5, the server will issue a challenge
e.g. in the below example, the challenge is

PDIwMTM3LjEyNjc1ODUxMDBAaW5kaW1haWwub3JnPg==

if you decode this, i.e.

% echo PDIwMTM3LjEyNjc1ODUxMDBAaW5kaW1haWwub3JnPg== | base64 -d
<20137 .1267585100="" indimail.org="">

The response for the challenge can be generated using the cram-md5 shell script which we created above. i.e.

% ./cram-md5 PDIwMTM3LjEyNjc1ODUxMDBAaW5kaW1haWwub3JnPg==
cG9zdG1hc3RlckBleGFtcGxlLmNvbSBjZWU4Mzk3YWIxMjNhMGQ0ZjNhN2ZkZGJiOWNiODcxOQ==
% telnet 0 smtp
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 indimail.org (NO UCE) ESMTP IndiMail 1.137 3 Mar 2010 08:28:17 +0530
auth cram-md5
334 PDIwMTM3LjEyNjc1ODUxMDBAaW5kaW1haWwub3JnPg==
cG9zdG1hc3RlckBleGFtcGxlLmNvbSBjZWU4Mzk3YWIxMjNhMGQ0ZjNhN2ZkZGJiOWNiODcxOQ==
235 ok, go ahead (#2.0.0)

Please do take a look at Erwin Hoffman's excellent tutorial on the same subject at
http://www.fehcom.de/qmail/smtpauth.html