Would anybody be interested in a package which combines qmail with other packages like courier-imap for IMAP/POP3, bogofilter for SPAM control, fetchmail for hosts with intermittent connectivity, ability for a single domain to have users across multiple hosts (even across different geographical locations) and tools to manage virtual domains ?
As a hobby way back in 2001 I started playing with qmail. Over the years, i learnt how to use qmail and put together a complete solution which worked quite well for me and can work for large ISP installations (> 8 million+ users in a single domain across multiple hosts and more than 4 million deliveries per day). Maybe some of the stuff could prove useful for someone who had the same situation as me (very little budget to buy costly NAS filers, availability of simple hardware, etc). So with this thoughts, I am attempting my first big contribution to open source. I am not sure how one releases code. But I have opened a project called IndiMail at http://sourceforge.net/projects/indimail. The thought is to write a set of utilities to manage virtual domains using DJB's functions (stralloc, substdio, etc). There will be a host schema (hostcntrl) to store the host, storage information for all users. Deliveries will lookup this schema to make deliveries for local users.
Since the code uses lot of GPL code, this code itself will be GPL.
There is no warranty implied or otherwise with this package. I believe in OpenSource Philosophy and this is simply an attempt to give back to the OpenSource community. Other than that I do not desire any monetary or material benefit out of this.
What will you find in this package
Integrated solution of the following packages qmail, serialmail, qmailanalog, dotforward, fastforward, mess822, daemontools, ucspi-tcp, Indimail - Management of Virtual domains, Courier IMAP/POP3, Bogofilter - A Bayesian Spam Filter, Fetchmail, other useful utilities.
If any modification or addition to been done to DJB's code, it will been done in the same style (not using the standard C library and using only DJB's functions like substdio, stralloc, etc). All man pages will be modified to reflect changes.
This package will allow to split users in a single domain across multiple hosts without using NFS. I haven't looked at qmail-ldap but maybe this package would something like qmail-ldap but with MySQL as the backend for storing user information. The tools for administering virtual domains/users will have a lot in similarity to the vpopmail package from inter7.com. The table structures will be the same. Plenty of code (for managing virtual domains/users) can been borrowed from there. Since the user get split across multiple hosts, IndiMail will also provide two proxies for IMAP & POP3 protocol. The proxy servers will run using daemontools and are by default be set under supervise. You will be able to use any IMAP/POP3 server behind the proxy. IndiMail will also allow you to add any proprietary mail server serving some of the users of any domain which you want to configure under indimail. This will makes it easy to migrate from an existing proprietary mail server like MS Exchange in 5 steps. The Steps will be
- You simply set up a new IndiMail installation and add the existing domain
- Add the IP address and SMTP port of the Exchange Server in MySQL table host_table and smtp_port respectively.
- Add existing users on the Exchange Server in a MySQL table hostcntrl (either manually or using the utility hostcntrl
- Set up SMTP, IMAP Proxy, POP3 Proxy on the IndiMail Server.
- Change the MX to point to the IndiMail server
Some of the features available in this package (Probably most of these would not be available in any patch out there). These changes are being coded/done by me.
- Indentation of djb's code (using indent) so that a mortal like me could understand it :)
- use getpwnam to use uids/gids from /etc/passwd, /etc/group (allows me to transfer the installation to any host regardless of the ids in /etc/passwd)
- configurable control directory (using CONTROLDIR environment variable) (allows me to have multiple running copies of qmail using a single binary installation)
- configurable queue directory (using QUEUEDIR environment variable) (allows me to have multiple queues on a host with a single qmail installation).
qmail-multi (queue load balancer) uses qmail-queue to deposit mails across multiple queues. Each queue has its own qmail-send process. You can spread the individual queues across multiple filesystems on different controllers to maximize on IO throughput.
A queue in indimail is configurable by three environment variables QUEUE_BASE, QUEUE_COUNT, and QUEUE_START.
A queue in indimail is defined as a collection of multiple queues. Each queue in the collection can have one or more SMTP listener but a single delivery (qmail-send) processes. It is possible to have the entire queue collection without a delivery process (e.g. SMTP on port 366 – ODMR). The QUEUE_COUNT can be defined based on how powerful your host is (IO bandwidth, etc).
- ETRN, ATRN, ODMR (RFC 2645) support
- accesslist - restrictions between mail transactions between email ids (you can decide who can send mails to whom)
- bodycheck - checks on header/body on incoming emails (for spam, virus security and other needs)
- hostaccess - provides domain, IP address pair access control. e.g. you can define from which set of addresses mail from yahoo.com will be accepted.
- chkrcptdomains - rcpt check on selective domains
- NULLQUEUE, qmail-nullqueue (blackhole support - like qmail-queue but mails go into a blackhole). I typically uses this in conjuction with envrules to trash the mail into blackhole without spending any disk IO.
- envrules - recipient/sender based set or unset environment variables (qmail-smtpd for senders, qmail-inject for senders, qmail-local for recipients, qmail-remote for recipients, qmail-send for bounce recipients) any variables which controls the behaviour of qmail-smtpd, qmail-inject, qmail-local, qmail-remote e.g. NODNSCHECKS, DATABYTES, RELAYCLIENT, BADMAILFROM, etc can be defined individually for a particular recipient, bounce recipient or sender rather than only in the run file or control files
- qmail-multi - run multiple filters (qmail-smtpd) (something like qmail-qfilter). Also distributes mails across multiple queues to do a load balancing act. qmail-multi allowed me to process massive rate of incoming mails at my earlier job with a ISP.
- envheaders - Any thing defined here e.g. Return-Path, qmail-queue sets Return-Path as an environment variable with the value found in Return-Path header in the email. This environment variable gets passed across the queue and is also available to qmail-local, qmail-remote
- logheaders - Any header defined in this control file, gets written to file descriptor 2 with the value found in the email.
- removeheaders - Any header defined here, qmail-queue will remove that header from the email
- quarantine or QUARANTINE env variable causes qmail-queue to replace the recipient list with the value defined in the control file or the environment variable. Additionally an environment variable X-Quarantine-ID: is set which holds the orignal recipient list.
- Added ability in qmail-queue to do line processing. Line processing allows qmail-queue to do some of the stuff mentioned above
- plugins support for QHPSI interface (qmail-queue). qmail-queue will use dlopen to load any shared objected defined by PLUGINDIR environment. Multiple plugins can be loaded. For details see man qmail-queue
- QMAILREMOTE - Run executable defined by this instead of qmail-remote (qmail-remote) QMAILLOCAL - Run executable defined by this instead of qmail-local (qmail-local).Theoretically one can exploit QMAILLOCAL, QMAILREMOTE variables to route mails for a domain across multiple mail stores.
- qmail-rspawn hack which connects to MySQL and keeps the connection open. This gives qmail-rspawn to do high speed user lookups in MySQL and to deliver the mail for a single domain split across multiple mail stores.
- Message Submission Port (port 587) RFC 2476
- Integrated Authenticted SMTP with Indimail (PLAIN, LOGIN, CRAM-MD5, pop-bef-smtp)
- qmail-remote can do User Based Routing via SMTPROUTE environment. qmail-rspawn has the ability to connect to MySQL and set SMTPROUTES. This gives split a domain across multiple hosts without using NFS to mount multiple filesystems on any host. One can even use a shell script, set the environment variable and deliver mails to users across multiple hosts. I call this dynamic SMTPROUTE
- duplicate eliminator using 822header
- qmail-remote has configurable TCP timeout table (max_tolerance, min_backoff periods can be configured in smtproutes)
- Ability to change concurrency of tcpserver without restarting tcpserver
- Ability to restrict connections per IP
- multilog replaced buffer funtions with substdio
- supervise can run script shutdown if present on svc -d
- rfc3834 compliance for qmail-autoresponder (provide Auto-Submitted, In-Reply-To, References fields (RFC 3834))
- ability to add disclaimer to messages.
- Proxy for IMAP and POP3. Allows IMAP/POP3 protocol for users in a domain to be split across multiple hosts. Also allows seamless integration of proprietary email servers with indimail.
- Dbserver - serves as a high performance user lookup daemon for qmail-smtpd (rcpt checks, authenticated SMTP, RELAY check). Even the IMAP, POP3 authentication gets served by dbserver. dbserver preforks configurable number of daemons which opens multiple connections to MySQL and keep the connection open. This gives dbserver a decent database performance when handling millions of lookups in few hours. dbserver uses a fifo to communicate with qmail-smtpd
- indisrvr - Was written to ease mail server administration across multiple hosts. Allows ones to create, delete, modify users and run any command as defined in variables.c. indisrvr listens on a AF_INET socket.
- svctool - A simple tool which helps you to configure any configuration item in indimail (creation of supervise scripts, qmail configuration, installation of all default MySQL tables, creation of default aliases, users, etc)
The credit for inspiration to write/put together this package is entire due to Dr. Daniel Bernstein. Had he not written qmail the way it is written, i wouldn't probably have fallen in love with the code. In no way is this package endorsed by DJB. Though I have put in lot of effort putting together this package, any bugs or flaws could be entirely due to me. So I would be glad if the community could further improve this. Since I have gained so much personally as well as professionally due to qmail, the least I can do is to put back my experiences and learnings for anyone who would be interested. I release my entire code as Open Source GPL.
My contribution is minuscule compared to what I have picked up from others in building the packages. Since this was being done for my own personal use, over the years, I may have missed mentioning some parts of the code that has been written by others. In case you know that I have missed out giving credit to anyone for any of the code or idea, let me know and I will be glad to add it here.
Dr. Daniel Bernstein for the following original packages which are there are http://cr.yp.to
The site http://qmail.org and the qmail mailing list for wealth of information
- Russel Nelson - qmail-lint, qmail-dk, big-todo, antivirus patch for qmail-smtpd, selfhelo, reject relay probes, logselect
- inter7.com - vpopmail - Chris Johnson, Ken Jones, Bill Shupp, Tom Collins
- Chris Johnson - tarpitting, RELAYMAILFROM patch
- Chris Kennedy – Blackhole
- Charles Cazabon - doublebounce-trim-patch http://www.qmail.org/doublebounce-trim.patch.
- Klaus Reimer - bouncecontrol patch
- Johannes Erdfelt - big-concurrency patch
- Scott Gifford - 0.0.0.0 patch
- Chuck Foster - binding outgoing connections to local interface
- Andre Oppermann – ext-todo
- Andreas Aardal Hanssen - ext-todo and big-todo
- Bruce Guenter - QMAILQUEUE, qmail-qstat, syncdir, rate limiting autoresponder based on Eric's Huss's Design, qmail-qfilter, QUEUE_EXTRA patch
- Dave Sill - Life with qmail, inst_check script
- Erwin Hoffman - QHPSI, Authenticated SMTP, RFC1870 SMTP SIZE, SPAMCONTROL
- Gerrit Pape - Man pages for ucspi-tcp, daemontools-0.76
- Peter Samuels – tai64nunix
- John Levine - patch to matchup.c to accept tai64n dates, Auth SMTP for ofmipd.
- Georg Lehner - qmailanalog better integrated with multilog
- Chris Garrigues - pretty-print Received: lines.
- Matt Ranney – qmail-lagcheck
- Michele Beltrame - qmhandle http://sourceforge.net/projects/qmhandle
- Eric Huss - queue-fix (with patch by Matthew Harrel to work for big-todo)
- Eric Huss - qmail-qmqpc timeout patch
- William E Baxtar - qtools - http://www.superscript.com/qtools/intro.html
- Linux Magic - qmail-remove http://www.linuxmagic.com/opensource/qmail/qmail-remove/
- Alex Kramarov – qmail-print-queue
- Folkert van Heusden – multitail
- Russ Nelson/Ivan Kohler – mbox2maildir.pl
- Tetsu Ushijima - maildirdeliver - http://www.din.or.jp/~ushijima/maildirdeliver.html
- Evan Champion - patch to condredirect
- John Saunders - patch to date822fmt.c (emit dates in local timezones), newline patch
- Robert Sander - RECIPIENT Extension
- Chuck Foster - bindroutes patch
- Nick Leverton - holdremote patch
- Fred Lindberg - Preserve MIME-ness of message when bouncing MIME message
- Krzysztof Dabrowski - Authenticated SMTP Patch
- Frederik Vermeulen – starttls
- Frank DENIS - patch to truncate bounce messages
- Matthias Andree - Found the 'qmail-local tab' bug and introduced the 'sendmail -N dsn' compatibility
- Scott Gifford - ipme, moreipme patch
- Charles Cazabon - patch to enforce single recipients on bounces.
- Jay Soffian - auth smtp patch for qmail-remote
- Adrian Ho - RFC 2821 in qmail-remote
- Ingo Rohloff - SMTP authentication support to serialsmtp
- Ward Vandewege - badrcptto patch
- Kazinori Fujiwara - ipv6 patch
- Len Budney – qscanq
- Christophe Saout - SPF checker
- David Phillips - qmail's sendmail's -f set the default for the username like sendmail does.
- Mark Belnap - Bounce Lifetime patch
- Christopher K. Davis' DNS patch handles a problem qmail has with DNS responses larger than the standard
- 512 bytes. http://www.ckdhr.com/ckd/qmail-103.patch and
- Mark Delany - wildmat patch
- Nagy Balazs - patch to ensure that the domain name on the envelope sender is a valid DNS name.
- re-read concurrencylocal concurrency remote on SIGHUP.
- Erik Sjölund - qmail-local tab patch
- Frank Denis - Patched qmail-send to limit the size of bounces, MAXRECIPIENT, maxhop control file
- Paul Gregg - ENFORCE_FQDN_HELO patch 29/08/2003
- qmail-remove from LinuxMagic is the development arm of Wizard Tower TechnoServices Ltd
- qtools - from www.superscript.com, SIGINT to multilog to stop writing to log
- Qmail Holdremote Patch by Nick Leverton
- ripMIME/altermime by PLDaniels,
- evaluate - evaluate algebraic strings(C) 2000-2002 Kyzer/CSG. http://www.kyzer.me.uk/code/evaluate/
- make seekable patch "Marcus Williams" marcus at quintic.co.uk.
- daemontools patches from (additional signals to svc, svscan logging, http://www.gluelogic.com/code/daemontools/)
- Save/discard logs with multilog (SIGINT/SIGHUP to toggle) - http://www.superscript.com/patches/intro.html
- timed log rotation for multilog - Jos Backus
- Alin-Adrian Anton - Fixed qmail-smtpd vulnurability for very long header lines
- http://www.esat.kuleuven.ac.be/~vermeule/qmail/tls.patch STARTTLS
- http://members.elysium.pl/brush/qmail-smtpd-auth/ SMTP AUTH
- http://www.netable.com/~dburkes/qmail-smtpd-requireauth/ Require AUTH
- http://www.elysium.pl/members/brush/cmd5checkpw/ CRAM-MD5 Checkpassword
- RFC-2554, RFC-2222 compliance
- Rask Ingemann Lambertsen - who provided the original RELAY Patch
- Markus Stumpf - provided the original LOGGING patch
- Charles Cazabon - Author of the NULL-Sender modifcation
- Bjoern Kalkbrenner - Initial auther of the qmail-smtp-auth-send patch.
- Peter Ladwig - had the idea to use hard tarpitting in case of too many invalid RECIPIENTS.
- Flash Secure Menu Shell - WWW: http://www.netsoc.ucd.ie/flash/ Author: Steve Fegan
- mpack/unpack from CMU