What is IndiMail

What is IndiMail

Would anybody be interested in a package which combines qmail with other packages like courier-imap for IMAP/POP3, bogofilter for SPAM control, fetchmail for hosts with intermittent connectivity, ability for a single domain to have users across multiple hosts (even across different geographical locations) and tools to manage virtual domains ?

As a hobby way back in 2001 I started playing with qmail. Over the years, i learnt how to use qmail and put together a complete solution which worked quite well for me and can work for large ISP installations (> 8 million+ users in a single domain across multiple hosts and more than 4 million deliveries per day). Maybe some of the stuff could prove useful for someone who had the same situation as me (very little budget to buy costly NAS filers, availability of simple hardware, etc). So with this thoughts, I am attempting my first big contribution to open source. I am not sure how one releases code. But I have opened a project called IndiMail at http://sourceforge.net/projects/indimail. The thought is to write a set of utilities to manage virtual domains using DJB's functions (stralloc, substdio, etc). There will be a host schema (hostcntrl) to store the host, storage information for all users. Deliveries will lookup this schema to make deliveries for local users.

Since the code uses lot of GPL code, this code itself will be GPL.

Disclaimer

There is no warranty implied or otherwise with this package. I believe in OpenSource Philosophy and this is simply an attempt to give back to the OpenSource community. Other than that I do not desire any monetary or material benefit out of this.

What will you find in this package

Integrated solution of the following packages qmail, serialmail, qmailanalog, dotforward, fastforward, mess822, daemontools, ucspi-tcp, Indimail - Management of Virtual domains, Courier IMAP/POP3, Bogofilter - A Bayesian Spam Filter, Fetchmail, other useful utilities.

If any modification or addition to been done to DJB's code, it will been done in the same style (not using the standard C library and using only DJB's functions like substdio, stralloc, etc). All man pages will be modified to reflect changes.

This package will allow to split users in a single domain across multiple hosts without using NFS. I haven't looked at qmail-ldap but maybe this package would something like qmail-ldap but with MySQL as the backend for storing user information. The tools for administering virtual domains/users will have a lot in similarity to the vpopmail package from inter7.com. The table structures will be the same. Plenty of code (for managing virtual domains/users) can been borrowed from there. Since the user get split across multiple hosts, IndiMail will also provide two proxies for IMAP & POP3 protocol. The proxy servers will run using daemontools and are by default be set under supervise. You will be able to use any IMAP/POP3 server behind the proxy. IndiMail will also allow you to add any proprietary mail server serving some of the users of any domain which you want to configure under indimail. This will makes it easy to migrate from an existing proprietary mail server like MS Exchange in 5 steps. The Steps will be

1. You simply set up a new IndiMail installation and add the existing domain
2. Add the IP address and SMTP port of the Exchange Server in MySQL table host_table and smtp_port respectively.
3. Add existing users on the Exchange Server in a MySQL table hostcntrl (either manually or using the utility hostcntrl
4. Set up SMTP, IMAP Proxy, POP3 Proxy on the IndiMail Server.
5. Change the MX to point to the IndiMail server
   

Feature List

Some of the features available in this package (Probably most of these would not be available in any patch out there). These changes are being coded/done by me.

1. Indentation of djb's code (using indent) so that a mortal like me could understand it :)
2. use getpwnam to use uids/gids from /etc/passwd, /etc/group (allows me to transfer the installation to any host regardless of the ids in /etc/passwd)
3. configurable control directory (using CONTROLDIR environment variable) (allows me to have multiple running copies of qmail using a single binary installation)
4. configurable queue directory (using QUEUEDIR environment variable) (allows me to have multiple queues on a host with a single qmail installation).

   qmail-multi (queue load balancer) uses qmail-queue to deposit mails across multiple queues. Each queue has its own qmail-send process. You can spread the individual queues across multiple filesystems on different controllers to maximize on IO throughput.
   

   A queue in indimail is configurable by three environment variables QUEUE_BASE, QUEUE_COUNT, and QUEUE_START.

   A queue in indimail is defined as a collection of multiple queues. Each queue in the collection can have one or more SMTP listener but a single delivery (qmail-send) processes. It is possible to have the entire queue collection without a delivery process (e.g. SMTP on port 366 – ODMR). The QUEUE_COUNT can be defined based on how powerful your host is (IO bandwidth, etc).
   

5. ETRN, ATRN, ODMR (RFC 2645) support
6. accesslist - restrictions between mail transactions between email ids (you can decide who can send mails to whom)
7. bodycheck - checks on header/body on incoming emails (for spam, virus security and other needs)
8. hostaccess - provides domain, IP address pair access control. e.g. you can define from which set of addresses mail from yahoo.com will be accepted.
   
9. chkrcptdomains - rcpt check on selective domains
10. NULLQUEUE, qmail-nullqueue (blackhole support - like qmail-queue but mails go into a blackhole). I typically uses this in conjuction with envrules to trash the mail into blackhole without spending any disk IO.
    
11. envrules - recipient/sender based set or unset environment variables (qmail-smtpd for senders, qmail-inject for senders, qmail-local for recipients, qmail-remote for recipients, qmail-send for bounce recipients) any variables which controls the behaviour of qmail-smtpd, qmail-inject, qmail-local, qmail-remote e.g. NODNSCHECKS, DATABYTES, RELAYCLIENT, BADMAILFROM, etc can be defined individually for a particular recipient, bounce recipient or sender rather than only in the run file or control files
    
12. qmail-multi - run multiple filters (qmail-smtpd) (something like qmail-qfilter). Also distributes mails across multiple queues to do a load balancing act. qmail-multi allowed me to process massive rate of incoming mails at my earlier job with a ISP.
13. envheaders - Any thing defined here e.g. Return-Path, qmail-queue sets Return-Path as an environment variable with the value found in Return-Path header in the email. This environment variable gets passed across the queue and is also available to qmail-local, qmail-remote
14. logheaders - Any header defined in this control file, gets written to file descriptor 2 with the value found in the email.
15. removeheaders - Any header defined here, qmail-queue will remove that header from the email
16. quarantine or QUARANTINE env variable causes qmail-queue to replace the recipient list with the value defined in the control file or the environment variable. Additionally an environment variable X-Quarantine-ID: is set which holds the orignal recipient list.
17. Added ability in qmail-queue to do line processing. Line processing allows qmail-queue to do some of the stuff mentioned above
18. plugins support for QHPSI interface (qmail-queue). qmail-queue will use dlopen to load any shared objected defined by PLUGINDIR environment. Multiple plugins can be loaded. For details see man qmail-queue
19. QMAILREMOTE - Run executable defined by this instead of qmail-remote (qmail-remote) QMAILLOCAL - Run executable defined by this instead of qmail-local (qmail-local).Theoretically one can exploit QMAILLOCAL, QMAILREMOTE variables to route mails for a domain across multiple mail stores.
20. qmail-rspawn hack which connects to MySQL and keeps the connection open. This gives qmail-rspawn to do high speed user lookups in MySQL and to deliver the mail for a single domain split across multiple mail stores.
21. Message Submission Port (port 587) RFC 2476
22. Integrated Authenticted SMTP with Indimail (PLAIN, LOGIN, CRAM-MD5, pop-bef-smtp)
23. qmail-remote can do User Based Routing via SMTPROUTE environment. qmail-rspawn has the ability to connect to MySQL and set SMTPROUTES. This gives split a domain across multiple hosts without using NFS to mount multiple filesystems on any host. One can even use a shell script, set the environment variable and deliver mails to users across multiple hosts. I call this dynamic SMTPROUTE
24. duplicate eliminator using 822header
25. qmail-remote has configurable TCP timeout table (max_tolerance, min_backoff periods can be configured in smtproutes)
26. Ability to change concurrency of tcpserver without restarting tcpserver
27. Ability to restrict connections per IP
28. multilog replaced buffer funtions with substdio
29. supervise can run script shutdown if present on svc -d
30. rfc3834 compliance for qmail-autoresponder (provide Auto-Submitted, In-Reply-To, References fields (RFC 3834))
31. ability to add disclaimer to messages.
32. Proxy for IMAP and POP3. Allows IMAP/POP3 protocol for users in a domain to be split across multiple hosts. Also allows seamless integration of proprietary email servers with indimail.
    
33. Dbserver - serves as a high performance user lookup daemon for qmail-smtpd (rcpt checks, authenticated SMTP, RELAY check). Even the IMAP, POP3 authentication gets served by dbserver. dbserver preforks configurable number of daemons which opens multiple connections to MySQL and keep the connection open. This gives dbserver a decent database performance when handling millions of lookups in few hours. dbserver uses a fifo to communicate with qmail-smtpd
34. indisrvr - Was written to ease mail server administration across multiple hosts. Allows ones to create, delete, modify users and run any command as defined in variables.c. indisrvr listens on a AF_INET socket.
35. svctool - A simple tool which helps you to configure any configuration item in indimail (creation of supervise scripts, qmail configuration, installation of all default MySQL tables, creation of default aliases, users, etc)
    

Other features are as below which I have incorporated manually by hand as and when they came up at Qmail List <qmail@list.cr.yp.to>

CREDITS

The credit for inspiration to write/put together this package is entire due to Dr. Daniel Bernstein. Had he not written qmail the way it is written, i wouldn't probably have fallen in love with the code. In no way is this package endorsed by DJB. Though I have put in lot of effort putting together this package, any bugs or flaws could be entirely due to me. So I would be glad if the community could further improve this. Since I have gained so much personally as well as professionally due to qmail, the least I can do is to put back my experiences and learnings for anyone who would be interested. I release my entire code as Open Source GPL.

My contribution is minuscule compared to what I have picked up from others in building the packages. Since this was being done for my own personal use, over the years, I may have missed mentioning some parts of the code that has been written by others. In case you know that I have missed out giving credit to anyone for any of the code or idea, let me know and I will be glad to add it here.

Dr. Daniel Bernstein for the following original packages which are there are http://cr.yp.to

1. qmail-1.03
2. dot-forward
3. fast-forward
4. qmailanalog
5. serialmail
6. ucspi-tcp
7. daemontools
8. mess822
9. checkpassword
   

My ex colleagues Sushant TS, Murali Panchapakesan, Govind Raghuram, Ramya Krishnan, Anuradha TP, Premnath Sah who came with lot of ideas and suggestion and snippets of code. A great number of bugs were identified by Ramya who became an expert in using strace after this.

The site http://qmail.org and the qmail mailing list for wealth of information

1. Russel Nelson - qmail-lint, qmail-dk, big-todo, antivirus patch for qmail-smtpd, selfhelo, reject relay probes, logselect
2. inter7.com - vpopmail - Chris Johnson, Ken Jones, Bill Shupp, Tom Collins
3. Chris Johnson - tarpitting, RELAYMAILFROM patch
4. Chris Kennedy – Blackhole
5. Charles Cazabon - doublebounce-trim-patch http://www.qmail.org/doublebounce-trim.patch.
6. Klaus Reimer - bouncecontrol patch
7. Johannes Erdfelt - big-concurrency patch
8. Scott Gifford - 0.0.0.0 patch
9. Chuck Foster - binding outgoing connections to local interface
10. Andre Oppermann – ext-todo
11. Andreas Aardal Hanssen - ext-todo and big-todo
12. Bruce Guenter - QMAILQUEUE, qmail-qstat, syncdir, rate limiting autoresponder based on Eric's Huss's Design, qmail-qfilter, QUEUE_EXTRA patch
13. Dave Sill - Life with qmail, inst_check script
14. Erwin Hoffman - QHPSI, Authenticated SMTP, RFC1870 SMTP SIZE, SPAMCONTROL
15. Gerrit Pape - Man pages for ucspi-tcp, daemontools-0.76
16. Peter Samuels – tai64nunix
17. John Levine - patch to matchup.c to accept tai64n dates, Auth SMTP for ofmipd.
18. Georg Lehner - qmailanalog better integrated with multilog
19. http://www.magma.com.ni/moin/TipsAnd/QmailAnalog
20. Chris Garrigues - pretty-print Received: lines.
21. ftp://ftp.foxharp.boston.ma.us/pub/pgf/qmail/mailroute.pl
22. Matt Ranney – qmail-lagcheck
23. Michele Beltrame - qmhandle http://sourceforge.net/projects/qmhandle
24. Eric Huss - queue-fix (with patch by Matthew Harrel to work for big-todo)
25. Eric Huss - qmail-qmqpc timeout patch
26. William E Baxtar - qtools - http://www.superscript.com/qtools/intro.html
27. Linux Magic - qmail-remove http://www.linuxmagic.com/opensource/qmail/qmail-remove/
28. Alex Kramarov – qmail-print-queue
29. Folkert van Heusden – multitail
30. Russ Nelson/Ivan Kohler – mbox2maildir.pl
31. Tetsu Ushijima - maildirdeliver - http://www.din.or.jp/~ushijima/maildirdeliver.html
32. Evan Champion - patch to condredirect
33. John Saunders - patch to date822fmt.c (emit dates in local timezones), newline patch
34. Robert Sander - RECIPIENT Extension
35. Chuck Foster - bindroutes patch
36. Nick Leverton - holdremote patch
37. Fred Lindberg - Preserve MIME-ness of message when bouncing MIME message
38. Krzysztof Dabrowski - Authenticated SMTP Patch
39. Frederik Vermeulen – starttls
40. Frank DENIS - patch to truncate bounce messages
41. Matthias Andree - Found the 'qmail-local tab' bug and introduced the 'sendmail -N dsn' compatibility
42. Scott Gifford - ipme, moreipme patch
43. Charles Cazabon - patch to enforce single recipients on bounces.
44. Jay Soffian - auth smtp patch for qmail-remote
45. Adrian Ho - RFC 2821 in qmail-remote
46. Ingo Rohloff - SMTP authentication support to serialsmtp
47. Ward Vandewege - badrcptto patch
48. Kazinori Fujiwara - ipv6 patch
49. Len Budney – qscanq
50. Christophe Saout - SPF checker
51. David Phillips - qmail's sendmail's -f set the default for the username like sendmail does.
52. Mark Belnap - Bounce Lifetime patch
53. Christopher K. Davis' DNS patch handles a problem qmail has with DNS responses larger than the standard
54. 512 bytes. http://www.ckdhr.com/ckd/qmail-103.patch and
55. Mark Delany - wildmat patch
56. Nagy Balazs - patch to ensure that the domain name on the envelope sender is a valid DNS name.
57. re-read concurrencylocal concurrency remote on SIGHUP.
58. Erik Sjölund - qmail-local tab patch
59. Frank Denis - Patched qmail-send to limit the size of bounces, MAXRECIPIENT, maxhop control file
60. Paul Gregg - ENFORCE_FQDN_HELO patch 29/08/2003
61. qmail-remove from LinuxMagic is the development arm of Wizard Tower TechnoServices Ltd
62. qtools - from www.superscript.com, SIGINT to multilog to stop writing to log
63. Qmail Holdremote Patch by Nick Leverton
64. ripMIME/altermime by PLDaniels,
65. evaluate - evaluate algebraic strings(C) 2000-2002 Kyzer/CSG. http://www.kyzer.me.uk/code/evaluate/
66. make seekable patch "Marcus Williams" marcus at quintic.co.uk.
67. daemontools patches from (additional signals to svc, svscan logging, http://www.gluelogic.com/code/daemontools/)
68. Save/discard logs with multilog (SIGINT/SIGHUP to toggle) - http://www.superscript.com/patches/intro.html
69. timed log rotation for multilog - Jos Backus
70. Alin-Adrian Anton - Fixed qmail-smtpd vulnurability for very long header lines
71. http://www.esat.kuleuven.ac.be/~vermeule/qmail/tls.patch STARTTLS
72. http://members.elysium.pl/brush/qmail-smtpd-auth/ SMTP AUTH
73. http://www.netable.com/~dburkes/qmail-smtpd-requireauth/ Require AUTH
74. http://www.elysium.pl/members/brush/cmd5checkpw/ CRAM-MD5 Checkpassword
75. RFC-2554, RFC-2222 compliance
76. Rask Ingemann Lambertsen - who provided the original RELAY Patch
77. Markus Stumpf - provided the original LOGGING patch
78. Charles Cazabon - Author of the NULL-Sender modifcation
79. Bjoern Kalkbrenner - Initial auther of the qmail-smtp-auth-send patch.
80. Peter Ladwig - had the idea to use hard tarpitting in case of too many invalid RECIPIENTS.
81. Flash Secure Menu Shell - WWW: http://www.netsoc.ucd.ie/flash/ Author: Steve Fegan
82. mpack/unpack from CMU