Friday, April 22, 2011

Database Connections made by IndiMail

IndiMail uses MySQL for storing information of virtual domain users. The table 'indimail' stores important user information like password, access permissions, quota and the mailbox path. Most of user related queries have to lookup the 'indimail' table in MySQL.
Rather than making individual connections to MySQL for extracting information from the 'indimail' table, IndiMail programs use the service of the inlookup(8) server. Programs use an API function inquery() to request service. InLookup is a connection pooling server to serve requests for inquery() function. It is implemented over two fifos. One fixed fifo for reading the query and reading the path of a randomly generated fifo. The randomly generated fifo is used for writing the result of the query back. inlookup(8) creates a read FIFO determined by the environment variable INFIFO. If INFIFO is not defined, the default FIFO used is /var/indimail/inquery/infifo. inlookup(8) then goes into an infinite loop reading this FIFO. If INFIFO is not an absolute path, inlookup(8) uses environment variable FIFODIR to look for fifo named by INFIFO variable. Inlookup(8) can be configured by setting environment variables in /service/inlookup.info/variables
  • inlookup helps in optimizing connection to MySQL(1), by keeping the connections persistent.
  • It also maintains the query result in a double link list.
  • It uses binary tree algorithm to search the cache before actually sending the query to the database.
  • IndiMail clients send requests for MySQL(1) queries to inlookup through the function inquery() using a fifo.
  • The inquery() API uses the InLookup service only if the environment variable QUERY_CACHE is set. If this environment variable is not set, the inquery() function makes a direct connecton to MySQL.
  • Clients which are currently using inquery are qmail-smtpd(1), proxyimap(8), proxypop3(8), vchkpass(8) and authindi(8).
  • inlookup(8) service is one of the reasons why IndiMail is able to serve million+ users using commodity hardware.
The program inquerytest simulates all the queries which inlookup supports and can be used as a test/diagnostic tool for submitting queries to inlookup. e.g
% sudo inquerytest -q 3 -i "" user@example.com



Friday, April 15, 2011

Plugin for qmail-smtpd

IndiMail now has a plugin functionality for qmail-smtpd. You can write your own plugin to carry out tasks during the MAIL, RCPT or the DATA phase of SMTP.  See the man page for plugin_init(3) for details.

NAME
plugin_init() - Template for Dynamic SMTP Plugins

SYNTAX
#include smtp_plugin.h

char *from_plug(char *rip, char *from, char **mesg);

char *rcpt_plug(char *rip, char *from, char *rcpt, char **mesg);

char *data_plug(char *local, char *rip, char *rhost, char *rinfo, char **mesg);

PLUGIN *plugin_init();

typedef struct
{
int (*mail_func) (char *, char *, char **);
int (*rcpt_func) (char *, char *, char *, char **);
int (*data_func) (char *, char *, char *, char *, char **);
} PLUGIN;

DESCRIPTION
PLUGIN structure has three components: mail_func is a pointer to function to be exe‐
cuted in the SMTP MAIL session. rcpt_func is a pointer to function to be executed in
the SMTP RCPT session. data_func is a pointer to function to be executed in the SMTP
DATA session.

To write a SMTP plugin you have to write the plugin_init() function. Depending on
which phase of SMTP (MAIL, RCPT, DATA) you want to call your function, you have to
write the from_plug, rcpt_plug, data_plug functions.

The plugin_init() function can be written as below

PLUGIN *
plugin_init()
{
static PLUGIN plug;
PLUGIN *ptr;

ptr = &plug;
ptr->mail_func = from_plug;
ptr->rcpt_func = rcpt_plug;
ptr->data_func = data_plug;
return &plug;
}

To compile the plugin you can use gcc(1).

gcc -shared -rdynamic -nostartfiles -fPIC -s -O4 -o smtpd-plugin.so smtp_plugin.o

RETURN VALUE
The functions from_plug, rcpt_plug, data_plug must return 0 on success. These func‐
tions should return 1 to terminate the sesson with a message. You can set your own
message by assigning mesg variable. If you have the below function in smtpd-plugin.so

int
rcpt_plug(char *remoteip, char *from, char *rcpt, char **mesg)
{
if (!strstr(rcpt, "@yahoo.com"))
{
*mesg = "530 We are serious and don't Yahoo (#5.7.1)
return (1);
}
return (0);
}

SEE ALSO
qmail-smtpd(8), dlopen(3), gcc(1)

Friday, April 1, 2011

The AdminClient Protocol

IndiMail provides close to around 300 different programs as part of a flexible Enterprise Messaging Platform. You can carry administer the entire platform with around 45 of these programs. A program called indisrvr(8) provides a way for users to secure execute these commands from any remote location. To execute these programs, you need to have an admin account on the IndiMail server. These accounts can be created by the mgmtpass(8) program. Once you have an admin account on the IndiMail server, you can further restrict users to certain programs using the vpriv(8) program. vpriv can further modify privileges by allowing only certain options within a allowed program.

To execute these programs on the IndiMail server, you need to connect to port 4000 and use the adminclient protocol. The adminclient protocol is described below by showing a conversation between a client and the server 'indisrvr'


s - denotes server
c - denotes client

        s: "Login: "
        c: "userid\n"
        s: "Password: "
        c: "password\n"
        s: "OK\n"
        c: "index command arg1 arg2 ...\n"
        s:
        c: "\n"
        s: "RETURNSTATUS[return value of command]\n"

To execute 'vuserinfo' you will need to do the following.

        % telnet 0 4000
        Trying 0.0.0.0...
        Connected to 0.
        Escape character is '^]'.
        Login: admin
        Password: benhur20
        OK
        7 vuserinfo -n manvendra@indimail.org
        name          : manvendra@indimail.org
        
        RETURNSTATUS0

Index value of '7' was used for vuserinfo. A privileged user 'admin' with password 'benhur20' was used to execute vuserinfo. Each command has an index. The values are given as below.

Index Commands
----- --------
00    vadduser
01    vpasswd
02    vdeluser
03    vsetuserquota
04    vbulletin
05    vmoduser
06    valias
07    vuserinfo
08    vipmap
09    vacation
10    vmoveuser
11    vrenameuser
12    crc
13    vcfilter
14    indiversion
15    vsmtp
16    dbinfo
17    vhostid
18    mgmtpass
19    inquerytest
20    printdir
21    shit
22    vaddaliasdomain
23    vadddomain
24    vcalias
25    vcaliasrev.sh
26    vconvert
27    vdeldomain
28    vrenamedomain
29    vdominfo
30    vfstab
31    vgroup
32    vatrn
33    vpriv
34    vlimit
35    hostcntrl
36    execmysql
37    updatefile
38    vreorg
39    vdeloldusers
40    ipchange
41    svctool
42    clearopensmtp
43    hostsync
44    inquerytest
45    vmoddomain

If you have a user provisioning interface written in your favourite language, you can connect to the port 4000 and execute the above programs. IndiMail also provides you a command line program 'adminclient' which can execute any of the above programs. Read the man page for adminclient for further details.

Saturday, January 8, 2011

Distributing your outgoing mails from Multiple IP addresses

Some mail providers like hotmail, yahoo restrict the number of connections from a single IP and the number of mails that can be delivered in an hour from a single IP. To increase your ability to deliver large number of genuine emails from your users to such sites, you may want to send out mails from multiple IP addresses.

IndiMail has the ability to call a custom program instead of qmail-local(8) or qmail-remote(8). This is done by defining the environment variable QMAILLOCAL or QMAILREMOTE. qmail-remote(8) can use the environment variable OUTGOINGIP to set the IP address of the local interface when making outgoing connections. By writing a simple script and setting QMAILREMOTE environment variable pointing to this script, one can randomly chose an IP address from the control file

/var/indimail/control/outgoingip

The script below also allows you to define multiple outgoing IP addresses for a single host. e.g. you can create the control file to send out mails from multiple IPs only for the domain hotmail.com

/var/indimail/control/outgoingip.hotmail.com

Let us name the below script balance_outgoing

% su
# echo "/var/indimail/bin/balance_outgoing" > /service/qmail-send.25/variables/QMAILREMOTE
# svc -d /service/qmail-send.25
# svc -u /service/qmail-send.25
# exit
%

Finally the balance_outgoing script can be placed with execute bit in /var/indimail/bin

#!/bin/sh
# This scripts expects qmail-remote arguments on command line
# argv0          - qmail-remote
# argv1          - host   (host)
# argv2          - sender (sender)
# argv3          - qqeh   (qmail queue extra header)
# argv4          - size
# argv5 .. argvn - recipients
#
#
host=$1
sender=$2
qqeh=$3
size=$4
shift 4

cd /var/indimail
if [ " $CONTROLDIR" = " " ] ; then
    FN=/var/indimail/control/filterargs
else
    FN=$CONTROLDIR/filterargs
fi
if [ -n "$SPAMFILTER" -o -n "$FILTERARGS" -o -f $FN ] ; then
    # execute spawn-filter if you have filters defined for remote/local deliveries
    PROG="bin/spawn-filter"
else
    PROG="bin/qmail-remote"
fi
if [ " $CONTROLDIR" = " " ] ; then
    if [ -f /var/indimail/control/outgoingip.$host ] ; then
        IP=(`cat /var/indimail/control/outgoingip.$host`)
    elif [ -f /var/indimail/control/outgoingip ] ; then
        IP=(`cat /var/indimail/control/outgoingip`)
    else
        exec -a qmail-remote $PROG "$host" "$sender" "$qqeh" $size $*
    fi
else
    if [ -f $CONTROLDIR/outgoingip.$host ] ; then
        IP=(`cat $CONTROLDIR/outgoingip.$host`)
    elif [ -f $CONTROLDIR/outgoingip ] ; then
        IP=(`cat $CONTROLDIR/outgoingip`)
    else
        exec -a qmail-remote $PROG "$host" "$sender" "$qqeh" $size $*
    fi
fi
IP_COUNT=${#IP[*]}
if [ $IP_COUNT -gt 1 ] ; then
    i=`expr $RANDOM % $IP_COUNT`
    export OUTGOINGIP=${IP[$i]}
fi
exec -a qmail-remote $PROG "$host" "$sender" "$qqeh" $size $*

Monday, August 2, 2010

Handling Bounces yourself

IndiMail allows a mechanism by which you can use your own script/program to handle bounces. All bounces in IndiMail is generated by qmail-send. qmail-send generates a bounce when qmail-lspawn or qmail-rspawn reports a permanent failed delivery. A bounce is generated by qmail-send by injecting a new mail in the queue using qmail-queue. This bounce generation by qmail-send can be modified in three ways

1. Using environment variable BOUNCEPROCESSOR

When you define the environment variable BOUNCEPROCESSOR as a valid path to a program or script, the program gets called whenever a delivery fails permanently. The program runs with the uid qmails and is passed the following five arguments

bounce_file bounce_report bounce_sender original_recipient bounce_recipient

To set BOUNCEPROCESSOR, you would do the following

# echo "path_of_your_bounce_processor" > /service/qmail-send.25/variables/BOUNCEPROCESSOR

There are few email marketing companies who are using BOUNCEPROCESSOR to insert the status of all bounces in MySQL table for their email marketing campaigns.

2 Using environment variable BOUNCERULES or control files bounce.envrules.

Using envrules, you can set specific environment variables only for bounced recipients. The format of  this  file  is  of  the  form

pat:envar1=val,envar2=val,...]

where pat is a regular expression which matches a bounce recipient.  envar1, envar2 are list of environment variables to be set. If var is omitted, the environment variable is unset.

e.g.

support@indimail.org:CONTROLDIR=control2,QMAILQUEUE=/var/indimail/bin/qmail-nullqueue

causes all bounces generated for the sender support@indimail.org to be discarded.

3. Using BOUNCEQUEUE environment variable to queue bounces


qmail-send  uses  qmail-queue to queue bounces and aliases/forwards. This can be changed by using QMAILQUEUE environment variable. If a different queue program is desired for bounces, it can be set by using BOUNCEQUEUE environment variable.

e.g

# echo /var/indimail/bin/qmail-nullqueue > /service/qmail-send.25/variables/BOUNCEQUEUE

disables bounces system-wide. Though disabling bounces may not be the right thing to do but in some situations where bounces are not at all needed, disabling bounces will surely result in performance improvements of your system, especially so if your system does mass-mailing.

Tuesday, July 13, 2010

Delivery Instructions for a Virtual Domain

IndiMail uses a modified version of qmail as the MTA. For local deliveries, qmail-lspawn reads a series of local delivery commands from descriptor 0, invokes qmail-local to perform the deliveries. qmail-local reads a mail message and delivers to to a user by the procedure described in dot-qmail(5). IndiMail uses vdelivermail as the local delivery agent.

A virtual domain is created by the command vadddomain(1).

% vadddomain example.com some_password

The above command creates a virtual domain with delivery instructions in /var/indimail/domains/example.com/.qmail-default file. A line in this file is of the form

/var/indimail/sbin/vdelivermail '' delivery_instruction_for_non_existing_user

The delivery_instruction_for_non_existing_user can have one of the following 5 forms

  1. delete
  2. bounce-no-mailbox
  3. Maildir
  4. emailAddress
  5. IPaddress
Using delete as the delivery instruction causes IndiMail to discard all mails addressed to non-existing users. The original sender does not get notified of the delivery. On a real messaging system serving real users, you will not want to do this.

The instruction bounce-no-mailbox causes a bounce to be generated to the sender in case an email is addressed to a non-existing user. This is the most common usage in .qmail-default which most IndiMail installations will have

The instruction Maildir causes emails to be addressed to non-existing users to be saved in a Maildir. Here Maildir should refer to a full path of an existing Maildir.

The instruction emailAddress causes emails to be addressed to non-existing users to be forwarded to an email address emailAddress.

The instruction IPaddress causes emails to be addressed to non-existing users to be redirected to a remote SMTP server at IP IPaddress. The format of IPaddress is domain:ip:port where domain is the domain name, ip is the IP address of the remote SMTP server and port is the SMTP port on the remote SMTP server. It is expected that the non-existing user is present on the remote system. This type of delivery is used by IndiMail on a clustered setup. In a clustered setup, users are distributed across multiple server. A particular user will be located only on one particular server. However, the same domain will be present on multiple servers.

In the delivery instruction in .qmail-default, you can replace vdelivermail with vfilter to perform in-line filtering use IndiMail's poweful vfilter. You can create filters using the program vcfilter.

Thursday, July 8, 2010

Relaying - How does IndiMail handle relaying securely

A SMTP server is responsible for accepting mails from a sender and processing it for delivery to one or more recipients. In most situations, for domains which are under your administrative control (native addresses), the SMTP server should accept mails without authentication. However, when a mail is submitted for delivery to domains which are not under your administrative control, you should accept mails only after it satisfies security considerations like having the sender authenticate itself. This is to prevent abuse of external domains using your SMTP server. A SMTP server which accepts mails for external domains without any authentication is called an open relay. The act of accepting mails for external domains for delivery is called relaying.

The default configuration of IndiMail configures the SMTP as a closed system. Hence to be able to send mails to external domains, you need to setup mechanisms for relaying.

There are many methods. Choose any of the below after studying them. I prefer 3 or 4 for security reasons.
  1. Have Sender's IP addresses in tcp.smtp file
  2. Use control file relayclients for IP addresses of clients allowed to relay mail through this host.
  3. Configure IndiMail to use MySQL relay table (good security). This is implemented on POP3/IMAP before SMTP
  4. Use authenticated SMTP (good security)
  5. For allowing relay to specific domains use control file relaydomains
  6. For allowing specific users (native addresses) use control file relaymailfrom
  • NOTE: you should use 1 & 2 only if if the host having the sender's IP is under your control and you have good security policies for the host (however what is a good security can be very subjective)
Using tcp.smtp

Your startup script for the qmail smtp server must use the tcpserver -x file option similar to this startup line.

env - PATH="/var/indimail/bin" tcpserver -H -R -x /var/indimail/etc/tcp.smtp.cdb \
-c 20 -u 555 -g 555 0 smtp /var/indimail/bin/qmail-smtpd 2>&1

IndiMail uses -x option to tcpserver and hence you need not bother about the above line. You however need to edit /var/indimail/etc/tcp.smtp and put in lines for all static IP's that you will always want to relay access to.

127.0.0.:allow,RELAYCLIENT=””
10.1.1.:allow,RELAYCLIENT=””

The above lines will cause RELAYCLIENT environment variable to be set for localhost and all machines on the 10.1.1 class and hence allow to relay through. Remember that any user on hosts on 10.1.1 class will be able to relay mails. You many not want this. The line having 127.0.0. will allow any client on the IndiMail host to use SMTP and relay mails.

If you add any IP to tcp.smtp, you have to rebuild a cdb database tcp.smtp.cdb. You can run the following command

% sudo /var/indimail/bin/qmailctl cdb


NOTE: Remember that you are exposed to unrestricted relaying from any of the IP addresses listed in tcp.smtp

Using control file relayclients

IP addresses of clients allowed to relay mail  through  this  host.   Each address should be followed by a colon and an (optional) string that should be appended to each incoming recipient address, just as  with  the  RELAYCLIENT environment variable.  Nearly always, the optional string should be null. The filename can be overriden by  the  environment  variable  RELAYCLIENTS.

Addresses in relayclients may be wildcarded (2nd line in the example below):

 192.168.0.1:
 192.168.1.:

Using MySQL relay table

Run the command /var/indimail/bin/clearopensmtp in the cron every 30 Minutes

By default every time who uses IndiMail's  POP3 or IMAP service and authenticates, the following happens:
  1. On successful authentication, IMAP/POP3 daemon inserts entry into relay table, inserting email, IP address and timestamp.
  2. If CHECKRELAY environment variable is enabled, SMTP checks the relay table for a entry within minutes specified by the RELAY_CLEAR_MINUTES environment variable. If the entry is there, RELAYCLIENT environment variable is set, which allows relaying. At this point, the SMTP server will allow that IP to relay for 60 Mins (default)
clearopensmtp will clear all IP which have not authenticated in the past RELAY_CLEAR_MINUTESclearopensmtp should be enabled in cron to run every 30 minutes.

Set up Authenticated SMTP

IndiMail also provides you authenticated SMTP providing AUTH PLAIN, AUTH LOGIN and AUTH CRAM-MD5 methods. Whenever a user successfully authenticates through SMTP, the RELAYCLIENT environment variable gets set. qmail-smtpd uses the RELAYCLIENT environment variable to allow relaying.

Most of the email clients like thunderbird, evolution, outlook, outlook express have options to use authenticated SMTP.

For a tutorial on authenticated SMTP, you can refer to
http://indimail.blogspot.com/2010/03/authenticated-smtp-tutorial.html

Using control file relaydomains

Host  and  domain  names  allowed  to  relay mail through this host.  Each  
address should be followed by a colon and an (optional) string that should 
be  appended  to  each incoming recipient address, just as with the RELAY\
CLIENT environment variable.  Nearly always, the optional string should be 
null. 
Addresses in relaydomains may be wildcarded:


  heaven.af.mil:
  .heaven.af.mil:

Using control file relaymailfrom

envelope  senders (MAIL FROM) listed in this file will be allowed to relay independently of the RELAYCLIENT environment variable. Entries in relaymailfrom can be E-Mail addresses, or just the domain (with the @ sign).

Unlike relaydomains native addresses should be entered.  A line in  relay mailfrom  may be of the form @host, meaning every address at host.  relaymailfrom can also be in cdb format. If relaymailfrom.cdb  is  present,  it will be searched first.

Examples:
  joeblow@domain1.com
  @domain2.com

If you use the control file /var/indimail/control/relaymailfrom, you should really know what you are doing. Any mail from having a domain component of the address matching any domain in this file, relaying will be allowed without any authentication. You can most probably use this only if you have a closed SMTP server to which access from outside is not possible.

IndiMail Queue Mechanism

Indimail has the ability of configuring multiple local and remote queues. A queue is a location on your hard disk where email are deposited ...