Understanding
POP and IMAP
The preceding discussion has outlined some
key points of mail delivery. To fully understand pull mail protocols,
though, it's useful to study this issue a bit more to better comprehend how
pull mail protocols can be used and how they interact with other e-mail
delivery systems. The two pull mail protocols described in this chapter are the
Post Office Protocol (POP) and the Internet Message Access Protocol (IMAP). The
discussion to this point has applied to both protocols, but there are important
differences between them, as described next. You might not want or need to use
both POP and IMAP, or you might have some active reason to not use one or the other protocol. Understanding the
tasks for which each protocol is most useful is very important when planning
your pull mail server implementation.
Pull
Mail's Place in the Mail Delivery System
As noted earlier, a pull mail server computer also normally
functions as a push mail serverwithout the push mail server, the pull mail
server won't have any e-mail to deliver to clients when it's queried. Where
does the mail come from, though? In most cases, the mail originates on one
computer, passes through one or more mail server computers using push mail
protocols, and arrives at the pull mail server system, where it waits for a
client to read the mail via the pull mail protocol.Unlike some protocols, e-mail delivery (especially push, but
also pull) is based on the idea of a relay.
Instead of delivering a message directly to the ultimate recipient computer,
the e-mail system is designed to allow each computer to pass the message closer
to the destination. Indeed, the sender may not be able to identify the true and ultimate
destination of the message. For instance, if you send a message to sammy@threeroomco.com , you may be able to
tell by checking Domain Name System (DNS) entries that the message will be sent
to the computer called mail.threeroomco.com .
This system, though, could be configured to forward mail addressed to sammy to another computer, such as gingko.threeroomco.com . The
user sammy might then use a pull
mail protocol to read the mail from a different computer, such as larch.threeroomco.com .
Similarly, when you send a message, you configure your mail package (let's say
it's on trilobite.pangaea.edu )
to relay the mail through some computer's push mail server (let's call it franklin.pangaea.edu ). That
computer could be configured to relay mail through another server ( osgood.pangaea.edu ). The
result can be a fairly long chain of mail servers involved in the delivery of a
message, as illustrated by href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11fig01#ch11fig01"> Figure 11.1 . Most of these transfers use the
Simple Mail Transfer Protocol (SMTP) push mail protocol; they're initiated by
the sender. Assuming the network and all intervening servers are functioning
smoothly, mail moves quickly from trilobite.pangaea.edu
to gingko.threeroomco.com .
This penultimate system, though, may sit on the message for quite some time,
waiting for a pull mail client system such as larch.threeroomco.com to retrieve the message. For
this reason, pull mail server computers may need large hard disks to store the
messages. As described shortly, this is particularly true of IMAP servers.
Figure 11.1. E-mail delivery can
involve several push mail relay systems, often concluding with a pull mail
transfer to the ultimate recipient.
mail delivery. Mail may involve as few as one computer (for mail from a user of
a computer to another user of the same system), or many more than are depicted
in href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11fig01#ch11fig01"> Figure 11.1 . There might be more systems
within the sender's domain or more within the recipient's domain. Other domains
may be involved in the transfer, especially if mail for specific users is
forwarded from one system to another, say because users have graduated from
college or moved on from one job to another. As described in the upcoming
section, "href="http:// /?xmlid=0-201-77423-2/ch11lev1sec5#ch11lev1sec5"> Using Fetchmail ," it's even possible for
a pull mail transfer to occur in something other than the last leg of the
mail's journey. In such a configuration, one system pulls mail using Fetchmail
or a similar program, then passes it on again, perhaps to a local mail queue
maintained by a push mail server on the same system, or possibly to another
computer altogether.One important point to remember is that pull mail servers are
used for retrieving mail, not sending it. Sending mail is accomplished with a push
mail protocol like SMTP. A single mail server computer
may operate as both a push and pull server, so an end user may both retrieve
incoming mail from and send outgoing mail through a single computer, but the
protocols involved are different. Some organizations separate these two
functions. For instance, you might specify franklin.pangaea.edu as your outgoing (SMTP) mail
server, but use ponyexpress.pangaea.edu
as your incoming (POP or IMAP) mail server. The latter computer would also run
an SMTP server, but it might not be accessible to end users; it might be used
only for accepting mail from the organization's externally-accessible push mail
server.
Storing
Mail: On the Client or the Server
As described earlier, one role of a pull mail server is to
store mail until the recipient can retrieve it. At that time, the recipient's
mail reader contacts the mail server and downloads the messages. You might
infer from this description that the mail is then deleted from the mail server.
This inference is often correct, but it need not be. There are circumstances in
which the mail server might retain a copy of the mail: With either POP or IMAP, you can configure a mail reader to
download messages but leave the originals on the server. You might do this if
you regularly use two different mail readers or computers. With POP, however,
this procedure can quickly become awkward, because POP has only very primitive
provisions for identifying and manipulating individual messages. Therefore, as
the number of messages on the server grows, this method can become extremely
awkward. With IMAP, you can organize messages into message
"folders"on the server. Thus, if you
regularly use several different computers, you can organize your messages on
the IMAP server and not wade through duplicate messages when you switch
computers. IMAP also includes the ability to retrieve only message headers (information such as the subject and sender
address) without the message bodies (the bulk
of the text and attachments), so you need not waste undue bandwidth to check
your mail.These distinctions between POP and IMAP are unimportant in
some situations, but crucial in others. If you use just one computer and want
to store and organize your messages locally, POP and IMAP both do the job. If
you switch computers regularly or use multiple mail packages on a single
computer, IMAP can be much more convenient. On the other hand, the network
transfers involved mean that storing messages on the server could cause you to
wait longer to read any given message, particularly if you store many messages
and read old messages on a regular basis. From the point of view of
administering the pull mail server, though, IMAP offers more challenges,
because it increases the demand for disk storage and network bandwidth, as
users may recover the same message multiple times. (To be sure, though, users
who use IMAP just like POP won't impose such additional overhead for IMAP; it's
only if users take advantage of IMAP's extra capabilities that you'll see
increased demands on the server's hardware.)
A
Sample POP Session
POP is actually several different protocols. The most common
version today is POP-3, which uses TCP port 110. (The earlier POP-2 uses port
109.) Like many Internet protocols, POP is built around text-based commands
sent by the client and text-based responses sent by the server. POP-3 supports
about a dozen commands, including USER
(to specify a username), PASS
(to specify a password), RETR
(to retrieve a message), DELE
(to delete a message), and QUIT
(to terminate a session). Mail readers that support POP can send these and
other POP commands to a POP server, capture the results, and use the results to
store mail and present it to users in an intelligible fashion.As an example of POP in use, consider href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list01#ch11list01"> Listing 11.1 , which shows a simple POP session
in which a single message is retrieved. This listing uses telnet to connect to the POP-3 port and
issue commands manually. Most POP sessions hide these details behind the mail
reader, but it's informative to see what the mail reader and POP-3 server do to
exchange messages.
Listing
11.1 A sample POP-3 session
$ telnet nessus 110 Trying 192.168.1.3... Connected to nessus.rodsbooks.com. Escape character is '^]'. +OK POP3 nessus.rodsbooks.com v7.64 server ready USER rodsmith +OK User name accepted, password please PASS password +OK Mailbox open, 1 messages RETR 1 +OK 531 octets >From rodsmith Wed Aug 8 14:38:46 2001 Return-Path: <ben@pangaea.edu> Delivered-To: rodsmith@nessus.rodsbooks.com Received: from speaker.rodsbooks.com (speaker.rodsbooks.com [192.168.1.1]) by nessus.rodsbooks.com (Postfix) with SMTP id EB2A01A2BD for <rodsmith@nessus.rodsbooks.com>; Wed, 8 Aug 2001 14:38:26 -0400 (EDT) Message-Id: <20010808183826.EB2A01A2BD@nessus.rodsbooks.com> Date: Wed, 8 Aug 2001 14:38:26 -0400 (EDT) From: ben@pangaea.edu To: undisclosed-recipients:; Status: This is a test message. . DELE 1 +OK Message deleted QUIT +OK Sayonara Connection closed by foreign host.
As you can see from href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list01#ch11list01"> Listing 11.1 , POP refers to messages by
number. In this example, the server had only one message available, as
indicated by the message +OK mailbox open, 1
messages . Messages may be retrieved or deleted by number, as in the
RETR 1 and DELE 1 commands. Messages must be
retrieved in full or not at allPOP provides no mechanism for discovering a
message's sender, length, or other important information prior to retrieving
it. Much of this information is included in the
message itself, though. In this example, the bulk of the message consists of
its headers lines that include information on
the servers through which the message has passed, the sender's address, and so
on. A real message is more likely to consist of more lines of the message than
of headers.NOTE

The headers shown in href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list01#ch11list01"> Listing 11.1 illustrate a feature of e-mail
that's both flexible and frustrating: It can be difficult to determine the
true sender of the mail. You'll note From:
and Return-Path: headers that
indicate the message was sent by ben@pangaea.edu .
Those who are used to reading e-mail headers, though, know that these headers
are easily forged. Every mail server adds a Received:
header to specify who the server is and from where it received the message. I
sent this sample message directly from one computer on my network to another
one, and the Received: header
reflects this fact, showing the message having come from speaker.rodsbooks.com and
being delivered to nessus.rodsbooks.com .
No computers in pangaea.edu
were involved in the transfer.
A Sample IMAP Session
In very broad outline, IMAP fills the same role as POPboth
are pull mail protocols. IMAP, however, provides finer-grained control over
messages. IMAP includes the ability to examine headers before retrieving the
body of the message, so a user can decide whether to download a message. (From
the user's point of view, a mail reader may present information on the
available messages, allowing the user to choose which to read in detail at any
given moment.) These additional features require additional commands: a total
of about two dozen in IMAP-4, the current version of the protocol, which uses
TCP port 143. href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list02#ch11list02"> Listing 11.2 shows an IMAP retrieval session
that's roughly equivalent to href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list01#ch11list01"> Listing 11.1 , except that it includes a
command to copy the message into an IMAP folder.
Listing
11.2 A sample IMAP-4 session
$ telnet nessus 143 Trying 192.168.1.3... Connected to nessus.rodsbooks.com. Escape character is '^]'. * OK nessus.rodsbooks.com IMAP4rev1 v12.264.phall server ready A1 LOGIN rodsmith password A1 OK LOGIN completed A2 SELECT Inbox * 1 EXISTS * NO Trying to get mailbox lock from process 29559 * 1 RECENT * OK [UIDVALIDITY 997295985] UID validity status * OK [UIDNEXT 4] Predicted next UID * FLAGS (\Answered \Flagged \Deleted \Draft \Seen) * OK [PERMANENTFLAGS (\* \Answered \Flagged \Deleted \Draft \Seen)] Permanent flags * OK [UNSEEN 1] first unseen message in /var/spool/mail/rodsmith A2 OK [READ-WRITE] SELECT completed A3 FETCH 1 BODY[HEADER] * 1 FETCH (BODY[HEADER] {494} >From rodsmith Wed Aug 8 16:02:47 2001 Return-Path: <ben@pangaea.edu> Delivered-To: rodsmith@nessus.rodsbooks.com Received: from speaker.rodsbooks.com (speaker.rodsbooks.com [192.168.1.1]) by nessus.rodsbooks.com (Postfix) with SMTP id 2C7121A2BD for <rodsmith@nessus.rodsbooks.com>; Wed, 8 Aug 2001 16:02:25 -0400 (EDT) Message-Id: <20010808200225.2C7121A2BD@nessus.rodsbooks.com> Date: Wed, 8 Aug 2001 16:02:25 -0400 (EDT) From: ben@pangaea.edu To: undisclosed-recipients:; ) * 1 FETCH (FLAGS (\Recent \Seen)) A3 OK FETCH completed A4 FETCH 1 BODY[TEXT] * 1 FETCH (BODY[TEXT] {25} This is a test message. ) A4 OK FETCH completed A5 COPY 1 demos A5 OK COPY completed A6 LOGOUT * BYE nessus.rodsbooks.com IMAP4rev1 server terminating connection A6 OK LOGOUT completed Connection closed by foreign host.
href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list02#ch11list02"> Listing 11.2 illustrates some of the added features and complexity of IMAP over POP. IMAP
requires the client to submit numbered commands, such as A6 LOGOUT rather than simply LOGOUT . This detail is unimportant from the user's point of view, because
the mail reader handles it. IMAP allows the client to retrieve the message
headers separately from the body, as illustrated by the A3 and A4 commands in
href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list02#ch11list02"> Listing 11.2 . The
use of folders requires the client to move into a folder (the A2 command in href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list02#ch11list02"> Listing 11.2 )
before reading messages, and allows the client to copy messages into folders
(the A5 command in href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list02#ch11list02"> Listing 11.2 ). href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list02#ch11list02"> Listing 11.2 illustrates a smaller fraction of what IMAP can do than does href="http:// /JVXSL.asp?x=1&mode=section&sortKey=insertDate&sortOrder=desc&view=&xmlid=0-201-77423-2/ch11lev1sec2&open=true&title=New%20This%20Week&catid=&s=1&b=1&f=1&t=1&c=1&u=1#ch11list01#ch11list01"> Listing 11.1 with
respect to POP. There are many variants on the way these commands can be used,
such as different message features that can be retrieved with FETCH . For
more information on what these commands can do, consult an IMAP reference, such
as the IMAP RFC ( target="_blank">http://www.ietf.org/rfc/rfc2060.txt ).Although a quick look at the low-level
commands used in IMAP can be informative, you probably don't need to be too
concerned about them as an IMAP administrator. Some of these commands do have
consequences, however. Specifically, the ability of IMAP to use folders means
that the IMAP server must have somewhere to store those folders. The most common
IMAP server is the University of Washington IMAP (UW IMAP; href="http://www.washington.edu/imap/" target="_blank">http://www.washington.edu/imap/ ). This package stores IMAP folders in the user's home directory,
except for the INBOX folder, which corresponds to the standard mail spool directory (at /var/spool/mail/username on most systems). When a user first connects using IMAP, there will
be no folders defined, aside from the default INBOX . Users can create
folders using appropriate mail reader commands, and the UW IMAP server responds
by creating a file in the user's home directory in which mail messages placed
in that folder will reside. Other packages may store mail folders in other
locations; consult their documentation for details. You should be aware of
where these messages are stored so that you can allocate adequate disk space
for them, particularly on large servers or those on which users will be storing
large messages.
Determining Which to Use
The choice of POP or IMAP is partly a matter
of personal preference, partly one of the capabilities of your user's mail
readers, and partly one of your available resources. POP minimizes your need
for disk space and network speed on the mail server, because users will be more
likely to download and permanently store e-mail on their local computers rather
than leave the e-mail on the server for later perusal or archival purposes. For
this reason, an IMAP server is likely to require more disk space on the server,
and possibly more network bandwidth to clients. This server may be preferable
to at least some users, though. The vast majority of modern mail readers
support POP. IMAP support is common, but not as prevalent as is POP support. Therefore,
if you provide IMAP, it's usually a good idea to provide POP as well. The added
load on the server should be minimal, but it will allow users who prefer mail
readers that don't support IMAP to retrieve their mail. A POP client may clean
out a user's inbox, but it won't disrupt folders created via IMAP. On the other
hand, running POP also means leaving another port open. If the POP server turns
out to have a security flaw, this could allow miscreants access to your system.WARNING

By default, both POP and IMAP send all
information, including passwords, in an unencrypted (cleartext) form. Therefore,
if possible you should use POP or IMAP passwords only
for e-mail retrieval, not for other purposes, such as regular logins. There
are secure variants of POP and IMAP available, such as versions that encrypt
the transfers using the Secure Sockets Layer (SSL).
These are more difficult to configure, but well worth the effort if security
is a concern. This is particularly important when e-mail is likely to be
accessed over the Internet, rather than restricted to a local network segment
that you control. If you want to restrict access to your pull mail servers to
your local network, you can apply TCP Wrappers or xinetd rules, as described in href="http:// /?xmlid=0-201-77423-2/ch04#ch04"> Chapter 4 ,
Starting Servers. Another option is to create packet filter firewall rules,
as described in href="http:// /?xmlid=0-201-77423-2/ch25#ch25"> Chapter 25 ,
Configuring iptables.