/ / how to parse IMAP

So I'm writing an IMAP client for Emacs. The main reason is that I want an email client works as if by osmosis, like the one on my phone—whenever it has a connection, messages magically enter a local cache, where they stay available even if the connection is lost. There are many ways to achieve that with existing Emacs email clients, possibly augmented with external tools, but I want one that works with minimal setup, and requires nothing but Emacs itself.

The first problem encountered when writing an IMAP client is parsing what the server is sending. There are two IMAP parsers in the Emacs source tree already (imap.el and nnimap.el), both fairly closely tied to their respective clients, and both synchronous—they block Emacs while they are waiting for data from the server. I want to address both of those points: my IMAP client is going to have a reusable asynchronous parser module.

Obviously, the first step in figuring out how to parse the protocol is to read the IMAP RFC. Naturally, you'd skip ahead to the section "Formal syntax", which defines in ABNF what various messages should look like: a sequence number followed by one of a few keywords followed by a quoted string or an atom, etc etc. That's where I started, adding a special case for each command response into my parser code.

But then I thought that this wasn't the right way to do it. The parser module wouldn't be independent of the module that uses it, since the calling module might want to use an extension that the parser module doesn't know about yet. Also, I would constantly need to come up with sensible data structures to represent the responses with.

So I decided to create a parser that would be able to parse IMAP responses with as little knowledge as possible about what those responses should look like. It turns out that most of IMAP consists of atoms, strings and lists. I figured that I should be able to parse everything following these simple rules:

  • If it starts with a double quote, parse it as a quoted string.
  • If it starts with a ( or [, descend, parse recursively and return it as a list.
  • Otherwise, treat it as an atom, read until the next space character or closing ) or ], and return it as a string.

For example, this LIST response:

* LIST (\HasNoChildren) "." INBOX

gets parsed to this:

("*" "LIST" ("\\HasNoChildren") "." "INBOX")

This mostly worked. One exception is that if the second word of a line is one of OK, NO, BAD, BYE or PREAUTH, then the rest of the line (described by resp-text in the RFC) is free-form human-readable text, optionally preceded by a resp-text-code. For example, this is part of a response to a SELECT command from the Dovecot server:

* OK [UNSEEN 6] First unseen.
* OK [UIDVALIDITY 1381959933] UIDs valid
* OK [UIDNEXT 10] Predicted next UID
1 OK [READ-WRITE] Select completed (0.003 secs).

There is no reason why this free-form text couldn't contain unbalanced parentheses or anything else that might confuse a parser, and besides it doesn't make sense to split the text into words anyway. So that's a special case for the parser, and gets parsed to this:

("*" :ok :code "UNSEEN"      :data "6"          :text "First unseen.")
("*" :ok :code "UIDVALIDITY" :data "1381959933" :text "UIDs valid")
("*" :ok :code "UIDNEXT"     :data "10"         :text "Predicted next UID")
("1" :ok :code "READ-WRITE"  :data nil          :text "Select completed (0.003 secs).")

Then there is BODY, which when given as a fetch or message attribute is followed immediately by an opening square bracket, e.g. BODY[]. I decided to treat it as if there were a space between BODY and the bracket, and return them as two elements, the string "BODY" and the list of items parsed inside the brackets.

And all of this applies in principle to every single line, except that a physical line can end with a byte count in curly braces (e.g. {42}), which means that the following bytes are literal data to be treated as part of the current logical line. Fortunately, this is independent from the parsing itself, so I have a function that splits the data by logical lines and passes everything to the parse function.

So far, my little client is able to select a mailbox, search for unread messages and fetch the messages, and my parser is sufficient for all of that.