Having to deal with Microsoft's outlook has been a growing pain with my computing needs. Of course, this is pertinent only for work email. For personal email, I use Migadu which has good standards for email unlike Microsoft's.

At work, I can only use either the official Outlook webmail or Thunderbird since those are the only clients that are whitelisted by the IT department at my university. Thunderbird is slightly better (it has a few problems of its own but I digress) but I would like to have an offline copy of all my emails, fully searchable with an indexed database provided by notmuch, which uses the excellent Xapian library for indexing. Notmuch is shipped as an emacs package in the form of notmuch.el, which is exactly what I like given that I'm moving most of my computing needs into emacs¹.

I've taken inspiration from Fergus' excellent blogpost which implements the same in neomutt. The initial setup process is identical to Fergus' blogpost.

It should be noted that notmuch is only a mail indexer. It does not fetch mail, nor does it send mail. Those are handled by isync and msmtp, which are commandline programs.

Access authentication with O365

Since Thunderbird is still allowed to access O365's authentication, we will use the client ID provided from it (this is publicly available from Thunderbird's source code)². For authentication for IMAP and SMTP, we will use XOAUTH, which is a simple authentication and security layer (SASL) more suited for email clients.

Tools like isync rely on SASL for authentication, for which we'll require an XOAUTH SASL plugin: https://github.com/moriyoshi/cyrus-sasl-xoauth2.

If you're on Arch, then this plugin is installable from the AUR:

paru -S cyrus-sasl-xoauth2-git

If not, then we can build it from source:

git clone https://github.com/moriyoshi/cyrus-sasl-xoauth2
cd cyrus-sasl-xoauth2/
./autogen.sh   # checks for requirements
./configure    # configure (here we use the defaults)
make           # compile
sudo make install     # install for root user

Token generation for OAuth

Microsoft has conditionally allowed API access to clients and the only way to get access for any client is to register the application in the Microsot Application portal, which is something only my org's IT department can do³. Fortunately, we can use Thunderbird's credentials to generate an access token for our purposes.

Another issue that comes up is the finite lifetime of these access tokens (typically an hour or two), which is cumbersome to generate each time one needs to download email. Fortunately, mutt comes with mutt_oauth2.py, a script to help in the automatic renewal of this token. This script helps in decrypting the token file and renewing it everytime it is invoked.

Fetching the initial token is fairly simple but before we do it, there are certain changes that we need to do to mutt_oauth2.py.

Look into the ENCRYPTION_PIPE variable (line 48) and make sure to specify your gpg key. If you don't have one, then generate one:

gpg --full-generate-key

Select the default options (ed25519 cypher if it allows) and make sure that the expiration of the key is set to 0 (no expiration). This makes your life easier but of course for security reasons, this is not a good practice.

Bring up your public key:

gpg --list-keys

Copy the second line (after the pub line) and paste it into mutt_oauth2.py

ENCRYPTION_PIPE = ['gpg', '--encrypt', '--recipient', 'YOUR-GPG-KEY']

After this, look at line 79. This includes a client_id for microsoft. For Thunderbird, the client id is 9e5f94bc-e8a4-4e73-b8be-63364c29d753. Leave the client_secret value as is. It should look like so:

'client_id': '9e5f94bc-e8a4-4e73-b8be-63364c29d753',
'client_secret': '',

Now we are ready to generate the access token. Simply run:

./mutt_oauth2.py OUTPUT_TOKEN_FILE --verbose --authorize

A series of prompts should appear. Select microsoft, localhostauthcode (or authcode), followed by your email address, after which you should automatically be redirected to a browser for adding your credentials for authentication. Doing so, should reveal the access token in your terminal. No need to copy it, it is present in your OUTPUT_TOKEN_FILE which is encrypted. To obtain the decrypted token, we run the script again with just the file name as an argument

./mutt_oauth2.py OUTPUT_TOKEN_FILE

A nice thing is that running this script with the token file also checks for expiry and renews it if necessary. This can be done multiple times without worry.

UPDATE: About 20 days after I set this up, for whatever reason the token file expired on my personal laptop. I'm not entirely sure what happened but the fix for now was to regenerate the token again. I'll need to debug the exact cause to prevent it from happening again.

Offline email setup

Now that we are done with the difficult part, we move onto setting up isync (the binary is called mbsync) which is available in most distributions. This program basically downloads your emails offline and synchronizes it with your email's remote IMAP servers. It supports multiple accounts too. The program is configurable with a ~/.mbsyncrc file⁴. I highly recommend reading the manual for it via man mbsync. Here's a starting template:

# -- Global defaults
# These will be applied to all accounts

# Create new mail in either location, so if the remote or local has mail
# and the other does not, then create it
Create Both
# never remote mail from either side
Remove None
# Remove messages marked for deletion from the local side on the
# remote. Do not ever delete local if marked for deletion in the
# remote. Prevents admins from deleting one's local mail
Expunge Far
Sync All
SyncState *
CopyArrivalDate yes
# No maximum number of messages
MaxMessages 0

# -- First email account
IMAPAccount university
Host outlook.office365.com
Port 993
User my-username@university.edu
AuthMechs XOAUTH2
PassCmd "~/.local/bin/mutt_oauth2.py ~/.local/bin/TOKEN"
# Use TLS
TLSType IMAPS
SystemCertificates yes
Timeout 10

# A store defines a collection of mailboxes, so we associate the
# remote IMAP mailbox with the account we just configured
IMAPStore university-remote
# Associate the store with our account
Account university

# Where are we going to keep the mail associated with this account
MaildirStore university-local
SubFolders Verbatim
Path ~/.local/share/mail/university/
Inbox ~/.local/share/mail/university/INBOX

# Channel to synchronize everything except "Sent" email
Channel uni-main
Far :university-remote:
Near :university-local:
# Which directories to synchronize
Patterns INBOX "Deleted Items" "Drafts"

# Channel to synchronize "Sent" email
Channel uni-sent
Far :university-remote:"Sent Items"
Near :university-local:Sent

# Group the two channels and sync everything with `mbsync NAME`
Group university
Channel uni-main
Channel uni-sent

# -- Second email account
IMAPAccount personal
...

Once you have everything configured, we can start downloading all our email by running:

mbsync university

The first time mbsync is run, it will take a fair amount of time, since you'll be downloading a lot of email (assuming you've been using your account for a long time). The subsequent runs would be just incremental downloads, so it's much faster.

After navigating to your local Maildir directory (~/.local/share/mail/university in my case), you should see all your emails downloaded and categorized into inbox, drafts, sent, and deleted-items folders.

A way to automate this to run a script periodically. I use systemd-timers where I have a script which is periodically run every 15 mins and notifies me if there's new email (I'll cover this later in the blogpost).

Sending mail via SMTP

Sending email is done via msmtp. It is configured by editing ~/.msmtprc⁵.

defaults
auth on
tls on
tls_trust_file system
logfile ~/.cache/msmtp.log
timeout 10

# -- First account
account university
tls_starttls on
host smtp.office365.com
port 587
auth xoauth2
user my-username@university.edu
passwordeval "~/.local/bin/mutt_oauth2.py ~/.local/bin/TOKEN"
from my-username@university.edu

# -- Second account
account personal

This is fairly self-explanatory. One thing to note is that tls_starttls should be on for this to work. I'm not very familiar with email security protocols like TLS and STARTTLS but from my understanding, STARTTLS is an automatic upgrade (if supported) from the more insecure TLS protocol. TLS however needs to be enabled for any email service, otherwise it won't work.

Now, we can test our configuration by sending email directly from the commandline. Save the contents of a file called example-mail as follows:

From: my-username@university.edu
To: user@example.com
Subject: Hello World

This is a test for msmtp.

And send the email by running:

cat example-mail | msmtp -t -a university

Indexing email offline with notmuch

We have receiving and sending email configured. How about reading and "using" email via an email client. In more technical terms, an email client is called a mail user agent (MUA), which is typically seen in email headers. This includes clients like Thunderbird, Apple mail, neomutt, etc. We will configure notmuch which indexes email and provides an interface for emacs natively with notmuch.el.

Notmuch is entirely tag based. Basic tags exist like unread, inbox, draft, sent but you can customize it with more tags to filter out emails from a particular email address, from a particular range of date, etc. Tags are heavily used and it's a different paradigm of handling email compared to your typical webmail interface. I won't go over the details of tagging in notmuch here for the sake of brevity (to be covered in another blogpost).

When first setting up notmuch, simply run:

notmuch setup

This will prompt you for basic information like your name, email address, Maildir directory, etc. Once this is done, notmuch needs to index your Maildir. Run the following:

notmuch new

This should be fairly fast, even if your Maildir is big. Notmuch is quick like that.

Now you can search an email entry via notmuch search. For example, I can search emails from a particular sender like so:

notmuch search 'from:friend@university.edu'

I would again advise you to go through the man pages (man notmuch-search) for more details.

Emacs setup for notmuch

Simply running M-x notmuch should give you a hello screen, where you can then select tags like inbox, unread to view their respective email. I have a fairly customized config for notmuch.el but below I provide a fairly good starting point for a more minimal notmuch interface⁶:

;; Notmuch for email
(use-package notmuch
  :load-path "/usr/share/emacs/site-lisp/"
  :ensure nil
  :defer t
  :commands (notmuch notmuch-mua-new-mail)
  :init
  ;; Search
  (setq notmuch-search-oldest-first nil)
  :config
  ; General UI
  (setq notmuch-show-logo nil
	notmuch-column-control 1.0
	notmuch-hello-auto-refresh t
	notmuch-hello-recent-searches-max 20
	notmuch-hello-thousands-separator ""
	notmuch-hello-sections '(notmuch-hello-insert-saved-searches)
	notmuch-show-all-tags-list t)

  ; Search
  (setq notmuch-search-result-format
        '(("date" . "%12s  ")
          ("count" . "%-7s  ")
          ("authors" . "%-20s  ")
          ("subject" . "%-80s  ")
          ("tags" . "(%s)")))
  (setq notmuch-tree-result-format
        '(("date" . "%12s  ")
          ("authors" . "%-20s  ")
          ((("tree" . "%s")
            ("subject" . "%s"))
           . " %-80s  ")
          ("tags" . "(%s)")))
  (setq notmuch-show-empty-saved-searches t)

  ; Tags
  (setq notmuch-archive-tags nil ; I don't archive email
	notmuch-message-replied-tags '("+replied")
	notmuch-message-forwarded-tags '("+forwarded")
	notmuch-show-mark-read-tags '("-unread")
	notmuch-draft-tags '("+draft")
	notmuch-draft-folder "university/Drafts"
	notmuch-draft-save-plaintext 'ask)

  ; Email composition
  (setq notmuch-mua-compose-in 'new-window)
  (setq notmuch-mua-hidden-headers nil)
  (setq notmuch-address-command 'internal)
  (setq notmuch-address-use-company nil)
  (setq notmuch-always-prompt-for-sender t)
  (setq notmuch-mua-cite-function
	'message-cite-original-without-signature)
  (setq notmuch-mua-user-agent-function nil)

  (setq notmuch-show-relative-dates t)
  (setq notmuch-show-all-multipart/alternative-parts nil)
  (setq notmuch-show-indent-messages-width 0)
  (setq notmuch-show-indent-multipart nil)
  (setq notmuch-show-part-button-default-action 'notmuch-show-view-part)
  (setq notmuch-wash-wrap-lines-length 120)
  (setq notmuch-unthreaded-show-out nil)
  (setq notmuch-message-headers '("To" "Cc" "Subject" "Date"))
  (setq notmuch-message-headers-visible t)

  :bind
  ( :map global-map
    ("C-c m m" . notmuch)
    ("C-x m" . notmuch-mua-new-mail) ; override `compose-mail'
    :map notmuch-search-mode-map
    ("/" . notmuch-search-filter) ; alias for l
    ("r" . notmuch-search-reply-to-thread) ; easier to reply to all by default
    ("R" . notmuch-search-reply-to-thread-sender)
    :map notmuch-show-mode-map
    ("r" . notmuch-show-reply) ; easier to reply to all by default
    ("R" . notmuch-show-reply-sender)
    :map notmuch-hello-mode-map
    ("J" . notmuch-jump-search)))

You're free to look at my init.el for my own customization for notmuch.el.

Automating email synchronization with systemd timers

Let's automate the email sync to happen every 15 mins (or whatever frequency you prefer). First we create a shell-script to download email via mbsync and also index the email with notmuch. Let's name it it syncmail.sh and place it in $HOME/.local/bin/.

#!/bin/sh
set -eu

# Set environment variable for notmuch config file
export NOTMUCH_CONFIG="$HOME/.config/notmuch/notmuch-config"

mbsync university

before=$(notmuch count --lastmod '*' | cut -f3)
notmuch new > /dev/null

query="lastmod:$((before + 1)).. and path:university/**"
count=$(notmuch count "$query")

if [ "$count" -gt 0 ]; then
    body=$(notmuch search --format=json --limit=5 "$query" \
	       | jq -r '.[] | "\(.authors): \(.subject)"')
    notify-send "New mail: $count" "$body"
fi

In addition to downloading email and indexing, the script also notifies through notify-send if there's new email along with a sender and subject in the notification. Make sure to make the script executable with

chmod +x ./syncmail.sh

Then we write the systemd script to run this script periodically. First we create a syncmail.service file in $HOME/.config/systemd/user/:

[Unit]
Description=Sync mail with mbsync and index with notmuch

[Service]
Type=oneshot
ExecStart=%h/.local/bin/syncmail.sh

Then we create a timer file (make sure it is named syncmail.timer) for actually running the script every n minutes.

[Unit]
Description=Run syncmail every 15 minutes

[Timer]
OnBootSec=2m
OnUnitActiveSec=15m
Persistent=true

[Install]
WantedBy=timers.target

The advantage over cronjobs is that if your system is asleep or shutdown, then booting up accomodates accordingly and runs the script after m minutes (2 minutes in my case).

Finally we enable the script with systemctl:

systemctl --user daemon-reload
systemctl --user enable --now syncmail.timer

Concluding remarks

Now you should have a working setup for using offline email for Microsoft's O365. This is fairly minimal but if you'd like to have encrypted emails, addressbooks, etc., please take a look at the original blogpost from Fergus.

A basic emailing etiquette I like to follow is to use plaintext as opposed to html. Read more on why this is better.