Adding a References Header in Postfix: Better Gmail Conversation View

Posted on by

Background

If you didn’t know, Google recently changed the way it groups emails into threads, or as it calls them, conversations. Previously Gmail used the subject and sender information to group emails. This new method relies on explicit linking through the References header, which is much better.

Lots of our network equipment sends email notifications without the References header, and this confuses conversation view when the emails have the same subject. This causes problems especially when an appliance sends emails about different events all with the same subject line. It makes it hard to group the separate events.

My Solution

All we have to do is somehow add a References header with a random value (random because if a References header exists but is unique, the message won’t be threaded even if it has the same subject and is from the same sender as another message). I decided to use part of the Message-ID header as my source of random data (not being able to generate random data the way this solution works). Message-ID is a unique id for each message. This suits my purpose as only when the References header matches up with a previously sent message’s corresponding Message-ID header will threading occur.

If you use an on-premises SMTP server you may be in luck, because you might be able to process the messages before they are sent.

At work we use an onsite Postfix SMTP server for handling the sending emails from photocopiers, network appliances, web filters etc. None of them use the References header which is painful. Using the header_checks config statement in /etc/postfix/main.cf, we can easily add this in.

According to RFC 2076 the References header has the following purpose:

In email: reference to other related messages, in Usenet News: reference to replied-to-articles.

According to RFC 822, the References header is a phrase or msg-id. A msg-id is something that looks like an email address surrounded by angled brackets, eg: <12345.12345@somedomain.com.au>. What I decided to do was to take the value of the Message-ID header and mash it up a bit then use that as the value for the new References header. The resulting config is reproduced below:

In the file /etc/postfix/main.cf:

...
header_checks=pcre:/etc/postfix/header_checks
...

In the file /etc/postfix/header_checks (which you may have to create):

/^Message-ID:\s+<([0-9])\.([0-9])@mydomain.com.au>/ PREPEND References: <$2$1@unique.dummy.id.com.au>

This statement starts with a regular expression that looks for a Message-ID header and captures the required information out of it. Then there is the PREPEND command which prepends the following text to the header. The prepended text is itself a regex which pieces together the parts extracted in the first regex. The effect of this is to insert a dummy References header.

So far this is working in testing as expected and will be moved to the production SMTP server soon.

Limitations

The solution I present above has a few limitations:

  • if the Message-ID header doesn’t exist this won’t work
  • it doesn’t check for the existence of a References header before making a modification. In my case I know the References header doesn’t exist so I am ok for now. This may break in the future
  • I can’t limit the application of this modification by sender or any other value
  • If the Message-ID header is not made up of two strings of digits separated by a full stop, this won’t work.

Other ways this could work is a Postfix filter or milter. I am not sure about them, but they warrant further investigation. When my solution breaks, I will have a look into it.

Fin

I hope this helps someone who, like me, likes conversation view in Gmail, but has to deal with many emails a day that get grouped into conversations when they really shouldn’t.

References

https://support.plesk.com/hc/en-us/articles/115004515714-How-to-rewrite-headers-in-outgoing-mail-messages

Tagged vs. Untagged VLANs

Posted on by

I finally got it through my head today the difference between adding a port to a VLAN in tagged vs. untagged mode.

Tagged means that traffic with the VLAN ID already attached—that is tagged traffic—will be passed. Untagged means that traffic without the VLAN ID already attached will have the VLAN ID inserted—that is, it will be tagged—and then passed. That way if there is a device that you want to be on a particular VLAN, put the port the device is connected to into untagged mode for whatever VLAN you want, and the traffic from that device will be tagged with the VLAN ID. Tagged ports are trunked ports, that is you can assign multiple VLANs to a port and it will allow all traffic with the appropriate tags to pass.

Please feel free to correct me in the comments if you think this is incorrect.

Reset Profile Manager in OS X El Capitan

Posted on by

If you’ve needed to fully reset Profile Manager and you’re running OS X El Capitan and the server app version 5.1.5, you’ll not find a whole lot of information. Luckily the process is almost exactly the same as outlined for Mavericks in this official Apple support page. We only have to change the last line, because the shell script it refers to has moved in El Capitan. Follow these steps to fully reset the Profile Manager:

sudo serveradmin stop devicemgr
sudo serverctl disable service=com.apple.DeviceManagement.devicemgrd
sudo serverctl disable service=com.apple.DeviceManagement.postgres
sudo mv /Library/Server/ProfileManager/Config/ServiceData/Data/PostgreSQL ~/.Trash
sudo mv /Library/Server/ProfileManager/Config/ServiceData/Data/backup ~/.Trash
sudo /Applications/Server.app/Contents/ServerRoot/usr/libexec/deviceManagerCommon.sh

I hope this helps you out. Let me know if there are any problems.

Executing an arbitrary function from a PHP file on pfSense

Posted on by

I accidentally overwrote the bind service’s startup script in /usr/local/etc/rc.d/. I had no backup, and I couldn’t remember the contents of the file. I had a look at the pfSense github packages repository and had a look through the bind section. I found bind.inc which had a function called bind_write_rcfile(). I toyed around with the idea of copying and pasting the code by hand, replacing the variables with their values (or what I thought were the values). I remembered that pfSense has a developer shell which allows you to execute PHP code. You type in the PHP and then type exec and press enter. The code gets executed. Yay. I required_once('bind.inc') and then pressed enter, then typed exec and again pressed enter. I jumped into the shell and checked the contents of the file, and hey presto! it was fixed.

This post is for my future reference. I hope it helps you too!

Manny in Hospital

Relay for Life

Posted on by

Some months ago, almost my entire family took part in the Relay for Life, a 24-hour fundraising event. I posted about it beforehand. We raised over AU$20,000. Below is a speech I gave during the formalities.

On Fighting Back

Since my son Emmanuel was diagnosed with liver cancer when he was 11 months old, I have been reflecting on the idea of the ‘battle with cancer’ and how it could apply to one so young. Was he too young to be able to battle anything, let alone cancer? Does he even know what is going on? Is he capable of fighting back? I am not sure if Little Manny is in the way you or I are, but that doesn’t matter because I am not really a good fighter, I give up too easily. But Manny has shown me what it means to be a good fighter. He has shown me to always be positive, to keep keeping on, not lie down and let things overwhelm me, to walk forward with my head held high and my eyes on the horizon. Live in the moment, don’t wallow in misery for too long or you’ll miss out on what’s happening now, which you may never be able to get back. After our experiences with Manny, I know that if I ever have to go through what he has, I will know what to do. He has been the example for me.

He has been poked, pricked, prodded, pushed, pulled, and pumped full of more drugs than I care to remember. He has had major surgery which has left him with half a liver (the only organ that is capable of regenerating). I’ve never had a general anaesthetic; Manny’s had about 5. He’s been through the wash, wrung out and hung to dry. Multiple times. But he’s not let that stop him. Cancer? Pah, he says! Chemotherapy? Bring it on, he cries! All Manny cares about are his broom-brooms and throwing balls around the house. And milk from Mummy, but that’s another story. Nothing has stopped him smiling and laughing up at us for too long, or waving at the nurses and doctors, or at any random person for that matter. He still gets sick with excitement when he gets his favourite truck to play with, or when its time to read ‘Where is the Green Sheep?’ or any of his other favourite books (again). Manny has the resilience of children, but he also has a friendly and outgoing nature that makes the battles faced by friends and family more easy to bear.

We are all fighters, all of us here. Just by being here, taking part in an event such as this, we are all part of the global search, the fight for a cure for cancer. Everyone that donated or helped or supported is part of the fight. And the greatest weapon we have is hope. We must never lose hope. We must never lose hope.

Going Up Quickly

Posted on by

Untitled

Time-saving Bash Techniques

If you are a nerd and love hacking away at your UNIX-like machine’s command line, then chances are you know that things can get really repetitive very quickly. One of my pet peeves is changing-directory upwards towards the root directory by typing cd ../../ or some such command. Some people alias two dots to the cd command to go up one level, 3 dots to go up 3 levels, or whatever. To me this is clunky. I wanted something more flexible. So up was born. up is a Bash function that takes one argument: the number of directories you want to go up. If you do not specify an argument, up will take you up one directory.

So if you are at /this/really/deep/directory/looking/around, instead of typing lots of dots and slashes to get to /this/really/deep, just type up 3. Simple and Quick. Unless you are really, really deep, you’ll probably only ever have to type a maximum of 5 keystrokes to go up to any level, whereas with cd, you’ll have to type ../ as many times as levels you want to go up. That’s at least 6 (5 if you leave off the trailing slash) if you want to go up only one level. There’s some Big-O notation just waiting to happen from that scenario.

Check out the code:


#! /bin/bash
function up () {
levels=$1
if [ -z "$levels" ]; then
levels=1
fi
# Test if $levels is a number; the -eq operator expects a number, and will
# output an error if one is not found. Any output the STDERR is redirected
# to the bit bucket (/dev/null)
if [ "$levels" -eq "$levels" ] 2> /dev/null; then
if [ "$levels" -eq "0" ]; then
levels=1
fi
for (( c=1; c<=levels; c++ ))
do
cd ../
done
else
echo up: expected a number, not $levels
fi
}

view raw

up.sh

hosted with ❤ by GitHub

Include that in your .bashrc.

Very simple, but not without its problems. If you use the cd - command to go to the last directory you were in, this will break that because it cds multiple times. A solution to that is to build a string with the corresponding number of ../ entries you want to go up, then passing that as an argument to cd. I am sure there are more problems, and if you want to contribute to the betterment of nerdkind by improving the function, please fork my gist and issue a pull-request.

Help Find a Cure

Posted on by

No-one is safe from cancer, not even children, even very young children. I am no expert with anything medical, but I know this for a fact.

My 1-year-old son Emmanuel has a form of liver cancer called Hepatoblastoma. Thankfully this type of cancer is highly curable and he has a good prognosis. So many others don’t however. We have met many people over the last month whose children are back in hospital with their second bout of cancer. One of them was first diagnosed with cancer when he was 16 days old; he is now only 7 months and the cancer has returned. My heart goes out to all of them, and they inspire me, who are facing their difficulties with tremendous strength and courage.

We count ourselves as lucky that we are in such a good hospital, and that our country’s health care system means that we don’t have to pay for treatment. We have a huge support network consisting of friends, family, and people we don’t even know. We are lucky because Manny’s cancer is curable. We are most lucky because we have such a gorgeous little boy. Manny has been so good through the intrusiveness of all the tests before being diagnosed, and now through all his treatment. He is a little champion, and is an inspiration.

The Cancer Council holds an annual event: Relay for Life, a 24 hour fundraising walking event. I am taking part this year and need donations! You can do so by visiting my fundraising page and then clicking “Sponsor Me”. Anything is highly appreciated.

I’d love to hear from you especially if your child has had Hepatblastoma, or if you had it when you were younger.

Press All the Things…

Posted on by

So I finally managed to get my hands onto a nipping press. It is a lovely little cast iron baby, with a turn-wheel instead of a handle. It’s in pretty good nick: the paint is peeling in a couple of places and the screw is a bit rusty (luckily it only looks like surface rust).

You may remember that I was planning to make my own nipping press from wood. Well, that’s not going to happen any more is it? It’s probably a blessing in disguise so I’m not that worried about it. I may still build one and sell it. Would you be interested in buying a wooden nipping press? I have a few projects on at the moment, so one less is no worries.

I’ll post some more pictures once I’ve cleaned it up a bit.

Happy bookbinding!

All My Dotfiles Are Belong To You

Posted on by

Under a Blood Red Moon

As I mentioned in my last post, I am a bit of a computer nerd. The last couple of months have seen my head buried in my Mac’s Terminal tapping out and learning old-style commands: grep this, ack that, and vim you. I say old-style but the computer terminal is still as current and as useful as it was back in the ’60s and ’70s when it was the only way to access computing power. Technically, Terminal (and other software like it) is not a real terminal, it is a piece of software emulating—or pretending to be—a terminal. There is even a terminal emulator available (for a fee) that looks just like an actual vintage computer screen—complete with flickering, fuzzy letters, and curved edges—called Cathode. It is visually spectacular, but a little OTT for me.

Sharing is Caring

I recently decided to join the growing community of dotfilers on GitHub, a web frontend for using git, a distributed version control system (VCS). VCSs are used where it is handy to keep track of files in a project as they change. At various intervals one commits changes made to the project into the local repository. This way, one has a record of all the changes made to a file over the length of time the project is worked on.

Using GitHub and git is a great way of sharing stuff, giving back to the community—it is a social coding site. It is also a great way of sharing stuff between one’s own machines, a bit like Dropbox but more nerdy. This makes it ideal to keep something like your dotfiles. All you have to do is upload (or push in git-speak) on one machine, and then download (or clone) your files on another. Et voila! You have your familiar environment set up and ready to go. What’s really great is that one can make changes on this other machine, push the changes to GitHub, and grab them on the original machine. Cool.

But What Are Dotfiles?

On a UNIX-like OS (think all the Linuxes, Mac OS X, and more I don’t know), dotfiles are simply files whose name starts with a dot, otherwise known as a full stop, or period. They are usually not displayed in a directory listing, or in file browsers like OS X’s Finder. Prepending a dot to a file’s name is a way of hiding that file from a basic directory listing.

They are use mainly for storing application settings and configuration data, or for data that shouldn’t be as easily accessible as a plain file. Some common dot-files are the so-called RC files, named after runcom files, of which .vimrc and .bash_profile (.bashrc on Linux etc.) are a few examples.

All My Dotfiles…

So anyhow, all that was just to say that I have my dotfiles on GitHub. I don’t expect that to make much if a difference to most of you, but you never know. I hope it does. Merry Christmas.

Adhesive Binding is Perfect

Posted on by

It’s Been a While

So this is the post where I apologise to all my readers for not posting for so long: sorry! I have a good excuse—he is 6 months old soon. I have had a busy time at work, and have been engaged in other creative pursuits, bookbinding being just one of many creative things I love to do.

Believe it or not I love writing computer programs. I just love it. I started off at Uni teaching myself C++, I learnt Pascal, HTML, JavaScript, did some Prolog, and a whole lot of Windows API programming and ASP with VBScript. I can get myself into trouble with a bit of ConTeXt, SQL, PHP, and CSS. Now I code in C#, and I am learning Objective-C and Cocoa, Regular Expressions, and I am becoming a Bash nerd.

Anyway, I don’t want to freak you out or anything.

Pictures are Worth Heaps of Words

I thought that it would be an excellent idea to not type another word, and just let my pictures do the talking. So here is an image gallery of the steps involved in adhesive binding.