Wednesday, November 19, 2008
Finally! Sync it all with Dropbox!
I work on an iMac at If/Then, my own iMac at home, and sometimes off of my PowerBook G4. Working from Subversion and checking my email is a breeze, but pretty much everything else is annoying when it comes to sharing data. I don’t want to fork over the money for MobileMe because I’d only want it for syncing. I don’t want to make the big shift to products like SugarSync or PiWorx because that feels like too much. But alas! I finally have a simple solution that I’m using to keep my 1Password keychain, Address Book contacts, and Things data all up to date.
I came across DropBox with the same disinterest and skepticism that I’ve had with other syncing solutions. The makers of my now-precious 1Password announced that they’d be discontinuing a web service that sync’d 1Password across several computers. Upset and looking for alternatives, I took some advice and tried using DropBox to keep my 1Password data updated on the machines I use. I just switched my 1Password data to the Agile Keychain format, chose to store the keychain in my DropBox, and I was done! It worked so well I thought I’d try and use DropBox for some other stuff.
1Password made it easy. I just chose to store my keychain in my dropbox and I was done.
Oh my contacts! Finally I could have Address Book share the same data and sync any changes to my iPhone from my primary/home computer! It was as simple as turning the Address Book’s data folder into a symbolic link. The app acts like it normally would, but secretly the data it uses gets stored in the DropBox and therefore updated each time the Address Book gets an edit. For me, it was two lines via the terminal:
cd /Users/matt/Library/Application\ Support/
ln -s /Users/matt/Dropbox/SyncData/AddressBook/ ./AddressBook
Since this worked just fine, I tried the same with Things. It stores its data in a single XML file, but I made the whole directory a symbolic link anyway:
cd /Users/matt/Library/Application\ Support/Cultured\ Code/
ln -s /Users/matt/Dropbox/SyncData/Things/ ./Things
The only catch here is that I need to remember to close Things when I’ll be moving to another computer. I ran into a problem once where I opened up my laptop and it wrote to the Things data before it had a chance to pull down updates. This was easily fixed by going to the DropBox web panel, selecting a previous revision of that file, and restoring it. It immediately pushed the revision to all three machines. Impressive! So even if something gets messed up, DropBox is quietly keeping revisions of everything. I love it.
I’d like to follow suit with my Safari bookmarks, but I only want to update one file (bookmarks.plist) rather than Safari’s entire support directory. I tried making only this file a symbolic link, but Safari is quick to overwrite it with a real file which destroys the link. I don’t want my browsing history, cookies, cache, etc. to get synced, but I’m not sure what else to do. If anybody out on the internet has an idea, feel free to leave a comment. (Also, if anybody besides me actually reads my posts, I’d be thrilled to find that out.)
Tuesday, November 04, 2008
Monday, October 20, 2008
A Mail Server: Postfix, Courier, MySQL, SpamAssassin, Procmail, Maildrop, Postini
Setting up a mail server hasn't been a simple affair. Fortunately, PickledOnion's email tutorials gave me a jump start on configuring my Hardy slice as a mail server. After using Postfix and MySQL to get virtual mail delivery working, several offers for replica watches and discount meds reminded me that I needed junk filtering. Here I'll detail the process that I went through to configure email handling on my slice.
Categorizing Mail with SpamAssassin
At the time, SpamAssassin seemed like the best way to filter out junk. I'd heard of it before and configured SpamAssassin settings on shared hosting accounts at Hostgator and DigitalSpace. I'd used it and knew it worked; I just had to figure out how to set it up. I read a few warnings about SpamAssassin being a bit of a memory hog, but how bad could it be? I'll come back to that later. It was easy enough to use
I populated SpamAssassin's whitelist and lowered the minimum SPAM score. Immediately SpamAssassin started doing a great job of telling me what was SPAM and what wasn't. There were very few false positives, which were quickly remedied with more whitelisting.
The next problem was keeping SPAM from my inbox. SpamAssassin was doing a great job of categorizing SPAM, but I needed some way to avoid seeing the SPAM alongside my legitimate messages. Every new SPAM message to my inbox was an urgent reminder of two things:
- I needed to separate junk from real mail.
- With my own server, I just have to do everything!
Sorting with Procmail (An Attempt)
Webmin, which has at times been extremely helpful and deserves its own post, had a menu item under SpamAssassin for Procmail delivery. It didn't take much searching and reading to know that Procmail was just the filtering device that I was looking for. I installed Procmail and repeatedly configured
Filtering with Maildrop
My understanding of virtual delivery was lacking and I couldn't get Procmail to work. It seemed like most of the example Procmail scripts I found assumed that you were using local delivery with real users (as opposed to virtual users like I was using via MySQL). I convinced myself that Procmail didn't play nice with virtual delivery (wrong) and looked for alternatives. Despite looking for the wrong reason, I found Maildrop and eventually got that installed and filtering mail.
Here's how I configured maildrop and got it working:
/etc/maildroprc
DEFAULT="$HOME/Maildir"
logfile "/var/log/maildrop.log"
SHELL="/bin/bash"
PATH=/bin:/usr/bin:/usr/local/bin:/usr/lib/courier/bin/
INBOX="/home/vmail/$DEFAULT"
DEFAULT="$INBOX"
SPAMFLD="$INBOX.Junk/"
# create the trash directory if it does not exist
`test -d "$SPAMFLD"`
if( $RETURNCODE == 1 )
{
`maildirmake "$SPAMFLD"`
}
# filter message through spamassassin's spamc agent
xfilter "/usr/bin/spamc -f"
if ( /^X-Spam-Status: Yes,/)
{
DELETE_THRESHOLD=9.0
/score=(\d+)/
if ($MATCH1 >= $DELETE_THRESHOLD)
{
to /dev/null
}
else
{
to "$SPAMFLD"
}
}
/etc/postfix/master.cf
maildrop unix - n n - - pipe
flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
/etc/postfix/main.cf
mailbox_command = /usr/bin/maildrop -d ${USER} ${RECIPIENT}
virtual_transport = maildrop
It's imporant to note here that
Maxing Out the Slice
Everything was finally working just like I wanted. The only minor detail was that my memory usage seemed high, and my virtual memory usage was -- well -- being used on occasion. Typically, I was running at about 230MB out of the available 256MB. In the Slicehost Campfire chatroom, I posted my memory usage statistics and promptly received some concerned feedback. My swap usage was over 100MB, which means that my memory usage was so high the system had to start dumping memory contents to the disk's swap space. I was maxing out the 256MB of memory on my Slice, and I needed to tune some processes and avoid utilizing virtual memory.
I restricted my Apache limits a bit, knowing that none of my sites would generate heavy traffic. I reduced the number of SpamAssassin children running on the Slice. Both of these adjustments helped a bit, but my memory usage was still high. I restricted SSH, saslauthd, and Courier, but that proved to be pretty stupid. Mail.app would suffer from SSL errors and I'd get occasional authentication problems. I put those back quickly.
Weeks passed and I closely watched my memory usage. I didn't want to sacrifice functionality, but my memory usage was high and occasionally pushed into swap territory; mostly as little as 80KB but sometimes up to 4-5MB. My top processes were Apache (which I couldn't cut any further), SpamAssassin, and MySQL. Switching Apache for Lighttpd
Filtering Mail with Google Apps/Postini
I read a helpful post by unicks in Joyent's forums that explained how Google Apps/Postini offered an excellent mail filtering service that's fairly easy to implement. I could still have complete control over the physical mail on my server, but with the benefit of up-to-date, state-of-the-art junk and virus filtering. Most importantly, I could drop SpamAssassin and save some memory.
All I had to do to configure this was to edit my DNS settings and point my MX servers to Postini. I then restricted Postfix's allowed domains to Postini, so any mail that comes through my inbox has first passed through one of the more sophisticated SPAM filtering services on the market. The service will be <$20/year, which is also a huge plus.
Regaining Memory
Rather than paying to upgrade to a slice with more RAM, I've offloaded my SPAM filtering to Postini -- which I trust more than myself to stay up-to-date and effective. My average physical memory usage now hovers at around 125MB, which is about 80MB less than my SpamAssassin setup took. My swap usage gets to about 80KB sometimes, but that doesn't worry me too much.
Sunday, October 19, 2008
ServerMonitor Dashboard Widget
Since getting my slice up and running smoothly, I frequently log in to Webmin just to make sure that my server is behaving. I'm mostly concerned with my memory and CPU usage, and any quick increase in disk usage warrants investigation. Logging into Webmin all the time doesn't seem ideal and I couldn't find a Dashboard widget that displayed server vitals -- so I made my own. It's pretty basic, and even a little bit of a hack -- but it works for me.
How it Works
There are two scripts that work together to provide the widget with server vitals:
- stats.sh: this is a bash script that runs
free for memory usage anddf for disk usage. The output is scrubbed and made into a simple string for PHP to parse. - getstats.php: this PHP script parses the string from stats.sh and outputs XML for our Dashboard widget to read. With this version, you must specify the IP address for each desktop machine.
How to Configure This Widget
- Upload
getstats.php andstats.sh to a web-accessible folder somewhere on your server. - Edit
$allowed_ips withingetstats.php . Add the IP addresses of any desktop machines that will be using the widget. Example:$allowed_ips = array( "254.26.34.190", "192.168.0.1" ); - Import the widget to your Leopard Dashboard, and add the public address to
getstats.php/snippet> to the widget's settings pane.
Any feedback on how I could/should improve this is welcome. I plan on having a slightly more intuitive feedback when entering the URL for getstats.php.
Download: ServerMonitor0.95.zip
Sunday, October 12, 2008
Slicehost!
In September, I switched from a Baby shared hosting account at HostGator to my very own slice at Slicehost. It's been an eye-opening plunge into the world of unmanaged hosting, but the Slicehost support team has played a critical role in supporting me while I get my Slice running smoothly, securely, and efficiently. I'm maxing out a 256MB slice with LAMP, a mail server, DAV, and SVN.
The great relief is that every single problem I've had been my own, not the limitations or drawbacks of the hosting company. I was impressed with the responsiveness and offering of my HostGator account, but felt limited by not having DAV or SVN access. Slicehost, at least thus far, has simply followed through with its promise: rock-solid, high-performance service with unbeatable support. They've gone out of their way to help me troubleshoot and configure my Slice, understanding full well that I've been learning as I go. The risk of course is that your server is yours ... to completely screw up. I've had a few late-night Postfix/Courier experiments go poorly, but there has always been someone around to help. (Which is amazing -- I really mean always.)
I plan on posting some articles with what I've learned from my own experience. If you're starting out with Slicehost, however, PickledOnion's tutorials are a must.