That has been on my status page for a few weeks now and I have only been twittering via IM (routing it through my own Jabber server). That sucks. While they have been triumphing the fact they have made it through WWDC, can someone please fix that IM thing?!

Or maybe it is not necessary. As of yesterday I am now updating my status on Jaiku (thanks to Glenn for giving me the invite). Currently you can’t sign up new account there but hopefully soon after they have moved to Google AppEngine. At least their IM bot works…

The next task is to automatically post my Jaiku status to Twitter. Most cross-posting tools are online based, and you can only instruct Jaiku to sync with Twitter but not the other way around. Well I didn’t really look long enough because “reinventing the wheel” here is actually relatively cheap, as both Jaiku and Twitter has well published RESTful API. I quickly do up a Python script that takes new entries from my Jaiku feed and publish them onto Twitter.

Here’s the code:

#!/usr/bin/env python

import anydbm
import feedparser
import os
import re
import sys
import urllib
import urllib2

# Configurable variables
JAIKU_USERNAME = 'scottyang'
TWITTER_USERNAME = 'scottyang'
TWITTER_PASSWORD = 'mypassword'

# Other constant
TWITTER_ENDPOINT = 'http://twitter.com/statuses/update.json'

def main():
    db = anydbm.open(os.path.join(os.environ.get('HOME', '/tmp'),
        '.jaiku2twitter.db'), 'c')
    feed = feedparser.parse('http://%s.jaiku.com/feed/rss' % JAIKU_USERNAME,
        etag=(db.get('etag') or None))
    if feed.get('status') == 304:
        pass
    elif feed.bozo:
        print 'Error: %s' % feed.bozo_exception
    else:
        auth = urllib2.HTTPBasicAuthHandler()
        auth.add_password('Twitter API',
            'twitter.com',
            TWITTER_USERNAME,
            TWITTER_PASSWORD)
        opener = urllib2.build_opener(auth)

        for entry in reversed(feed.entries):
            link = str(entry.link)
            if link.startswith('http://%s.jaiku.com/presence' % JAIKU_USERNAME):
                if link not in db:
                    value = entry.summary
                    value = re.search(r'^<p>(.*?)</p>', value, re.M | re.S)
                    if value:
                        value = value.group(1).strip()
                        print 'Jaiku->Twitter: %s' % value
                        response = opener.open(TWITTER_ENDPOINT,
                            'status=%s' % urllib.quote(value))
                        response.read()
                        db[link] = '1'

        if feed.etag:
            db['etag'] = feed.etag

        db.sync()
        db.close()

if __name__ == '__main__':
    main()

Or you can download it here. It requires the Universal Feed Parser, and it creates a BSDdb file in your home directory to track what entries have it posted. It supports etag to hopefully reduce load on Jaiku. Tested under Python 2.4 and 2.5. You just need to fill in your Jaiku username, Twitter username/password, and then instruct cron to run it every now and then. I run every 10 minutes to publish my presence/status on Jaiku to Twitter.

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency’s sake, Twitter was built with technologies and practises that are more appropriate to a content management system. Over the last year and a half we’ve tried to make our system behave like a messaging system as much as possible, but that’s introduced a great deal of complexity and unpredictability.

The fundamental problem of the Twitter architecture is trying to massage a state-less request/response system that is most suitable for read-intensive applications, to a write-intensive message-based system. Basically — getting Ruby on Rails to do what Jabber/XMPP Server was designed to do — something is not right! Twitter not being able to scale is not that they haven’t thrown in enough boxes, or they haven’t cached the right content in the right place, or they haven’t implemented a right cluster solution for the database. Not even because RoR is slow, in the way Tech Crunch puts it. It does not scale because it was not architectured to what it was supposed to do. A paradigm shift is required. Maybe they should have started with an Erlang backend in the beginning, like what Facebook did with their chat system.

Re-architecture an existing system is never fun though. Not many can afford a second team doing a rewrite. Rewriting component by component is not really optimal, especially if the original design was not well componentised (for scalability’s sake, it’s often not!) At work we have also been talking about rewriting the backends (to increase scalability + server utilisation) and frontends (employ various RIA techniques), and it has been very frustrating when you lay it out “what needs to be done”, but management’s asking “can it be done bit by bit?” or “does it add any functionality?” (mind you that our SaaS product is maybe 500x more complicated than Twitter although requiring only 1/100000′s scalability). Yes. Frustrating.

Anyway. Good luck to Twitter because I really want to use it. Except it is not working right now. Message via Jabber is broken…

It is just so much less work than blogging. No more logging into WordPress. No more thinking about a catching title. No more battling with my broken English. Since I use TwitterFox on Firefox, and bound the key to <F9>, I just press the key to bring up the window, type in a short message, press enter and I have done some micro-blogging! In fact, due to the 140 character limitation on Twitter I actually cannot talk about something deep and meaningful. Far too easy for me.

Twitter now also has integration with Facebook, where your new tweets can automatically update your Facebook status. While it sounds useful, it really severely limits what you can tweet about, as Facebook automatically prefixes “Scott is” in front of my status.

Anyway. Don’t be alerted when you have not seen me blogging for a while. Check my Twitter feed for latest updates.

(The actual reason why I have not been blogging is of course my own lazyness :)

Signing up is a breeze — clicking on the link from the invitation email, choose a username, password, fill up a few more fields and you are almost there. Then uploading a picture. Then… Oh wait! I have just hit a page telling me there is a bug in their code!

This is NOT a bug, it is the Admiral Ackbar from the Rebel Alliance!!! I guess Pownce must be from the dark side.

Still, as I don’t really get Twitter, I am not sure about the compelling point of Pownce either. Sure it has more function than Twitter. There is even an Adobe Flex-based desktop client. But if I really need IM, I already have Jabber, Google Talk, MSN, ICQ and Yahoo Messenger thank you very much. Meanwhile, Twitter over XMPP still sucks. When can I get something that actually works?

Anyway. I’ve got 6 Pownce invites to give away. Tell me you want it in comments.

To be fair, it sounds like most of Twitter’s problems are database problems, so maybe it doesn’t matter what language they use”

Even Alex stated it clearly in the Twitter interview that database has been the bottleneck, just like most share-nothing web development platform that was designed to be highly scalable in the first place! Ruby is slow. Python is a tad better, but in a heavily IO bounded web-based application, both will be more than fast enough. How to scale the database? It is not trivial and very domain specific, and that is what Alex is complaining about.

It is Evan Williams‘ (of Blogger and Odeo fame) latest creation, and is currently hosted on TextDrive‘s accelerators. They call it micro-blogging, where you can post plain text of up to 140 characters using either the web site, your phone via SMS (if you are in US), Google Talk, or its web service API. Moreover, it combines with social networking where you get notification whenever your friends posted something.

So there we have it — a bunch of guys and girls writing short snippets of text about what they do everyday.

• Just woken up, on the wrong side of bed (8 hours ago)
• Had Indomie for breakfast. 3rd pack this week, and it is only Tuesday (7 hours ago)
• On the bus to work. Cloudy. 24″C at most. (6 hours ago)
• @friend Yeah that was cool. (3 hours ago)
• Still working. A boring day. (2 hours ago)
• …

Yes — that would be a typical Twitter time line for a made-up individual (definitely not mine). Pretty much useless to those who do not know the person who is posting these. Yet everyone is claiming that Twitter is so addictive. However I just don’t get it (not yet).

Twitter replacing emails? Is scalability and fail-safe delivery in the designing of the protocol? Can I type in long paragraphs, attaching documents and PGP sign my signature? My PGP signature would be more than 140 characters…

Use Twitter productively, as ToDo list, people and business management, a short newsletter, etc. As it is posted on LifeHack, they do look like lots of “hacks” to tweak Twitter into something it wasn’t designed for.

I know. It is a simple product. Cool people behind it. It has an API (but which Web 2.0 app hasn’t?). People love it… Meanwhile, I am twittering here, trying to figure out why people are so crazy about it.