girtby.net

Let’s solve Twitter’s scalability problems, shall we?

So, like most people, I don’t know much about the problems there and certainly don’t have any solutions to suggest. But I do know there are a certain class of solutions which aren’t on the table.

If you look at Twitter from a suitably high vantage point you see real-time communication between small groups. People entering short messages and having these messages appear at their peers a small time later. There’s also a central archive, but I’ve heard Twitter described as “public Instant-Messaging” and this seems to characterise it best for me.

In short, Twitter seems more suited to peer-to-peer communication than to client-server. What sort of protocol would it use? I can imagine a protocol which would be probably UDP-based, and which would send tweets to followers either directly from peers or perhaps through a local aggregation point. Large groups of followers could perhaps even use UDP multicast. Archive servers could be reached through network anycast addresses, to allow for greater decentralisation. IPv6 to get universal connectivity. And so on; fill in your own pet network technology here, there are certainly lots of potential solutions.

Instead of these, clients communicate directly with the Twitter servers using HTTP. Not only that, but they poll for updates. Bit of an architectural blunder, you might think. Well not really. In fact I don’t think the Twitter designers had any choice.

Once upon a time it was possible to deploy new application-layer protocols on the Internet. But those times have passed, it seems. These days, it’s HTTP(S) or nothing. And this is not the protocol you would choose for carrying tweets, if you had the choice. So the fact that twitter works at all over this sub-optimal application-layer protocol is quite an achievement.

This is a great example of the many ways in which innovation can be stifled by enforcing a lowest-common-denominator.

The impact is of course more widespread than just Twitter. In fact, the so-called end-to-end principle which was one of the fundamental founding principles of the Internet is now all but abandoned in practice. Geoff Huston examines the issue in some detail in a recent article, and it is highly recommended.

Of course, there are no easy answers, either for Twitter or the next application to suffer due to the proliferation of network middleware. But it’s certainly an issue that does need to be more prominent.

(This post is an obvious departure from my usual style of blatant attack pieces in order to score traffic and fame for myself. Normal service will resume shortly.)

As you know, I’m a fan of the DRM free music. In fact it seems that I’ve blogged about it each time I’ve discovered a new website that sells the stuff. And the latest discovery is emusic. They have hits and some misses.

Continue reading...

A few years ago I purchased an Arc-AAA led flashlight, mainly on the advice of Dan. It was an excellent piece of kit. Small enough to carry everywhere but bright enough to be useful. For years it was the only thing that hung off my keychain (besides keys of course). A month ago it died due to a battery leak.

I wasn’t hopeful of saving it. Since I bought my unit, the guy had stopped selling them for a while, and then been bought out by another company. Nevertheless I wrote to the new company and asked if it could be fixed. They said sure, send it in. A few weeks later, this arrived:

A brand new Arc-AAA, mysteriously labeled an Arc-P.

The new one is slightly heavier I think, but feels more rugged. The knurled metal case is that dull greenish-grey colour, as might be carried by Brown from Spook Country. The light is activated by twisting the head, and the new unit has a lot more resistance which makes it a lot less likely to turn on in your pocket. It’s at least as bright as the old Arc-AAA, in other words surprisingly good for the size (and capable of out-shining any AAA Maglite). If you want to get really mental I see there’s now a “premium” version which is even brighter.

The quality of the unit itself and of the support from the vendor gets two thumbs up from me.

To justify my occasional lapses into Apple fanboy-ism, I offer the following for your consideration.

Exhibit A: Apple, 24 years ago

Exhibit B: Microsoft, today

Yeah, I know, who cares about packaging? But howcome so few companies get it right?

And in the long run, I think it *is* important. The message you send with the packaging of your product is one of the care you have put in to producing it. And of the importance of the customer’s time in getting up and running quickly.

Message received, Apple.

If you’re at all interested in computing technology you can’t help but be amazed at the advances in CPU power over the last few decades, Moore’s Law, blah blah blah. But a few seconds pondering this invariably provokes the question as to how long this party can last.

The commonly accepted wisdom is that CPUs have gotten about as fast as they are likely to go in terms of sheer clock speed, and now manufacturers are turning to multiprocessing to provide more processing power for a given price point. The recent Intel price drops which made the quad-core Q6600 CPU available for less than AUD400 are a highly relevent (and welcome) data point to illustrate this trend.

This raises lots of hairy questions for developers, such as “how are we going to design our software to run efficiently in a multi-processing environment?” The previously-linked wide finder experiment is an attempt to explore some of these issues. And it’s pretty obvious that so far there is no silver bullet.

But wait, it gets worse. I will point you to a long but highly thought-provoking presentation from Herb Sutter. Turns out we are already hitting major architectural hurdles in the form of memory access limitations, and we’ll need to find some solutions for these before tackling the parallel computation problem.

Sutter’s presentation is deeply technical, but still quite accessible, and delivered with an engaging style that makes it required viewing. Highly recommended.

I recently had some experience diagnosing some memory-related performance problems (not quite in the same class as that discussed by Sutter, but similar) and I have to say there is a serious deficit in the development tools for these kinds of problems. Currently we need to look aggregate behaviour over multiple iterations to isolate some of these problems, and this is a difficult and error-prone approach. For example, check out Sutter’s technique to discover the memory cache line size in code. In the future it would be great if we could monitor cache misses, pipeline stalls, page faults, and other performance-impacting events within the debugger.

These issues also make me wonder about how higher-level languages are going to provide appropriate abstractions to avoid the performance problems. For example, garbage collection is a major win for programmer productivity but it does encourage memory usage patterns that are not always conducive to performance given architectural limitations in the underlying hardware. The same abstraction problems affect C/C++ of course but at least there is the option to go “bare-metal” where necessary.

Whatever the answers are here, it’s certain there are some interesting times ahead for developers.

Amazon recently opened their MP3 store (in “beta” of course). It is awesome.

I don’t know about you, but I certainly didn’t see it coming. Previous Amazon efforts with downloaded content were pretty lame, and I had no idea that they were going to come back with an offering that was as good as this.

Although I am a very happy bleep.com customer, their selection of music is limited to the Warp label plus a couple of degrees of separation. So I’ve been looking around at other DRM-free music stores, but few have excited me enough to become customers.

Amazon’s catalogue of music is quite large and hopefully getting larger. Most importantly, they have an extensive back-catalogue into which I have been delving in my quest to go legit, and to reclaim old legitimately-purchased albums that have got lost along the way. The purchase and download process is very easy and efficient.

I can also get individual songs from those long-lost albums. Somehow though I managed to turn on the infamous one-click purchase, so rather than bundling up a bunch of individual tracks and purchasing them en masse, Amazon conducts a separate transaction for each one. But no matter.

So if you have a US credit card, do yourself a favour and check it out. This, plus the new Radiohead release, make me think we’ve finally reached the point where purchasing legitimate DRM-free music is easier than stealing it.

A wise man once told me that he deliberately avoided cultivating expensive tastes, because they are ultimately unsatisfying unless you are uncommonly wealthy. I think this is probably good advice in general, and particularly good advice for cars, wine, cameras, and home A/V equipment. But for headphones, not so much. Sure, you get the urge to upgrade, but you won’t end up mortgaging your house to do so. Although you can of course go overboard if you want to, the point of diminishing returns can be reached relatively cheaply.

Dan likes to make the analogy with motorcycles. They perform better than cars, but at a fraction of the price. This is true also of headphones versus a full-blown home Hi-Fi system. However, unlike motorcycles, headphones are more practical than their full-priced competitors. You can even carry them around!

The only thing you really lose with the headphone experience is that whole-body listening sensation. You don’t get to feel that thump in your chest on really deep bass hits. Instead it’s all in your head. Which is by no means unpleasant, just slightly different to a live performance or even a good Hi-Fi system.

In case it’s not painfully obvious, this post is about headphones. It contains some recommendations and advice about specific types and models, but mainly the message I want to get across is that headphones are an inexpensive and invaluable conduit to musical satisfaction. For a few hundred measly bucks you can experience something close to musical perfection. Without even bothering the neighbours. (Although you can do that too, just by singing along…)

Continue reading...

Some blog updates, the details of which will no doubt enthrall and excite very few of you.

Invisible Changes

Under the covers, the entire girtby site (such as it is) is now stored in an SVK repository depot. SVK is a great little distributed version control system built on top of Subversion. I want to say more about it in future, but for now it is really helping me maintain this site.

I use it like this. First I set up local mirrors of various Subversion repositories. The mephisto repository, the 1-2-stable branch of rails, and the repository for my one and only plugin. With a single svk sync -a, I can download the most recent changes in all of those repositories, and access it locally.

So then I copy the source from each of these repositories into a new local branch. Any changes I make on this local branch are, of course, isolated from the mirrored sources that I just downloaded.

When time comes to update, I can just merge across from the appropriate mirrored repository into my local branch. The really cool thing is that this merge is generally painless because SVK manages to track merge points (a well-known limitation of Subversion). This means it can do a three-way diff and hence get the merge right ninety-nine times out of a hundred. Plus the Mephisto codebase has extensive unit tests that really help the confidence level.

Result: updating to the latest Mephisto, rails, etc is extremely easy. You wordpress guys really should do yaself a favour.

Visible Changes

There have also been some subtle changes to the girtby stylesheet. Hopefully it looks better, although you be the judge.

Thanks to the version control I can tell you in exact detail what was changed, but I’ll spare you from that. Suffice it to say: some font changes, de-cluttering, and general tidyup. There is more to do here.

Of course the most visible change is the new Markdown WYSIWYM comment editor, WMD. Let me know what you think of it. I’m very impressed so far, this is how WYSIWYM editors should be done.

It should be pointed out that the preview mode may not accurately match your comment when it published. This is because WMD uses its own, JavaScript, implementation of Markdown. The rest of the blog uses the ruby implementation, which is not as good, frankly. Until I get around to fixing the ruby version (yeah, right), please accept my apologies if your comment comes out wonky. Or more wonky than you anticipated.

So with the change of job, I’m eager to keep in touch with all (well, some) of my former colleagues. In the past this has taken the form of sporadic individual emails of decreasing frequency. So, I’m experimenting with the latest craze, social networking sites. Some very early impressions follow.

Continue reading...

Take a look at this comparison of server operating systems from mon.itor.us. What does it tell you?

To me, this data is almost entirely useless.

First off, I’m annoyed by the use of the term “uptime” here. To me, uptime is a measure of time. Specifically, the length of time for which a given server is “up”. Wikipedia backs me up here. Netcraft measure uptime. What these folks seem to be measuring is availability: the percentage probability that the server will respond at any given instant.

So after getting over that, what to make of the information from mon.itor.us? Well they quite correctly acknowledge that the OS is only one of many factors determining uptime. However they haven’t isolated any of the other factors.

They have assumed that the quality of hosting ISP, and hence network connectivity, will be constant across all operating systems. However this is not at all obvious to me; instead I would imagine that different hosting companies will prefer one OS over another, hence the distribution will not be even.

A possibly more important factor in determining availability of the server as a whole is the web application itself. The simple static page sites are intrinsicly less complex, and hence more reliable than highly interactive database-driven web applications. Again it is assumed that the distribution of these different types of sites is constant for all operating systems, and again it is not obvious to me that this holds.

The primary conclusion they draw from this survey is that Linux is more reliable than Windows. This seems like a bit of a long bow to draw, especially given some significant WTFs in the data. 6.4% of the sites ran Minix? Really? This seems entirely counterintuitive to me. Not dissing Minix or anything, but is it really more prevalent out there than Solaris? Also, Minix really has a lower availability than Windows? How is this possible, given that the former is explicitly designed for resilience and reliability, and the latter is famous for instability?

In short, it looks like this data is worth what you paid for it, and it probably means whatever you want it to mean.

Do you have a todo.txt file? Are you like most alpha geeks?

I used to have one. In fact I had lots of them. One for each machine I used regularly.

Continue reading...

The features I want out of an online music store are:

• No DRM. At all. Seriously.
• MP3 encoded at a high bitrate (diskspace is cheap).
• Optional lossless encoding.
• Reasonable prices.
• Must be legit (rules out allofmp3 for example).
• Decent website with online previews.
• Appropriate embedded meta-data.

I am happy to say that although it is not perfect, Bleep is pretty close to satisfying every item on my list. (hat tip: Uninnovate)

Bleep is a spin-off of the hugely influential electronic music label Warp Records. Hence the catalogue is heavy with Warp and similar electronic artists. Not the sort of thing you’re likely to hear on commercial radio.

The website is probably the least appealing thing about Bleep. It looks pretty — thanks to TDR design — but it’s a usability nightmare. No scroll bars, fixed layout, weird coverart-based navigation. However it is quite searchable and there are online previews of all songs in their entirety (but served up in 30 second chunks). Also, you can embed colour-coordinated previews in your own site, like this:

That’s a track from Autechre’s Confield, specifically chosen because 30 seconds is probably all you’re going be able to listen to. (That album is particularly impenetrable to mere mortals, although I am a big fan of some of their other stuff.)

I’d also like to see Bleep provide some user reviews, artist background info, and notable previous works. Preferably outsourced to another site that specialises in this stuff.

On the plus side, all tracks are available as MP3 encoded in LAME using the canonical --alt-preset standard option. Some are also available with FLAC encoding, which is a nice touch. (Unfortunately FLAC doesn’t seem to be playable in iTunes yet.)

MP3 encoded albums are all US$9.95 from what I can see. FLAC encoding costs a little more. Individual tracks are also available, but I don’t know why you’d bother.

In the name of research I bought an album and EP. The purchase experience was quite pleasant although after concluding the paypal-based payment the download did not start automatically as promised. No worries though, there’s a downloads page where you can download all the songs you have paid for but not received yet.

I would like the ability to re-download purchases but Bleep does not seem to allow this. No matter, just burn a CD and keep it with all the other receipts.

The download is delivered as a big honkin’ zip file. You unzip, import to iTunes, sync to your iPod, and play happily ever after.

This is how music should be sold online. Recommended.

We are now in the home stretch and will soon arrive at the end of the Bush administration. Not a moment too soon.

In the last 6 or so years we have seen just so much insanity out of the white house that it is hard to keep track of. The horrendous Patriot Act almost seems a quaint historical footnote in face of some of the other abominations that have been laid before us.

For my own benefit I was going to go back through the archives and come up with a list of the worst of the worst. The acts and deeds of the Bush cronies that scrape the bottom of the barrel. Just to get it all straight in my own mind, you understand.

I promise that I was going to try to be fair and balanced. That is, ignore the scandals which were not foreseeable, or relatively harmless, or caused only indirectly, or through the incompetence of a lowly subordinate. Foleygate wouldn’t make the list, for example. Abu Ghraib would not make the list, but the response to it would. “Mission Accomplished” was mostly a political stunt with relatively minor repercussions that wouldn’t make the list. I wasn’t even going to get mad at the Bush gang for the useless Congress or obsequious media. You see that I am trying to be generous.

But after toying with the list for a bit, I now have to admit defeat. I’m going to cut and run. Going back through the details of madness like Executive Order 13233 or immigration reform, or the underhanded media tactics, or the assault on science or … enough. It’s just too much.

So I can’t say with absolute certainty after sober reflection and contemplation that the current enactment of the Military Commissions Act is the absolute lowest point of the Bush administration. But it’s pretty fucking bad.

Olbermann’s rant is eloquent and apparently a better response than I can manage. It’s passionate and rhetorical but worth listening to.

Roll on 2009.

Even before the latest version of Google Talk was released, it was already the best Instant Messaging service in the world. Here’s why.

• Ubiquity. Every Gmail web page has a chat client built-in. This rocks.
• Server-side archives. None of the other IM services include this, yet it is so handy. “What was that Youtube video that Fred sent me yesterday?”
• Open Standards. Under the hood, it’s Jabber. So I can use prettymuch any client I like, even iChat. No reverse-engineering the protocols, no turf wars.
• Firewall friendly. By this I mean it supports the universal firewall bypass protocol (ie HTTP). The other IM services also support this, but for me it only seems to work for the official clients, and not gaim-based clients for example.
• Multiple simultaneous logins. So obviously The Right Way To Do It.
• The default client is functional and minimal. It isn’t butt ugly, it doesn’t have ads, and its installer doesn’t pimp toolbars or other such crapware.

AIM, Yahoo and MSN collectively had a huge head start on Google Talk. So instead of adding the above features, why were they all wasting time building smilies, themes, “IMvironments”, icons and other nonsense? Beats me.

Screen sharing is a feature I really want to see in an IM service. Preferably, it should be based on an open standard like VNC. This would be a lifesaver for those of us who perform remote troubleshooting. As far as I know none of the current crop of IM services have this feature.

Based on their proven ability to add sensible features first, I’m betting Google Talk will be the first to support screen sharing. I suggested it, maybe you should too?

Now I know you have no reason to trust my taste in music. I haven't written much about it — except in passing — so you have no way of knowing whether my music recommendations will have any meaning for you.

I would like to be able to point to my last.fm profile and say that you could get a good idea of the type of stuff I listen to, but it's not really accurate. Mainly because last.fm is blocked by the censorware used at my work, so none of the stuff I listen to there is recorded. Also the top artists list hasn't been updated since May, for some reason.

But forget all that bollocks and take my word for it that music from the Monotonik net label is really good.

Net label? Yes, net label. As in, music label on the net. As in, MP3 format with no DRM, for all to download free of charge. Creative commons licensed, even.

Monotonik specialise in what they call "headphone music". This is a term which may be confusing to you but makes sense to me. Its electronic music which you listen to late at night with the headphones on. Just like I'm doing now. There are a huge variety of artists and styles mostly revolving around the IDM genre.

Go and get the torrent of all their 2006 releases for a start. Do yaself a favour.