What we want to do is a) detect any files which don’t contain tweets, b) remove the redundant user profile, and c) combine the results into a single file.

Well, friends, here is a shell script to do exactly that. You’ll need zsh and xsltproc, both of which are standard on MacOS X and most sane Linuxen.

Output is on stdout, so just redirect to your filename of choice:

$ tweetcombine user_timeline.xml\?count=100\&page=* \
    > tweet_archive.xml

Here’s the script:

#!/bin/zsh

# Combine all of the twitter user_timeline.xml files specified on the command line into a single output
# Written by Alastair Rankine, http://girtby.net
# Licensed as Creative Commons BY-SA

input_args=()
for f in ${(on)*}; do
    [[ -f $f ]] || exit "Not a file: $f"
    input_args+="<input>${f//&/&amp;}</input>"
done

xsltproc - <<EOF
<?xml version="1.0"?>
<!DOCTYPE inputs [
  <!ATTLIST xsl:stylesheet id ID #REQUIRED>
]>
<?xml-stylesheet type="text/xml" href="#style1"?>
<inputs>
  ${input_args}

  <xsl:stylesheet id="style1" version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    <xsl:output type="xml" indent="yes"/>

    <xsl:template match="*">
      <xsl:copy>
        <xsl:copy-of select="@*"/>
        <xsl:apply-templates/>
      </xsl:copy>
    </xsl:template>

    <xsl:template match="statuses">
      <xsl:apply-templates/>
    </xsl:template>

    <xsl:template match="user"/>

    <xsl:template match="xsl:stylesheet"/>

    <xsl:template match="input">
      <xsl:choose>
        <xsl:when test="document(.)/statuses">
          <xsl:apply-templates select="document(.)"/>
        </xsl:when>
        <xsl:otherwise>
          <xsl:message terminate="yes"><xsl:value-of select="."/> does not contain statuses element</xsl:message>
        </xsl:otherwise>
      </xsl:choose>
    </xsl:template>

    <xsl:template match="inputs">
      <statuses type="array">
        <xsl:apply-templates/>
      </statuses>
    </xsl:template>

  </xsl:stylesheet>
</inputs>
EOF

I think this method of sticking filename arguments into an XSL document with an embedded stylesheet is quite a powerful way of processing XML documents with shell scripts. (Probably should put the <input> tags into a separate namespace though…)