On Thursday I’m making the yearly pilgrimage to PyCon in Atlanta. This will be my third year and I’m sure it’ll be better than ever.

For me the real highlights of this conference are the legendary open space sessions. The level of interaction and learning at these really sets it apart from other conferences. Last year I attended a number of crackers including one on Cassandra and big data scalability with Jonathan Ellis. It’s great to see that Jonathan is delivering a scheduled talk on database scalability, and I’m sure there will be a fair number of open space sessions dedicated to NoSQL databases.

Looking at the 2010 schedule these are my other top picks:

1. Deployment, development, packaging, and a little bit of the cloud (Ian Bicking)
2. Powerful Pythonic Patterns (Alex Martelli)
3. Understanding the Python GIL (David Beazley)
4. Mastering Team Play: Four powerful examples of composing Python tools (Raymond Hettinger)
5. Unladen Swallow: fewer coconuts, faster Python (Collin Winter)

I’ll have my camera with me again this year so watch out for pics under the pycon and pycon2010 tags on Flickr.

Based on an example provided with the Twitter library for Python I cobbled together the following script to add my latest tweets to this site. It’s called from a cron job that I run on an occasional basis. My script linkifies hashtags and @username tokens in tweets so that you can see search results or user information.

Why did I not use one of the WordPress widgets? Well writing scripts like this is fun, and some widgets don’t seem to play too well with my Thesis theme. One thing to note is that getting the shell script setup under some cron configurations can take a while if you aren’t using it on a regular basis. It’s operation is also different between Ubuntu Server and Joyent’s Accelerator platform.

Up next: a similar script to process my latest bookmarks on Delicious.com.

Main script (tweets.py)

#!/usr/bin/python

import codecs, re, getopt, sys, twitter

TEMPLATE = """
<li>
  <span class="twitter-text">%s</span>
  <span class="twitter-relative-created-at"><a href="http://twitter.com/%s/statuses/%s">Posted %s</a></span>
</li>
"""

def Usage():
  print 'Usage: %s [options] twitterid' % __file__
  print
  print '  This script fetches a users latest twitter update and stores'
  print '  the result in a file as an XHTML fragment'
  print
  print '  Options:'
  print '    --help -h : print this help'
  print '    --output : the output file [default: stdout]'

def FetchTwitter(user, output):
  assert user
  statuses = twitter.Api().GetUserTimeline(user=user, count=7)

  xhtml = []
  for status in statuses:
  	  status.text = Linkify(status.text)
  	  xhtml.append(TEMPLATE % (status.text, status.user.screen_name, status.id, status.relative_created_at))

  if output:
    Save(''.join(xhtml), output)
  else:
    print ''.join(xhtml)

def Linkify(tweet):
    tweet = re.sub(r'(\A|\s)@(\w+)', r'\1@\2', tweet)
    return re.sub(r'(\A|\s)#(\w+)', r'\1#\2', tweet)

def Save(xhtml, output):
  out = codecs.open(output, mode='w', encoding='utf-8',
                    errors='xmlcharrefreplace')
  out.write(xhtml)
  out.close()

def main():
  try:
    opts, args = getopt.gnu_getopt(sys.argv[1:], 'ho', ['help', 'output='])
  except getopt.GetoptError:
    Usage()
    sys.exit(2)
  try:
    user = args[0]
  except:
    Usage()
    sys.exit(2)
  output = None
  for o, a in opts:
    if o in ("-h", "--help"):
      Usage()
      sys.exit(2)
    if o in ("-o", "--output"):
      output = a
  FetchTwitter(user, output)

if __name__ == "__main__":
  main()

Shell script executed as cron job (tweets.sh)

/usr/bin/python /path/to/tweets.py brianly --output /path/to/output/twittertimeline.htm

When you use more than one computer on a daily basis, keeping files in sync between them is a constant problem. I’m familiar with tools like Subversion and Mercurial that make it relatively easy to keep code in sync between machines, but these require explicit actions that I often want to control. When it comes to photos, office documents, and other binary files that I don’t want to actively manage these developer tools just get in my way. Thankfully a Y Combinator startup called Dropbox is trying to solve this problem and doing a pretty good job at it.

Dropbox is very simple to use. Just drop a file in a folder and it’ll be uploaded to the central server. If your other computer is connected to Dropbox it’ll be updated with the file almost immediately. It is worth noting that these updates are almost instant, even for large files. Whilst a sync tool like this is all very simple in theory, the execution is near flawless. I’ve tried out other tools in the past and none of them had seen the attention to detail that the Dropbox team made for their application. It get’s better when you discover that Dropbox works just as well across Windows, Mac, and Linux.

I’m just about to shell out for one of their paid accounts so that I can keep my photos in sync between my Mac and Windows desktop. I’ll post an update on my experience in a couple of months. I’m sure I’ll try out some of the neat tricks detailed on their wiki.

I’ve just been through an office move at a client site and packing it up got me thinking about the choices that companies make for their staff when it comes to computer hardware. Most users can benefit in some way from having faster computers, with bigger screens, and better input peripherals. Corporations have a tendency to make conservative choices and to pay above the market price for standard hardware. Whilst frustrating for normal users, this is an absolute performance killer for developers like me.

Compare the specifications of my home machine to my work computer:

My home configuration can be purchased for around $1200 today, and is many times more powerful than the work configuration. The ability to run multiple virtual machines as if they are real desktops gives a fantastic boost to my developer productivity. The screen real estate is beneficial for rapidly updating web pages, and the fast hard drives ensures that Windows keeps up with what I’m doing.

Since I don’t run Antivirus at home I run as a user without administrative rights. If I need to test out new pieces of software from the Internet I use a Windows XP Virtual Machine running Windows OneCare Antivirus. Removing the performance overhead of antivirus means that I get the most performance out of my hardware whilst still staying safe. Did you know that simply running Windows with a standard user account can eliminate the threat of most Windows malware?

My advice to anyone buying hardware for development use is to buy the best performing hardware you can afford. Don’t focus on any single component as skipping a dual screen configuration is not worth the price of a faster processor.

I’ve been rebuilding my Dell PowerEdge 830 server after a meltdown last week. This time I decided to go with Ubuntu Server 8.10 as the base operating system with Windows running under VMware Server 2.0.

Since this is a pretty small server, I’m always checking up on resource usage and top is a useful tool for this purpose. I find that I’d like to take actions based on what is happening in top and need to keep another shell open to issue commands. Wouldn’t it be nice if there was a better version of top, that acted a little like the Windows Task Manager?

On Linux, there is an improved version of this tool called htop. This offers colour coding, CPU meters, and other nice things through the shell. Go check it out if you run Linux servers.

• Microsoft InfoMesa Project Whiteboard for your data
• Fabric – like Capistrano but written in Python
• A Visual Introduction to Screen
• retlang – Google Code
• Tim Sneath : The Bumper List of Windows 7 Secrets
• update-engine – Google Code

Recently I was migrating several Subversion repositories from a Windows server over to a Solaris one and ran into a rather frustrating issue. Actually it’s quite an interesting problem, but was frustrating due to the size of the repositories involved, and my fear that repositories were borked.

It all started with my decision to fire up PowerShell rather than cmd.exe to do the initial dump of the repostories. The following code was executed to dump a repository:

svnadmin dump repositorypath > repository.dump

All seems to go to plan but this is where a silent corruption of the dump file occurs. Not suspecting that the dump file was malformed, I copied repository.dump over to the Solaris box. I issued the standard commands to create a new repository and load the dump file:

svnadmin create repository/path
svnadmin load repository/path < repository.dump

After executing the second command I got an error from svnadmin complaining about a malformed header on the dump file:

svnadmin: Malformed dumpfile header

My first instinct was to re-copy the dump file and try again, but I thought I should dump and load the repository on my Windows server. Lo and behold this fell over with the same error. At this point I was thinking something was really wrong with my SVN install and repositories, but then I stumbled on this newsgroup discussion.

It turns out that my problems were the result of how Windows PowerShell handles binary data that is piped on the command line. I really want to look into the details of this, but it would appear that the new found ability to pass .NET objects on the command line can mess with old school expectations of how the command line works.

After a very quick install on a Dell XPS M1710 laptop, and a few hours of messing around, I have come to the conclusion that this is going to be the best release of Windows yet. At the same time, it’s not very exciting for the alpha geek in me. Perhaps it is good to have regular stable releases and leave the fireworks to user applications?

Before I list some of the things that I like, or didn’t as the case may be, I have a tip for anyone trying to get Aero running on the M1710. For some reason, Beta 1 doesn’t have drivers in the box for the NVIDIA GeForce 9700M GT. Just download the mobile drivers from the NVIDIA website and install the Vista version using the “Have a disk…” option. Ignore any warnings about compatibility, the Vista drivers are close enough and I would expect NVIDIA to release some beta drivers for Windows 7.

What I liked

Performance improvements

This has been a problem for a large number of Vista users and Microsoft have made big strides according to my unscientific testing. Little things like searching for programs on the Start menu is noticeably quicker than Vista on the same machine. No doubt anti-virus vendors are working on ways to reduce the performance.

System protection

You can now reserve a percentage of disk space to storage of old versions of user and system files (just like Recycle Bin has done since Windows 95). This is something I would want to increase for someone like my Mum.

Task bar application integration

The thumbnails introduced in Vista are now more useful since you view individual tabs in an application like Internet Explorer 8. When you have a long running task like a file copy, the progress is shown directly on the task bar when minimised.

Easier wireless network access

Clicking the network icon in the task notification area now shows a list of network connections. This makes it much easier to connect, and is similar to the experience in Apple OS X.

Notification area grouping

I find the pollution of the task notification area to be a real pain. Every application thinks it belongs there and needs to be visible. With Vista you could force some items to be hidden, in Windows 7 they are grouped under a single icon. It only takes simple stuff to improve the overall experience.

Requires more thought

Taskbar application identification

So is Internet Explorer running, or do you have a shortcut pinned to the task bar? It’s hard to see on the task bar, but I can see the rationale for this design decision. It would be nice if there was an option for clearer identification, such as the name of the application appearing on the task bar. I suspect more use of Windows 7 will result in a change in my expectation for this aspect of task bar operation.

IE 8 rendering issues

There is a still a lot of work to do here. I suspect that a lot of people will enable compatibility mode to enable sites to load. One site with problems was GMail (I’m using Google Apps for e-mail).

I manage my bookmarks using a Delicious Bookmarklet, and I’ll post some of the best on my blog each week.

• Hiring for an Agile Team: Possible Questions
• FontExplorer® X – The new professional font management software
• TextMate Blog » Working With History in Bash
• Using MVC to Unit Test WPF Applications
• Mole v4.2 For Visual Studio « Karl On WPF – .Net
• SharePoint How To: Create a Custom Media Player Field Control
• Vertebra from Engine Yard

Maybe this year will be the one where I get back on the path to regular blogging. Between 2002 and 2004 I was a fairly frequent blogger and then things got substantially busier when I moved to the US. The obligatory first post these days seems to be some background information on the blogger, but I’m leaving that for my about page.

Instead I’ll just talk a little about some highlights of 2008. By far the biggest highlight of the year was my wedding in October – Sarah really is the greatest thing that ever happened to me. I find it hard to believe that in a couple of weeks we will be celebrating our fourth year together, as they say ‘time flies when you are having fun!’.

The other main highlight of 2008 was my cycling progress and I completed 3 big rides: the 5 Boro Bike Tour, a metric centrury in Lancaster, and the Lance Armstrong ride in Philadelphia. Cycling was something that Sarah got me interested in, and I am looking forward to lots more in 2009!