I was asked to take a few photos for this event by fellow pythonista Tom Panzarella. I think they turned out OK for the most part – although the EF 70-200 L IS II saved my arse in the end. It was difficult shooting with the lighting setup at the venue, but I’ll be back there to get some more practice.

All of the photos are over on Flickr and they were even linked to by Technically Philly!

This is mainly for me own reference, but it is probably something that other people will find useful. If you attempt to use the rlcompleter module on the Mac you’ll find that tab completion does not work correctly. Apparently this is because Apple ship a neutered version of readline.

To get completion working you have to use a different argument to readline.parse_and_bind(). The base example uses:

readline.parse_and_bind("tab: complete")

Change that line to:

readline.parse_and_bind ("bind ^I rl_complete")

Following the recent PhillyPUG meetup I was trying to install scrapy on an old MacBook Pro running OS X 10.6 (Snow Leopard) and ran into a number of problems with the lxml dependency. This is the parser used to extract data from pages that scrapy downloads so you are not going to get very far without it.

It seems that the compilation problems experienced when installing with pip result from an attempt to build a universal binary. If you have Xcode 4 installed then you lose some of this capability and need to make sure that the correct architecture is specified.

Architecture Fix

Setting the architecture is something you can do in your bash profile, executing it under a new bash ensures that the build script picks it up.

sudo bash
export ARCHFLAGS='-arch i386 -arch x86_64'
pip install lxml # test it
pip install scrapy --upgrade # fix the failed scrapy install

Original Error

brianly$ sudo pip install lxml --upgrade
Downloading/unpacking lxml
  Downloading lxml-2.3.tar.gz (3.2Mb): 3.2Mb downloaded
  Running setup.py egg_info for package lxml
    Building lxml version 2.3.
    Building without Cython.
    Using build configuration of libxslt 1.1.24
    warning: no previously-included files found matching '*.py'
Installing collected packages: lxml
  Found existing installation: lxml 2.2.2
    Uninstalling lxml:
      Successfully uninstalled lxml
  Running setup.py install for lxml
    Building lxml version 2.3.
    Building without Cython.
    Using build configuration of libxslt 1.1.24
    building 'lxml.etree' extension
    gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -I/usr/include/libxml2 -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.6-universal-2.6/src/lxml/lxml.etree.o -w -flat_namespace
    /usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed
    Installed assemblers are:
    /usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64
    /usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
    src/lxml/lxml.etree.c:161594: fatal error: error writing to -: Broken pipe
    compilation terminated.
    lipo: can't open input file: /var/tmp//ccYr9GpX.out (No such file or directory)
    error: command 'gcc-4.2' failed with exit status 1
    Complete output from command /usr/bin/python -c "import setuptools;__file__='/Users/brianly/dev/github/pyconscrape/build/lxml/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /tmp/pip-axeEA7-record/install-record.txt:
    Building lxml version 2.3.

Building without Cython.

Using build configuration of libxslt 1.1.24

running install

running build

running build_py

creating build

creating build/lib.macosx-10.6-universal-2.6

creating build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/__init__.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/_elementpath.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/builder.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/cssselect.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/doctestcompare.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/ElementInclude.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/pyclasslookup.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/sax.py -> build/lib.macosx-10.6-universal-2.6/lxml

copying src/lxml/usedoctest.py -> build/lib.macosx-10.6-universal-2.6/lxml

creating build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/__init__.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/_dictmixin.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/_diffcommand.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/_html5builder.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/_setmixin.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/builder.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/clean.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/defs.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/diff.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/ElementSoup.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/formfill.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/html5parser.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/soupparser.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

copying src/lxml/html/usedoctest.py -> build/lib.macosx-10.6-universal-2.6/lxml/html

creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron

copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron

creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources

creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/rng

copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/rng

creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl

copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl

copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl

creating build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-10.6-universal-2.6/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

running build_ext

building 'lxml.etree' extension

creating build/temp.macosx-10.6-universal-2.6

creating build/temp.macosx-10.6-universal-2.6/src

creating build/temp.macosx-10.6-universal-2.6/src/lxml

gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64 -pipe -I/usr/include/libxml2 -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.6-universal-2.6/src/lxml/lxml.etree.o -w -flat_namespace

/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed

Installed assemblers are:

/usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64

/usr/bin/../libexec/gcc/darwin/i386/as for architecture i386

src/lxml/lxml.etree.c:161594: fatal error: error writing to -: Broken pipe

compilation terminated.

lipo: can't open input file: /var/tmp//ccYr9GpX.out (No such file or directory)

error: command 'gcc-4.2' failed with exit status 1

----------------------------------------
  Rolling back uninstall of lxml
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/basecommand.py", line 126, in main
    self.run(options, args)
  File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/commands/install.py", line 228, in run
    requirement_set.install(install_options, global_options)
  File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 1104, in install
    requirement.rollback_uninstall()
  File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 487, in rollback_uninstall
    self.uninstalled.rollback()
  File "/Library/Python/2.6/site-packages/pip-1.0.1-py2.6.egg/pip/req.py", line 1417, in rollback
    pth.rollback()
AttributeError: 'str' object has no attribute 'rollback'

Storing complete log in /Users/brianly/.pip/pip.log

If you want need a 64-bit version of this module for Google App Engine, or another project, you can run into a couple of issues. Sticking to 32-bit versions of Python and PIL avoids these issues so that may be the best direction for newbies.

The downloads on the PIL site are for 32-bit versions of Python which means that you hit runtime issues as soon as the Python script files attempt to load the 32-bit libraries into 64-bit Python. Annoyingly the installer is unaware of the architecture for the Python installation it finds and will leave you with a broken PIL install. Just uninstall it from Control Panel. My first thought was to try an build from the sources but I figured someone has run into this problem.

After some searching I found a site which offers pre-built versions of many Python modules for 64-bit architectures including PIL. If you are in any way concerned about performance, or security a better solution might be to build from source or use versions from a company offering supported versions.

I downloaded the version for Python 2.5 and then discovered that it would not install. What is interesting is that the 32-bit installer was able to find my 64-bit Python, but the 64-bit installer was unable to find it.

After some more searching it turns out that the installer takes it’s cue from a key in the registry and the Registry virtualization in Windows x64 was confusing it. To resolve the issue I had to export the “HKLM\SOFTWARE\Wow6432Node\Python\PythonCore\2.5” key, remove the “Wow6432Node\” string using a text editor, and re-import the key so that it was at “HKLM\SOFTWARE\Python\PythonCore\2.5”.

After doing this the 64-bit installer for PIL detected my Python installation and I was up and running.

On Thursday I’m making the yearly pilgrimage to PyCon in Atlanta. This will be my third year and I’m sure it’ll be better than ever.

For me the real highlights of this conference are the legendary open space sessions. The level of interaction and learning at these really sets it apart from other conferences. Last year I attended a number of crackers including one on Cassandra and big data scalability with Jonathan Ellis. It’s great to see that Jonathan is delivering a scheduled talk on database scalability, and I’m sure there will be a fair number of open space sessions dedicated to NoSQL databases.

Looking at the 2010 schedule these are my other top picks:

1. Deployment, development, packaging, and a little bit of the cloud (Ian Bicking)
2. Powerful Pythonic Patterns (Alex Martelli)
3. Understanding the Python GIL (David Beazley)
4. Mastering Team Play: Four powerful examples of composing Python tools (Raymond Hettinger)
5. Unladen Swallow: fewer coconuts, faster Python (Collin Winter)

I’ll have my camera with me again this year so watch out for pics under the pycon and pycon2010 tags on Flickr.

Based on an example provided with the Twitter library for Python I cobbled together the following script to add my latest tweets to this site. It’s called from a cron job that I run on an occasional basis. My script linkifies hashtags and @username tokens in tweets so that you can see search results or user information.

Why did I not use one of the WordPress widgets? Well writing scripts like this is fun, and some widgets don’t seem to play too well with my Thesis theme. One thing to note is that getting the shell script setup under some cron configurations can take a while if you aren’t using it on a regular basis. It’s operation is also different between Ubuntu Server and Joyent’s Accelerator platform.

Up next: a similar script to process my latest bookmarks on Delicious.com.

Main script (tweets.py)

#!/usr/bin/python

import codecs, re, getopt, sys, twitter

TEMPLATE = """
<li>
  <span class="twitter-text">%s</span>
  <span class="twitter-relative-created-at"><a href="http://twitter.com/%s/statuses/%s">Posted %s</a></span>
</li>
"""

def Usage():
  print 'Usage: %s [options] twitterid' % __file__
  print
  print '  This script fetches a users latest twitter update and stores'
  print '  the result in a file as an XHTML fragment'
  print
  print '  Options:'
  print '    --help -h : print this help'
  print '    --output : the output file [default: stdout]'

def FetchTwitter(user, output):
  assert user
  statuses = twitter.Api().GetUserTimeline(user=user, count=7)

  xhtml = []
  for status in statuses:
  	  status.text = Linkify(status.text)
  	  xhtml.append(TEMPLATE % (status.text, status.user.screen_name, status.id, status.relative_created_at))

  if output:
    Save(''.join(xhtml), output)
  else:
    print ''.join(xhtml)

def Linkify(tweet):
    tweet = re.sub(r'(\A|\s)@(\w+)', r'\1@\2', tweet)
    return re.sub(r'(\A|\s)#(\w+)', r'\1#\2', tweet)

def Save(xhtml, output):
  out = codecs.open(output, mode='w', encoding='utf-8',
                    errors='xmlcharrefreplace')
  out.write(xhtml)
  out.close()

def main():
  try:
    opts, args = getopt.gnu_getopt(sys.argv[1:], 'ho', ['help', 'output='])
  except getopt.GetoptError:
    Usage()
    sys.exit(2)
  try:
    user = args[0]
  except:
    Usage()
    sys.exit(2)
  output = None
  for o, a in opts:
    if o in ("-h", "--help"):
      Usage()
      sys.exit(2)
    if o in ("-o", "--output"):
      output = a
  FetchTwitter(user, output)

if __name__ == "__main__":
  main()

Shell script executed as cron job (tweets.sh)

/usr/bin/python /path/to/tweets.py brianly --output /path/to/output/twittertimeline.htm

• Microsoft InfoMesa Project Whiteboard for your data
• Fabric – like Capistrano but written in Python
• A Visual Introduction to Screen
• retlang – Google Code
• Tim Sneath : The Bumper List of Windows 7 Secrets
• update-engine – Google Code

Congratulations to Dino, Harry, and the rest of the IronPython team! They just released IronPython 2.0 which has parity with CPython 2.5. This release of IronPython includes release 0.9 of the Dynamic Language Runtime (DLR).

Download it from CodePlex.

IronPython was one of the factors that impacted my decision to attend PyCon. Microsoft are approaching the release of version 2.0 which will have parity with CPython 2.5. The production versions already are close to full Python 2.4 support making it a viable platform for use in a lot of places where I would typically use C#.

Going into the conference I was looking forward to the Sunday session with Jim Hugunin but there turned out to be some more treats for the IronPython developer. Feihong Hsu ran a session on Python.NET and how you can bridge from CPython to the .NET platform, taking advantage of rich Windows APIs. Michael Foord spoke on Silverlight as well as his company’s spreadsheet which embeds IronPython.

Feihong organised an open space session for Saturday evening after the PyWin32 gathering to talk about Python.NET and we were joined by the IronPython developers and management (Dino Viehland, Harry Pierson, Jim Hugunin and others). We discussed a number of aspects of IronPython and progress towards the 2.0 release. It looks like this may be complete in October given that they released the first beta last week. Again Michael Foord had something interesting to say on what Resolver Systems are doing.

Michael presented an open source project called IronClad. This is quite an insane assortment of code from C# to Python to assembler all in the name of accessing Python modules written in C. To date they have the bzip2 module running but are working on support for modules like NumPy which are important to their customers.

After the open space session we headed into Chicago for dinner at India House. This gave us a chance to find out some more stuff about the IronPython implementation, and other factoids. Dino hinted that he was working on getting Django up and running. Little did we know he was going to be demoing this to the crowd on Sunday.

Sunday saw Jim’s big talk and I managed to get a few photos. It wasn’t easy, but I think these turned out a bit better than earlier shots at the conference. Dino showed off the fairly minimal changes needed to get Django running on IronPython and Jim demoed the IronPython interpreter running under Dynamic Silverlight.

After the keynote, Dino gave a me a quick run through of the IronPython and DLR source code. This was very interesting and it gave me a real step up in understanding what goes on under the covers. Thanks Dino!

Today was the Python tutorial day. Given that I haven’t spent a lot of time reading or writing Python code I thought it would be a good idea to attend some of these tutorials. Since they kicked off at 9am it was a bit of a challenge making it on time. My Southwest flight from Philly last night arrived late, and then I had a big trip around to the other side of Chicago. If you need to attend a conference near O’Hare, try to fly into that airport. All I wanted to do was to lie in bed for a few more hours

When I made my tutorial selections I was hoping to attend a Python for Java Developers session. This would have been useful given my experience with C# but it seems that I was in the minority and it was cancelled. I switched to the Django session but I think I may have been better attending the session on performance optimisation.

Registration wasn’t too busy today since the main conference crew won’t arrive till Friday. I got a PyCon bag and some flyers but the T-Shirts weren’t ready. Apparently they’ll be available tomorrow, I’d hate to miss out on one!

Python 101 Tutorial (Steve Holden)

It turns out that Steve was another British ex-pat living here in the US. It gets weirder in that he lectured at Manchester University for a number of years. Given Steve’s position in the community I expected a sharp introduction to Python. It didn’t disappoint and I picked up a fair bit. The ‘slice’ mechanism looks really useful, I wonder if it can be implemented with any of the new C# features?

Getting Started with Django (Jacob Kaplan-Moss)

I was expecting this session to be a little more exciting. Jacob has some fine ideas about how Python frameworks should be built but his presentation style is not as striking as DHH. Since this was an introductory session I can’t complain too much but I really want to hear some more about Django deployment and debugging over the next few days.

Internet Programming with Python (Wesley Chun)

At this point I was pretty exhausted. This tutorial seemed to be geared toward newbies to network programming, rather than a best practice session on leveraging Python for internet programming.