=================================
Bytecode Hacks: A Cautionary Tale
=================================

Jason Kirtland
August 12th, 2008
for the Portland Python User Group

Intro
-----

A project I was working on had an enormous collection of unit tests that had
accumulated over years.  Python's unittest can run setUp() and tearDown()
methods before and after each test case, but in this project the testing
base class had been extended with setup and teardown code that could be run
around each test class, allowing expensive static initialization to be
shared by each test case in the class.

The unittest implementation doesn't make it easy to implement this kind of
extension, particularly when it comes to sharing whatever you've statically
initialized with the individual test cases.  At some point, someone noted
the expedient method of simply stashing the work done in the class-level
setUp() routine in global variables, and then cleaning them up in the
class-level tearDown().  It worked, but was error prone and sometimes led
to conflicts, or modules holding references to objects when the tearDown()
code didn't do its job properly.

The pattern became entrenched in about 1,500 fairly involved tests.  I
wanted to remove the globals and somehow rein in the scoping of the
resources.  But I needed to balance that goal against the ease of use of
what was already there in the code.  Problems aside, the globals approach
made very readable test cases, with little boilerplate or cruft to distract
from the code that was actually being exercised in the test.

This is where we switch over to a self-guided tour.  Below are snapshots
from the refactoring process and some brief commentary.  What's not included
here is the extended unittest base class.  It's actual implementation isn't
important to the story.  For these purposes, just imagine that it has a
mechanism to keep state between test method invocations, a granular
implementation of class-level setup and teardown, and the details can be
freely tweaked and rewritten as needed to support the attempts below.

original.py             full code snippet
original2.py            narrowed in
as_attrs.py             move to self.tablename
as_attrs2.py            reduce assignments by returning dicts
inverted.py             reduce lookup with x = self.get_all('name')
inverted2.py            don't return dicts- can monitor these creations by
                        snapshotting metadata and cls.__subclasses__()
                        behind the scenes.
                        the road to magic begins.
bytehacked.py           not DRY: drop half of the boilerplate: x = self.lookup()
bytehacked2.py          dis.dis(assigning function returns)
                        runnable.
inject.py               a simple matter of programming
globals1.py             the ideal- globals-like ease of use, but not globals
globals2.py             dis.dis(function using unassigned names)
                        runnable, 2 parts
resolve.py              another SMOP
globals3.py             final decorator

License
-------

These sample files are directly derived from SQLAlchemy and work I've done
on SQLAlchemy, some of it committed and some not.  This presentation is
licensed under the same terms as SQLAlchemy itself, namely the MIT License:

Copyright (c) 2008 Jason Kirtland, SQLAlchemy developers and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.