Welcome to snakeoil. Now what is it?

Snakeoil is a library with the rough intentions of holding common python patterns/implementations, optimized common functionality, functionality to ease supporting multiple python versions (specifically python2.5 through 3.3), and finally well tested implementations of rather hard problems to solve.

Snakeoil’s naming was chosen partially as a python analog of liboil (per cpu assembly optimizations of common operations), and also as a partial warning riffing on the common meaning of Snake Oil; specifically a supposed miracle cure that has no effect. To be clear, this package does have some very powerful functionality and optimizations available, and its usage is well known to speed things up: the pkgcore package manager from which this library was derived, makes heavy usage of these optimizations- further in a simple test usage of snakeoil.caching.WeakInstMeta to portage’s Atom class, emerge -ep system was sped up by 7% with a 20% memory reduction- specifically via just adding that metaclass to their Atom class, a one line change.

The point the authors are trying to make however is that it is not a magic solve all. Badly performant code will still be badly performant code if it’s due to algorithmic idiocies- swapping in a snakeoil implementation for what affects 3% of your runtime won’t do anything for the 97% that is due to a bad quadratic algorithm for example. Nor will using the more crazy functionality we’ve got make your code necessarily any better to read- while snakeoil endeavours to make its functionality simple, and to simplify consuming code, its usage does not magically make horrible code better.

That said, a python mantra is “we’re all adults here”. Snakeoil fully subscribes to this philosophy- the warnings provided are merely to try and make it clear to people that snakeoil while powerful, if used in a thoughtless fashion is not advisable.

Please note that snakeoil while relying on extensions in certain spots, does not require the extensions to function- it will just fallback to native python implementations if it cannot find its extensions to use.

Community

Snakeoil now lives in full at https://github.com/pkgcore/snakeoil. Issue tracking, wiki, groups, source, are all available from there.

Getting the source (downloading releases or trunk)

Snakeoil vcs of choice is git, and our source can be checked out at https://github.com/pkgcore/snakeoil

All releases are available at https://github.com/pkgcore/snakeoil/releases, with release news available at Release Notes.

As for dependencies, snakeoil basically just requires python3.8 and up.

Snakeoil intentions

Following is a rough breakdown of the core areas snakeoil aims to cover. This is not comprehensive- generally speaking things that are useful have a way of winding up being added to snakeoil for reuse elsewhere.

Python annoyances and hard issues

The philosophy behind snakeoil is essentially that things should just work- to that end, and via a fair amount of unit testing, snakeoil internals will cover up the nasty details of how to do something while presenting a simple/no surprises api for consumers. Good examples of this are:

  • snakeoil.caching.WeakInstMeta, a metaclass allowing you to inline instance reuse for immutable objects. Essentially, why write multiple factory implementations? Why not push it directly into instance generation itself so that the instance sharing is transparent, not requiring the consumer to know that it’s occuring?

  • snakeoil.obj.DelayedInstantiation(); a object proxy implementation that is effectively fully transparent for any non-builtin target proxying it does. While proxying implementations exist, the authors are aware of none that are reusable in this fashion, nor any that address the full issue of slotted methods (see the module for full details).

For the issues described above, these are not the simplest problems to solve- people typically solve this on an adhoc basis, partially solving the issue but never fully addressing it. The purpose of snakeoil’s functionality in that vein is to solve it once and for all, in one spot, with the best possible implementation. Let developers worry about their problem at hand, rather than worry about solving an issue someone else has already addressed essentially.

Supporting multiple python versions can be a pain

Another facet of snakeoil functionality is python compatibility- snakeoil.compatibility is a separate module that exists to address compatibility issues across py2.7 to py3.4- whether it be intern moving to sys.intern, there is a significant amount of functionality in there to help with these issues. Note that compatibility functionality that goes into that module isn’t the only compatibility bits- that’s just the general grab bag for it. snakeoil.currying is another example (primarily targeting functools.partial, although providing more functionality than just that limited usage).

Optimizations

Snakeoil provides a fair amount of optimized implementations. snakeoil.osutils is a good spot to start for file operations/path manipulations, optimized sequence flattening (snakeoil.sequences.iflatten_instance), and optimized common patterns for class implementations (for example the snakeoil.klass.jit_attr() decorator which converts secondary calls to the cached attribute into c level lookup speeds). There is a fair bit more than just those examples- it’s in the readers interest to peruse our module docs, there is a fair amount available.

For importation speed issues, we provide snakeoil.demandload - for anyone who has used bzrlib.lazy_import or hg’s equivalent implementation, this should be familiar. In effect, it allows you to delay importation till it’s actually needed. While it’s not obvious, for well written scripts the time required for importation can be come a large problem leaving people either the option of splitting up all functionality (which can be very problematic from a codebase comprehension standpoint), or suffering the performance degradation.

As such, having a lazy importer is rather useful- your code still flows the same, just the import is done only when something actually needs to access that functionality.

While we provide speed optimized implementations, we also provide functionality for reduction of memory usage- for codebases with a large number of small dictionaries, snakeoil.mappings.make_SlottedDict_kls() can reduce the memory requirement in the range of 75-95%. For codebases that make extensive use of __slots__ for memory reasons, it’s advised that they take a look at snakeoil.test.test_slot_shadowing

Avoiding Boilerplate (functionality to help with DRY- Don’t Repeat Yourself)

Finally, for folks who have wrote a significantly large python codebase they wind up finding themselves repeatedly writing the same type of functionality, over and over. Things like __eq__ methods, attribute/class attribute aliasing for backwards compatibility, JIT properties, and cloning documentation from preexisting sources. If interested, snakeoil.klass is a good place to start reading.

General Contents:

Indices and tables