snakeoil.data_source module

A far more minimal form of file protocol encapsulation location and encoding into it

The primary use for data_source’s is to encapsulate the following issues into a single object:

• is the data actually on disk (thus can I use more efficient ops against the file?).

• what is the preferred encoding?

• py3k compatibility concerns (bytes versus text file handles)

Note that all file like handles returned from text_fileobj() and bytes_fileobj() have a required additional attribute- exceptions, either a single Exception class, or a tuple of Exception classes that can be thrown by that file handle during usage.

This requirement exists purely to allow the consuming code to avoid having to know anything about the backing of the file like object.

The proper way to use such a filehandle is as follows:

>>> from snakeoil.data_source import data_source
>>> source = data_source("It's a fez. I wear a fez now. Fezes are cool.", mutable=False)
>>> handle = source.text_fileobj()
>>> handle.write("You graffitied the oldest cliff face in the universe.")
Traceback (most recent call last):
TypeError:
>>> # if this where a normal file, it would be an IOError- it's impossible to guess the
>>> # correct exception to intercept, so instead we rely on the handle telling us what
>>> # we should catch;
>>> try:
...   handle.write("You wouldn't answer your phone.")
... except handle.exceptions as e:
...   print("we caught the exception.")
we caught the exception.

class snakeoil.data_source.base

Bases: object

base data_source class; implementations of the protocol are advised to derive from this.

Variables:

path – If None, no local path is available- else it’s the ondisk path to the data

bytes_fileobj(writable=False)

get a bytes level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

path = None

text_fileobj(writable=False)

get a text level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

transfer_to_data_source(write_source)

transfer_to_path(path)

weakref

class snakeoil.data_source.bytes_data_source(data, mutable=False)

Bases: data_source

Bytes data source.

This does auto-conversion between bytes/text as needed.

class snakeoil.data_source.bz2_source(path, mutable=False)

Bases: base

locally accessible bz2 archive

Literally a bz2 file on disk.

bytes_fileobj(writable=False)

get a bytes level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

mutable

path

text_fileobj(writable=False)

get a text level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

class snakeoil.data_source.data_source(data, mutable=False)

Bases: base

base class encapsulating a purely virtual data source lacking an on disk location.

Whether this be due to transformation steps necessary (pulling the data out of an archive for example), or the data being generated on the fly, this classes’s derivatives text_data_source and bytes_data_source are likely what you should be using for direct creation.

Variables:

• data – the raw data- should either be a string or bytes depending on your derivative

• path – note that path is None for this class- no on disk location available.

bytes_fileobj(writable=False)

get a bytes level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

data

mutable

text_fileobj(writable=False)

get a text level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

class snakeoil.data_source.invokable_data_source(data)

Bases: data_source

data source that takes a callable instead of the actual data item

The callable takes a single argument- a boolean, True if a text fileobj is requested, False if None

Note that this instance is explicitly readonly.

bytes_fileobj(writable=False)

get a bytes level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

text_fileobj(writable=False)

get a text level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

classmethod wrap_function(invokable, returns_text=True, returns_handle=False, encoding_hint=None)

Helper function to automatically convert a function that returns text or bytes into appropriate callable

Parameters:

• invokable – a callable that returns either text, or bytes, taking no args

• returns_text – True if the data returned is text/basestring, False if Not

• returns_handle – True if the object returned is a handle, False if not. Note that returns_text still has meaning here- returns_text indicates what sort of data the handle returns from read invocations.

• encoding_hint – the preferred encoding to use for encoding

Returns:

invokable_data_source instance

class snakeoil.data_source.local_source(path, mutable=False, encoding=None)

Bases: base

locally accessible data source

Literally a file on disk.

buffering_window = 32768

bytes_fileobj(writable=False)

get a bytes level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

encoding

mutable

path

text_fileobj(writable=False)

get a text level filehandle for for this data

Parameters:

writable – whether or not we need to write to the handle

Raise:

TypeError if immutable and write is requested

Returns:

file handle like object

class snakeoil.data_source.text_data_source(data, mutable=False)

Bases: data_source

Text data source.

This does auto-conversion between bytes/text as needed.