snakeoil.pickling module

pickling convenience module

snakeoil.pickling.dump_stream(handle, stream)

given a filehandle to write to, write pickled objects to it.

This is useful in conjunction with iter_stream() to deserialize the results of this function- specifically you use dump_stream to flush it to disk as you go, and iter_stream to load it back as you go.

Parameters:

• handle – file like object to write to

• stream – iterable of objects to pickle and write to handle

snakeoil.pickling.iter_stream(stream)

given a filehandle to consume from, yield pickled objects from it.

This is useful in conjunction with dump_stream() to serialize items as you go, rather than in one single shot.

Parameters:

stream – file like object to continually try consuming pickled data from until EOF is reached.