snakeoil.osutils package

Submodules

• snakeoil.osutils.mount module
  • mount()
  • umount()
• snakeoil.osutils.native_readdir module
  • listdir_dirs()
  • listdir_files()
  • readdir()
  • stat_swallow_enoent()

Module contents

OS related functionality

This module is primarily optimized implementations of various filesystem operations, written for posix specifically. If this is a non-posix system (or extensions were disabled) it falls back to native python implementations that yield no real speed gains.

A rough example of the performance benefits, collected from a core2 2.4GHz running python 2.6.5, w/ an EXT4 FS on a 160GB x25-M for the FS related invocations (it’s worth noting the IO is pretty fast in this setup- for slow IO like nfs, the speedup for extension vs native for listdir* functionality is a fair bit larger).

Rough stats:

python -m timeit code snippet	native	extension time
join(“/usr/portage”, “dev-util”, “bsdiff”, “ChangeLog”)	2.8 usec	0.36 usec
normpath(“/usr/portage/foon/blah/dar”)	5.52 usec	0.15 usec
normpath(“/usr/portage//foon/blah//dar”)	5.66 usec	0.15 usec
normpath(“/usr/portage/./foon/../blah/”)	5.92 usec	0.15 usec
listdir_files(“/usr/lib64”) # 2338 entries, 990 syms	18.6 msec	4.17 msec
listdir_files(“/usr/lib64”, False) # same dir content	16.9 msec	1.48 msec
readfile(“/etc/passwd”) # 1899 bytes	20.4 usec	4.05 usec
readfile(“tmp-file”) # 1MB	300 usec	259 usec
list(readlines(“/etc/passwd”)) # 1899 bytes, 34 lines	37.3 usec	12.8 usec
list(readlines(“/etc/passwd”, False)) # leave whitespace	26.7 usec	12.8 usec

If you’re just invoking join or normpath, or reading a file or two a couple of times, these optimizations are probably overkill. If you’re doing lots of path manipulation, reading files, scanning directories, etc, these optimizations start adding up pretty quickly.

snakeoil.osutils.abspath(path)

resolve a path absolutely, including symlink resolving.

Note that if it’s a symlink and the target doesn’t exist, it’ll still return the target.

Parameters:

path – filepath to resolve.

Raises:

EnvironmentError – some errno other than an ENOENT or EINVAL is encountered

Returns:

the absolute path calculated against the filesystem

snakeoil.osutils.abssymlink(path)

Return the absolute path of a symlink

Parameters:

path – filepath to resolve

Returns:

resolved path

Raises:

EnvironmentError – with errno=ENINVAL if the requested path isn’t a symlink

snakeoil.osutils.ensure_dirs(path, gid=-1, uid=-1, mode=511, minimal=True)

ensure dirs exist, creating as needed with (optional) gid, uid, and mode.

Be forewarned- if mode is specified to a mode that blocks the euid from accessing the dir, this code will try to create the dir.

Parameters:

• path – directory to ensure exists on disk

• gid – a valid GID to set any created directories to

• uid – a valid UID to set any created directories to

• mode – permissions to set any created directories to

• minimal – boolean controlling whether or not the specified mode must be enforced, or is the minimal permissions necessary. For example, if mode=0755, minimal=True, and a directory exists with mode 0707, this will restore the missing group perms resulting in 757.

Returns:

True if the directory could be created/ensured to have those permissions, False if not.

snakeoil.osutils.join(a, *p)

Join two or more pathname components, inserting ‘/’ as needed. If any component is an absolute path, all previous path components will be discarded. An empty last part will result in a path that ends with a separator.

snakeoil.osutils.listdir(path=None)

Return a list containing the names of the files in the directory.

path can be specified as either str, bytes, or a path-like object. If path is bytes,

the filenames returned will also be bytes; in all other circumstances the filenames returned will be str.

If path is None, uses the path=’.’. On some platforms, path may also be specified as an open file descriptor;

the file descriptor must refer to a directory. If this functionality is unavailable, using it raises NotImplementedError.

The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.

snakeoil.osutils.listdir_dirs(path, followSymlinks=True)

Return a list of all subdirectories within a directory

Parameters:

• path – directory to scan

• followSymlinks – this controls if symlinks are resolved. If True and the symlink resolves to a directory, it is returned, else if False it isn’t returned.

Returns:

list of directories within path

snakeoil.osutils.listdir_files(path, followSymlinks=True)

Return a list of all files within a directory

Parameters:

• path – directory to scan

• followSymlinks – this controls if symlinks are resolved. If True and the symlink resolves to a file, it is returned, else if False it isn’t returned.

Returns:

list of files within path

snakeoil.osutils.normpath(mypath: str) → str

normalize path- //usr/bin becomes /usr/bin, /usr/../bin becomes /bin

see os.path.normpath() for details- this function differs from os.path.normpath only in that it’ll convert leading ‘//’ into ‘/’

snakeoil.osutils.pjoin(a, *p)

Join two or more pathname components, inserting ‘/’ as needed. If any component is an absolute path, all previous path components will be discarded. An empty last part will result in a path that ends with a separator.

snakeoil.osutils.readdir(path)

Given a directory, return a list of (filename, filetype)

see d_type_mappings for the translation used

Parameters:

path – path of a directory to scan

Returns:

list of (filename, filetype)

snakeoil.osutils.supported_systems(*systems)

Decorator limiting functions to specified systems.

Supported platforms are passed as string arguments. When run on any other system (determined using sys.platform), the function fails immediately with NotImplementedError.

Example usage:

>>> from snakeoil.osutils import supported_systems
>>> @supported_systems('linux', 'darwin')
>>> def func(param):
...     return True
>>>
>>> if sys.platform.startswith(('linux', 'darwin')):
>>>     assert func() == True

NotImplementedError is raised on platforms that aren’t supported.

>>> @supported_systems('nonexistent')
>>> def func2(param):
...     return False
>>>
>>> func2()
Traceback (most recent call last):
    ...
NotImplementedError: func2 not supported on nonexistent

snakeoil.osutils.unlink_if_exists(path)

wrap os.unlink, ignoring if the file doesn’t exist

Parameters:

path – a non directory target to ensure doesn’t exist