Engines

ListEngine

class acid.engines.ListEngine

Storage engine that backs onto a sorted list of (key, value) tuples. Lookup is logarithmic while insertion is linear. Primarily useful for unit testing.

URL scheme for acid.open(): “list:/”

items = None

Sorted list of (key, value) tuples.

size = None

Size in bytes for stored items, i.e. sum(len(k)+len(v) for k, v in items).

SkiplistEngine

class acid.engines.SkiplistEngine(maxsize=65535)

Storage engine that backs onto a Skip List, lookup and insertion are logarithmic. This is like ListEngine but scales well, with overhead approaching a regular dict (113 bytes/record vs. 69 bytes/record on amd64). Supports around 23k inserts/second or 44k lookups/second, and tested up to 2.8 million keys.

maxsize:
Maximum expected number of elements. Inserting more will result in performance degradation.

URL scheme for acid.open(): “skiplist:/[;maxsize=N]”

LmdbEngine

class acid.engines.LmdbEngine(env=None, **kwargs)

Storage engine that uses the OpenLDAP “Lightning” MDB library via the py-lmdb module.

env:
lmdb.Environment to use. If None, the remaining keyword args are passed to the lmdb.Environment constructor.
db:
Named database handle to use, or None to use the main database.

URL scheme for acid.open(): “lmdb:/path/to/env.lmdb[;p1[;p2=v2]]”. URL parameters:

map_size=N:
Maximum environment map size in MiB; default 4194304MiB.
readonly:
Open environment read-only; default read-write.
nometasync:
Disable meta page fsync; default enabled.
nosync
Disable sync; default enabled.
map_async
Use MS_ASYNC with msync, writemap equivalent to nosync; default use MS_SYNC.
noreadahead:
Instruct LMDB to disable the OS filesystem readahead mechanism, which may improve random read performance when a database is larger than RAM; readahead is enabled by default.
writemap:

Use writeable memory mapping; default disabled.

Note: When enabled, defective operating systems like OS X that do not fully support sparse files will attempt to zero-fill the database file to match map_size at close. Avoid use of writemap there.

nomeminit:
Avoid 0-initialization of malloc buffers when writemap is disabled. Improves performance at the cost of nondeterministic slack areas in the database file, and potential security consequences (e.g. accidentally persisting free’d cleartext passwords).
max_readers=N:
Maximum concurrent read threads; default 126.

PlyvelEngine

class acid.engines.PlyvelEngine(db=None, lock=None, _snapshot=None, **kwargs)

Storage engine that uses Google LevelDB via the Plyvel module.

Read transactions are implemented using snapshots, and write transactions are implemented using an Engine-internal threading.Lock. Note that write batches are not used, since a consistent view of the partially mutated database is required within a transaction.

db:
If specified, should be a plyvel.DB instance for an already open database. Otherwise, the remaining keyword args are passed to the plyvel.DB constructor.
lock:
If not None, specifies some instance satisfying the threading.Lock interface to use as an alternative lock instead of the Engine-internal write lock. This can be used to synchronize writes with some other aspect of the application, or to replace the lock with e.g. a greenlet-friendly implementation.

URL scheme for acid.open(): “leveldb:/path/to/env.ldb[p1[;p2=v2]]”. URL parameters:

paranoid_checks:
Enable “paranoid” engine checks; default off.
write_buffer_size=N:
Size in MiB of level 0 memtable, higher values improve bulk loads; default: 4MiB.
max_open_files=N:
Maximum number of OS file descriptors to open; default 1000.
lru_cache_size=N:
Size of block cache in MiB; default 8MiB.
block_size=N:
Approximate size in KiB of user data packed per block. Note that the size specified here corresponds to uncompressed data, the actual size of the unit read from disk may be smaller if compression is enabled. Default 4KiB
block_restart_interval=N:
Number of keys between restart points for delta encoding of keys; default 16.
compression=[compressor]:
Block compression scheme to use; default “snappy”.
bloom_filter_bits=N:
Use a bloom filter policy with approximately the specified number of bits per key. A good value is 10, which yields a filter with ~1% false positive rate. Default: no bloom filter policy.

KyotoEngine

class acid.engines.KyotoEngine(db=None, path=None)

Storage engine that uses Kyoto Cabinet. Note a treedb must be used.

TraceEngine

class acid.engines.TraceEngine(engine, trace_path=None, _fp=None)

Storage engine that wraps another to provide a complete trace of interactions between Acid and the external engine. Used for debugging and producing crash reports.

engine:
lmdb.engines.Engine to wrap.
trace_path:
String filesystem path to overwrite with a new trace log.

Each line written to trace_path contains the following fields separated by a single space character, with the line itself terminated by a single newline character.

Fields:

  • Monotonically incrementing transaction count. Initial value is 1 for operations performed against the main Engine class.
  • String operation identifier.
  • Optionally a hex-encoded string key. Possibly the empty string.
  • Optionally a hex-encoded string value. Possibly the empty string.

Valid operation identifiers:

close:
The engine or transaction was closed.
get:
Engine.get() is about to be invoked for key.
got:
Engine.get() was invoked, and returned value. Value may be the string None if no record was returned.
put:
Engine.put() is about to be invoked with key and value.
delete:
Engine.delete() is about to be invoked with key.
abort:
Engine.abort() is about to be invoked.
begin:
Engine.begin() is about to be invoked. key may be the string True if a write transaction was requested, otherwise False.
commit:
Engine.commit() is about to be invoked.
iter:
Engine.iter() is about to be invoked for key. If value is True, then reverse iteration was requested, otherwise False.
fetch:
The next element is about to be retrieved from an iterator returned by Engine.iter().
yield:
The element retrieved by the last fetch is about to be yielded; key and value are the key and value returned by the engine.

Engine Interface

acid.engines.register(klass)

Register a new engine class that supports Engine.from_url(). Allows the engine to be uniformly constructed from a simple string using acid.open(). May be used as a decorator.

class acid.engines.Engine

A storage engine or transaction is any object that implements the following methods. Engines need not inherit from this class, but doing so enables various default method implementations. All key and value variables below are NUL-safe bytestrings.

abort()

Abort the active database transaction. The default implmentation does nothing.

begin(write=False)

Start a database transaction, returning an Engine instance requests should be directed to for the duration of the transaction. The default implementation does nothing, and returns itself.

close()

Close the database connection. The default implementation does nothing.

commit()

Commit the active database transaction. The default implementation does nothing.

delete(key)

Delete key if it exists.

classmethod from_url(dct)

Attempt to parse dct as a reference to the engine. If the reference is valid, return a new engine instance, otherwise return None. URLs should be of the form engine:/path?p1=v1;p2=v2 or engine://<netloc>/path?p1=v1;p2=v2. dct is a dict with fields:

  • scheme: URL scheme (e.g. “engine”).
  • netloc: Empty string if URL was in first form, netloc if URL was in second form.
  • path: String path.
  • params: Dict with keys p1 and p2, values are strings if the parameter included a value, otherwise True.
get(key)

Return the value of key or None if it does not exist.

iter(key, reverse=False)

Yield (key, value) tuples in key order, starting at key and moving in a fixed direction.

Key order must match the C memcmp() function.

key:
Starting key. The first yielded element should correspond to this key if it exists, or the next highest key, or the highest key in the store.
reverse:
If False, iteration proceeds until the lexicographically highest key is reached, otherwise it proceeds until the lowest key is reached.
pop(key)

Delete key if it exists, returning the previous value, if any, otherwise None. The default implementation is uses get() and delete().

put(key, value)

Set the value of key to value, overwriting any prior value.

replace(key, value)

Replace the value of key with value, returning its prior value. If key previously didn’t exist, return None instead. The default implementation is implemented using get() and put().

source = None

If present and not None, indicates the source object responsible for producing data in the buffers returned by this engine. The source object must implement the Memsink Protocol. This allows acid.keylib.Key and acid.structlib.Struct to present the result to the user without performing any copies.