Storage Services

The HDF5 Storage Service

class pypet.storageservice.HDF5StorageService(filename=None, file_title='Experiment')

Storage Service to handle the storage of a trajectory/parameters/results into hdf5 files.

Normally you will not interact with the storage service directly but via the trajectory, see pypet.trajectory.SingleRun.f_store() and pypet.trajectory.Trajectory.f_load().

The service is not thread safe. For multiprocessing the service needs to be wrapped either by the LockWrapper or with a combination of QueueStorageServiceSender and QueueStorageServiceWriter.

The storage service supports two operations store and load.

Requests for these two are always passed as msg, what_to_store_or_load, *args, **kwargs

For example:

>>> HDF5StorageService.load(pypetconstants.LEAF, myresult, load_only=['spikestimes','nspikes'])

For a list of supported items see store() and load().

load(msg, stuff_to_load, *args, **kwargs)

Loads a particular item from disk.

The storage service always accepts these parameters:

Parameters:

• trajectory_name – Name or current trajectory and name of top node in hdf5 file
• filename – Name of the hdf5 file

The following messages (first argument msg) are understood:

• pypet.pypetconstants.TRAJECTORY (‘TRAJECTORY’)

  Loads a trajectory.

  param stuff_to_load:  The trajectory param as_new:Whether to load trajectory as new param load_params:  How to load parameters and config, see also Loading. param load_derived_params:  How to load derived parameters param load_results:  How to load results

• pypet.pypetconstants.LEAF (‘LEAF’)

  Loads a parameter or result.

  param stuff_to_load:  

  The item to be loaded

  param load_only:  

  If you load a result, you can partially load it and ignore the rest. Just specify the name of the data you want to load. You can also provide a list, for example load_only=’spikes’, load_only=[‘spikes’,’membrane_potential’].

  Throws a ValueError if data cannot be found.

• pypet.pypetconstants.TREE (‘TREE’)

  Loads a whole sub tree

  param stuff_to_load:  The parent node (!) not the one where loading starts! param child_name:  Name of child node, that should be loaded param recursive:  Whether to load recursively the sub tree below child param load_data:  How to load stuff, see also Loading. param trajectory:  The trajectory object

• pypet.pypetconstants.LIST (‘LIST’)

  Analogous to storing lists

Raises:NoSuchServiceError if message or data is not understood

store(msg, stuff_to_store, *args, **kwargs)

Stores a particular item to disk.

The storage service always accepts these parameters:

Parameters:

• trajectory_name – Name or current trajectory and name of top node in hdf5 file
• filename – Name of the hdf5 file
• file_title – If file needs to be created, assigns a title to the file.

The following messages (first argument msg) are understood:

• pypet.pypetconstants.PREPARE_MERGE (‘PREPARE_MERGE’):

  Called to prepare a trajectory for merging, see also ‘MERGE’ below.

  Will also be called if merging cannot happen within the same hdf5 file.

  param stuff_to_store:  Trajectory that is about to be extended by another one param changed_parameters:  Dictionary containing all parameters that were enlarged due to merging.

• pypet.pypetconstants.MERGE (‘MERGE’)

  Note that before merging within HDF5 file, the storage service will be called with msg=’PREPARE_MERGE’ before, see above.

  Raises a Value Error if the two trajectories are not stored within the very same hdf5 file. Then the current trajectory needs to perform the merge slowly item by item.

  Merges two trajectories, parameters are:

  param stuff_to_store:  The trajectory data is merged into param other_trajectory_name:  Name of the other trajectory param rename_dict:  Dictionary containing the old result and derived parameter names in the other trajectory and their new names in the current trajectory. param move_nodes:  Whether to move the nodes from the other to the current trajectory param delete_trajectory:  Whether to delete the other trajectory after merging.

• pypet.pypetconstants.BACKUP (‘BACKUP’)

  param stuff_to_store:  Trajectory to be backed up param backup_filename:  Name of backup file. If None The backup filename will be the same folder as your hdf5 file adding backup_XXXXX.hdf5 where XXXXX is the name of your current trajectory.

• pypet.pypetconstants.TRAJECTORY (‘TRAJECTORY’)

  Stores the whole trajectory

  param stuff_to_store:  The trajectory to be stored

• pypet.gobally.SINGLE_RUN (‘SINGE_RUN’)

  param stuff_to_store:  The single run to be stored

• pypet.pypetconstants.LEAF or pypetconstants.UPDATE_LEAF (‘LEAF’ or ‘UPDATE_LEAF’)

  Stores a parameter or result. Use msg = ‘UPDATE_LEAF’ if a parameter was expanded (due to merging or expanding the trajectory) to modify it’s data.

  Modification of results is not supported (yet). Everything stored to disk is set in stone!

  Note that anything that is supported by the storage service and that is stored to disk will be perfectly recovered. For instance, you store a tuple of numpy 64 bit integers, you will get a tuple of numpy 64 bit integers after loading!

  param stuff_to_sore:  

  Result or parameter to store

  In order to determine what to store, the function ‘_store’ of the parameter or result is called. This function returns a dictionary with name keys and data to store. In order to determine how to store the data, the storage flags are considered, see below.

  The function ‘_store’ has to return a dictionary containing values only from the following objects:

  • python natives (int,str,bool,float,complex),
  • numpy natives, arrays and matrices of type np.int8-64, np.uint8-64, np.float32-64, np.complex, np.str
  • python lists and tuples of the previous types (python natives + numpy natives and arrays) Lists and tuples are not allowed to be nested and must be homogeneous, i.e. only contain data of one particular type. Only integers, or only floats, etc.
  • python dictionaries of the previous types (not nested!), data can be heterogeneous, keys must be strings. For example, one key-value pair of string and int and one key-value pair of string and float, and so on.
  • pandas DataFrames
  • ObjectTable

  The keys from the ‘_store’ dictionaries determine how the data will be named in the hdf5 file.

  param store_flags:  

  Flags describing how to store data.

  ARRAY (‘ARRAY’)

  Store stuff as array

  CARRAY (‘CARRAY’)

  Store stuff as carray

  TABLE (‘TABLE’)

  Store stuff as pytable

  DICT (‘DICT’)

  Store stuff as pytable but reconstruct it later as dictionary on loading

  FRAME (‘FRAME’)

  Store stuff as pandas data frame

  Storage flags can also be provided by the parameters and results themselves if they implement a function ‘_store_flags’ that returns a dictionary with the names of the data to store as keys and the flags as values.

  If no storage flags are provided, they are automatically inferred from the data. See pypet.HDF5StorageSerive.TYPE_FLAG_MAPPING for the mapping from type to flag.

• pypet.pypetconstants.REMOVE (‘REMOVE’)

  Removes an item from disk. Empty group nodes, results and non-explored parameters can be removed.

  param stuff_to_store:  The item to be removed. param remove_empty_groups:  Whether to also remove groups that become empty due to removal Default is False.

• pypet.pypetconstants.GROUP (‘GROUP’)

  param stuff_to_store:  The group to store

• pypet.pypetconstants.REMOVE_INCOMPLETE_RUNS (‘REMOVE_INCOMPLETE_RUNS’)

  Removes all data from hdf5 file that is from an incomplete run.

  param stuff_to_store:  The trajectory

• pypet.pypetconstants.TREE

  Stores a single node or a full sub tree

  param stuff_to_store:  Node to store param recursive:  Whether to store recursively the whole sub-tree

• pypet.pypetconstants.LIST

  Stores several items at once

  param stuff_to_store:  Iterable whose items are to be stored. Iterable must contain tuples, for example [(msg1,item1,arg1,kwargs1),(msg2,item2,arg2,kwargs2),...]

Raises:NoSuchServiceError if message or data is not understood

The Multiprocessing Wrappers

class pypet.storageservice.LockWrapper(storage_service, lock)

For multiprocessing in LOCK mode, augments a storage service with a lock.

The lock is acquired before storage or loading and released afterwards.

load(*args, **kwargs)

Acquires a lock before loading and releases it afterwards.

store(*args, **kwargs)

Acquires a lock before storage and releases it afterwards.

class pypet.storageservice.QueueStorageServiceSender

For multiprocessing with WRAP_MODE_QUEUE, replaces the original storage service.

All storage requests are send over a queue to the process running the QueueStorageServiceWriter.

Does not support loading of data!

send_done()

Signals the writer that it can stop listening to the queue

store(*args, **kwargs)

Puts data to store on queue.

class pypet.storageservice.QueueStorageServiceWriter(storage_service, queue)

Wrapper class that listens to the queue and stores queue items via the storage service.

run()

Starts listening to the queue.

Empty Storage Service for Debugging

class pypet.storageservice.LazyStorageService

This lazy guy does nothing! Only for debugging purposes.

Ignores all storage and loading requests and simply executes pass instead.

load(*args, **kwargs)

Nope, I won’t care, dude!

store(*args, **kwargs)

Do whatever you want, I won’t store anything!