Parameters and Results

This module contains implementations of result and parameter containers.

Results and parameters are the leaf nodes of the Trajectory tree. Instances of results can only be found under the subtree traj.results, whereas parameters are used to handle data kept under traj.config, traj.parameters, and traj.derived_parameters.

Result objects can handle more than one data item and heterogeneous data. On the contrary, parameters only handle single data items. However, they can contain ranges - arrays of homogeneous data items - to allow parameter exploration.

The module contains the following parameters:

  • BaseParameter

    Abstract base class to define the parameter interface

  • Parameter

    Standard parameter that handles a variety of different data types.

  • ArrayParameter

    Parameter class for larger numpy arrays and python tuples

  • SparseParameter

    Parameter for Scipy sparse matrices

  • PickleParameter

    Parameter that can handle all objects that can be pickled

The module contains the following results:

  • BaseResult

    Abstract base class to define the result interface

  • Result

    Standard result that handles a variety of different data types

  • SparseResult

    Result that can handle Scipy sparse matrices

  • PickleResult

    Result that can handle all objects that can be pickled

Moreover, part of this module is also the ObjectTable. This is a specification of pandas DataFrames which maintains data types. It prevents auto-conversion of data to numpy data types, like python integers to numpy 64 bit integers, for instance.

Parameter

class pypet.parameter.Parameter(full_name, data=None, comment='')[source]

The standard container that handles access to simulation parameters.

Parameters are simple container objects for data values. They handle single values as well as the so called exploration range. An array containing multiple values which are accessed one after the other in individual simulation runs.

Parameter exploration is usually initiated through the trajectory see :func:~pypet.trajectory.Trajectory.f_explore and :func:~pypet.trajectory.Trajectory.f_expand.

To access the parameter’s data value one can call the f_get() method.

Parameters support the concept of locking. Once a value of the parameter has been accessed, the parameter cannot be changed anymore unless it is explicitly unlocked using f_unlock(). Locking prevents parameters from being changed during runtime of a simulation.

Supported data values for the parameter are

  • python natives (int, long, str, bool, float, complex),
  • numpy natives, arrays and matrices of type np.int8-64, np.uint8-64, np.float32-64, np.complex, np.str
  • python homogeneous non-nested tuples and lists

Note that for larger numpy arrays it is recommended to use the ArrayParameter.

In case you create a new parameter you can pass the following arguments:

Parameters:
  • full_name – The full name of the parameter. Grouping can be achieved by using colons.
  • data –

    A data value that is handled by the parameter. It is checked whether the parameter f_supports() the data. If not a TypeError is thrown. If the parameter becomes explored, the data value is kept as a default. After simulation the default value will be restored.

    The data can be accessed as follows:

    >>> param.f_get()
42

    Or using >>> param.data 42

    [It is not v_data because the data is supposed to be part of the trajectory tree or extension of the natural naming scheme and not considered as an attribute/variable of the parameter container.]

    To change the data after parameter creation one can call f_set():

    >>> param.f_set(43)
>>> param.f_get()
43
  • comment –

    A useful comment describing the parameter. The comment can be changed later on using the ‘v_comment’ variable.

    >>> param.v_comment = 'Example comment'
>>> print param.v_comment
'Example comment'
Raises:

TypeError: If data is not supported by the parameter.

Example usage:

>>> param = Parameter('traffic.mobiles.ncars',data=42, comment='I am a neat example')
f_ann_to_str()

Returns annotations as string

Equivalent to v_annotations.f_ann_to_str()

f_empty()[source]

Erases all data in the parameter.

Does not erase data from disk. So if the parameter has been stored with a service to disk and is emptied, it can be restored by loading from disk.

Raises:ParameterLockedException: If the parameter is locked.
f_get()[source]

Returns the current data value of the parameter and locks the parameter.

Raises:TypeError if the parameter is empty

Example usage:

>>> param = Parameter('groupA.groupB.myparam', comment='I am a neat example')
>>> param.f_set(44.0)
>>> param.f_get()
44.0:
f_get_annotations(*args)

Returns annotations

Equivalent to v_annotations.f_get(*args)

f_get_class_name()

Returns the name of the class i.e. return self.__class__.__name__

f_get_default()[source]

Returns the default value of the parameter and locks it.

f_get_range(copy=True)[source]

Returns a python iterable containing the exploration range.

Parameters:copy – If the range should be copied before handed over to avoid tempering with data

Example usage:

>>> param = Parameter('groupA.groupB.myparam',data=22, comment='I am a neat example')
>>> param._explore([42,43,43])
>>> param.f_get_range()
(42,43,44)
Raises:TypeError: If parameter is not explored.
f_get_range_length()

Returns the length of the parameter range.

Raises TypeError if the parameter has no range.

Does not need to be implemented if the parameter supports __len__ appropriately.

f_has_range()[source]

If the parameter has a range.

Does not have to be True if the parameter is explored. The range might be removed during pickling to save memory. Accordingly, v_explored remains True whereas f_has_range is False.

f_is_empty()[source]

True if no data has been assigned to the parameter.

Example usage:

>>> param = Parameter('myname.is.example', comment='I am _empty!')
>>> param.f_is_empty()
True
>>> param.f_set(444)
>>> param.f_is_empty()
False

True if no data has been assigned to the parameter.

Example usage:

>>> param = Parameter('myname.is.example', comment='I am _empty!')
>>> param.f_is_empty()
True
>>> param.f_set(444)
>>> param.f_is_empty()
False
f_lock()

Locks the parameter and forbids further manipulation.

Changing the data value or exploration range of the parameter are no longer allowed.

f_set(data)[source]

Sets a data value for a parameter.

Example usage:

>>> param = Parameter('groupA.groupB.myparam', comment='I am a neat example')
>>> param.f_set(44.0)
>>> param.f_get()
44.0
Raises:

ParameterLockedException: If parameter is locked

TypeError: If the type of the data value is not supported by the parameter
f_set_annotations(*args, **kwargs)

Sets annotations

Equivalent to calling v_annotations.f_set(*args,**kwargs)

f_supports(data)[source]

Checks if input data is supported by the parameter.

f_supports_fast_access()

Checks if parameter supports fast access.

A parameter supports fast access if it is NOT empty!

f_unlock()

Unlocks the locked parameter.

Please use it very carefully, or best do not use this function at all. There should better be no reason to unlock a locked parameter! The only exception I can think of is to unlock a large derived parameter after usage to subsequently call f_empty() to clear memory.

f_val_to_str()

String summary of the value handled by the parameter.

Note that representing the parameter as a string accesses its value, but for simpler debugging, this does not lock the parameter or counts as usage!

Calls __repr__ of the contained value.

func

Alternative naming, you can use node.func.name instead of node.f_func

v_annotations

Annotation feature of a trajectory node.

Store some short additional information about your nodes here. If you use the standard HDF5 storage service, they will be stored as hdf5 node attributes.

v_branch

The name of the branch/subtree, i.e. the first node below the root.

The empty string in case of root itself.

v_comment

Should be a nice descriptive comment

v_depth

Depth of the node in the trajectory tree.

v_explored

Whether parameter is explored.

Does not necessarily have to be similar to f_has_range() since the range can be deleted on pickling and the parameter remains explored.

v_full_copy

Whether or not the full parameter including the range or only the current data is copied during pickling.

If you run your simulations in multiprocessing mode, the whole trajectory and all parameters need to be pickled and are sent to the individual processes. Each process than runs an individual point in the parameter space. As a consequence, you do not need the full ranges during these calculations. Thus, if the full copy mode is set to False the parameter is pickled without the range array and you can save memory.

If you want to access the full range during individual runs, you need to set v_full_copy to True.

It is recommended NOT to do that in order to save memory and also do obey the philosophy that individual simulation runs are independent.

Example usage:

>>> import pickle
>>> param = Parameter('examples.fullcopy', data=333, comment='I show you how the copy mode works!')
>>> param._explore([1,2,3,4])
>>> dump=pickle.dumps(param)
>>> newparam = pickle.loads(dump)
>>> newparam.f_get_range()
TypeError

>>> param.v_full_copy=True
>>> dump = pickle.dumps(param)
>>> newparam=pickle.loads(dump)
>>> newparam.f_get_range()
(1,2,3,4)
v_full_name

The full name, relative to the root node.

The full name of a trajectory or single run is the empty string since it is root.

v_is_group

Whether node is a group or not (i.e. it is a leaf node)

v_is_leaf

Whether node is a leaf or not (i.e. it is a group node)

v_is_parameter

Whether the node is a parameter or not (i.e. a result)

v_is_root

Whether the group is root (True for the trajectory and a single run object)

v_location

Location relative to the root node.

The location of a trajectory or single run is the empty string since it is root.

v_locked

Whether or not the parameter is locked and prevents further modification

v_name

Name of the node

v_run_branch

If this node is hanging below a branch named run_XXXXXXXXX.

The branch name is either the name of a single run (e.g. ‘run_00000009’) or ‘trajectory’.

v_stored

Whether or not this tree node has been stored to disk before.

vars

Alternative naming, you can use node.vars.name instead of node.v_name

ArrayParameter

class pypet.parameter.ArrayParameter(full_name, data=None, comment='')[source]

Similar to the Parameter, but recommended for large numpy arrays and python tuples.

The array parameter is a bit smarter in memory management than the parameter. If a numpy array is used several times within an exploration, only one numpy array is stored by the default HDF5 storage service. For each individual run references to the corresponding numpy array are stored.

Since the ArrayParameter inherits from Parameter it also supports all other native python types.

IDENTIFIER = '__rr__'

Identifier to mark stored data as an array

f_supports(data)[source]

Checks if input data is supported by the parameter.

SparseParameter

class pypet.parameter.SparseParameter(full_name, data=None, comment='')[source]

Parameter that handles Scipy csr, csc, bsr and dia sparse matrices.

Sparse Parameter inherits from ArrayParameter and supports arrays and native python data as well.

Uses similar memory management as its parent class.

DIA_NAME_LIST = ['format', 'data', 'offsets', 'shape']

Data names for serialization of dia matrices

IDENTIFIER = '__spsp__'

Identifier to mark stored data as a sparse matrix

OTHER_NAME_LIST = ['format', 'data', 'indices', 'indptr', 'shape']

Data names for serialization of csr, csc, and bsr matrices

f_supports(data)[source]

Sparse matrices support Scipy csr, csc, bsr and dia matrices and everything their parent class the ArrayParameter supports.

PickleParameter

class pypet.parameter.PickleParameter(full_name, data=None, comment='', protocol=2)[source]

A parameter class that supports all picklable objects, and pickles everything!

If you use the default HDF5 storage service, the pickle dumps are stored to disk. Works similar to the array parameter regarding memory management (Equality of objects is based on object id).

There is no straightforward check to guarantee that data is picklable, so you have to take care that all data handled by the PickleParameter supports pickling.

You can pass the pickle protocol via protocol=2 to the constructor or change it with the v_protocol property. Default protocol is 0. Note that after storage to disk changing the protocol has no effect. If the parameter is loaded, v_protocol is set to the protocol used to store the data.

f_supports(data)[source]

There is no straightforward check if an object can be pickled and this function will always return True.

So you have to take care in advance that the item can be pickled.

v_protocol

The protocol used to pickle data, default is 0.

See pickle documentation for the protocols.

Result

class pypet.parameter.Result(full_name, *args, **kwargs)[source]

Light Container that stores basic python and numpy data.

Note that no sanity checks on individual data is made (only on outer data structure) and you have to take care, that your data is understood by the storage service. It is assumed that results tend to be large and therefore sanity checks would be too expensive.

Data that can safely be stored into a Result are:

  • python natives (int, long, str, bool, float, complex),
  • numpy natives, arrays and matrices of type np.int8-64, np.uint8-64, np.float32-64, np.complex, np.str
  • python lists and tuples of the previous types (python natives + numpy natives and arrays) Lists and tuples are not allowed to be nested and must be homogeneous, i.e. only contain data of one particular type. Only integers, or only floats, etc.
  • python dictionaries of the previous types (not nested!), data can be heterogeneous, keys must be strings. For example, one key-value pair of string and int and one key-value pair of string and float, and so on.
  • pandas DataFrames
  • ObjectTable

Note that containers should NOT be empty (like empty dicts or lists) at the time they are saved to disk. The standard HDF5 storage service cannot store empty containers! The Result emits a warning if you hand over an empty container.

Data is set on initialisation or with f_set()

Example usage:

>>> res = Result('supergroup.subgroup.myresult', comment='I am a neat example!'         [1000,2000], {'a':'b','c':333}, hitchhiker='Arthur Dent')

In case you create a new result you can pass the following arguments:

Parameters:
  • fullanme – The fullname of the result, grouping can be achieved by colons,
  • comment –

    A useful comment describing the result. The comment can later on be changed using the v_comment variable

    >>> param.v_comment
'I am a neat example!'
  • args – Data that is handled by the result. The first positional argument is stored with the name of the result. Following arguments are stored with name_X where X is the position of the argument.
  • kwargs –

    Data that is handled by the result, it is kept by the result under the names specified by the keys of kwargs.

    >>> res.f_get(0)
[1000,2000]
>>> res.f_get(1)
{'a':'b','c':'d'}
>>> res.f_get('myresult')
[1000,2000]
>>> res.f_get('hitchhiker')
'ArthurDent'
>>> res.f_get('myresult','hitchhiker')
([1000,2000], 'ArthurDent')

    Can be changed or more can be added via f_set()

    >>> result.f_set('Uno',x='y')
>>> result.f_get(0)
'Uno'
>>> result.f_get('x')
'y'

    Alternative method to put and retrieve data from the result container is via __getattr__ and __setattr__:

    >>> res.ford = 'prefect'
>>> res.ford
'prefect'
Raises:

TypeError:

If the data format in args or kwargs is not known to the result. Checks type of outer data structure, i.e. checks if you have a list or dictionary. But it does not check on individual values within dicts or lists.
f_ann_to_str()

Returns annotations as string

Equivalent to v_annotations.f_ann_to_str()

f_empty()[source]

Removes all data from the result or parameter.

If the result has already been stored to disk via a trajectory and a storage service, the data on disk is not affected by f_empty.

Yet, this function is particularly useful if you have stored very large data to disk and you want to free some memory on RAM but still keep the skeleton of your result or parameter.

Note that freeing RAM requires that all references to the data are deleted. If you reference the data somewhere else in your code, the data is not erased from RAM.

f_get(*args)[source]

Returns items handled by the result.

If only a single name is given, a single data item is returned. If several names are given, a list is returned. For integer inputs the result returns resultname_X.

If the result contains only a single entry you can call f_get() without arguments. If you call f_get() and the result contains more than one element a ValueError is thrown.

If the requested item(s) cannot be found an AttributeError is thrown.

Parameters:args – strings-names or integers
Returns:Single data item or tuple of data

Example:

>>> res = Result('supergroup.subgroup.myresult', comment='I am a neat example!'         [1000,2000], {'a':'b','c':333}, hitchhiker='Arthur Dent')
>>> res.f_get('hitchhiker')
'Arthur Dent'
>>> res.f_get(0)
[1000,2000]
>>> res.f_get('hitchhiker', 'myresult')
('Arthur Dent', [1000,2000])
f_get_annotations(*args)

Returns annotations

Equivalent to v_annotations.f_get(*args)

f_get_class_name()

Returns the class name of the parameter or result or group.

Equivalent to obj.__class__.__name__

f_is_empty()[source]

True if no data has been put into the result.

Also True if all data has been erased via f_empty().

f_remove(*args)[source]

Removes *args from the result

f_set(*args, **kwargs)[source]

Method to put data into the result.

Parameters:
  • args – The first positional argument is stored with the name of the result. Following arguments are stored with name_X where X is the position of the argument.
  • kwargs – Arguments are stored with the key as name.
Raises:

TypeError if outer data structure is not understood.

Example usage:

>>> res = Result('supergroup.subgroup.myresult', comment='I am a neat example!')
>>> res.f_set(333,42.0, mystring='String!')
>>> res.f_get('myresult')
333
>>> res.f_get('myresult_1')
42.0
>>> res.f_get(1)
42.0
>>> res.f_get('mystring')
'String!'
f_set_annotations(*args, **kwargs)

Sets annotations

Equivalent to calling v_annotations.f_set(*args,**kwargs)

f_set_single(name, item)[source]

Sets a single data item of the result.

Raises TypeError if the type of the outer data structure is not understood. Note that the type check is shallow. For example, if the data item is a list, the individual list elements are NOT checked whether their types are appropriate.

Parameters:
  • name – The name of the data item
  • item – The data item
Raises:

TypeError

Example usage:

>>> res.f_set_single('answer', 42)
>>> res.f_get('answer')
42
f_supports_fast_access()[source]

Whether or not the result supports fast access.

A result supports fast access if it contains exactly one item with the name of the result.

f_to_dict(copy=True)[source]

Returns all handled data as a dictionary.

Parameters:copy – Whether the original dictionary or a shallow copy is returned.
Returns:Data dictionary
f_translate_key(key)[source]

Translates integer indices into the appropriate names

f_val_to_str()[source]

Summarizes data handled by the result as a string.

Calls __repr__ on all handled data. Data is NOT ordered.

Truncates the string if it is longer than pypetconstants.HDF5_STRCOL_MAX_VALUE_LENGTH

Returns:string
func

Alternative naming, you can use node.func.name instead of node.f_func

v_annotations

Annotation feature of a trajectory node.

Store some short additional information about your nodes here. If you use the standard HDF5 storage service, they will be stored as hdf5 node attributes.

v_branch

The name of the branch/subtree, i.e. the first node below the root.

The empty string in case of root itself.

v_comment

Should be a nice descriptive comment

v_depth

Depth of the node in the trajectory tree.

v_full_name

The full name, relative to the root node.

The full name of a trajectory or single run is the empty string since it is root.

v_is_group

Whether node is a group or not (i.e. it is a leaf node)

v_is_leaf

Whether node is a leaf or not (i.e. it is a group node)

v_is_parameter

Whether the node is a parameter or not (i.e. a result)

v_is_root

Whether the group is root (True for the trajectory and a single run object)

v_location

Location relative to the root node.

The location of a trajectory or single run is the empty string since it is root.

v_name

Name of the node

v_run_branch

If this node is hanging below a branch named run_XXXXXXXXX.

The branch name is either the name of a single run (e.g. ‘run_00000009’) or ‘trajectory’.

v_stored

Whether or not this tree node has been stored to disk before.

vars

Alternative naming, you can use node.vars.name instead of node.v_name

SparseResult

class pypet.parameter.SparseResult(full_name, *args, **kwargs)[source]

Handles Scipy sparse matrices.

Supported Formats are csr, csc, bsr, and dia.

Subclasses the standard result and can also handle all data supported by Result.

IDENTIFIER = '__spsp__'

Identifier string to label sparse matrix data

f_ann_to_str()

Returns annotations as string

Equivalent to v_annotations.f_ann_to_str()

f_empty()

Removes all data from the result or parameter.

If the result has already been stored to disk via a trajectory and a storage service, the data on disk is not affected by f_empty.

Yet, this function is particularly useful if you have stored very large data to disk and you want to free some memory on RAM but still keep the skeleton of your result or parameter.

Note that freeing RAM requires that all references to the data are deleted. If you reference the data somewhere else in your code, the data is not erased from RAM.

f_get(*args)

Returns items handled by the result.

If only a single name is given, a single data item is returned. If several names are given, a list is returned. For integer inputs the result returns resultname_X.

If the result contains only a single entry you can call f_get() without arguments. If you call f_get() and the result contains more than one element a ValueError is thrown.

If the requested item(s) cannot be found an AttributeError is thrown.

Parameters:args – strings-names or integers
Returns:Single data item or tuple of data

Example:

>>> res = Result('supergroup.subgroup.myresult', comment='I am a neat example!'         [1000,2000], {'a':'b','c':333}, hitchhiker='Arthur Dent')
>>> res.f_get('hitchhiker')
'Arthur Dent'
>>> res.f_get(0)
[1000,2000]
>>> res.f_get('hitchhiker', 'myresult')
('Arthur Dent', [1000,2000])
f_get_annotations(*args)

Returns annotations

Equivalent to v_annotations.f_get(*args)

f_get_class_name()

Returns the class name of the parameter or result or group.

Equivalent to obj.__class__.__name__

f_is_empty()

True if no data has been put into the result.

Also True if all data has been erased via f_empty().

f_remove(*args)

Removes *args from the result

f_set(*args, **kwargs)

Method to put data into the result.

Parameters:
  • args – The first positional argument is stored with the name of the result. Following arguments are stored with name_X where X is the position of the argument.
  • kwargs – Arguments are stored with the key as name.
Raises:

TypeError if outer data structure is not understood.

Example usage:

>>> res = Result('supergroup.subgroup.myresult', comment='I am a neat example!')
>>> res.f_set(333,42.0, mystring='String!')
>>> res.f_get('myresult')
333
>>> res.f_get('myresult_1')
42.0
>>> res.f_get(1)
42.0
>>> res.f_get('mystring')
'String!'
f_set_annotations(*args, **kwargs)

Sets annotations

Equivalent to calling v_annotations.f_set(*args,**kwargs)

f_set_single(name, item)[source]

Sets a single data item of the result.

Raises TypeError if the type of the outer data structure is not understood. Note that the type check is shallow. For example, if the data item is a list, the individual list elements are NOT checked whether their types are appropriate.

Parameters:
  • name – The name of the data item
  • item – The data item
Raises:

TypeError

Example usage:

>>> res.f_set_single('answer', 42)
>>> res.f_get('answer')
42
f_supports_fast_access()

Whether or not the result supports fast access.

A result supports fast access if it contains exactly one item with the name of the result.

f_to_dict(copy=True)

Returns all handled data as a dictionary.

Parameters:copy – Whether the original dictionary or a shallow copy is returned.
Returns:Data dictionary
f_translate_key(key)

Translates integer indices into the appropriate names

f_val_to_str()

Summarizes data handled by the result as a string.

Calls __repr__ on all handled data. Data is NOT ordered.

Truncates the string if it is longer than pypetconstants.HDF5_STRCOL_MAX_VALUE_LENGTH

Returns:string
func

Alternative naming, you can use node.func.name instead of node.f_func

v_annotations

Annotation feature of a trajectory node.

Store some short additional information about your nodes here. If you use the standard HDF5 storage service, they will be stored as hdf5 node attributes.

v_branch

The name of the branch/subtree, i.e. the first node below the root.

The empty string in case of root itself.

v_comment

Should be a nice descriptive comment

v_depth

Depth of the node in the trajectory tree.

v_full_name

The full name, relative to the root node.

The full name of a trajectory or single run is the empty string since it is root.

v_is_group

Whether node is a group or not (i.e. it is a leaf node)

v_is_leaf

Whether node is a leaf or not (i.e. it is a group node)

v_is_parameter

Whether the node is a parameter or not (i.e. a result)

v_is_root

Whether the group is root (True for the trajectory and a single run object)

v_location

Location relative to the root node.

The location of a trajectory or single run is the empty string since it is root.

v_name

Name of the node

v_run_branch

If this node is hanging below a branch named run_XXXXXXXXX.

The branch name is either the name of a single run (e.g. ‘run_00000009’) or ‘trajectory’.

v_stored

Whether or not this tree node has been stored to disk before.

vars

Alternative naming, you can use node.vars.name instead of node.v_name

PickleResult

class pypet.parameter.PickleResult(full_name, *args, **kwargs)[source]

Result that digest everything and simply pickles it!

Note that it is not checked whether data can be pickled, so take care that it works!

You can pass the pickle protocol via protocol=2 to the constructor or change it with the v_protocol property. Default protocol is 0.

Note that after storage to disk changing the protocol has no effect. If the parameter is loaded, v_protocol is set to a protocol used to store an item. Note that items are reconstructed from a dictionary and the protocol is taken from the first one found in the dictionary. This is a rather arbitrary choice. Yet, the underlying assumption is that all items were pickled with the same protocol, which is the general case.

f_set_single(name, item)[source]

Adds a single data item to the pickle result.

Note that it is NOT checked if the item can be pickled!

v_protocol

The protocol used to pickle data, default is 0.

See pickle documentation for the protocols.

Object Table

class pypet.parameter.ObjectTable(data=None, index=None, columns=None, copy=False)[source]

Wrapper class for pandas DataFrames.

It creates data frames with dtype=object.

Data stored into an object table preserves its original type when stored to disk. For instance, a python int is not automatically converted to a numpy 64 bit integer (np.int64).

The object table serves as a data structure to hand data to a storage service.

Example Usage:

>>> ObjectTable(data={'characters':['Luke', 'Han', 'Spock'], 'Random_Values' :[42,43,44] })

Creates the following table:

Index Random_Values characters
0 42 Luke
1 43 Han
2 44 Spock

The Abstract Base Classes of Parameters and Results

These classes serve as a reference if you want to implement your own parameter or result. Therefore, also private functions are listed.

class pypet.parameter.BaseParameter(full_name, comment='')[source]

Abstract class that specifies the methods for a trajectory parameter.

Parameters are simple container objects for data values. They handle single values as well as ranges of potential values. These range arrays contain multiple values which are accessed one after the other in individual simulation runs.

Parameter exploration is usually initiated through the trajectory see f_explore() and f_expand().

To access the parameter’s data value one can call the f_get() method.

Parameters support the concept of locking. Once a value of the parameter has been accessed, the parameter cannot be changed anymore unless it is explicitly unlocked using f_unlock(). This prevents parameters from being changed during runtime of a simulation.

If multiprocessing is desired the parameter must be picklable!

Parameters:
  • full_name – The full name of the parameter in the trajectory tree, groupings are separated by a colon: fullname = ‘supergroup.subgroup.paramname’
  • comment – A useful comment describing the parameter: comment = ‘Some useful text, dude!’
__all_slots__ = {'__weakref__', '_annotations', '_branch', '_comment', '_depth', '_explored', '_full_copy', '_full_name', '_func', '_is_leaf', '_is_parameter', '_locked', '_logger', '_name', '_run_branch', '_stored', '_vars'}
__class__

alias of pypet.slots.MetaSlotMachine

__delattr__

Implement delattr(self, name).

__dir__()

Includes all slots in the dir method

__eq__

Return self==value.

__format__()

Default object formatter.

__ge__

Return self>=value.

__getattribute__

Return getattr(self, name).

__getitem__(item)[source]

Equivalent to f_get_range()[idx]

Raises:TypeError if parameter has no range
__getstate__()

Called for pickling.

Removes the logger to allow pickling and returns a copy of __dict__.

__gt__

Return self>value.

__hash__

Return hash(self).

__init__(full_name, comment='')[source]

Initialize self. See help(type(self)) for accurate signature.

__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__

Return self<=value.

__lt__

Return self<value.

__module__ = 'pypet.parameter'
__ne__

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__()[source]

Return repr(self).

__setattr__

Implement setattr(self, name, value).

__setstate__(statedict)

Called after loading a pickle dump.

Restores __dict__ from statedict and adds a new logger.

__sizeof__()

Size of object in memory, in bytes.

__slots__ = ('_locked', '_full_copy', '_explored')
__str__()[source]

String representation of the Parameter

Output format is:<class_name> full_name (len:X, `comment): value`. If comment is the empty string, the comment is omitted. If the parameter is not explored the length is omitted.

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

_annotations
_branch
_comment
_depth
_equal_values(val1, val2)[source]

Checks if the parameter considers two values as equal.

This is important for the trajectory in case of merging. In case you want to delete duplicate parameter points, the trajectory needs to know when two parameters are equal. Since equality is not always implemented by values handled by parameters in the same way, the parameters need to judge whether their values are equal.

The straightforward example here is a numpy array. Checking for equality of two numpy arrays yields a third numpy array containing truth values of a piecewise comparison. Accordingly, the parameter could judge two numpy arrays equal if ALL of the numpy array elements are equal.

In this BaseParameter class values are considered to be equal if they obey the function nested_equal(). You might consider implementing a different equality comparison in your subclass.

Raises:TypeError: If both values are not supported by the parameter.
_expand(iterable)[source]

Similar to _explore() but appends to the exploration range.

Parameters:

iterable – An iterable specifying the exploration range.
Raises:

ParameterLockedException: If the parameter is locked

TypeError: If the parameter did not have a range before

Example usage:

>>> param = Parameter('groupA.groupB.myparam', data=3.13, comment='I am a neat example')
>>> param._explore([3.0,2.0,1.0])
>>> param._expand([42.0,43.0])
>>> param.f_get_range()
(3.0,2.0,1.0,42.0,43.0)

ABSTRACT: Needs to be defined in subclass

_explore(iterable)[source]

The method to explore a parameter and create a range of entries.

Parameters:

iterable –

An iterable specifying the exploration range

For example:

>>> param = Parameter('groupA.groupB.myparam',data=22.33,              comment='I am a neat example')
>>> param._explore([3.0,2.0,1.0])

Raises:

ParameterLockedException: If the parameter is locked

TypeError: If the parameter is already explored

ABSTRACT: Needs to be defined in subclass

_explored
_full_copy
_full_name
_func
_is_leaf
_is_parameter
_load(load_dict)

Method called by the storage service to reconstruct the original result.

Data contained in the load_dict is equal to the data provided by the result or parameter when previously called with _store().

Parameters:load_dict – The dictionary containing basic data structures, see also _store().

ABSTRACT: Needs to be implemented by subclass

_load_flags()

Currently not used because I let the storage service infer how to load stuff from the data itself.

If you write your own parameter or result you can implement this function to make specifications on how to load data, see also pypet.storageservice.HDF5StorageService.store().

Returns:{} (Empty dictionary)
_locked
_logger
_name
_rename(full_name)

Renames the tree node

_restore_default()[source]

Restores original data if changed due to exploration.

If a Parameter is explored, the actual data is changed over the course of different simulations. This method restores the original data assigned before exploration.

ABSTRACT: Needs to be defined in subclass

_run_branch
_set_details(depth, branch, run_branch)

Sets some details for internal handling.

_set_logger(name=None)

Adds a logger with a given name.

If no name is given, name is constructed as type(self).__name__.

_set_parameter_access(idx=0)[source]

Sets the current value according to the idx in the exploration range.

Prepares the parameter for further usage, and tells it which point in the parameter space should be accessed by calls to f_get().

Parameters:

idx –

The index within the exploration range.

If the parameter has no range, the single data value is considered regardless of the value of idx. Raises ValueError if the parameter is explored and idx>=len(param).
Raises:

ValueError:

If the parameter has a range and idx is larger or equal to the length of the parameter.

Example usage:

>>> param = Parameter('groupA.groupB.myparam',data=22.33, comment='I am a neat example')
>>> param._explore([42.0,43.0,44.0])
>>> param._set_parameter_access(idx=1)
>>> param.f_get()
43.0

ABSTRACT: Needs to be defined in subclass

_shrink()[source]

If a parameter is explored, i.e. it has a range, the whole exploration range is deleted.

Note that this function does not erase data from disk. So if the parameter has been stored with a service to disk and is shrunk, it can be restored by loading from disk.

Raises:

ParameterLockedException: If the parameter is locked

TypeError: If the parameter has no range

ABSTRACT: Needs to be defined in subclass

_store()

Method called by the storage service for serialization.

The method converts the parameter’s or result’s value(s) into simple data structures that can be stored to disk. Returns a dictionary containing these simple structures.

Understood basic structures are

  • python natives (int, long, str,bool,float,complex)
  • python lists and tuples
  • numpy natives arrays, and matrices of type np.int8-64, np.uint8-64, np.float32-64, np.complex, np.str
  • python dictionaries of the previous types (flat not nested!)
  • pandas data frames
  • object tables (see ObjectTable)
Returns:A dictionary containing basic data structures.

ABSTRACT: Needs to be implemented by subclass

_store_flags()

Currently not used because I let the storage service infer how to store stuff from the data itself.

If you write your own parameter or result you can implement this function to make specifications on how to store data, see also pypet.storageservice.HDF5StorageService.store().

Returns:{} (Empty dictionary)
_stored
_values_of_same_type(val1, val2)[source]

Checks if two values agree in type.

For example, two 32 bit integers would be of same type, but not a string and an integer, nor a 64 bit and a 32 bit integer.

This is important for exploration. You are only allowed to explore data that is of the same type as the default value.

One could always come up with a trivial solution of type(val1) is type(val2). But sometimes your parameter does want even more strict equality or less type equality.

For example, the Parameter has a stricter sense of type equality regarding numpy arrays. In order to have two numpy arrays of the same type, they must also agree in shape. However, the ArrayParameter, considers all numpy arrays as of being of same type regardless of their shape.

Moreover, the SparseParameter considers all supported sparse matrices (csc, csr, bsr, dia) as being of the same type. You can make explorations using all these four types at once.

The difference in how strict types are treated arises from the way parameter data is stored to disk and how the parameters hand over their data to the storage service (see pypet.parameter.BaseParameter._store()).

The Parameter puts all it’s data in an ObjectTable which has strict constraints on the column sizes. This means that numpy array columns only accept numpy arrays with a particular size. In contrast, the array and sparse parameter hand over their data as individual items which yield individual entries in the hdf5 node. In order to see what I mean simply run an experiment with all 3 parameters, explore all of them, and take a look at the resulting hdf5 file!

However, this BaseParameter class implements the straightforward version of type(val1) is type(val2) to consider data to be of the same type.

Raises:TypeError: if both values are not supported by the parameter.
_vars
f_ann_to_str()

Returns annotations as string

Equivalent to v_annotations.f_ann_to_str()

f_empty()[source]

Erases all data in the parameter.

Does not erase data from disk. So if the parameter has been stored with a service to disk and is emptied, it can be restored by loading from disk.

Raises:ParameterLockedException: If the parameter is locked.

ABSTRACT: Needs to be defined in subclass

f_get()[source]

Returns the current data value of the parameter and locks the parameter.

Raises:TypeError if the parameter is empty

Example usage:

>>> param = Parameter('groupA.groupB.myparam', comment='I am a neat example')
>>> param.f_set(44.0)
>>> param.f_get()
44.0:

ABSTRACT: Needs to be defined in subclass

f_get_annotations(*args)

Returns annotations

Equivalent to v_annotations.f_get(*args)

f_get_class_name()[source]

Returns the name of the class i.e. return self.__class__.__name__

f_get_default()[source]

Returns the default value of the parameter and locks it.

f_get_range(copy=True)[source]

Returns an iterable to iterate over the values of the exploration range.

Note that the returned values should be either a copy of the exploration range unless explicetly requested otherwise.

Parameters:copy – If range should be copied to avoid tempering with data.
Returns:Iterable
Raises:TypeError if the parameter is not explored

Example usage:

>>> param = Parameter('groupA.groupB.myparam',data=22, comment='I am a neat example')
>>> param._explore([42,43,43])
>>> param.f_get_range()
(42,43,44)

ABSTRACT: Needs to be defined in subclass

f_get_range_length()[source]

Returns the length of the parameter range.

Raises TypeError if the parameter has no range.

Does not need to be implemented if the parameter supports __len__ appropriately.

f_has_range()[source]

Returns true if the parameter contains a range array.

Not necessarily equal to v_explored if the range is removed on pickling due to v_full_copy=False.

ABSTRACT: Needs to be defined in subclass

f_is_empty()[source]

True if no data has been assigned to the parameter.

Example usage:

>>> param = Parameter('myname.is.example', comment='I am _empty!')
>>> param.f_is_empty()
True
>>> param.f_set(444)
>>> param.f_is_empty()
False
f_lock()[source]

Locks the parameter and forbids further manipulation.

Changing the data value or exploration range of the parameter are no longer allowed.

f_set(data)[source]

Sets a data value for a parameter.

Example usage:

>>> param = Parameter('groupA.groupB.myparam', comment='I am a neat example')
>>> param.f_set(44.0)
>>> param.f_get()
44.0
Raises:

ParameterLockedException: If parameter is locked

TypeError: If the type of the data value is not supported by the parameter

ABSTRACT: Needs to be defined in subclass

f_set_annotations(*args, **kwargs)

Sets annotations

Equivalent to calling v_annotations.f_set(*args,**kwargs)

f_supports(data)[source]

Checks whether the data is supported by the parameter.

f_supports_fast_access()[source]

Checks if parameter supports fast access.

A parameter supports fast access if it is NOT empty!

f_unlock()[source]

Unlocks the locked parameter.

Please use it very carefully, or best do not use this function at all. There should better be no reason to unlock a locked parameter! The only exception I can think of is to unlock a large derived parameter after usage to subsequently call f_empty() to clear memory.

f_val_to_str()[source]

String summary of the value handled by the parameter.

Note that representing the parameter as a string accesses its value, but for simpler debugging, this does not lock the parameter or counts as usage!

Calls __repr__ of the contained value.

func

Alternative naming, you can use node.func.name instead of node.f_func

v_annotations

Annotation feature of a trajectory node.

Store some short additional information about your nodes here. If you use the standard HDF5 storage service, they will be stored as hdf5 node attributes.

v_branch

The name of the branch/subtree, i.e. the first node below the root.

The empty string in case of root itself.

v_comment

Should be a nice descriptive comment

v_depth

Depth of the node in the trajectory tree.

v_explored

Whether parameter is explored.

Does not necessarily have to be similar to f_has_range() since the range can be deleted on pickling and the parameter remains explored.

v_full_copy

Whether or not the full parameter including the range or only the current data is copied during pickling.

If you run your simulations in multiprocessing mode, the whole trajectory and all parameters need to be pickled and are sent to the individual processes. Each process than runs an individual point in the parameter space. As a consequence, you do not need the full ranges during these calculations. Thus, if the full copy mode is set to False the parameter is pickled without the range array and you can save memory.

If you want to access the full range during individual runs, you need to set v_full_copy to True.

It is recommended NOT to do that in order to save memory and also do obey the philosophy that individual simulation runs are independent.

Example usage:

>>> import pickle
>>> param = Parameter('examples.fullcopy', data=333, comment='I show you how the copy mode works!')
>>> param._explore([1,2,3,4])
>>> dump=pickle.dumps(param)
>>> newparam = pickle.loads(dump)
>>> newparam.f_get_range()
TypeError

>>> param.v_full_copy=True
>>> dump = pickle.dumps(param)
>>> newparam=pickle.loads(dump)
>>> newparam.f_get_range()
(1,2,3,4)
v_full_name

The full name, relative to the root node.

The full name of a trajectory or single run is the empty string since it is root.

v_is_group

Whether node is a group or not (i.e. it is a leaf node)

v_is_leaf

Whether node is a leaf or not (i.e. it is a group node)

v_is_parameter

Whether the node is a parameter or not (i.e. a result)

v_is_root

Whether the group is root (True for the trajectory and a single run object)

v_location

Location relative to the root node.

The location of a trajectory or single run is the empty string since it is root.

v_locked

Whether or not the parameter is locked and prevents further modification

v_name

Name of the node

v_run_branch

If this node is hanging below a branch named run_XXXXXXXXX.

The branch name is either the name of a single run (e.g. ‘run_00000009’) or ‘trajectory’.

v_stored

Whether or not this tree node has been stored to disk before.

vars

Alternative naming, you can use node.vars.name instead of node.v_name

class pypet.parameter.BaseResult(full_name, comment='')[source]

Abstract base API for results.

Compared to parameters (see BaseParameter) results are also initialised with a full name and a comment. Yet, results can contain more than a single value and heterogeneous data.

__all_slots__ = {'__weakref__', '_annotations', '_branch', '_comment', '_depth', '_full_name', '_func', '_is_leaf', '_is_parameter', '_logger', '_name', '_run_branch', '_stored', '_vars'}
__class__

alias of pypet.slots.MetaSlotMachine

__delattr__

Implement delattr(self, name).

__dir__()

Includes all slots in the dir method

__eq__

Return self==value.

__format__()

Default object formatter.

__ge__

Return self>=value.

__getattribute__

Return getattr(self, name).

__getstate__()

Called for pickling.

Removes the logger to allow pickling and returns a copy of __dict__.

__gt__

Return self>value.

__hash__

Return hash(self).

__init__(full_name, comment='')[source]

Initialize self. See help(type(self)) for accurate signature.

__init_subclass__()

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

__le__

Return self<=value.

__lt__

Return self<value.

__module__ = 'pypet.parameter'
__ne__

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__

Return repr(self).

__setattr__

Implement setattr(self, name, value).

__setstate__(statedict)

Called after loading a pickle dump.

Restores __dict__ from statedict and adds a new logger.

__sizeof__()

Size of object in memory, in bytes.

__slots__ = ()
__str__()

String representation of the parameter or result.

If not specified in subclass this is simply the full name.

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

__weakref__

list of weak references to the object (if defined)

_annotations
_branch
_comment
_depth
_full_name
_func
_is_leaf
_is_parameter
_load(load_dict)

Method called by the storage service to reconstruct the original result.

Data contained in the load_dict is equal to the data provided by the result or parameter when previously called with _store().

Parameters:load_dict – The dictionary containing basic data structures, see also _store().

ABSTRACT: Needs to be implemented by subclass

_load_flags()

Currently not used because I let the storage service infer how to load stuff from the data itself.

If you write your own parameter or result you can implement this function to make specifications on how to load data, see also pypet.storageservice.HDF5StorageService.store().

Returns:{} (Empty dictionary)
_logger
_name
_rename(full_name)

Renames the tree node

_run_branch
_set_details(depth, branch, run_branch)

Sets some details for internal handling.

_set_logger(name=None)

Adds a logger with a given name.

If no name is given, name is constructed as type(self).__name__.

_store()

Method called by the storage service for serialization.

The method converts the parameter’s or result’s value(s) into simple data structures that can be stored to disk. Returns a dictionary containing these simple structures.

Understood basic structures are

  • python natives (int, long, str,bool,float,complex)
  • python lists and tuples
  • numpy natives arrays, and matrices of type np.int8-64, np.uint8-64, np.float32-64, np.complex, np.str
  • python dictionaries of the previous types (flat not nested!)
  • pandas data frames
  • object tables (see ObjectTable)
Returns:A dictionary containing basic data structures.

ABSTRACT: Needs to be implemented by subclass

_store_flags()

Currently not used because I let the storage service infer how to store stuff from the data itself.

If you write your own parameter or result you can implement this function to make specifications on how to store data, see also pypet.storageservice.HDF5StorageService.store().

Returns:{} (Empty dictionary)
_stored
_vars
f_ann_to_str()

Returns annotations as string

Equivalent to v_annotations.f_ann_to_str()

f_empty()

Removes all data from the result or parameter.

If the result has already been stored to disk via a trajectory and a storage service, the data on disk is not affected by f_empty.

Yet, this function is particularly useful if you have stored very large data to disk and you want to free some memory on RAM but still keep the skeleton of your result or parameter.

Note that freeing RAM requires that all references to the data are deleted. If you reference the data somewhere else in your code, the data is not erased from RAM.

ABSTRACT: Needs to be implemented by subclass

f_get_annotations(*args)

Returns annotations

Equivalent to v_annotations.f_get(*args)

f_get_class_name()

Returns the class name of the parameter or result or group.

Equivalent to obj.__class__.__name__

f_is_empty()

Returns true if no data is handled by a result or parameter.

ABSTRACT: Needs to be implemented by subclass

f_set_annotations(*args, **kwargs)

Sets annotations

Equivalent to calling v_annotations.f_set(*args,**kwargs)

f_supports_fast_access()

Whether or not fast access can be supported by the parameter or result.

ABSTRACT: Needs to be implemented by subclass.

f_val_to_str()

Returns a string summarizing the data handled by the parameter or result

ABSTRACT: Needs to be implemented by subclass, otherwise the empty string is returned.

func

Alternative naming, you can use node.func.name instead of node.f_func

v_annotations

Annotation feature of a trajectory node.

Store some short additional information about your nodes here. If you use the standard HDF5 storage service, they will be stored as hdf5 node attributes.

v_branch

The name of the branch/subtree, i.e. the first node below the root.

The empty string in case of root itself.

v_comment

Should be a nice descriptive comment

v_depth

Depth of the node in the trajectory tree.

v_full_name

The full name, relative to the root node.

The full name of a trajectory or single run is the empty string since it is root.

v_is_group

Whether node is a group or not (i.e. it is a leaf node)

v_is_leaf

Whether node is a leaf or not (i.e. it is a group node)

v_is_parameter

Whether the node is a parameter or not (i.e. a result)

v_is_root

Whether the group is root (True for the trajectory and a single run object)

v_location

Location relative to the root node.

The location of a trajectory or single run is the empty string since it is root.

v_name

Name of the node

v_run_branch

If this node is hanging below a branch named run_XXXXXXXXX.

The branch name is either the name of a single run (e.g. ‘run_00000009’) or ‘trajectory’.

v_stored

Whether or not this tree node has been stored to disk before.

vars

Alternative naming, you can use node.vars.name instead of node.v_name