The Trajectory and Group Nodes

Trajectory

class pypet.trajectory.Trajectory(name='my_trajectory', add_time=False, comment='', dynamic_imports=None, wildcard_functions=None, storage_service=None, **kwargs)[source]

The trajectory manages results and parameters.

The trajectory provides all functionality to define how the parameter space of your simulation should be explored. During single runs based on a particular parameter point, the functionality fo the trajectory is reduced.

You can add four types of data to the trajectory:

  • Config:

    These are special parameters specifying modalities of how to run your simulations. Changing a config parameter should NOT have any influence on the results you obtain from your simulations.

    They specify runtime environment parameters like how many CPUs you use for multiprocessing etc.

    In fact, if you use the default runtime environment of this project, the environment will add some config parameters to your trajectory.

    The method to add more config is f_add_config()

    Config parameters are put into the subtree traj.config (with traj being your trajectory instance).

  • Parameters:

    These are your primary ammunition in numerical simulations. They specify how your simulation works. They can only be added before the actual running of the simulation exploring the parameter space. They can be added via f_add_parameter() and be explored using f_explore(). Or to expand an existing trajectory use f_expand().

    Your parameters should encompass all values that completely define your simulation, I recommend also storing random number generator seeds as parameters to guarantee that a simulation can be repeated exactly the way it was run the first time.

    Parameters are put into the subtree traj.parameters.

  • Derived Parameters:

    They are not much different from parameters except that they can be added anytime.

    Conceptually this encompasses stuff that is intermediately computed from the original parameters. For instance, as your original parameters you have a random number seed and some other parameters. From these you compute a connection matrix for a neural network. This connection matrix could be stored as a derived parameter.

    Derived parameters are added via f_add_derived_parameter().

    Derived parameters are put into the subtree traj.derived_parameters. They are further sorted into traj.derived_parameters.runs.run_XXXXXXXX if they were added during a single run. XXXXXXXX is replaced by the index of the corresponding run, for example run_00000001.

  • Results:

    Result are added via the f_add_result(). They are kept under the subtree traj.results and are further sorted into traj.results.runs.run_XXXXXXXX if they are added during a single run.

There are several ways to access the parameters and results, to learn about these, fast access, and natural naming see Accessing Data in the Trajectory.

In case you create a new trajectory you can pass the following arguments:

Parameters:
  • name – Name of the trajectory, if add_time=True the current time is added as a string to the parameter name.
  • add_time – Boolean whether to add the current time in human readable format to the trajectory name.
  • comment – A useful comment describing the trajectory.
  • dynamic_imports

    If you’ve written a custom parameter that needs to be loaded dynamically during runtime, this needs to be specified here as a list of classes or strings naming classes and there module paths. For example: dynamic_imports = [‘pypet.parameter.PickleParameter’,MyCustomParameter]

    If you only have a single class to import, you do not need the list brackets: dynamic_imports = ‘pypet.parameter.PickleParameter’

  • wildcard_functions – Dictionary of wildcards like $ and corresponding functions that are called upon finding such a wildcard.
  • storage_service – Pass a storage service used by the Trajectory. Alternatively, pass a constructor and other **kwargs are passed onto the constructor.
  • kwargs – Other arguments passed to the storage service constructor
Raises:
ValueError: If the name of the trajectory contains invalid characters or

not all additional keyword arguments are used.

TypeError: If the dynamically imported classes are not classes or strings.

Example usage:

>>> traj = Trajectory('ExampleTrajectory', dynamic_imports=['Some.custom.class'],    comment = 'I am a neat example!', storage_service=HDF5StorageService,     filename='experiment.hdf5', file_title='Experiments')
f_add_config(*args, **kwargs)[source]

Adds a config parameter under the current group.

Similar to f_add_parameter().

If current group is the trajectory the prefix ‘config’ is added to the name.

ATTENTION: This function is not available during a single run!

f_add_config_group(*args, **kwargs)[source]

Adds an empty config group under the current node.

Adds the full name of the current node as prefix to the name of the group. If current node is the trajectory (root), the prefix ‘config’ is added to the full name.

The name can also contain subgroups separated via colons, for example: name=subgroup1.subgroup2.subgroup3. These other parent groups will be automatically be created.

ATTENTION: This function is not available during a single run!

f_add_parameter(*args, **kwargs)[source]

Adds a parameter under the current node.

There are two ways to add a new parameter either by adding a parameter instance:

>>> new_parameter = Parameter('group1.group2.myparam', data=42, comment='Example!')
>>> traj.f_add_parameter(new_parameter)

Or by passing the values directly to the function, with the name being the first (non-keyword!) argument:

>>> traj.f_add_parameter('group1.group2.myparam', 42, comment='Example!')

If you want to create a different parameter than the standard parameter, you can give the constructor as the first (non-keyword!) argument followed by the name (non-keyword!):

>>> traj.f_add_parameter(PickleParameter,'group1.group2.myparam', data=42, comment='Example!')

The full name of the current node is added as a prefix to the given parameter name. If the current node is the trajectory the prefix ‘parameters’ is added to the name.

Note, all non-keyword and keyword parameters apart from the optional constructor are passed on as is to the constructor.

Moreover, you always should specify a default data value of a parameter, even if you want to explore it later.

ATTENTION: This function is not available during a single run!

f_add_parameter_group(*args, **kwargs)[source]

Adds an empty parameter group under the current node.

Can be called with f_add_parameter_group('MyName', 'this is an informative comment') or f_add_parameter_group(name='MyName', comment='This is an informative comment') or with a given new group instance: f_add_parameter_group(ParameterGroup('MyName', comment='This is a comment')).

Adds the full name of the current node as prefix to the name of the group. If current node is the trajectory (root), the prefix ‘parameters’ is added to the full name.

The name can also contain subgroups separated via colons, for example: name=subgroup1.subgroup2.subgroup3. These other parent groups will be automatically created.

ATTENTION: This function is not available during a single run!

f_add_to_dynamic_imports(dynamic_imports)[source]

Adds classes or paths to classes to the trajectory to create custom parameters.

param dynamic_imports:
 

If you’ve written custom parameter that needs to be loaded dynamically during runtime, this needs to be specified here as a list of classes or strings naming classes and there module paths. For example: dynamic_imports = [‘pypet.parameter.PickleParameter’,MyCustomParameter]

If you only have a single class to import, you do not need the list brackets: dynamic_imports = ‘pypet.parameter.PickleParameter’

ATTENTION: This function is not available during a single run!

f_add_wildcard_functions(func_dict)[source]

#TODO

f_backup(**kwargs)[source]

Backs up the trajectory with the given storage service.

Arguments of kwargs are directly passed to the storage service, for the HDF5StorageService you can provide the following argument:

param backup_filename:
 

Name of file where to store the backup.

In case you use the standard HDF5 storage service and backup_filename=None, the file will be chosen automatically. The backup file will be in the same folder as your hdf5 file and named ‘backup_XXXXX.hdf5’ where ‘XXXXX’ is the name of your current trajectory.

ATTENTION: This function is not available during a single run!

f_copy(copy_leaves=True, with_links=True)[source]

Returns a shallow copy of a trajectory.

Parameters:
  • copy_leaves

    If leaves should be shallow copied or simply referred to by both trees. Shallow copying is established using the copy module.

    Accepts the setting 'explored' to only copy explored parameters. Note that v_full_copy determines how these will be copied.

  • with_links – If links should be ignored or followed and copied as well
Returns:

A shallow copy

f_delete_item(item, *args, **kwargs)[source]

Deletes a single item, see f_delete_items()

f_delete_items(iterator, *args, **kwargs)[source]

Deletes items from storage on disk.

Per default the item is NOT removed from the trajectory.

Links are NOT deleted on the hard disk, please delete links manually before deleting data!

Parameters:
  • iterator – A sequence of items you want to remove. Either the instances themselves or strings with the names of the items.
  • remove_from_trajectory – If items should also be removed from trajectory. Default is False.
  • args – Additional arguments passed to the storage service
  • kwargs

    Additional keyword arguments passed to the storage service

    If you use the standard hdf5 storage service, you can pass the following additional keyword argument:

    param delete_only:
     You can partially delete leaf nodes. Specify a list of parts of the result node that should be deleted like delete_only=[‘mystuff’,’otherstuff’]. This wil only delete the hdf5 sub parts mystuff and otherstuff from disk. BE CAREFUL, erasing data partly happens at your own risk. Depending on how complex the loading process of your result node is, you might not be able to reconstruct any data due to partially deleting some of it.

    Be aware that you need to specify the names of parts as they were stored to HDF5. Depending on how your leaf construction works, this may differ from the names the data might have in your leaf in the trajectory container.

    If the hdf5 nodes you specified in delete_only cannot be found a warning is issued.

    Note that massive deletion will fragment your HDF5 file. Try to avoid changing data on disk whenever you can.

    If you want to erase a full node, simply ignore this argument or set to None.

    param remove_from_item:
     If data that you want to delete from storage should also be removed from the items in iterator if they contain these. Default is False.
    param recursive:
     If you want to delete a group node and it has children you need to set recursive to True. Default is `False.

Deletes a single link see f_delete_links()

Deletes several links from the hard disk.

Links can be passed as a string 'groupA.groupB.linkA' or as a tuple containing the node from which the link should be removed and the name of the link (groupWithLink, 'linkA').

f_expand(build_dict, fail_safe=True)[source]
Similar to f_explore(), but can be used to enlarge

already completed trajectories.

Please ensure before usage, that all explored parameters are loaded!

param build_dict:
 

Dictionary containing the expansion

param fail_safe:
 

If old ranges should be deep-copied in order to allow to restore the original exploration if something fails during expansion. Set to False if deep-copying your parameter ranges causes errors.

raises:

TypeError: If not all explored parameters are enlarged

AttributeError: If keys of dictionary cannot be found in the trajectory

NotUniqueNodeError:

If dictionary keys do not unambiguously map to single parameters

ValueError: If not all explored parameter ranges are of the same length

ATTENTION: This function is not available during a single run!

f_explore(build_dict)[source]

Prepares the trajectory to explore the parameter space.

To explore the parameter space you need to provide a dictionary with the names of the parameters to explore as keys and iterables specifying the exploration ranges as values.

All iterables need to have the same length otherwise a ValueError is raised. A ValueError is also raised if the names from the dictionary map to groups or results and not parameters.

If your trajectory is already explored but not stored yet and your parameters are not locked you can add new explored parameters to the current ones if their iterables match the current length of the trajectory.

Raises an AttributeError if the names from the dictionary are not found at all in the trajectory and NotUniqueNodeError if the keys not unambiguously map to single parameters.

Raises a TypeError if the trajectory has been stored already, please use f_expand() then instead.

Example usage:

>>> traj.f_explore({'groupA.param1' : [1,2,3,4,5], 'groupA.param2':['a','b','c','d','e']})

Could also be called consecutively:

>>> traj.f_explore({'groupA.param1' : [1,2,3,4,5]})
>>> traj.f_explore({'groupA.param2':['a','b','c','d','e']})

NOTE:

Since parameters are very conservative regarding the data they accept (see Values supported by Parameters), you sometimes won’t be able to use Numpy arrays for exploration as iterables.

For instance, the following code snippet won’t work:

import numpy a np
from pypet.trajectory import Trajectory
traj = Trajectory()
traj.f_add_parameter('my_float_parameter', 42.4,
                     comment='My value is a standard python float')

traj.f_explore( { 'my_float_parameter': np.arange(42.0, 44.876, 0.23) } )

This will result in a TypeError because your exploration iterable np.arange(42.0, 44.876, 0.23) contains numpy.float64 values whereas you parameter is supposed to use standard python floats.

Yet, you can use Numpys tolist() function to overcome this problem:

traj.f_explore( { 'my_float_parameter': np.arange(42.0, 44.876, 0.23).tolist() } )

Or you could specify your parameter directly as a numpy float:

traj.f_add_parameter('my_float_parameter', np.float64(42.4),
                       comment='My value is a numpy 64 bit float')

ATTENTION: This function is not available during a single run!

f_finalize_run(store_meta_data=True, clean_up=True)[source]

Can be called to finish a run if manually started.

Does NOT reset the index of the run, i.e. f_restore_default should be called manually if desired.

Does NOT store any data (except meta data) so you have to call f_store manually before to avoid data loss.

Parameters:
  • store_meta_data – If meta data like the runtime should be stored
  • clean_up – If data added during the run should be cleaned up. Only works if turn_into_run was set to True.
f_find_idx(name_list, predicate)[source]

Finds a single run index given a particular condition on parameters.

ONLY useful for a single run if v_full_copy` was set to ``True. Otherwise a TypeError is thrown.

Parameters:
  • name_list – A list of parameter names the predicate applies to, if you have only a single parameter name you can omit the list brackets.
  • predicate – A lambda predicate for filtering that evaluates to either True or False
Returns:

A generator yielding the matching single run indices

Example:

>>> predicate = lambda param1, param2: param1==4 and param2 in [1.0, 2.0]
>>> iterator = traj.f_find_idx(['groupA.param1', 'groupA.param2'], predicate)
>>> [x for x in iterator]
[0, 2, 17, 36]
f_get_config(fast_access=False, copy=True)[source]

Returns a dictionary containing the full config names as keys and the config parameters or the config parameter data items as values.

Parameters:
  • fast_access – Determines whether the parameter objects or their values are returned in the dictionary.
  • copy – Whether the original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all! Not Copying and fast access do not work at the same time! Raises ValueError if fast access is true and copy false.
Returns:

Dictionary containing the config data

Raises:

ValueError

f_get_derived_parameters(fast_access=False, copy=True)[source]
Returns a dictionary containing the full parameter names as keys and the parameters
or the parameter data items as values.
Parameters:
  • fast_access – Determines whether the parameter objects or their values are returned in the dictionary.
  • copy – Whether the original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all! Not Copying and fast access do not work at the same time! Raises ValueError if fast access is true and copy false.
Returns:

Dictionary containing the parameters.

Raises:

ValueError

f_get_explored_parameters(fast_access=False, copy=True)[source]
Returns a dictionary containing the full parameter names as keys and the parameters

or the parameter data items as values.

IMPORTANT: This dictionary always contains all explored parameters as keys. Even when they are not loaded, in this case the value is simply None. fast_access only works if all explored parameters are loaded.

Parameters:
  • fast_access – Determines whether the parameter objects or their values are returned in the dictionary.
  • copy – Whether the original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all! Not Copying and fast access do not work at the same time! Raises ValueError if fast access is true and copy false.
Returns:

Dictionary containing the parameters.

Raises:

ValueError

f_get_from_runs(name, include_default_run=True, use_indices=False, fast_access=False, with_links=True, shortcuts=True, max_depth=None, auto_load=False)[source]

Searches for all occurrences of name in each run.

Generates an ordered dictionary with the run names or indices as keys and found items as values.

Example:

>>> traj.f_get_from_runs(self, 'deep.universal_answer', use_indices=True, fast_access=True)
OrderedDict([(0, 42), (1, 42), (2, 'fortytwo), (3, 43)])
param name:String description of the item(s) to find. Cannot be full names but the part of the names that are below a run_XXXXXXXXX group.
param include_default_run:
 If results found under run_ALL should be accounted for every run or simply be ignored.
param use_indices:
 If True the keys of the resulting dictionary are the run indices (e.g. 0,1,2,3), otherwise the keys are run names (e.g. run_00000000, run_000000001)
param fast_access:
 Whether to return parameter or result instances or the values handled by these.
param with_links:
 If links should be considered
param shortcuts:
 If shortcuts are allowed and the trajectory can hop over nodes in the path.
param max_depth:
 Maximum depth (relative to start node) how search should progress in tree. None means no depth limit. Only relevant if shortcuts are allowed.
param auto_load:
 If data should be loaded from the storage service if it cannot be found in the current trajectory tree. Auto-loading will load group and leaf nodes currently not in memory and it will load data into empty leaves. Be aware that auto-loading does not work with shortcuts.
return:Ordered dictionary with run names or indices as keys and found items as values. Will only include runs where an item was actually found.

ATTENTION: This function is not available during a single run!

f_get_parameters(fast_access=False, copy=True)[source]
Returns a dictionary containing the full parameter names as keys and the parameters
or the parameter data items as values.
Parameters:
  • fast_access – Determines whether the parameter objects or their values are returned in the dictionary.
  • copy – Whether the original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all! Not Copying and fast access do not work at the same time! Raises ValueError if fast access is true and copy false.
Returns:

Dictionary containing the parameters.

Raises:

ValueError

f_get_results(fast_access=False, copy=True)[source]

Returns a dictionary containing the full result names as keys and the corresponding result objects or result data items as values.

Parameters:
  • fast_access – Determines whether the result objects or their values are returned in the dictionary. Works only for results if they contain a single item with the name of the result.
  • copy – Whether the original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all! Not Copying and fast access do not work at the same time! Raises ValueError if fast access is true and copy false.
Returns:

Dictionary containing the results.

Raises:

ValueError

f_get_run_information(name_or_idx=None, copy=True)[source]

Returns a dictionary containing information about a single run.

ONLY useful during a single run if v_full_copy` was set to ``True. Otherwise only the current run is available.

The information dictionaries have the following key, value pairings:

  • completed: Boolean, whether a run was completed

  • idx: Index of a run

  • timestamp: Timestamp of the run as a float

  • time: Formatted time string

  • finish_timestamp: Timestamp of the finishing of the run

  • runtime: Total runtime of the run in human readable format

  • name: Name of the run

  • parameter_summary:

    A string summary of the explored parameter settings for the particular run

  • short_environment_hexsha: The short version of the environment SHA-1 code

If no name or idx is given then a nested dictionary with keys as run names and info dictionaries as values is returned.

Parameters:
  • name_or_idx – str or int
  • copy – Whether you want the dictionary used by the trajectory or a copy. Note if you want the real thing, please do not modify it, i.e. popping or adding stuff. This could mess up your whole trajectory.
Returns:

A run information dictionary or a nested dictionary of information dictionaries with the run names as keys.

f_get_run_names(sort=True)[source]

Returns a list of run names.

ONLY useful for a single run during multiprocessing if v_full_copy` was set to ``True. Otherwise only the current run is available.

Parameters:sort – Whether to get them sorted, will only require O(N) [and not O(N*log N)] since we use (sort of) bucket sort.
f_get_wildcards()[source]

Returns a list of all defined wildcards

f_idx_to_run(name_or_idx)[source]

Converts an integer idx to the corresponding single run name and vice versa.

Note during a single run ONLY useful if v_full_copy was set to True.

Parameters:name_or_idx – Name of a single run or an integer index
Returns:The corresponding idx or name of the single run

Example usage:

>>> traj.f_idx_to_run(4)
'run_00000004'
>>> traj.f_idx_to_run('run_00000000')
0
f_is_completed(name_or_id=None)[source]

Whether or not a given run is completed.

If no run is specified it is checked whether all runs were completed.

param name_or_id:
 Nam or id of a run to check
return:True or False

ATTENTION: This function is not available during a single run!

f_is_empty()[source]
Whether no results nor parameters have been added yet to the trajectory
(ignores config).

ATTENTION: This function is not available during a single run!

f_is_wildcard(wildcard)[source]

Checks if a given wildcard is really a wildcard.

f_iter_runs(start=0, stop=None, step=1, yields='name')[source]

Makes the trajectory iterate over all runs.

param start:

Start index of run

param stop:

Stop index, leave None for length of trajectory

param step:

Stepsize

param yields:

What should be yielded: 'name' of run, idx of run or 'self' to simply return the trajectory container.

You can also pick 'copy' to get shallow copies (ie the tree is copied but no leave nodes except explored ones.) of your trajectory, might lead to some of overhead.

Note that after a full iteration, the trajectory is set back to normal.

Thus, the following code snippet

for run_name in traj.f_iter_runs():

     # Do some stuff here...

is equivalent to

for run_name in traj.f_get_run_names(sort=True):
    traj.f_set_crun(run_name)

    # Do some stuff here...

traj.f_set_crun(None)
return:Iterator over runs. The iterator itself will return the run names but modify the trajectory in each iteration and set it back do normal in the end.

ATTENTION: This function is not available during a single run!

f_load(name=None, index=None, as_new=False, load_parameters=2, load_derived_parameters=1, load_results=1, load_other_data=1, recursive=True, load_data=None, max_depth=None, force=False, dynamic_imports=None, with_run_information=True, with_meta_data=True, storage_service=None, **kwargs)[source]

Loads a trajectory via the storage service.

If you want to load individual results or parameters manually, you can take a look at f_load_items(). To only load subtrees check out f_load_child().

For f_load you can pass the following arguments:

param name:

Name of the trajectory to be loaded. If no name or index is specified the current name of the trajectory is used.

param index:

If you don’t specify a name you can specify an integer index instead. The corresponding trajectory in the hdf5 file at the index position is loaded (counting starts with 0). Negative indices are also allowed counting in reverse order. For instance, -1 refers to the last trajectory in the file, -2 to the second last, and so on.

param as_new:

Whether you want to rerun the experiments. So the trajectory is loaded only with parameters. The current trajectory name is kept in this case, which should be different from the trajectory name specified in the input parameter name. If you load as_new=True all parameters are unlocked. If you load as_new=False the current trajectory is replaced by the one on disk, i.e. name, timestamp, formatted time etc. are all taken from disk.

param load_parameters:
 

How parameters and config items are loaded

param load_derived_parameters:
 

How derived parameters are loaded

param load_results:
 

How results are loaded

You can specify how to load the parameters, derived parameters and results as follows:

Note that in all cases except pypet.pypetconstants.LOAD_NOTHING, annotations will be reloaded if the corresponding instance is created or the annotations of an existing instance were emptied before.

param recursive:
 

If data should be loaded recursively. If set to None, this is equivalent to set all data loading to :const:`pypet.pypetconstants.LOAD_NOTHING.

param load_data:
 

As the above, per default set to None. If not None the setting of load_data will overwrite the settings of load_parameters, load_derived_parameters, load_results, and load_other_data. This is more or less or shortcut if all types should be loaded the same.

param max_depth:
 

Maximum depth to load nodes (inclusive).

param force:

pypet will refuse to load trajectories that have been created using pypet with a different version number or a different python version. To force the load of a trajectory from a previous version simply set force = True. Note that it is not checked if other versions of packages differ from previous experiments, i.e. numpy, scipy, etc. But you can check for this manually. The versions of other packages can be found under 'config.environment.name_of_environment.versions.package_name'.

param dynamic_imports:
 

If you’ve written a custom parameter that needs to be loaded dynamically during runtime, this needs to be specified here as a list of classes or strings naming classes and there module paths. For example: dynamic_imports = [‘pypet.parameter.PickleParameter’,MyCustomParameter]

If you only have a single class to import, you do not need the list brackets: dynamic_imports = ‘pypet.parameter.PickleParameter’

The classes passed here are added for good and will be kept by the trajectory. Please add your dynamically imported classes only once.

param with_run_information:
 

If information about the individual runs should be loaded. If you have many runs, like 1,000,000 or more you can spare time by setting with_run_information=False. Note that f_get_run_information and f_idx_to_run do not work in such a case. Moreover, setting v_idx does not work either. If you load the trajectory without this information, be careful, this is not recommended.

param wiht_meta_data:
 

If meta data should be loaded.

param storage_service:
 

Pass a storage service used by the trajectory. Alternatively pass a constructor and other **kwargs are passed onto the constructor. Leave None in combination with using no other kwargs, if you don’t want to change the service the trajectory is currently using.

param kwargs:

Other arguments passed to the storage service constructor. Don’t pass any other kwargs and storage_service=None, if you don’t want to change the current service.

ATTENTION: This function is not available during a single run!

f_load_item(item, *args, **kwargs)[source]

Loads a single item, see also f_load_items()

f_load_items(iterator, *args, **kwargs)[source]

Loads parameters and results specified in iterator.

You can directly list the Parameter objects or just their names.

If names are given the ~pypet.naturalnaming.NNGroupNode.f_get method is applied to find the parameters or results in the trajectory. Accordingly, the parameters and results you want to load must already exist in your trajectory (in RAM), probably they are just empty skeletons waiting desperately to handle data. If they do not exist in RAM yet, but have been stored to disk before, you can call f_load_skeleton() in order to bring your trajectory tree skeleton up to date. In case of a single run you can use the f_load_child() method to recursively load a subtree without any data. Then you can load the data of individual results or parameters one by one.

If want to load the whole trajectory at once or ALL results and parameters that are still empty take a look at f_load(). As mentioned before, to load subtrees of your trajectory you might want to check out f_load_child().

To load a list of parameters or results with f_load_items you can pass the following arguments:

Parameters:
  • iterator – A list with parameters or results to be loaded.
  • only_empties – Optional keyword argument (boolean), if True only empty parameters or results are passed to the storage service to get loaded. Non-empty parameters or results found in iterator are simply ignored.
  • args – Additional arguments directly passed to the storage service
  • kwargs

    Additional keyword arguments directly passed to the storage service (except the kwarg only_empties)

    If you use the standard hdf5 storage service, you can pass the following additional keyword arguments:

    param load_only:
     If you load a result, you can partially load it and ignore the rest of data items. Just specify the name of the data you want to load. You can also provide a list, for example load_only=’spikes’, load_only=[‘spikes’,’membrane_potential’].

    Be aware that you need to specify the names of parts as they were stored to HDF5. Depending on how your leaf construction works, this may differ from the names the data might have in your leaf in the trajectory container.

    A warning is issued if data specified in load_only cannot be found in the instances specified in iterator.

    param load_except:
     Analogous to the above, but everything is loaded except names or parts specified in load_except. You cannot use load_only and load_except at the same time. If you do a ValueError is thrown.

    A warning is issued if names listed in load_except are not part of the items to load.

f_load_skeleton()[source]

Loads the full skeleton from the storage service.

This needs to be done after a successful exploration in order to update the trajectory tree with all results and derived parameters from the individual single runs. This will only add empty results and derived parameters (i.e. the skeleton) and load annotations.

ATTENTION: This function is not available during a single run!

f_lock_derived_parameters()[source]

Locks all non-empty derived parameters

ATTENTION: This function is not available during a single run!

f_lock_parameters()[source]

Locks all non-empty parameters

ATTENTION: This function is not available during a single run!

f_merge(other_trajectory, trial_parameter=None, remove_duplicates=False, ignore_data=(), backup=True, move_data=False, delete_other_trajectory=False, keep_info=True, keep_other_trajectory_info=True, merge_config=True, consecutive_merge=False, slow_merge=False)[source]

Merges another trajectory into the current trajectory.

Both trajectories must live in the same space. This means both need to have the same parameters with similar types of values.

Note that links are also merged. There are exceptions: Links found under a generic run group called run_ALL or links linking to a node under such a group are NOT merged and simply skipped, because there is no straightforward way to resolve the link.

param other_trajectory:
 

Other trajectory instance to merge into the current one.

param trial_parameter:
 

If you have a particular parameter that specifies only the trial number, i.e. an integer parameter running form 0 to T1 and 0 to T2, the parameter is modified such that after merging it will cover the range 0 to T1+T2+1. T1 is the number of individual trials in the current trajectory and T2 number of trials in the other trajectory.

param remove_duplicates:
 

Whether you want to remove duplicate parameter points. Requires N1 * N2 (quadratic complexity in single runs). A ValueError is raised if no runs would be merged.

param ignore_data:
 

List of full names of data that should be ignored and not merged.

param backup:

If True, backs up both trajectories into files chosen automatically by the storage services. If you want to customize your backup use the f_backup function instead.

param move_data:
 

Tells the storage service to move data from one trajectory to the other instead of copying it.

If you use the HDF5 storage service and both trajectories are stored in the same file, merging is performed fast directly within the file. You can choose if you want to copy nodes (‘move_nodes=False`) from the other trajectory to the current one, or if you want to move them. Accordingly, the stored data is no longer accessible in the other trajectory.

param delete_other_trajectory:
 

If you want to delete the other trajectory after merging.

param keep_info:
 

If True, information about the merge is added to the trajectory config tree under config.merge.

param merge_config:
 

Whether or not to merge all config parameters under .config.git, .config.environment, and .config.merge of the other trajectory into the current one.

param keep_other_trajectory_info:
 

Whether to keep information like length, name, etc. of the other trajectory in case you want to keep all the information. Setting of keep_other_trajectory_info is irrelevant in case keep_info=False.

param consecutive_merge:
 

Can be set to True if you are about to merge several trajectories into the current one within a loop to avoid quadratic complexity. But remember to store your trajectory manually after all merges. Also make sure that all parameters and derived parameters are available in your current trajectory and load them before the consecutive merging. Also avoid specifying a trial_parameter and set backup=False to avoid quadratic complexity in case of consecutive merges.

param slow_merge:
 

Enforces a slow merging. This means all data is loaded one after the other to memory and stored to disk. Otherwise it is tried to directly copy the data from one file into another without explicitly loading the data.

If you cannot directly merge trajectories within one HDF5 file, a slow merging process is used. Results are loaded, stored, and emptied again one after the other. Might take some time!

Annotations of parameters and derived parameters under .derived_parameters.trajectory are NOT merged. If you wish to extract the annotations of these parameters you have to do that manually before merging. Note that annotations of results and derived parameters of single runs are copied, so you don’t have to worry about these.

ATTENTION: This function is not available during a single run!

f_merge_many(other_trajectories, ignore_data=(), move_data=False, delete_other_trajectory=False, keep_info=True, keep_other_trajectory_info=True, merge_config=True, backup=True)[source]

Can be used to merge several other_trajectories into your current one.

IMPORTANT backup=True only backs up the current trajectory not any of the other_trajectories. If you need a backup of these, do it manually.

Parameters as for f_merge().

ATTENTION: This function is not available during a single run!

f_migrate(new_name=None, in_store=False, new_storage_service=None, **kwargs)[source]

Can be called to rename and relocate the trajectory.

param new_name:New name of the trajectory, None if you do not want to change the name.
param in_store:Set this to True if the trajectory has been stored with the new name at the new file before and you just want to “switch back” to the location. If you migrate to a store used before and you do not set in_store=True, the storage service will throw a RuntimeError in case you store the Trajectory because it will assume that you try to store a new trajectory that accidentally has the very same name as another trajectory. If set to True and trajectory is not found in the file, the trajectory is simply stored to the file.
param new_storage_service:
 New service where you want to migrate to. Leave none if you want to keep the olde one.
param kwargs:Additional keyword arguments passed to the service. For instance, to change the file of the trajectory use filename='my_new_file.hdf5.

ATTENTION: This function is not available during a single run!

f_preset_config(config_name, *args, **kwargs)[source]

Similar to func:~pypet.trajectory.Trajectory.f_preset_parameter

ATTENTION: This function is not available during a single run!

f_preset_parameter(param_name, *args, **kwargs)[source]

Presets parameter value before a parameter is added.

Can be called before parameters are added to the Trajectory in order to change the values that are stored into the parameter on creation.

After creation of a parameter, the instance of the parameter is called with param.f_set(*args,**kwargs) with *args, and **kwargs provided by the user with f_preset_parameter.

Before an experiment is carried out it is checked if all parameters that were marked were also preset.

param param_name:
 The full name (!) of the parameter that is to be changed after its creation.
param args:Arguments that will be used for changing the parameter’s data
param kwargs:Keyword arguments that will be used for changing the parameter’s data

Example:

>>> traj.f_preset_parameter('groupA.param1', data=44)
>>> traj.f_add_parameter('groupA.param1', data=11)
>>> traj.parameters.groupA.param1
44

ATTENTION: This function is not available during a single run!

f_remove(recursive=True, predicate=None)[source]

Recursively removes all children of the trajectory

Parameters:
  • recursive – Only here for consistency with signature of parent method. Cannot be set to False because the trajectory root node cannot be removed.
  • predicate – Predicate which can evaluate for each node to True in order to remove the node or False if the node should be kept. Leave None if you want to remove all nodes.
f_remove_item(item, recursive=False)[source]

Removes a single item, see f_remove_items()

f_remove_items(iterator, recursive=False)[source]

Removes parameters, results or groups from the trajectory.

This function ONLY removes items from your current trajectory and does not delete data stored to disk. If you want to delete data from disk, take a look at f_delete_items().

This will also remove all links if items are linked.

Parameters:
  • iterator – A sequence of items you want to remove. Either the instances themselves or strings with the names of the items.
  • recursive – In case you want to remove group nodes, if the children should be removed, too.
f_restore_default()[source]
Restores the default value in all explored parameters and sets the
v_idx property back to -1 and v_crun to None.

ATTENTION: This function is not available during a single run!

f_set_crun(name_or_idx)[source]

Can make the trajectory behave as during a particular single run.

It allows easier data analysis.

Has the following effects:
  • v_idx and v_crun are set to the appropriate index and run name
  • All explored parameters are set to the corresponding value in the exploration ranges, i.e. when you call f_get() (or fast access) on them you will get in return the value at the corresponding v_idx position in the exploration range.
  • If you perform a search in the trajectory tree, the trajectory will only search the run subtree under results and derived_parameters with the corresponding index. For instance, if you use f_set_crun(‘run_00000007’) or f_set_crun(7) and search for traj.results.z this will search for z only in the subtree traj.results.run_00000007. Yet, you can still explicitly name other subtrees, i.e. traj.results.run_00000004.z will still work.

ATTENTION: This function is not available during a single run!

f_set_properties(**kwargs)[source]

Sets properties like v_fast_access.

For example: traj.f_set_properties(v_fast_access=True, v_auto_load=False)

f_shrink(force=False)[source]
Shrinks the trajectory and removes all exploration ranges from the parameters.

Only possible if the trajectory has not been stored to disk before or was loaded as new.

param force:Usually you cannot shrink the trajectory if it has been stored to disk, because there’s no guarantee that it is actually shrunk if there still exist explored parameters on disk. In case you are certain that you did not store explored parameters to disk set or you deleted all of them from disk set force=True.
raises:TypeError if the trajectory was stored before.

ATTENTION: This function is not available during a single run!

f_start_run(run_name_or_idx=None, turn_into_run=True)[source]

Can be used to manually allow running of an experiment without using an environment.

Parameters:
  • run_name_or_idx – Can manually set a trajectory to a particular run. If None the current run the trajectory is set to is used.
  • turn_into_run – Turns the trajectory into a run, i.e. reduces functionality but makes storing more efficient.
f_store(only_init=False, store_data=2, max_depth=None)[source]

Stores the trajectory to disk and recursively all data in the tree.

Parameters:
  • only_init – If you just want to initialise the store. If yes, only meta information about the trajectory is stored and none of the groups/leaves within the trajectory. Alternatively, you can pass recursive=False.
  • store_data

    Only considered if only_init=False. Choose of the following:

    • pypet.pypetconstants.STORE_NOTHING: (0)
      Nothing is store.
    • pypet.pypetconstants.STORE_DATA_SKIPPING: (1)
      Speedy version of normal STORE_DATA will entirely skip groups (but not their children) and leaves if they have been stored before. No new data is added in this case.
    • pypet.pypetconstants.STORE_DATA: (2)
      Stores every group and leave node. If they contain data that is not yet stored to disk it is added.
    • pypet.pypetconstants.OVERWRITE_DATA: (3)
      Stores all groups and leave nodes and will delete all data on disk and overwrite it with the current data in RAM.

      NOT RECOMMENDED! Overwriting data on disk fragments the HDF5 file and yields badly compressed large files. Better stick to the concept write once and read many!

If you use the HDF5 Storage Service usually (STORE_DATA (2)) only novel data is stored to disk. If you have results that have been stored to disk before only new data items are added and already present data is NOT overwritten.

Overwriting (OVERWRITE_DATA (3)) existing data with the HDF5 storage service is not recommended due to fragmentation of the HDF5 file. Better stick to the concept write once, but read often.

If you want to store individual parameters or results, you might want to take a look at f_store_items(). To store whole subtrees of your trajectory check out f_store_child(). Note both functions require that your trajectory was stored to disk with f_store at least once before.

ATTENTION: Calling f_store during a single run the behavior is different.

To avoid re-storing the full trajectory in every single run, which is redundant, only sub-trees of the trajectory are really stored.

The storage serivce looks for new data that is added below groups called run_XXXXXXXXXX and stores it where XXXXXXXXX is the index of this run. The only_init parameter is ignored in this case. You can avoid this behavior by using the argument from below.

Parameters:max_depth – Maximum depth to store tree (inclusive). During single runs max_depth is also counted from root.
f_store_item(item, *args, **kwargs)[source]

Stores a single item, see also f_store_items().

f_store_items(iterator, *args, **kwargs)[source]

Stores individual items to disk.

This function is useful if you calculated very large results (or large derived parameters) during runtime and you want to write these to disk immediately and empty them afterwards to free some memory.

Instead of storing individual parameters or results you can also store whole subtrees with f_store_child().

You can pass the following arguments to f_store_items:

Parameters:
  • iterator – An iterable containing the parameters or results to store, either their names or the instances. You can also pass group instances or names here to store the annotations of the groups.
  • non_empties – Optional keyword argument (boolean), if True will only store the subset of provided items that are not empty. Empty parameters or results found in iterator are simply ignored.
  • args – Additional arguments passed to the storage service
  • kwargs

    If you use the standard hdf5 storage service, you can pass the following additional keyword argument:

    param overwrite:
     List names of parts of your item that should be erased and overwritten by the new data in your leaf. You can also set overwrite=True to overwrite all parts.

    For instance:

    >>> traj.f_add_result('mygroup.myresult', partA=42, partB=44, partC=46)
    >>> traj.f_store()
    >>> traj.mygroup.myresult.partA = 333
    >>> traj.mygroup.myresult.partB = 'I am going to change to a string'
    >>> traj.f_store_item('mygroup.myresult', overwrite=['partA', 'partB'])
    

    Will store ‘mygroup.myresult’ to disk again and overwrite the parts ‘partA’ and ‘partB’ with the new values 333 and ‘I am going to change to a string’. The data stored as partC is not changed.

    Be aware that you need to specify the names of parts as they were stored to HDF5. Depending on how your leaf construction works, this may differ from the names the data might have in your leaf in the trajectory container.

    Note that massive overwriting will fragment and blow up your HDF5 file. Try to avoid changing data on disk whenever you can.

Raises:

TypeError:

If the (parent) trajectory has never been stored to disk. In this case use pypet.trajectory.f_store() first.

ValueError: If no item could be found to be stored.

Note if you use the standard hdf5 storage service, there are no additional arguments or keyword arguments to pass!

f_to_dict(fast_access=False, short_names=False, nested=False, copy=True, with_links=True)[source]

Returns a dictionary with pairings of (full) names as keys and instances/values.

Parameters:
  • fast_access – If True, parameter values are returned instead of the instances. Works also for results if they contain a single item with the name of the result.
  • short_names – If true, keys are not full names but only the names. Raises a ValueError if the names are not unique.
  • nested – If true, a nested dictionary is returned.
  • copy – If fast_access=False and short_names=False you can access the original data dictionary if you set copy=False. If you do that, please do not modify anything! Raises ValueError if copy=False and fast_access=True or short_names=True.
  • with_links – If links should be ignored
Returns:

dictionary

Raises:

ValueError

f_wildcard(wildcard='$', run_idx=None)[source]

#TODO

v_auto_load

Whether the trajectory should attempt to load data on the fly.

v_auto_run_prepend

If during run the runs.run_XXXXXXXX should be prepended if it is missing.

Is not considered for f_add_leaf and f_add_group which never prepend.

v_comment

Should be a nice descriptive comment

v_crun

Run name if you want to access the trajectory as a single run.

You can turn the trajectory to behave as during a single run if you set v_crun to a particular run name. Note that only string values are appropriate here, not indices. Check the v_idx property if you want to provide an index.

Alternatively instead of directly setting v_crun you can call f_set_crun:().

Set to None to make the trajectory to turn everything back to default.

v_crun_

” Similar to v_crun but returns 'run_ALL' if v_crun is None.

v_environment_hexsha

If the trajectory is used with an environment this returns the SHA-1 code of the environment.

v_environment_name

If the trajectory is used with an environment this returns the name of the environment.

v_fast_access

Whether parameter instances (False) or their values (True) are returned via natural naming.

Works also for results if they contain a single item with the name of the result.

Default is True.

v_full_copy

Whether trajectory is copied fully during pickling or only the current parameter space point.

Note if the trajectory is copied as a whole, also during a single run you can access the full parameter space.

Changing v_full_copy will also change v_full_copy of all explored parameters!

v_idx

Index if you want to access the trajectory as during a single run.

You can turn the trajectory to behave as if during the execution of your runs if you set v_idx to a particular index. Note that only integer values are appropriate here, not names of runs.

Alternatively instead of directly setting v_idx you can call f_set_crun:().

Set to -1 to make the trajectory turn everything back to default.

v_is_run

True mak if trajectory is used during a single run initiated by an environment.

Accordingly, the functionality of the trajectory is reduced.

v_iter_recursive

Whether using __iter__ should iterate only immediate children or recursively all nodes.

v_max_depth

The maximum depth the tree should be searched if shortcuts are allowed.

Set to None if there should be no depth limit.

v_no_clobber

If f_add_leaf should not throw an error in case something is added that is already part of the Trajectory. If True no error is thrown and the new data is ignored.

v_python

The version of python as a string that was used to create the trajectory

v_shortcuts

Whether shortcuts are allowed if accessing data via natural naming or squared bracket indexing.

v_standard_leaf

The standard constructor used if you add a generic leaf.

The constructor is only used if you do not add items under the usual four subtrees (parameters, derived_parameters, config, results).

v_standard_parameter

The standard parameter used for parameter creation

v_standard_result

The standard result class used for result creation

v_storage_service

The service that can store the trajectory to disk or wherever.

Default is None or if a filename was provided on construction the HDF5StorageService.

v_time

Formatted time string of the time the trajectory or run was created.

v_timestamp

Float timestamp of creation time

v_version

The version of pypet that was used to create the trajectory

Whether links should be considered in case using natural naming or squared bracket indexing

pypet.trajectory.load_trajectory(name=None, index=None, as_new=False, load_parameters=2, load_derived_parameters=1, load_results=1, load_other_data=1, recursive=True, load_data=None, max_depth=None, force=False, dynamic_imports=None, new_name='my_trajectory', add_time=True, wildcard_functions=None, with_run_information=True, storage_service=<class 'pypet.storageservice.HDF5StorageService'>, **kwargs)[source]

Helper function that creates a novel trajectory and loads it from disk.

For the parameters see f_load().

new_name and add_time are only used in case as_new is True. Accordingly, they determine the new name of trajectory.

NNGroupNode

class pypet.naturalnaming.NNGroupNode(full_name='', trajectory=None, comment='')[source]

A group node hanging somewhere under the trajectory or single run root node.

You can add other groups or parameters/results to it.

f_add_group(*args, **kwargs)[source]

Adds an empty generic group under the current node.

You can add to a generic group anywhere you want. So you are free to build your parameter tree with any structure. You do not necessarily have to follow the four subtrees config, parameters, derived_parameters, results.

If you are operating within these subtrees this simply calls the corresponding adding function.

Be aware that if you are within a single run and you add items not below a group run_XXXXXXXX that you have to manually save the items. Otherwise they will be lost after the single run is completed.

f_add_leaf(*args, **kwargs)[source]

Adds an empty generic leaf under the current node.

You can add to a generic leaves anywhere you want. So you are free to build your trajectory tree with any structure. You do not necessarily have to follow the four subtrees config, parameters, derived_parameters, results.

If you are operating within these subtrees this simply calls the corresponding adding function.

Be aware that if you are within a single run and you add items not below a group run_XXXXXXXX that you have to manually save the items. Otherwise they will be lost after the single run is completed.

Adds a link to an existing node.

Can be called as node.f_add_link(other_node) this will add a link the other_node with the link name as the name of the node.

Or can be called as node.f_add_link(name, other_node) to add a link to the other_node and the given name of the link.

In contrast to addition of groups and leaves, colon separated names are not allowed, i.e. node.f_add_link('mygroup.mylink', other_node) does not work.

f_ann_to_str()

Returns annotations as string

Equivalent to v_annotations.f_ann_to_str()

f_children()[source]

Returns the number of children of the group

f_contains(item, with_links=True, shortcuts=False, max_depth=None)[source]

Checks if the node contains a specific parameter or result.

It is checked if the item can be found via the f_get() method.

Parameters:
  • item

    Parameter/Result name or instance.

    If a parameter or result instance is supplied it is also checked if the provided item and the found item are exactly the same instance, i.e. id(item)==id(found_item).

  • with_links – If links are considered.
  • shortcuts – Shortcuts is False the name you supply must be found in the tree WITHOUT hopping over nodes in between. If shortcuts=False and you supply a non colon separated (short) name, than the name must be found in the immediate children of your current node. Otherwise searching via shortcuts is allowed.
  • max_depth – If shortcuts is True than the maximum search depth can be specified. None means no limit.
Returns:

True or False

f_debug()[source]

Creates a dummy object containing the whole tree to make unfolding easier.

This method is only useful for debugging purposes. If you use an IDE and want to unfold the trajectory tree, you always need to open the private attribute _children. Use to this function to create a new object that contains the tree structure in its attributes.

Manipulating the returned object does not change the original tree!

f_dir_data()[source]

Returns a list of all children names

f_get(name, fast_access=False, with_links=True, shortcuts=True, max_depth=None, auto_load=False)[source]

Searches and returns an item (parameter/result/group node) with the given name.

Parameters:
  • name – Name of the item (full name or parts of the full name)
  • fast_access – Whether fast access should be applied.
  • with_links – If links are considered. Cannot be set to False if auto_load is True.
  • shortcuts – If shortcuts are allowed and the trajectory can hop over nodes in the path.
  • max_depth – Maximum depth relative to starting node (inclusive). None means no depth limit.
  • auto_load – If data should be loaded from the storage service if it cannot be found in the current trajectory tree. Auto-loading will load group and leaf nodes currently not in memory and it will load data into empty leaves. Be aware that auto-loading does not work with shortcuts.
Returns:

The found instance (result/parameter/group node) or if fast access is True and you found a parameter or result that supports fast access, the contained value is returned.

Raises:

AttributeError: If no node with the given name can be found

NotUniqueNodeError

In case of forward search if more than one candidate node is found within a particular depth of the tree. In case of backwards search if more than one candidate is found regardless of the depth.

DataNotInStorageError:

In case auto-loading fails

Any exception raised by the StorageService in case auto-loading is enabled

f_get_all(name, max_depth=None, shortcuts=True)[source]

Searches for all occurrences of name under node.

Links are NOT considered since nodes are searched bottom up in the tree.

Parameters:
  • node – Start node
  • name – Name of what to look for, can be separated by colons, i.e. 'mygroupA.mygroupB.myparam'.
  • max_depth – Maximum search depth relative to start node. None for no limit.
  • shortcuts – If shortcuts are allowed, otherwise the stated name defines a consecutive name.For instance. 'mygroupA.mygroupB.myparam' would also find mygroupA.mygroupX.mygroupB.mygroupY.myparam if shortcuts are allowed, otherwise not.
Returns:

List of nodes that match the name, empty list if nothing was found.

f_get_annotations(*args)

Returns annotations

Equivalent to v_annotations.f_get(*args)

f_get_children(copy=True)[source]

Returns a children dictionary.

Parameters:copy – Whether the group’s original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all!
Returns:Dictionary of nodes
f_get_class_name()

Returns the class name of the parameter or result or group.

Equivalent to obj.__class__.__name__

f_get_default(name, default=None, fast_access=True, with_links=True, shortcuts=True, max_depth=None, auto_load=False)[source]

Similar to f_get, but returns the default value if name is not found in the trajectory.

This function uses the f_get method and will return the default value in case f_get raises an AttributeError or a DataNotInStorageError. Other errors are not handled.

In contrast to f_get, fast access is True by default.

f_get_groups(copy=True)[source]

Returns a dictionary of groups hanging immediately below this group.

Parameters:copy – Whether the group’s original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all!
Returns:Dictionary of nodes
f_get_leaves(copy=True)[source]

Returns a dictionary of all leaves hanging immediately below this group.

Parameters:copy – Whether the group’s original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all!
Returns:Dictionary of nodes

Returns a link dictionary.

Parameters:copy – Whether the group’s original dictionary or a shallow copy is returned. If you want the real dictionary please do not modify it at all!
Returns:Dictionary of nodes
f_get_parent()[source]

Returns the parent of the node.

Raises a TypeError if current node is root.

f_groups()[source]

Returns the number of immediate groups of the group

f_has_children()[source]

Checks if node has children or not

f_has_groups()[source]

Checks if node has groups or not

f_has_leaves()[source]

Checks if node has leaves or not

Checks if node has children or not

f_iter_leaves(with_links=True)[source]

Iterates (recursively) over all leaves hanging below the current group.

Parameters:with_links – If links should be ignored, leaves hanging below linked nodes are not listed.
Returns:Iterator over all leaf nodes
f_iter_nodes(recursive=True, with_links=True, max_depth=None, predicate=None)[source]

Iterates recursively (default) over nodes hanging below this group.

Parameters:
  • recursive – Whether to iterate the whole sub tree or only immediate children.
  • with_links – If links should be considered
  • max_depth – Maximum depth in search tree relative to start node (inclusive)
  • predicate

    A predicate function that is applied for each node and only returns the node if it evaluates to True. If False and you iterate recursively also the children are spared.

    Leave to None if you don’t want to filter and simply iterate over all nodes.

    For example, to iterate only over groups you could use:

    >>> traj.f_iter_nodes(recursive=True, predicate=lambda x: x.v_is_group)
    

    To blind out all runs except for a particular set, you can simply pass a tuple of run indices with -1 referring to the run_ALL node.

    For instance

    >>> traj.f_iter_nodes(recursive=True, predicate=(0,3,-1))
    

    Will blind out all nodes hanging below a group named run_XXXXXXXXX (including the group) except run_00000000, run_00000003, and run_ALL.

Returns:

Iterator over nodes

f_leaves()[source]

Returns the number of immediate leaves of the group

Returns the number of links of the group

f_load(recursive=True, load_data=2, max_depth=None)[source]

Loads a group from disk.

Parameters:
  • recursive – Default is True. Whether recursively all nodes below the current node should be loaded, too. Note that links are never evaluated recursively. Only the linked node will be loaded if it does not exist in the tree, yet. Any nodes or links of this linked node are not loaded.
  • load_data – Flag how to load the data. For how to choose ‘load_data’ see Loading.
  • max_depth – In case recursive is True, you can specify the maximum depth to load load data relative from current node.
Returns:

The node itself.

f_load_child(name, recursive=False, load_data=2, max_depth=None)[source]

Loads a child or recursively a subtree from disk.

Parameters:
  • name – Name of child to load. If grouped (‘groupA.groupB.childC’) the path along the way to last node in the chain is loaded. Shortcuts are NOT allowed!
  • recursive – Whether recursively all nodes below the last child should be loaded, too. Note that links are never evaluated recursively. Only the linked node will be loaded if it does not exist in the tree, yet. Any nodes or links of this linked node are not loaded.
  • load_data – Flag how to load the data. For how to choose ‘load_data’ see Loading.
  • max_depth – In case recursive is True, you can specify the maximum depth to load load data relative from current node. Leave None if you don’t want to limit the depth.
Returns:

The loaded child, in case of grouping (‘groupA.groupB.childC’) the last node (here ‘childC’) is returned.

f_remove(recursive=True, predicate=None)[source]

Recursively removes the group and all it’s children.

Parameters:
  • recursive – If removal should be applied recursively. If not, node can only be removed if it has no children.
  • predicate – In case of recursive removal, you can selectively remove nodes in the tree. Predicate which can evaluate for each node to True in order to remove the node or False if the node should be kept. Leave None if you want to remove all nodes.
f_remove_child(name, recursive=False, predicate=None)[source]

Removes a child of the group.

Note that groups and leaves are only removed from the current trajectory in RAM. If the trajectory is stored to disk, this data is not affected. Thus, removing children can be only be used to free RAM memory!

If you want to free memory on disk via your storage service, use f_delete_items() of your trajectory.

Parameters:
  • name – Name of child, naming by grouping is NOT allowed (‘groupA.groupB.childC’), child must be direct successor of current node.
  • recursive – Must be true if child is a group that has children. Will remove the whole subtree in this case. Otherwise a Type Error is thrown.
  • predicate – Predicate which can evaluate for each node to True in order to remove the node or False if the node should be kept. Leave None if you want to remove all nodes.
Raises:

TypeError if recursive is false but there are children below the node.

ValueError if child does not exist.

Removes a link from from the current group node with a given name.

Does not delete the link from the hard drive. If you want to do this, checkout f_delete_links()

f_set_annotations(*args, **kwargs)

Sets annotations

Equivalent to calling v_annotations.f_set(*args,**kwargs)

f_store(recursive=True, store_data=2, max_depth=None)[source]

Stores a group node to disk

Parameters:
  • recursive – Whether recursively all children should be stored too. Default is True.
  • store_data – For how to choose ‘store_data’ see Storing.
  • max_depth – In case recursive is True, you can specify the maximum depth to store data relative from current node. Leave None if you don’t want to limit the depth.
f_store_child(name, recursive=False, store_data=2, max_depth=None)[source]

Stores a child or recursively a subtree to disk.

Parameters:
  • name – Name of child to store. If grouped (‘groupA.groupB.childC’) the path along the way to last node in the chain is stored. Shortcuts are NOT allowed!
  • recursive – Whether recursively all children’s children should be stored too.
  • store_data – For how to choose ‘store_data’ see Storing.
  • max_depth – In case recursive is True, you can specify the maximum depth to store data relative from current node. Leave None if you don’t want to limit the depth.
Raises:

ValueError if the child does not exist.

f_to_dict(fast_access=False, short_names=False, nested=False, with_links=True)[source]

Returns a dictionary with pairings of (full) names as keys and instances as values.

This will iteratively traverse the tree and add all nodes below this group to the dictionary.

Parameters:
  • fast_access – If True parameter or result values are returned instead of the instances.
  • short_names – If true keys are not full names but only the names. Raises a ValueError if the names are not unique.
  • nested – If dictionary should be nested
  • with_links – If links should be considered
Returns:

dictionary

Raises:

ValueError

func

Alternative naming, you can use node.func.name instead of node.f_func

kids

Alternative naming, you can use node.kids.name instead of node.name for easier tab completion.

v_annotations

Annotation feature of a trajectory node.

Store some short additional information about your nodes here. If you use the standard HDF5 storage service, they will be stored as hdf5 node attributes.

v_branch

The name of the branch/subtree, i.e. the first node below the root.

The empty string in case of root itself.

v_comment

Should be a nice descriptive comment

v_depth

Depth of the node in the trajectory tree.

v_full_name

The full name, relative to the root node.

The full name of a trajectory or single run is the empty string since it is root.

v_is_group

Whether node is a group or not (i.e. it is a leaf node)

v_is_leaf

Whether node is a leaf or not (i.e. it is a group node)

v_is_root

Whether the group is root (True for the trajectory and a single run object)

v_location

Location relative to the root node.

The location of a trajectory or single run is the empty string since it is root.

v_name

Name of the node

v_root

Link to the root of the tree, i.e. the trajectory

v_run_branch

If this node is hanging below a branch named run_XXXXXXXXX.

The branch name is either the name of a single run (e.g. ‘run_00000009’) or ‘trajectory’.

v_stored

Whether or not this tree node has been stored to disk before.

vars

Alternative naming, you can use node.vars.name instead of node.v_name

ParameterGroup

class pypet.naturalnaming.ParameterGroup(full_name='', trajectory=None, comment='')[source]

Group node in your trajectory, hanging below traj.parameters.

You can add other groups or parameters to it.

f_add_parameter(*args, **kwargs)[source]

Adds a parameter under the current node.

There are two ways to add a new parameter either by adding a parameter instance:

>>> new_parameter = Parameter('group1.group2.myparam', data=42, comment='Example!')
>>> traj.f_add_parameter(new_parameter)

Or by passing the values directly to the function, with the name being the first (non-keyword!) argument:

>>> traj.f_add_parameter('group1.group2.myparam', 42, comment='Example!')

If you want to create a different parameter than the standard parameter, you can give the constructor as the first (non-keyword!) argument followed by the name (non-keyword!):

>>> traj.f_add_parameter(PickleParameter,'group1.group2.myparam', data=42, comment='Example!')

The full name of the current node is added as a prefix to the given parameter name. If the current node is the trajectory the prefix ‘parameters’ is added to the name.

Note, all non-keyword and keyword parameters apart from the optional constructor are passed on as is to the constructor.

Moreover, you always should specify a default data value of a parameter, even if you want to explore it later.

f_add_parameter_group(*args, **kwargs)[source]

Adds an empty parameter group under the current node.

Can be called with f_add_parameter_group('MyName', 'this is an informative comment') or f_add_parameter_group(name='MyName', comment='This is an informative comment') or with a given new group instance: f_add_parameter_group(ParameterGroup('MyName', comment='This is a comment')).

Adds the full name of the current node as prefix to the name of the group. If current node is the trajectory (root), the prefix ‘parameters’ is added to the full name.

The name can also contain subgroups separated via colons, for example: name=subgroup1.subgroup2.subgroup3. These other parent groups will be automatically created.

f_apar(*args, **kwargs)

Adds a parameter under the current node.

There are two ways to add a new parameter either by adding a parameter instance:

>>> new_parameter = Parameter('group1.group2.myparam', data=42, comment='Example!')
>>> traj.f_add_parameter(new_parameter)

Or by passing the values directly to the function, with the name being the first (non-keyword!) argument:

>>> traj.f_add_parameter('group1.group2.myparam', 42, comment='Example!')

If you want to create a different parameter than the standard parameter, you can give the constructor as the first (non-keyword!) argument followed by the name (non-keyword!):

>>> traj.f_add_parameter(PickleParameter,'group1.group2.myparam', data=42, comment='Example!')

The full name of the current node is added as a prefix to the given parameter name. If the current node is the trajectory the prefix ‘parameters’ is added to the name.

Note, all non-keyword and keyword parameters apart from the optional constructor are passed on as is to the constructor.

Moreover, you always should specify a default data value of a parameter, even if you want to explore it later.

ConfigGroup

class pypet.naturalnaming.ConfigGroup(full_name='', trajectory=None, comment='')[source]

Group node in your trajectory, hanging below traj.config.

You can add other groups or parameters to it.

f_aconf(*args, **kwargs)

Adds a config parameter under the current group.

Similar to f_add_parameter().

If current group is the trajectory the prefix ‘config’ is added to the name.

f_add_config(*args, **kwargs)[source]

Adds a config parameter under the current group.

Similar to f_add_parameter().

If current group is the trajectory the prefix ‘config’ is added to the name.

f_add_config_group(*args, **kwargs)[source]

Adds an empty config group under the current node.

Adds the full name of the current node as prefix to the name of the group. If current node is the trajectory (root), the prefix ‘config’ is added to the full name.

The name can also contain subgroups separated via colons, for example: name=subgroup1.subgroup2.subgroup3. These other parent groups will be automatically be created.

DerivedParameterGroup

class pypet.naturalnaming.DerivedParameterGroup(full_name='', trajectory=None, comment='')[source]

Group node in your trajectory, hanging below traj.derived_parameters.

You can add other groups or parameters to it.

f_add_derived_parameter(*args, **kwargs)[source]

Adds a derived parameter under the current group.

Similar to f_add_parameter()

Naming prefixes are added as in f_add_derived_parameter_group()

f_add_derived_parameter_group(*args, **kwargs)[source]

Adds an empty derived parameter group under the current node.

Adds the full name of the current node as prefix to the name of the group. If current node is a single run (root) adds the prefix ‘derived_parameters.runs.run_08%d%’ to the full name where ‘08%d’ is replaced by the index of the current run.

The name can also contain subgroups separated via colons, for example: name=subgroup1.subgroup2.subgroup3. These other parent groups will be automatically be created.

f_adpar(*args, **kwargs)

Adds a derived parameter under the current group.

Similar to f_add_parameter()

Naming prefixes are added as in f_add_derived_parameter_group()

ResultGroup

class pypet.naturalnaming.ResultGroup(full_name='', trajectory=None, comment='')[source]

Group node in your trajectory, hanging below traj.results.

You can add other groups or results to it.

f_add_result(*args, **kwargs)[source]

Adds a result under the current node.

There are two ways to add a new result either by adding a result instance:

>>> new_result = Result('group1.group2.myresult', 1666, x=3, y=4, comment='Example!')
>>> traj.f_add_result(new_result)

Or by passing the values directly to the function, with the name being the first (non-keyword!) argument:

>>> traj.f_add_result('group1.group2.myresult', 1666, x=3, y=3,comment='Example!')

If you want to create a different result than the standard result, you can give the constructor as the first (non-keyword!) argument followed by the name (non-keyword!):

>>> traj.f_add_result(PickleResult,'group1.group2.myresult', 1666, x=3, y=3, comment='Example!')

Additional arguments (here 1666) or keyword arguments (here x=3, y=3) are passed onto the constructor of the result.

Adds the full name of the current node as prefix to the name of the result. If current node is a single run (root) adds the prefix ‘results.runs.run_08%d%’ to the full name where ‘08%d’ is replaced by the index of the current run.

f_add_result_group(*args, **kwargs)[source]

Adds an empty result group under the current node.

Adds the full name of the current node as prefix to the name of the group. If current node is a single run (root) adds the prefix ‘results.runs.run_08%d%’ to the full name where ‘08%d’ is replaced by the index of the current run.

The name can also contain subgroups separated via colons, for example: name=subgroup1.subgroup2.subgroup3. These other parent groups will be automatically be created.

f_ares(*args, **kwargs)

Adds a result under the current node.

There are two ways to add a new result either by adding a result instance:

>>> new_result = Result('group1.group2.myresult', 1666, x=3, y=4, comment='Example!')
>>> traj.f_add_result(new_result)

Or by passing the values directly to the function, with the name being the first (non-keyword!) argument:

>>> traj.f_add_result('group1.group2.myresult', 1666, x=3, y=3,comment='Example!')

If you want to create a different result than the standard result, you can give the constructor as the first (non-keyword!) argument followed by the name (non-keyword!):

>>> traj.f_add_result(PickleResult,'group1.group2.myresult', 1666, x=3, y=3, comment='Example!')

Additional arguments (here 1666) or keyword arguments (here x=3, y=3) are passed onto the constructor of the result.

Adds the full name of the current node as prefix to the name of the result. If current node is a single run (root) adds the prefix ‘results.runs.run_08%d%’ to the full name where ‘08%d’ is replaced by the index of the current run.