Utils

Exploration Functions

Module containing factory functions for parameter exploration

pypet.utils.explore.cartesian_product(parameter_dict, combined_parameters=())

Generates a Cartesian product of the input parameter dictionary.

For example:

>>> print cartesian_product({'param1':[1,2,3], 'param2':[42.0, 52.5]})
{'param1':[1,1,2,2,3,3],'param2': [42.0,52.5,42.0,52.5,42.0,52.5]}

Parameters:

• parameter_dict – Dictionary containing parameter names as keys and iterables of data to explore.
• combined_parameters –

  Tuple of tuples. Defines the order of the parameters and parameters that are linked together. If an inner tuple contains only a single item, you can spare the inner tuple brackets.

  For example:

  >>> print cartesian_product( {'param1': [42.0, 52.5], 'param2':['a', 'b'], 'param3' : [1,2,3]}, ('param3',('param1', 'param2')))
  {param3':[1,1,2,2,3,3],'param1' : [42.0,52.5,42.0,52.5,42.0,52.5], 'param2':['a','b','a','b','a','b']}
  

Returns:

Dictionary with cartesian product lists.

pypet.utils.explore.find_unique_points(explored_parameters)

Takes a list of explored parameters and finds unique parameter combinations.

If parameter ranges are hashable operates in O(N), otherwise O(N**2).

Parameters:explored_parameters – List of explored parameters Returns:List of tuples, first entry being the parameter values, second entry a list containing the run position of the unique combination.

Utility Functions

HDF5 File Compression

You can use the following function to compress an existing HDF5 file that already contains a trajectory. This only works under Linux.

pypet.compact_hdf5_file(filename, name=None, index=None, keep_backup=True)

Can compress an HDF5 to reduce file size.

The properties on how to compress the new file are taken from a given trajectory in the file. Simply calls ptrepack from the command line. (Se also https://pytables.github.io/usersguide/utilities.html#ptrepackdescr)

Currently only supported under Linux, no guarantee for Windows usage.

Parameters:

• filename – Name of the file to compact
• name – The name of the trajectory from which the compression properties are taken
• index – Instead of a name you could also specify an index, i.e -1 for the last trajectory in the file.
• keep_backup – If a back up version of the original file should be kept. The backup file is named as the original but _backup is appended to the end.

Returns:

The return/error code of ptrepack

Progressbar

Simple progressbar that can be used during a for-loop (no initialisation necessary). It displays progress and estimates remaining time.

pypet.progressbar(index, total, percentage_step=10, logger='print', log_level=20, reprint=True, time=True, length=20, fmt_string=None, reset=False)

Plots a progress bar to the given logger for large for loops.

To be used inside a for-loop at the end of the loop:

for irun in range(42):
    my_costly_job() # Your expensive function
    progressbar(index=irun, total=42, reprint=True) # shows a growing progressbar

There is no initialisation of the progressbar necessary before the for-loop. The progressbar will be reset automatically if used in another for-loop.

Parameters:

• index – Current index of for-loop
• total – Total size of for-loop
• percentage_step – Steps with which the bar should be plotted
• logger – Logger to write to - with level INFO. If string ‘print’ is given, the print statement is used. Use None if you don’t want to print or log the progressbar statement.
• log_level – Log level with which to log.
• reprint – If no new line should be plotted but carriage return (works only for printing)
• time – If the remaining time should be estimated and displayed
• length – Length of the bar in = signs.
• fmt_string – A string which contains exactly one %s in order to incorporate the progressbar. If such a string is given, fmt_string % progressbar is printed/logged.
• reset – If the progressbar should be restarted. If progressbar is called with a lower index than the one before, the progressbar is automatically restarted.

Returns:

The progressbar string or None if the string has not been updated.

Multiprocessing Directory Creation

Function that calls os.makedirs but takes care about race conditions if multiple processes or threads try to create the directories at the same time.

pypet.racedirs(path)

Like os.makedirs but takes care about race conditions

Merging many Trajectories

You can easily merge several trajectories located in one directory into one with

pypet.merge_all_in_folder(folder, ext='.hdf5', dynamic_imports=None, storage_service=None, force=False, ignore_data=(), move_data=False, delete_other_files=False, keep_info=True, keep_other_trajectory_info=True, merge_config=True, backup=True)

Merges all files in a given folder.

IMPORTANT: Does not check if there are more than 1 trajectory in a file. Always uses the last trajectory in file and ignores the other ones.

Trajectories are merged according to the alphabetical order of the files, i.e. the resulting merged trajectory is found in the first file (according to lexicographic ordering).

Parameters:

• folder – folder (not recursive) where to look for files
• ext – only files with the given extension are used
• dynamic_imports – Dynamic imports for loading
• storage_service – storage service to use, leave None to use the default one
• force – If loading should be forced.
• delete_other_files – Deletes files of merged trajectories

All other parameters as in f_merge_many of the trajectory.

Returns:The merged traj

Manual Runs

If you don’t want to use an Environment but manually schedule runs, take a look at the following decorator:

pypet.manual_run(turn_into_run=True, store_meta_data=True, clean_up=True)

Can be used to decorate a function as a manual run function.

This can be helpful if you want the run functionality without using an environment.

Parameters:

• turn_into_run – If the trajectory should become a single run with more specialized functionality during a single run.
• store_meta_data – If meta-data like runtime should be automatically stored
• clean_up – If all data added during the single run should be removed, only works if turn_into_run=True.

General Equality Function and Comparisons of Parameters and Results

Module containing utility functions to compare parameters and results

pypet.utils.comparisons.get_all_attributes(instance)

Returns an attribute value dictionary much like __dict__ but incorporates __slots__

pypet.utils.comparisons.nested_equal(a, b)

Compares two objects recursively by their elements.

Also handles numpy arrays, pandas data and sparse matrices.

First checks if the data falls into the above categories. If not, it is checked if a or b are some type of sequence or mapping and the contained elements are compared. If this is not the case, it is checked if a or b do provide a custom __eq__ that evaluates to a single boolean value. If this is not the case, the attributes of a and b are compared. If this does not help either, normal == is used.

Assumes hashable items are not mutable in a way that affects equality. Based on the suggestion from HERE, thanks again Lauritz V. Thaulow :-)

pypet.utils.comparisons.parameters_equal(a, b)

Compares two parameter instances

Checks full name, data, and ranges. Does not consider the comment.

Returns:True or False Raises:ValueError if both inputs are no parameter instances

pypet.utils.comparisons.results_equal(a, b)

Compares two result instances

Checks full name and all data. Does not consider the comment.

Returns:True or False Raises:ValueError if both inputs are no result instances