Changelog

pypet 0.5.1

  • Updated package description to automatically convert md to rst for pypi
  • Updated pngpath for Sphinx

pypet 0.5.0

  • Fix to work with pandas 1.0
  • Fix to work with brian2 2.3
  • Fix to work with Python 3.7 and 3.8
  • Removal expectedrows and filters option for HDF5Storage.put as this is no longer supported by pandas

pypet 0.4.3

pypet 0.4.2

  • BUG FIX: Brian2Parameters can now be merged [Issue #50]

pypet 0.4.1

  • BUG FIX: Fixed slow f_get_from_all_runs with run groups
  • More intuitive behaviro for f_to_dict to only start at initiating group node.
  • BUG FIX: brian 2 removed get unit fast, this has been replaced with a pypet implementation

pypet 0.4.0

  • BUG FIX: Correct progressbar printing in Python3 using reprint
  • ENH: New nested option for f_to_dict
  • Python 3 Support ONLY!!!

pypet 0.3.0

  • Support for BRIAN2
  • New Brian2Parameter
  • New Brian2Result and Brian2MontitorResult
  • New BRIAN2 Network framework
  • BUG FIX: Auto loading parameters will set the parameter to the current run.

pypet 0.2.0

  • No longer beta

pypet 0.2b1

  • ENH: Pypet can now apply fast merging if trajectories reside in different files. If one still wants to apply slow merging, one can set slow_merge=True.
  • ENH: New merge_all_in_folder function
  • The range is now a list that is by default copied upon request. To avoid quadratic complexity with multiple calls to f_expand.
  • ENH: Meta data of single runs is no longer stored during the single run to avoid overhead and guarantee faster runs
  • For more stability continuing is no longer supported for the 'QUEUE' and 'PIPE' wrap modes.
  • Queue retries are now logged in debug mode.
  • One can now choose what f_iter_runs should return.
  • New f_run_map and f_pipeline_map functionality for iterable arguments.
  • New wrap mode 'LOCAL' to ensure data is only stored by the main program
  • Immediate post-processing is now supported by all wrap modes.
  • PIPE wrapping is no longer supported under Windows
  • The add_time parameter of the trajectory is by default now False
  • New niceness feature to prioritize child processes
  • New scoop feature to allow multiprocessing with the scoop package.
  • Passing log_multiproc=False to an environment disables multiprocess logging.
  • Parameters and Results can nest data in HDF5 themselves using dot notation.
  • Removed BrianDurationParameter
  • Parameters now support lists and no longer make numpy arrays unwritable
  • Results are now always sorted according to the run indices (except for immediate post-processing)
  • Process ID and hostname are also added to the logfiles’ naming scheme
  • freeze_pool_input has been renamed to freeze_input because it can be used with SCOOP as well.
  • New 'NETLOCK' wrap mode, to allow locking over a network and usage of servers with a shared home directory.
  • Prefixes are no longer mandatory if using func and vars: traj.f_iter_runs() is equivalent to traj.func.iter_runs() or myresult.v_name is equivalent to myresult.vars.name
  • BUG FIX: If a trajectory is loaded as new the explored parameters are now correctly loaded
  • New graceful_exit mode to allow to stop your programs via CTRL+C while finishing all your active runs first and storing all your remaining data.
  • New v_no_clobber property that allows to ignore additions to the trajectory that are already there.
  • New rts_X abbreviation that translates a given run index into the correspondin run set, e.g. rts_4007 gives run_set_00004`.
  • log_stdout is False by default.
  • pypet can now be combined with DEAP optimization.
  • All deprecated functions have been removed!
  • Lazy adding is no longer supported!

pypet 0.2b.0

  • Erroneous Release due to PyPI fault :/

pypet 0.1b.12

  • Renaming of the MultiprocContext’s start function to f_start
  • BUG FIX: Correct lock acquiring for multiprocessing with StorageContextManager
  • BUG FIX: v_full_copy is now False by default
  • BUG FIX: v_full_copy is no longer automatically set to True when using freeze_pool_input.
  • New consecutive_merge parameter for f_merge to allow faster merging of several trajectories.
  • New f_merge_many function to merge many trajectories at once.
  • New experimental 'PIPE' wrapping which is even faster (but more unreliable) than 'QUEUE'.

pypet 0.1b.11

  • If one wants the old logging method, log_config should not be specified, i.e. setting it to None is no longer sufficient`
  • BUG FIX: Connection loss between the queue manager and the pool/processes has been resolved. This caused a minor slowing down of multiprocessing using a queue.
  • New freeze_pool_input functionality for faster starting of single runs if using a pool.

pypet 0.1b.10

  • New v_crun_ property simply returning 'run_ALL' if v_crun is None.
  • BUG FIX: Removed recursive evaluation due to usage of itertools.chain during recursive node traversal
  • max_depth is now also supported by f_store, f_store_child, f_load, f_load_child
  • Loading and Storing internally are no longer truly recursive but iteratively handled.
  • New v_auto_run_prepend property of the trajectory to switch off auto run name prepending if desired.
  • The trajectory no longer relies on evil eval to construct a class. Instead it relies on the global scope.
  • Better counting of loading and storing nodes to display rate in nodes/s
  • BUG FIX: Minor bug in the progressbar has been fixed to detect automatic resets.
  • Now support for non-nested empty containers: Empty dictionary, empty list, empty tuple and empty numpy array. All of them supported by the ArrayParameter and normal Results.
  • Support for Sparse matrices containing NO data (i.e. only zeros).
  • Performance optimization for storage and loading
  • Improved test handling and parsing in pypet.tests
  • Environment now supports git_fail option to fail if there are not committed changes instead of triggering a new commit.
  • Users can now define their own functions to produce run-names
  • Likewise users can define their onw wildcards
  • The lazy version of adding elements (traj.par.x = 42, ‘A comment’) now needs to be turned on by the user via (`traj.v_lazy_adding = True) before it can be used.
  • HDF5_STRCOL_MAX_ARRAY_LENGTH has been renamed to HDF5_STRCOL_MAX_RANGE_LENGTH
  • The summary tables have been shortened. Now there’s no distinction anymore between the actual runs and everything else.
  • Moreover, data added to summary tables is no longer deleted. There also exists a maximum length for these tables (1000).
  • The overview about the explored parameters in each run has been removed (due to size)
  • Summary tables are now only based on the comments not the names!
  • One can pass an estimate for memory that each run needs to better protect the memory cap.
  • All tree nodes except the trajectory now use __slots__ for faster and more compact creation.
  • You can now request to load a trajectory without run_information to save time for huge trajectories
  • Trajectories use ordered dictionaries to remember what was added during a single run. Accordingly, now every data added during a single run regardless if they were added below a group run_XXXXXXXX is stored.
  • BUG FIX: The ‘QUEUE’ wrapping no longer waits for chunks of data, but starts storing immediately. Thus, if you have fast simulations, the storage service no longer waits until the end of you simulation before it starts storing data. In order to avoid overhead, the hdf5 is simply kept open until the queue is closed.
  • BUG FIX: If log_stdout=True, the original stream is restored instead of sys.__stdout__. Thus, using another Python interpreter with a redirection of stdout and calling f_disable_logging no longer disables print completely.
  • Refactored ‘QUEUE’ wrapping. The user can now decide the maximum size of the Storage Queue.
  • CAP values are now in %, so choose values between 0.0 and 100.0
  • BUG FIX: Links removed during single runs are now no longer stored
  • BUG FIX: pypet. is no longer prepended to unpickled logger names. Accordingly, pypet logger names are now fully qualified names like pypet.trajectory.Trajectory.

pypet 0.1b.9

  • BUG FIX: Fixed backwards compatibility
  • BUG FIX: Metadata is loaded only once
  • Results no longer support the v_no_data_string property
  • Data of Results is no longer sorted in case of calling f_val_to_string
  • In accordance with the python default to call __repr__ for displaying contained objects, f_val_to_str calls repr on the contained data in parameters and results.
  • Added informative __repr__ for the most commonly used classes
  • The (annoyingly long) keyword dynamically_imported_classes is changed to dynamic_imports. For backwards compatibility, the old keyword can still be used.
  • New f_get_default method, now one can specify a default value that should be returned if the requested data is not found in the trajectory
  • progressbar displays the run and remaining time.
  • New LINK features allowing group nodes to refer to other parts of the tree
  • The SingleRun has been removed and all interactions are with real Trajectory objects, but the API remained the same.
  • All pypet relevant imported packages will be stored by the trajectory
  • Internally the queue no longer needs to be re-passed to the QueueSender, allowing for easier custom multiprocessing
  • New MultiprocessWrapper (aka a light-weight environment for multiprocessing) for custom multiprocessing
  • StorageServices provide an multiproc_safe attribute to enable the user to check if they work in a multi-process safe environment
  • Environments can be used as context managers to disable the logging to files after the experiment.
  • Environments provide the v_log_path property to get the current log path
  • BUG FIX: Trajectories with only a single explored parameter can now be merged several times
  • Backwards search no longer supported!
  • f_get_all now supports shortcuts and abbreviations like crun or par
  • $ always translates to the run the trajectory is set to, also for adding new items to the tree
  • If the current run is not set, traj.v_crun is set to None
  • Moreover, f_iter_nodes and f_iter_leaves is no longer affected by the setting of a current run and always return all nodes and leaves
  • The iteration functions from above now allow for a predicate function to filter potential nodes
  • Storing a leaf or a group via traj.f_store_item(item, overwrite=True) now also replaces all annotations and comments
  • Passing overwrite_file=True to an environment will overwrite the hdf5 file.
  • remove_empty_groups is no longer supported
  • All messages logged by pypet are now no longer using the root logger but one called ‘pypet’.
  • Better customization of logging. The user can now pass a list of logger_names and corresponding log_levels which are logged to files in the log_path.
  • The environment no longer adds config information about hdf5 to the trajectory directly. This is now done by the service itself.
  • The keyword arguments passed to the environment regarding the storage service are no longer handled by the environment but are directly passed to the service.
  • BUG FIX: Fixed merging of result summaries that are only found in one trajectory
  • BUG FIX: Log files are now closed when the handlers are removed
  • BUG FIX: max_depth is now really always in relation to the start node and not in relation to intermediate results
  • API change for f_migrate to match new concept of storage service
  • Short function names for item additions like f_apar besides f_add_parameter.
  • Abbreviations like par and dpar can now also be used for item creation and are always translated
  • To streamline the API you can now no longer specify the name of backup files for merging
  • Locked parameters can no longer be loaded and must be unlocked before.
  • Parameters are no longer required to implement __len__ because it can be ambiguous, instead they must implement f_get_range_length function.
  • BUG FIX: crun is now also accepted for adding of data and not only for requests
  • Setting ncores=0 lets pypet determine the number of CPUs automatically (requires psutil).

pypet 0.1b.8

  • Support for python 3.3 and 3.4!
  • Proper handling of unicode strings (well, see above^^)
  • Checking if names of leaf and group nodes only contain alphanumeric characters
  • PickleParameter and PickleResult now explicitly store the pickle protocol because retrieval from the pickle dump is not always possible in python 3.
  • Children of groups are no longer listed via __dir__ in case of debugging to prevent unwanted locking.
  • Better support for PyTables 2 and 3 with same code base.
  • pypet and pypet.brian now provide the __all__ list.

pypet 0.1b.7

  • StreamToLogger has moved to the pypetlogging.py module. The mplogging module was deleted.
  • The Network Manager now accepts custom network constructors
  • A SingleRun can now provide a v_filename and v_as_run property. Both cannot be changed and the latter simply returns the name of the run.
  • Better testing on travis
  • Better support for pandas 0.14.1
  • Now you can import most of the objects directly from the pypet package, e.g. from pypet import Trajectory instead of from pypet.trajectory import Trajectory.

pypet 0.1b.6

  • The storage service prints status updates for loading and storing trees
  • v_as_run is not longer None for a trajectory but run_ALL
  • The Queue storage writer now stores batches of single runs to avoid re-opening of files as much as possible
  • Faster Loading of data
  • Support for PyTables 3.1.1
  • pypet stores the name of the main script as a config parameter
  • Data of Parameters can be accessed via .data or param[‘data’]. Same holds for results that only contain a single data item.
  • Parameters provide the function f_get_default to return the default value if the parameter is not empty.
  • Large dictionaries and Object Tables are now split into chunks of 512 items
  • In case an object table has more than 32 columns, another table is created to store the data item types (which is faster than storing all of the types as hdf5 node attributes)

pypet 0.1b.5

  • New auto load feature
  • BUG FIX: When parameters are emptied, the default value is set to None (and no longer kept)
  • Now items are only saved once, if the node already exist on disk, storage is refused (Previously new data was added if it was not within the leaf before, but this can lead to strange inconsistencies).
  • BUG FIX: f_has_children of a group node, now returns the correct result
  • Refactored continuing of trajectories. Now this is based on dill and works also with data that cannot be pickled. f_continue_run is renamed f_continue to emphasize this change in API
  • Picking the search strategy and using v_check_uniqueness is no longer supported. Sorry for the inconvenience. So forward search will always check if it finds 2 nodes with the same name within the same depth, and skip search if the next tree depth is entered.
  • f_contains of group nodes has per default shortcuts=False
  • There exists now the abstract class HasLogger in pypetlogging.py that establishes a unified logging framework
  • Likewise the loggers of all network components are now private objects _logger and so is the function _set_logger.
  • BUG FIX: f_get_run_information now works without passing any arguments
  • Trajectories no longer accept a file_tile on initialisation
  • One can now decide if trajectories should be automatically stored and if data should be cleaned up after every run
  • BUG FIX: Storage of individual items during a single run do no longer require a full storage of the single run container.
  • If automatic storage is enabled, trajectories are now stored at the end of the experiment, no longer before the starting of the single runs
  • You can use the $ character to decide where the HDF5 file tree should branch out for the individual runs
  • v_creator_name is now called v_run_branch (since single runs can also create items that are not part of a run branch, so this is no longer misleading`.
  • Results and parameters now issue a warning when they have been stored and you change their data
  • Parameters now have a property v_explored which is True for explored parameters even if the range has been removed
  • By default backwards_search is turned off!
  • Brian parameters no longer store the storage_mode explicitly
  • BUG FIX: Wildcard character now always defaults to run_ALL if trajectory is not set to particular run
  • BUG FIX: Now names with XXXrun_ are again allowed only run_ are forbidden.

pypet 0.1b.4

  • Annotations and Results now support __setitem__, which is analogue to f_get and f_set
  • Group Nodes can now contain comments as well
  • Comments are only stored to HDF5 if they are not the empty string
  • Large Overview Tables are off by default
  • BrianDurationParameter was removed and the annotations are used instead. Use a normal BrianParameter and instead of v_order use v_annotations.order
  • The user is advised to use environment.f_run(manager.run_network), instead of environment.f_run(run_network, manager)
  • Now there is the distinction between small, large, and summary tables
  • BrianMonitorResult: Mean and variance values of State and MultiState Monitors are only extracted if they were recorded (which for newer BRIAN versions is only the case if you do NOT record traces)
  • Log Level can be passed to environment
  • BUG FIX: Scalars are no longer autoconverted by the storage service to zero-length numpy arrays
  • Names of loggers have been shortened
  • The trajectory now contains the functions f_delete_item and f_delete_items to erase stuff from disk. f_remove_items and f_remove_item no longer delete data from disk.
  • Loading and deletions of items can now be made with SingleRuns as well.
  • f_iter_nodes now iterates by default recursively all nodes
  • A group node now supports __iter__ which simply calls f_iter_nodes NON recursively
  • The structure of the trees are slightly changed. Results and derived parameters added with the trajectory are no longer assigned the prefix trajectory. Results and derived parameters added during single runs are now sorted into runs.run_XXXXXXXXX.
  • Useless shortcuts have been removed.
  • New Backwards search functionality
  • New f_get_all functionality to find all items in a tree matching a particular search string
  • Pandas Series and Panels are now supported, too!
  • Now Pypet allows compression of HDF5 files, this can yield a massive reduction in memory space.
  • tr, cr, current_run, param, dparam are no longer supported as a shortcuts
  • __getitem__ is equivalent to __getattr__
  • Now one can specify a maximum depth for accessing items.
  • Now one can specify if shortcuts, i.e. hopping over parts of the tree, are allowed.
  • New trajectroy attributes v_backwards_search, v_max_depth, v_shortcuts and v_iter_recursive. v_iter_recursive specifies the behavior of __iter__.
  • __contains__ now greps is arguments from the attributes v_max_depth and v_shortcuts.
  • log_stdout parameter of the Environment determines if STDOUT and STDERROR will be logged to files. Can be disabled to allow better usage of pypet with interactive consoles.
  • git commits only happen on changes and not all the time
  • Now one can specify CPU, RAM and Swap cap levels if no pool is used. If these cap levels are crossed pypet won’t start new processes.
  • f_load now has an argument load_all to quickly load all subtrees with the same setting. Also f_load now accepts a filename as well
  • New post-processing and pipeline modes!

pypet 0.1b.3

  • BUG FIX: Now f_run and f_continue_run of an environment return the results produced by runfunc
  • You can enforce a type convert for exploration
  • Added lazy_debug option for the environment
  • If you don’t specify a filename, the environment defaults to a file with the name of the trajectory
  • New multiprocessing mode (use_pool=False for environment) to spawn individual processes for each run. Useful if data cannot be pickled.
  • New Brian framework with NetworkManager, NetworkComponent, NetworkAnalyser, NetworkRunner and DurationParameter
  • Components, Analysers and the network runner of the manager are now publicly available
  • Every component now provides the function set_logger to enable logging and instantiate a logger for self.logger

pypet 0.1b.2

  • DefaultReplacementError is now called PresettingError
  • Now the runtime of single runs is measured and stored.
  • __getitem__ of the trajectory always returns the instance and fast access is not applied
  • PickleResult and PickleParameter support the choice of protocol
  • Explored Sparse matrices are stored under a slightly different name to disk.
  • BUG FIX: BFS works properly
  • BUG FIX: f_iter_runs is now affected if f_as_run is chosen
  • Annotations support __iter__
  • Annotations support __getitem__
  • BrianMonitorResult, the property and columns ‘times’ for the Spike and StateSpikeMonitor has been renamed ‘spiketimes’.
  • Results support __iter__
  • BrianMonitorResult, the name of state variables in array mode is changed to varname+’_%Xd’, instead of varname+’_idx%08d’, and ‘spiketimes_%08d’ to ‘spiketimes_%Xd’ and X is chosen in accordance with the number of neurons
  • BUG FIX: nested_equal now supports Object Tables containing numpy arrays
  • Better categorizations of the utility functions
  • Comments are no longer limited in size
  • New Brian Result
  • Storage Service - in case of purging - now sets the comment to the result with the lowest index, in case of multiprocessing.
  • Old API names are kept, but emit a depricated warning.
  • The exploration array is now termed range. Accordingly, the function f_is_array is renamed f_has_range and f_get_array renamed to f_get_range.
  • v_leaf renamed to v_is_leaf
  • f_is_root renamed to v_is_root and changed to property
  • v_fast_accessible renamed to f_supports_fast_access and changed to function
  • v_parameter changed to v_is_parameter

pypet 0.1b.1

  • Support for long types
  • Documentation for the f_find_idx function
  • The parameters trajectory_name and trajectory_index in f_load have been renamed to name and index

pypet 0.1b.0

  • Group nodes support __getitem__

  • SparseResult

  • If you merge a trajectory, all environment settings of both are kept.

  • More information about the environment is added to the trajectory

  • BUG FIX:

    Recall of trajectory comments from disks yielded numpy strings instead of python strings This could cause trouble if the comment is empty!

  • Git Integration, you can now make autocommits for every experiment

  • New Sparse Parameter, for scipy sparse matrices

  • BUG FIX: Loading of Trajectory metadata, now v_time is loaded correctly

  • BUG FIX: Expand no longer repeats already run experiments

  • More fine grain overview tables

  • Comments for runs are only added once and not every run

  • The overview tables are now found in the group overview

  • Test are operating in a temp directory

  • Now you can have fast access with results if they contain only a single entry with the name of the result

  • New trajectory function f_as_run that makes the trajectory belief it is a particular single run and results and derived parameters of other runs are blinded out.

  • Every group node now knows how to store and load itself via f_load and f_store.

  • Storage of data is now analogous to loading with constants in (1,2,3). 1 Storing data only of items not been stored before, 2 for storing data as previously known. 3 For overwriting data. For instance, traj.f_load(store_data=3) overwrites all data on disk.

  • f_update_skeleton is now f_load_skeleton to be more in line with naming scheme.

  • setattr no longer supports shortcuts, i.e. traj.x = 4 only works if x is directly below the trajectory root.

  • Using setattr with a tuple of exactly length 2 whereas the second element is a string, sets the value as well as a comment

pypet 0.1a.6

  • BUG FIX: (HDF5StorageService): storing a trajectory several times increased run and info table

pypet 0.1a.5

  • Removed unnecessary imports
  • Better documentation

pypet 0.1a.4

  • Adding positional results will add them with the result name

pypet 0.1a.3

  • Better handling of filenames, now relative paths are considered

pypet 0.1a.2

  • Added automatic version grapping in setup.py

pypet 0.1a.1

  • BaseParameter supports now __getitem__ if it is an array