skrl-v1.4.0

[1.4.0] - Unreleased

Added

• Utilities to operate on Gymnasium spaces (Box, Discrete, MultiDiscrete, Tuple and Dict)
• parse_device static method in ML framework configuration (used in library components to set up the device)
• Model instantiator support for different shared model structures in PyTorch
• Support for automatic mixed precision training in PyTorch
• init_state_dict method to initialize model's lazy modules in PyTorch
• Model instantiators fixed_log_std parameter to define immutable log standard deviations
• Define the stochastic_evaluation trainer config to allow the use of the actions returned by the agent's model
  as-is instead of deterministic actions (mean-actions in Gaussian-based models) during evaluation.
  Make the return of deterministic actions the default behavior.

Changed

• Call agent's pre_interaction method during evaluation
• Use spaces utilities to process states, observations and actions for all the library components
• Update model instantiators definitions to process supported fundamental and composite Gymnasium spaces
• Make flattened tensor storage in memory the default option (revert changed introduced in version 1.3.0)
• Drop support for PyTorch versions prior to 1.10 (the previous supported version was 1.9)
• Update KL Adaptive learning rate scheduler implementation to match Optax's behavior in JAX
• Update AMP agent to use the environment's terminated and truncated data, and the KL Adaptive learning rate scheduler
• Update runner implementations to support definition of arbitrary agents and their models
• Speed up PyTorch implementation:
  • Disable argument checking when instantiating distributions
  • Replace PyTorch's BatchSampler by Python slice when sampling data from memory

Changed (breaking changes: style)

• Format code using Black code formatter (it's ugly, yes, but it does its job)

Fixed

• Move the batch sampling inside gradient step loop for DQN, DDQN, DDPG (RNN), TD3 (RNN), SAC and SAC (RNN)
• Model state dictionary initialization for composite Gymnasium spaces in JAX
• Add missing reduction parameter to Gaussian model instantiator
• Optax's learning rate schedulers integration in JAX implementation
• Isaac Lab wrapper's multi-agent state retrieval with gymnasium 1.0
• Treat truncation signal when computing 'done' (environment reset)

Removed

• Remove OpenAI Gym (gym) from dependencies and source code. skrl continues to support gym environments,
  it is just not installed as part of the library. If it is needed, it needs to be installed manually.
  Any gym-based environment wrapper must use the convert_gym_space space utility to operate

skrl-v1.3.0

[1.3.0] - 2024-09-11

Added

• Distributed multi-GPU and multi-node learning (JAX implementation)
• Utilities to start multiple processes from a single program invocation for distributed learning using JAX
• Model instantiators return_source parameter to get the source class definition used to instantiate the models
• Runner utility to run training/evaluation workflows in a few lines of code
• Wrapper for Isaac Lab multi-agent environments
• Wrapper for Google Brax environments

Changed

• Move the KL reduction from the PyTorch KLAdaptiveLR class to each agent that uses it in distributed runs
• Move the PyTorch distributed initialization from the agent base class to the ML framework configuration
• Upgrade model instantiator implementations to support CNN layers and complex network definitions,
  and implement them using dynamic execution of Python code
• Update Isaac Lab environment loader argument parser options to match Isaac Lab version
• Allow to store tensors/arrays with their original dimensions in memory and make it the default option

Changed (breaking changes)

• Decouple the observation and state spaces in single and multi-agent environment wrappers and add the state
  method to get the state of the environment
• Simplify multi-agent environment wrapper API by removing shared space properties and methods

Fixed

• Catch TensorBoard summary iterator exceptions in TensorboardFileIterator postprocessing utils
• Fix automatic wrapper detection issue (introduced in previous version) for Isaac Gym (previews),
  DeepMind and vectorized Gymnasium environments
• Fix vectorized/parallel environments reset method return values when called more than once
• Fix IPPO and MAPPO act method return values when JAX-NumPy backend is enabled

skrl-v1.2.0

[1.2.0] - 2024-06-23

Added

• Define the environment_info trainer config to log environment info (PyTorch implementation)
• Add support to automatically compute the write and checkpoint intervals and make it the default option
• Single forward-pass in shared models
• Distributed multi-GPU and multi-node learning (PyTorch implementation)

Changed

• Update Orbit-related source code and docs to Isaac Lab

Fixed

• Move the batch sampling inside gradient step loop for DDPG and TD3
• Perform JAX computation on the selected device

skrl-v1.1.0

[1.1.0] - 2024-02-12

Added

• MultiCategorical mixin to operate MultiDiscrete action spaces

Changed (breaking changes)

• Rename the ManualTrainer to StepTrainer
• Output training/evaluation progress messages to system's stdout
• Get single observation/action spaces for vectorized environments
• Update Isaac Orbit environment wrapper

skrl-v1.0.0

[1.0.0] - 2023-08-16

Transition from pre-release versions (1.0.0-rc.1 and1.0.0-rc.2) to a stable version.

This release also announces the publication of the skrl paper in the Journal of Machine Learning Research (JMLR):
https://www.jmlr.org/papers/v24/23-0112.html

Summary of the most relevant features:

• JAX support
• New documentation theme and structure
• Multi-agent Reinforcement Learning (MARL)

skrl-v1.0.0-rc.2

[1.0.0-rc.2] - 2023-08-11

Added

• Get truncation from time_outs info in Isaac Gym, Isaac Orbit and Omniverse Isaac Gym environments
• Time-limit (truncation) boostrapping in on-policy actor-critic agents
• Model instantiators initial_log_std parameter to set the log standard deviation's initial value

Changed

• Structure environment loaders and wrappers file hierarchy coherently [breaking change]
• Drop support for versions prior to PyTorch 1.9 (1.8.0 and 1.8.1)

skrl-v1.0.0-rc.1

[1.0.0-rc.1] - 2023-07-25

Added

• JAX support (with Flax and Optax)
• RPO agent
• IPPO and MAPPO multi-agent
• Multi-agent base class
• Bi-DexHands environment loader
• Wrapper for PettingZoo and Bi-DexHands environments
• Parameters num_envs, headless and cli_args for configuring Isaac Gym, Isaac Orbit
  and Omniverse Isaac Gym environments when they are loaded

Changed

• Migrate to pyproject.toml Python package development
• Define ML framework dependencies as optional dependencies in the library installer
• Move agent implementations with recurrent models to a separate file
• Allow closing the environment at the end of execution instead of after training/evaluation
• Documentation theme from sphinx_rtd_theme to furo
• Update documentation structure and examples

Fixed

• Compatibility for Isaac Sim or OmniIsaacGymEnvs (2022.2.0 or earlier)
• Disable PyTorch gradient computation during the environment stepping
• Get categorical models' entropy
• Typo in KLAdaptiveLR learning rate scheduler
  (keep the old name for compatibility with the examples of previous versions.
  The old name will be removed in future releases)

skrl-v0.10.2

[0.10.2] - 2023-03-23

Changed

• Update loader and utils for OmniIsaacGymEnvs 2022.2.1.0
• Update Omniverse Isaac Gym real-world examples

skrl-v0.10.1

[0.10.1] - 2023-01-26

Fixed

• Tensorboard writer instantiation when write_interval is zero

skrl-v0.10.0

[0.10.0] - 2023-01-22

Added

• Isaac Orbit environment loader
• Wrap an Isaac Orbit environment
• Gaussian-Deterministic shared model instantiator