Implementation of Custom Algorithms

[sonotokay]

Hello there!

Just wondering if there are any tutorials/documentation regarding the implementation of Custom RL algorithms?
From what i understand, there is already a custom_algorithms file, where we could just create a class and input all the
needed parameters, along with the actual implementation for said algorithm, but it seems to still be a very complex task.

I do find the logging of the training metrics in weights and biases to be very nice, although i do still need
to brush up on that. And i haven't actually tried it yet, but i assume recorded runs are saved in Trackmania?

Thank you very much in advance for your time and help!

[yannbouteiller]

Hi!

There are a bunch of tutorials out there, if you just want to implement your own training algorithm but still use the readily implemented tmrl gym environment for TrackMania, I recommend looking into the competition tutorial script (it is a working bugged script in which the comments explain how to customize the TrackMania pipeline using the tmrl API).

Note: it requires a good mastery of deep RL. If you have never Implemented deep RL algorithms before, I would start with simple tutorials about how to implement, e.g., Q-learning in the "frozen lake" environment, DQN in Atari, or REINFORCE/DDPG/SAC/PPO in the classic MuJoCo continuous control tasks.

About the replays, there is an option for saving them in TrackMania, yes :) You can enable that in config.json ("ENV" > (...) > "save_replays"). They get saved in the "my replays" folder I believe.