Continue training at checkpoint

[Ariiio]

Sometimes colab just limits ones usage and training xl loras stops. This means if your training wasnt finished, you'd have to start from epoch 0 again. It would be nice it one could just continue at the most recent lora model. This is possible with non xl loras, so it should be doable for xl too right?

[Ariiio]

Just added it myself, add this at the top of the code

continue_from_lora = "" #@param {type:"string"} if continue_from_lora and not continue_from_lora.startswith("/content/drive/MyDrive"): continue_from_lora = os.path.join("/content/drive/MyDrive", continue_from_lora)

and then add this line "network_weights": continue_from_lora if continue_from_lora else None
at this code block:
"network_arguments": { "unet_lr": unet_lr, "text_encoder_lr": text_encoder_lr if not cache_text_encoder_outputs else 0, "network_dim": network_dim, "network_alpha": network_alpha, "network_module": network_module, "network_args": network_args, "network_train_unet_only": text_encoder_lr == 0 or cache_text_encoder_outputs, },