What's the impact of running MLM during training?

[shiweii2r]

Hi, thank you very much for sharing the code! From the code of run.py, in function train(), during each training epoch, if args.mlm_during is true, "Run MLM during training" part will run. But it doesn't change "model". Is it because this sentence "model.bert_model = pre_model.bert_model.bert", that "model" and "pre_model" share the same weights since they use the same physical address? Thank you very much!