Fine-tune evo to achieve a sequence-level prediction

[jingliezhou]

Could you supply a script or notebook demonstrating how to perform regression analysis using two sequences as inputs? The input format would be "[CLS]GCTTAGCGAGACTAAATTATATAGCAGCT[SEP]CTAACTGCAGCCCGCCCGTAT", and the prediction should utilize the hidden state associated with the "[CLS]" token.

Thank you,
Jinglie

[kawabata-tomoko]

Here is my solution: GitHub link. You may use [EOS] instead of [CLS] because EVO is a Decoder-Only model. The [CLS]/[BOS] token cannot be used to refer to the information of the whole sequence, especially for tokens that come after [CLS].