Clarification on Fine-Tuning with Special Tokens (e.g., <cas9>) in Evo Model

[MarcAmil30]

Hello,

In the paper, it states: "During pretraining, Evo uses an effective vocabulary of four tokens, one per base, from a total vocabulary of 512 characters. We use the additional characters to enable prompting with special tokens during generation with fine-tuned models."

I wanted to clarify the fine-tuning approach, particularly for the CRISPR-Cas application. Did you append prompt tokens like or to their respective sequences, e.g., ATAGCA...? If so, were the prompt tokens (e.g., <, c, a, s, 9, >) already part of the 512-character vocabulary, or did you need to expand or retrain the model to include these new tokens?

I’m also curious whether the model would need to learn how to handle these characters during fine-tuning, since they were not part of the sequences used in pretraining.

Thank you!

[brianhie]

We prepended a special token already in the vocab during finetuning. For cas, these are:

CAS_ID_TO_START_TOKEN = {
    'Cas9': '`',
    'Cas12': '!',
    'Cas13': '@',
}

The model did not see these during pretraining but did see them (only in the first char position) during finetuning!

[MarcAmil30]

Hello,

Thank you for your response! Just to confirm, during fine-tuning, an example sequence for Cas12 would look like !ATACGA..., with the ! token prepended. Is that correct?

Additionally, I wanted to clarify the 512-token vocabulary. Are characters such as [,], capital letters (aside from A, C, T, G), and numbers included in this vocabulary? How would I be able to see the full set of characters used in the vocabulary?

Thank you!

[UsmanMahmood8]

For DNA, what were the start, end, and pad tokens? If any.
I see that for score function bos token as 0 and pad token as 1 is used. But for generation example bos token is not being used. Is there a reason for that? Why special tokens are not being prepend in the geneeration example?