Moved the creation of the sorted vocabulary in "build_tokenizer()" #328
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Slight downside here is that sorting the vocab probably creates latency, and if the user isn't going to prompt the model we are paying the price needlessly. Maybe we can allocate it lazily? |
|
Yes, it can be done in "encode()" to ensure it is done only when needed. I'll change it. |
|
Done. Frankly, I don't think there will be many using llama2.c with no prompt, but I may be wrong. |
|
Just pushed a very similar commit ty |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the vocabulary is sorted every time the
encode()function is called.For the current version, it's not an issue since the program exits every time but if the generation code would be called within a loop to have a "chat" version, this qsort will be useless (and detrimental).
I've moved this to the
build_tokenizer()function as it seems to me that this is an operation that should logically be performed at the time when we set up the tokenizer.