Change clone files mechanism

[omer-dayan]

For S3 Model loading, we clone all files except weights files to a local, temporary, directory.

We do it using ignore pattern of ["*.pt", "*.safetensors", "*.bin"].
This raise a problem, users may upload large files to the S3 (For example, like in this issue: run-ai/runai-model-streamer#25 (comment))
The case is very valid, downloading all model repo from HF and directly upload it to S3 (Including .git which is a large dir)
Most of them are not needed.

One option is to still pull all files, but it can take time if the files are large (like the .git dir)
Option 2 - pull only relevant files, tokenizer files, however there is no clear list of which file names are relevant and may change in the future.
Option 3 - Pull only json files, from the assumption that if the filenames will change in the future most likely all the tokenizer metadata file will still be jsons.
Option 4 - ignore .git dir only

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[svantesorberg]

Just opened this issue and was directed here.

I'm of a different opinion, I don't think vLLM should make too many assumptions about file/folder names in a model directory. Afaik, huggingface models / transformers seem to be quite liberal w.r.t. what files can constitute a model. Some models have arbitrary python files, some have chat templates, etc.

Forgive me if I don't understand the context fully, but isn't the solution to the specific problem raised by users to not put unnecessary files in S3?