We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
we will finetune a 70B model that support long content with 800k, can vllm support to inference this model?
The text was updated successfully, but these errors were encountered:
Hi @yunll, yes if you have enough GPU memory available for a context length that large it will run. I tested Mistral 128k last week and was able to use its full length. Model for reference: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k
Sorry, something went wrong.
No branches or pull requests
Anything you want to discuss about vllm.
we will finetune a 70B model that support long content with 800k, can vllm support to inference this model?
The text was updated successfully, but these errors were encountered: