Bring any huggingface model to ollama !
As of February’2024, there are just approximately 100 models in ollama library.
On the other hand, huggingface has more than 500k models as of now -
So, then how can we leverage ollama to deploy other by default unsupported models from huggingface.
Let’s dive in.
Step 1 : Go to huggingface Models repo -
Step 2 — Search for the model you want to deploy.
Step 3: Go to “File and versions” tab and look for the gguf file of the model.
NOTE : Usually model creater uploads the gguf version of the file, if not then still you can generate the file yourself. Writing another blog for that.
Assuming that the gguf file is already uploaded on huggingface, just download it on your local machine (or on the machine where your ollama server is running).
Step 4: Once downloaded, create a Modelfile on your machine on the same folder where you downloaded the gguf file, and add this line in the Modelfile -
FROM "./sqlcoder-7b-q5_k_m.gguf"Step 5— Go to ‘Model Card’ tab and look for ‘Prompt’ section -
Usually every model has its own specific propmt template.
Copy the prompt template and add it in the Modelfile with the key TEMPLATE -
TEMPLATE """
### Task
Generate a SQL query to answer [QUESTION]{user_question}[/QUESTION]
### Database Schema
The query will run on a database with the following schema:
{table_metadata_string_DDL_statements}
### Answer
Given the database schema, here is the SQL query that [QUESTION]{user_question}[/QUESTION]
[SQL]
"""Modelfile looks like this now -
Step 6 — Run this command from ollama
ollama create sqlcoder-7b-2 -f Modelfile
And that’s it, its done. Start inferencing.
Happy Learning !
