llama.cpp
Adding a model requires few steps:
llama.cpp
After following these steps, you can open PR.
Also, it is important to check that the examples and main ggml backends (CUDA, METAL, CPU) are working with the new architecture, especially:
This step is done in python with a convert
script using the gguf library.
Depending on the model architecture, you can use either convert.py or convert-hf-to-gguf.py.
The convert script reads the model configuration, tokenizer, tensor names+data and converts them to GGUF metadata and tensors.
The required steps to implement for an HF model are:
Model.register
annotation in a new Model
subclass, example:@Model.register("MyModelForCausalLM")
class MyModel(Model):
model_arch = gguf.MODEL_ARCH.GROK
Add an enum entry in MODEL_ARCH
, the model human friendly name in MODEL_ARCH_NAMES
and the GGUF tensor names in MODEL_TENSORS
.
Example for falcon
model:
MODEL_ARCH.FALCON: [
MODEL_TENSOR.TOKEN_EMBD,
MODEL_TENSOR.OUTPUT_NORM,
MODEL_TENSOR.OUTPUT,
MODEL_TENSOR.ATTN_NORM,
MODEL_TENSOR.ATTN_NORM_2,
MODEL_TENSOR.ATTN_QKV,
MODEL_TENSOR.ATTN_OUT,
MODEL_TENSOR.FFN_DOWN,
MODEL_TENSOR.FFN_UP,
]
As a general rule, before adding a new tensor name to GGUF, be sure the equivalent naming does not already exist.
Once you have found the GGUF tensor name equivalent, add it to the tensor_mapping.py file.
If the tensor name is part of a repetitive layer/block, the key word bid
substitutes it.
Example for the normalization tensor in attention layers:
block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = {
# Attention norm
MODEL_TENSOR.ATTN_NORM: (
"gpt_neox.layers.{bid}.input_layernorm", # gptneox
"transformer.h.{bid}.ln_1", # gpt2 gpt-j refact qwen
"transformer.blocks.{bid}.norm_1", # mpt
...
)
}
transformer.blocks.{bid}.norm_1
will be mapped to blk.{bid}.attn_norm
in GGUF.
Depending on the model configuration, tokenizer, code and tensors layout, you will have to override:
Model#set_gguf_parameters
Model#set_vocab
Model#write_tensors
NOTE: Tensor names must end with .weight
suffix, that is the convention and several tools like quantize
expect this to proceed the weights.
llama.cpp
The model params and tensors layout must be defined in llama.cpp
:
llm_arch
LLM_TENSOR_NAMES
llm_load_hparams
llm_load_tensors
llama_rope_type
NOTE: The dimensions in ggml
are typically in the reverse order of the pytorch
dimensions.
This is the funniest part, you have to provide the inference graph implementation of the new model architecture in llama_build_graph
.
Have a look at existing implementation like build_llama
, build_dbrx
or build_bert
.
When implementing a new graph, please note that the underlying ggml
backends might not support them all, support for missing backend operations can be added in another PR.
Note: to debug the inference graph: you can use eval-callback.
https://github.com/ggerganov/ggml/blob/master/docs/gguf.md
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。