KVSplit
A patch for llama.cpp that quantizes KV cache keys and values at different precisions, cutting memory use so Apple Silicon Macs can run LLMs with longer contexts.
Open sourceCoding & Development
Visit website(opens in a new tab)
A patch for llama.cpp that quantizes KV cache keys and values at different precisions, cutting memory use so Apple Silicon Macs can run LLMs with longer contexts.