Your local AI needs memory for the whole workload
The model file is only the starting allocation. Test the expected context under real concurrency before deciding whether the machine has enough memory.
A model can load successfully, then run out of memory when a long document arrives during a second session.
A local AI workload includes more than the model file. Context needs memory beside runtime buffers. Other applications also use the machine, while simultaneous requests add their own allocation. The complete workload needs to fit at an acceptable response time.
My older note relied on inexpensive graphics cards with approximate model sizes. A memory budget is easier to reuse when those product lists age. Testing the intended work can show whether the equipment already available is sufficient.
Calculate the weights before the overhead
Seven billion parameters at exactly four bits each require 28 billion bits, about 3.5 decimal gigabytes. At sixteen bits each, they require about 14 gigabytes. Both calculations cover weights alone.
A file can contain metadata with mixed numerical representations. Its 4-bit label may describe a quantization method whose complete file occupies more space than the simple calculation. llama.cpp's documentation gives a Llama 3.1 8B Q4_K_M file of about 4.9 GB, while describing differences in speed with quality across methods. llama.cpp's quantization documentation
That figure measures a particular file. Runtime memory needs a separate estimate before choosing hardware, with space left for the rest of the application after loading the weights.
Include the working conversation
Many runtimes store earlier attention information in a key-value cache. Reusing it avoids some repeated computation at each generation step. Its memory cost depends on the model architecture, with the context length affecting how much is stored. Cache representation also changes the allocation.
Hugging Face documents offloaded caches as one strategy. Quantized caches provide another. Moving data or changing its representation can affect latency, so the memory saving needs testing against the response the application requires. Hugging Face's cache strategies
A greeting tests a much smaller workload than a document question. The latter can include retrieved passages beside the document, with conversation history occupying more context. Space also needs to remain for the response. Test those inputs after establishing that the model loads.
I'd record the configured context length beside the actual benchmark input. That gives a result like "it ran on my laptop" enough detail to be compared with another run.
Test the intended concurrency
One request can respond normally while several compete for resources. An editor may share the local service with a background workflow, even if only one person is using the computer.
Ollama's FAQ connects parallel requests with additional context allocation. Its example uses four concurrent 2K-context requests, requiring the allocation for an equivalent 8K context. It also describes queuing when memory cannot accommodate the work. Ollama's concurrency guidance
A home service needs a test with the simultaneous jobs it will permit. Occasional chat may fit a configuration that needs a queue when an automated workflow begins using it too.
Limiting concurrency can reduce memory pressure without a hardware purchase. It also changes waiting time. An overnight job can tolerate a queue that would interrupt an assistant used during editing, so the limit needs to follow the workload.
Define an acceptable result
Start with an input representative of the task. A document answer needs to locate the correct passage before explaining it accurately. A coding task may require a change that passes the relevant checks. These conditions define which completed responses count as usable.
Measure the first useful output separately from total completion time. Record peak memory, then check the answer against the acceptance condition. Repeat with the expected longer input under intended concurrency. Reproduction also requires the model's quantization with the runtime version, plus its settings.
Ollama's context documentation explains that larger contexts need more memory. It points to ollama ps for observing context allocation with CPU/GPU placement. That output lets the placement be checked directly. Ollama's context guidance
Tokens per second needs the task's acceptance condition beside it. An incorrect fast answer still needs repair. A batch job may tolerate completion time that feels too slow during interactive use.
Inspect the full offline path
Local inference gives control over where the model runs. Search can still call an external service. Synchronization may send data elsewhere too, as may a connected application. Each component needs inspection when the requirement is offline operation.
Ollama's OLLAMA_NO_CLOUD=1 setting disables its cloud features after a restart. Other components in the application keep their own configuration. Ollama's local-only configuration
After downloading the necessary model files, test the application while disconnected. The result shows whether that path can complete without external services under the tested conditions.
Choose the workload before estimating the hardware requirement. Measure its complete allocation with the surrounding software running, then decide whether the existing machine can support it.