RthItalia/nano_compact_3b_qkvfp16

Nano Compact 3B QKV-FP16

RthItalia/nano_compact_3b_qkvfp16 is the validated compact self-contained variant derived from Qwen/Qwen2.5-3B-Instruct.

This release is not the old overlay artifact zip. It is the final exported Hugging Face folder that loads directly with transformers plus trust_remote_code=True.

What This Variant Is

This runtime uses a mixed policy:

• q_proj, k_proj, v_proj: fp16
• o_proj and most of the remaining body: Nano compact format
• model.embed_tokens: quantized single copy
• lm_head: tied custom head over the quantized embeddings

The goal of this policy is practical balance between:

• disk size
• VRAM footprint
• quality relative to the true 8bit baseline

Validated Runtime Envelope

• model size: 2.3432 GB
• allocated after load: 2.3432 GB
• peak generation VRAM: about 2.44 GB

True 8bit baseline used for comparison:

• allocated after load: 3.1703 GB
• peak generation VRAM: about 3.21 GB

Quality Claim

The quality claim for this release is intentionally narrow:

• it was compared against the true 8bit baseline on a small internal prompt suite
• it is not claimed to match the full original model in every task
• it is not claimed to outperform the base model

More aggressive variants reached better size or VRAM numbers, but failed the quality gate against the true 8bit reference.
qkvfp16 was the first variant that restored acceptable behavior on the validation smoke suite while preserving a substantial memory advantage.

How To Load

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "RthItalia/nano_compact_3b_qkvfp16"

tok = AutoTokenizer.from_pretrained(
    repo_id,
    use_fast=True,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    device_map="cuda",
    dtype=torch.float16,
).eval()

Example

messages = [
    {"role": "user", "content": "Explain what a neural network is in exactly 3 simple sentences."}
]

text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inp = tok(text, return_tensors="pt").to(next(model.parameters()).device)

with torch.no_grad():
    out = model.generate(
        **inp,
        max_new_tokens=120,
        do_sample=False,
        repetition_penalty=1.08,
        eos_token_id=tok.eos_token_id,
        pad_token_id=tok.eos_token_id,
    )

print(tok.decode(out[0][inp["input_ids"].shape[-1]:], skip_special_tokens=True))

Requirements

pip install torch transformers accelerate safetensors

bitsandbytes is not required for this final exported winner variant at runtime.

Important Notes

• trust_remote_code=True is required.
• The custom runtime uses a NanoTiedHead implementation that ties output logits to the quantized embedding table without registering the embedding module twice.
• Custom linear layers use chunked forward paths to keep peak VRAM under control.

Limitations

• Validation was narrow and engineering-driven, not a full benchmark suite.
• This release is specifically tuned around Qwen/Qwen2.5-3B-Instruct.
• It should be treated as a compact experimental runtime artifact, not as a general scientific proof of broader architectural claims.

License Note

This release should be described as a dual-license or dual-layer research distribution.

Built with Qwen.

The intended reading is:

• Qwen-derived materials retain the relevant Qwen Research License obligations
• Nano-specific runtime, packaging, and documentation changes are additional repository-authored research components

For redistributed copies, keep:

• LICENSE
• NOTICE
• clear indication of modified files where applicable

The Hugging Face metadata still stays at license: other because this is not accurately described by a single simple SPDX identifier.