What are all the files that are being downloaded?

Beginners
1

Any model i try to download ( let’s say gpt-oss-20b) actually downloads a looot of files and large sized files but the model itself is probably only 20GB and end up with 50+GB

hf download openai/gpt-oss-20b
Downloading ‘.gitattributes’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/52373fe24473b1aa44333d318f578ae6bf04b49b.incomplete’
.gitattributes: 1.57kB [00:00, 3.02MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/52373fe24473b1aa44333d318f578ae6bf04b49b
Downloading ‘LICENSE’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/7a4a3ea2424c09fbe48d455aed1eaa94d9124835.incomplete’
LICENSE: 11.4kB [00:00, 17.0MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/7a4a3ea2424c09fbe48d455aed1eaa94d9124835
Downloading ‘README.md’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/f6e25105129ce88f068afff463e0db36024e9ae8.incomplete’
README.md: 7.09kB [00:00, 13.5MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/f6e25105129ce88f068afff463e0db36024e9ae8
Downloading ‘USAGE_POLICY’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/b030f63aecc61cbaf2316a7b6401254f4312df74.incomplete’
USAGE_POLICY: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 781kB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/b030f63aecc61cbaf2316a7b6401254f4312df74
Downloading ‘chat_template.jinja’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/dc7bb11927d29f653ba2740f2db2c688fd77592f.incomplete’
chat_template.jinja: 16.7kB [00:00, 20.1MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/dc7bb11927d29f653ba2740f2db2c688fd77592f
Downloading ‘config.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/8fb5a4a03376ab5a12afd94b4ed93da61edf5b1c.incomplete’
config.json: 1.81kB [00:00, 4.53MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/8fb5a4a03376ab5a12afd94b4ed93da61edf5b1c
Downloading ‘generation_config.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/86f91466555bd40e3de0b1edee3d5d82f4ccdbfe.incomplete’
generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████| 177/177 [00:00<00:00, 569kB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/86f91466555bd40e3de0b1edee3d5d82f4ccdbfe
Downloading ‘metal/model.bin’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/725b7f996b5a226d922f034a731aefc398652f670dcafc2efd78620509a0fef1.incomplete’
metal/model.bin: 100%|████████████████████████████████████████████████████████████████████████████████████| 13.8G/13.8G [05:51<00:00, 39.1MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/725b7f996b5a226d922f034a731aefc398652f670dcafc2efd78620509a0fef1
Downloading ‘model-00000-of-00002.safetensors’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/16d0f997dcfc4462089d536bffe51b4bcea2f872f5c430be09ef8ed392312427.incomplete’
model-00000-of-00002.safetensors: 100%|███████████████████████████████████████████████████████████████████| 4.79G/4.79G [01:31<00:00, 52.4MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/16d0f997dcfc4462089d536bffe51b4bcea2f872f5c430be09ef8ed392312427
Downloading ‘model-00001-of-00002.safetensors’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/4fbe328ab445455d6f58dc73852b85873bd626986310abd91cd4d2ce3245eaea.incomplete’
model-00001-of-00002.safetensors: 100%|███████████████████████████████████████████████████████████████████| 4.80G/4.80G [02:14<00:00, 35.6MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/4fbe328ab445455d6f58dc73852b85873bd626986310abd91cd4d2ce3245eaea
Downloading ‘model-00002-of-00002.safetensors’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/a18106b209e9ab35c3406db4f6f12a927364a058b21e9d1373d682e20674b303.incomplete’
model-00002-of-00002.safetensors: 100%|████████████████████████████████████████████████████████████████████| 4.17G/4.17G [00:35<00:00, 117MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/a18106b209e9ab35c3406db4f6f12a927364a058b21e9d1373d682e20674b303
Downloading ‘model.safetensors.index.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/ae08521471ab125be4af84d0e51ecfc245830119.incomplete’
model.safetensors.index.json: 36.4kB [00:00, 50.2MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/ae08521471ab125be4af84d0e51ecfc245830119
Downloading ‘original/config.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/bd6a33e55b68b6734c932c44bd98f376d492dad9.incomplete’
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 376/376 [00:00<00:00, 1.40MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/bd6a33e55b68b6734c932c44bd98f376d492dad9
Downloading ‘original/dtypes.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/07e91493fc3ebf901d7d18b1309795b3f5e466aa.incomplete’
dtypes.json: 13.1kB [00:00, 27.9MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/07e91493fc3ebf901d7d18b1309795b3f5e466aa
Downloading ‘original/model.safetensors’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/3340a61d1a0391e8c5b5d3463d18d4c48129a84bbc04a554c762c99020aa06ed.incomplete’
original/model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████| 13.8G/13.8G [04:08<00:00, 55.4MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/3340a61d1a0391e8c5b5d3463d18d4c48129a84bbc04a554c762c99020aa06ed
Downloading ‘special_tokens_map.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/73bd12e55e2004cdfff088f85092b39dba2ccdd0.incomplete’
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████| 98.0/98.0 [00:00<00:00, 345kB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/73bd12e55e2004cdfff088f85092b39dba2ccdd0
Downloading ‘tokenizer.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/0614fe83cadab421296e664e1f48f4261fa8fef6e03e63bb75c20f38e37d07d3.incomplete’
tokenizer.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 27.9M/27.9M [00:00<00:00, 46.8MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/0614fe83cadab421296e664e1f48f4261fa8fef6e03e63bb75c20f38e37d07d3
Downloading ‘tokenizer_config.json’ to ‘/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/c021cddb0a9dd35b1bf83a9f145be2d9b3757891.incomplete’
tokenizer_config.json: 4.20kB [00:00, 9.87MB/s]
Download complete. Moving file to /workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/blobs/c021cddb0a9dd35b1bf83a9f145be2d9b3757891
/workspace/.cache/huggingface/hub/models–openai–gpt-oss-20b/snapshots/6cee5e81ee83917806bbde320786a8fb61efebee

1 Like
2

A common reason for extra storage consumption is caching.

However, with major models like openai/gpt-oss-20b, it’s common for separate files for multiple platforms to be stored within the repository. Downloading the entire repository means all of these get downloaded…

hf download openai/gpt-oss-20b downloads a full snapshot of the repo (i.e., every file in the model repo), not “just one set of weights”. For openai/gpt-oss-20b, the repo contains multiple full-weight artifacts (hence ~41.3GB total).

Files in openai/gpt-oss-20b

Repo root

  • .gitattributes
  • LICENSE
  • README.md
  • USAGE_POLICY
  • chat_template.jinja
  • config.json
  • generation_config.json
  • model-00000-of-00002.safetensors
  • model-00001-of-00002.safetensors
  • model-00002-of-00002.safetensors
  • model.safetensors.index.json
  • special_tokens_map.json
  • tokenizer.json
  • tokenizer_config.json (Hugging Face)

metal/

original/

  • original/config.json
  • original/dtypes.json
  • original/model.safetensors (13.8GB) (Hugging Face)

Why this becomes ~40–50GB

This repo includes three large “model-weight” payloads:

  • Sharded safetensors in the root (model-0000*-of-*.safetensors) totaling ~13.8GB (Hugging Face)
  • A single-file safetensors copy under original/model.safetensors (~13.8GB) (Hugging Face)
  • A precompiled Metal binary under metal/model.bin (~13.8GB) intended for Apple Metal runtimes (Hugging Face)

That’s already ~41.4GB before small metadata/tokenizer files, which matches the repo size shown on the “Files” tab (~41.3GB). (Hugging Face)

If you want to avoid downloading everything

Use --include/--exclude patterns. (Hugging Face)

Examples:

  • Download only the “original” weights (minimal set recommended in OpenAI’s gpt-oss repo docs):

    hf download openai/gpt-oss-20b --include "original/*" --local-dir gpt-oss-20b/

    (GitHub)

  • Download everything except the Metal and original copies (keep only the root sharded safetensors + configs/tokenizer):

    hf download openai/gpt-oss-20b --exclude "metal/*" --exclude "original/*"

    (Hugging Face)

3

thank you so much!!

1 Like
4

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.