Hi Hugging Face team and community,
We’re experiencing a critical production incident on our Space where Persistent Storage (/data) suddenly became unavailable, and as a result our SQLite vote database appears to be lost.
This Space is a Korean TTS Arena where users vote and compare models, so the vote history is essentially community-built data accumulated over time. Losing it is extremely serious for us.
Space
What changed?
As far as we know: nothing on our side.
-
We did not modify Space settings, secrets, storage configuration, or deploy code changes intentionally around the incident.
-
The app started behaving as if persistent storage was disabled/detached.
What we see in logs
On startup, the Space prints:
-
“Persistent storage (/data) not available. Using ‘instance/’ (data may be lost on restart)”
-
Then it tries to restore the DB from an HF dataset backup, but that now fails with 401 Unauthorized / Repository Not Found.
-
After that it attempts to create a new DB and crashes with sqlite3.OperationalError: unable to open database file.
Log excerpt (full logs available):
===== Application Startup at 2026-02-04 03:55:05 =====
Running in a Hugging Face Space 🤗
⚠️ Warning: Persistent storage (/data) not available. Using 'instance/' (data may be lost on restart)
Database not found at instance/tts_arena.db, downloading from HF dataset (channelcorp/ko-tts-arena-db)...
Error downloading database from HF dataset: 401 Client Error. (Request ID: Root=1-6982c32a-2d6a493653b785c766f63865;76865d47-0b52-4f3d-a901-766738eceb48)
Repository Not Found for url: https://huggingface.co/datasets/channelcorp/ko-tts-arena-db/resolve/main/tts_arena.db
Invalid username or password. ⚠️
A new database will be created.
...
sqlite3.OperationalError: unable to open database file
What we tried
-
SSH into the Space and searched the filesystem, but we could not find the old DB anywhere.
-
Since
/datais not mounted, anything previously stored there is currently inaccessible from the running instance.
Questions / help needed (urgent)
-
Why would Persistent Storage (/data) suddenly become unavailable for a Space without any setting change on our side?
-
Is there any way to restore/recover the previous persistent volume (snapshots, backups, internal recovery, etc.)?
-
Is this potentially related to billing/permissions, a platform incident, or a region/infrastructure issue?
-
Separately: why would our dataset backup download suddenly return 401 / repo not found?
- Could secrets/tokens have been reset, or did access permissions change?
If any HF staff can help escalate this, we’d really appreciate it — this dataset represents real user activity and we’re trying to avoid losing that history permanently.
Thank you.