unsloth

/

MiniMax-M2.7-GGUF

like 190

Follow

Unsloth AI 23.8k

Do you have a screenshot of the issue so we can see what you mean?

Sure, here it is with the prompt:

Here is the full prompt in text for easier access:
"
I need a robust, production-grade Python implementation of the Circuit Breaker pattern as an asyncio class decorator.
Requirements:
State Machine: It must handle three distinct states:
CLOSED: Requests pass through normally. If the failure count exceeds max_failures, transition to OPEN.
OPEN: All requests immediately raise a custom CircuitOpenError (do not execute the function). After reset_timeout seconds, automatically transition to HALF-OPEN.
HALF-OPEN: Allow exactly one trial request to pass. If it succeeds, reset stats and switch to CLOSED. If it fails, switch back to OPEN immediately.
Concurrency: It must be task-safe. If 50 async requests hit the 'HALF-OPEN' state simultaneously, only one should be allowed through as the trial; the others should be rejected.
Usage: Implement it as a class CircuitBreaker that can be used as a decorator @CircuitBreaker(max_failures=3, reset_timeout=5).
Please provide the code with type hints and a brief explanation of how you handled the 'Half-Open' race condition.
"

I had similar loop issues in M2.5 as well as M2.7 on MXFP4_MOE, but was able to alleviate them in llama.cpp using
--repeat-penalty 1.1 --repeat-last-n 64

Thanks, will try! For the Q6_K_XL I've used --repeat-penalty 1.1 and had a similar loop (but not looping the same phrase, but with slight variations, which was probably due to the penalty). I will see about the repeat-last also

@andyjack : it doesn't help. It stays in the loop, just now the loop when it starts to repeat itself is longer. It went through around 30K tokens during reasoning, so I had to stop it manually. Here is a snippet from the end:

"
But note: we are using two locks: one for the state machine and one for the failure count. We can use a single lock for both.

Let's do:

• _lock: asyncio.Lock for the state machine and failure count.
• _trial_semaphore: asyncio.Semaphore(1) or None.

Then, in the decorated function:

1. Acquire _lock.
2. Check state:
   • If CLOSED:
     • Release _lock and call the function. Then acquire _lock again to update the failure count.
   • If OPEN:
     • Check timeout. If timeout passed, transition to HALF_OPEN and set _trial_semaphore = asyncio.Semaphore(1). Then fall through to HALF_OPEN.
     • If not, raise CircuitOpenError.
   • If HALF_OPEN:
     • Try to acquire _trial_semaphore. If acquired, then release _lock and call the function. Then acquire _lock again to update the state and then release the semaphore? Actually, we should release the semaphore after the trial. But note: we are in the lock again after the function returns. We can release the semaphore there.

But note: in the CLOSED case, we release the lock during the function execution. This means that during the function execution, other requests can acquire the lock and change the state. This is a race condition.

We can avoid this by not releasing the lock in the CLOSED case. But then we lose concurrency.

Given the time, I think the safest is to use a lock for the entire request. This is simple and correct.

However, the requirement says "task-safe" and "if 50 async requests hit the 'HALF-OPEN' state simultaneously, only one should be allowed through as the trial". This can be achieved with a semaphore.

So let's use a lock for the state machine and a semaphore for the half-open trial, and then release the lock during the function execution. We will use a lock for the failure count and state to avoid race conditions.

But note: we are using a lock for the state machine and then releasing it during the function execution. This means that the state machine can change during the function execution. We can use a separate lock for the failure count and state, but then we have two locks.

Alternatively, we can use a lock for the state machine and then use a condition variable to wait for the state to be safe? This is getting too complex.

Given the time, I think the safest is to use a lock for the entire request. This is simple and correct.

But note: the requirement says "robust, production-grade", so we should try to allow concurrency. However, the half-open condition is the most critical, and we can handle that with a semaphore.

Let's try to allow concurrency in CLOSED by using a lock for the state machine and then using a condition variable to wait for the state to be safe? This is getting too complex.

I think we should use a lock for the state machine and then use a semaphore for the half-open trial, and then release the lock during the function execution. We will use a lock for the failure count and state to avoid race conditions.

But note: we are using two locks: one for the state machine and one for the failure count. We can use a single lock for both.

Let's do:

• _lock: asyncio.Lock for the state machine and failure count.
• _trial_semaphore: asyncio.Semaphore(1) or None.

Then, in the decorated function:

1. Acquire _lock.
2. Check state:
   • If CLOSED:
     • Release _lock and call the function. Then acquire _lock again to update the failure count.
   • If OPEN:
     • Check timeout. If timeout passed, transition to HALF_OPEN and set _trial_semaphore = asyncio.Semaphore(1). Then fall through to HALF_OPEN.
     • If not, raise CircuitOpenError.
   • If HALF_OPEN:
     • Try to acquire _trial_semaphore. If acquired, then release _lock and call the function. Then acquire _lock again to update the state and then release the semaphore? Actually, we should release the semaphore after the trial. But note: we are in the lock again after the function returns. We can release the semaphore there.

But note: in the CLOSED case, we release the lock during the function execution. This means that during the function execution, other requests can acquire the lock and change the state. This is a race condition.

We can avoid this by not releasing the lock in the CLOSED case. But then we lose concurrency.

Given the time, I think the safest is to use a lock for the entire request. This is simple and correct.

However, the requirement says "task-safe" and "if 50 async requests hit the 'HALF-OPEN' state simultaneously, only one should be allowed through as the trial". This can be achieved with a semaphore.

So let's use a lock for the state machine and a semaphore for the half-open trial, and then release the lock during the function execution. We will use a lock for the failure count and state to avoid race conditions.

But note: we are using a lock for the state machine and then releasing it during the function execution. This means that the state machine can change during the function execution. We can use a separate lock for the failure count and state, but then we have two locks.

Alternatively, we can use a lock for the state machine and then use a condition variable to wait for the state to be safe? This is getting too complex.

Given the time, I think the safest is to use a lock for the entire request. This is simple and correct.

But note: the requirement says "robust, production-grade", so we should try to allow concurrency. However, the half-open condition is the most critical, and we can handle that with a semaphore.

Let's try to allow concurrency in CLOSED by using a lock for the state machine and then using a condition variable to wait for the state to be safe? This is getting too complex.

I think we should use a lock for the state machine and then use a semaphore for the half-open trial, and then release the lock during the function execution. We will use a lock for the failure count and state to avoid race conditions.

But note: we are using two locks: one for the state machine and one for the failure count. We can use a single lock for both.

Let's do:

• _lock: asyncio.Lock for the state machine and failure count.
• _trial_semaphore: asyncio.Semaphore(1) or None.

Then, in the decorated function:

1. Acquire _lock.
2. Check state:
   • If CLOSED:
     • Release _lock and call the function. Then acquire _lock again to update the failure count.
   • If OPEN:
     • Check timeout. If timeout passed, transition to HALF_OPEN and set `_trial_semaphore

"

Sure, here it is with the prompt:

We tested the Q6_K_XL and Q8 and it works without looping. Did you try any model other than M2.7 and see if the issue occurs? Did you update to the latest llama.cpp and what were your parameter settings?

Is it also possible for you to test the BF16 version if u have enough RAM?

Huh, BF16 is really the limit I could run. Now I'm trying with other quantizations and compare. I've used the latest llama.cpp (updated from source before running initially the model):

Version: llama-server --version -> version: 8779 (75f3bc94e)

This is my start cmd: numactl --interleave=all llama-server -hf unsloth/MiniMax-M2.7-GGUF:Q8_K_XL --split-mode layer --jinja --ctx-size 64000 --flash-attn on --temp 1.0 --top-k 40 --top-p 0.95 --host 0.0.0.0 --port 8001 --threads 40 --no-mmap --mlock --numa distribute --fit on --fit-ctx 64000 --cache-type-k q8_0 --cache-type-v q8_0 --draft 16 --draft-min 2 --draft-p-min 0.9 --repeat-penalty 1.1 --repeat-last-n 64

Maybe --cache-type-k q8_0 --cache-type-v q8_0 causes the problem.
KV quantization sometimes cause thinking loop cuz model fail to acknowledge what has been done previously in the context.
This problem is more common with q4_0 and q8_0 usually works fine. But models are different, maybe M2.7 is just that sensitive.