[WIP] Spec infer with EAGLE2

Motivation
Accelerate the model inference by speculative inference (EAGLE2).

Modifications
It will be provided soon.

Checklist

[ ] Format your code according to the Contributor Guide.
[ ] Add unit tests as outlined in the Contributor Guide.
[ ] Update documentation as needed, including docstrings or example tutorials.

hello, whether this code supports the multiple request sepc?

Hi @yukavio Is there any recent progress or plan for this?

Hi @yukavio Is there any recent progress or plan for this? Do you plan to support deepseek-v2?

I have implemented the draft and verify stages and tested them on a single request.  I am trying to migrate my code to the main branch due to the main branch has some significant changes about the controller and worker which are very important for my implementation.
My plan:
Migrate code and test it: 1-2 days.
Implement remained code of single request speculative decoding:  half or one week.
Implement remained code of speculative decoding with batch: one or two week.

THX, yukavio.I have some suggestions for this pr: 1. further more support more models, e.g. i think we should pop the eagle head from draft_extend_input_queue so we dont modify the origin llama model file. 2.i dont understand why we need so many SpecInfoPipline queue, spec only in decoding stage,if we dont need the draft_extend_input_queue at least.

Also i have another question, in the pr model_runner.py init kv cache twice in different tpworker, this results in the oom in gpu, if  we merge the draft and tagret kv cache to increase the gpu utilization?

Also i have another question, in the pr model_runner.py init kv cache twice in different tpworker, this results in the oom in gpu, if we merge the draft and tagret kv cache to increase the gpu utilization?

I have migrated the code to another branch :https://github.com/yukavio/sglang/tree/new_spec_infer and I will update the code to this PR lately. In the new implementation, I choose to run the draft worker and target model worker in one process instead of using many queues in SpecInfoPipline to communicate with draft work process and target process.
For memory management, I've fixed this bug in the new branch to ensure it won't raise an error during testing. But it may not be very efficient and I will improve it after I have finish the remained work in the plan.

@yukavio Hi yukavio Recently, SGLang has undergone some refactoring work. You need to merge the latest main to resolve the corresponding conflicts. Thanks!

@yukavio Hi, when is this PR expected to be merged? I've trained a draft model and am eager to try it out.

@yukavio Hi yukavio Recently, SGLang has undergone some refactoring work. You need to merge the latest main to resolve the corresponding conflicts. Thanks!

OK, I am fixing some bugs in batch inference now. I will update the code to main branch after fixing them. Personally, I think the updated code can be used as the first version. The community could review this version of the implementation.

@yukavio Hi, when is this PR expected to be merged? I've trained a draft model and am eager to try it out.

If all goes well I will finish the first version of development this week. When to merge into the main branch depends on community review and opinions.

I have updated the code. The new implementation has support draft model inference with cuda graph / speculative inference with batch. It can been tested by run python examples/runtime/engine/offline_batch_inference.py.
TODO:

To support more sampling arguments. (Only support greedy now)
Migrate the code and fix conflict.
Performance improvement.

Write some kernel to fuse the element-wise operation in EAGLE Tree building to accelerate it.
Try to overlap the progress of model inference and EAGLE Tree building and verify progress.

@yukavio Is CLI startup not supported currently? I encountered this error:
File "/opt/tiger/sglang/python/sglang/srt/server_args.py", line 613, in <dictcomp>
return cls(**{attr: getattr(args, attr) for attr in attrs})
^^^^^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'draft_runner_cache_size'

@yukavio Please fix the conflicts

@yukavio Is CLI startup not supported currently? I encountered this error:
File "/opt/tiger/sglang/python/sglang/srt/server_args.py", line 613, in <dictcomp>
return cls(**{attr: getattr(args, attr) for attr in attrs})
^^^^^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'draft_runner_cache_size'

Did your locally installed code switch to this branch? This error doesn't look like it should occur. Because I have already added this variable in server args.
https://github.com/sgl-project/sglang/pull/1498/files#diff-700b5118b493d60d7b5994857f5f1e6a7e842ad702392b8ab199945764dfc8edR123
And you can update the code for this branch because I just fixed a memory leak bug in batch inference.

@yukavio Please fix the conflicts
OK, I am doing this.

@yukavio Is CLI startup not supported currently? I encountered this error:
File "/opt/tiger/sglang/python/sglang/srt/server_args.py", line 613, in <dictcomp>
return cls(**{attr: getattr(args, attr) for attr in attrs})
^^^^^^^^^^^^^^^^^^^
AttributeError: 'Namespace' object has no attribute 'draft_runner_cache_size'

Did your locally installed code switch to this branch? This error doesn't look like it should occur. Because I have already added this variable in server args. https://github.com/sgl-project/sglang/pull/1498/files#diff-700b5118b493d60d7b5994857f5f1e6a7e842ad702392b8ab199945764dfc8edR123
And you can update the code for this branch because I just fixed a memory leak bug in batch inference.

@yukavio  I am using the latest code. This is my startup script, is there anything wrong with it?:
python3 -m sglang.launch_server --model $LOCAL_PATH --stream-interval 1 --max-prefill-tokens 16384  --trust-remote-code  --mem-frac $MEM_FRAC --tp $TP_SIZE --dp $DP_SIZE --kv-cache-dtype fp8_e5m2 --port $PORT0  --mem-fraction-static $MEM_FRACTION_STATIC --schedule-conservativeness $SCHEDULE_CONSERVATIVENESS --context-length $MODEL_LEN  --chunked-prefill-size $CHUNKED_PREFILL_SIZE \
        --draft-model-path /opt/tiger/eagle --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE >> server.log 2>&1

python3 -m sglang.launch_server --model $LOCAL_PATH --stream-interval 1 --max-prefill-tokens 16384  --trust-remote-code  --mem-frac $MEM_FRAC --tp $TP_SIZE --dp $DP_SIZE --kv-cache-dtype fp8_e5m2 --port $PORT0  --mem-fraction-static $MEM_FRACTION_STATIC --schedule-conservativeness $SCHEDULE_CONSERVATIVENESS --context-length $MODEL_LEN  --chunked-prefill-size $CHUNKED_PREFILL_SIZE \
        --draft-model-path /opt/tiger/eagle --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE >> server.log 2>&1

Sorry, I haven't tested it as a service before. You can start the service with this command: python3 -m sglang.launch_server --model Llama-2-7b-chat-hf  --stream-interval 1 --max-prefill-tokens 16384  --trust-remote-code  --mem-frac 0.8 --tp 1 --dp 1   --mem-fraction-static 0.8   --draft-model-path EAGLE-llama2-chat-7B  --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE. However, I found that there seems to be some problems with the return value in this way. I will fix this error later. If you want to test, please run it first through the form in offline_batch_inference.py. This PR is not ready now. I'm doing more testing and fixing the bugs I find.

python3 -m sglang.launch_server --model $LOCAL_PATH --stream-interval 1 --max-prefill-tokens 16384  --trust-remote-code  --mem-frac $MEM_FRAC --tp $TP_SIZE --dp $DP_SIZE --kv-cache-dtype fp8_e5m2 --port $PORT0  --mem-fraction-static $MEM_FRACTION_STATIC --schedule-conservativeness $SCHEDULE_CONSERVATIVENESS --context-length $MODEL_LEN  --chunked-prefill-size $CHUNKED_PREFILL_SIZE \
        --draft-model-path /opt/tiger/eagle --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE >> server.log 2>&1

Sorry, I haven't tested it as a service before. You can Download the draft model(https://huggingface.co/yuhuili/EAGLE-llama2-chat-7B) first and edit the config (change architectures from "LlamaForCausalLM" to "LlamaForCausalLMEagle". After that, you can start the service with this command: python3 -m sglang.launch_server --model Llama-2-7b-chat-hf --stream-interval 1 --max-prefill-tokens 16384 --trust-remote-code --mem-frac 0.8 --tp 1 --dp 1 --mem-fraction-static 0.8 --draft-model-path EAGLE-llama2-chat-7B --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE
However, I found that there seems to be some problems with the return value in this way. I will fix this error later. If you want to test, please run it first through the form in offline_batch_inference.py.
This PR is not ready now. I'm doing more testing and fixing the bugs I find.

@yukavio
After solving the issue mentioned above, a new problem has emerged.
[15:16:50 TP4] Traceback (most recent call last):
File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1075, in run_scheduler_process
scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in init
self.tp_worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in init
self.model_runner = ModelRunner(
^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in init
self.init_cuda_graphs()
File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 501, in init_cuda_graphs
self.cuda_graph_runner = CudaGraphRunner(self)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 165, in init
self.capture()
File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 194, in capture
) = self.capture_one_batch_size(bs, num_token, forward)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 217, in capture_one_batch_size
self.model_runner.attn_backend.init_forward_metadata_capture_cuda_graph(
TypeError: TritonAttnBackend.init_forward_metadata_capture_cuda_graph() takes 4 positional arguments but 5 were given
/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/multiprocessing/resource_tracker.py:104: UserWarning: resource_tracker: process died unexpectedly, relaunching.  Some resources might leak.
warnings.warn('resource_tracker: process died unexpectedly, '
Traceback (most recent call last):
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/multiprocessing/resource_tracker.py", line 209, in main
cache[rtype].remove(name)
KeyError: '/mp-0aw9znwq'

python3 -m sglang.launch_server --model $LOCAL_PATH --stream-interval 1 --max-prefill-tokens 16384  --trust-remote-code  --mem-frac $MEM_FRAC --tp $TP_SIZE --dp $DP_SIZE --kv-cache-dtype fp8_e5m2 --port $PORT0  --mem-fraction-static $MEM_FRACTION_STATIC --schedule-conservativeness $SCHEDULE_CONSERVATIVENESS --context-length $MODEL_LEN  --chunked-prefill-size $CHUNKED_PREFILL_SIZE \
        --draft-model-path /opt/tiger/eagle --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE >> server.log 2>&1

Sorry, I haven't tested it as a service before. You can Download the draft model(https://huggingface.co/yuhuili/EAGLE-llama2-chat-7B) first and edit the config (change architectures from "LlamaForCausalLM" to "LlamaForCausalLMEagle". After that, you can start the service with this command: python3 -m sglang.launch_server --model Llama-2-7b-chat-hf --stream-interval 1 --max-prefill-tokens 16384 --trust-remote-code --mem-frac 0.8 --tp 1 --dp 1 --mem-fraction-static 0.8 --draft-model-path EAGLE-llama2-chat-7B --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE
However, I found that there seems to be some problems with the return value in this way. I will fix this error later. If you want to test, please run it first through the form in offline_batch_inference.py.
This PR is not ready now. I'm doing more testing and fixing the bugs I find.

@yukavio  When using the offline_batch_inference.py, I encountered the following error:
[23:23:54 TP7] Traceback (most recent call last):
File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1085, in run_scheduler_process
scheduler.event_loop()
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 272, in event_loop
self.run_step()
File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 459, in run_step
result = self.run_batch(new_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 669, in run_batch
logits_output, next_token_ids = self.draft_worker.forward_batch_speculative_generate(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 67, in forward_batch_speculative_generate
self.forward_draft_extend(batch)
File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 42, in forward_draft_extend
logits_output = self.model_runner.forward(forward_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 536, in forward
return self.forward_extend(forward_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 520, in forward_extend
return self.model.forward(
^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 328, in forward
hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 289, in forward
torch.cat(
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 44 but got size 1 for tensor number 1 in the list.

python3 -m sglang.launch_server --model $LOCAL_PATH --stream-interval 1 --max-prefill-tokens 16384  --trust-remote-code  --mem-frac $MEM_FRAC --tp $TP_SIZE --dp $DP_SIZE --kv-cache-dtype fp8_e5m2 --port $PORT0  --mem-fraction-static $MEM_FRACTION_STATIC --schedule-conservativeness $SCHEDULE_CONSERVATIVENESS --context-length $MODEL_LEN  --chunked-prefill-size $CHUNKED_PREFILL_SIZE \
        --draft-model-path /opt/tiger/eagle --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE >> server.log 2>&1

Sorry, I haven't tested it as a service before. You can Download the draft model(https://huggingface.co/yuhuili/EAGLE-llama2-chat-7B) first and edit the config (change architectures from "LlamaForCausalLM" to "LlamaForCausalLMEagle". After that, you can start the service with this command: python3 -m sglang.launch_server --model Llama-2-7b-chat-hf --stream-interval 1 --max-prefill-tokens 16384 --trust-remote-code --mem-frac 0.8 --tp 1 --dp 1 --mem-fraction-static 0.8 --draft-model-path EAGLE-llama2-chat-7B --num-speculative-steps 5 --eagle-topk 8 --num-draft-tokens 64 --speculative-algorithm EAGLE
However, I found that there seems to be some problems with the return value in this way. I will fix this error later. If you want to test, please run it first through the form in offline_batch_inference.py.
This PR is not ready now. I'm doing more testing and fixing the bugs I find.

@yukavio When using the offline_batch_inference.py, I encountered the following error:
[23:23:54 TP7] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1085, in run_scheduler_process
    scheduler.event_loop()
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 272, in event_loop
    self.run_step()
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 459, in run_step
    result = self.run_batch(new_batch)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 669, in run_batch
    logits_output, next_token_ids = self.draft_worker.forward_batch_speculative_generate(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 67, in forward_batch_speculative_generate
    self.forward_draft_extend(batch)
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 42, in forward_draft_extend
    logits_output = self.model_runner.forward(forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 536, in forward
    return self.forward_extend(forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 520, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 328, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 289, in forward
    torch.cat(
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 44 but got size 1 for tensor number 1 in the list.

Sorry, I'm fixing some bugs. This may cause the code to be temporarily unavailable, I will let you know here once I have fixed it and tested it.

Polite, may I ask if the build_tree function does not have a Triton version implementation！

Polite, may I ask if the build_tree function does not have a Triton version implementation！
This kernel is difficult to implement with triton because triton don't support control the behavior of cuda threads.

@fengyang95  you can pull the new code and run offline_batch_inference.py again.

@fengyang95 You can pull the new code and run offline_batch_inference.py again. It is my code to create Engine with EAGLE:
sgl.Engine(model_path="Llama-2-7b-chat-hf", draft_model_path='EAGLE-llama2-chat-7B', disable_cuda_graph=False,
                     num_speculative_steps=2, eagle_topk=2, num_draft_tokens=2,  speculative_algorithm='EAGLE', mem_fraction_static=0.75,
                     dtype='bfloat16')

You can change the argument of num_speculative_steps / eagle_topk / num_draft_tokens to get better performance. The eagle_topk and num_draft_tokens should be power of 2 and eagle_topk should be less than 10, num_draft_tokens should be less or equal to 64.

thanks, I will try it asap

@fengyang95 You can pull the new code and run offline_batch_inference.py again. It is my code to create Engine with EAGLE:
sgl.Engine(model_path="Llama-2-7b-chat-hf", draft_model_path='EAGLE-llama2-chat-7B', disable_cuda_graph=False,
                     num_speculative_steps=2, eagle_topk=2, num_draft_tokens=2,  speculative_algorithm='EAGLE', mem_fraction_static=0.75,
                     dtype='bfloat16')

You can change the argument of num_speculative_steps / eagle_topk / num_draft_tokens to get better performance. The eagle_topk and num_draft_tokens should be power of 2 and eagle_topk should be less than 10, num_draft_tokens should be less or equal to 64.

@yukavio thanks, I will try it asap. Additionally, is fp8 supported? From the code, it seems that FP8 is supported, although I'm uncertain about how to properly quantify the draft model.

I haven't considered quantification yet in my implementation and testing. But I think this shouldn't be a difficult problem to solve. I'm currently busy merging the master branch and resolving conflicts. I may have to look at this issue later. Or you can implement it by yourself if you want😄.
ps. The time-consuming ratio of draft model inference is very low. The benefits of quantifying this may be modest.

Currently, it has not been supported.  But I think it is not very difficult
to support it. Personally, I hope to implement these features step by step
after reviewing and merging this PR.  This PR is already too large, making
it difficult to maintain.
Yuanpeng Li @.***> 于2024年10月29日周二 17:05写道：

@fengyang95 https://github.com/fengyang95 You can pull the new code and
run offline_batch_inference.py again. It is my code to create Engine with
EAGLE:
sgl.Engine(model_path="Llama-2-7b-chat-hf", draft_model_path='EAGLE-llama2-chat-7B', disable_cuda_graph=False,
num_speculative_steps=2, eagle_topk=2, num_draft_tokens=2,  speculative_algorithm='EAGLE', mem_fraction_static=0.75,
dtype='bfloat16')
You can change the argument of num_speculative_steps / eagle_topk /
num_draft_tokens to get better performance. The eagle_topk and
num_draft_tokens should be power of 2 and eagle_topk should be less than
10, num_draft_tokens should be less or equal to 64.
@yukavio https://github.com/yukavio thanks, I will try it asap.
Additionally, is fp8 supported? From the code, it seems that FP8 is
supported, although I'm uncertain about how to properly quantify the draft
model.
I haven't considered quantification yet in my implementation and testing.
But I think this shouldn't be a difficult problem to solve. I'm currently
busy merging the master branch and resolving conflicts. I may have to look
at this issue later. Or you can implement it by yourself if you want😄. ps.
The time-consuming ratio of draft model inference is very low. The benefits
of quantifying this may be modest.
My target model is quantized, is there a way to keep the draft model in
bf16 while the target model remains quantized?
—
Reply to this email directly, view it on GitHub
https://github.com/sgl-project/sglang/pull/1498#issuecomment-2443633545,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQELBMJEDVKIQP2TQETTYZDZ55FT7AVCNFSM6AAAAABOXVKUIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBTGYZTGNJUGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

@fengyang95 You can pull the new code and run offline_batch_inference.py again. It is my code to create Engine with EAGLE:
sgl.Engine(model_path="Llama-2-7b-chat-hf", draft_model_path='EAGLE-llama2-chat-7B', disable_cuda_graph=False,
                     num_speculative_steps=2, eagle_topk=2, num_draft_tokens=2,  speculative_algorithm='EAGLE', mem_fraction_static=0.75,
                     dtype='bfloat16')

You can change the argument of num_speculative_steps / eagle_topk / num_draft_tokens to get better performance. The eagle_topk and num_draft_tokens should be power of 2 and eagle_topk should be less than 10, num_draft_tokens should be less or equal to 64.

@yukavio thanks, I will try it asap. Additionally, is fp8 supported? From the code, it seems that FP8 is supported, although I'm uncertain about how to properly quantify the draft model.

I haven't considered quantification yet in my implementation and testing. But I think this shouldn't be a difficult problem to solve. I'm currently busy merging the master branch and resolving conflicts. I may have to look at this issue later. Or you can implement it by yourself if you want😄. ps. The time-consuming ratio of draft model inference is very low. The benefits of quantifying this may be modest.

My target model is quantized, is there a way to keep the draft model in bf16 while the target model remains quantized?

Currently, it has not been supported.  But I think it is not very difficult
to support it. Personally, I hope to implement these features step by step
after reviewing and merging this PR.  This PR is already too large, making
it difficult to maintain.

@yukavio  I conducted tests on DeepSeek-v2 + eagle (which I trained myself, using the llama architecture for the AR layer) with 8 H20 cards.
My draft model can produce normal results using VLLM, indicating that there should be no issues with the model structure.
Here are the errors when using cuda graph:
 File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 143, in __init__
    self.model_runner.attn_backend.init_cuda_graph_state(self.max_num_token)
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 72, in init_cuda_graph_state
    self.cuda_graph_attn_logits = torch.empty(
                                  ^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 GiB. GPU 1 has a total capacity of 95.00 GiB of which 16.12 GiB is free. Process 1714898 has 78.87 GiB memory in use. Of the allocated memory 76.60 GiB is allocated by PyTorch, and 42.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

When not using cuda graph, the errors are as follows:
[17:28:42 TP6] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1092, in run_scheduler_process
    scheduler.event_loop()
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 274, in event_loop
    self.run_step()
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 461, in run_step
    result = self.run_batch(new_batch)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 674, in run_batch
    logits_output, next_token_ids = self.draft_worker.forward_batch_speculative_generate(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 69, in forward_batch_speculative_generate
    self.forward_draft_extend(batch)
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 44, in forward_draft_extend
    logits_output = self.model_runner.forward(forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 534, in forward
    return self.forward_extend(forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 518, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 328, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 289, in forward
    torch.cat(
TypeError: expected Tensor as element 1 in argument 0, but got NoneType

@yukavio I conducted tests on DeepSeek-v2 + eagle (which I trained myself, using the llama architecture for the AR layer) with 8 H20 cards. My draft model can produce normal results using VLLM, indicating that there should be no issues with the model structure.
Here are the errors when using cuda graph:
 File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 143, in __init__
    self.model_runner.attn_backend.init_cuda_graph_state(self.max_num_token)
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 72, in init_cuda_graph_state
    self.cuda_graph_attn_logits = torch.empty(
                                  ^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 GiB. GPU 1 has a total capacity of 95.00 GiB of which 16.12 GiB is free. Process 1714898 has 78.87 GiB memory in use. Of the allocated memory 76.60 GiB is allocated by PyTorch, and 42.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

When not using cuda graph, the errors are as follows:
[17:28:42 TP6] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1092, in run_scheduler_process
    scheduler.event_loop()
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 274, in event_loop
    self.run_step()
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 461, in run_step
    result = self.run_batch(new_batch)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 674, in run_batch
    logits_output, next_token_ids = self.draft_worker.forward_batch_speculative_generate(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 69, in forward_batch_speculative_generate
    self.forward_draft_extend(batch)
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 44, in forward_draft_extend
    logits_output = self.model_runner.forward(forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 534, in forward
    return self.forward_extend(forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 518, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 328, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 289, in forward
    torch.cat(
TypeError: expected Tensor as element 1 in argument 0, but got NoneType

For the first case, you should try to use flash-infer as attention backend to fix it.
For the last case, It sames your target model is failed to capture the hidden states. You can add code like to your target model (DeepSeek-V2 or other) like: https://github.com/sgl-project/sglang/pull/1498/files#diff-f6f7943965e41f2d4081018071c87bc1e9f806d5d639579688eb5f6c02f250cdR320.
I will refine this implementation later to avoid the edit of model implementation.

I am using the latest code for offline testing。
At the beginning of the cycle，tps = 180 tokens/s，
But after a few cycles, there will be a serious decrease in speed，tps 180 tokens/s -> 57 tokens/s -> 38 tokens/s
import sglang as sgl
import time
import json

def main():
    # Sample prompts.
    prompts = [
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Where is the capital city of France? ASSISTANT:",
        "[INST] <<SYS>>\\nYou are a helpful assistant.\\n<</SYS>>\\n如何赚取 10000w？ [/INST]"
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: 你是谁呢？ ASSISTANT:"
    ]
    # Create a sampling params object.
    sampling_params = {"temperature": 0, "max_new_tokens": 256,}

    draft_model_path = "/mnt/data/model_hub/EAGLE-llama2-chat-7B"
    model_path = "/mnt/data/model_hub/Llama-2-7b-chat-hf"
    # Create an LLM.
    llm = sgl.Engine(model_path=model_path, draft_model_path=draft_model_path, disable_cuda_graph=False, num_speculative_steps=4, eagle_topk=4, num_draft_tokens=32, speculative_algorithm='EAGLE', mem_fraction_static=0.60)
    # llm = sgl.Engine(model_path=model_path, disable_cuda_graph=False, mem_fraction_static=0.60)
    #outputs = llm.generate(prompts, sampling_params)

    for _ in range(100):
        start = time.time()
        outputs = llm.generate(prompts, sampling_params)
        cos = time.time()-start
        # print(f"!!!! {json.dumps(outputs)}")
        completion_tokens = outputs[0]["meta_info"]["completion_tokens"]
        # Print the outputs.
        for prompt, output in zip(prompts, outputs):
            print(f"!!!!!!!!! tps =: {completion_tokens/cos}")

# The __main__ condition is necessary here because we use "spawn" to create subprocesses
# Spawn starts a fresh program every time, if there is no __main__, it will run into infinite loop to keep spawning processes from sgl.Engine
if __name__ == "__main__":
    main()

I am using the latest code for offline testing。 At the beginning of the cycle，tps = 180 tokens/s， But after a few cycles, there will be a serious decrease in speed，tps 180 tokens/s -> 57 tokens/s -> 38 tokens/s
import sglang as sgl
import time
import json

def main():
    # Sample prompts.
    prompts = [
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Where is the capital city of France? ASSISTANT:",
        "[INST] <<SYS>>\\nYou are a helpful assistant.\\n<</SYS>>\\n如何赚取 10000w？ [/INST]"
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: 你是谁呢？ ASSISTANT:"
    ]
    # Create a sampling params object.
    sampling_params = {"temperature": 0, "max_new_tokens": 256,}

    draft_model_path = "/mnt/data/model_hub/EAGLE-llama2-chat-7B"
    model_path = "/mnt/data/model_hub/Llama-2-7b-chat-hf"
    # Create an LLM.
    llm = sgl.Engine(model_path=model_path, draft_model_path=draft_model_path, disable_cuda_graph=False, num_speculative_steps=4, eagle_topk=4, num_draft_tokens=32, speculative_algorithm='EAGLE', mem_fraction_static=0.60)
    # llm = sgl.Engine(model_path=model_path, disable_cuda_graph=False, mem_fraction_static=0.60)
    #outputs = llm.generate(prompts, sampling_params)

    for _ in range(100):
        start = time.time()
        outputs = llm.generate(prompts, sampling_params)
        cos = time.time()-start
        # print(f"!!!! {json.dumps(outputs)}")
        completion_tokens = outputs[0]["meta_info"]["completion_tokens"]
        # Print the outputs.
        for prompt, output in zip(prompts, outputs):
            print(f"!!!!!!!!! tps =: {completion_tokens/cos}")

# The __main__ condition is necessary here because we use "spawn" to create subprocesses
# Spawn starts a fresh program every time, if there is no __main__, it will run into infinite loop to keep spawning processes from sgl.Engine
if __name__ == "__main__":
    main()

Thanks for the report. I will check and fix it later.

@yukavio I conducted tests on DeepSeek-v2 + eagle (which I trained myself, using the llama architecture for the AR layer) with 8 H20 cards. My draft model can produce normal results using VLLM, indicating that there should be no issues with the model structure.
Here are the errors when using cuda graph:
 File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 143, in __init__
    self.model_runner.attn_backend.init_cuda_graph_state(self.max_num_token)
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 72, in init_cuda_graph_state
    self.cuda_graph_attn_logits = torch.empty(
                                  ^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 GiB. GPU 1 has a total capacity of 95.00 GiB of which 16.12 GiB is free. Process 1714898 has 78.87 GiB memory in use. Of the allocated memory 76.60 GiB is allocated by PyTorch, and 42.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

When not using cuda graph, the errors are as follows:
[17:28:42 TP6] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1092, in run_scheduler_process
    scheduler.event_loop()
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 274, in event_loop
    self.run_step()
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 461, in run_step
    result = self.run_batch(new_batch)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 674, in run_batch
    logits_output, next_token_ids = self.draft_worker.forward_batch_speculative_generate(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 69, in forward_batch_speculative_generate
    self.forward_draft_extend(batch)
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 44, in forward_draft_extend
    logits_output = self.model_runner.forward(forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 534, in forward
    return self.forward_extend(forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 518, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 328, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 289, in forward
    torch.cat(
TypeError: expected Tensor as element 1 in argument 0, but got NoneType

For the first case, you should try to use flash-infer as attention backend to fix it.
For the last case, It sames your target model is failed to capture the hidden states. You can add code like to your target model (DeepSeek-V2 or other) like: https://github.com/sgl-project/sglang/pull/1498/files#diff-f6f7943965e41f2d4081018071c87bc1e9f806d5d639579688eb5f6c02f250cdR320. I will refine this implementation later to avoid the editing of model implementation.

@yukavio got another issue:
[19:16:21 TP5] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1090, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 167, in __init__
    self.capture()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 196, in capture
    ) = self.capture_one_batch_size(bs, num_token, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 255, in capture_one_batch_size
    run_once(self.capture_forward_mode)
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 249, in run_once
    return forward(input_ids, forward_batch.positions, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 662, in forward
    hidden_states = self.model(input_ids, positions, forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 631, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 578, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 473, in forward
    attn_output = self.attn(q_input, k_input, v_input, forward_batch)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/radix_attention.py", line 60, in forward
    return forward_batch.attn_backend.forward(q, k, v, self, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/__init__.py", line 39, in forward
    return self.forward_extend(q, k, v, layer, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 112, in forward_extend
    self.extend_attention_fwd(
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 308, in extend_attention_fwd
    grid = (batch_size, head_num, triton.cdiv(max_len_extend, BLOCK_M))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/triton/__init__.py", line 60, in cdiv
    return (x + y - 1) // y
            ~~^~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

FYI. Due to the large number of modifications to the attention backend part of the main branch. I need some time to fix the conflicts caused by code migration.

I am using the latest code for offline testing。 At the beginning of the cycle，tps = 180 tokens/s， But after a few cycles, there will be a serious decrease in speed，tps 180 tokens/s -> 57 tokens/s -> 38 tokens/s
import sglang as sgl
import time
import json

def main():
    # Sample prompts.
    prompts = [
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Where is the capital city of France? ASSISTANT:",
        "[INST] <<SYS>>\\nYou are a helpful assistant.\\n<</SYS>>\\n如何赚取 10000w？ [/INST]"
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: 你是谁呢？ ASSISTANT:"
    ]
    # Create a sampling params object.
    sampling_params = {"temperature": 0, "max_new_tokens": 256,}

    draft_model_path = "/mnt/data/model_hub/EAGLE-llama2-chat-7B"
    model_path = "/mnt/data/model_hub/Llama-2-7b-chat-hf"
    # Create an LLM.
    llm = sgl.Engine(model_path=model_path, draft_model_path=draft_model_path, disable_cuda_graph=False, num_speculative_steps=4, eagle_topk=4, num_draft_tokens=32, speculative_algorithm='EAGLE', mem_fraction_static=0.60)
    # llm = sgl.Engine(model_path=model_path, disable_cuda_graph=False, mem_fraction_static=0.60)
    #outputs = llm.generate(prompts, sampling_params)

    for _ in range(100):
        start = time.time()
        outputs = llm.generate(prompts, sampling_params)
        cos = time.time()-start
        # print(f"!!!! {json.dumps(outputs)}")
        completion_tokens = outputs[0]["meta_info"]["completion_tokens"]
        # Print the outputs.
        for prompt, output in zip(prompts, outputs):
            print(f"!!!!!!!!! tps =: {completion_tokens/cos}")

# The __main__ condition is necessary here because we use "spawn" to create subprocesses
# Spawn starts a fresh program every time, if there is no __main__, it will run into infinite loop to keep spawning processes from sgl.Engine
if __name__ == "__main__":
    main()

I have fixed it. You can update the code and rerun it.

@yukavio I conducted tests on DeepSeek-v2 + eagle (which I trained myself, using the llama architecture for the AR layer) with 8 H20 cards. My draft model can produce normal results using VLLM, indicating that there should be no issues with the model structure.
Here are the errors when using cuda graph:
 File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 143, in __init__
    self.model_runner.attn_backend.init_cuda_graph_state(self.max_num_token)
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 72, in init_cuda_graph_state
    self.cuda_graph_attn_logits = torch.empty(
                                  ^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 GiB. GPU 1 has a total capacity of 95.00 GiB of which 16.12 GiB is free. Process 1714898 has 78.87 GiB memory in use. Of the allocated memory 76.60 GiB is allocated by PyTorch, and 42.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

When not using cuda graph, the errors are as follows:
[17:28:42 TP6] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1092, in run_scheduler_process
    scheduler.event_loop()
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 274, in event_loop
    self.run_step()
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 461, in run_step
    result = self.run_batch(new_batch)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 674, in run_batch
    logits_output, next_token_ids = self.draft_worker.forward_batch_speculative_generate(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 69, in forward_batch_speculative_generate
    self.forward_draft_extend(batch)
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 44, in forward_draft_extend
    logits_output = self.model_runner.forward(forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 534, in forward
    return self.forward_extend(forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 518, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 328, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 289, in forward
    torch.cat(
TypeError: expected Tensor as element 1 in argument 0, but got NoneType

For the first case, you should try to use flash-infer as attention backend to fix it.
For the last case, It sames your target model is failed to capture the hidden states. You can add code like to your target model (DeepSeek-V2 or other) like: https://github.com/sgl-project/sglang/pull/1498/files#diff-f6f7943965e41f2d4081018071c87bc1e9f806d5d639579688eb5f6c02f250cdR320. I will refine this implementation later to avoid the editing of model implementation.

@yukavio got another issue:
[19:16:21 TP5] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1090, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 167, in __init__
    self.capture()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 196, in capture
    ) = self.capture_one_batch_size(bs, num_token, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 255, in capture_one_batch_size
    run_once(self.capture_forward_mode)
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 249, in run_once
    return forward(input_ids, forward_batch.positions, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 662, in forward
    hidden_states = self.model(input_ids, positions, forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 631, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 578, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 473, in forward
    attn_output = self.attn(q_input, k_input, v_input, forward_batch)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/radix_attention.py", line 60, in forward
    return forward_batch.attn_backend.forward(q, k, v, self, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/__init__.py", line 39, in forward
    return self.forward_extend(q, k, v, layer, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 112, in forward_extend
    self.extend_attention_fwd(
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 308, in extend_attention_fwd
    grid = (batch_size, head_num, triton.cdiv(max_len_extend, BLOCK_M))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/triton/__init__.py", line 60, in cdiv
    return (x + y - 1) // y
            ~~^~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

It looks like you are using triton_backend. EAGLE only supports the flash-infer backend.

@yukavio Hi yukavio Recently, SGLang has undergone some refactoring work. You need to merge the latest main to resolve the corresponding conflicts. Thanks!

I have

@yukavio Hi yukavio Recently, SGLang has undergone some refactoring work. You need to merge the latest main to resolve the corresponding conflicts. Thanks!

@yukavio Hi yukavio Recently, SGLang has undergone some refactoring work. You need to merge the latest main to resolve the corresponding conflicts. Thanks!

I have merged the main branch and fixed the corresponding conflicts. Do you know if I need to add a corresponding unit test? If necessary, EAGLE's tests need to rely on the specific model. Please tell me how to add dependent models.

I am using the latest code for offline testing。 At the beginning of the cycle，tps = 180 tokens/s， But after a few cycles, there will be a serious decrease in speed，tps 180 tokens/s -> 57 tokens/s -> 38 tokens/s
import sglang as sgl
import time
import json

def main():
    # Sample prompts.
    prompts = [
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Where is the capital city of France? ASSISTANT:",
        "[INST] <<SYS>>\\nYou are a helpful assistant.\\n<</SYS>>\\n如何赚取 10000w？ [/INST]"
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: 你是谁呢？ ASSISTANT:"
    ]
    # Create a sampling params object.
    sampling_params = {"temperature": 0, "max_new_tokens": 256,}

    draft_model_path = "/mnt/data/model_hub/EAGLE-llama2-chat-7B"
    model_path = "/mnt/data/model_hub/Llama-2-7b-chat-hf"
    # Create an LLM.
    llm = sgl.Engine(model_path=model_path, draft_model_path=draft_model_path, disable_cuda_graph=False, num_speculative_steps=4, eagle_topk=4, num_draft_tokens=32, speculative_algorithm='EAGLE', mem_fraction_static=0.60)
    # llm = sgl.Engine(model_path=model_path, disable_cuda_graph=False, mem_fraction_static=0.60)
    #outputs = llm.generate(prompts, sampling_params)

    for _ in range(100):
        start = time.time()
        outputs = llm.generate(prompts, sampling_params)
        cos = time.time()-start
        # print(f"!!!! {json.dumps(outputs)}")
        completion_tokens = outputs[0]["meta_info"]["completion_tokens"]
        # Print the outputs.
        for prompt, output in zip(prompts, outputs):
            print(f"!!!!!!!!! tps =: {completion_tokens/cos}")

# The __main__ condition is necessary here because we use "spawn" to create subprocesses
# Spawn starts a fresh program every time, if there is no __main__, it will run into infinite loop to keep spawning processes from sgl.Engine
if __name__ == "__main__":
    main()

Thanks for the report. I will check and fix it later.

if set disable_radix_cache= True,The speed of 1000 cycles is very stable!

one issue is EAGLE cannot support the use of certain sampling parameters, such as repetition_penalty.

I am using the latest code for offline testing。 At the beginning of the cycle，tps = 180 tokens/s， But after a few cycles, there will be a serious decrease in speed，tps 180 tokens/s -> 57 tokens/s -> 38 tokens/s
import sglang as sgl
import time
import json

def main():
    # Sample prompts.
    prompts = [
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Where is the capital city of France? ASSISTANT:",
        "[INST] <<SYS>>\\nYou are a helpful assistant.\\n<</SYS>>\\n如何赚取 10000w？ [/INST]"
        # "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: 你是谁呢？ ASSISTANT:"
    ]
    # Create a sampling params object.
    sampling_params = {"temperature": 0, "max_new_tokens": 256,}

    draft_model_path = "/mnt/data/model_hub/EAGLE-llama2-chat-7B"
    model_path = "/mnt/data/model_hub/Llama-2-7b-chat-hf"
    # Create an LLM.
    llm = sgl.Engine(model_path=model_path, draft_model_path=draft_model_path, disable_cuda_graph=False, num_speculative_steps=4, eagle_topk=4, num_draft_tokens=32, speculative_algorithm='EAGLE', mem_fraction_static=0.60)
    # llm = sgl.Engine(model_path=model_path, disable_cuda_graph=False, mem_fraction_static=0.60)
    #outputs = llm.generate(prompts, sampling_params)

    for _ in range(100):
        start = time.time()
        outputs = llm.generate(prompts, sampling_params)
        cos = time.time()-start
        # print(f"!!!! {json.dumps(outputs)}")
        completion_tokens = outputs[0]["meta_info"]["completion_tokens"]
        # Print the outputs.
        for prompt, output in zip(prompts, outputs):
            print(f"!!!!!!!!! tps =: {completion_tokens/cos}")

# The __main__ condition is necessary here because we use "spawn" to create subprocesses
# Spawn starts a fresh program every time, if there is no __main__, it will run into infinite loop to keep spawning processes from sgl.Engine
if __name__ == "__main__":
    main()

Thanks for the report. I will check and fix it later.

if set disable_radix_cache= True,The speed of 1000 cycles is very stable!

It is due to the draft model not supporting prefix share now. I am trying to support it.

one issue is EAGLE cannot support the use of certain sampling parameters, such as repetition_penalty.
Yes, but I personally have no plans to implement this feature in this PR at this time.

@yukavio I conducted tests on DeepSeek-v2 + eagle (which I trained myself, using the llama architecture for the AR layer) with 8 H20 cards. My draft model can produce normal results using VLLM, indicating that there should be no issues with the model structure.
Here are the errors when using cuda graph:
 File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 143, in __init__
    self.model_runner.attn_backend.init_cuda_graph_state(self.max_num_token)
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 72, in init_cuda_graph_state
    self.cuda_graph_attn_logits = torch.empty(
                                  ^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 GiB. GPU 1 has a total capacity of 95.00 GiB of which 16.12 GiB is free. Process 1714898 has 78.87 GiB memory in use. Of the allocated memory 76.60 GiB is allocated by PyTorch, and 42.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

When not using cuda graph, the errors are as follows:
[17:28:42 TP6] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1092, in run_scheduler_process
    scheduler.event_loop()
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 274, in event_loop
    self.run_step()
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 461, in run_step
    result = self.run_batch(new_batch)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 674, in run_batch
    logits_output, next_token_ids = self.draft_worker.forward_batch_speculative_generate(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 69, in forward_batch_speculative_generate
    self.forward_draft_extend(batch)
  File "/opt/tiger/sglang/python/sglang/srt/speculative/eagle_worker.py", line 44, in forward_draft_extend
    logits_output = self.model_runner.forward(forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 534, in forward
    return self.forward_extend(forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 518, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 328, in forward
    hidden_states = self.model(input_ids, positions, forward_batch, input_embeds)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/llama_eagle.py", line 289, in forward
    torch.cat(
TypeError: expected Tensor as element 1 in argument 0, but got NoneType

For the first case, you should try to use flash-infer as attention backend to fix it.
For the last case, It sames your target model is failed to capture the hidden states. You can add code like to your target model (DeepSeek-V2 or other) like: https://github.com/sgl-project/sglang/pull/1498/files#diff-f6f7943965e41f2d4081018071c87bc1e9f806d5d639579688eb5f6c02f250cdR320. I will refine this implementation later to avoid the editing of model implementation.

@yukavio got another issue:
[19:16:21 TP5] Traceback (most recent call last):
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 1090, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/scheduler.py", line 143, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/managers/tp_worker.py", line 62, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 141, in __init__
    self.init_cuda_graphs()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/model_runner.py", line 504, in init_cuda_graphs
    self.cuda_graph_runner = CudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 167, in __init__
    self.capture()
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 196, in capture
    ) = self.capture_one_batch_size(bs, num_token, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 255, in capture_one_batch_size
    run_once(self.capture_forward_mode)
  File "/opt/tiger/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 249, in run_once
    return forward(input_ids, forward_batch.positions, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 662, in forward
    hidden_states = self.model(input_ids, positions, forward_batch)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 631, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 578, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/models/deepseek_v2.py", line 473, in forward
    attn_output = self.attn(q_input, k_input, v_input, forward_batch)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/radix_attention.py", line 60, in forward
    return forward_batch.attn_backend.forward(q, k, v, self, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/__init__.py", line 39, in forward
    return self.forward_extend(q, k, v, layer, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_backend.py", line 112, in forward_extend
    self.extend_attention_fwd(
  File "/opt/tiger/sglang/python/sglang/srt/layers/attention/triton_ops/extend_attention.py", line 308, in extend_attention_fwd
    grid = (batch_size, head_num, triton.cdiv(max_len_extend, BLOCK_M))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/triton/__init__.py", line 60, in cdiv
    return (x + y - 1) // y
            ~~^~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

It looks like you are using triton_backend. EAGLE only supports the flash-infer backend.

@zhyncs hi, is MLA currently only supporting triton_backend? Does that mean Eagle and MLA cannot be used together?

Add a memo: cutex should be added to dependency list of Sglang after review.

@yukavio Can you resolve the conflicts?

@yukavio Can you resolve the conflicts?

Fixed. Could you please help me to trigger the CI?

CI has failed due to timeout.


Inform python version

Since python 3 is very popular right now, it would be helpful to inform the supported python version.

You're right, would be nice if you could do it for us, I think that keep this information at the README.md file, what you think?

Done: https://github.com/globocom/glog-cli/pull/7


Suggestions on Save Persistence

Proposal
Can different save times be allowed for different job-name

You can set a scrape interval for each job.


Getting TypeError: jQuery.ui is undefined after recent releases

I am now getting jQuery.ui on my collections. Culprit seems to be this PR #90

Ah, sorry mate.
Will provide a fix asap.

Same issue here!
What is the version we need to revert to while you fix this? :)

I just fixed it, going to release a new version.
A

Done, you can upgrade to 2.1.18
Have a good day!

'm stuck with 2.1.17 :(

@ninsuo can you put 2.1.18 as stable release? I cannot upgrade to it as it is now.

Humm, it is a stable release :thinking:
Which package manager are you using?

@ninsuo well when I do a composer update, it's stuck on 2.1.17.
If I update my composer.json to  "ninsuo/symfony-collection": "^2.1.18", I get this error
- The requested package ninsuo/symfony-collection ^2.1.18 exists as ninsuo/symfony-collection[1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.1.0, 2.1.1, 2.1.10, 2.1.11, 2
.1.12, 2.1.13, 2.1.14, 2.1.15, 2.1.16, 2.1.17, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.1.9, dev-master] but these are rejected by your constraint.

I am sorry but I don't understand...
alain@iMac /tmp/test
$ composer require ninsuo/symfony-collection
Using version ^2.1 for ninsuo/symfony-collection
./composer.json has been created
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing ninsuo/symfony-collection (2.1.18): Downloading (100%)
Writing lock file
Generating autoload files
alain@iMac /tmp/test

Can you try to remove your cache (empty directory returned by composer config cache-dir)?

Now it's OK... very strange, it's the first time I have to do this in years haha

Thanks a lot @ninsuo works great!

Yep, thanks :)


Added goreleaser changelog groups and update PR template

Description of Changes

Updated goreleaser with changelog groups
Updated PR template to include instructions and remove changelog checkbox.

Please check that the PR fulfills these requirements

[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)
[ ] Add a changelog entry (for non-trivial bug fixes / features)
[ ] CI passes

What's our plan with the changelog file? Seems redundant.

I agree. Probably add a big deprecated note at the top but keep it to preserve history.

I forgot to mention, Dependabot can be told how it should name and label pull requests https://github.com/observIQ/iris/pull/209/files

I forgot to mention, Dependabot can be told how it should name and label pull requests https://github.com/observIQ/iris/pull/209/files

I didn't know that I'll update.

CI failures are unrelated to changes going to merge.


Add MaybeUninit type

The standard library's MaybeUninit type does not currently support wrapping unsized types. This commit introduces a polyfill with the same behavior as MaybeUninit which does support wrapping unsized types.
In this commit, the only supported types are sized types and slice types. Later (as part of #29), we will add the ability to derive the AsMaybeUninit trait, which will extend support to custom DSTs.
Makes progress on #29

This was accidentally closed when renaming a branch. It's reopened in https://github.com/google/zerocopy/pull/319.


Simplify _unique_internal

Was retrieving the indices found by unique. However as those are not used, it makes no sense to get them. Hence we drop requesting that result from unique and simplify the surrounding code in _unique_internal.

Also set dtypes a bit more explicitly. Didn't seem like we had issues without dtypes being explicitly set, but there could have been conversions happening behind the scenes.

No objections from me

Ok to merge?


Responsiveness and media queries

Thank you for very good mixins.
Could you help me please. I would like to have responsive grid.
But i don't see any media queries. How I could add it?

flexiblegs-*-plus is the core. There is no need to breakpoint. You can write it yourself.
Example:
// xl
.products {
  .wrap;
  .wrap(gutter,40);
  .wrap(5);
  &__item {
    .col;
  }
}

// lg
@media (max-width: 1024px) {
  .products {
    .wrap(gutter,24);
    .wrap(4);
  }
}

// md
@media (max-width: 768px) {
  .products {
    .wrap(gutter,16);
    .wrap(3);
  }
}

// sm
@media (max-width: 667px) {
  .products {
    .wrap(gutter,8);
    .wrap(2);
  }
}

01

02

03

04

05

My english is really bad. I'm sorry for it :(

Thank you for your feedback. I actually did the same approach. I have decided, that responsiveness should be carefully planned. Any way thanks for good job.

:+1:


minimal embeded mruby

I have very small program.
How I can producing minimal execution file? I'm no need stdio, file access, random mrbgems/mruby-random, print, time etc. Now I get 744kB exec file.
My mruby repo is in ~/mruby
Now I compiling my code:
gcc -Wall -pedantic -O0 -I ~/mruby/include/ -I ~/mruby/include/mruby \
   t.c ~/mruby/build/host/lib/libmruby.a -o t -lm

How disabling STDIO? I uncomment
#define MRB_DISABLE_STDIO  line in ~/mruby/include/mrbconf.h but mruby not compilig.
CC    mrbgems/mruby-bin-mrbc/tools/mrbc/mrbc.c -> build/host/mrbgems/mruby-bin-mrbc/tools/mrbc/mrbc.o
mrbc.c:175:10: error: no member named 'f' in 'struct mrb_parser_state'
if (p->f) fclose(p->f);

If I link all *.o files i get error not found readline and add_history

Building mruby is a two-step process. You first need a minimal environment and a working mrbc binary to compile standard library and other gems. Having that in place, you rebuild your host and all other configurations. But the mrbc binary needs file IO so you can't disable it completely in mrbconf.h. Instead, you should create a new configuration in build_config.rb. There, you can disable all gems and features like IO, floating point number and so on. Bear in mind that you won't have any standard mruby binary without IO, so you'll need to compile your .rb scripts with host tools and use "minified" version as a library with precompiled mruby scripts only.

less build_config.rb | grep default
include the default GEMs
conf.gembox 'default'
include the default GEMs
conf.gembox 'default'
conf.gembox 'default'
conf.gembox 'default'
Where i must add/remowe to create my own small mruby?
Can You write any example? I use only unices system.

You can start by adding this to your build_config.rb:
MRuby::Build.new('mini') do |conf|
    toolchain :gcc
    conf.cc.flags << '-Os'
    conf.cc.defines = %w(MRB_DISABLE_STDIO MRB_WITHOUT_FLOAT)
end

Wow! minimal 1786084 regular host 9535240
I can disable ruby gems in conf.gembox ''
for example time and random ?

There are no gems enabled in the mini configuration I've posted. You must enable them manually, eg:
MRuby::Build.new('mini-with-class-ext') do |conf|
    toolchain :gcc
    conf.cc.flags << '-Os'
    conf.cc.defines = %w(MRB_DISABLE_STDIO MRB_WITHOUT_FLOAT)
    conf.gem :core => "mruby-class-ext"
end

Some gems won't work without IO (eg. parser).

thanks for all

@dabroz i run minimal example with integer.
I'm using conf.cc.defines = %w(MRB_DISABLE_STDIO)
my code https://github.com/mruby/mruby/issues/3846
$ ./t
Aborted (zrzut pamięci)
irep what nedded of gem?


Very helpful but needs more

This is a great bridge from the high level 365 business admin to intune
Please also indicate which 365 business device setting requires windows 10 business upgrade, how that ties to intune, and how to troubleshoot if it doesn’t work

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d3bdeba6-de08-d374-cf95-6b83bf21e81f
Version Independent ID: 10cac9c3-602e-bfa3-3519-5d4cb52b1d24
Content: How do protection features in Microsoft 365 Business map to Intune settings
Content Source: microsoft-365/business/map-protection-features-to-intune-settings.md
Service: o365-administration
GitHub Login: @Sirkkuw
Microsoft Alias: sirkkuw

@officedocsbot assign @mypil

@gkens - Thank you for submitting feedback.
I will get this issue over to the Microsoft 365 writing team for investigation.
Thank you for reporting and making the docs better. Much appreciated.
I made a note to request the team to update this when the work is complete.

@gkens The definition of Windows 10 Business gives us the answer : "Windows 10 Business is a set of cloud-services and device management capabilities that complement Windows 10 Pro and enable the centralized management and security controls of Microsoft 365 Business."
So in order to take advantage ot the centralized management and security controls of Microsoft 365 Business you need Windows 10 Business. And if you want to go into detail and see what are these servicies and capabilities exactly, you can find this information in the Microsoft 365 Business Service Description - https://docs.microsoft.com/en-us/office365/servicedescriptions/microsoft-365-business-service-description
The link with Intune, generally speaking, is having MDM capabilities (Windows, MacOS, iOS and Android), MDM Auto-enrollment, MAM for Office Apps, deploy device profiles and security configuration, etc.

Dear Sir or Madam,
It would be helpful to see my original comment along with your response.
I think I can summarize all of your responses as “our docs are fine just as they are, no changes needed.  Talk to support if you need help”
As such, I will probably stop trying to make them better by leaving comments as I am not feeling encouraged by your responses thus far.
Glenn Kime
Principal
Enspired IT of Ann Arbor
734-228-7077
From: VLG17<EMAIL_ADDRESS>Sent: Wednesday, April 24, 2019 7:58 AM
To: MicrosoftDocs/microsoft-365-docs<EMAIL_ADDRESS>Cc: Glenn Kime<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/microsoft-365-docs] Very helpful but needs more (#326)
@gkenshttps://github.com/gkens The definition of Windows 10 Business gives us the answer : "Windows 10 Business is a set of cloud-services and device management capabilities that complement Windows 10 Pro and enable the centralized management and security controls of Microsoft 365 Business."
So in order to take advantage ot the centralized management and security controls of Microsoft 365 Business you need Windows 10 Business. And if you want to go into detail and see what are these servicies and capabilities exactly, you can find this information in the Microsoft 365 Business Service Description - https://docs.microsoft.com/en-us/office365/servicedescriptions/microsoft-365-business-service-description
The link with Intune, generally speaking, is having MDM capabilities (Windows, MacOS, iOS and Android), MDM Auto-enrollment, MAM for Office Apps, deploy device profiles and security configuration, etc.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/microsoft-365-docs/issues/326#issuecomment-486197674, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALXT7WEHRAUYYTFUOVXLQITPSBDMFANCNFSM4HD6PWJA.

@gkens - Thank you for your response. Is there something within this article that you suggest to be added or edited? If yes, please share the appropriate text and where it should be placed, so we can have @VLG17 create a Pull Request to get it added.
Otherwise, if your issue is more of a feature request or for user feedback, you can submit via User Voice - https://office365.uservoice.com
For troubleshooting, you can visit support.microsoft.com, answers.microsoft.com or open a support ticket- https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please understand that this site is a product documentation site and not a support nor a feature request site. I hope this gets you pointed in the right direction. Thank you.

@officedocsbot close