Speculative Decoding

Speculative Decoding#

SGLang now provides an EAGLE-based (EAGLE-2/EAGLE-3) speculative decoding option. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines. Note: Currently, Speculative Decoding in SGLang is compatible with radix cache and chunked prefill.

Performance Highlights#

Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding. For further details please see the EAGLE3 paper.

Method

Throughput (tokens/s)

SGLang (w/o speculative, 1x H100)

158.34 tokens/s

SGLang + EAGLE-2 (1x H100)

244.10 tokens/s

SGLang + EAGLE-3 (1x H100)

373.25 tokens/s

EAGLE Decoding#

To enable EAGLE speculative decoding the following parameters are relevant:

  • speculative_draft_model_path: Specifies draft model. This parameter is required.

  • speculative_num_steps: Depth of autoregressive drafting. Increases speculation range but risks rejection cascades. Default is 5.

  • speculative_eagle_topk: Branching factor per step. Improves candidate diversity, will lead to higher acceptance rate, but more lead to higher memory/compute consumption. Default is 4.

  • speculative_num_draft_tokens: Maximum parallel verification capacity. Allows deeper tree evaluation but will lead to higher GPU memory usage. Default is 8.

These parameters are the same for EAGLE-2 and EAGLE-3.

EAGLE-2 decoding#

You can enable EAGLE-2 decoding by setting --speculative_algorithm EAGLE and choosing an appropriate model.

[1]:

from sglang.test.test_utils import is_in_ci

if is_in_ci():
    from patch import launch_server_cmd
else:
    from sglang.utils import launch_server_cmd

from sglang.utils import wait_for_server, print_highlight, terminate_process

import openai

[2]:

server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf  --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 3 \
    --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs 8
"""
)

wait_for_server(f"http://localhost:{port}")

WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-07-14 06:20:51] server_args=ServerArgs(model_path='meta-llama/Llama-2-7b-chat-hf', tokenizer_path='meta-llama/Llama-2-7b-chat-hf', tokenizer_mode='auto', skip_tokenizer_init=False, skip_server_warmup=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='meta-llama/Llama-2-7b-chat-hf', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, impl='auto', host='127.0.0.1', port=31925, nccl_port=None, mem_fraction_static=0.849, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=141262134, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm='EAGLE', speculative_draft_model_path='lmsys/sglang-EAGLE-llama2-chat-7B', speculative_num_steps=3, speculative_eagle_topk=4, speculative_num_draft_tokens=16, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=8, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=True, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, enable_triton_kernel_moe=False, warmups=None, disable_hybrid_swa_memory=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
[2025-07-14 06:20:52] Inferred chat template from model path: llama-2
[2025-07-14 06:21:03] Attention backend not set. Use flashinfer backend by default.
[2025-07-14 06:21:03] Init torch distributed begin.
[2025-07-14 06:21:04] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:21:05] Load weight begin. avail mem=61.98 GB
[2025-07-14 06:21:05] The weight of LmHead is not packed
[2025-07-14 06:21:05] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:01<00:01,  1.49s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.01s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.08s/it]

[2025-07-14 06:21:07] Load weight end. type=LlamaForCausalLM, dtype=torch.float16, avail mem=49.41 GB, mem usage=12.57 GB.
[2025-07-14 06:21:07] KV Cache is allocated. #tokens: 20480, K size: 5.00 GB, V size: 5.00 GB
[2025-07-14 06:21:07] Memory pool end. avail mem=39.22 GB
[2025-07-14 06:21:08] model doesn't have generation_config.json
[2025-07-14 06:21:08] Init torch distributed begin.
[2025-07-14 06:21:08] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:21:08] Load weight begin. avail mem=38.65 GB
[2025-07-14 06:21:08] The weight of LmHead is not packed
[2025-07-14 06:21:08] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.15s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.15s/it]

[2025-07-14 06:21:09] Load weight end. type=LlamaForCausalLMEagle, dtype=torch.float16, avail mem=37.72 GB, mem usage=0.93 GB.
[2025-07-14 06:21:09] KV Cache is allocated. #tokens: 20480, K size: 0.16 GB, V size: 0.16 GB
[2025-07-14 06:21:09] Memory pool end. avail mem=37.40 GB
[2025-07-14 06:21:10] max_total_num_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=200, context_len=4096, available_gpu_mem=37.63 GB
[2025-07-14 06:21:11] INFO:     Started server process [2622521]
[2025-07-14 06:21:11] INFO:     Waiting for application startup.
[2025-07-14 06:21:11] INFO:     Application startup complete.
[2025-07-14 06:21:11] INFO:     Uvicorn running on http://127.0.0.1:31925 (Press CTRL+C to quit)
[2025-07-14 06:21:11] INFO:     127.0.0.1:60920 - "GET /v1/models HTTP/1.1" 200 OK
[2025-07-14 06:21:12] INFO:     127.0.0.1:60928 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-07-14 06:21:12] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:21:12.175208


NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
[3]:

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(f"Response: {response}")

[2025-07-14 06:22:16] Prefill batch. #new-seq: 1, #new-token: 16, #cached-token: 1, token usage: 0.00, #running-req: 1, #queue-req: 0, timestamp: 2025-07-14T06:22:16.161953
[2025-07-14 06:22:46] INFO:     127.0.0.1:60938 - "POST /generate HTTP/1.1" 200 OK
[2025-07-14 06:22:46] The server is fired up and ready to roll!
[2025-07-14 06:22:49] INFO:     127.0.0.1:60942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Response: ChatCompletion(id='f96ed0fc9e6e4fba83cf8079ffa9846d', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=' Sure! Here are three countries and their capitals:\n\n1. Country: France\nCapital: Paris\n2. Country: Japan\nCapital: Tokyo\n3. Country: Brazil\nCapital: Brasília', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None, reasoning_content=None), matched_stop=2)], created=1752474169, model='meta-llama/Llama-2-7b-chat-hf', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=48, prompt_tokens=17, total_tokens=65, completion_tokens_details=None, prompt_tokens_details=None))
[4]:

terminate_process(server_process)

[2025-07-14 06:22:49] Child process unexpectedly failed with exitcode=9. pid=2623413
[2025-07-14 06:22:49] Child process unexpectedly failed with exitcode=9. pid=2622956

EAGLE-2 Decoding with torch.compile#

You can also enable torch.compile for further optimizations and optionally set --torch-compile-max-bs:

[5]:

server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf  --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 5 \
        --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.6 \
            --enable-torch-compile --torch-compile-max-bs 2
"""
)

wait_for_server(f"http://localhost:{port}")

WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-07-14 06:23:01] server_args=ServerArgs(model_path='meta-llama/Llama-2-7b-chat-hf', tokenizer_path='meta-llama/Llama-2-7b-chat-hf', tokenizer_mode='auto', skip_tokenizer_init=False, skip_server_warmup=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='meta-llama/Llama-2-7b-chat-hf', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, impl='auto', host='127.0.0.1', port=34447, nccl_port=None, mem_fraction_static=0.6, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=299598129, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm='EAGLE', speculative_draft_model_path='lmsys/sglang-EAGLE-llama2-chat-7B', speculative_num_steps=5, speculative_eagle_topk=8, speculative_num_draft_tokens=64, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=None, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=True, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=True, torch_compile_max_bs=2, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, enable_triton_kernel_moe=False, warmups=None, disable_hybrid_swa_memory=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
[2025-07-14 06:23:01] Inferred chat template from model path: llama-2
[2025-07-14 06:23:14] Attention backend not set. Use flashinfer backend by default.
[2025-07-14 06:23:14] Init torch distributed begin.
[2025-07-14 06:23:14] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:23:15] Load weight begin. avail mem=61.31 GB
[2025-07-14 06:23:15] The weight of LmHead is not packed
[2025-07-14 06:23:16] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:01<00:01,  1.42s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.02it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:02<00:00,  1.05s/it]

[2025-07-14 06:23:18] Load weight end. type=LlamaForCausalLM, dtype=torch.float16, avail mem=48.74 GB, mem usage=12.57 GB.
[2025-07-14 06:23:18] KV Cache is allocated. #tokens: 20480, K size: 5.00 GB, V size: 5.00 GB
[2025-07-14 06:23:18] Memory pool end. avail mem=38.54 GB
[2025-07-14 06:23:18] model doesn't have generation_config.json
[2025-07-14 06:23:18] Init torch distributed begin.
[2025-07-14 06:23:18] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:23:18] Load weight begin. avail mem=37.97 GB
[2025-07-14 06:23:18] The weight of LmHead is not packed
[2025-07-14 06:23:18] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.23s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.24s/it]

[2025-07-14 06:23:20] Load weight end. type=LlamaForCausalLMEagle, dtype=torch.float16, avail mem=37.04 GB, mem usage=0.93 GB.
[2025-07-14 06:23:20] KV Cache is allocated. #tokens: 20480, K size: 0.16 GB, V size: 0.16 GB
[2025-07-14 06:23:20] Memory pool end. avail mem=36.73 GB
[2025-07-14 06:23:20] max_total_num_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=200, context_len=4096, available_gpu_mem=36.91 GB
[2025-07-14 06:23:20] INFO:     Started server process [2630161]
[2025-07-14 06:23:20] INFO:     Waiting for application startup.
[2025-07-14 06:23:20] INFO:     Application startup complete.
[2025-07-14 06:23:20] INFO:     Uvicorn running on http://127.0.0.1:34447 (Press CTRL+C to quit)
[2025-07-14 06:23:21] INFO:     127.0.0.1:57200 - "GET /v1/models HTTP/1.1" 200 OK
[2025-07-14 06:23:21] INFO:     127.0.0.1:57202 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-07-14 06:23:21] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:23:21.935334
[2025-07-14 06:23:24] INFO:     127.0.0.1:57218 - "POST /generate HTTP/1.1" 200 OK
[2025-07-14 06:23:24] The server is fired up and ready to roll!


NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
[6]:

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(f"Response: {response}")

[2025-07-14 06:23:26] Prefill batch. #new-seq: 1, #new-token: 16, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:23:26.695276
[2025-07-14 06:23:27] INFO:     127.0.0.1:57232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Response: ChatCompletion(id='b4df4695f6a94ed0aac27daceb674e11', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=' Sure! Here are three countries and their capitals:\n\n1. Country: France\nCapital: Paris\n2. Country: Japan\nCapital: Tokyo\n3. Country: Brazil\nCapital: Brasília', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None, reasoning_content=None), matched_stop=2)], created=1752474207, model='meta-llama/Llama-2-7b-chat-hf', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=48, prompt_tokens=17, total_tokens=65, completion_tokens_details=None, prompt_tokens_details=None))
[7]:

terminate_process(server_process)

[2025-07-14 06:23:27] Child process unexpectedly failed with exitcode=9. pid=2630537
[2025-07-14 06:23:27] Child process unexpectedly failed with exitcode=9. pid=2630310

EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling#

By employing a truncated high-frequency token vocabulary in the draft model, Eagle speculative decoding reduces lm_head computational overhead while accelerating the pipeline without quality degradation. For more details, checkout the paper.

In our implementation, set --speculative-token-map to enable the optimization. You can get the high-frequency token in FR-Spec from this model. Or you can obtain high-frequency token by directly downloading these token from this repo.

Thanks for the contribution from Weilin Zhao and Zhousx.

[8]:

server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
    --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16
"""
)

wait_for_server(f"http://localhost:{port}")

WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-07-14 06:23:37] server_args=ServerArgs(model_path='meta-llama/Meta-Llama-3-8B-Instruct', tokenizer_path='meta-llama/Meta-Llama-3-8B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, skip_server_warmup=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, dtype='float16', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='meta-llama/Meta-Llama-3-8B-Instruct', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, impl='auto', host='127.0.0.1', port=36639, nccl_port=None, mem_fraction_static=0.7, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=763599065, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm='EAGLE', speculative_draft_model_path='lmsys/sglang-EAGLE-LLaMA3-Instruct-8B', speculative_num_steps=5, speculative_eagle_topk=8, speculative_num_draft_tokens=64, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map='thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt', ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=2, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=True, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, enable_triton_kernel_moe=False, warmups=None, disable_hybrid_swa_memory=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
[2025-07-14 06:23:38] Casting torch.bfloat16 to torch.float16.
[2025-07-14 06:23:49] Casting torch.bfloat16 to torch.float16.
[2025-07-14 06:23:50] Casting torch.bfloat16 to torch.float16.
[2025-07-14 06:23:50] Attention backend not set. Use flashinfer backend by default.
[2025-07-14 06:23:50] Init torch distributed begin.
[2025-07-14 06:23:52] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:23:53] Load weight begin. avail mem=61.36 GB
[2025-07-14 06:23:54] The weight of LmHead is not packed
[2025-07-14 06:23:54] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:04<00:12,  4.22s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:08<00:08,  4.28s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.27s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.02s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.47s/it]

[2025-07-14 06:24:08] Load weight end. type=LlamaForCausalLM, dtype=torch.float16, avail mem=63.52 GB, mem usage=-2.16 GB.
[2025-07-14 06:24:08] KV Cache is allocated. #tokens: 20480, K size: 1.25 GB, V size: 1.25 GB
[2025-07-14 06:24:08] Memory pool end. avail mem=60.83 GB
[2025-07-14 06:24:09] model doesn't have generation_config.json
[2025-07-14 06:24:09] Warning: User-specified context_length (8192) is greater than the derived context_length (2048). This may lead to incorrect model outputs or CUDA errors.
[2025-07-14 06:24:09] Init torch distributed begin.
[2025-07-14 06:24:09] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:24:09] Load weight begin. avail mem=60.25 GB
[2025-07-14 06:24:09] The weight of LmHead is not packed
[2025-07-14 06:24:09] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.06it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.06it/s]

[2025-07-14 06:24:10] Load weight end. type=LlamaForCausalLMEagle, dtype=torch.float16, avail mem=58.55 GB, mem usage=1.70 GB.
[2025-07-14 06:24:10] KV Cache is allocated. #tokens: 20480, K size: 0.04 GB, V size: 0.04 GB
[2025-07-14 06:24:10] Memory pool end. avail mem=58.47 GB
[2025-07-14 06:24:10] max_total_num_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=200, context_len=8192, available_gpu_mem=59.15 GB
[2025-07-14 06:24:11] INFO:     Started server process [2632095]
[2025-07-14 06:24:11] INFO:     Waiting for application startup.
[2025-07-14 06:24:11] INFO:     Application startup complete.
[2025-07-14 06:24:11] INFO:     Uvicorn running on http://127.0.0.1:36639 (Press CTRL+C to quit)
[2025-07-14 06:24:11] INFO:     127.0.0.1:51560 - "GET /v1/models HTTP/1.1" 200 OK
[2025-07-14 06:24:12] INFO:     127.0.0.1:51562 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-07-14 06:24:12] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:24:12.359907
[2025-07-14 06:24:14] INFO:     127.0.0.1:51568 - "POST /generate HTTP/1.1" 200 OK
[2025-07-14 06:24:14] The server is fired up and ready to roll!


NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
[9]:

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(f"Response: {response}")

[2025-07-14 06:24:16] Prefill batch. #new-seq: 1, #new-token: 17, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:24:16.643021
[2025-07-14 06:24:16] INFO:     127.0.0.1:51570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Response: ChatCompletion(id='8285545362554e4686864ad8c12f3938', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Here are 3 countries and their capitals:\n\n1. **France** - **Paris**\n2. **Japan** - **Tokyo**\n3. **Australia** - **Canberra**', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None, reasoning_content=None), matched_stop=128009)], created=1752474256, model='meta-llama/Meta-Llama-3-8B-Instruct', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=39, prompt_tokens=18, total_tokens=57, completion_tokens_details=None, prompt_tokens_details=None))
[10]:

terminate_process(server_process)

[2025-07-14 06:24:16] Child process unexpectedly failed with exitcode=9. pid=2633109

EAGLE-3 Decoding#

You can enable EAGLE-3 decoding by setting --speculative_algorithm EAGLE3 and choosing an appropriate model.

[11]:

server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct  --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 5 \
        --speculative-eagle-topk 8 --speculative-num-draft-tokens 32 --mem-fraction 0.6 \
        --cuda-graph-max-bs 2 --dtype float16
"""
)

wait_for_server(f"http://localhost:{port}")

WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-07-14 06:24:27] server_args=ServerArgs(model_path='meta-llama/Llama-3.1-8B-Instruct', tokenizer_path='meta-llama/Llama-3.1-8B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, skip_server_warmup=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, dtype='float16', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='meta-llama/Llama-3.1-8B-Instruct', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, impl='auto', host='127.0.0.1', port=31365, nccl_port=None, mem_fraction_static=0.6, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=673303888, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm='EAGLE3', speculative_draft_model_path='jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B', speculative_num_steps=5, speculative_eagle_topk=8, speculative_num_draft_tokens=32, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=2, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=True, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, enable_triton_kernel_moe=False, warmups=None, disable_hybrid_swa_memory=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
[2025-07-14 06:24:27] Casting torch.bfloat16 to torch.float16.
[2025-07-14 06:24:38] Casting torch.bfloat16 to torch.float16.
[2025-07-14 06:24:39] Casting torch.bfloat16 to torch.float16.
[2025-07-14 06:24:39] Attention backend not set. Use flashinfer backend by default.
[2025-07-14 06:24:39] Init torch distributed begin.
[2025-07-14 06:24:39] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:24:40] Load weight begin. avail mem=78.50 GB
[2025-07-14 06:24:40] The weight of LmHead is not packed
[2025-07-14 06:24:41] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:04<00:12,  4.24s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:08<00:08,  4.21s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:12<00:04,  4.15s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  2.99s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:13<00:00,  3.43s/it]

[2025-07-14 06:24:55] Load weight end. type=LlamaForCausalLM, dtype=torch.float16, avail mem=63.48 GB, mem usage=15.02 GB.
[2025-07-14 06:24:55] KV Cache is allocated. #tokens: 20480, K size: 1.25 GB, V size: 1.25 GB
[2025-07-14 06:24:55] Memory pool end. avail mem=60.68 GB
[2025-07-14 06:24:55] model doesn't have generation_config.json
[2025-07-14 06:24:55] Warning: User-specified context_length (131072) is greater than the derived context_length (2048). This may lead to incorrect model outputs or CUDA errors.
[2025-07-14 06:24:55] Init torch distributed begin.
[2025-07-14 06:24:55] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:24:55] Load weight begin. avail mem=60.11 GB
[2025-07-14 06:24:55] The weight of LmHead is not packed
[2025-07-14 06:24:55] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.82it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.82it/s]

[2025-07-14 06:24:56] Load weight end. type=LlamaForCausalLMEagle3, dtype=torch.float16, avail mem=58.29 GB, mem usage=1.82 GB.
[2025-07-14 06:24:56] KV Cache is allocated. #tokens: 20480, K size: 0.04 GB, V size: 0.04 GB
[2025-07-14 06:24:56] Memory pool end. avail mem=58.21 GB
[2025-07-14 06:24:56] max_total_num_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=200, context_len=131072, available_gpu_mem=58.94 GB
[2025-07-14 06:24:57] INFO:     Started server process [2635161]
[2025-07-14 06:24:57] INFO:     Waiting for application startup.
[2025-07-14 06:24:57] INFO:     Application startup complete.
[2025-07-14 06:24:57] INFO:     Uvicorn running on http://127.0.0.1:31365 (Press CTRL+C to quit)
[2025-07-14 06:24:58] INFO:     127.0.0.1:33082 - "GET /v1/models HTTP/1.1" 200 OK
[2025-07-14 06:24:58] INFO:     127.0.0.1:52864 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-07-14 06:24:58] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:24:58.578782
[2025-07-14 06:25:01] INFO:     127.0.0.1:52866 - "POST /generate HTTP/1.1" 200 OK
[2025-07-14 06:25:01] The server is fired up and ready to roll!


NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
[12]:

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(f"Response: {response}")

[2025-07-14 06:25:03] Prefill batch. #new-seq: 1, #new-token: 42, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:25:03.150916
[2025-07-14 06:25:03] INFO:     127.0.0.1:52878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Response: ChatCompletion(id='a2c6a6f970254e5f98732c681973cb73', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Here are 3 countries and their capitals:\n\n1. Country: Japan\n Capital: Tokyo\n\n2. Country: Australia\n Capital: Canberra\n\n3. Country: Brazil\n Capital: Brasília', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None, reasoning_content=None), matched_stop=128009)], created=1752474303, model='meta-llama/Meta-Llama-3.1-8B-Instruct', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=43, prompt_tokens=43, total_tokens=86, completion_tokens_details=None, prompt_tokens_details=None))
[13]:

terminate_process(server_process)

[2025-07-14 06:25:03] Child process unexpectedly failed with exitcode=9. pid=2635466
[2025-07-14 06:25:03] Child process unexpectedly failed with exitcode=9. pid=2635317

Multi Token Prediction#

We support MTP(Multi-Token Prediction) in SGLang by using speculative decoding. We use Xiaomi/MiMo-7B-RL model as example here (deepseek mtp usage refer to deepseek doc)

[14]:

server_process, port = launch_server_cmd(
    """
    python3 -m sglang.launch_server --model-path XiaomiMiMo/MiMo-7B-RL --host 0.0.0.0 --trust-remote-code \
    --speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
    --mem-fraction 0.5
"""
)

wait_for_server(f"http://localhost:{port}")

WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-07-14 06:25:14] server_args=ServerArgs(model_path='XiaomiMiMo/MiMo-7B-RL', tokenizer_path='XiaomiMiMo/MiMo-7B-RL', tokenizer_mode='auto', skip_tokenizer_init=False, skip_server_warmup=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='XiaomiMiMo/MiMo-7B-RL', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, impl='auto', host='0.0.0.0', port=32560, nccl_port=None, mem_fraction_static=0.5, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=97911973, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm='EAGLE', speculative_draft_model_path=None, speculative_num_steps=1, speculative_eagle_topk=1, speculative_num_draft_tokens=2, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=None, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=True, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, enable_triton_kernel_moe=False, warmups=None, disable_hybrid_swa_memory=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
[2025-07-14 06:25:25] Attention backend not set. Use flashinfer backend by default.
[2025-07-14 06:25:25] Init torch distributed begin.
[2025-07-14 06:25:26] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:25:27] Load weight begin. avail mem=78.50 GB
[2025-07-14 06:25:27] The weight of LmHead is not packed
[2025-07-14 06:25:27] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:01,  1.65it/s]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.46it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.39it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.49it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.48it/s]

[2025-07-14 06:25:30] Load weight end. type=MiMoForCausalLM, dtype=torch.bfloat16, avail mem=64.28 GB, mem usage=14.21 GB.
[2025-07-14 06:25:30] KV Cache is allocated. #tokens: 20480, K size: 1.41 GB, V size: 1.41 GB
[2025-07-14 06:25:30] Memory pool end. avail mem=61.23 GB
[2025-07-14 06:25:30] Init torch distributed begin.
[2025-07-14 06:25:30] Init torch distributed ends. mem usage=0.00 GB
[2025-07-14 06:25:30] Load weight begin. avail mem=60.54 GB
[2025-07-14 06:25:30] The weight of LmHead is not packed
[2025-07-14 06:25:30] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:00,  5.35it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00,  8.89it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00,  8.47it/s]

[2025-07-14 06:25:31] Load weight end. type=MiMoMTP, dtype=torch.bfloat16, avail mem=57.83 GB, mem usage=2.71 GB.
[2025-07-14 06:25:31] KV Cache is allocated. #tokens: 20480, K size: 1.41 GB, V size: 1.41 GB
[2025-07-14 06:25:31] Memory pool end. avail mem=54.88 GB
[2025-07-14 06:25:31] max_total_num_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=200, context_len=32768, available_gpu_mem=56.97 GB
[2025-07-14 06:25:32] INFO:     Started server process [2636216]
[2025-07-14 06:25:32] INFO:     Waiting for application startup.
[2025-07-14 06:25:32] INFO:     Application startup complete.
[2025-07-14 06:25:32] INFO:     Uvicorn running on http://0.0.0.0:32560 (Press CTRL+C to quit)
[2025-07-14 06:25:33] INFO:     127.0.0.1:44264 - "GET /v1/models HTTP/1.1" 200 OK
[2025-07-14 06:25:33] INFO:     127.0.0.1:44274 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-07-14 06:25:33] Prefill batch. #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:25:33.717718
[2025-07-14 06:25:34] INFO:     127.0.0.1:44278 - "POST /generate HTTP/1.1" 200 OK
[2025-07-14 06:25:34] The server is fired up and ready to roll!


NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
[15]:

import requests

url = f"http://localhost:{port}/v1/chat/completions"

data = {
    "model": "XiaomiMiMo/MiMo-7B-RL",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
}

response = requests.post(url, json=data)
print_highlight(response.json())

[2025-07-14 06:25:38] Prefill batch. #new-seq: 1, #new-token: 26, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-14T06:25:38.545887
[2025-07-14 06:25:39] Decode batch. #running-req: 1, #token: 92, token usage: 0.00, accept len: 1.79, cuda graph: False, gen throughput (token/s): 9.99, #queue-req: 0, timestamp: 2025-07-14T06:25:39.153387
[2025-07-14 06:25:39] Decode batch. #running-req: 1, #token: 158, token usage: 0.01, accept len: 1.65, cuda graph: False, gen throughput (token/s): 101.40, #queue-req: 0, timestamp: 2025-07-14T06:25:39.804271
[2025-07-14 06:25:40] Decode batch. #running-req: 1, #token: 223, token usage: 0.01, accept len: 1.62, cuda graph: False, gen throughput (token/s): 99.96, #queue-req: 0, timestamp: 2025-07-14T06:25:40.454543
[2025-07-14 06:25:41] Decode batch. #running-req: 1, #token: 287, token usage: 0.01, accept len: 1.60, cuda graph: False, gen throughput (token/s): 100.24, #queue-req: 0, timestamp: 2025-07-14T06:25:41.093013
[2025-07-14 06:25:41] Decode batch. #running-req: 1, #token: 355, token usage: 0.02, accept len: 1.70, cuda graph: False, gen throughput (token/s): 105.50, #queue-req: 0, timestamp: 2025-07-14T06:25:41.737563
[2025-07-14 06:25:42] Decode batch. #running-req: 1, #token: 415, token usage: 0.02, accept len: 1.50, cuda graph: False, gen throughput (token/s): 91.07, #queue-req: 0, timestamp: 2025-07-14T06:25:42.396403
[2025-07-14 06:25:43] Decode batch. #running-req: 1, #token: 486, token usage: 0.02, accept len: 1.77, cuda graph: False, gen throughput (token/s): 109.16, #queue-req: 0, timestamp: 2025-07-14T06:25:43.046834
[2025-07-14 06:25:43] Decode batch. #running-req: 1, #token: 551, token usage: 0.03, accept len: 1.62, cuda graph: False, gen throughput (token/s): 98.61, #queue-req: 0, timestamp: 2025-07-14T06:25:43.706001
[2025-07-14 06:25:44] Decode batch. #running-req: 1, #token: 622, token usage: 0.03, accept len: 1.77, cuda graph: False, gen throughput (token/s): 110.56, #queue-req: 0, timestamp: 2025-07-14T06:25:44.348182
[2025-07-14 06:25:44] INFO:     127.0.0.1:44288 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{'id': 'a356c2e15c024452a263be0770030f46', 'object': 'chat.completion', 'created': 1752474344, 'model': 'XiaomiMiMo/MiMo-7B-RL', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': '\nOkay, so the user is asking, "What is the capital of France?" Let me start by recalling what I know about France. France is a country in Europe, right? It\'s known for things like the Eiffel Tower, the Louvre Museum, and the French Riviera. Now, the capital... Hmm, I think it\'s Paris. But wait, maybe I should double-check that. Sometimes countries have capitals that aren\'t the most well-known city. For example, the capital of Japan is Tokyo, which is a big city, but maybe France\'s capital is similar.\n\nI remember that Paris has a lot of historical significance. The Tour de France starts there, and it\'s mentioned in a lot of movies and books. Also, the French government is based in Paris. The president of France lives in the Elysée Palace, which is in Paris. The national assembly is there too. So that makes sense. \n\nWait, could there be any confusion? Let me think. Are there any other major cities in France? Lyon is a big industrial city, but I don\'t think it\'s the capital. Marseille is a major port, but again, not the capital. Lille is more northern, but not the capital. So Paris definitely seems to be the main city. \n\nI should also consider if there have been any changes. Has the capital of France ever been moved? From what I remember, Paris has been the capital since the French Revolution. Before that, maybe some other cities were important, like during the Middle Ages, but I don\'t think so. \n\nAnother way to verify: think about the country\'s abbreviation. France is often abbreviated as "F" for France, but sometimes "FR" for France Republic. But that\'s not relevant here. Maybe check a map. Paris is definitely the largest city in France. According to population stats, Paris has over 2 million people in the city proper, and the metro area is around 12 million. That\'s much larger than other cities like Lyon or Marseille. \n\nSo putting it all together: Paris is the capital, largest city, and a major cultural and political hub. The answer should be straightforward. I don\'t see any reason to doubt this. Maybe the user is taking a basic geography test or just curious. Either way, the answer is Paris. \n\nWait, just to be thorough, maybe check if there\'s a district or area within Paris that serves as the capital. But the question is about the capital of the country, which is Paris itself. The city has administrative divisions, but the capital as a whole is Paris. \n\nNo, I think I\'m confident. The answer is Paris.\n\nThe capital of France is **Paris**. This historic city is home to major political institutions, cultural landmarks like the Louvre Museum and the Eiffel Tower, and serves as the center of the country\'s economy, politics, and entertainment life.', 'reasoning_content': None, 'tool_calls': None}, 'logprobs': None, 'finish_reason': 'stop', 'matched_stop': 151645}], 'usage': {'prompt_tokens': 26, 'total_tokens': 627, 'completion_tokens': 601, 'prompt_tokens_details': None}}
[16]:

terminate_process(server_process)

[2025-07-14 06:25:44] Child process unexpectedly failed with exitcode=9. pid=2636500

References#

EAGLE process is as follows:

  • Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence \((f_1, ..., f_k)\) and the token sequence \((t_2, ..., t_{k+1})\).

  • The next token is then sampled from \(p_{k+2}=\text{LMHead}(f_{k+1})\). Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the speculative_eagle_topk parameter—to ensure a more coherent connection of context, and are given as input again.

  • EAGLE-2 additionally uses the draft model to evaluate how probable certain branches in the draft tree are, dynamically stopping the expansion of unlikely branches. After the expansion phase, reranking is employed to select only the top speculative_num_draft_tokens final nodes as draft tokens.

  • EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner.

This enhances drafting accuracy by operating on the features instead of tokens for more regular inputs and passing the tokens from the next timestep additionally to minimize randomness effects from sampling. Furthermore the dynamic adjustment of the draft tree and selection of reranked final nodes increases acceptance rate of draft tokens further. For more details see EAGLE-2 and EAGLE-3 paper.

For guidance how to train your own EAGLE model please see the EAGLE repo.

Contents