OpenAI APIs - Vision#
SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference. This tutorial covers the vision APIs for vision language models.
SGLang supports vision language models such as Llama 3.2, LLaVA-OneVision, and QWen-VL2
As an alternative to the OpenAI API, you can also use the SGLang offline engine.
Launch A Server#
Launch the server in your terminal and wait for it to initialize.
Remember to add --chat-template llama_3_vision
to specify the vision chat template, otherwise the server only supports text, and performance degradation may occur.
We need to specify --chat-template
for vision language models because the chat template provided in Hugging Face tokenizer only supports text.
[1]:
from sglang.test.test_utils import is_in_ci
if is_in_ci():
from patch import launch_server_cmd
else:
from sglang.utils import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process
vision_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
--chat-template=llama_3_vision
"""
)
wait_for_server(f"http://localhost:{port}")
INFO 03-20 18:37:33 __init__.py:190] Automatically detected platform cuda.
The following error message 'operation scheduled before its operands' can be ignored.
[2025-03-20 18:37:39] server_args=ServerArgs(model_path='meta-llama/Llama-3.2-11B-Vision-Instruct', tokenizer_path='meta-llama/Llama-3.2-11B-Vision-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='meta-llama/Llama-3.2-11B-Vision-Instruct', chat_template='llama_3_vision', completion_template=None, is_embedding=False, revision=None, host='127.0.0.1', port=38808, mem_fraction_static=0.88, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, stream_interval=1, stream_output=False, random_seed=460504660, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=5, speculative_eagle_topk=4, speculative_num_draft_tokens=8, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, enable_flashinfer_mla=False, enable_flashmla=False, flashinfer_mla_disable_ragged=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False)
[2025-03-20 18:37:46] Use chat template for the OpenAI-compatible API server: llama_3_vision
INFO 03-20 18:37:50 __init__.py:190] Automatically detected platform cuda.
INFO 03-20 18:37:50 __init__.py:190] Automatically detected platform cuda.
The following error message 'operation scheduled before its operands' can be ignored.
The following error message 'operation scheduled before its operands' can be ignored.
[2025-03-20 18:38:20 TP0] Overlap scheduler is disabled for multimodal models.
[2025-03-20 18:38:20 TP0] Automatically reduce --mem-fraction-static to 0.836 because this is a multimodal model.
[2025-03-20 18:38:20 TP0] Automatically turn off --chunked-prefill-size for mllama.
[2025-03-20 18:38:20 TP0] Init torch distributed begin.
[2025-03-20 18:38:20 TP0] Init torch distributed ends. mem usage=0.00 GB
[2025-03-20 18:38:20 TP0] Load weight begin. avail mem=78.81 GB
[2025-03-20 18:38:21 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:01<00:04, 1.10s/it]
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:02<00:03, 1.18s/it]
Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:03<00:02, 1.07s/it]
Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:03<00:00, 1.26it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:04<00:00, 1.21it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:04<00:00, 1.10it/s]
[2025-03-20 18:38:26 TP0] Load weight end. type=MllamaForConditionalGeneration, dtype=torch.bfloat16, avail mem=58.63 GB, mem usage=20.18 GB.
[2025-03-20 18:38:26 TP0] KV Cache is allocated. #tokens: 20480, K size: 1.56 GB, V size: 1.56 GB
[2025-03-20 18:38:26 TP0] Memory pool end. avail mem=55.18 GB
[2025-03-20 18:38:32 TP0] max_total_num_tokens=20480, chunked_prefill_size=-1, max_prefill_tokens=16384, max_running_requests=200, context_len=131072
[2025-03-20 18:38:32] INFO: Started server process [1419972]
[2025-03-20 18:38:32] INFO: Waiting for application startup.
[2025-03-20 18:38:32] INFO: Application startup complete.
[2025-03-20 18:38:32] INFO: Uvicorn running on http://127.0.0.1:38808 (Press CTRL+C to quit)
[2025-03-20 18:38:33] INFO: 127.0.0.1:56444 - "GET /v1/models HTTP/1.1" 200 OK
[2025-03-20 18:38:33] INFO: 127.0.0.1:56454 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-03-20 18:38:33 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:38:36] INFO: 127.0.0.1:56458 - "POST /generate HTTP/1.1" 200 OK
[2025-03-20 18:38:36] The server is fired up and ready to roll!
NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
Using cURL#
Once the server is up, you can send test requests using curl or requests.
[2]:
import subprocess
curl_command = f"""
curl -s http://localhost:{port}/v1/chat/completions \\
-d '{{
"model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
"messages": [
{{
"role": "user",
"content": [
{{
"type": "text",
"text": "What’s in this image?"
}},
{{
"type": "image_url",
"image_url": {{
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
}}
}}
]
}}
],
"max_tokens": 300
}}'
"""
response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)
response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)
[2025-03-20 18:38:42 TP0] Prefill batch. #new-seq: 1, #new-token: 6463, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:38:42] INFO: 127.0.0.1:55074 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-03-20 18:38:43 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 6462, token usage: 0.32, #running-req: 0, #queue-req: 0,
[2025-03-20 18:38:43 TP0] Decode batch. #running-req: 1, #token: 6472, token usage: 0.32, gen throughput (token/s): 3.49, #queue-req: 0,
[2025-03-20 18:38:44 TP0] Decode batch. #running-req: 1, #token: 6512, token usage: 0.32, gen throughput (token/s): 63.37, #queue-req: 0,
[2025-03-20 18:38:44] INFO: 127.0.0.1:55080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Using Python Requests#
[3]:
import requests
url = f"http://localhost:{port}/v1/chat/completions"
data = {
"model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What’s in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
},
},
],
}
],
"max_tokens": 300,
}
response = requests.post(url, json=data)
print_highlight(response.text)
[2025-03-20 18:38:45 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 6462, token usage: 0.32, #running-req: 0, #queue-req: 0,
[2025-03-20 18:38:45 TP0] Decode batch. #running-req: 1, #token: 6465, token usage: 0.32, gen throughput (token/s): 35.64, #queue-req: 0,
[2025-03-20 18:38:45 TP0] Decode batch. #running-req: 1, #token: 6505, token usage: 0.32, gen throughput (token/s): 67.02, #queue-req: 0,
[2025-03-20 18:38:46 TP0] Decode batch. #running-req: 1, #token: 6545, token usage: 0.32, gen throughput (token/s): 60.94, #queue-req: 0,
[2025-03-20 18:38:47 TP0] Decode batch. #running-req: 1, #token: 6585, token usage: 0.32, gen throughput (token/s): 66.27, #queue-req: 0,
[2025-03-20 18:38:47] INFO: 127.0.0.1:55088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Using OpenAI Python Client#
[4]:
from openai import OpenAI
client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
response = client.chat.completions.create(
model="meta-llama/Llama-3.2-11B-Vision-Instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is in this image?",
},
{
"type": "image_url",
"image_url": {
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
},
},
],
}
],
max_tokens=300,
)
print_highlight(response.choices[0].message.content)
[2025-03-20 18:38:47 TP0] Prefill batch. #new-seq: 1, #new-token: 11, #cached-token: 6452, token usage: 0.32, #running-req: 0, #queue-req: 0,
[2025-03-20 18:38:48 TP0] Decode batch. #running-req: 1, #token: 6487, token usage: 0.32, gen throughput (token/s): 31.54, #queue-req: 0,
[2025-03-20 18:38:48] INFO: 127.0.0.1:55100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
Multiple-Image Inputs#
The server also supports multiple images and interleaved text and images if the model supports it.
[5]:
from openai import OpenAI
client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
response = client.chat.completions.create(
model="meta-llama/Llama-3.2-11B-Vision-Instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true",
},
},
{
"type": "image_url",
"image_url": {
"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png",
},
},
{
"type": "text",
"text": "I have two very different images. They are not related at all. "
"Please describe the first image in one sentence, and then describe the second image in another sentence.",
},
],
}
],
temperature=0,
)
print_highlight(response.choices[0].message.content)
[2025-03-20 18:38:49 TP0] Prefill batch. #new-seq: 1, #new-token: 12895, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:38:50 TP0] Decode batch. #running-req: 1, #token: 12930, token usage: 0.63, gen throughput (token/s): 21.58, #queue-req: 0,
[2025-03-20 18:38:50] INFO: 127.0.0.1:42408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[6]:
terminate_process(vision_process)
[2025-03-20 18:38:50] Child process unexpectedly failed with an exit code 9. pid=1420480
Chat Template#
As mentioned before, if you do not specify a vision model’s --chat-template
, the server uses Hugging Face’s default template, which only supports text.
We list popular vision models with their chat templates:
meta-llama/Llama-3.2-Vision uses
llama_3_vision
.Qwen/Qwen2-VL-7B-Instruct uses
qwen2-vl
.LlaVA-OneVision uses
chatml-llava
.LLaVA-NeXT uses
chatml-llava
.Llama3-LLaVA-NeXT uses
llava_llama_3
.LLaVA-v1.5 / 1.6 uses
vicuna_v1.1
.