OpenAI APIs - Vision#
SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference. This tutorial covers the vision APIs for vision language models.
SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and more.
As an alternative to the OpenAI API, you can also use the SGLang offline engine.
Launch A Server#
Launch the server in your terminal and wait for it to initialize.
[1]:
from sglang.test.test_utils import is_in_ci
if is_in_ci():
from patch import launch_server_cmd
else:
from sglang.utils import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process
vision_process, port = launch_server_cmd(
"""
python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct
"""
)
wait_for_server(f"http://localhost:{port}")
WARNING:sglang.srt.server_args:Multimodal model: Dynamically adjusted --mem-fraction-static from: 0.874 to: 0.784.
[2025-07-19 07:58:08] server_args=ServerArgs(model_path='Qwen/Qwen2.5-VL-7B-Instruct', tokenizer_path='Qwen/Qwen2.5-VL-7B-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, skip_server_warmup=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='Qwen/Qwen2.5-VL-7B-Instruct', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, impl='auto', host='127.0.0.1', port=39486, nccl_port=None, mem_fraction_static=0.7835956249999999, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=1021573117, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=None, cuda_graph_bs=None, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=False, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', hicache_io_backend='', hicache_storage_backend=None, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, enable_triton_kernel_moe=False, warmups=None, disable_hybrid_swa_memory=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False, enable_pdmux=False, sm_group_num=3)
You have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename the file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.
[2025-07-19 07:58:10] You have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename the file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.
[2025-07-19 07:58:10] Inferred chat template from model path: qwen2-vl
You have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename the file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.
[2025-07-19 07:58:24] You have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename the file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.
[2025-07-19 07:58:24] Attention backend not explicitly specified. Use flashinfer backend by default.
[2025-07-19 07:58:24] Init torch distributed begin.
[2025-07-19 07:58:25] Init torch distributed ends. mem usage=0.00 GB
[2025-07-19 07:58:27] Load weight begin. avail mem=78.50 GB
[2025-07-19 07:58:27] Multimodal attention backend not set. Use sdpa.
[2025-07-19 07:58:27] Using sdpa as multimodal attention backend.
[2025-07-19 07:58:28] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:00<00:02, 1.63it/s]
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:01<00:01, 1.62it/s]
Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:01<00:01, 1.61it/s]
Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:02<00:00, 1.57it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 2.01it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:02<00:00, 1.80it/s]
[2025-07-19 07:58:31] Load weight end. type=Qwen2_5_VLForConditionalGeneration, dtype=torch.bfloat16, avail mem=35.46 GB, mem usage=43.03 GB.
[2025-07-19 07:58:31] KV Cache is allocated. #tokens: 20480, K size: 0.55 GB, V size: 0.55 GB
[2025-07-19 07:58:31] Memory pool end. avail mem=34.10 GB
[2025-07-19 07:58:32] max_total_num_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=200, context_len=128000, available_gpu_mem=33.52 GB
[2025-07-19 07:58:33] INFO: Started server process [614480]
[2025-07-19 07:58:33] INFO: Waiting for application startup.
[2025-07-19 07:58:33] INFO: Application startup complete.
[2025-07-19 07:58:33] INFO: Uvicorn running on http://127.0.0.1:39486 (Press CTRL+C to quit)
[2025-07-19 07:58:33] INFO: 127.0.0.1:42386 - "GET /v1/models HTTP/1.1" 200 OK
[2025-07-19 07:58:34] INFO: 127.0.0.1:42392 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-07-19 07:58:34] Prefill batch. #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-19T07:58:34.119571
[2025-07-19 07:58:35] INFO: 127.0.0.1:42398 - "POST /generate HTTP/1.1" 200 OK
[2025-07-19 07:58:35] The server is fired up and ready to roll!
NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
Using cURL#
Once the server is up, you can send test requests using curl or requests.
[2]:
import subprocess
curl_command = f"""
curl -s http://localhost:{port}/v1/chat/completions \\
-H "Content-Type: application/json" \\
-d '{{
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
"messages": [
{{
"role": "user",
"content": [
{{
"type": "text",
"text": "What’s in this image?"
}},
{{
"type": "image_url",
"image_url": {{
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
}}
}}
]
}}
],
"max_tokens": 300
}}'
"""
response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)
response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)
[2025-07-19 07:58:39] Prefill batch. #new-seq: 1, #new-token: 307, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-19T07:58:39.254760
[2025-07-19 07:58:40] Decode batch. #running-req: 1, #token: 340, token usage: 0.02, cuda graph: False, gen throughput (token/s): 5.26, #queue-req: 0, timestamp: 2025-07-19T07:58:40.229231
[2025-07-19 07:58:40] Decode batch. #running-req: 1, #token: 380, token usage: 0.02, cuda graph: False, gen throughput (token/s): 59.38, #queue-req: 0, timestamp: 2025-07-19T07:58:40.902913
[2025-07-19 07:58:41] Decode batch. #running-req: 1, #token: 420, token usage: 0.02, cuda graph: False, gen throughput (token/s): 55.95, #queue-req: 0, timestamp: 2025-07-19T07:58:41.617885
[2025-07-19 07:58:41] INFO: 127.0.0.1:42410 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"4d52923496a54e33be77ba98d5388130","object":"chat.completion","created":1752911921,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The image portrays a man standing on the tailgate of a yellow SUV parked on a city street. He is wearing a yellow hoodie and is engaged in an unusual task—ironing a pair of jeans while balancing them vertically on a makeshift setup consisting of levers and supports attached to the tailgate. The iron box he is holding is also reaching out horizontally, an unlikely maneuver for an ironing task. Around him, there are several taxis parked nearby, and urban scenery like sidewalks, buildings, and storefronts suggest that this is happening in a busy metropolitan street.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151645}],"usage":{"prompt_tokens":307,"total_tokens":421,"completion_tokens":114,"prompt_tokens_details":null}}
[2025-07-19 07:58:42] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 306, token usage: 0.01, #running-req: 0, #queue-req: 0, timestamp: 2025-07-19T07:58:42.010835
[2025-07-19 07:58:42] Decode batch. #running-req: 1, #token: 346, token usage: 0.02, cuda graph: False, gen throughput (token/s): 36.99, #queue-req: 0, timestamp: 2025-07-19T07:58:42.699384
[2025-07-19 07:58:43] Decode batch. #running-req: 1, #token: 386, token usage: 0.02, cuda graph: False, gen throughput (token/s): 59.64, #queue-req: 0, timestamp: 2025-07-19T07:58:43.370063
[2025-07-19 07:58:43] INFO: 127.0.0.1:57084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"bf8b1cff111e47edbf72de984c411269","object":"chat.completion","created":1752911923,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The image depicts a man dressed in a yellow shirt, ironing clothes while leaning over the open trunk of a yellow taxi parked in an urban street setting. The taxi bears logos and advertisements that suggest its affiliations with major taxi service companies. The street is lined with buildings featuring shops and storefronts, and another taxi vehicle can be seen in the background. It's a humorous and unusual scene, as taxis are generally designed for transporting vehicles not passengers engaging in domestic tasks like ironing clothes!","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151645}],"usage":{"prompt_tokens":307,"total_tokens":406,"completion_tokens":99,"prompt_tokens_details":null}}
Using Python Requests#
[3]:
import requests
url = f"http://localhost:{port}/v1/chat/completions"
data = {
"model": "Qwen/Qwen2.5-VL-7B-Instruct",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What’s in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
},
},
],
}
],
"max_tokens": 300,
}
response = requests.post(url, json=data)
print_highlight(response.text)
[2025-07-19 07:58:44] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 306, token usage: 0.01, #running-req: 0, #queue-req: 0, timestamp: 2025-07-19T07:58:44.098866
[2025-07-19 07:58:44] Decode batch. #running-req: 1, #token: 327, token usage: 0.02, cuda graph: False, gen throughput (token/s): 36.44, #queue-req: 0, timestamp: 2025-07-19T07:58:44.467701
[2025-07-19 07:58:45] Decode batch. #running-req: 1, #token: 367, token usage: 0.02, cuda graph: False, gen throughput (token/s): 58.92, #queue-req: 0, timestamp: 2025-07-19T07:58:45.146546
[2025-07-19 07:58:45] INFO: 127.0.0.1:57098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"b6943887102e46648570b76288890b53","object":"chat.completion","created":1752911925,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The image depicts a man performing an unconventional use of a taxi. Instead of driving, he is leaning onto the taxi's rear door and appears to be pressing an item with an iron, utilizing the car door as an unusual and improvised pressing/ironing surface. The taxi has a yellow cab body, similar to those typically found in busy urban areas. There's a blue cloth or garment under the board he is pressing.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151645}],"usage":{"prompt_tokens":307,"total_tokens":392,"completion_tokens":85,"prompt_tokens_details":null}}
Using OpenAI Python Client#
[4]:
from openai import OpenAI
client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
response = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is in this image?",
},
{
"type": "image_url",
"image_url": {
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
},
},
],
}
],
max_tokens=300,
)
print_highlight(response.choices[0].message.content)
[2025-07-19 07:58:46] Prefill batch. #new-seq: 1, #new-token: 292, #cached-token: 15, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-19T07:58:46.120399
[2025-07-19 07:58:46] Decode batch. #running-req: 1, #token: 322, token usage: 0.02, cuda graph: False, gen throughput (token/s): 28.49, #queue-req: 0, timestamp: 2025-07-19T07:58:46.550535
[2025-07-19 07:58:47] Decode batch. #running-req: 1, #token: 362, token usage: 0.02, cuda graph: False, gen throughput (token/s): 58.64, #queue-req: 0, timestamp: 2025-07-19T07:58:47.232639
[2025-07-19 07:58:47] Decode batch. #running-req: 1, #token: 402, token usage: 0.02, cuda graph: False, gen throughput (token/s): 58.46, #queue-req: 0, timestamp: 2025-07-19T07:58:47.916874
[2025-07-19 07:58:48] INFO: 127.0.0.1:57102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
The image shows a person on the back of a yellow taxi, interacting with what appears to be a stand-up ironing board and fabric. The individual is actively working on pressing or adjusting something (likely clothing) on the board. The taxi is parked on a busy street with other taxis in the background. This seems to depict humorously unconventional use of a public vehicle, perhaps integrating a side business like laundry pickup and service, or it could be staged as performative art or a publicity stunt.
Multiple-Image Inputs#
The server also supports multiple images and interleaved text and images if the model supports it.
[5]:
from openai import OpenAI
client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
response = client.chat.completions.create(
model="Qwen/Qwen2.5-VL-7B-Instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true",
},
},
{
"type": "image_url",
"image_url": {
"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png",
},
},
{
"type": "text",
"text": "I have two very different images. They are not related at all. "
"Please describe the first image in one sentence, and then describe the second image in another sentence.",
},
],
}
],
temperature=0,
)
print_highlight(response.choices[0].message.content)
[2025-07-19 07:58:48] Prefill batch. #new-seq: 1, #new-token: 2532, #cached-token: 14, token usage: 0.00, #running-req: 0, #queue-req: 0, timestamp: 2025-07-19T07:58:48.782378
[2025-07-19 07:58:50] Decode batch. #running-req: 1, #token: 2581, token usage: 0.13, cuda graph: False, gen throughput (token/s): 17.23, #queue-req: 0, timestamp: 2025-07-19T07:58:50.238324
[2025-07-19 07:58:50] INFO: 127.0.0.1:57116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
The first image shows a man ironing clothes on the back of a taxi in a busy urban street. The second image is a stylized logo featuring the letters "SGL" with a book and a computer icon incorporated into the design.
[6]:
terminate_process(vision_process)
[2025-07-19 07:58:50] Child process unexpectedly failed with exitcode=9. pid=615392
[2025-07-19 07:58:50] Child process unexpectedly failed with exitcode=9. pid=615144