OpenAI APIs - Vision

OpenAI APIs - Vision#

SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference. This tutorial covers the vision APIs for vision language models.

SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and more:

As an alternative to the OpenAI API, you can also use the SGLang offline engine.

Launch A Server#

Launch the server in your terminal and wait for it to initialize.

Remember to add --chat-template llama_3_vision to specify thevision chat template, otherwise, the server will only support text (images won’t be passed in), which can lead to degraded performance.

We need to specify --chat-template for vision language models because the chat template provided in Hugging Face tokenizer only supports text.

[1]:

from sglang.test.test_utils import is_in_ci

if is_in_ci():
    from patch import launch_server_cmd
else:
    from sglang.utils import launch_server_cmd

from sglang.utils import wait_for_server, print_highlight, terminate_process

vision_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
    --chat-template=llama_3_vision
"""
)

wait_for_server(f"http://localhost:{port}")

[2025-04-13 23:27:18] server_args=ServerArgs(model_path='meta-llama/Llama-3.2-11B-Vision-Instruct', tokenizer_path='meta-llama/Llama-3.2-11B-Vision-Instruct', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='meta-llama/Llama-3.2-11B-Vision-Instruct', chat_template='llama_3_vision', completion_template=None, is_embedding=False, revision=None, host='127.0.0.1', port=36614, mem_fraction_static=0.88, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, stream_interval=1, stream_output=False, random_seed=631966815, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, enable_llama4_multimodal=None, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, deepep_mode='auto', enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, enable_flashinfer_mla=False, enable_flashmla=False, flashinfer_mla_disable_ragged=False, warmups=None, n_share_experts_fusion=0, disable_shared_experts_fusion=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998, disaggregation_transfer_backend='mooncake', disable_fast_image_processor=False)
[2025-04-13 23:27:21] Use chat template for the OpenAI-compatible API server: llama_3_vision
[2025-04-13 23:27:29 TP0] Overlap scheduler is disabled for multimodal models.
[2025-04-13 23:27:29 TP0] Attention backend not set. Use flashinfer backend by default.
[2025-04-13 23:27:29 TP0] Automatically reduce --mem-fraction-static to 0.792 because this is a multimodal model.
[2025-04-13 23:27:29 TP0] Automatically turn off --chunked-prefill-size for multimodal model.
[2025-04-13 23:27:29 TP0] Init torch distributed begin.
[2025-04-13 23:27:30 TP0] Init torch distributed ends. mem usage=0.00 GB
[2025-04-13 23:27:30 TP0] Load weight begin. avail mem=78.58 GB
[2025-04-13 23:27:30 TP0] Ignore import error when loading sglang.srt.models.llama4.
[2025-04-13 23:27:31 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:00<00:03,  1.27it/s]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:01<00:02,  1.19it/s]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:02<00:01,  1.16it/s]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:03<00:00,  1.14it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:03<00:00,  1.47it/s]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:03<00:00,  1.32it/s]

[2025-04-13 23:27:35 TP0] Load weight end. type=MllamaForConditionalGeneration, dtype=torch.bfloat16, avail mem=55.29 GB, mem usage=23.29 GB.
[2025-04-13 23:27:35 TP0] KV Cache is allocated. #tokens: 20480, K size: 1.56 GB, V size: 1.56 GB
[2025-04-13 23:27:36 TP0] Memory pool end. avail mem=51.05 GB
[2025-04-13 23:27:36 TP0]

CUDA Graph is DISABLED.
This will cause significant performance degradation.
CUDA Graph should almost never be disabled in most usage scenarios.
If you encounter OOM issues, please try setting --mem-fraction-static to a lower value (such as 0.8 or 0.7) instead of disabling CUDA Graph.

[2025-04-13 23:27:37 TP0] max_total_num_tokens=20480, chunked_prefill_size=-1, max_prefill_tokens=16384, max_running_requests=200, context_len=131072
[2025-04-13 23:27:38] INFO:     Started server process [2793158]
[2025-04-13 23:27:38] INFO:     Waiting for application startup.
[2025-04-13 23:27:38] INFO:     Application startup complete.
[2025-04-13 23:27:38] INFO:     Uvicorn running on http://127.0.0.1:36614 (Press CTRL+C to quit)
[2025-04-13 23:27:39] INFO:     127.0.0.1:52246 - "GET /v1/models HTTP/1.1" 200 OK
[2025-04-13 23:27:39] INFO:     127.0.0.1:52258 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-04-13 23:27:39 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-04-13 23:27:41] INFO:     127.0.0.1:52270 - "POST /generate HTTP/1.1" 200 OK
[2025-04-13 23:27:41] The server is fired up and ready to roll!


NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.

Using cURL#

Once the server is up, you can send test requests using curl or requests.

[2]:

import subprocess

curl_command = f"""
curl -s http://localhost:{port}/v1/chat/completions \\
  -d '{{
    "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
    "messages": [
      {{
        "role": "user",
        "content": [
          {{
            "type": "text",
            "text": "What’s in this image?"
          }},
          {{
            "type": "image_url",
            "image_url": {{
              "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
            }}
          }}
        ]
      }}
    ],
    "max_tokens": 300
  }}'
"""

response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)


response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)

[2025-04-13 23:27:44 TP0] Prefill batch. #new-seq: 1, #new-token: 6462, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-04-13 23:27:45] INFO:     127.0.0.1:52278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"893a1dc4b20b4c4487596314a8b3b87d","object":"chat.completion","created":1744586864,"model":"meta-llama/Llama-3.2-11B-Vision-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"The image shows a man ironing clothes on the back of a yellow taxi cab on a city street.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":128009}],"usage":{"prompt_tokens":6462,"total_tokens":6484,"completion_tokens":22,"prompt_tokens_details":null}}
[2025-04-13 23:27:46 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 6461, token usage: 0.32, #running-req: 0, #queue-req: 0,
[2025-04-13 23:27:46 TP0] Decode batch. #running-req: 1, #token: 6474, token usage: 0.32, gen throughput (token/s): 4.65, #queue-req: 0,
[2025-04-13 23:27:46 TP0] Decode batch. #running-req: 1, #token: 6514, token usage: 0.32, gen throughput (token/s): 63.93, #queue-req: 0,
[2025-04-13 23:27:47 TP0] Decode batch. #running-req: 1, #token: 6554, token usage: 0.32, gen throughput (token/s): 63.09, #queue-req: 0,
[2025-04-13 23:27:48] INFO:     127.0.0.1:52284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"8024b686193d4c19a734e0490f7c3e03","object":"chat.completion","created":1744586865,"model":"meta-llama/Llama-3.2-11B-Vision-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"This image depicts a man ironing clothes on an ironing board that is propped up on the back of a yellow taxi cab. The man is wearing a yellow jacket and is standing in the middle of the street, with the ironing board and iron in front of him. The taxi behind him has a license plate that reads \"TAXI\" on the front and back, and there are other taxi cabs in the background. The image appears to be a humorous or ironic representation of a person doing laundry in an unusual location, possibly as a commentary on the busy and chaotic nature of city life.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":128009}],"usage":{"prompt_tokens":6462,"total_tokens":6585,"completion_tokens":123,"prompt_tokens_details":null}}

Using Python Requests#

[3]:

import requests

url = f"http://localhost:{port}/v1/chat/completions"

data = {
    "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
                    },
                },
            ],
        }
    ],
    "max_tokens": 300,
}

response = requests.post(url, json=data)
print_highlight(response.text)

[2025-04-13 23:27:48 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 6461, token usage: 0.32, #running-req: 0, #queue-req: 0,
[2025-04-13 23:27:48 TP0] Decode batch. #running-req: 1, #token: 6472, token usage: 0.32, gen throughput (token/s): 41.38, #queue-req: 0,
[2025-04-13 23:27:49 TP0] Decode batch. #running-req: 1, #token: 6512, token usage: 0.32, gen throughput (token/s): 64.04, #queue-req: 0,
[2025-04-13 23:27:49] INFO:     127.0.0.1:52286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
{"id":"42a7717ee8fd4655a60c18ed91cc59ff","object":"chat.completion","created":1744586868,"model":"meta-llama/Llama-3.2-11B-Vision-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"This image shows a man ironing clothes on the back of a yellow cab. The image shows a man standing on the street, wearing a yellow jacket, ironing a piece of clothing on an ironing board that is attached to the back of a yellow taxi cab, with another yellow taxi driving past.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":128009}],"usage":{"prompt_tokens":6462,"total_tokens":6524,"completion_tokens":62,"prompt_tokens_details":null}}

Using OpenAI Python Client#

[4]:

from openai import OpenAI

client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true"
                    },
                },
            ],
        }
    ],
    max_tokens=300,
)

print_highlight(response.choices[0].message.content)

[2025-04-13 23:27:49 TP0] Prefill batch. #new-seq: 1, #new-token: 11, #cached-token: 6451, token usage: 0.31, #running-req: 0, #queue-req: 0,
[2025-04-13 23:27:50] INFO:     127.0.0.1:52938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
The image depicts a man ironing clothes on the back of a yellow taxi cab in the middle of a city street.

Multiple-Image Inputs#

The server also supports multiple images and interleaved text and images if the model supports it.

[5]:

from openai import OpenAI

client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.2-11B-Vision-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true",
                    },
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png",
                    },
                },
                {
                    "type": "text",
                    "text": "I have two very different images. They are not related at all. "
                    "Please describe the first image in one sentence, and then describe the second image in another sentence.",
                },
            ],
        }
    ],
    temperature=0,
)

print_highlight(response.choices[0].message.content)

[2025-04-13 23:27:51 TP0] Prefill batch. #new-seq: 1, #new-token: 12894, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-04-13 23:27:51 TP0] Decode batch. #running-req: 1, #token: 12899, token usage: 0.63, gen throughput (token/s): 16.86, #queue-req: 0,
[2025-04-13 23:27:52 TP0] Decode batch. #running-req: 1, #token: 12939, token usage: 0.63, gen throughput (token/s): 63.09, #queue-req: 0,
[2025-04-13 23:27:52 TP0] Decode batch. #running-req: 1, #token: 12979, token usage: 0.63, gen throughput (token/s): 62.76, #queue-req: 0,
[2025-04-13 23:27:53] INFO:     127.0.0.1:52952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
The first image shows a man in a yellow shirt and shorts, standing on a sidewalk, holding a large orange and white sign that says "SGI" in bold letters. The man is wearing a yellow shirt and shorts, and the sign is held up high, with the letters "SGI" standing out prominently.

The second image shows a yellow taxi cab driving down the street, with the word "SGI" written on the side of the car in large letters. The taxi is driving away from the camera, and the word "SGI" is written in a bold, orange font.
[6]:

terminate_process(vision_process)

[2025-04-13 23:27:53] Child process unexpectedly failed with an exit code 9. pid=2793793

Chat Template#

As mentioned before, if you do not specify a vision model’s --chat-template, the server uses Hugging Face’s default template, which only supports text.

We list popular vision models with their chat templates:

Contents