OpenAI APIs - Embedding#
SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the OpenAI API Reference.
This tutorial covers the embedding APIs for embedding models, such as
Launch A Server#
Launch the server in your terminal and wait for it to initialize. Remember to add --is-embedding to the command.
[1]:
from sglang.test.test_utils import is_in_ci
if is_in_ci():
from patch import launch_server_cmd
else:
from sglang.utils import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process
embedding_process, port = launch_server_cmd(
"""
python -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-7B-instruct \
--host 0.0.0.0 --is-embedding
"""
)
wait_for_server(f"http://localhost:{port}")
INFO 03-20 18:35:04 __init__.py:190] Automatically detected platform cuda.
The following error message 'operation scheduled before its operands' can be ignored.
[2025-03-20 18:35:10] server_args=ServerArgs(model_path='Alibaba-NLP/gte-Qwen2-7B-instruct', tokenizer_path='Alibaba-NLP/gte-Qwen2-7B-instruct', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='Alibaba-NLP/gte-Qwen2-7B-instruct', chat_template=None, completion_template=None, is_embedding=True, revision=None, host='0.0.0.0', port=32753, mem_fraction_static=0.88, max_running_requests=200, max_total_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, stream_interval=1, stream_output=False, random_seed=121136990, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=5, speculative_eagle_topk=4, speculative_num_draft_tokens=8, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, enable_flashinfer_mla=False, enable_flashmla=False, flashinfer_mla_disable_ragged=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False)
[2025-03-20 18:35:16] Downcasting torch.float32 to torch.float16.
INFO 03-20 18:35:22 __init__.py:190] Automatically detected platform cuda.
INFO 03-20 18:35:23 __init__.py:190] Automatically detected platform cuda.
The following error message 'operation scheduled before its operands' can be ignored.
The following error message 'operation scheduled before its operands' can be ignored.
[2025-03-20 18:35:34 TP0] Downcasting torch.float32 to torch.float16.
[2025-03-20 18:35:34 TP0] Overlap scheduler is disabled for embedding models.
[2025-03-20 18:35:34 TP0] Downcasting torch.float32 to torch.float16.
[2025-03-20 18:35:34 TP0] Init torch distributed begin.
[2025-03-20 18:35:34 TP0] Init torch distributed ends. mem usage=0.00 GB
[2025-03-20 18:35:34 TP0] Load weight begin. avail mem=59.71 GB
[2025-03-20 18:35:35 TP0] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/7 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 14% Completed | 1/7 [00:01<00:09, 1.53s/it]
Loading safetensors checkpoint shards: 29% Completed | 2/7 [00:03<00:09, 1.89s/it]
Loading safetensors checkpoint shards: 43% Completed | 3/7 [00:04<00:06, 1.55s/it]
Loading safetensors checkpoint shards: 57% Completed | 4/7 [00:06<00:05, 1.75s/it]
Loading safetensors checkpoint shards: 71% Completed | 5/7 [00:08<00:03, 1.85s/it]
Loading safetensors checkpoint shards: 86% Completed | 6/7 [00:10<00:01, 1.92s/it]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:12<00:00, 1.89s/it]
Loading safetensors checkpoint shards: 100% Completed | 7/7 [00:12<00:00, 1.83s/it]
[2025-03-20 18:35:48 TP0] Load weight end. type=Qwen2ForCausalLM, dtype=torch.float16, avail mem=47.49 GB, mem usage=12.22 GB.
[2025-03-20 18:35:48 TP0] KV Cache is allocated. #tokens: 20480, K size: 0.55 GB, V size: 0.55 GB
[2025-03-20 18:35:48 TP0] Memory pool end. avail mem=46.12 GB
[2025-03-20 18:35:48 TP0] max_total_num_tokens=20480, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=200, context_len=131072
[2025-03-20 18:35:49] INFO: Started server process [1414653]
[2025-03-20 18:35:49] INFO: Waiting for application startup.
[2025-03-20 18:35:49] INFO: Application startup complete.
[2025-03-20 18:35:49] INFO: Uvicorn running on http://0.0.0.0:32753 (Press CTRL+C to quit)
[2025-03-20 18:35:49] INFO: 127.0.0.1:38614 - "GET /v1/models HTTP/1.1" 200 OK
[2025-03-20 18:35:50] INFO: 127.0.0.1:38628 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-03-20 18:35:50 TP0] Prefill batch. #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:35:51] INFO: 127.0.0.1:38642 - "POST /encode HTTP/1.1" 200 OK
[2025-03-20 18:35:51] The server is fired up and ready to roll!
NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
We are running those notebooks in a CI parallel environment, so the throughput is not representative of the actual performance.
Using cURL#
[2]:
import subprocess, json
text = "Once upon a time"
curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \
-d '{{"model": "Alibaba-NLP/gte-Qwen2-7B-instruct", "input": "{text}"}}'"""
text_embedding = json.loads(subprocess.check_output(curl_text, shell=True))["data"][0][
"embedding"
]
print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
[2025-03-20 18:35:54 TP0] Prefill batch. #new-seq: 1, #new-token: 4, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:35:54] INFO: 127.0.0.1:38658 - "POST /v1/embeddings HTTP/1.1" 200 OK
Text embedding (first 10): [0.00830841064453125, 0.0006928443908691406, -0.008087158203125, -0.0006895065307617188, 0.0143890380859375, -0.0089874267578125, 0.01238250732421875, 0.0020923614501953125, 0.006198883056640625, -0.0030384063720703125]
Using Python Requests#
[3]:
import requests
text = "Once upon a time"
response = requests.post(
f"http://localhost:{port}/v1/embeddings",
json={"model": "Alibaba-NLP/gte-Qwen2-7B-instruct", "input": text},
)
text_embedding = response.json()["data"][0]["embedding"]
print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
[2025-03-20 18:35:54 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 3, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:35:54] INFO: 127.0.0.1:38668 - "POST /v1/embeddings HTTP/1.1" 200 OK
Text embedding (first 10): [0.00830841064453125, 0.0007052421569824219, -0.00809478759765625, -0.0006909370422363281, 0.01438140869140625, -0.00897216796875, 0.0123748779296875, 0.002086639404296875, 0.006198883056640625, -0.003055572509765625]
Using OpenAI Python Client#
[4]:
import openai
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
# Text embedding example
response = client.embeddings.create(
model="Alibaba-NLP/gte-Qwen2-7B-instruct",
input=text,
)
embedding = response.data[0].embedding[:10]
print_highlight(f"Text embedding (first 10): {embedding}")
[2025-03-20 18:35:54 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 3, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:35:54] INFO: 127.0.0.1:38682 - "POST /v1/embeddings HTTP/1.1" 200 OK
Text embedding (first 10): [0.00830841064453125, 0.0007052421569824219, -0.00809478759765625, -0.0006909370422363281, 0.01438140869140625, -0.00897216796875, 0.0123748779296875, 0.002086639404296875, 0.006198883056640625, -0.003055572509765625]
Using Input IDs#
SGLang also supports input_ids as input to get the embedding.
[5]:
import json
import os
from transformers import AutoTokenizer
os.environ["TOKENIZERS_PARALLELISM"] = "false"
tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-7B-instruct")
input_ids = tokenizer.encode(text)
curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \
-d '{{"model": "Alibaba-NLP/gte-Qwen2-7B-instruct", "input": {json.dumps(input_ids)}}}'"""
input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][
0
]["embedding"]
print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
[2025-03-20 18:36:00 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 3, token usage: 0.00, #running-req: 0, #queue-req: 0,
[2025-03-20 18:36:00] INFO: 127.0.0.1:34442 - "POST /v1/embeddings HTTP/1.1" 200 OK
Input IDs embedding (first 10): [0.00830841064453125, 0.0007052421569824219, -0.00809478759765625, -0.0006909370422363281, 0.01438140869140625, -0.00897216796875, 0.0123748779296875, 0.002086639404296875, 0.006198883056640625, -0.003055572509765625]
[6]:
terminate_process(embedding_process)