SGLang Frontend Language#

SGLang frontend language can be used to define simple and easy prompts in a convenient, structured way.

Launch A Server#

Launch the server in your terminal and wait for it to initialize.

[1]:

from sglang import assistant_begin, assistant_end
from sglang import assistant, function, gen, system, user
from sglang import image
from sglang import RuntimeEndpoint
from sglang.lang.api import set_default_backend
from sglang.srt.utils import load_image
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import print_highlight, terminate_process, wait_for_server

server_process, port = launch_server_cmd(
    "python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}")
print(f"Server started on http://localhost:{port}")

[2025-11-14 15:11:49] INFO utils.py:148: Note: detected 112 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2025-11-14 15:11:49] INFO utils.py:151: Note: NumExpr detected 112 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
[2025-11-14 15:11:49] INFO utils.py:164: NumExpr defaulting to 16 threads.

[2025-11-14 15:11:55] INFO utils.py:148: Note: detected 112 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2025-11-14 15:11:55] INFO utils.py:151: Note: NumExpr detected 112 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
[2025-11-14 15:11:55] INFO utils.py:164: NumExpr defaulting to 16 threads.
[2025-11-14 15:11:57] WARNING server_args.py:1204: Attention backend not explicitly specified. Use fa3 backend by default.
[2025-11-14 15:11:57] INFO trace.py:69: opentelemetry package is not installed, tracing disabled
[2025-11-14 15:11:57] WARNING memory_pool_host.py:36: Current platform not support pin_memory
[2025-11-14 15:12:03] INFO utils.py:148: Note: detected 112 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2025-11-14 15:12:03] INFO utils.py:151: Note: NumExpr detected 112 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
[2025-11-14 15:12:03] INFO utils.py:164: NumExpr defaulting to 16 threads.
[2025-11-14 15:12:05] INFO trace.py:69: opentelemetry package is not installed, tracing disabled
[2025-11-14 15:12:05] WARNING memory_pool_host.py:36: Current platform not support pin_memory
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-11-14 15:12:09] INFO utils.py:148: Note: detected 112 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2025-11-14 15:12:09] INFO utils.py:151: Note: NumExpr detected 112 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
[2025-11-14 15:12:09] INFO utils.py:164: NumExpr defaulting to 16 threads.
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:02,  1.46it/s]
[2025-11-14 15:12:11] INFO trace.py:69: opentelemetry package is not installed, tracing disabled
[2025-11-14 15:12:11] WARNING memory_pool_host.py:36: Current platform not support pin_memory
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.33it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.26it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.30it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.31it/s]

Capturing batches (bs=1 avail_mem=62.72 GB): 100%|██████████| 3/3 [00:00<00:00,  9.72it/s]

NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
To reduce the log length, we set the log level to warning for the server, the default log level is info.
We are running those notebooks in a CI environment, so the throughput is not representative of the actual performance.

Server started on http://localhost:35054

Set the default backend. Note: Besides the local server, you may use also OpenAI or other API endpoints.

[2]:

set_default_backend(RuntimeEndpoint(f"http://localhost:{port}"))

Basic Usage#

The most simple way of using SGLang frontend language is a simple question answer dialog between a user and an assistant.

[3]:

@function
def basic_qa(s, question):
    s += system(f"You are a helpful assistant than can answer questions.")
    s += user(question)
    s += assistant(gen("answer", max_tokens=512))

[4]:

state = basic_qa("List 3 countries and their capitals.")
print_highlight(state["answer"])

Here are three countries with their capitals:

1. France - Paris
2. Japan - Tokyo
3. Brazil - Brasília

Multi-turn Dialog#

SGLang frontend language can also be used to define multi-turn dialogs.

[5]:

@function
def multi_turn_qa(s):
    s += system(f"You are a helpful assistant than can answer questions.")
    s += user("Please give me a list of 3 countries and their capitals.")
    s += assistant(gen("first_answer", max_tokens=512))
    s += user("Please give me another list of 3 countries and their capitals.")
    s += assistant(gen("second_answer", max_tokens=512))
    return s


state = multi_turn_qa()
print_highlight(state["first_answer"])
print_highlight(state["second_answer"])

Sure! Here are three countries along with their respective capitals:

1. **France** - Paris
2. **Australia** - Canberra
3. **Brazil** - Brasília

Of course! Here is another list of three countries and their capitals:

1. **Spain** - Madrid
2. **India** - New Delhi
3. **Japan** - Tokyo

Control flow#

You may use any Python code within the function to define more complex control flows.

[6]:

@function
def tool_use(s, question):
    s += assistant(
        "To answer this question: "
        + question
        + ". I need to use a "
        + gen("tool", choices=["calculator", "search engine"])
        + ". "
    )

    if s["tool"] == "calculator":
        s += assistant("The math expression is: " + gen("expression"))
    elif s["tool"] == "search engine":
        s += assistant("The key word to search is: " + gen("word"))


state = tool_use("What is 2 * 2?")
print_highlight(state["tool"])
print_highlight(state["expression"])

calculator

2 * 2. You don't really need a calculator for this, as it's a simple multiplication, but here’s the calculation:

\[ 2 * 2 = 4 \]

So, 2 * 2 equals 4.

Parallelism#

Use fork to launch parallel prompts. Because sgl.gen is non-blocking, the for loop below issues two generation calls in parallel.

[7]:

@function
def tip_suggestion(s):
    s += assistant(
        "Here are two tips for staying healthy: "
        "1. Balanced Diet. 2. Regular Exercise.\n\n"
    )

    forks = s.fork(2)
    for i, f in enumerate(forks):
        f += assistant(
            f"Now, expand tip {i+1} into a paragraph:\n"
            + gen("detailed_tip", max_tokens=256, stop="\n\n")
        )

    s += assistant("Tip 1:" + forks[0]["detailed_tip"] + "\n")
    s += assistant("Tip 2:" + forks[1]["detailed_tip"] + "\n")
    s += assistant(
        "To summarize the above two tips, I can say:\n" + gen("summary", max_tokens=512)
    )


state = tip_suggestion()
print_highlight(state["summary"])

1. **Balanced Diet**:
- **Importance**: Crucial for overall health and well-being.
- **Key Components**: A mix of vegetables, fruits, whole grains, lean proteins, and healthy fats.
- **Benefits**: Provides essential nutrients, supports immune function, helps manage weight, and reduces the risk of chronic diseases.
- **Additional Tips**: Pay attention to portion sizes, limit processed foods and sugars, and enjoy a variety of flavors and textures.

2. **Regular Exercise**:
- **Importance**: Crucial for maintaining overall health and wellness.
- **Key Components**: Activities such as walking, cycling, or swimming, which can be moderate in intensity.
- **Benefits**: Supports healthy weight, strengthens muscles and bones, improves cardiovascular health, and enhances mental health.
- **Additional Tips**: Consistency is key, and combining different types of exercises (aerobic, strength training, flexibility) can provide a well-rounded fitness routine.

By integrating these two tips into your daily routine, you can significantly improve your overall health and quality of life.

Constrained Decoding#

Use regex to specify a regular expression as a decoding constraint. This is only supported for local models.

[8]:

@function
def regular_expression_gen(s):
    s += user("What is the IP address of the Google DNS servers?")
    s += assistant(
        gen(
            "answer",
            temperature=0,
            regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)",
        )
    )


state = regular_expression_gen()
print_highlight(state["answer"])

208.67.222.222

Use regex to define a JSON decoding schema.

[9]:

character_regex = (
    r"""\{\n"""
    + r"""    "name": "[\w\d\s]{1,16}",\n"""
    + r"""    "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n"""
    + r"""    "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n"""
    + r"""    "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n"""
    + r"""    "wand": \{\n"""
    + r"""        "wood": "[\w\d\s]{1,16}",\n"""
    + r"""        "core": "[\w\d\s]{1,16}",\n"""
    + r"""        "length": [0-9]{1,2}\.[0-9]{0,2}\n"""
    + r"""    \},\n"""
    + r"""    "alive": "(Alive|Deceased)",\n"""
    + r"""    "patronus": "[\w\d\s]{1,16}",\n"""
    + r"""    "bogart": "[\w\d\s]{1,16}"\n"""
    + r"""\}"""
)


@function
def character_gen(s, name):
    s += user(
        f"{name} is a character in Harry Potter. Please fill in the following information about this character."
    )
    s += assistant(gen("json_output", max_tokens=256, regex=character_regex))


state = character_gen("Harry Potter")
print_highlight(state["json_output"])

{
"name": "Harry Potter",
"house": "Gryffindor",
"blood status": "Half-blood",
"occupation": "student",
"wand": {
"wood": "Holmwood",
"core": "Phoenix Feather",
"length": 11.25
},
"alive": "Alive",
"patronus": "Stag",
"bogart": "Evil smiling pin"
}

Batching#

Use run_batch to run a batch of prompts.

[10]:

@function
def text_qa(s, question):
    s += user(question)
    s += assistant(gen("answer", stop="\n"))


states = text_qa.run_batch(
    [
        {"question": "What is the capital of the United Kingdom?"},
        {"question": "What is the capital of France?"},
        {"question": "What is the capital of Japan?"},
    ],
    progress_bar=True,
)

for i, state in enumerate(states):
    print_highlight(f"Answer {i+1}: {states[i]['answer']}")

100%|██████████| 3/3 [00:00<00:00, 35.92it/s]

Answer 1: The capital of the United Kingdom is London.

Answer 2: The capital of France is Paris.

Answer 3: The capital of Japan is Tokyo.

Streaming#

Use stream to stream the output to the user.

[11]:

@function
def text_qa(s, question):
    s += user(question)
    s += assistant(gen("answer", stop="\n"))


state = text_qa.run(
    question="What is the capital of France?", temperature=0.1, stream=True
)

for out in state.text_iter():
    print(out, end="", flush=True)

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of France?<|im_end|>
<|im_start|>assistant
The capital of France is Paris.<|im_end|>

Complex Prompts#

You may use {system|user|assistant}_{begin|end} to define complex prompts.

[12]:

@function
def chat_example(s):
    s += system("You are a helpful assistant.")
    # Same as: s += s.system("You are a helpful assistant.")

    with s.user():
        s += "Question: What is the capital of France?"

    s += assistant_begin()
    s += "Answer: " + gen("answer", max_tokens=100, stop="\n")
    s += assistant_end()


state = chat_example()
print_highlight(state["answer"])

The capital of France is Paris.

[13]:

terminate_process(server_process)

SGLang Frontend Language

Contents