OME API

ModelType defines the architecture family of the model (e.g., "bert", "gpt2", "llama"). This value typically corresponds to the "model_type" field in a Hugging Face model's config.json. It is used to identify the transformer architecture and inform runtime selection and tokenizer behavior.

ModelFramework specifies the underlying framework used by the model, such as "ONNX", "TensorFlow", "PyTorch", "Transformer", or "TensorRTLLM". This value helps determine the appropriate runtime for model serving.

ModelArchitecture specifies the concrete model implementation or head, such as "LlamaForCausalLM", "GemmaForCausalLM", or "MixtralForCausalLM". This is often derived from the "architectures" field in Hugging Face config.json.

Quantization defines the quantization scheme applied to the model weights, such as "fp8", "fbgemm_fp8", or "int4". This influences runtime compatibility and performance.

ModelParameterSize indicates the total number of parameters in the model, expressed in human-readable form such as "7B", "13B", or "175B". This can be used for scheduling or runtime selection.

ModelCapabilities of the model, e.g., "TEXT_GENERATION", "TEXT_SUMMARIZATION", "TEXT_EMBEDDINGS"

Configuration of the model, stored as generic JSON for flexibility.

Storage configuration for the model

ModelExtension is the common extension of the model

MaxTokens is the maximum number of tokens that can be processed by the model

Additional metadata for the model

HuggingFaceSecretReference is a reference to a Kubernetes Secret containing the Hugging Face API key. The referenced Secret must reside in the same namespace as the BenchmarkJob. This field replaces the raw HuggingFaceAPIKey field for improved security.

Endpoint is the reference to the inference service to benchmark.

ServiceMetadata records metadata about the backend model server or service being benchmarked. This includes details such as server engine, version, and GPU configuration for filtering experiments.

Task specifies the task to benchmark, pattern: -to- (e.g., "text-to-text", "image-to-text").

TrafficScenarios contains a list of traffic scenarios to simulate during the benchmark. If not provided, defaults will be assigned via genai-bench.

NumConcurrency defines a list of concurrency levels to test during the benchmark. If not provided, defaults will be assigned via genai-bench.

MaxTimePerIteration specifies the maximum time (in minutes) for a single iteration. Each iteration runs for a specific combination of TrafficScenarios and NumConcurrency.

MaxRequestsPerIteration specifies the maximum number of requests for a single iteration. Each iteration runs for a specific combination of TrafficScenarios and NumConcurrency.

AdditionalRequestParams contains additional request parameters as a map.

Dataset is the dataset used for benchmarking. It is optional and only required for tasks other than "text-to-".

OutputLocation specifies where the benchmark results will be stored (e.g., object storage).

ResultFolderName specifies the name of the folder that stores the benchmark result. A default name will be assigned if not specified.

Pod defines the pod configuration for the benchmark job. This is optional, if not provided, default values will be used.

State represents the current state of the benchmark job: "Pending", "Running", "Completed", "Failed".

StartTime is the timestamp for when the benchmark job started.

CompletionTime is the timestamp for when the benchmark job completed, either successfully or unsuccessfully.

LastReconcileTime is the timestamp for the last time the job was reconciled by the controller.

FailureMessage contains any error messages if the benchmark job failed.

Details provide additional information or metadata about the benchmark job.

Minimum number of replicas, defaults to 1 but can be set to 0 to enable scale-to-zero.

Maximum number of replicas for autoscaling.

ScaleTarget specifies the integer target value of the metric type the Autoscaler watches for. concurrency and rps targets are supported by Knative Pod Autoscaler (https://knative.dev/docs/serving/autoscaling/autoscaling-targets/).

ScaleMetric defines the scaling metric type watched by autoscaler possible values are concurrency, rps, cpu, memory. concurrency, rps are supported via Knative Pod Autoscaler(https://knative.dev/docs/serving/autoscaling/autoscaling-metrics).

ContainerConcurrency specifies how many requests can be processed concurrently, this sets the hard limit of the container concurrency(https://knative.dev/docs/serving/autoscaling/concurrency).

TimeoutSeconds specifies the number of seconds to wait before timing out a request to the component.

CanaryTrafficPercent defines the traffic split percentage between the candidate revision and the last ready revision

Labels that will be add to the component pod. More info: http://kubernetes.io/docs/user-guide/labels

Annotations that will be add to the component pod. More info: http://kubernetes.io/docs/user-guide/annotations

The deployment strategy to use to replace existing pods with new ones. Only applicable for raw deployment mode.

Latest revision name that is in ready state

Latest revision name that is created

Previous revision name that is rolled out with 100 percent traffic

Latest revision name that is rolled out with 100 percent traffic

Traffic holds the configured traffic distribution for latest ready revision and previous rolled out revision.

URL holds the primary url that will distribute traffic over the provided traffic targets. This will be one the REST or gRPC endpoints that are available. It generally has the form http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}

REST endpoint of the component if available.

Addressable endpoint for the InferenceService

This spec provides a full PodSpec for the decoder component Allows complete customization of the Kubernetes Pod configuration including containers, volumes, security contexts, affinity rules, and other pod settings.

ComponentExtensionSpec defines deployment configuration like min/max replicas, scaling metrics, etc. Controls scaling behavior and resource allocation for the decoder component.

Runner container override for customizing the main container This is essentially a container spec that can override the default container Defines the main decoder container configuration, including image, resource requests/limits, environment variables, and command.

Leader node configuration (only used for MultiNode deployment) Defines the pod and container spec for the leader node that coordinates distributed token generation in multi-node deployments.

Worker nodes configuration (only used for MultiNode deployment) Defines the pod and container spec for worker nodes that perform distributed token generation tasks as directed by the leader.

URL represents the endpoint URL for the inference service.

APIFormat specifies the type of API, such as "openai" or "oci-cohere".

ModelName specifies the name of the model being served at the endpoint. Useful for endpoints that require model-specific configuration. For instance, for openai API, this is a required field in the payload

InferenceService holds a Kubernetes reference to an internal inference service.

Endpoint holds the details of a direct endpoint for an external inference service, including URL and API details.

This spec provides a full PodSpec for the engine component Allows complete customization of the Kubernetes Pod configuration including containers, volumes, security contexts, affinity rules, and other pod settings.

ComponentExtensionSpec defines deployment configuration like min/max replicas, scaling metrics, etc. Controls scaling behavior and resource allocation for the engine component.

Runner container override for customizing the engine container This is essentially a container spec that can override the default container Defines the main model runner container configuration, including image, resource requests/limits, environment variables, and command.

Leader node configuration (only used for MultiNode deployment) Defines the pod and container spec for the leader node that coordinates distributed inference in multi-node deployments.

Worker nodes configuration (only used for MultiNode deployment) Defines the pod and container spec for worker nodes that perform distributed processing tasks as directed by the leader.

Name of component to which the failure relates (usually Pod name)

High level class of failure

Detailed error message

Internal Revision/ID of model, tied to specific Spec contents

Time failure occurred or was discovered

Exit status from the last termination of the container

Reference to the base model that this weight is fine-tuned from

ModelType of the fine-tuned weight, e.g., "Distillation", "Adapter", "Tfew"

HyperParameters used for fine-tuning, stored as generic JSON for flexibility

ModelExtension is the common extension of the model

Configuration of the fine-tuned weight, stored as generic JSON for flexibility

Storage configuration for the fine-tuned weight

TrainingJobID is the ID of the training job that produced this weight

Name of the secret containing the Hugging Face API key. The secret must reside in the same namespace as the BenchmarkJob.

Name specifies the name of the inference service to benchmark.

Namespace specifies the Kubernetes namespace where the inference service is deployed. Cross-namespace references are allowed but require appropriate RBAC permissions.

Predictor defines the model serving spec It specifies how the model should be deployed and served, handling inference requests. Deprecated: Predictor is deprecated and will be removed in a future release. Please use Engine and Model fields instead.

Engine defines the serving engine spec This provides detailed container and pod specifications for model serving. It allows defining the model runner (container spec), as well as complete pod specifications including init containers, sidecar containers, and other pod-level configurations. Engine can also be configured for multi-node deployments using leader and worker specifications.

Decoder defines the decoder spec This is specifically used for PD (Prefill-Decode) disaggregated serving deployments. Similar to Engine in structure, it allows for container and pod specifications, but is only utilized when implementing the disaggregated serving pattern to separate the prefill and decode phases of inference.

Model defines the model to be used for inference, referencing either a BaseModel or a custom model. This allows models to be managed independently of the serving configuration.

Runtime defines the serving runtime environment that will be used to execute the model. It is an inference service spec template that determines how the service should be deployed. Runtime is optional - if not defined, the operator will automatically select the best runtime based on the model's size, architecture, format, quantization, and framework.

Router defines the router spec

KedaConfig defines the autoscaling configuration for KEDA Provides settings for event-driven autoscaling using KEDA (Kubernetes Event-driven Autoscaling), allowing the service to scale based on custom metrics or event sources.

Conditions for the InferenceService

• EngineRouteReady: engine route readiness condition;
• DecoderRouteReady: decoder route readiness condition;
• PredictorReady: predictor readiness condition;
• RoutesReady (serverless mode only): aggregated routing condition, i.e. endpoint readiness condition;
• LatestDeploymentReady (serverless mode only): aggregated configuration condition, i.e. latest deployment readiness condition;
• Ready: aggregated condition;

Addressable endpoint for the InferenceService

URL holds the url that will distribute traffic over the provided traffic targets. It generally has the form http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}

Statuses for the components of the InferenceService

Model related statuses

EnableKeda determines whether KEDA autoscaling is enabled for the InferenceService.

• true: KEDA will manage the autoscaling based on the provided configuration.
• false: KEDA will not be used, and autoscaling will rely on other mechanisms (e.g., HPA).

PromServerAddress specifies the address of the Prometheus server that KEDA will query to retrieve metrics for autoscaling decisions. This should be a fully qualified URL, including the protocol and port number.

Example: http://prometheus-operated.monitoring.svc.cluster.local:9090

CustomPromQuery defines a custom Prometheus query that KEDA will execute to evaluate the desired metric for scaling. This query should return a single numerical value that represents the metric to be monitored.

Example: avg_over_time(http_requests_total{service="llama"}[5m])

ScalingThreshold sets the numerical threshold against which the result of the Prometheus query will be compared. Depending on the ScalingOperator, this threshold determines when to scale the number of replicas up or down.

Example: "10" - The Autoscaler will compare the metric value to 10.

ScalingOperator specifies the comparison operator used by KEDA to decide whether to scale the Deployment. Common operators include:

• "GreaterThanOrEqual": Scale up when the metric is >= ScalingThreshold.
• "LessThanOrEqual": Scale down when the metric is <= ScalingThreshold.

This operator defines the condition under which scaling actions are triggered based on the evaluated metric.

Example: "GreaterThanOrEqual"

Pod specification for the leader node This overrides the main PodSpec when specified Allows customization of the Kubernetes Pod configuration specifically for the leader node.

Runner container override for customizing the main container This is essentially a container spec that can override the default container Provides fine-grained control over the container that executes the leader node's coordination logic.

How many copies of this predictor's models failed to load recently

Total number copies of this predictor's models that are currently loaded

DisplayName is the user-friendly name of the model

Whether the model is enabled or not

Vendor of the model, e.g., "NVIDIA", "Meta", "HuggingFace"

CompartmentID is the compartment ID of the model

Name of the format in which the model is stored, e.g., "ONNX", "TensorFlow SavedModel", "PyTorch", "SafeTensors"

Version of the model format. Used in validating that a runtime supports a predictor. It Can be "major", "major.minor" or "major.minor.patch".

Name of the library in which the model is stored, e.g., "ONNXRuntime", "TensorFlow", "PyTorch", "Transformer", "TensorRTLLM"

Version of the library. Used in validating that a runtime supports a predictor. It Can be "major", "major.minor" or "major.minor.patch".

Name of the model being referenced Identifies the specific model to be used for inference.

Kind of the model being referenced Defaults to ClusterBaseModel Specifies the Kubernetes resource kind of the referenced model.

APIGroup of the resource being referenced Defaults to ome.io Specifies the Kubernetes API group of the referenced model.

Optional FineTunedWeights references References to fine-tuned weights that should be applied to the base model.

High level state string: Pending, Standby, Loading, Loaded, FailedToLoad

Minimum size of the model in bytes

Maximum size of the model in bytes

Specific ClusterServingRuntime/ServingRuntime name to use for deployment.

Whether the available predictor endpoints reflect the current Spec or is in transition

State information of the predictor's model.

Details of last failure, when load of target model is failed or blocked.

Model copy information of the predictor's model.

LifeCycle is an enum of Deprecated, Experiment, Public, Internal

Status of the model weight

Name of the referenced object

Namespace of the referenced object

Image specifies the container image to use for the benchmark job.

List of environment variables to set in the container.

List of sources to populate environment variables in the container.

Pod volumes to mount into the container's filesystem.

Compute Resources required by this container. Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

If specified, the pod's tolerations.

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

If specified, the pod's scheduling constraints

List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes

List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes

List of initialization containers belonging to the pod. Init containers are executed in order prior to containers being started. If any init container fails, the pod is considered to have failed and is handled according to its restartPolicy. The name for an init container or normal container must be unique among all containers. Init containers may not have Lifecycle actions, Readiness probes, Liveness probes, or Startup probes. The resourceRequirements of an init container are taken into account during scheduling by finding the highest request/limit for each resource type, and then using the max of of that value or the sum of the normal containers. Limits are applied to init containers in a similar fashion. Init containers cannot currently be added or removed. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated.

List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated actions such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod's ephemeralcontainers subresource.

Restart policy for all containers within the pod. One of Always, OnFailure, Never. In some contexts, only a subset of those values may be permitted. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy

Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds.

Optional duration in seconds the pod may be active on the node relative to StartTime before the system will actively try to mark it failed and kill associated containers. Value must be a positive integer.

Set DNS policy for the pod. Defaults to "ClusterFirst". Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to 'ClusterFirstWithHostNet'.

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

DeprecatedServiceAccount is a deprecated alias for ServiceAccountName. Deprecated: Use serviceAccountName instead.

AutomountServiceAccountToken indicates whether a service account token should be automatically mounted.

NodeName indicates in which node this pod is scheduled. If empty, this pod is a candidate for scheduling by the scheduler defined in schedulerName. Once this field is set, the kubelet for this node becomes responsible for the lifecycle of this pod. This field should not be used to express a desire for the pod to be scheduled on a specific node. https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodename

Host networking requested for this pod. Use the host's network namespace. If this option is set, the ports that will be used must be specified. Default to false.

Use the host's pid namespace. Optional: Default to false.

Use the host's ipc namespace. Optional: Default to false.

Share a single process namespace between all of the containers in a pod. When this is set containers will be able to view and signal processes from other containers in the same pod, and the first process in each container will not be assigned PID 1. HostPID and ShareProcessNamespace cannot both be set. Optional: Default to false.

SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field.

ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod

Specifies the hostname of the Pod If not specified, the pod's hostname will be set to a system-defined value.

If specified, the fully qualified Pod hostname will be "...svc.". If not specified, the pod will not have a domainname at all.

If specified, the pod's scheduling constraints

If specified, the pod will be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.

If specified, the pod's tolerations.

HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts file if specified.

If specified, indicates the pod's priority. "system-node-critical" and "system-cluster-critical" are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default.

The priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority.

Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy.

If specified, all readiness gates will be evaluated for pod readiness. A pod is ready when all its containers are ready AND all conditions specified in the readiness gates have status equal to "True" More info: https://git.k8s.io/enhancements/keps/sig-network/580-pod-readiness-gates

RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the "legacy" RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class

EnableServiceLinks indicates whether information about services should be injected into pod's environment variables, matching the syntax of Docker links. Optional: Defaults to true.

PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset.

Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md

TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed.

If true the pod's hostname will be configured as the pod's FQDN, rather than the leaf name (the default). In Linux containers, this means setting the FQDN in the hostname field of the kernel (the nodename field of struct utsname). In Windows containers, this means setting the registry value of hostname for the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters to FQDN. If a pod does not have FQDN, this has no effect. Default to false.

Specifies the OS of the containers in the pod. Some pod and container fields are restricted if this is set.

If the OS field is set to linux, the following fields must be unset: -securityContext.windowsOptions

If the OS field is set to windows, following fields must be unset:

• spec.hostPID
• spec.hostIPC
• spec.hostUsers
• spec.securityContext.appArmorProfile
• spec.securityContext.seLinuxOptions
• spec.securityContext.seccompProfile
• spec.securityContext.fsGroup
• spec.securityContext.fsGroupChangePolicy
• spec.securityContext.sysctls
• spec.shareProcessNamespace
• spec.securityContext.runAsUser
• spec.securityContext.runAsGroup
• spec.securityContext.supplementalGroups
• spec.securityContext.supplementalGroupsPolicy
• spec.containers[*].securityContext.appArmorProfile
• spec.containers[*].securityContext.seLinuxOptions
• spec.containers[*].securityContext.seccompProfile
• spec.containers[*].securityContext.capabilities
• spec.containers[*].securityContext.readOnlyRootFilesystem
• spec.containers[*].securityContext.privileged
• spec.containers[*].securityContext.allowPrivilegeEscalation
• spec.containers[*].securityContext.procMount
• spec.containers[*].securityContext.runAsUser
• spec.containers[*].securityContext.runAsGroup

Use the host's user namespace. Optional: Default to true. If set to true or not present, the pod will be run in the host user namespace, useful for when the pod needs a feature only available to the host user namespace, such as loading a kernel module with CAP_SYS_MODULE. When set to false, a new userns is created for the pod. Setting false is useful for mitigating container breakout vulnerabilities even allowing users to run their containers as root without actually having root privileges on the host. This field is alpha-level and is only honored by servers that enable the UserNamespacesSupport feature.

SchedulingGates is an opaque list of values that if specified will block scheduling the pod. If schedulingGates is not empty, the pod will stay in the SchedulingGated state and the scheduler will not attempt to schedule the pod.

SchedulingGates can only be set at pod creation time, and be removed only afterwards.

ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume them by name.

This is an alpha field and requires enabling the DynamicResourceAllocation feature gate.

This field is immutable.

This field points to the location of the model which is mounted onto the pod.

Runtime version of the predictor docker image

Protocol version to use by the predictor (i.e. v1 or v2 or grpc-v1 or grpc-v2)

Container enables overrides for the predictor. Each framework will have different defaults that are populated in the underlying container spec.

Model spec for any arbitrary framework.

This spec is dual purpose.

1. Provide a full PodSpec for custom predictor. The field PodSpec.Containers is mutually exclusive with other predictors (i.e. TFServing).
2. Provide a predictor (i.e. TFServing) and specify PodSpec overrides, you must not provide PodSpec.Containers in this case.

Component extension defines the deployment configurations for a predictor

WorkerSpec for the predictor, this is used for multi-node serving without Ray Cluster

PodSpec defines the container configuration for the router

ComponentExtensionSpec defines deployment configuration like min/max replicas, scaling metrics, etc.

This is essentially a container spec that can override the default container

Additional configuration parameters for the runner This can include framework-specific settings

Container spec for the runner Provides complete Kubernetes container configuration for the primary execution container.

Engine specifies the backend model server engine. Supported values: "vLLM", "SGLang", "TGI".

Version specifies the version of the model server (e.g., "0.5.3").

GpuType specifies the type of GPU used by the model server. Supported values: "H100", "A100", "MI300", "A10".

GpuCount indicates the number of GPU cards available on the model server.

List of containers belonging to the pod. Containers cannot currently be added or removed. Cannot be updated.

List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

If specified, the pod's scheduling constraints

If specified, the pod's tolerations.

Labels that will be add to the pod. More info: http://kubernetes.io/docs/user-guide/labels

Annotations that will be add to the pod. More info: http://kubernetes.io/docs/user-guide/annotations

ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod

If specified, the pod will be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.

Use the host's ipc namespace. Optional: Default to false.

Set DNS policy for the pod. Defaults to "ClusterFirst". Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to 'ClusterFirstWithHostNet'.

Host networking requested for this pod. Use the host's network namespace. If this option is set, the ports that will be used must be specified. Default to false.

Name of the runtime being referenced Identifies the specific runtime environment to be used for model execution.

Kind of the runtime being referenced Defaults to ClusterServingRuntime Specifies the Kubernetes resource kind of the referenced runtime. ClusterServingRuntime is a cluster-wide runtime, while ServingRuntime is namespace-scoped.

APIGroup of the resource being referenced Defaults to ome.io Specifies the Kubernetes API group of the referenced runtime.

Model formats and version supported by this runtime

ModelSizeRange is the range of model sizes supported by this runtime

Set to true to disable use of this runtime

Router configuration for this runtime

Engine configuration for this runtime

Decoder configuration for this runtime

Supported protocol versions (i.e. openAI or cohere or openInference-v1 or openInference-v2)

PodSpec for the serving runtime

WorkerPodSpec for the serving runtime, this is used for multi-node serving without Ray Cluster

Path is the absolute path where the model will be downloaded and stored on the node.

SchemaPath is the path to the model schema or configuration file within the storage system. This can be used to validate the model or customize how it's loaded.

Parameters contain key-value pairs to override default storage credentials or configuration. These values are typically used to configure access to object storage or mount options.

StorageKey is the name of the key in a Kubernetes Secret used to authenticate access to the model storage. This key will be used to fetch credentials during model download or access.

StorageUri specifies the source URI of the model in a supported storage backend. Supported formats:

• OCI Object Storage: oci://n/{namespace}/b/{bucket}/o/{object_path}
• Persistent Volume: pvc://{pvc-name}/{sub-path}
• Vendor-specific: vendor://{vendor-name}/{resource-type}/{resource-path} This field is required.

NodeSelector defines a set of key-value label pairs that must be present on a node for the model to be scheduled and downloaded onto that node.

NodeAffinity describes the node affinity rules that further constrain which nodes are eligible to download and store this model, based on advanced scheduling policies.

TODO this field is being used as model format name, and this is not correct, we should deprecate this and use Name from ModelFormat Name of the model

ModelFormat of the model, e.g., "PyTorch", "TensorFlow", "ONNX", "SafeTensors"

DEPRECATED: This field is deprecated and will be removed in future releases.

Version of the model format. Used in validating that a runtime supports a predictor. It Can be "major", "major.minor" or "major.minor.patch".

ModelFramework of the model, e.g., "PyTorch", "TensorFlow", "ONNX", "Transformers"

ModelArchitecture of the model, e.g., "LlamaForCausalLM", "GemmaForCausalLM", "MixtralForCausalLM"

Quantization of the model, e.g., "fp8", "fbgemm_fp8", "int4"

Set to true to allow the ServingRuntime to be used for automatic model placement if this model format is specified with no explicit runtime.

Priority of this serving runtime for auto selection. This is used to select the serving runtime if more than one serving runtime supports the same model format. The value should be greater than zero. The higher the value, the higher the priority. Priority is not considered if AutoSelect is either false or not specified. Priority can be overridden by specifying the runtime in the InferenceService.

Size of the worker, this is the number of pods in the worker.

PodSpec for the worker

PodSpec for the worker Allows customization of the Kubernetes Pod configuration specifically for worker nodes.

Size of the worker, this is the number of pods in the worker. Controls how many worker pod instances will be deployed for horizontal scaling.

Runner container override for customizing the main container This is essentially a container spec that can override the default container Provides fine-grained control over the container that executes the worker node's processing logic.