Version

Route to External Providers

On this page

This document is for an unreleased version of Modelplane.

This document applies to the Modelplane main branch and not to the latest release v0.4.

API: modelplane.ai/v1alpha1 · ModelEndpoint

A ModelEndpoint is a single reachable inference endpoint that a ModelService can route to. Modelplane creates one for each of your replicas automatically, but you can also create one by hand to point at an inference endpoint Modelplane doesn’t run, most often a SaaS provider like Together or Baseten. A service treats both the same, so you can front your own replicas and an external provider as one model, splitting traffic between them or keeping the provider as a backup.

Routing to an external provider

Create a ModelEndpoint with the five things the manifest numbers:

# Modelplane composes a ModelEndpoint per ModelReplica. Write one by hand only
# to register a model it doesn't run, like this one at Together, so a
# ModelService can fan over both.
apiVersion: modelplane.ai/v1alpha1
kind: ModelEndpoint
metadata:
  name: kimi-k2-together
  namespace: ml-team
  labels:
    # 1. A label of your own for a ModelService to select on. Any label works.
    modelplane.ai/external-provider: together
spec:
  # 2. Scheme and host, no path. An https origin gets TLS originated to it.
  #    This must be a name rather than an address: Envoy AI Gateway only applies
  #    per-backend model rewriting, credentials and priority failover when every
  #    backend in a route is addressed by hostname.
  origin: https://api.together.xyz
  api:
    # 3. The API this backend speaks, and the path it serves it under. The
    #    gateway translates an Anthropic client's request for an OpenAI
    #    backend, but not the reverse, so an Anthropic backend serves only
    #    Anthropic clients. Most providers serve /v1; Groq serves /openai/v1.
    schema: OpenAI
    prefix: /v1
  # 4. The name Together knows this model by. The gateway rewrites the request
  #    body's model to it, so a caller keeps naming the ModelService and gets
  #    back whichever model actually served.
  model: moonshotai/Kimi-K2-Instruct
  # 5. Together's API key, which the gateway attaches on the way out. The
  #    caller's identity doesn't reach Together: Modelplane strips that header
  #    for any backend it doesn't operate, while still recording the caller in
  #    the usage record.
  credential:
    method: APIKey
    apiKey:
      secretRef:
        name: together-api-key

Then point a ModelService at it. Selecting modelplane.ai/external-provider: together routes to the provider; adding a second entry for a deployment fronts both as one model, splitting its traffic between them by weight:

# A ModelService's endpoints list combines: one entry can select your own
# deployment's replicas while another selects an external ModelEndpoint, so a
# SaaS provider sits under the same model name as your own replicas. At the same
# priority the two split traffic by weight.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: kimi-k2
  namespace: ml-team
spec:
  endpoints:
  - name: kimi-k2
    selector:
      matchLabels:
        modelplane.ai/deployment: kimi-k2          # your own replicas
  - name: together
    selector:
      matchLabels:
        modelplane.ai/external-provider: together  # the endpoint above

To keep the provider as a backup instead, see failover tiers .

Anything speaking the OpenAI or Anthropic API works. origin is the scheme and host to reach it at, with no path; api.prefix is the path the provider serves those APIs under, and api.schema which of the two it speaks. Only those change between providers.