InferNext

How it works

Three questions, in the order
they have to be asked.

Evaluate, benchmark, promote is not a slogan for a pipeline diagram. Each step answers something the next one assumes, and the whole point is that the thing you validate is the thing that ships.

Step one

Evaluate: does it work on your task?

A test endpoint is an isolated place to put a model while you are still deciding about it. You deploy an open-weight model to one, point your own eval harness at it, and take the result. It is meant to be disposable, and nothing else runs on it while your evaluation does.

The dataset and the scoring are yours. That is the part that matters: a published benchmark measures a general capability on someone else's data, and the number you actually need is accuracy on your inputs, judged the way your application judges them.

The output of this step is a decision about a specific model on a specific task, not a ranking.

Step two

Benchmark: what will it do under your traffic?

A model that passes your evals can still be wrong for you. Benchmarking drives the same endpoint with the traffic shape you actually see, and records what production monitoring would record.

  • Time to first token

    What the user experiences as responsiveness, measured across the distribution rather than as an average.

  • Throughput under concurrency

    Tokens per second as concurrent requests rise, which is where a configuration that looked fine on one request stops looking fine.

  • Cost per request

    Derived from the capacity the run actually held, so it can be compared against what the workload is worth to you.

  • The traffic shape itself

    Prompt lengths, output lengths and arrival pattern are part of the result. A number without the traffic that produced it is not portable.

Step three

Promote: ship the thing you measured.

Promotion moves the artifact that produced the results you accepted onto a dedicated or private endpoint. An artifact here is more than weights:

Test endpoint

Where the evals and the benchmark ran

The validated artifact

Weights, quantisation, context length, batching and scheduler settings, serving engine and version

Dedicated or private endpoint

Production, running the identical artifact

Nothing is rebuilt in between, because a rebuild is where drift gets in. If a setting was not part of what you validated, it is not part of what runs.

Where it runs

Our regions, or your cloud account.

InferNext regions
Capacity reserved for one customer and operated by us. Nothing else is scheduled onto it, so a queue you did not create cannot appear in your latency.
Your cloud account
The same stack deployed inside infrastructure you already own, for cases where model traffic and data are not allowed to leave it.
Underneath
Kubernetes for scheduling and lifecycle, open-source inference servers doing the serving. No proprietary runtime to port your model into.
Billing
By the hour for the capacity you hold, rather than per token.

Questions about a specific model or a specific traffic shape are the useful kind. Send them to founders@infernext.ai.