Production AI inference: assess the operating case

OutsourcingVN is operated by Netbase JSC, which scopes and builds AI services of this kind, so we have an interest in your decision. The assessment method below applies whoever runs it. It is written for CTOs, platform owners and product leads deciding whether a model that works in a demo can carry live traffic.

Once the operating case is settled, Cloud Platform and DevOps Engineering builds the repeatable infrastructure, GPU or queue capacity and observability the workload runs on, often on AWS, the cloud Netbase's own delivered platform work already runs on. The reliability and incident readiness guide covers what happens when it fails in production, and the data security and compliance guide covers the access and data-location questions a hosted or self-hosted route raises.

What this guide covers

• What exactly is the workload?
• Which serving route fits?
• How do you run a bounded assessment?
• A worked example: one model, two workloads
• What has Netbase delivered around inference?
• Which questions should you ask a supplier?
• What failure modes should you watch for?
• Common questions
• How this guide is sourced
• Bring the workload description

What exactly is the workload?

Write the workload down before anyone names a platform. The description needs: input types and their size range, how requests arrive (steady, bursty, overnight batch), the longest useful wait for an answer, what happens downstream if the answer is late or wrong, the data classification, the model and prompt version, and the fallback when inference is unavailable.

Two numbers shape almost every later choice. The first is the deadline: a customer-facing chat needs the first words back quickly and can stream the rest, while an overnight document queue only needs to finish before the morning shift. The second is context length. Long prompts, long retrieved passages and long outputs consume memory on the serving hardware, reduce how many requests can be batched together, and raise the operating cost of each request. A workload that looks cheap on short test prompts can behave very differently once real documents are attached, which is why the retrieval guide and the multimodal document guide feed directly into sizing.

Which serving route fits?

Self-hosted engines such as vLLM expose an OpenAI-compatible HTTP interface and batch concurrent requests continuously, which is why they are a common choice when a team wants to run open models on its own hardware. Their documentation describes configuration, not your throughput. Only a load test with your prompts, context lengths and concurrency tells you how many GPUs the peak needs. The model selection guide covers how to keep the model choice reversible.

A worked example: one model, two workloads

A logistics company wants one model for two jobs: extracting fields from overnight delivery documents, and answering customer questions about shipment status in a web chat.

Walked through the table, the document job lands on batch processing. Its deadline is the 6 a.m. shift, so it can use a larger model, retry failures and send a sample of outputs to a reviewer before records update. The owner is the operations manager, and the key proof is idempotency: a retried document must never create a second record.

The chat job lands on a hosted API or managed endpoint. Its deadline is seconds, so the test measures time to first token under peak evening concurrency, and the acceptance cases include a customer asking about someone else's shipment (refuse), a tracking number that does not exist (say so, do not invent a status), and a complaint (hand to staff). The owner is the customer service lead. When the provider rate-limits, the chat shows a "we will reply by email" state instead of an error.

Same model family, two different routes, owners and acceptance rules. Treating them as one "AI platform" decision would have sized the chat for batch economics or the batch for chat latency.

Which questions should you ask a supplier?

What failure modes should you watch for?

How this guide is sourced

Statements about Netbase map to approved claims backed by attested company facts listed under Sources, in their registered wording. External guidance comes from the vLLM serving documentation and NIST's generative AI profile, each dated below. The worked example is illustrative and describes no client. Self-hosted serving appears here as a growth capability, offered through a scoped assessment and sized on your own load test. This page does not estimate capacity, availability or cost for your workload. See the methodology for how claims are reviewed.

Bring the workload description

Start with the workload sheet from this guide, a small approved case set and the deadline your users actually tolerate, then submit a project. The emerging technology adoption guide, the edge deployment assessment and the other guides cover neighbouring decisions, and an AI Workflow Blueprint is the assessment route when the workload is still unclear. OutsourcingVN is operated by Netbase JSC and is Netbase's own outsourcing-services platform.

Related services and solutions

Sources

• vLLM documentation, OpenAI-compatible server
• NIST, AI Risk Management Framework: Generative AI Profile