WWNIT SOLUTIONS
RUEN

Hi! I'm Ivan.

I build services and automate processes. I help with infrastructure, launch and ongoing support.

Solutions

Service development, infrastructure and automation for a specific challenge.

Cases

Real challenges, the solutions and the project outcomes.

Process

From the first conversation to launch and support.

Journal

Engineering notes and articles from practice.

Research

Technical studies, experiments and scientific publications.

Resume

Work experience, skills and a PDF resume.

Have a project in mind?

Get in touch through Telegram or email.

WWNIT Solutions
RUEN

Cases

Back
WWN
IT SOLUTIONSPROJECT FILE02 / 05

LLMOPS · REFERENCE ARCHITECTURE

Preparing an AI assistant for company use

A reference design for an internal AI service: user and application access, load monitoring and response evaluation before updates.

Reference architecture. Deployment readiness and performance are evaluated in a pilot.

01

Challenge

A test chat is not yet an everyday work tool

An assistant may answer test questions well, but daily company use also needs access rules, load limits and error diagnostics. The team must also check, before an update, whether a new model or prompt gives worse answers.

02

Solution

A shared AI service with clear operating rules

Applications access models through a shared gateway that checks access and applies limits. Monitoring shows service health, and traces allow individual requests to be investigated. Updates are evaluated using an agreed set of questions.

03

In practice

Requests are served; updates are evaluated separately

User requests and version releases follow separate processes. Evaluating an update allows the team to assess a change before it reaches everyday use.

  1. An application sends a request through the gateway with access rules and limits.

  2. The model produces a response; metrics and traces help investigate errors and delays.

  3. Before an update, the team compares versions on test requests and evaluates quality, speed and resource use.

Technical diagram and tools
Technical diagramLOGICAL CONNECTIONS

→ action or data flow · — functional relationship. Repeated blocks refer to the same component.

Model request

  1. ApplicationAI assistant
    Request
  2. Kong EnterpriseAI gateway · access and limits
    Access and limits
  3. vLLMInference in Kubernetes

Load and errors

  1. vLLMInference in Kubernetes
    Metrics
  2. Prometheus / GrafanaMetrics and alerts

Response diagnostics

  1. ApplicationAI assistant
    Traces
  2. LangfuseTraces and evaluation

Change evaluation

  1. Git / CI/CDVersions and releases
    Version comparison
  2. LangfuseTraces and evaluation

Release

  1. Git / CI/CDVersions and releases
    After evaluation
  2. vLLMInference in Kubernetes

Reference architecture. Requests, telemetry and releases are shown separately. The metrics arrow shows data flow, not which side initiates the network request.

Application
AI assistant
Kong Enterprise
AI gateway · access and limits
vLLM
Inference in Kubernetes
Prometheus / Grafana
Metrics and alerts
Langfuse
Traces and evaluation
Git / CI/CD
Versions and releases

OIDC · AI Rate Limiting Advanced · Langfuse · Prometheus · Grafana

Pilot scenario

Reference design: a text assistant with vLLM in Kubernetes and a self-managed Kong Gateway Enterprise with licensed AI plugins. Corporate identity and application policies control access; the inference endpoint is not exposed directly. Langfuse collects application traces and evaluation results inside the same environment. A pilot validates quality, p95 latency, time to first token, GPU use and recovery after a failure. Token limits are not a GPU cost estimate. Model licensing, gateway subscription availability, hardware, data masking and retention must be agreed before implementation.

04

Pilot validation

Confirm that the design fits the company

The intended outcome is an internal AI service the team can operate and diagnose. This is a proposed design: answer quality, latency and running cost must be validated in a pilot using the company’s actual tasks.

  • Evaluate answers on an agreed set of requests.
  • Measure latency, GPU use and recovery after failure.
  • Agree access, licences, hardware and data retention rules.