GPU inference on Azure Local
An inference service has to answer quickly, every time. We build the GPU layer, the model servers and the monitoring, and measure throughput before users arrive.
Scope of the work
- Select GPUs and machines for the target models and request volume
- Prepare hosts, drivers and GPU node pools
- Deploy model servers with health checks and scaling
- Load test for latency and throughput
- Monitor GPU use, queue length and errors
- An inference endpoint on your own hardware
- A load test report
- Dashboards and alerts for the service
Billed at $150.00 per hour. Ask Keith for an estimate.
What we check first.
Measure, do not assume
Tokens per second depend on the model, the quantization and the server. We test your combination.
Batching and concurrency
The same GPU serves far more users with the right server settings.
A single GPU is a single point of failure
If the service matters, it needs a second GPU on a second machine.
Four steps, written down.
A call with Keith
You describe the goal, the estate and the deadline. Keith turns it into a written scope with an hour estimate at the hourly rate.
A written design by Gus
Gus reviews the environment and gives you a written design with each decision and its reason.
Hands-on implementation
Gus implements in your tenant and your racks, remotely or on site, with every change logged as it is made.
Documents and a named contact
You receive the as-built record and the runbooks. Keith stays on as your account and project manager.
Asked before every engagement.
Which GPUs do you recommend?
It depends on the model size and the budget. We start from what Azure Local and the hardware vendor support, then size against your models.
Can small GPUs be useful?
Yes. Modest data center GPUs handle embedding, speech and smaller language models well.
What does the consulting cost?
Consulting is billed at $150.00 per hour. Keith sends a written hour estimate after the scoping call, and work starts when you approve it.
Often scoped together.
One rate. Two people.
Consulting is billed at $150.00 per hour. Keith runs sales, project management and your account. Gus does the architecture, the design and the implementation.
per consulting hour, for every consulting service