How Industry Makes LLM Apps Faster, Cheaper, and More Reliable

Building an LLM application is easy in the beginning.

You choose a model, write a prompt, send a request, and get a response. The first version may work perfectly on your laptop. It may even look impressive when you show it to someone.

Then real users arrive.

The application becomes slow. The API bill starts increasing. Some users receive unnecessarily long answers. Others complain that the quality is inconsistent. When multiple requests arrive together, the system starts struggling.

This is where LLM optimisation becomes important.

In production, the question is not only:

“Can the model answer the question?”

The real questions are:

- Can it answer quickly?

- Can it handle many users at the same time?

- Can we afford to run it every day?

- Can we maintain good quality?

- Can we understand what is slowing it down?

That is why companies optimise LLM applications. They try to balance quality, cost, latency, and throughput.

Understanding the important terms

Before discussing optimisation techniques, let’s understand the basic terms.

Cost

Cost is the amount of money spent to run the application.

When using an external API, you may pay based on input and output tokens. When hosting your own model, you may pay for GPUs, servers, storage, and networking.

A model that works well for ten test users may become extremely expensive when thousands of users start using it every day.

Latency

Latency is the waiting time for one request.

If a user asks a question and receives a response in two seconds, the experience feels comfortable. If the response takes fifteen seconds, the user may assume that the application is broken.

Latency includes:

- The time before the first token appears.

- The time required to generate the complete response.

- The time spent in retrieval, database calls, tool calls, or other services.

Throughput

Throughput means how much work the system can handle over a period of time.

Imagine a support assistant used by 500 employees. It may work perfectly when one person uses it. But what happens when 100 people send queries at the same time?

Throughput helps answer that question.

A system with good throughput can handle more requests without becoming extremely slow or failing completely.

Quality

Quality means whether the response is actually useful.

A fast and cheap answer is not valuable if it is wrong. Similarly, the most powerful model is not always the best choice if it is too slow or expensive for the use case.

Production optimisation is about finding the right balance.

The first optimisation: choose the right model

The first and most important optimisation is choosing the right model for the task.

Many beginners use the most powerful model for everything. This is understandable. If a model gives excellent answers, why not use it everywhere?

Because every task does not require the same level of intelligence.

Consider a customer-support application. It may need to:

- Identify whether a ticket is related to billing or delivery.

- Extract an order number.

- Rewrite a message politely.

- Answer a complicated policy question.

- Investigate a customer complaint across multiple systems.

These tasks are not equally difficult.

A small and fast model may be perfectly capable of classifying a ticket or extracting an order number. A larger model may be more useful for a complicated question that requires reasoning across multiple documents.

This is called model routing.

The application sends simple requests to a smaller and cheaper model, while difficult requests go to a more capable model.

For example:

- Simple classification: small model.

- Information extraction: small model.

- Short summarisation: medium model.

- Complex reasoning: larger model.

This approach can reduce cost and improve speed without sending every task to the most expensive model.

In an interview, saying “I will use the biggest model” is usually a weak answer. A stronger answer is:

“I will select or route models based on task complexity. Routine tasks can use smaller models, while difficult reasoning tasks can use larger models.”

That shows that you are thinking about the complete system, not just model quality.

The second optimisation: reduce unnecessary tokens

Tokens are the basic units that language models process. They may represent parts of words, complete words, or punctuation.

Tokens matter because they affect both cost and speed.

The more text you send to a model:

- The more you may pay.

- The longer the model may take.

- The more likely it is to get distracted by irrelevant information.

This is especially important in RAG applications.

A beginner may retrieve ten large chunks from a document and send all of them to the model. The model receives a lot of information, but only a small portion may be useful for answering the user’s question.

A better approach is to send:

- Fewer chunks.

- More relevant chunks.

- Only the amount of context required for the task.

Make prompts shorter

Many prompts contain repeated instructions, unnecessary background information, and long explanations that do not improve the result.

A prompt can often be made more efficient by:

- Removing repeated instructions.

- Writing direct requirements.

- Defining the expected output format.

- Avoiding unnecessary examples.

For instance, instead of using a long instruction every time, you can write:

“Answer using only the provided context. If the answer is not present, say that you do not know. Respond in five bullet points.”

This is shorter and easier for the model to follow.

Control the output

You do not always need a long answer.

If an application needs a category such as billing, technical, or account, there is no reason to allow the model to produce a long paragraph.

You can use:

- Output limits.

- Structured responses.

- JSON formats.

- Short-answer instructions.

- Fixed fields.

This reduces unnecessary output and makes the application easier to process.

The goal is not to use as few tokens as possible at any cost. The goal is to remove waste while keeping the information needed for a good answer.

The third optimisation: caching

Caching means saving a result so that you do not have to calculate it again.

You already experience caching in everyday technology. When you open a website you have visited before, some content may load faster because it was stored locally.

The same idea can be used in LLM applications.

Imagine an HR assistant that receives these questions every week:

- How many casual leaves can I take?

- What is the work-from-home policy?

- When will salaries be credited?

- How do I apply for reimbursement?

If the answer to a question has already been generated and the underlying information has not changed, the application may be able to reuse it.

This saves:

- An LLM call.

- Time.

- Money.

- System capacity.

Exact caching

Exact caching works when the same question appears again.

For example, if the question “What is the leave policy?” appears again, the stored response can be returned.

Semantic caching

Users do not always ask questions using the same words.

One person may ask:

“What is the work-from-home policy?”

Another may ask:

“Can employees work remotely?”

These questions may have a similar meaning. Semantic caching helps identify such similarities instead of checking only whether the wording is identical.

Caching must be used carefully, though.

If the information changes frequently, an old answer may become incorrect. A price, account balance, or delivery status should not be cached for too long.

A good production system defines:

- What can be cached.

- How long it can be cached.

- When the cache must be cleared.

The fourth optimisation: batching and throughput

Suppose a company wants to summarise 50,000 customer reviews. Sending each review separately may work, but it may also be slow and inefficient.

For tasks that do not require an immediate response, the system can process multiple requests together. This is called batching.

Instead of asking the model to process one item at a time, the application groups several items and processes them efficiently.

Batching is useful for:

- Document summarisation.

- Classification.

- Generating embeddings.

- Content moderation.

- Report generation.

- Background processing.

It improves throughput because the hardware can process more work during the same period.

But batching is not always suitable for live conversations.

If a user is waiting for an answer, the application should not wait until 20 other requests arrive before processing theirs. For real-time chat, low latency is usually more important. For overnight document processing, throughput may be more important than immediate response time.

This gives us an important production trade-off:

- Live chat prioritises latency.

- Bulk processing prioritises throughput.

- Both require different optimisation strategies.

The fifth optimisation: quantization

Quantization is mainly relevant when you host an open-source model yourself.

A model stores a large number of mathematical values. In simple terms, quantization stores these values using fewer bits.

This makes the model:

- Smaller.

- Easier to fit into GPU memory.

- Cheaper to run.

- Sometimes faster.

Imagine you want to host a model on your own server. The full version may require an expensive GPU. A quantized version may fit on a smaller and more affordable machine.

There is a trade-off, though.

Reducing numerical precision may cause a small drop in quality. The effect depends on the model and the task. For a simple classification task, the difference may be unimportant. For a difficult reasoning task, it may matter more.

That is why quantization should be tested rather than applied blindly.

You should check:

- Whether quality remains acceptable.

- How much memory is saved.

- How latency changes.

- Whether the cost reduction is worth it.

Choosing how to serve a model

You generally have three choices.

Use an external API

This is the easiest way to start.

You do not manage GPUs or model infrastructure. You send requests to a provider and pay based on usage.

This is convenient, but cost can increase as usage grows. You also have less control over the infrastructure.

Host the model yourself

This gives you more control over:

- The model.

- Privacy.

- Hardware.

- Deployment.

- Performance tuning.

But you also become responsible for servers, scaling, security, updates, and failures.

Run a smaller model locally

This may be useful when low latency, privacy, or offline usage is important.

The model may be less capable, but it can respond quickly without sending data to an external service.

The correct choice depends on the application. There is no single deployment option that is best for every company.

The sixth optimisation: faster inference

This is the more advanced part of LLM optimisation.

When a model generates text, it performs a large number of calculations. Several techniques help it avoid unnecessary work or use hardware more efficiently.

Streaming

Streaming allows the application to show the response as it is being generated.

Without streaming, the user may see nothing for five seconds and then receive the complete answer. With streaming, the first words may appear quickly, even though the complete answer still takes five seconds.

The total generation time may not change much, but the experience feels faster.

KV caching

During a conversation, the model repeatedly processes earlier parts of the conversation. KV caching allows the system to reuse some of the previous computation instead of starting from zero every time.

This is especially helpful for:

- Long conversations.

- Repeated prompts.

- Applications with large shared instructions.

Speculative decoding

Speculative decoding uses a smaller and faster model to guess possible next tokens. A larger model then checks those guesses.

When the guesses are correct, text can be generated faster without fully relying on the larger model for every token.

The basic idea is simple:

Use a fast model to suggest. Use the powerful model to verify.

You may not implement these techniques yourself, but as an AI engineer, you should understand the problems they solve.

Measure before you optimise

One of the biggest mistakes in LLM applications is optimising without measuring.

A developer may add caching, change the model, shorten the prompt, or quantize the model without knowing whether the change actually helped.

Before optimising, establish a baseline.

Measure:

- Average response time.

- p50 and p95 latency.

- Time to first token.

- Cost per request.

- Tokens used per request.

- Requests handled per second.

- GPU or server utilisation.

- Answer quality.

Why measure p95 latency?

Because averages can hide bad experiences. If most requests take two seconds but some users wait 15 seconds, the average may still look acceptable. p95 helps you understand the slower experiences that affect a significant portion of users.

Also, never measure only speed and cost. If optimisation reduces quality, it may not be a real improvement.

A useful optimisation result might look like this:

Before optimisation:

- Cost per request: ₹2.00.

- Average latency: 6 seconds.

- p95 latency: 10 seconds.

- Quality: 82%.

After optimisation:

- Cost per request: ₹0.80.

- Average latency: 2.8 seconds.

- p95 latency: 5 seconds.

- Quality: 81%.

This may be a good result if the one-point quality reduction is acceptable for the use case.

But for a medical or financial application, even a small quality drop may not be acceptable.

Optimisation always depends on the application.

A simple production example

Imagine an internal company assistant.

The first version uses a powerful model for every question. It retrieves large document chunks, sends them in a long prompt, and generates detailed answers every time.

During testing, everything looks fine.

After launch:

- Employees ask repeated questions.

- Some prompts contain unnecessary context.

- Response times increase.

- Monthly cost becomes difficult to predict.

- The system struggles during busy hours.

Now the team improves it step by step:

1. Simple questions are routed to a smaller model.

2. Prompts are shortened.

3. Retrieval returns only the most relevant chunks.

4. Frequently asked questions are cached.

5. Background document processing is batched.

6. A smaller self-hosted model handles selected routine tasks.

7. Streaming is added to improve the user experience.

8. Cost, latency, throughput, and answer quality are tracked continuously.

The result is not necessarily the most powerful system possible.

It is a system that is fast enough, affordable enough, and reliable enough for the actual users.

That is what production optimisation means.

How to explain this in an interview

If an interviewer asks:

“How would you optimise an LLM application?”

Do not immediately name five tools.

Start with the problem.

You can say:

“First, I would establish a baseline for quality, latency, throughput, and cost. Then I would choose the smallest model that meets the quality requirement, reduce unnecessary context, add caching for repeated requests, use batching for non-real-time workloads, and consider quantization or efficient serving if we host the model ourselves. After every change, I would evaluate quality again.”

That answer is much stronger than:

“I will use a faster model.”

It shows that you understand the trade-offs.

A good AI engineer does not blindly optimise for speed. They optimise for the needs of the product.

Final thought

LLM optimisation is not about making every application use the smallest model or the fewest tokens.

It is about removing unnecessary work.

Do not use a powerful model for a simple task.

Do not send irrelevant context.

Do not call the model again when the answer can be reused.

Do not process bulk workloads one request at a time.

Do not host a large model without considering memory and infrastructure cost.

Do not claim that an optimisation worked until you measure it.

In a demo, the model is the centre of attention.

In production, the entire system matters.

That is why industry cares about model routing, token optimisation, caching, batching, quantization, inference acceleration, and measurement. These concepts help turn an LLM application from an impressive prototype into a system that can actually serve users reliably.

And that is the real goal of optimisation:

Not the most powerful AI application.

The most useful one that can operate sustainably in the real world.