Deployment and Scaling

Going from Prototype to Production

There's a massive gap between an AI feature that works on your laptop and one that works reliably for thousands of users. Your prototype probably calls the API directly, has minimal error handling, and runs on a single server. Production needs to handle users hitting it simultaneously, APIs going down unexpectedly, traffic spikes after a product launch, and all of this while keeping response times fast and costs manageable.

This module walks you through closing that gap systematically, so you can ship with confidence.

The Production Readiness Checklist

Before deploying any AI feature, run through this checklist:

• Secrets management. All API keys are stored in environment variables or a secrets manager — never in code, config files, or Git.
• Error handling. Every API call has retry logic, timeouts, and graceful fallbacks.
• Rate limiting. Your application limits how many AI requests each user can make per minute/hour/day.
• Monitoring. You're tracking response times, error rates, token usage, and costs in real time.
• Fallback plan. If the AI API goes down completely, your app still works — maybe with reduced functionality, but it doesn't crash.
• Cost controls. Budget alerts are set, and there's a hard ceiling that prevents runaway spending.
• Logging. You're logging requests and responses (with sensitive data redacted) for debugging and evaluation.

If any of these items are missing, you're not ready for production. Fix them first.

Handling Concurrent Users

When one user hits your AI feature, it's simple — one request, one response. When 500 users hit it in the same minute, things get complicated fast.

The core challenge: AI API calls are slow compared to regular backend operations. A typical database query takes 5-50 milliseconds. An AI API call takes 1-10 seconds. If your server handles each request one at a time, user #500 is waiting a very long time.

The solution: async processing. Your backend should make AI API calls asynchronously, so it can handle many concurrent requests without blocking.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def handle_request(user_prompt):
    # This doesn't block — other requests can be handled while waiting
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_prompt}]
    )
    return response.choices[0].message.content

Using async clients means your server can juggle hundreds of in-flight API requests simultaneously, instead of processing them one by one.

Rate Limiting and Queuing

Rate limiting protects both you and the AI provider. Without it, a single overactive user (or a bot) could exhaust your API quota and break the feature for everyone else.

User-level rate limits. Limit each user to a reasonable number of requests — for example, 20 AI requests per minute. This prevents abuse while allowing normal usage.

Application-level rate limits. AI providers impose their own rate limits on your account. If you're approaching these limits, you need a queue system that holds excess requests and processes them as capacity becomes available.

# Simple queue-based approach
from redis import Redis
from rq import Queue

queue = Queue(connection=Redis())

def process_ai_request(user_id, prompt):
    # Check user rate limit
    if get_request_count(user_id, window="1m") > 20:
        return {"error": "Rate limit exceeded. Try again shortly."}

    # Queue the request instead of processing immediately
    job = queue.enqueue(call_ai_api, prompt, timeout=30)
    return {"job_id": job.id, "status": "processing"}

Why queuing matters: Without a queue, traffic spikes cause failures. With a queue, requests wait in line and get processed smoothly. The user might wait a few extra seconds during peak times, but they get a response instead of an error.

Fallback Strategies

What happens when the AI API goes down? This isn't hypothetical — every major AI provider has outages. Your product needs a plan.

Provider fallback. If OpenAI is down, route requests to Anthropic (or vice versa). This requires writing your integration to support multiple providers, but it's the most robust approach. The user never knows a failover happened.

Model fallback. If the premium model is unavailable or rate-limited, fall back to a cheaper, faster model. The response quality might be slightly lower, but it's infinitely better than an error message.

Cached response fallback. For common queries, return a cached response from a previous successful request. "Here's a recent answer to a similar question" is better than "Service unavailable."

Graceful degradation. If no AI response is possible, disable the AI feature and show the non-AI version of the experience. A search page without AI-powered summaries is still a functional search page.

async def get_ai_response(prompt):
    providers = [
        ("openai", call_openai),
        ("anthropic", call_anthropic),
    ]
    for name, provider_fn in providers:
        try:
            return await provider_fn(prompt)
        except Exception as e:
            log_error(f"{name} failed: {e}")
            continue

    # All providers failed — try cache
    cached = get_cached_response(prompt)
    if cached:
        return cached + "\n\n(Using cached response)"

    return "This feature is temporarily unavailable. Please try again."

Monitoring Latency and Errors

In production, you need visibility into how your AI features are performing in real time.

Latency tracking. Measure how long each AI request takes, from the moment the user clicks to the moment they see a response. Set alerts for when average latency exceeds your target (e.g., > 5 seconds). Track p50 (median), p95, and p99 latency — the average can look fine while 5% of users have a terrible experience.

Error rate monitoring. Track the percentage of requests that fail. A healthy AI feature should have an error rate under 1%. If it spikes above 5%, something is wrong and you need to investigate immediately.

Token usage tracking. Monitor tokens consumed per request. A sudden spike might indicate a bug (like accidentally including an entire document in every prompt) that's both degrading quality and running up costs.

User experience metrics. Track thumbs up/down ratios, regeneration rates, and feature abandonment. These tell you how users perceive quality, which automated metrics alone can't capture.

The Staged Rollout Approach

Don't flip a switch and expose your AI feature to all users at once. Use a staged rollout:

Stage 1: Internal testing (1-2 weeks). Your team uses the feature and reports issues. This catches the obvious problems.

Stage 2: Beta users (1-2 weeks). Roll out to 5-10% of users, ideally ones who've opted into beta testing. Monitor all metrics closely.

Stage 3: Gradual expansion. Increase to 25%, then 50%, then 100%, watching metrics at each stage. If something goes wrong, you can roll back to the previous stage quickly.

Stage 4: Full launch. Once metrics are stable across all users, you're in full production. Continue monitoring — AI features can degrade subtly over time as user behaviour evolves.

This staged approach means that if something goes wrong, it only affects a fraction of your users instead of everyone. It also gives you real production data to optimise against before hitting full scale.