[TEST-PERF-02] Load Testing of AI Gateways and Verifying Ollama CPU Limits

Testing: Load and Stability

Published

October 14, 2026

NoteTask Brief
  • Components: Full AI Circuit (Ports 11435, 11436, 11437, 11438).
  • Tools: Apache JMeter / k6 (JavaScript-based).
  • Task Type: QA / Performance Testing.
  • Status: Under Analysis

Task Description

Since the local machine learning models (Llama 3.2 Vision, Whisper, Qwen) are deployed in Docker utilizing CPU-bound computations, they hit a strict throughput threshold. It is necessary to conduct load testing across the AI gateways, identify the system crash point, and verify the behavioral stability of the strict timeouts (20 and 40 seconds).

Step-by-Step Instructions

  1. Preparing Test Samples: Assemble a set of static testing payloads: a single Base64-encoded receipt image (for the Vision gateway) and a single .wav audio file encoded in Base64 (for the STT gateway).
  2. Writing the k6 Script (Scenario Example): Develop a load testing script that simulates a gradual ramp-up of virtual users:
    • Stage 1: 1 virtual user (baseline benchmark).
    • Stage 2: 5 concurrent requests per second (nominal system load).
    • Stage 3: 20 concurrent requests per second (peak stress load).
  3. Target Metrics for Monitoring (Expected Result):
    • Verify that when the 40-second execution threshold is exceeded, the Python gateway does not crash with an internal Error 500 but instead reliably returns the configured fallback_response structure.
    • Record RAM consumption patterns and host machine CPU core utilization rates using the docker stats command line utility.