[TEST-INTEGRATION-01] Integration Testing of the Ingestion Pipeline with AI Mocks and Kafka

Testing: Backend and Queues

NoteTask Brief

• Components: product_process_service (Go), ai_receipt_vision_gateway (Python).
• Tools: Go test, Testcontainers (for Kafka/Postgres), WireMock (for Ollama mock).
• Task Type: QA / Backend Automation.
• Status: Under Analysis

Task Description

It is necessary to write integration tests to validate the end-to-end product processing pipeline. The tests must guarantee that when media data is transmitted, the Go service interacts properly with the AI gateways, correctly pushes messages to Kafka, and commits the final result into PostgreSQL.

Step-by-Step Instructions

1. Environment Isolation (Testcontainers): Within the Go test file, configure the automated launch of clean, temporary PostgreSQL and Kafka Docker containers before the execution of the test suite begins.
2. Creating AI Gateway Mocks (WireMock): Instead of invoking the actual Llama model (which has a long execution latency), configure WireMock on ports 11435 and 11437 to emulate three specific responses:
   • Success: Return a valid product JSON payload within 100 ms.
   • Timeout (Fallback Test): Delay the response for 45 seconds to validate the VISION_TIMEOUT trigger.
   • Profanity/Explicit Content: Return an invalid status structure from the profanity filter.
3. Data Consistency Verification:
   • Write a test case that verifies that upon receiving a successful response from the AI mock, the message is accurately delivered to the b2b_products_intake Kafka topic.
   • Verify that the Go background worker successfully consumes the message from Kafka and performs an INSERT statement into the digital_employees table.