Retail Catalog Enrichment Blueprint

A GenAI-powered catalog enrichment system that transforms basic product images into comprehensive, rich catalog entries using NVIDIA Nemotron 3 Nano Omni for content analysis, Nemotron 3.5 Lightning for intelligent prompt planning, FLUX Kontext model for generating high-quality product variations, and TRELLIS model for 3D asset generation.

Demo

▶️ Watch the Demo Video

Architecture

Key Features

• AI-Powered Analysis: NVIDIA Nemotron 3 Nano Omni for intelligent omni-modal product understanding
• Smart Categorization: Automatic classification into predefined product categories
• Intelligent Prompt Planning: Context-aware image variation planning based on regional aesthetics
• Multi-Language Support: Generate product titles and descriptions in 10 regional locales
• Cultural Image Generation: Create culturally-appropriate product backgrounds (Spanish courtyards, Mexican family spaces, British formal settings)
• Quality Evaluation: Automated VLM-based quality assessment of generated images with detailed scoring
• 3D Asset Generation: Transform 2D product images into interactive 3D GLB models using Microsoft TRELLIS
• Product FAQ Generation: Automatically generate product FAQs from enriched catalog data, with optional product manual PDF upload for richer FAQs (up to 10) via stateless targeted RAG
• Policy Compliance: Upload policy PDFs and automatically check product listings against them using RAG + Milvus
• Protocol Schema Export: Export enriched product data as ACP (Agentic Commerce Protocol) and UCP (Unified Commerce Protocol) compliant schemas with LLM-extracted structured attributes
• Rich VLM Product JSON: Generate an image-grounded JSON object with detailed visual product attributes in a dedicated UI tab
• Product Web Insights: Run fixed Exa searches and a single Nemotron synthesis pass to summarize current product pros, cons, use cases, and online insights in a dedicated UI tab
• Modular API: Separate endpoints for VLM analysis, rich product JSON, FAQ generation, product web insights, image generation, 3D asset generation, and protocol schema export

Documentation

• API Documentation - Detailed API endpoints, parameters, and examples
• Docker Deployment Guide - Docker and Docker Compose setup instructions
• Product Requirements (PRD) - Product requirements and feature specifications
• Policy Compliance - How policy compliance checking works
• Product Manual for FAQs - How product manual PDFs enrich FAQ generation
• Product Web Insights - Exa + Nemotron product web research feature
• AI Agent Guidelines - Instructions for AI assistants working on this project

Tech Stack

Backend:

• FastAPI + Uvicorn
• Python 3.11+

Frontend:

• Next.js 15 with React 19
• TypeScript
• Kaizen UI (KUI) design system
• Model-viewer for 3D assets

AI Models:

• NVIDIA Nemotron 3 Nano Omni (omni-modal VLM)
• NVIDIA Nemotron 3.5 Lightning (prompt planning LLM)
• NVIDIA Embeddings (Policy Compliance)
• Exa API (external web search and retrieved content)
• FLUX models (image generation)
• Microsoft TRELLIS (3D generation)

Infrastructure:

• Docker & Docker Compose
• NVIDIA NIM containers
• HuggingFace model hosting
• Milvus vector database for policy PDF retrieval

Minimum System Requirements

Hardware Requirements

For self-hosting the NIM microservices locally, the following GPU requirements apply:

Model	Purpose	Minimum GPU	Recommended GPU
Nemotron 3 Nano Omni	Vision-Language / Omni-Modal Analysis	1× A100	1× H100
Nemotron 3.5 Lightning	Prompt Planning (LLM)	1× A100	1× H100
nv-embedqa	Embeddings (Policy Compliance)	1× A100	1× H100
FLUX Kontext Dev	Image Generation	1× H100	1× H100
Microsoft TRELLIS	3D Asset Generation	1× L40S	1× H100

Total recommended setup: 3× H100 + 1× L40S (or 4× H100 for uniform configuration). Embeddings model can be deploy on the same GPU as Flux or Trellis models.

Deployment Options

• Docker 28.0+
• Docker compose

Quick Start

Prerequisites

• Python 3.11+
• uv package manager
• NVIDIA API key for VLM/LLM services
• HuggingFace token for FLUX image generation
• Optional Exa API key for product web insights

Environment Setup

Copy the example env file and fill in your keys:

cp .env.example .env

Getting API Keys:

• NVIDIA API Key: Get one here
• HuggingFace Token: Get one here
• Exa API Key: Get one here if you want Web Insights enabled

The FLUX.1-Kontext-Dev NIM uses a model that is for non-commercial use. Contact sales@blackforestlabs.ai for commercial terms.

Make sure you have accepted https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev and https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev-onnx License Agreements and Acceptable Use Policy, check if your HF token has correct permissions.

Local Development (Without Docker)

1. Install uv (if not already installed):

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Create and activate virtual environment:

   uv venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3. Install dependencies:

   uv pip install -e .

4. Configure NVIDIA NIM endpoints:

   IMPORTANT: Self-Hosted NIMs Required

   For local development, you must self-host the following NVIDIA NIM containers:

   • Nemotron 3 Nano Omni (omni-modal VLM)
   • Nemotron 3.5 Lightning (prompt planning LLM)
   • FLUX Kontext dev (image generation)
   • TRELLIS (3D asset generation)

   Update the URLs in shared/config/config.yaml to point to your self-hosted NIM endpoints:

   vlm:
     url: "http://localhost:8001/v1"  # Your VLM NIM endpoint
     model: "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning"
   
   llm:
     url: "http://localhost:8002/v1"  # Your LLM NIM endpoint
     model: "nvidia/nemotron-3.5-lightning"
   
   flux:
     url: "http://localhost:8003/v1/infer"  # Your FLUX NIM endpoint
   
   trellis:
     url: "http://localhost:8004/v1/infer"  # Your TRELLIS NIM endpoint
   
   embeddings:
     url: "http://localhost:8005/v1" #Your Embeddings NIM endpoint
     model: "nvidia/nv-embedqa-e5-v5"

   See the Docker Deployment Guide for instructions on deploying these NIMs.

5. Run the backend:

   uvicorn --app-dir src backend.main:app --host 127.0.0.1 --port 8000 --reload

6. Run the frontend (optional):

   cd src/ui
   pnpm install
   pnpm dev

The frontend at http://localhost:3000.

Docker Deployment (Self-Hosted NIMs)

The Docker deployment includes all required self-hosted NVIDIA NIM containers (Nemotron 3 Nano Omni, Nemotron 3.5 Lightning, FLUX, and TRELLIS). If you want to use uploaded policy PDFs in the UI, start the companion Milvus stack from docker-compose.rag.yml as well. The shared/config/config.yaml is pre-configured with the correct service URLs for Docker networking.

Published ports bind to 127.0.0.1 by default because this repository is a reference blueprint without application-level authentication or rate limiting. Do not expose the stack directly to an untrusted network. A controlled remote deployment may set CATALOG_BIND_ADDRESS=0.0.0.0 only when an authenticated, TLS-enabled, rate-limited gateway or equivalent network controls protect every published service. See SECURITY.md for the supported deployment scope and security assumptions.

For complete Docker deployment instructions, see the Docker Deployment Guide.

Quick Docker Start:

1. Create .env file with required credentials:

   NGC_API_KEY=your_ngc_api_key_here
   HF_TOKEN=your_huggingface_token_here

   The local policy RAG stack provides development-only MinIO credentials when MINIO_ACCESS_KEY and MINIO_SECRET_KEY are omitted. Override both values for any shared or production-like deployment.

2. Create cache directories:

   export LOCAL_NIM_CACHE=~/.cache/nim
   mkdir -p "$LOCAL_NIM_CACHE"
   chmod a+w "$LOCAL_NIM_CACHE"

3. Create the shared Docker network:

   docker network create catalog-network || true

4. Start the policy RAG stack:

   docker compose -f docker-compose.rag.yml up -d

5. Start the application stack:

   docker compose up -d

6. Access the application:

   • Frontend: http://localhost:3000
   • Backend API through the gateway: http://localhost:3000/api
   • Health Check through the gateway: http://localhost:3000/api/health
   • Milvus: localhost:19530
   • MinIO Console: http://localhost:9001

API Endpoints

The system provides the following endpoints:

• POST /vlm/analyze - Fast VLM/LLM analysis
• POST /vlm/rich-product - Rich image-grounded product JSON from Nemotron 3 Nano Omni
• POST /vlm/faqs - Product FAQ generation (supports optional manual knowledge)
• POST /vlm/manual/extract - Extract knowledge from a product manual PDF for FAQ enrichment
• POST /research/product-insights - Source-backed product web insights dashboard using Exa + Nemotron
• POST /generate/variation - Image generation with FLUX
• POST /generate/3d - 3D asset generation with TRELLIS
• POST /protocols/generate - ACP & UCP protocol schema generation

Image Input Guidance

• Recommended image size: For best results, use product images that are ideally 500×500 pixels or higher (JPEG or PNG).

For detailed API documentation with request/response examples, see API Documentation.

License

GOVERNING TERMS: The Blueprint scripts are governed by Apache License, Version 2.0, and enables use of separate open source and proprietary software governed by their respective licenses: Nemotron-3-Nano-Omni-30B-A3B-Reasoning, Nemotron-3.5-Lightning-30B-A3B, nv-embedqa-e5-v5, FLUX.1-Kontext-Dev, and Microsoft TRELLIS.

ADDITIONAL INFORMATION: FLUX.1-Kontext-Dev license: https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev/blob/main/LICENSE.md.

Third-Party Community Consideration: The FLUX Kontext model is not owned or developed by NVIDIA. This model has been developed and built to a third-party’s requirements for this application and use case; see link to: black-forest-labs/FLUX.1-Kontext-dev Model Card - https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.