ECHO – MULTIMODAL AI SYSTEM FOR THE INTERACTIVE EXPLORATION OF ANCIENT EGYPT

ECHO is a multimodal AI system designed to explore Ancient Egypt through computer vision, natural language processing, and generative AI.

The system allows users to upload images of Egyptian landmarks, statues, or hieroglyphs and receive intelligent recognition, historical explanations, interactive conversations, and generated visual storytelling.

This project was developed as a Graduation Project in Artificial Intelligence.

Project Overview

Ancient Egyptian history is rich but often difficult to explore interactively. ECHO bridges this gap by combining:

• Computer Vision
• NLP
• Generative AI

The system transforms static historical content into an intelligent interactive experience.

System Architecture

ECHO uses a modern Microservices Architecture to separate lightweight routing/CRUD operations from heavy AI model inferences.

flowchart LR
    %% Aesthetics
    classDef actor fill:none,stroke:none
    classDef ui fill:#bfdbfe,stroke:#3b82f6,stroke-width:2px,color:#000
    classDef gw fill:#1e293b,stroke:#cbd5e1,stroke-width:2px,color:#fff
    classDef pipe fill:#bfdbfe,stroke:#3b82f6,stroke-width:2px,color:#000
    classDef db fill:#bfdbfe,stroke:#3b82f6,stroke-width:2px,color:#000
    classDef ext fill:none,stroke:none

    Actor(["👤<br/>Actor"]):::actor

    subgraph Vercel [▲ Vercel]
        FE(("N")):::ui
    end
    style Vercel fill:#e0f2fe,stroke:#bae6fd,stroke-width:2px,color:#000

    BE["⚡"]:::gw

    subgraph Runpod [📦 runpod]
        Chat["Chatbot Pipeline"]:::pipe
        Hiero["Hieroglyphics<br/>Translation<br/>Pipeline"]:::pipe
        Video["Video Generation<br/>Pipeline"]:::pipe
        Rec["Entity Recognition<br/>Pipeline"]:::pipe
    end
    style Runpod fill:#e0f2fe,stroke:#bae6fd,stroke-width:2px,color:#000

    subgraph External [External Services]
        Groq["⚡ groq"]:::ext
        R2("☁️ Cloudflare R2 Storage"):::ext
        DB[("🐘 PostgreSQL")]:::db
    end
    style External fill:none,stroke:none,color:#000

    %% Connections
    Actor --- FE
    
    FE -- "API Request<br/>Data Result" --> BE

    BE -- "User Prompt<br/>Audio/Text Response" --> Chat
    BE -- "Inscription Image<br/>English Text" --> Hiero
    BE -- "Video Request<br/>MP4 Video" --> Video
    BE -- "Entity Image<br/>Entity Description" --> Rec

    Chat -- "Enhanced Prompt + Context<br/>LLM Response" --> Groq
    
    Chat -- "Query Entity Context<br/>Entity Context" --> DB
    Video -- "Fetch Images<br/>Image Files" --> R2
    Video -- "Query Images via Text Embeddings<br/>Images URL" --> DB
    Rec -- "Query Metadata<br/>Entity Metadata" --> DB

Loading

Main Components & Deployment:

• Frontend (frontend): Next.js application hosted on Vercel for edge delivery.
• Backend Microservices (Hosted on RunPod):
  • API Gateway (src/app): Built with FastAPI. Handles frontend authentication, database CRUD operations, and forwards AI-heavy requests to the dedicated microservices.
  • Recognition API (src/recognition_api): Dedicated microservice for Landmark and Statue recognition Using CNNs.
  • Chatbot API (src/chatbot_api): Dedicated microservice for RAG, Groq LLM streaming, and Text-To-Speech generation.
  • Video Generation API (src/video_generation_api): Dedicated microservice for automated historical video compilation.
  • Hieroglyph Detection API (src/hieroglyph_api): Dedicated microservice to detect, classify, and translate Ancient Egyptian hieroglyphs using CNNs and Transformers.
• Databases: PostgreSQL.
• Cloud Storage: Cloudflare R2 Storage.

Project Structure

ECHO/
|-- alembic/                   # Database Migrations
|-- data/                      # Local data sets & assets
|-- experiments/               # Notebooks & model experiments
|-- frontend/                  # Next.js Application
|-- infra/                     # Dockerfiles & infrastructure config
|-- requirements/              # Python dependencies & environments
|-- scripts/                   # Utility & automation scripts
|-- src/                       # Microservices Workspace
|   |-- app/                   # API Gateway & Orchestrator
|   |-- chatbot_api/           # Chatbot Microservice
|   |-- db/                    # Database connection & sessions
|   |-- db_models/             # SQLAlchemy ORM Models
|   |-- hieroglyph_api/        # Hieroglyph Translation Microservice
|   |-- ml_models/             # Machine Learning models & weights
|   |-- recognition_api/       # Landmark & Statue Recognition Microservice
|   |-- video_generation_api/  # Video Generation Microservice
|-- docker-compose.yml         # Docker Compose configuration for AI modules
|-- start_all.sh               # Shell script to start all services natively
|-- README.md

Installation & Running

Because every feature in ECHO is containerized, you can effortlessly run all backend AI modules using Docker.

1. Clone the repository

git clone https://github.com/karimtawfikk/ECHO.git
cd ECHO

2. Configure Environment

Ensure you have .env properly configured in the root directory based on .env.example, including your PostgreSQL connection string, Hugging Face Token, and all required API keys.

3. Running the AI Microservices (Docker)

All AI microservices (Recognition, Chatbot, Video, Hieroglyph) can be spun up at once:

docker compose up --build -d

Note: Ensure your Docker host has NVIDIA Container Toolkit installed to utilize GPUs.

4. Running the API Gateway (Native)

To run the main server orchestrator natively:

python -m venv venv
# Windows: venv\Scripts\activate
# Mac/Linux: source venv/bin/activate

pip install -r requirements/main.txt -r requirements/chatbot.txt -r requirements/video.txt -r requirements/hieroglyph.txt -r requirements/recognition.txt
uvicorn src.app.main:app --reload --host 0.0.0.0 --port 8010

(Alternatively, you can start all services natively by running ./start_all.sh on Linux/Mac)

5. Running the Frontend

Start the Next.js application:

cd frontend
npm install
npm run dev

---

Core Modules

1. Landmark & Statue Recognition

Users upload an image of a historical landmark or statue. The system extracts visual embeddings and retrieves structured metadata from PostgreSQL.

2. Historical Video Generation

The system uses text embeddings from generated scripts via LLM to retrieve relevant historical images and dynamically compiles them with generated Text-to-Speech (TTS) narration to create short educational videos based on verified historical scripts.

3. Conversational Historical Chatbot

Users can interact with the recognized entity through a conversational interface using Agentic RAG. The system grounds responses in stored metadata and maintains historical accuracy.

4. Hieroglyph Translation

Users upload an image containing hieroglyphs. The system detects symbols, classifies them, and generates structured translations via LLM reasoning.

API Routes & Endpoints

ECHO provides a unified API Gateway which exposes the following core endpoints (all prefixed with /api/v1):

Feature / Category	Endpoint (prefixed with /api/v1)	Method	Purpose
Recognition	/recognize/	POST	Submits an image for entity (landmark/statue) recognition.
Chatbot	/chat/init	POST	Initializes a new chat session.
Chatbot	/chat/chat	POST	Sends a message to the AI and receives a response.
Chatbot	/chat/info	GET	Retrieves session information.
Chatbot	/chat/transcribe	POST	Handles voice-to-text transcription.
Video Generation	/video/generate	POST	Triggers the generation of an educational video.
Video Generation	/video/status/{entity_name}	GET	Checks video generation status.
Video Generation	/video/stream/{entity_name}	GET	Streams the generated video.
Hieroglyphs Translation	/hieroglyphs/translate	POST	Translates an image containing hieroglyphs.
Hieroglyphs Translation	/hieroglyphs/translate/stream	POST	Streams the translation process.
Entities	/entities/trending	GET	Retrieves trending historical entities.
Entities	/entities/all	GET	Retrieves all entities.
Entities	/entities/details	GET	Retrieves specific entity details.
Assets & Users	/assets/r2/{key:path}	GET	Fetch assets from Cloudflare R2 storage.
Assets & Users	/assets/r2-history/{key:path}	GET	Fetch asset history from Cloudflare R2 storage.
Assets & Users	/assets/upload/history	POST	Stores asset history.
Assets & Users	/assets/delete-account/{user_id}	DELETE	Deletes user account and data.

Database Design

The system contains structured data about Landmarks, Pharaohs, User Profiles, Conversations, Chat Messages, Recognition History, and Translation History. Relationships are modeled using SQLAlchemy ORM and version-controlled using Alembic migrations, hosted on Supabase.

AI Techniques Used

• Convolutional Neural Networks (CNNs)
• Object Detection
• Transformers & Large Language Models (LLMs)
• Retrieval-Augmented Generation (RAG)
• Multimodal Embeddings & Vector Similarity Search
• Density-Based Spatial Clustering (DBSCAN)
• Text-to-Speech (TTS) and Speech-to-Text (STT) Synthesis