A comprehensive real estate intelligence platform for the UAE market. This platform scrapes property listings from major UAE property portals, processes and normalizes the data, calculates investment opportunity scores, and presents insights through an interactive dashboard.
- 🔍 Web Scraping: Automated scraping from Bayut and Dubizzle with anti-detection measures
- 🔄 ETL Pipeline: Extract, transform, and load property data with deduplication
- 📊 Scoring Engine: Multi-factor investment opportunity scoring algorithm
- 🗺️ Interactive Maps: Heatmap visualization of property opportunities
- 📈 Market Analytics: Real-time market statistics and trends
- 🐳 Containerized: Full Docker support with docker-compose
- 🚀 Production Ready: Nginx, health checks, rate limiting, caching
┌─────────────────────────────────────────────────────────────────┐
│ Frontend (Next.js) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │Dashboard │ │ Areas │ │Properties│ │ Map Heatmap │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │
└─────────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ API (FastAPI + Redis) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │ Areas │ │Properties│ │ Scores │ │ Market Stats │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │
└─────────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ PostgreSQL + PostGIS Database │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │ Areas │ │Properties│ │ Scores │ │ Price History │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │
└─────────────────────────────┬───────────────────────────────────┘
▲
│
┌─────────────────────────────┴───────────────────────────────────┐
│ Data Pipeline │
│ ┌──────────────┐ ┌───────────────┐ ┌───────────────────────┐ │
│ │ Scraper │─▶│ ETL │─▶│ Scoring Engine │ │
│ │ (Playwright)│ │ Pipeline │ │ (Multi-factor) │ │
│ └──────────────┘ └───────────────┘ └───────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
- Docker and Docker Compose
- Node.js 20+ (for local frontend development)
- Python 3.11+ (for local backend development)
- Mapbox API token (optional, for maps)
# Clone the repository
git clone https://github.com/yourusername/uae-property-platform.git
cd uae-property-platform
# Copy environment file
cp .env.example .env
# Edit .env with your configuration
# - Set POSTGRES_PASSWORD to a secure value
# - Add your NEXT_PUBLIC_MAPBOX_TOKEN for maps# Start core services (database, redis, api, frontend)
docker-compose up -d
# Wait for services to be healthy
docker-compose ps
# View logs
docker-compose logs -fThe database schema is automatically applied on first run. To seed initial area data:
docker-compose exec api python -c "
from db.schema import seed_areas
seed_areas()
"# Run the scraper
docker-compose --profile scraper up scraper
# Run ETL pipeline
docker-compose --profile etl up etl
# Calculate scores
docker-compose --profile scoring up scoring- Frontend Dashboard: http://localhost:3000
- API Documentation: http://localhost:8000/docs
- API Health Check: http://localhost:8000/health
uae-property-platform/
├── api/ # FastAPI backend
│ └── main.py # API endpoints
├── scraper/ # Web scraping module
│ ├── browser.py # Playwright browser management
│ ├── manager.py # Scraper orchestration
│ ├── main.py # CLI entry point
│ ├── sites/ # Site-specific scrapers
│ │ ├── base.py # Base scraper class
│ │ ├── bayut.py # Bayut.com scraper
│ │ └── dubizzle.py # Dubizzle.com scraper
│ └── utils/
│ └── storage.py # Data storage utilities
├── etl/ # ETL pipeline
│ ├── extract.py # Data extraction
│ ├── transform.py # Data transformation
│ ├── load.py # Database loading
│ └── pipeline.py # Pipeline orchestration
├── scoring/ # Scoring engine
│ ├── indicators.py # Indicator calculations
│ ├── model.py # Scoring model
│ └── run.py # CLI runner
├── db/ # Database
│ └── schema.sql # PostgreSQL schema
├── frontend/ # Next.js frontend
│ ├── src/
│ │ ├── app/ # App router pages
│ │ ├── components/ # React components
│ │ └── lib/ # Utilities and API client
│ └── package.json
├── docker/ # Docker configurations
│ ├── api.Dockerfile
│ ├── frontend.Dockerfile
│ ├── scraper.Dockerfile
│ ├── etl.Dockerfile
│ └── nginx.conf
├── docker-compose.yml # Docker compose config
├── requirements.txt # Python dependencies
└── README.md
| Variable | Description | Default |
|---|---|---|
POSTGRES_USER |
Database username | propintel |
POSTGRES_PASSWORD |
Database password | propintel_secret |
POSTGRES_DB |
Database name | propintel |
REDIS_URL |
Redis connection URL | redis://redis:6379/0 |
API_SECRET_KEY |
API secret key | - |
NEXT_PUBLIC_API_URL |
API URL for frontend | http://localhost:8000 |
NEXT_PUBLIC_MAPBOX_TOKEN |
Mapbox API token | - |
SCRAPER_HEADLESS |
Run browser headless | true |
SCRAPER_DELAY_MIN |
Min delay between requests (sec) | 2 |
SCRAPER_DELAY_MAX |
Max delay between requests (sec) | 5 |
The opportunity score is calculated using multiple weighted factors:
| Factor | Weight | Description |
|---|---|---|
| Listing Count | 0.20 | Market activity indicator |
| Demand Index | 0.25 | Supply/demand balance |
| Price Value | 0.25 | Price competitiveness |
| Permit Growth | 0.15 | Development activity |
| Price Trend | 0.15 | Market momentum |
Score Interpretation:
- 🟢 80-100: Excellent opportunity
- 🔵 60-79: Good opportunity
- 🟡 40-59: Average opportunity
- 🔴 0-39: Below average
For production, set up cron jobs or use a scheduler:
# Crontab example - Run daily at 2 AM
0 2 * * * cd /path/to/project && docker-compose --profile scraper up scraper && docker-compose --profile etl up etl && docker-compose --profile scoring up scoringOr use the included scheduler service (coming soon).
curl -sSL https://dokploy.com/install.sh | sh- Log into Dokploy dashboard
- Create new project "UAE Property Platform"
- Add services from docker-compose.yml
Add environment variables in Dokploy:
- Database credentials
- API keys
- Domain settings
# Deploy all services
dokploy deploy
# Or deploy individually
dokploy deploy api
dokploy deploy frontend- Add your domain in Dokploy
- Enable automatic SSL via Let's Encrypt
- Configure nginx routing
| Endpoint | Method | Description |
|---|---|---|
/areas/top |
GET | Top investment areas |
/area/{id} |
GET | Area details with stats |
/properties |
GET | Property listings (paginated) |
/properties/{id} |
GET | Property details |
/stats/summary |
GET | Market summary by city |
/stats/market |
GET | Overall market statistics |
/scores/latest |
GET | Latest opportunity scores |
/geo/heatmap |
GET | Heatmap data for maps |
/health |
GET | Service health check |
# Backend
cd uae-property-platform
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt
uvicorn api.main:app --reload
# Frontend
cd frontend
npm install
npm run dev# Python tests
pytest
# Frontend tests
cd frontend && npm testThis platform is intended for educational and research purposes. Ensure compliance with:
- Website terms of service
- Data protection regulations (UAE PDPL)
- Rate limiting and ethical scraping practices
Always respect robots.txt and implement appropriate delays between requests.
MIT License - see LICENSE file for details.
Contributions are welcome! Please read our contributing guidelines before submitting PRs.
Built with ❤️ for the UAE real estate community