Tunisia Job Recommendation System

A machine learning-powered job recommendation system that matches CVs with job postings from LinkedIn using advanced NLP techniques and similarity algorithms.

🎯 Project Overview

This system scrapes job postings from LinkedIn (Tunisia), extracts skills from job descriptions, processes uploaded CVs, and recommends the most relevant jobs using two different matching algorithms:

1. Embedding-based Similarity (Semantic matching using transformers)
2. Nominal Distance (Feature-based matching)

---

📸 Screenshots

Upload Interface

Users can upload their CV (PDF format), select the number of job recommendations, and choose between two matching algorithms:

Job Recommendations - AI Engineer CV

Recommended jobs for an AI Engineer profile using the embedding-based similarity method:

Job Recommendations - Software Engineer CV

Recommended jobs for a Software Engineer profile:

Job Recommendations - Accounting CV

Recommended jobs for an Accounting profile showing cross-domain matching capabilities:

---

🏗️ Architecture

Backend (ML_backend)

• Framework: FastAPI
• ML Model: sentence-transformers/all-MiniLM-L6-v2
• Data Processing: pandas, torch
• Web Scraping: Apify Client

Frontend (ML_frontend)

• Framework: Next.js with TypeScript
• UI Components: shadcn/ui
• Styling: TailwindCSS

---

📊 ML Pipeline

1. Job Scraping (scraper.py)

Process:

• Uses Apify's LinkedIn Jobs Scraper to collect job postings from Tunisia
• Scrapes 1000 jobs per run with configurable keywords and location
• Extracts job metadata: title, description, employment type, industries, job function, seniority level

Key Features:

• Automated scraping with infinite loop (runs every 24 hours)
• Duplicate removal
• Data stored in ScrapedData.csv

API Integration:

• External Skill Extraction API
• Processes job descriptions in batches of 5
• Extracts top 5 skills per job posting

Error Handling:

• Retry mechanism for failed skill extractions
• 2-minute cooldown between retries
• Handles API rate limits gracefully

---

2. External AI API: Skill Extraction Service

Deployed on: Google Cloud Run
This is a separate FastAPI service that uses Llama 3.1 405B (Meta's large language model) via Google Vertex AI to extract structured information from text.

Technology Stack:

• LLM Model: Llama 3.1 405B Instruct (via Vertex AI MaaS)
• Framework: FastAPI with async support
• PDF Processing: PyMuPDF (fitz)
• Async Processing: Python asyncio for concurrent requests

Key Implementation Details:

API Endpoints:

A. /extract-skills - Job Description Skill Extraction

Method: POST
Input: List of job description texts (batch processing)
Output: List of extracted skills for each description

Core Implementation:

@app.post("/extract-skills")
async def process_job_descriptions(job_description_texts: List[str]):
    llm = GenerativeModel(MODEL_NAME)
    
    # Process all descriptions concurrently using asyncio
    tasks = [extract_skills(text, llm) for text in job_description_texts]
    results = await asyncio.gather(*tasks)
    
    return results

async def extract_skills(job_text: str, model: GenerativeModel):
    prompt = f"""
    From the following job description, identify the top 5 technical skills.
    Return ONLY a numbered list of skills, no extra text or markdown.
    
    Job Description: {job_text}
    """
    
    generation_config = GenerationConfig(temperature=0.1, max_output_tokens=128)
    response = await model.generate_content_async(prompt, generation_config=generation_config)
    
    # Parse and clean skills
    raw_skills = response.text.strip().split('\n')
    skills_dict = {}
    for i, skill in enumerate(raw_skills):
        clean_skill = ''.join(filter(lambda char: char.isalnum() or char.isspace(), skill)).strip()
        if clean_skill:
            skills_dict[f"skill{i+1}"] = clean_skill
    
    return SkillsResponse(skills=skills_dict)

Key Features:

• Async processing: Uses asyncio.gather() for concurrent LLM calls
• Temperature 0.1: Low randomness for consistency
• Max tokens 128: Sufficient for 5 skills
• Error handling: Returns {"error": "message"} if extraction fails

Example Request:

[
  "Looking for Python developer with Django experience...",
  "Senior Java engineer needed for microservices..."
]

Example Response:

[
  {
    "skills": {
      "skill1": "Python",
      "skill2": "Django",
      "skill3": "REST APIs",
      "skill4": "PostgreSQL",
      "skill5": "Docker"
    }
  },
  {
    "skills": {
      "skill1": "Java",
      "skill2": "Spring Boot",
      "skill3": "Microservices",
      "skill4": "Kubernetes",
      "skill5": "AWS"
    }
  }
]

B. /extract-from-cv - CV Information Extraction

Method: POST
Input: PDF file (multipart/form-data)
Output: Structured candidate information

Core Implementation:

@app.post("/extract-from-cv")
async def extract_from_cv(file: UploadFile = File(...)):
    # Extract text from PDF using PyMuPDF
    pdf_bytes = await file.read()
    pdf_document = fitz.open(stream=pdf_bytes, filetype="pdf")
    cv_text = ""
    for page in pdf_document:
        cv_text += page.get_text()
    pdf_document.close()
    
    # Prompt for structured extraction
    prompt = f"""
    From the following CV text, extract the specified information.
    Fields: 'employmentType', 'industries', 'jobFunction', 'seniorityLevel', 
            'title', and 5 main technical skills.
    
    Return ONLY a valid JSON object. Infer missing fields from context.
    If cannot infer, use "any data".
    
    CV Text: {cv_text}
    """
    
    llm = GenerativeModel(MODEL_NAME)
    generation_config = GenerationConfig(temperature=0.1, max_output_tokens=512)
    response = await llm.generate_content_async(prompt, generation_config=generation_config)
    
    # Parse JSON response
    json_string = response.text.strip().replace("```json", "").replace("```", "").strip()
    extracted_data = json.loads(json_string)
    
    # Normalize industries if list
    if isinstance(extracted_data.get("industries"), list):
        extracted_data["industries"] = ", ".join(extracted_data["industries"])
    
    return CVResponse(**extracted_data)

Key Features:

• PDF Processing: PyMuPDF extracts text from all pages
• Temperature 0.1: Consistent extraction
• Max tokens 512: Enough for complete CV analysis
• Smart inference: LLM infers missing fields from context
• Normalization: Handles list responses (joins into strings)

Example Response:

{
  "employmentType": "Full-time",
  "industries": "Software Development, Cloud Computing",
  "jobFunction": "Engineering",
  "seniorityLevel": "Mid-Senior level",
  "title": "Senior Backend Developer",
  "skill1": "Python",
  "skill2": "FastAPI",
  "skill3": "PostgreSQL",
  "skill4": "Docker",
  "skill5": "AWS"
}

Why Llama 3.1 405B?

• Advanced reasoning: Better understanding of job descriptions and CVs
• Structured output: Reliable JSON generation
• Context awareness: Infers missing information intelligently
• Low temperature: Consistent, deterministic outputs
• Async processing: Handles multiple requests concurrently

---

3. Skill Extraction Pipeline Integration

In Scraper (scraper.py):

• Sends job descriptions to /extract-skills in batches of 5
• Uses asyncio for concurrent processing
• Cleans extracted skills by removing numbering (e.g., "1. Python" → "Python")

Data Cleaning:

• Fills missing skills with "No data"
• Uses mode imputation for missing categorical values
• Removes duplicates and handles infinite values

Output:

• Cleaned data saved to DATA.csv
• Contains: employmentType, industries, jobFunction, seniorityLevel, title, skill1-skill5

---

4. CV Processing in Main API

API Endpoint: /test (in main.py)

Input:

• PDF file (CV)
• Choice: 1 (embedding method) or 0 (nominal distance method)
• Amount: number of job recommendations to return

Process:

1. Validates PDF format
2. Forwards CV to external API: /extract-from-cv
3. Receives structured candidate data
4. Proceeds to matching algorithm based on choice
5. Returns top N job recommendations

---

5. Method 1: Embedding-Based Similarity (Semantic Matching)

Model: sentence-transformers/all-MiniLM-L6-v2

Process:

A. Job Embedding Generation (embedding.py)

1. Load job data from CSV
2. Combine all job fields into single text: "field1 | field2 | field3..."
3. Tokenize using AutoTokenizer
4. Generate embeddings using AutoModel
5. Apply mean pooling on token embeddings
6. Normalize embeddings (L2 normalization)
7. Store embeddings in 'embededData.csv'

B. CV Matching

1. Combine all CV fields into single text
2. Generate CV embedding using same model
3. Apply mean pooling and normalization
4. Compute cosine similarity between CV and all job embeddings
5. Rank jobs by similarity score (descending)
6. Return top N matches

Mean Pooling Function:

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

Advantages:

• Captures semantic meaning
• Understands context and synonyms
• Better for complex skill matching
• Language-agnostic similarity

---

6. Method 2: Nominal Distance (Feature-Based Matching)

Process:

1. Extract key features from CV and jobs:
   - employmentType
   - industries
   - jobFunction
   - seniorityLevel
   - title
   - skill1, skill2, skill3, skill4, skill5

2. Calculate nominal distance for each job:
   distance = 1 - (matching_features / total_features)
   
3. Sort jobs by distance (ascending - lower is better)
4. Return top N matches

Distance Calculation:

def nominal_distance(row, target, cols):
    m = 0  # matching count
    p = len(cols)  # total features
    
    for col in cols:
        if row[col] == target[col]:
            m += 1
    
    similarity = m / p
    distance = 1 - similarity
    return distance

Advantages:

• Fast computation
• Exact feature matching
• Interpretable results
• No model training required

---

🔧 Technical Details

Data Flow

LinkedIn Jobs → Apify Scraper → Raw Job Data
                                      ↓
                            Skill Extraction API
                                      ↓
                              Cleaned Job Data
                                      ↓
                        ┌─────────────┴─────────────┐
                        ↓                           ↓
              Generate Embeddings          Store Features
                        ↓                           ↓
                  embededData.csv              DATA.csv
                        ↓                           ↓
                        └─────────────┬─────────────┘
                                      ↓
                            User Uploads CV
                                      ↓
                          Extract CV Skills/Info
                                      ↓
                        ┌─────────────┴─────────────┐
                        ↓                           ↓
              Cosine Similarity          Nominal Distance
                        ↓                           ↓
                        └─────────────┬─────────────┘
                                      ↓
                          Top N Job Recommendations

Key Technologies

Machine Learning:

• Transformers: HuggingFace sentence-transformers
• PyTorch: Deep learning framework
• Embeddings: 386-dimensional vectors
• Similarity Metric: Cosine similarity

Data Processing:

• pandas: Data manipulation and CSV handling
• numpy: Numerical operations
• ast.literal_eval: Safe evaluation of string representations

Web Framework:

• FastAPI: Modern async API framework
• CORS Middleware: Cross-origin resource sharing
• File Upload: Multipart form data handling

---

📁 File Structure

ML_backend/
├── main.py                 # FastAPI application & recommendation logic
├── scraper.py             # Job scraping & skill extraction pipeline
├── embedding.py           # Embedding generation script
├── justfixing.py          # Data cleaning utility
├── requirements.txt       # Python dependencies
├── Dockerfile            # Docker container configuration
├── cloudbuild.yaml       # Google Cloud Build configuration
├── .env                  # Environment variables (not in git)
├── .env.example          # Environment variables template
├── ScrapedData.csv       # Raw scraped job data
├── DATA.csv              # Cleaned job data with skills
└── embededData.csv       # Job data with embeddings

---

🚀 Deployment

Local Development

# Install dependencies
pip install -r requirements.txt

# Run the API server
uvicorn main:app --reload --port 8080

Docker

# Build image
docker build -t ml-backend .

# Run container
docker run -p 8080:8080 --env-file .env ml-backend

Google Cloud Run

# Deploy using Cloud Build
gcloud builds submit --config cloudbuild.yaml --substitutions=_APIFY_API_KEY="your_api_key"

---

🔐 Environment Variables

APIFY_API_KEY=your_apify_api_key_here

---

📊 Data Schema

Job Data (DATA.csv)

Column	Type	Description
employmentType	string	Full-time, Part-time, Contract, etc.
industries	string	Industry sector
jobFunction	string	Job category/function
seniorityLevel	string	Entry, Mid, Senior, Executive
title	string	Job title
skill1-skill5	string	Top 5 required skills

Embedded Data (embededData.csv)

• All columns from DATA.csv
• embedding: 384-dimensional vector (list of floats)

---

🎯 API Endpoints

POST /test

Request:

• file: PDF file (CV)
• choice: int (1 = embedding, 0 = nominal distance)
• amount: int (number of recommendations)

Response:

[
  {
    "employmentType": "Full-time",
    "industries": "Information Technology",
    "jobFunction": "Engineering",
    "seniorityLevel": "Mid-Senior level",
    "title": "Senior Python Developer",
    "skill1": "Python",
    "skill2": "Django",
    "skill3": "REST APIs",
    "skill4": "PostgreSQL",
    "skill5": "Docker"
  },
  ...
]

---

🔄 Scraping Schedule

• Frequency: Every 24 hours
• Jobs per run: 1000
• Location: Tunisia
• Automatic: Runs in infinite loop

---

🧪 Model Performance

Embedding Model: sentence-transformers/all-MiniLM-L6-v2

• Dimensions: 386
• Max Sequence Length: 256 tokens
• Performance: ~14ms per sentence on CPU
• Quality: High semantic understanding

---

🛠️ Future Improvements

• Add user feedback loop for recommendation quality
• Implement hybrid approach (combine both methods)
• Add more job sources beyond LinkedIn
• Fine-tune embedding model on job-specific data
• Add job location filtering
• Implement caching for faster responses
• Add authentication and user profiles
• Track application success rates

---

📝 License

This project is for educational and research purposes.

---

👥 Contributors

Developed as part of a job recommendation system for the Tunisian job market.