Comprehensive Meilisearch Index Configuration Guide for Efficient Bulk Ingestion

[bloomers7800]

This guide outlines best practices and actionable configuration blocks for setting up Meilisearch indexes to efficiently ingest hundreds of millions of documents. It is designed for automation by an AI agent and includes YAML/JSON examples, operational tips, and monitoring advice to ensure success in bulk imports.

1. Minimal Index Settings

searchableAttributes:

{
  "searchableAttributes": ["title", "description"]
}

Only include fields you need to search.

filterableAttributes:

{
  "filterableAttributes": ["status", "category"]
}

Limit to fields you will actually filter on.

displayedAttributes:

{
  "displayedAttributes": ["id", "title", "summary"]
}

Omit large or unnecessary fields from results to reduce payload size.

sortableAttributes (if needed):

{
  "sortableAttributes": ["date"]
}

---

2. Settings API: Apply via curl

Set searchableAttributes:

curl -X PATCH 'http://localhost:7700/indexes/YOUR_INDEX/settings' \
  --header 'Content-Type: application/json' \
  --data-binary '{"searchableAttributes": ["title", "description"]}'

Set filterableAttributes:

curl -X PATCH 'http://localhost:7700/indexes/YOUR_INDEX/settings' \
  --header 'Content-Type: application/json' \
  --data-binary '{"filterableAttributes": ["status", "category"]}'

Set displayedAttributes:

curl -X PATCH 'http://localhost:7700/indexes/YOUR_INDEX/settings' \
  --header 'Content-Type: application/json' \
  --data-binary '{"displayedAttributes": ["id", "title", "summary"]}'

Set sortableAttributes:

curl -X PATCH 'http://localhost:7700/indexes/YOUR_INDEX/settings' \
  --header 'Content-Type: application/json' \
  --data-binary '{"sortableAttributes": ["date"]}'

---

3. Batch & Parallel Ingestion Example

• Batch size: 10,000–100,000 (example: 50,000)
• Parallel workers: 4–8

---

4. Python Async Ingestion Script Example

import aiohttp
import asyncio

MEILI_URL = 'http://localhost:7700/indexes/YOUR_INDEX/documents'
API_KEY = 'your_api_key'
BATCH_SIZE = 50000
CONCURRENCY = 6

def chunker(seq, size):
    return (seq[pos:pos + size] for pos in range(0, len(seq), size))

async def send_batch(session, batch):
    async with session.post(MEILI_URL, json=batch, headers={'X-Meili-API-Key': API_KEY}) as resp:
        result = await resp.json()
        return result['taskUid']

async def ingest_all(documents):
    async with aiohttp.ClientSession() as session:
        batches = list(chunker(documents, BATCH_SIZE))
        sem = asyncio.Semaphore(CONCURRENCY)
        async def sem_send(batch):
            async with sem:
                return await send_batch(session, batch)
        tasks = [sem_send(batch) for batch in batches]
        return await asyncio.gather(*tasks)

---

5. Monitoring Commands

CPU & RAM:

htop

Disk I/O:

iotop

Swap:

free -h
swapon --show

---

6. Temporary Performance Settings

Disable ranking rules, synonyms, typo-tolerance:

curl -X PATCH 'http://localhost:7700/indexes/YOUR_INDEX/settings' \
  --header 'Content-Type: application/json' \
  --data-binary '{"rankingRules": ["words", "typo"], "synonyms": {}, "typoTolerance": false}'

---

7. Post-Ingestion Restore

Restore ranking rules, synonyms, typo-tolerance:

curl -X PATCH 'http://localhost:7700/indexes/YOUR_INDEX/settings' \
  --header 'Content-Type: application/json' \
  --data-binary '{"rankingRules": ["words", "typo", "proximity", "attribute", "sort", "exactness"], "typoTolerance": true}'

---

8. Troubleshooting Checklist

• Monitor /tasks endpoint for failed batches
• If memory nears 100%, reduce batch size and parallelism
• If task queue slows, wait for completion before sending more
• Ensure 2-3× expected index size is free on disk
• If OOM errors, add swap or split index

---

Copy-paste these code blocks and adjust for your project. For automation, script the above steps in your ingestion pipeline.