[Bug]: Embedded message arrays will hit 16MB BSON limit — guaranteed production outage #148

Area

Database / persistence

Description

Both Chat.js and DirectMessageThread.js use unbounded embedded arrays to store messages. MongoDB has a hard 16 MB document size limit. Active channels or DM threads will inevitably hit this limit, causing all write operations to that document to fail.

Files:

• server/src/models/Chat.js:9-17 — channels[].chat_details array grows without limit
• server/src/models/DirectMessageThread.js:5-13 — messages array grows without limit

Calculation

• Average message size: ~200 bytes (content + sender_id + timestamp + metadata)
• A moderately active channel with 100 messages/day = ~20KB/day
• In 2 years: ~73,000 messages = ~14.6 MB — dangerously close to the 16 MB limit
• A busy channel with 1000 messages/day: hits the limit in ~80 days

Impact

Once a channel or DM thread hits 16 MB:

1. Every message send/edit/delete fails with a MongoDB BSONObjectTooLarge error
2. The affected channel/DM becomes completely unusable
3. There is no cleanup or archival mechanism
4. Since messages are embedded arrays, there is no easy migration path without downtime

Related issues

• No pagination or cursor-based message fetching: get_messages and get_direct_messages return the ENTIRE message array
• Cache stores full message arrays — duplicates the memory problem in Redis
• No soft-delete or archival strategy: messages accumulate forever

Expected behavior

• Messages should be stored as separate documents in a messages collection (not embedded arrays)
• Use cursor-based pagination for fetching messages (limit 50-100 per request)
• Implement infinite scroll or load more in the frontend
• Add indexes on { channel_id, timestamp } for efficient ranged queries

Severity: CRITICAL — guaranteed production outage for active servers