Add a Monolog log formatter optimized for CloudWatch

[mnapoli]

Edit: moved the class to https://github.com/brefphp/monolog-bridge

TODO:

• Create PR in https://github.com/brefphp/laravel-bridge to install the package by default Require bref/monolog-bridge laravel-bridge#184
• Create PR in https://github.com/brefphp/symfony-bridge to install the package by default Require bref/monolog-bridge symfony-bridge#70

The goal of this is to improve the overall logs experience with CloudWatch.

The usual format is to log unstructured text. This sucks because:

• we can't filter by log level (e.g. look for errors)
• we can't easily filter and see all logs of a request/invocation via the AWS request ID (e.g. as log metadata)
• some log messages are split across multiple lines
• exception traces especially are split across loads of lines, not easy to navigate in CloudWatch or other line-based viewers

Here's an example:

A better approach is to switch to JSON-formatted logs:

• we can add metadata to log records (e.g. request ID to filter/see all logs of a request, or filter by log level)
• exception traces are organized as sub-metadata of the exception record, keeping things easier to navigate

But this isn't perfect:

• depending on the tool you use to view the logs, the UX can be very very poor
• it's hard to explore the logs quickly because now all info is nested in a log record, with no hierarchy to preview "just the message" quickly
• Monolog's JSON formatter adds tons of info to the JSON record

Here's an example (Bref Cloud formats the JSON payload FYI):

CloudWatch has some great feature though:

This is the log message {"key":"value"}

The log message above will automatically get parsed as text (the first part) and a JSON object attached (the second part).

That means we can have a simple log message + a structured JSON object.

On top of that, with CloudWatch:

• we don't need to log the timestamp, CloudWatch automatically timestamps log records
• I don't think we need to log the "channel" name: IMO channels make sense to split web logs from jobs or artisan commands, but in Lambda these are separate functions/logs anyway, so I think we can remove that part too

This is why I want to add this CloudWatchFormatter optimized for Bref users.

As you can see in the screenshots below, it is easy to navigate the logs textually. But it's also possible to have a lot more information nested in the log records via the JSON object.

In CloudWatch:

In Bref Cloud:

In CloudWatch Logs Insights, we can see that keys of the JSON objects are correctly detected:

That allows us to create advanced queries, for example to search for "ERROR" logs with a specific exception class:

Looking forward

I'm releasing this in Bref v2 so that we can all test this. If this works as well as I hope, the plan is to make this the default format in Laravel with the Bref-Laravel bridge.

I am also working on a v2 for https://github.com/brefphp/logger that uses the same format: brefphp/logger#11