At work, I encountered a strange issue where LogDNA was missing some log entries. This was very annoying since the whole point of logs is to have a full picture when debugging issues.

At first I chalked it up to the LogDNA service being flaky, but over time, I noticed this was a systemic issue and so I decided to investigate.

As my assigned projects are in Python, we use the logging module and LogDNA’s Python Handler to ingest our log messages. If you look into the module carefully, you’ll notice that it uses a thread in a self-callback-loop to flush it’s buffers.

Because AWS Lambda currently uses Linux’s process freezer, this is highly problematic if not managed carefully. The Linux freezer will suspend all threads under our process upon the exit of the handler function, which means – the thread callback never happens; the log buffer never gets flushed; and log messages become lost.

To solve this problem, you can implement a simple decorator that flushes all handlers attached to the root logger. Attach this decorator to the Lambda handler function, so that all loggers are flushed before control is returned to the Lambda executor and before our process gets frozen.

This ensures that the flush thread is joined before continuing, flushing the log buffers and ingesting our log messages. No more lost logs!