Posts for Observability

• Sunday, 8 Feb 2026 11:33 pm

  How to read and write application logs

  Tom MacWright, recently:

  Every application that I've worked on eventually just generates several 'flavors' of log messages out of stdout and stderr and logs stop being useful because they're filled with 'junk' like request logs.

  I’ve tried structured JSON logs with pino, tried tslog, Betterstack, Axiom, and never got it. We've never had a team member that really got value out of logs. I've never really gotten value out of logs. I often wonder if servers should emit logs at all, and instead we should just do telemetry and metrics?

  Servers should absolutely emit logs. So should scripts, and so should periodic jobs. Anything that does anything should write logs, because how else are you going to know what happened? Are you going to go into your database and check that a value was updated? Are you going to email your user and ask them whether they got the sign-up email?

  Now — admittedly, I don't like structured JSON logs. I don't think they're very useful, because they're not very legible. Because they're just JSON, devs like to put all sorts of crap into them. Much better are simple human-readable logs, something like this:

  2026-02-08 19:04:32.123456 Log_Level=INFO Process=<process_name> logReference=archiver:user:complete - User record user_id=123 was set to status=archived in duration=0.002 - internalID=20260208190432123_ABC123_10

  A couple of points to call out, here:

  • Including field=value pairs in the log doesn't really harm readability, but it does make logs queryable. To make an implicit assumption explicit: your logs should be ingested by some kind of aggregator that allows you to index and query; trying to read through a 28-million-line plaintext file is a recipe for a headache. You can route all stdout to a file timestamped by the hour and then ask Claude to write you a simple Go service that indexes those files to your DB of choice and provides an API for querying. You probably don't need Splunk.
  • Logs ought to have something like an internalID to allow you to join across all logs for a specific request. One really good pattern I've seen is the above, using a microsecond-specific timestamp + 6 random characters + some optional suffix (in this case maybe the index of the record being processed by a nightly job that archives inactive users). This allows you to do a query like SELECT duration FROM logs WHERE internalID LIKE "20260208%" to get a sense for how long requests took on a given day, or SELECT * from logs WHERE internalID=<some_internalID> to see the story of a single request.
  • Most logging solutions write log as strings and output JSON, but this is backwards. Maintain a list of strings for outputting and write logs as objects/hashes/dicts:

  const logs = {
    "archiver:user:complete": {
      "level": "INFO",
      "msg": "User record user_id={userID} was set to status={status} in duration={duration} - internalID={internalID}",
    },
    // ...
  }
  
  // elsewhere...
  logger.writeLog("archiver:user:complete": {
    userId: "123",
    lineNumber: 10,
    status: "archived",
    duration: 0.002,
    internalID: "20260208190432123_ABC123_10"
  });

  There is some value in keeping different types of logs, although I think most logs should fall into only two categories: INFO and ERROR. Almost all logs should fall into the INFO category; ERRORs are written when something unexpected has happened and indicates that a problem needs to be fixed. If you are getting no ERROR logs you have either been very diligent in fixing problems or not diligent enough at logging. I guess this is kind of what Sentry does, so if you already have Sentry then maybe you don't even need ERROR logs!

  Logs should be written a) when something is being measured, like the number of datastore query results, b) when one module or service hands off to another, c) when logic diverges from the happy path, or d) when something has gone wrong. I want to know these things.

  I tend to believe that anything more than this is overthinking. The goal here is to be able to say with certainty what your application is doing, with whatever level of granularity allows you to best deliver whatever service you're, uh, delivering. Can you tell that I can't think up a good way to end my blog post about logging.

  Code Observability