After having used ECS at several projects I have been playing with Kubernetes for a while. There is a lot more tooling around kubernetes, and its YAML files are certainly a lot more pleasant to write than CloudFormation.
I think I noticed when working with a system that had some stability issues was that often missed some critical events such as pods running out of memory and restarting. Normally I use Sentry to track and create alerts for application errors, so it made sense to use that here as well. There were some existing tools to log Kubernetes events in Sentry, but they all had some issues that were dealbreakers for my use:
- getsentry/sentry-kubernetes is the official Sentry kubernetes reporter. Unfortuantely this is not actively maintained and suffers from a major memory leak.
- stevelacy/go-sentry-kubernetes is an alternative go implementation. This only watches for Pod status changes, so it will miss events such as missing volumes, ingress errors, etc.
This led me to create a k8s-sentry. My design goals were simple: use as little memory and CPU as possible, at a minumum support the same features as sentry-kubernetes, and make it easy for people to use. This quickly led to a the first design decision: use Go as implementation language, since that is the standard for Kubernetes tooling and has the most mature API library
In order to capture all types of errors k8s-sentry use two different methods in parallel:
- watch all pods, and check for pods that failed completely or have a failing container.
- monitor events, looking for events that indicate an abnormal situation
To help with deployment I provided manifests for both cluster-wide and single-namespace. There include a separate service account with the right roles if you have RBAC enabled.