There are error metrics in the Arc Lamp client code (wmf-config/profiler.php, currently in Statsd until T240685). These are plotted in Grafana: Arc Lamp and included in Logstash: Perf Team. However, we currently have no alerting on this.
Proposed outline:
- Review error metric code in wmf-config.
- Decide on an alert threshold.
- Enable alerts for Arc Lamp error metrics (either in Puppet on Graphite, or via Grafana by the dash to perf team stuff)