At the end of June we rolled out Paglo’s log collection service. Chris has previously discussed how you can use this service for searching and analysis.

I want to cover another topic: alerts. Paglo has an Alerts application. Any query you can construct against your Paglo database can be used to generate an alert. We have extended the Alerts app to allow you to create alerts that trigger when certain log searches return results.

When you create an alert you specify the conditions to generate an alert. You can select whether this alert is triggered by the results of a PQL query, a free text search of your PQL database via an Assert search, or a free text Log search.

Here you can see that we have selected Log search and want an alert generated when either core dumped or ReportCrash appear in any log source.

Note: search terms are not case sensitive, but the logic operators OR, AND, NOT must be uppercase.

We use those terms specifically because on FreeBSD and Mac OS X which I am monitoring those are the strings that will appear in a system log file when a program crashes in a generally unhappy way.

The left and right parentheses group the search terms core and dumped together. The word OR will match any log messages where either set of terms appears. So a log message must have the words core and dumped or the word ReportCrash somewhere in it. Note that the logical operator OR must be in uppercase.

There is a final important element to log search based alerts: When the alert is saved, and when the alert is triggered it saves a bookmark of sorts that represents a certain position in your entire database of logs. When the alert is checked it only searches log messages that occurred in time after that bookmark.

Therefore an alert will only fire whenever a log message appears that matches the search terms after the alert has been created. Shortly after, when the alert is next tested, we only look for log messages after the one that caused the alert to fire. This will cause the alert to clear. The alert will then remain clear until another log message appears that matches the search terms.

The email that the Alerts app will generate will have just the log messages that caused the alert to trigger in it so you will have some idea of the specifics of what caused the alert to fire. Here is an example of the email generated by the Alerts app when the alert we defined above fires: