In Bosun notifications are used for both new alert incidents and when an alert is acked/closed/etc. If you don't want the other events to trigger a notification add runOnActions = false to the notification definition.
runOnActions = false
See also: