Basic Usage

Integrations

Integrations allow you to connect PagerTree to other 3rd party tools. Integrations create alerts and assign them to a account user, router or team.

High Level Alert Workflow Integration

For a full list of our supported integrations, please see the integrations page.

Integrations are open sourced and can be found on Github.

Integration Options

Every integration type will have different integration options that can be selected. At a minimum the following must be selected:

  • Name - A friendly name for your integration.
  • Urgency - The urgency of alerts created by the integration.
  • Destinations - Where alerts should be sent after they are created by the integration.

For integration specific options, follow the instructions on the new integration form.

Integration Options

Templates

Each integration supports customizing the alert title and description using handlebars substitution.

Data

{
  integration: {...} // the integration object
  data: {
    ... // what was sent by the integration
    
    // email integration special attributes
    to: //the recipients of the email joined by a semi-colon
    from: // the sender of the email joined by a semi-colon
    subject: // the email subject
    message: // the sanitized body of the e-mail
  } 
}

Integration Logs

Integration Rules

Troubleshooting

Integration Logs

  1. See the logs section on the integration page to see what PagerTree has received from your 3rd party tool.
  2. Click any log to see more details of what was received.

Integration Logs Integration Log

If you don't see any logs, check your 3rd Party Configuration. A couple of common issues:

  • The endpoint url was not correctly copy & pasted.
  • The 3rd party tool does not have access to the internet.
  • Internal firewall is blocking traffic.

If the logs aren't showing up, it's because PagerTree never received requests from your tool. If you do see logs, but it is still not behaving as expected, please check the source.

Check the Source

Check the source code for your specific integration to see why it might not be working the way you expect.

Integration adapters can be found in this folder. Specifically you should check the adapter_action function. It's normally close to the top of the file and looks like this:

def adapter_action
  if _is_create?
    :create
  elsif _is_resolve?
    :resolve
  else
    :other
  end
end

def _is_create?
  _state == "triggered"
end

def _is_resolve?
  _state == "resolved"
end

def _state
  adapter_incoming_request_params.dig("state")
end

Alert Aggregation

To reduce noise PagerTree will try to aggregate incoming alerts by thirdparty_id. The thirdparty_id will differ for each integration, see the source code for your integration to see what the thirdparty_id consists of.

When a "create" request comes in from a 3rd party system, PagerTree check's to see if there is another alert (from the same integration) with the same thirdparty_id in the (queued|open|acknowledged|dropped) state. If there is, PagerTree will increment the dedup count on the existing alert. Else, PagerTree will proceed to create the new alert.

Below is pseudo code showing how PagerTree tries to aggregate the alerts.

alert = integration.alerts
          .order(created_at: :desc)
          .find_by(thirdparty_id: thirdparty_id)

case action
  when :create
    if alert && (alert.status_queued? || 
        alert.status_open? || 
        alert.status_acknowledged? || 
        alert.status_dropped?)
      # alert already exists, increment dedup count
      alert.dedup_count += 1
    else
      # create a new alert
      alert = Alert.new
    end
  end
  # ...
end

Most integrations should handle unique ids for you, but if your integration requires you (ex: webhook integration) or your system (ex: Datadog) to manage the thirdparty_ids. Make sure the following are true:

  1. The 3rd party system is sending the "resolved" message to PagerTree. This will auto "resolve" the alert in PagerTree.
  2. Make sure your team is marking the alert as "resolved" in PagerTree. This will allow the system to create a new alert with the same ID.

Alert Aggregation Example #1 (Simple)

This example shows how a properly working integration would send information. Sequentially opening and resolving alerts.

  1. Integration I1 sends "create" request with thirdparty_id 123.
  2. Alert A1 is created.
  3. Integration I1 sends "resolve" request with thirdparty_id 123.
  4. Alert A1 is resolved.
  5. Integration I1 sends "create" request with thirdparty_id 123.
  6. A new alert (Alert A2) is created.

Alert Aggregation Example #2 (Dedup)

This example shows how a noisy integration might send information. Duplicating create events.

  1. Integration I1 sends "create" request with thirdparty_id 123.
  2. Alert A1 is created.
  3. Integration I1 sends "create" request with thirdparty_id 123.
  4. Existing alert (Alert A1) is found. The existing alert's dedup count is incremented.
  5. Integration I1 sends "resolve" request with thirdparty_id 123.
  6. Alert A1 is resolved.
  7. Integration I1 sends "create" request with thirdparty_id 123.
  8. A new alert (Alert A2) is created.

Alert Aggregation Example #3 (Multiple Integrations)

This example shows how integrations thirdparty_id namespace is unique.

  1. Integration I1 sends "create" request with thirdparty_id 123.
  2. Alert A1 is created.
  3. Integration I2 sends "create" request with thirdparty_id 123.
  4. Alert A2 is created.
  5. Integration I1 sends "resolve" request with thirdparty_id 123.
  6. Alert A1 is resolved.
  7. Integration I2 continuously sends "create" requests with thirdparty_id 123.
  8. Alert A2 dedup count is incremented until it is resolved by "Integration I2" or manually by a User.

Alert Aggregation Example #4 (User Intervention)

This example shows how a user can mark the alert as resolved to reset the deduplication flow.

  1. Integration I1 sends "create" request with thirdparty_id 123.
  2. Alert A1 is created.
  3. User acknowledges Alert A1.
  4. Integration I1 sends "create" request with thirdparty_id 123.
  5. Existing Alert A1 is found. The existing alert's dedup count is incremented.
  6. User manually resolves the alert.
  7. Alert A1 is resolved.
  8. Integration I1 sends "create" request with thirdparty_id 123.
  9. A new alert (Alert A2) is created...

FAQs

Previous
Escalation Policies