Okta Incident Response Guide

Where is the System Log?

Accessing the System Log requires access to the Admin section of Okta, and it is under the Reports tab. Make sure that when giving users access to the System Log it is in the form of a read only administrator user, so they cannot make changes to your Okta configuration. From the Reports page you can view a number of custom dashboards, as well as run a number of saved searches from the “System Log” section on the right side. You can also just go straight to the System Log by clicking on “System Log”.

Alternatively, consider exporting these logs via the API to a third-party log storage and analysis system. Okta maintains integrations with a number of third party systems like Splunk.

What is the System Log?

The Okta System Log web console is comprised of three sections.

At the top you have the search bar. Here you can specify a filter or search, set the timeframe for the query, and if needed modify the time zone to use.

Below the search bar is a chart displaying event count mapped over the timeframe of the search. This is useful for visually spotting trends. In addition you can click the link for “show event trends by category” to see charts mapping events by target, actor, or event type. 

The third section is the event viewer where you can review the events that match your query for the given timeframe. Note, at the top left of the event viewer is a button that will also map events geographically. This is shown in the screenshot below.

Within the event viewer you can expand events of interest to view more details simply by clicking on the row. You can also export the events in the viewer by clicking “Download CSV”.

Log Structure

Each log is separated into five sections, though not all 5 will be used by every event type.

• Actor – Describes the user, app, client, or other entity who performed an action on a target
• Client – Describes the client performing the action. This includes things like User Agent, device and browser type, and network zone info
• Event – What is being logged, as well as any associated outcomes (ex: authentication successful or authentication failed)
• Request – Logs information about the HTTP request that was sent to the Okta service. This also stores geographical context surrounding the IP address such as country and approximate latitude and longitude
• Target –The entity upon which an actor performs an action. Targets may be one of multiple things: a user, application, group, token, etc

A quick word about events. Event types categorize events by action and are recorded in a log’s 'eventType' attribute. This is a very important field. Event types take the general form of 'event_space.type.action'. Some example event types include:

User

• Session
  - Start (user.session.start)
  - Access_Admin_App (user.session.access_admin_app)
• Account
  - Update_Password (user.account.update_password)
  - Reset_Password (user.account.reset_password)
• MFA
  - Deactivate (user.mfa.deactivate)
  - Update (user.mfa.update)

In addition to user.* there are also event spaces like: group.*, application.*, and policy.*. For more information on event spaces see our developer documentation.

Advanced Queries

If you are unsure of how to filter, or having trouble figuring out the syntax, be sure to check out the "Advanced Filter" window.

This will open a window that aids you in building your filter, and it also has a helpful autocomplete feature.

In addition, you can use the operators described here to improve your queries. For some additional examples of how to craft custom queries review this support article.

Now that you have an understanding of the types of things you can find in the System Log, and how to access the System Log, let's talk about responding to security incidents.

As always, you should be looking to do the following things as you go through your Incident Response process:

• Extract Indicators of Compromise and record them
  - IP addresses
  - HTTP user agents
  - geographical areas
  - specific applications being targeted
• Identify compromised user accounts
• Determine what those accounts did within Okta
  - Pivot on session IDs
  - View apps accessed and timestamps associated with that access
  - Review for any Admin access or configuration changes to Okta

Note: Okta serves as access control to downstream applications but has no visibility into what occurs within them. If you see that a suspected compromised user used SSO to access a particular application, you will need to review the logs in that downstream application to understand what actions were carried out in that application. Correlating between Okta activity and other application logs can be a powerful tool for a security team.

As you are initiating your investigation, or are already involved in responding to an incident, try to remember the attacker lifecycle and use it to shape your actions and add context to your findings.
Let's walk through the stages at a high level, from the perspective of an Okta administrator.

Initial Reconnaissance

Early stages of an attack can be difficult to separate from everyday use of your Okta platform. Usually once you've identified some suspicious events you can begin to pivot and view other related activity. However, there are some patterns you should be on the lookout for.

• Large numbers of authentication attempts for invalid or non-existent users. This activity can be consistent with username enumeration
  - You can view this activity by looking for events of the type 'user.session.start' with the outcome 'UNKNOWN_USER'
  - Exact query = eventType eq "user.session.start" and outcome.reason eq "UNKNOWN_USER"
• Brute force or Password Spraying attacks can be spotted by looking for a few different patterns such as: high failure rate, excessive account lockouts, etc.
  - Track user lockouts with the query: eventType eq "user.account.lock"
  - Track failed authentications due to incorrect password with the query: eventType eq "user.session.start" and outcome.reason eq "INVALID_CREDENTIALS"

Initial Compromise

Once an attacker has gained control of account credentials they will then access the Okta account. This can be identified by searching for events as described in the section titled "User Authentication" in the previous section, specifically:

• eventType eq "user.session.start" and outcome.result eq "SUCCESS"  - correct username and password entered
• eventType sw "user.authentication.auth" - covers most authentication events such as MFA or RichClient (note the sw means 'starts with')
• Once you've identified a suspicious user session, track what occurred during that session by searching for other logs associated with the external session id: authenticationContext.externalSessionId eq "session_id"

Establish Foothold, Escalate Privileges

Once an attacker has gained access to the Okta platform, depending on the level of that access they will begin to establish a foothold. If possible, they may also attempt to escalate their privileges to an Administrator role.

You can audit access to the Admin section of Okta by looking for these events:

eventType eq "user.session.access_admin_app" - user attempted to access Admin section of Okta

An attacker may also attempt to remove protections and security controls from an account in a number of ways such as: updating or resetting passwords, unlocking accounts, modifying network zones, or removing MFA. Here are a couple of examples:

• eventType eq "user.mfa.factor.deactivate" - factor removed
• eventType eq "user.account.reset_password" - account password reset by Admin
• eventType eq "user.account.update_password" - account password changed by user
• eventType eq "zone.update" - network zone changed

Attackers may also try to grant themselves additional privileges, or add their user to specific groups to get at their objective. Search for these with the following queries:

• eventType eq "user.account.privilege.grant" - this is the event associated with the granting of Administrator privileges to a user.
• eventType eq "group.user_membership.add" - user added to group membership

Accomplish Mission

Once an attacker has gained access to Okta, and acquired the necessary privileges, they will then move to complete their objective. What the attackers' objective is can vary greatly, but it usually involves a third-party system that Okta controls access to. Okta cannot log what happens within these external systems. Examples of these objectives might include:

• Exfiltrating internal documents from a cloud file storage service.
• Modifying or changing payroll or finance information
• Accessing internal tools and systems, like those that manage infrastructure
• Retrieving the contents of internal communications and email

Once you've identified the malicious session, you can review the System Log to spot what they accessed and when, and then pivot into those downstream applications to review more detailed activity logs. Utilizing the Okta session ID within the System Log can be extremely helpful here.

Example Queries

To get you started, here are some example queries you might find useful. These statements can be combined with other statements using and/or logic. 

• Show all users that have attempted to log into the Okta platform in a given timeframe
  - eventType eq "user.session.start"
• Show all authentication attempts from a specific IP
  - eventType eq "user.session.start" and client.ipAddress eq "192.0.0.1"
• Show all apps accessed from a specific IP
  - eventType eq "user.authentication.sso" and client.ipAddress eq "192.0.0.1"
• Show all events associated with a particular HTTP user agent
  - client.userAgent.rawUserAgent eq "Python-urllib/2.7"
• Show all activity related to a user session
  - authenticationContext.externalSessionId eq "session_id"
• Show failed OTP (one-time pass) MFA attempts
  - eventType eq "user.authentication.auth_via_mfa" and outcome.result eq "FAILURE"
• Show all events where actor submitted legitimate username and password
  - eventType eq "user.session.start" and outcome.result eq "SUCCESS"
• Show all events related to a specific user (can be by username or associated email)
  - actor.alternateId eq "[email protected]"
  - actor.displayName eq "John Smith"
• Show all attempted access to the Admin app
  - eventType eq "user.session.access_admin_app"
• Show all applications a user accessed via SSO
  - eventType eq "user.authentication.sso" and actor.alternateId eq "[email protected]"
• Show all events associated with a given IP address
  - client.IpAddress eq "192.0.0.1"
  - request.ipChain.ip eq "192.0.0.1"
• Show all Okta activity from a specific country
  - request.ipChain.geographicalContext.country eq "United States"
• Show all SSO authentications to a particular application
  - target.displayName eq "App Name" and eventType eq "user.authentication.sso"
• Show events associated with the granting of Administrator access to a user. (Note there are multiple types of Administrator privileges, as described on this support page)
  - eventType eq "user.account.privilege.grant" and debugContext.debugData.privilegeGranted eq "Super administrator"

I hope that this guide gives you the confidence and basic skills required to conduct an investigation and respond to an incident using the Okta System Log. You should now know where the System Log events are stored, what the format looks like, and how to search for specific events of interest. In addition, I provide some basic examples of actions that can be taken within the platform to deal with an incident.

The Okta System Log supports searching and filtering, but to really take your incident response capabilities to the next level we recommend exporting them to a full featured log analysis system. Currently supported integrations are detailed in our documentation and we also provide an API for custom integrations. The benefits of exporting to a dedicated system include data retention past 6 months, the ability to calculate metrics and statistics around usage, pattern analysis, and alert generation for events of interest.