Okta Incident Response Guide
A critical part of responding to a security incident is gathering relevant information and facts that can be used to shape your response. You have to answer questions like: who accessed Okta, when, what did they do, and how did they gain access? To facilitate the Incident Response process, Okta provides detailed audit logs within the System Log. This information is available through the Admin web console, and also via the API.
Within this document we’ll cover what the System Log is and where to find it, how to translate logs to actual user activity, and how you can leverage the System Log during a security incident. We’ll also review some of the actions you can take to respond to an incident identified within the System Log.
For additional information, or if you need help, the following resources are available:
System Log 101
Where is the System Log?
Accessing the System Log requires access to the Admin section of Okta, and it is under the Reports tab. Make sure that when giving users access to the System Log it is in the form of a read only administrator user, so they cannot make changes to your Okta configuration. From the Reports page you can view a number of custom dashboards, as well as run a number of saved searches from the “System Log” section on the right side. You can also just go straight to the System Log by clicking on “System Log”.
Alternatively, consider exporting these logs via the API to a third-party log storage and analysis system. Okta maintains integrations with a number of third party systems like Splunk.
What is the System Log?
The Okta System Log web console is comprised of three sections.
At the top you have the search bar. Here you can specify a filter or search, set the timeframe for the query, and if needed modify the time zone to use.
Below the search bar is a chart displaying event count mapped over the timeframe of the search. This is useful for visually spotting trends. In addition you can click the link for “show event trends by category” to see charts mapping events by target, actor, or event type.
The third section is the event viewer where you can review the events that match your query for the given timeframe. Note, at the top left of the event viewer is a button that will also map events geographically. This is shown in the screenshot below.
Within the event viewer you can expand events of interest to view more details simply by clicking on the row. You can also export the events in the viewer by clicking “Download CSV”.
Log Structure
Each log is separated into five sections, though not all 5 will be used by every event type.
- Actor – Describes the user, app, client, or other entity who performed an action on a target
- Client – Describes the client performing the action. This includes things like User Agent, device and browser type, and network zone info
- Event – What is being logged, as well as any associated outcomes (ex: authentication successful or authentication failed)
- Request – Logs information about the HTTP request that was sent to the Okta service. This also stores geographical context surrounding the IP address such as country and approximate latitude and longitude
- Target –The entity upon which an actor performs an action. Targets may be one of multiple things: a user, application, group, token, etc
A quick word about events. Event types categorize events by action and are recorded in a log’s 'eventType' attribute. This is a very important field. Event types take the general form of 'event_space.type.action'. Some example event types include:
User
- Session
- Start (user.session.start)
- Access_Admin_App (user.session.access_admin_app) - Account
- Update_Password (user.account.update_password)
- Reset_Password (user.account.reset_password) - MFA
- Deactivate (user.mfa.deactivate)
- Update (user.mfa.update)
In addition to user.* there are also event spaces like: group.*, application.*, and policy.*. For more information on event spaces see our developer documentation.
Advanced Queries
If you are unsure of how to filter, or having trouble figuring out the syntax, be sure to check out the "Advanced Filter" window.
This will open a window that aids you in building your filter, and it also has a helpful autocomplete feature.
In addition, you can use the operators described here to improve your queries. For some additional examples of how to craft custom queries review this support article.
Understanding the System Log
User Authentication - Browser (Interactive)
What does it look like when a user accesses the Okta platform? The most common type of authentication is interactive through the web browser, and that is what will be described below.
There are several different events that will be logged.
- When a user authenticates to Okta it will generate an event of the type 'user.session.start'. This means they entered the correct username and password credentials
- When a user attempts to authenticate, the Okta platform will determine what if any policy should be applied to that authentication event. This will determine for example, if a user should receive an MFA challenge. You can review these events by filtering for events with the type 'policy.evaluate_sign_on'. If an MFA authentication is required by policy you will see an event of the type user.authentication.auth_via_mfa
- When a user accesses the Admin section of Okta, it will generate an event of the type 'user.session.access_admin_app'
The flow above refers specifically to a user accessing the Okta web console, however there are other authentication flows you may run into. These can include for example 'user.authentication.auth_via_richclient' when a user accesses Okta through a third party application such as Outlook. Some other common types of authentication events would be 'user.authentication.auth_via_radius' or 'user.authentication.auth_via_AD_agent'. The type of authentication events you will see is highly dependent on your Okta configuration and setup and we will not cover all flows here.
Let’s walk through what an actual authentication would look like in the System Log. I've abbreviated the logs down to specific items of interest.
Log 1
- Actor - John Smith
- Client - Chrome on OSX, Geography, IP address, Network Zone
- Event - user.session.start
-Outcome - Success - Request - IP Address
- Target – NA
Log 2
- Actor - John Smith
- Client - Chrome on OSX, Geography, IP address, Network Zone
- Event - policy.evaluate_sign_on
- Outcome - Challenge - Request - IP Address
- Target - Authentication Policy Object
Log 3
- Actor - John Smith
- Client - Chrome on OSX, Geography, IP, Network Zone
- Event - user.authentication.auth_via_mfa
-Outcome - Success - Request - IP Address
- Target - NA
Session IDs
When you successfully authenticate to Okta you will be granted a session with a unique identifier. This unique identifier will be included with every event associated with that session under the field "ExternalSessionId". You can use this to track activity that is related to suspicious authentications, and get a good idea of what occurred during that session. For more information about Session IDs please see this support page.
Application Access
When a user has an established session, we log when the Okta platform grants access to applications using single sign-on (SSO). These generate an event with the type 'user.authentication.sso'. This allows you to track who is accessing which applications and when.
Example: John Smith authenticates to Okta and then clicks on the icon for Box. The Okta SSO service logs him into the Box service. This would generate a log that takes the following form:
- Actor - John Smith
- Client - Chrome on OSX, Geography, IP address, Network Zone
- Event - user.authentication.sso
-Outcome - Success - Request - IP Address
- Target1 - Box (AppInstance)
- Target2 - John Smith (AppUser)
AppUsers and AppInstances
Take a look at the targets of the example event above. In this case the target is both an AppUser and AppInstance. Within Okta you may have multiple instances of the same application, for example multiple Box environments. Also the same Okta user could have multiple Box accounts. Okta differentiates between these environments and accounts using AppInstance and AppUser entities. So in the example SSO event, John Smith accessed a particular Box instance as a particular user within that instance.
The Incident Lifecycle
Now that you have an understanding of the types of things you can find in the System Log, and how to access the System Log, let's talk about responding to security incidents.
As always, you should be looking to do the following things as you go through your Incident Response process:
- Extract Indicators of Compromise and record them
- IP addresses
- HTTP user agents
- geographical areas
- specific applications being targeted - Identify compromised user accounts
- Determine what those accounts did within Okta
- Pivot on session IDs
- View apps accessed and timestamps associated with that access
- Review for any Admin access or configuration changes to Okta
Note: Okta serves as access control to downstream applications but has no visibility into what occurs within them. If you see that a suspected compromised user used SSO to access a particular application, you will need to review the logs in that downstream application to understand what actions were carried out in that application. Correlating between Okta activity and other application logs can be a powerful tool for a security team.
As you are initiating your investigation, or are already involved in responding to an incident, try to remember the attacker lifecycle and use it to shape your actions and add context to your findings.
Let's walk through the stages at a high level, from the perspective of an Okta administrator.
Initial Reconnaissance
Early stages of an attack can be difficult to separate from everyday use of your Okta platform. Usually once you've identified some suspicious events you can begin to pivot and view other related activity. However, there are some patterns you should be on the lookout for.
- Large numbers of authentication attempts for invalid or non-existent users. This activity can be consistent with username enumeration
- You can view this activity by looking for events of the type 'user.session.start' with the outcome 'UNKNOWN_USER'
- Exact query = eventType eq "user.session.start" and outcome.reason eq "UNKNOWN_USER" - Brute force or Password Spraying attacks can be spotted by looking for a few different patterns such as: high failure rate, excessive account lockouts, etc.
- Track user lockouts with the query: eventType eq "user.account.lock"
- Track failed authentications due to incorrect password with the query: eventType eq "user.session.start" and outcome.reason eq "INVALID_CREDENTIALS"
Initial Compromise
Once an attacker has gained control of account credentials they will then access the Okta account. This can be identified by searching for events as described in the section titled "User Authentication" in the previous section, specifically:
- eventType eq "user.session.start" and outcome.result eq "SUCCESS" - correct username and password entered
- eventType sw "user.authentication.auth" - covers most authentication events such as MFA or RichClient (note the sw means 'starts with')
- Once you've identified a suspicious user session, track what occurred during that session by searching for other logs associated with the external session id: authenticationContext.externalSessionId eq "session_id"
Establish Foothold, Escalate Privileges
Once an attacker has gained access to the Okta platform, depending on the level of that access they will begin to establish a foothold. If possible, they may also attempt to escalate their privileges to an Administrator role.
You can audit access to the Admin section of Okta by looking for these events:
eventType eq "user.session.access_admin_app" - user attempted to access Admin section of Okta
An attacker may also attempt to remove protections and security controls from an account in a number of ways such as: updating or resetting passwords, unlocking accounts, modifying network zones, or removing MFA. Here are a couple of examples:
- eventType eq "user.mfa.factor.deactivate" - factor removed
- eventType eq "user.account.reset_password" - account password reset by Admin
- eventType eq "user.account.update_password" - account password changed by user
- eventType eq "zone.update" - network zone changed
Attackers may also try to grant themselves additional privileges, or add their user to specific groups to get at their objective. Search for these with the following queries:
- eventType eq "user.account.privilege.grant" - this is the event associated with the granting of Administrator privileges to a user.
- eventType eq "group.user_membership.add" - user added to group membership
Accomplish Mission
Once an attacker has gained access to Okta, and acquired the necessary privileges, they will then move to complete their objective. What the attackers' objective is can vary greatly, but it usually involves a third-party system that Okta controls access to. Okta cannot log what happens within these external systems. Examples of these objectives might include:
- Exfiltrating internal documents from a cloud file storage service.
- Modifying or changing payroll or finance information
- Accessing internal tools and systems, like those that manage infrastructure
- Retrieving the contents of internal communications and email
Once you've identified the malicious session, you can review the System Log to spot what they accessed and when, and then pivot into those downstream applications to review more detailed activity logs. Utilizing the Okta session ID within the System Log can be extremely helpful here.
Example Queries
To get you started, here are some example queries you might find useful. These statements can be combined with other statements using and/or logic.
- Show all users that have attempted to log into the Okta platform in a given timeframe
- eventType eq "user.session.start" - Show all authentication attempts from a specific IP
- eventType eq "user.session.start" and client.ipAddress eq "192.0.0.1" - Show all apps accessed from a specific IP
- eventType eq "user.authentication.sso" and client.ipAddress eq "192.0.0.1" - Show all events associated with a particular HTTP user agent
- client.userAgent.rawUserAgent eq "Python-urllib/2.7" - Show all activity related to a user session
- authenticationContext.externalSessionId eq "session_id" - Show failed OTP (one-time pass) MFA attempts
- eventType eq "user.authentication.auth_via_mfa" and outcome.result eq "FAILURE" - Show all events where actor submitted legitimate username and password
- eventType eq "user.session.start" and outcome.result eq "SUCCESS" - Show all events related to a specific user (can be by username or associated email)
- actor.alternateId eq "[email protected]"
- actor.displayName eq "John Smith" - Show all attempted access to the Admin app
- eventType eq "user.session.access_admin_app" - Show all applications a user accessed via SSO
- eventType eq "user.authentication.sso" and actor.alternateId eq "[email protected]" - Show all events associated with a given IP address
- client.IpAddress eq "192.0.0.1"
- request.ipChain.ip eq "192.0.0.1" - Show all Okta activity from a specific country
- request.ipChain.geographicalContext.country eq "United States" - Show all SSO authentications to a particular application
- target.displayName eq "App Name" and eventType eq "user.authentication.sso" - Show events associated with the granting of Administrator access to a user. (Note there are multiple types of Administrator privileges, as described on this support page)
- eventType eq "user.account.privilege.grant" and debugContext.debugData.privilegeGranted eq "Super administrator"
Response
Once you've identified the scope of the incident, it is time to take action. As always you should follow your established Incident Response process as well as all relevant corporate policies. These are some Okta-specific tools you can employ.
Containment
Once you've identified an intrusion or compromised users, you may need to take administrative action. There are a couple approaches you could take:
- Move the user to a special group where they have extremely limited access, and restrictive authentication policy
- Suspend the user temporarily
- Kill user session to force a new authentication adhering to new policies
- Require MFA challenges for SSO authentication to sensitive applications for all users
- Require MFA for all access to the Okta administrator page
- Block malicious IP addresses using Network Zones
-Alternatively, create Network Zones for your Corporate IP space and only allow access from those Zones for the duration of the Incident
For user account actions go to the People section of the Admin console. From there you can deactivate, suspend, or kill a session.
Remediation
Once you've contained the incident, it's time to remediate and restore normal function for your users.
- Reset suspected compromised users' passwords
- Confirm users control the authentication factors registered to the account, or reset MFA settings
- Return users to their normal groups
- Unsuspend users after regaining control over accounts
- Restore authentication policies and network zones
After Action
After detecting, investigating, and responding to a security incident it’s important to review your current Okta configuration and policies. Ask yourself:
- Are you using industry best practices for your password policy?
- Are you correlating the Okta System Log events with other security events?
- Have you defined Network Zones for your business and incorporated them into your authentication policies?
- Do you have MFA enabled for your privileged users?
- Do all of your applications that support SAML have it enabled (a quick way to check this is to go to Reports page and then click “SAML Capable Apps”)?
- Are your sensitive applications protected by additional MFA challenges?
- Are you using strong authentication factors for MFA challenges?
-You can review permitted factors in the tab Security -> Multifactor -> Factor Types
-Okta recommends using Okta Verify or U2F factors for MFA
Consider using the API to automate the actions described in this section for faster response time and scalability.
Closing Thoughts
I hope that this guide gives you the confidence and basic skills required to conduct an investigation and respond to an incident using the Okta System Log. You should now know where the System Log events are stored, what the format looks like, and how to search for specific events of interest. In addition, I provide some basic examples of actions that can be taken within the platform to deal with an incident.
The Okta System Log supports searching and filtering, but to really take your incident response capabilities to the next level we recommend exporting them to a full featured log analysis system. Currently supported integrations are detailed in our documentation and we also provide an API for custom integrations. The benefits of exporting to a dedicated system include data retention past 6 months, the ability to calculate metrics and statistics around usage, pattern analysis, and alert generation for events of interest.