CYBER ; SEARCH TIME OPERATON SEQUENCE

SUMMARISE : Search-time operation sequence The following table presents the search-time operation sequence as a list. After the list you can find more information about each operation in the sequence, as well as examples of the search-time operations order. Each operation can have configurations that reference fields derived by operations that precede them in the sequence. However, those same configurations cannot contain fields that are derived by operations that follow them in the sequence. You can configure most of these operations through Splunk Web, although some configuration options are available only by making manual edits to configuration files. Make all manual file-based operation configurations on the search-head tier. This list does not include index-time operations, such as default and indexed field extraction. Index-time operations precede all search-time operations. See Index-time versus search time in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual. Search-time operation order Operation name Configurable in Splunk Web? Location of file configuration 7 1 2 3 4 5 6 7 8 Inline field extraction (no field transform) Field extraction that uses a field transform Automatic key-value field extraction Field filters Field aliasing Calculated fields Lookups Event types Tags Yes Yes No Yes Yes Yes Yes Yes Yes EXTRACT-<class> in a stanza in the props.conf file. REPORT-<class> in a stanza in the props.conf file. In stanzas in the props.conf file, where KV_MODE is set to a valid value other than none. If no KV_MODE value is specified for a stanza, it is set to auto by default. fieldFilterName stanza in the field_filters.conf file. FIELDALIAS-<class> in a stanza in the props.conf file. EVAL-<fieldname> in a stanza in the props.conf file. LOOKUP-<class> in a stanza in the props.conf file. In a stanza in the eventtypes.conf file. 9 See About configuration files for an overview of configuration file usage. In a stanza in the tags.conf file. Example of search-time operations order Consider calculated fields. Calculated field operations are in the middle of the search-time operation sequence. Splunk software performs several other operations ahead of them, and it performs several more operations after them. Calculated fields derive new fields by running the values of fields that already exist in an event through an eval formula. This means that a calculated field formula cannot include fields in its formula that are added to your events by operations that follow it in the search-time operation sequence. For example, when you design an eval expression for a calculated field, you can include extracted fields in the expression, because field extractions are processed at the start of the search-time operation sequence. By the time Splunk software processes calculated fields, the field extractions exist and the calculated field operation can complete correctly. However, make sure an eval expression for a calculated field never includes fields that are added through a lookup operation. Splunk software always performs calculated field operations ahead of lookup operations. This means that fields added through lookups at search time are unavailable when Splunk software processes calculated fields. You will get an error message if your calculated field eval expression includes fields that are added through lookups. Inline field extractions Inline field extractions are explicit field extractions that do not include a field transform reference. An explicit field extraction is a field extraction that is configured to extract a specific field or set of fields. Each inline field extraction configuration is specific to events belonging to a particular host, source, or source type. This operation does not include automatic key-value field extractions. Automatic key-value field extractions are their own operation category. Splunk Web management To create and manage inline field extractions, follow these steps: 8 In Splunk Web, go to Settings. 1. 2. 3. Navigate to Settings > Fields > Field extractions. Create a new field extraction or open an existing field extraction. You can also use the field extractor utility to design inline field extractions. Configure inline field extractions Create EXTRACT-<class> configurations within props.conf file stanzas. Restrictions Splunk software processes all inline field extractions belonging to a specific host, source, or source type in lexicographical order according to their <class> value. This means that you cannot reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ZZZ, but you can reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ddd. See Lexicographical processing of field extraction configurations. Because inline field extractions are near the top of the search-time operation sequence, they cannot reference fields that are derived and added to events by other search-time operations that come later. See also See Build field extractions with the field extractor to create inline field extractions in Splunk Web. The field extractor does not require you to understand how to write regular expressions. See Use the Field Extractions page to create inline field extractions in Splunk Web using the Field Extractions page in Settings. See Create and maintain search-time field extractions through configuration files to configure inline field extractions in the props.conf file.

Summary: Search-Time Operation Sequence in Splunk

Splunk's search-time operations allow dynamic data processing after indexing. Each operation has specific configurations and a defined sequence that impacts how fields are referenced. Below is an overview:

Operation Sequence Overview:
- Configurations of earlier operations can reference fields derived at that point but not fields created by subsequent operations.
- Configurations can be managed via Splunk Web or manually edited in configuration files (on the search-head tier).
Sequence of Operations:
- Inline Field Extraction: Extract specific fields explicitly without a transform reference. Configured using EXTRACT-<class> in props.conf.
- Field Transform Extractions: Use transforms referenced in REPORT-<class> in props.conf.
- Automatic Key-Value Extraction: Applied automatically when KV_MODE is enabled in props.conf.
- Field Filters: Managed in field_filters.conf.
- Field Aliasing: Aliases fields using FIELDALIAS-<class> in props.conf.
- Calculated Fields: Derived fields using eval formulas, configured as EVAL-<fieldname> in props.conf.
- Lookups: External data enrichment, managed with LOOKUP-<class> in props.conf.
- Event Types and Tags: Categorize events and add tags using eventtypes.conf and tags.conf.
Key Considerations:
- Inline extractions are processed early and cannot depend on fields added later in the sequence.
- Calculated fields can reference fields extracted earlier but not those added by lookups.
Examples and Usage:
- Calculated fields rely on fields created earlier (e.g., inline or key-value extractions).
- Lookup fields cannot be used in calculated fields due to their later position in the sequence.
Configuration Management:
- Inline field extractions can be managed via Splunk Web under Settings > Fields > Field Extractions or manually edited in props.conf.
- Lexicographical order applies to field extraction configurations for a specific host, source, or source type.
Resources:
- Splunk Web tools like the Field Extractor simplify configuration.
- Manual file edits provide advanced customization but require understanding props.conf syntax and processing rules.

This structured sequence ensures accurate and efficient field extraction, enrichment, and transformation in Splunk.

SUMMARISE: Field extraction that uses a field transform Field extraction configurations that reference a field transform are always processed by Splunk software after it processes inline field extractions. Like inline field extractions, each transform-referencing field extraction is explicitly configured to extract a specific field or set of fields. Each transform-referencing field extraction configuration is specific to events belonging to a particular host, source, or source type. This operation does not include automatic key-value field extractions. Automatic key-value field extractions are their own operation category. Splunk Web management To create and manage field extractions that use field transforms, follow these steps: 1. 2. 3. In Splunk Web, go to Settings. Navigate to Settings > Fields. In the Field Extractions and Field Transformations pages, set up the field extraction. 9 Configure field extraction that uses a field transform Create REPORT-<class> configurations within props.conf file stanzas. The REPORT-<class> configurations include a reference to an additional configuration in the transforms.conf file. Restrictions Splunk software processes all inline field extractions belonging to a specific host, source, or source type in lexicographical order according to their <class> value. This means that you cannot reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ZZZ, but you can reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ddd. See Lexicographical processing of field extraction configurations. Transform-referencing field extraction configurations can reference fields that are extracted through inline field extraction operations. They cannot reference fields that are derived and added to events by automatic key-value field extractions and other operations that take place later in the search-time operation sequence. See also See Use the field transformations page to create the transforms.conf part of a transform-referencing search-time field extraction. See Use the field extractions page to create the props.conf file part of a transform-referencing search-time field extraction. See Create and maintain search-time field extractions through configuration files to configure transform-referencing field extractions in transforms.conf file and props.conf files. See Extracting a field that was already extracted during inline field extraction to learn how to preserve values for fields that are extracted twice during the search-time operation sequence. Automatic key-value field extraction A field extraction configuration that uses the KV_MODE setting to automatically extract fields for events associated with a specific host, source, or source type. Automatic key-value field extraction is not explicit in that you cannot configure it to find a specific field or set of fields. It looks for any key-value patterns in events that it can find and extracts them as field-value pairs. You can configure key-value field extraction to extract fields from structured data formats like JSON, CSV, and table-formatted events. You can also disable search-time key-value field extraction for specific hosts, sources, and source types. Automatic key-value extraction always takes place after explicit field extraction methods, like in inline field extraction and transform--referencing field extraction. Splunk Web management You can configure the KV_MODE setting for source types through Splunk Web. KV_MODE defaults to automatic key-value field extraction for all source types unless it is set to another value. For example, if you want to disable search-time key-value field extraction for a specific source type, you must set KV_MODE to none for that source type. Here is how you can edit or update KV_MODE for a source type in Splunk Web. 1. In Splunk Web, go to Settings. 10 2. 3. 4. 5. 6. 7. Navigate to Settings > Source types. Locate a source type that you want to edit. Select Edit for that source type. Select Advanced. If KV _MODE is not among the settings listed for the source type you are editing, you can add it by selecting New setting and entering KV_MODE into the Name cell. When KV_MODE is not present it means that Splunk software applies the default for KV_MODE to the source type. KV_MODE is set to auto by default, which means that Splunk software runs automatic key-value field extraction at search time for any data with the source type. Change the Value of KV_MODE as necessary. See Configure automatic key-value field extraction for configuration details. Select Save to save your changes to the source type configuration. If you need to disable JSON field extraction for a source type without disabling automatic key-value field extraction for the source type, you can use this method to add the AUTO_KV_JSON setting with a Value of false to the source type configuration. For more information about editing source types with Splunk Web, see Manage source types in the Splunk Cloud Platform Getting Data In manual. Configure automatic key-value field extraction Set up automatic key-value field extractions for a specific host, source, or source type by finding or creating the appropriate stanza in the props.conf file and setting KV_MODE to auto, auto_escaped, multi, json, xml, or none. When KV_MODE is not set for a props.conf file stanza, that stanza has KV_MODE=auto by default. When KV_MODE is set to auto or auto_escaped, automatic JSON field extraction takes place alongside other automatic key-value field extractions. If you need to disable JSON field extraction without changing the KV_MODE value from auto, add AUTO_KV_JSON=false to the stanza. When not set, AUTO_KV_JSON defaults to true. Restrictions Splunk software processes automatic key-value field extractions in the order that it finds them in events. See also See Configure automatic key-value field extraction. Field filters Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms. Field filters control the search results that are visible to users at search time. You can apply a field filter to all nonexempt roles, which then affects the results of searches run by users assigned with those roles. Field filters do not modify the source event in the index, but remove or replace specific indexed, _raw, search time, and default fields and their values at search time when those fields appear in search results. You can remove specific fields and their values by redacting them with a null value. Alternatively, you can redact the value of a specific field by replacing it with a custom string such as XXXX, or you can obfuscate the field value by replacing it with a hash using SHA-256 or SHA-512 (SHA-2 family) hash 11 functions. By default, to create, edit, or delete field filters, you must be a member of the admin or sc_admin role. To view field filters, you must be a member of the admin, sc_admin, or power user role. See Define roles on the Splunk platform with capabilities in Securing Splunk Platform. Splunk Web management To create and manage field filters using Splunk Web, follow these steps: 1. 2. Select Settings > Field filters from Users And Authentication. Select New field filter. See Create field filters using Splunk Web in Securing the Splunk Platform. Configuration file management To create and manage field filters using configuration files, update the field_filters.conf file by adding action = <actions> to the <fieldFilterName> stanza for each field filter. To maintain data integrity and security, do not use configuration files to exempt roles from field filters. Instead, use Splunk Web or Splunk platform REST API endpoints to exempt roles from field filters. See Create field filters using configuration files in Securing the Splunk Platform. Splunk platform REST API management You can use the Splunk platform REST API to create and manage field filters. See the following in the Splunk Cloud Platform REST API Reference Manual: authorization/fieldfilters authorization/fieldfilters/{name} authorization/roles authorization/roles/{name} Restrictions Since field filters are in the middle of the search-time operation sequence, field filters affect search-time operations that come later for fields that are filtered. For example, if the user field is replaced with XXX, then field aliasing for user extracts the value XXX instead of the user's name. This process has particular implications for downstream operations that depend on the value of the field that is changed by a field filter. If your searches look for particular fields that are added through operations such as calculated fields, lookups, or tags, or you search on specific event types, be aware that field filters that redact or obfuscate your fields can cause your searches to break. If you want to use these kinds of operations with a field that is filtered, configure field filters to replace the field value with a hash, which preserves the statistical uniqueness of the field and makes it available to operations that come later in the search-time sequence. Alternatively, you might need to re-evaluate search operations that are used together with field filters. The following are operations that can be affected by field-value obfuscation and break existing searches when used with field filters: 12 Operation Calculated fields Lookups Event types tag command See also In Securing the Splunk Platform: Protect PII, PHI, and other sensitive data with field filters Define roles on the Splunk platform with capabilities Field aliasing Description The eval expression that calculates the field might break when it gets field values that are now XXX. Lookups add field-value combinations from lookup tables to event data and can break if Splunk software is unable to match field-value combinations in your event data with field-value combinations in external lookup tables. The search that defines the event type might be looking to match field values that are now XXX. If the value of a field for a tag's field-value pair is replaced with XXX, the tag is no longer applied. Field aliasing is the application of field alias configurations, which enable you to reference a single field in a search by multiple alternate names, or aliases. Each field alias configuration is specific to events belonging to a particular host, source, or source type. Splunk Web management To create and manage field aliases, follow these steps: 1. 2. 3. In Splunk Web, go to Settings. Navigate to Settings > Fields > Field aliases. Create a new field extraction or open an existing field alias. Configure field aliasing Create FIELDALIAS-<class> configurations in props.conf file stanzas. Restrictions Splunk software processes field aliases belonging to a specific host, source, or source type in lexicographical order. See Lexicographical processing of field extraction configurations. You can create aliases for fields that are extracted at index time or search time. You cannot create aliases for fields that are added to events by search-time operations that follow the field aliasing process, like lookups and calculated fields. See also Create field aliases in Splunk Web Configure field aliases with props.conf Calculated fields Configurations that create one or more fields through the calculation of eval expressions and add those fields to events. The eval expression can use values of fields that are already present in the event due to index-time or search-time field 13 extraction processes. Each calculated field configuration is specific to events belonging to a particular host, source, or source type. Splunk Web management To create and manage calculated fields, follow these steps: 1. 2. 3. In Splunk Web, go to Settings. Navigate to Settings > Fields > Calculated fields. Create a new calculated field or open an existing calculated field. Configure calculated fields Create calculated fields by adding EVAL-<fieldname> configurations to props.conf file stanzas. Restrictions All EVAL-<fieldname> configurations within a single props.conf stanza are processed in parallel instead of sequentially. This means you can't chain together calculated field expressions where the evaluation of one calculated field is used in the expression for the next calculated field. Calculated fields can reference all types of field extractions. They can't reference lookups, event types, or tags. You can't create a calculated field that is scoped to an aliased host, source, or source type. See Creation of a calculated field on an aliased source is not supported in the Knowledge Manager Manual. See also About calculated fields Create calculated fields with Splunk Web Configure calculated fields with props.conf Lookups Configurations that add fields from lookup tables to events when the lookup table fields are matched with one or more fields already present in those events. There are four types of lookup configurations: CSV lookups External lookups KV store lookups Geospatial lookups Each lookup configuration is specific to events belonging to a particular host, source, or source type. Splunk Web management To create and manage your lookups, follow these steps: 1. 2. In Splunk Web, go to Settings. Navigate to Settings > Lookups. 14 3. Create a new lookup definition or open an existing lookup definition. Configure lookups Define lookups that automatically add fields to events in search results by creating a LOOKUP-<class> configuration in the props.conf file. Each LOOKUP-<class> includes a reference to a [<lookup_name>] stanza in the transforms.conf file. Restrictions Splunk software processes lookups belonging to a specific host, source, or source type in lexicographical order. See Lexicographical processing of field extraction configurations. Lookup configurations can reference fields that are added to events by field extractions, field aliases, and calculated fields. They cannot reference event types and tags. See also About lookups

Summary: Field Extractions, Transformations, and Filters in Splunk

Field Extractions with Field Transforms

Processing Order: Transform-referencing field extractions occur after inline extractions but before automatic key-value extractions.
Configuration:
- Defined for specific hosts, sources, or source types.
- Configured via props.conf (REPORT-<class>) and transforms.conf files.
Restrictions:
- Fields extracted inline can be referenced in transforms, but not automatic key-value extractions.
- Inline extractions are processed in lexicographical order.

Automatic Key-Value Extractions

Functionality: Automatically extracts key-value pairs in events (e.g., JSON, CSV).
Configuration:
- Controlled by the KV_MODE setting in props.conf or Splunk Web.
- Defaults to KV_MODE=auto unless specified otherwise.
- JSON extraction can be disabled independently (AUTO_KV_JSON=false).
Processing Order: Takes place after explicit extractions.

Field Filters

Purpose: Redact, replace, or hash field values at search time without modifying original indexed data.
Configuration:
- Managed through Splunk Web, configuration files (field_filters.conf), or REST API.
- Filters apply to search-time operations (e.g., calculated fields, lookups) and can impact subsequent operations.
Restrictions: Operations dependent on filtered fields (e.g., lookups, calculated fields) may break.

Field Aliasing

Purpose: Create alternate names for fields.
Configuration:
- Managed via props.conf or Splunk Web under "Field Aliases."
- Processed in lexicographical order.
Restrictions: Cannot alias fields added by later search-time operations (e.g., lookups).

Calculated Fields

Purpose: Add fields based on eval expressions using existing fields.
Configuration:
- Defined in props.conf (EVAL-<fieldname>).
- Managed via Splunk Web under "Calculated Fields."
Restrictions: Parallel processing disallows chaining calculated fields. Cannot reference lookups or tags.

Lookups

Purpose: Enrich events by adding fields from lookup tables.
Types: CSV, KV store, geospatial, external lookups.
Configuration:
- Defined in props.conf (LOOKUP-<class>) and transforms.conf.
- Managed via Splunk Web under "Lookups."
Restrictions: Cannot reference event types or tags.

Key Takeaways

Processing Hierarchy: Inline extractions → Transform-referencing extractions → Automatic key-value extractions.
Configuration Flexibility: Extensive customization via configuration files (props.conf, transforms.conf) or Splunk Web.
Field Filters Impact: Can affect downstream operations, requiring careful configuration.
Field Aliases and Calculated Fields: Offer search-time flexibility but are limited by their processing order.

For detailed steps and examples, refer to Splunk's documentation on specific configurations.

SUMMARISE: Event types Configurations that add event type field-value pairs to events that match the search strings that define the event types. Splunk Web management After you run a search, save it as an event type. You can also define and maintain event types in Settings > Event types. Configure event types Configure event types in eventtypes.conf file stanzas. Restrictions Splunk software processes event types first by priority score and then by lexicographical order. So it processes all event types with a Priority of 1 first, and applies them to events in lexicographical order. Then it processes event types with a Priority of 2, and so on. Search strings that define event types cannot reference tags. Event types are always processed and added to events before tags. See also About event types Define event types in Splunk Web Automatically find and build event types Configure event types directly in eventtypes.conf Tags Configurations that add tags to specific field-value pairs in events. 15 Splunk Web management You can add tags directly to field-value pairs in search results. You can also define and maintain tags in Settings > Tags. Configure tags Configure tags in tags.conf file stanzas. Restrictions Splunk software applies tags to field-value pairs in events in lexicographical order, first by the field value, and then by the field name. See Lexicographical processing of field extraction configurations. You can apply tags to any field-value pair in an event, whether it is extracted at index time, extracted at search time, or added through some other method, such as an event type, lookup, or calculated field. See also Tag field value pairs in Search Define and manage tags in Settings Lexicographical processing of knowledge object configurations Splunk software processes the following knowledge objects in lexicographical order, according to the host, source, or source type they belong to: Inline field extractions Field extractions that use a field transform Field aliases Event types, after they are sorted according to priority Lookups Lexicographical order Splunk software also processes tags in lexicographical order, but they are not associated with a specific host, source, or source type. Lexicographical order sorts items based on the values used to encode the items in computer memory. In Splunk software, this is almost always UTF-8 encoding, which is a superset of ASCII. Numbers are sorted before letters. Numbers are sorted based on the first digit. For example, the numbers 10, 9, 70, 100 are sorted lexicographically as 10, 100, 70, 9. Uppercase letters are sorted before lowercase letters. Symbols are not standard. Some symbols are sorted before numeric values. Other symbols are sorted before or after letters. Splunk software also uses lexicographical ordering to determine configuration file precedence among app directories. See Configuration file precedence in the Splunk Enterprise Admin Manual. 16 Example Splunk software extracts inline field extractions to a specific host, source, or source type in ASCII sort order. This means that when it processes inline field extractions belonging to the access_combined_wcookies source type, it processes an extraction called REPORT-BBB before REPORT-ZZZ, then processes REPORT-ZZZ before REPORT-aaa, and so on. This means that you cannot reference a field extracted by REPORT-aaa in the field extraction definition for REPORT-BBB. For example, this configuration doesn't work because the first_ten field is extracted after the first_two field, due to the field extraction process ordering (aaa < ZZZ). [splunkd] EXTRACT-aaa = ^(?<first_ten>.{10}) EXTRACT-ZZZ = (?<first_two>.{2}) in first_ten The following configuration works because the first_ten field is extracted before the first_two field, due to the field extraction process ordering (ZZZ > mmm). [mongod] EXTRACT-ZZZ = ^(?<first_ten>.{10}) EXTRACT-mmm = (?<first_two>.{2}) in first_ten Here is a search you can use to verify these configuration issues. index=_internal (sourcetype=splunkd OR sourcetype=mongod) | stats values(first_ten) values(first_two) by sourcetype The process order within a single props.conf file The Splunk Enterprise Admin Manual contains several topics about configuration file administration. One of these topics, Attribute precedence within a single props.conf file in the Splunk Enterprise Admin Manual, includes information about knowledge object processing order, as well as the following topics: Precedence between sets of stanzas affecting the same host, source, or source type. Overriding the default lexicographical order in the props.conf file. Precedence for events with multiple attribute assignments

Summary: Event Types, Tags, and Lexicographical Processing in Splunk

Event Types

Definition: Configurations that add specific field-value pairs to events based on matching search strings.
Management:
- Create or maintain event types in Splunk Web (Settings > Event types) or directly in the eventtypes.conf file.
- Save search results as event types for reuse.
Processing:
- Prioritized by priority score, then processed in lexicographical order.
- Event types are applied before tags and cannot reference tags.
References:
- Splunk documentation includes guides for defining, building, and configuring event types.

Lexicographical Processing

Order of Processing:
- Knowledge objects (e.g., inline field extractions, field aliases, event types, tags, and lookups) are processed in lexicographical order.
- Sorting is based on UTF-8 encoding (numbers → uppercase letters → lowercase letters).
Application in Configurations:
- Configuration precedence applies to files in app directories and specific stanzas (e.g., props.conf and eventtypes.conf).
- Precedence rules help resolve conflicts when multiple configurations affect the same host, source, or source type.

Example of Configuration Order:

Invalid Configuration: Field extracted later cannot depend on a field extracted earlier due to lexicographical order.
Valid Configuration: Reordering extractions to comply with processing order resolves dependencies.
- Example: EXTRACT-ZZZ precedes EXTRACT-mmm in lexicographical order.

Use Cases:

Validate configurations using appropriate Splunk searches (e.g., index=_internal queries).
Leverage knowledge object precedence rules to manage complex configurations effectively.

Summary: Best Practices for Naming Knowledge Objects in Splunk

Importance of Unique Names for Knowledge Objects

Knowledge objects (e.g., field extractions, lookups, field aliases) should have unique names to avoid name collision issues.
Name collisions can prevent configurations from being applied correctly, especially during search-time processing.

Scenarios of Name Collisions

Within the Same Host, Source, or Source Type:
- If multiple configurations with the same name exist in a props.conf stanza, only the last configuration is applied.
- Example:
  - Conflict: Two lookups named LOOKUP-table override each other.
  - Solution: Rename configurations to reflect their purpose (e.g., LOOKUP-logs and LOOKUP-location).
Across Different Hosts, Sources, or Source Types:
- Overlapping events are affected by one configuration, typically determined by configuration file precedence.
- Example:
  - Host-specific lookups may override source-type-specific lookups if events match both criteria.
Across Different Apps:
- Configurations with the same name in different apps are applied in reverse lexicographical order of app directories.
- Example:
  - Rename conflicting configurations (e.g., FIELDALIAS-sshd → FIELDALIAS-global_sshd).

Lexicographical Ordering in Splunk

Order Based on UTF-8 Encoding:
- Numbers < Uppercase Letters < Lowercase Letters.
- Example: 10, 100, 70, 9 → 10, 100, 70, 9.

Naming Conventions

Purpose: To avoid confusion, improve organization, and make the purpose of knowledge objects clear.
Benefits:
- Helps distinguish between similar knowledge objects.
- Identifies team, technology, or purpose at a glance.
Develop Early: Consistent naming conventions minimize confusion and ensure scalability.

Example Naming Convention for Reports

Combine attributes like:
- Group: Team or department (e.g., SEG, NOC).
- Search Type: Alert, Report, Summary.
- Platform: Technology targeted (e.g., Windows, Network).
- Category: Focus area (e.g., CPU, Security).
- Time Interval: Frequency of the search.
- Description: Brief, meaningful context.
Sample Reports:
- SEG_Alert_Windows_Eventlog_15m_Failures
- NOC_Summary_Network_Security_24hr_Top_src_ip

Key Takeaway

Adopting unique names and consistent naming conventions for knowledge objects ensures smooth operations, avoids name conflicts, and simplifies management within Splunk deployments.

SUMMARISE: Manage knowledge object permissions This topic assumes that as a knowledge manager you have an ''admin'' role or a role with an equivalent permission set. As a Knowledge Manager, you can set knowledge object permissions to restrict or expand access to the variety of knowledge objects in your Splunk deployment. In some cases you will determine that certain specialized knowledge objects should only be used by people in a particular role, within a specific app. In others you will make universally useful knowledge objects globally available to users in all apps in your Splunk platform implementation. As with all aspects of knowledge management, you want to carefully consider the implications of the access restrictions and expansions that you enact. When a Splunk user first creates a new report, event type, transaction, or similar knowledge object, it is only available to that user. To make that object available to more people, Splunk Web provides the following options, which you can take advantage of if your permissions enable you to do so. You can: Make the knowledge object available globally to all users of all apps (also referred to as "promoting" an object). Make the knowledge object available to all roles associated with an app. Restrict (or expand) access to global or app-specific knowledge objects by role. Set read/write permissions at the app level for roles, to enable users to share or delete knowledge objects they do not own. By default, only people with a power or admin role can share and promote knowledge objects. This makes you and your fellow knowledge managers gatekeepers with approval capability over the sharing of new knowledge objects. Users with the admin role can change permissions for any knowledge object. Users with the power role can change permissions for the objects that they own. For information about giving roles other than admin and power the ability to set knowledge object permissions, see Enable a role other than Admin and Power to set permissions and share objects. How do permissions affect knowledge object usage? To illustrate how these choices can affect usage of a knowledge object, imagine that Finn , a user of a (fictional) Network Security app with an admin-level "Firewall Manager" role, creates a new event type named firewallbreach, which finds events that indicate firewall breaches. Here's a series of permissions-related issues that could come up, and the actions and results that would follow: Issue Action Result 21 Issue When Finn first creates firewallbreach, it is only available to Finn. Other users cannot see it or work with it. Finn decides to share it with their fellow Network Security app users. A bit later on, Mary, the knowledge manager, realizes that only people with the Firewall Manager role should have the ability to edit or update the firewallbreach event type. At some point a few people who have grown used to using the firewallbreach event type in the Network Security app decide they would also like to use it in the context of the Windows app. Action Finn updates the permissions of the firewallbreach event type so that it is available to all users of the Network Security app, regardless of role. They also set up the new event type so that all Network Security users can edit its definition. Mary restricts the ability to edit the event type to the Firewall Manager role. Result Anyone using Splunk Enterprise in the Network Security app context can see, work with, and edit the firewallbreach event type. Users of other apps in the same Splunk deployment have no idea it exists. Users of the Network Security app can use the firewallbreach event type in transactions, searches, dashboards, and so on, but now the only people that can edit the knowledge object are those with the Firewall Manager role and people with admin-level permissions (such as the knowledge manager). Splunk users in other app contexts remain unaware of the event type. They make their case to the knowledge manager, who promptly promotes the firewallbreach event type to global availability. Permissions - Getting started To change the permissions for a knowledge object, follow these steps: 1. Now, everyone that uses this Splunk deployment can use the firewallbreach event type, no matter what app context they happen to be in. But the ability to update the event type definition is still confined to admin-level users and users with the Firewall Manager role. In Splunk Web, navigate to the page for the type of knowledge object that you want to update permissions for. 2. 3. 4. Find the knowledge object that you created (use the filtering fields at the top of the page if necessary) and open its permissions dialog. In some cases you will need to click a Permissions link to do this. In other cases you need to make a menu selection such as Edit > Edit Permissions or Manage > Edit Permissions. If you are on a listing page you can also expand the object row and click Edit for Permissions. On the Permissions page for the knowledge object in question, perform the actions in the following subsections depending on how you'd like to change the object's permissions. Click Save to save your changes. 22 Make an object available to users of all apps To make an object globally available to users of all apps in your Splunk deployment: 1. 2. 3. 4. Navigate to the Permissions page for the knowledge object (following the instructions above). For Display for, select All apps. In the Permissions section, for Everyone, select a permission of either Read or Write. Option Definition Read Write When this is selected for a role, people with the role can see and use the object, but not update its definition. For example, when a role only has Read permission for a report, people with that role can see the report in the Reports listing page and they can run it. They cannot update the report's search string, change its time range, or save their changes. When this is selected for a role, people with the role can view, use, and update the defining details of the knowledge object as necessary. If neither Read or Write is selected for a role, people with that role cannot see or use the knowledge object. Save the permission change. Make an object available to all users of its app All knowledge objects are associated with an app. When you create a new knowledge object, it is associated with the app context that you are in at the time. In other words, if you are using the Search & Reporting app when you create the object, the object will be listed in Settings with Search & Reporting as its App column value. This means that if you restrict its sharing permissions to the app level it will only be available to users of the Search & Reporting app. When you create a new object, you are given the option of keeping it private, sharing it with users of the app that you're currently using, or sharing it globally with all users. Opt to make the app available to "this app only" to restrict its usage to users of that app, when they are in that app context. If you have write permissions for an object that already exists, you can change its permissions so that it is only available to users of its app by following these steps. 23 1. 2. 3. 4. Navigate to the Permissions page for the knowledge object (following the instructions in "Permissions - Getting Started," above). For Display for, select App. In the Permissions section, for Everyone, select a permission of either Read or Write. Option Read Write Definition When this is selected for a role, people with that role can see and use the object, but not update its definition. For example, when a role only has Read permission for a report, people with that role can see the report in the Reports listing page and they can run it. They cannot update the report's search string, change its time range, or save their changes. When this is selected for a role, people with the role can view, use, and update the defining details of the knowledge object as necessary. If neither Read or Write is selected for a role, people with that role cannot see or use the knowledge object. Save the permission change. Moving or cloning a knowledge object You may run into situations where you want users of an app to be able to access a particular knowledge object that belongs to a different app, but you do not want to share that object globally with all apps. There are two ways you can do this: by cloning the object, or by moving it. Option Clone Move Definition Make a copy of a knowledge object. The copy has all of the same settings as the original object, which you can keep or modify. You can keep it in the same app as the object you're cloning, or you can put it in a new app. If you add the cloned object to the same app as the original, give it a different name. You can keep the original name if you add the object to an app that doesn't have a knowledge object of the same type with that name. You can clone any object, even if your role does not have write permissions for it. Move an existing knowledge object to another app. Removes the object from its current app and places it in an app that you determine. Once there, you can set its permissions so that it is private, globally available, or only available to users of that app. The ability to move an app is connected to the same permissions that determine whether you can delete an app. You can only move a knowledge object if you have created that object and have write permissions for the app to which it belongs. Switching the app context of an knowledge object by moving it can have downstream consequences for objects that have been associated with it. See Disable or delete knowledge objects. You can find the Clone and Move controls on the Settings pages for various knowledge object types. To clone or move an object, find the object in its list and click Clone or Move. Restrict knowledge object access by app and role You can use this method to lock down various knowledge objects from alteration by specific roles. You can arrange things so users in a particular role can use the knowledge object but not update it--or you can set it up so those users cannot see the object at all. In the latter case, the object will not show up for them in Splunk Web, and they will not find any results when they search on it. If you want restrict the ability to see or update a knowledge object by role, simply navigate to the Permissions page for the object. If you want members of a role to: Be able to use the object and update its definition, give that role Read and Write access. Be able to use the object but be unable to update it, give that role Read access only (and make sure that Write is unchecked for the Everyone role). Be unable to see or use the knowledge object at all, leave Read and Write unchecked for that role (and unchecked for the Everyone role as well). 24 See About role-based user access in Securing Splunk Enterprise. Enable a role other than admin and power to set permissions and share objects By default, only the admin and power roles can set permissions for knowledge objects. Follow these steps to give another role the ability to set knowledge object permissions. This allows a user who has this role to set permissions for the knowledge objects that they own. Only a user with the admin role can change permissions for knowledge objects that are owned by another user. The ability to set permissions for knowledge objects is controlled at the app level. It is not connected to a role capability like other actions such as scheduling searches or changing default input settings. You have to give roles write permissions to an app to enable people with those roles to manage the permissions of knowledge objects created in the context of that app. Steps 1. 2. 3. 4. 5. From the Splunk Home page, select any app in the Apps Panel to open the app. Click on the Applications menu in the Splunk bar, and select Manage Apps. Find the app that you want to adjust permissions for and open its Permissions settings. On the Permissions page for the app, give the role Read and Write permissions. Click Save to save your changes. Users whose roles have write permissions to an app can also delete knowledge objects that are associated with that app. For more information, see Disable or delete knowledge objects. Set permissions for categories of knowledge objects You can set role-based permissions for specific knowledge object types by making changes to the default.meta file. For example, you can give all user roles the ability to set permissions for all saved searches in a specific app. See Set permissions for objects in a Splunk App on the Splunk Developer Portal.

Summary: Managing Knowledge Object Permissions in Splunk

Overview: Knowledge managers in Splunk with an admin role (or equivalent permissions) can manage permissions for knowledge objects to control user access across apps. Permissions determine whether objects like reports or event types are private, app-specific, or globally accessible.

Permission Management Options:

Scope of Accessibility:
- Private: Objects are visible only to their creator by default.
- App-specific: Accessible to users within a specific app.
- Global: Accessible across all apps for all users.
Actions by Knowledge Managers:
- Share objects globally or app-specifically.
- Restrict or expand access by role.
- Set read (view and use) and write (modify) permissions.
Role-Based Permissions:
- Admins can manage all objects.
- Power users can manage permissions for objects they own.
- Other roles can be configured to manage permissions via app-level adjustments.

Illustrative Example:

A user creates an event type, firewallbreach:
- Initially, only the creator can access it.
- Permissions can be expanded to app users or globally.
- Edit permissions can be restricted to specific roles (e.g., Firewall Manager).

Steps for Managing Permissions:

Navigate to the knowledge object's Permissions page.
Adjust visibility (Private, App, or All Apps).
Configure read or write permissions for roles.
Save changes.

Advanced Options:

Clone or Move Objects:
- Clone: Duplicate objects across apps or contexts.
- Move: Transfer objects to different apps, adjusting permissions accordingly.
Restrict by Role and App:
- Customize role-based access to prevent editing or hide objects from specific roles.
Enable Role-Based Permission Management:
- Grant roles write permissions at the app level to allow them to set permissions for their created objects.

Configuring Permissions via Default Files:

Use default.meta to set permissions for specific categories of knowledge objects.

Key Considerations:

Changes in app context or permissions can impact associated objects.
Carefully manage global or app-specific access to balance utility and security.

For detailed steps, refer to Splunk's official documentation or manage permissions via Splunk Web.

SUMMARISE:About deleting users who own knowledge objects If you delete a user from your Splunk deployment, the objects that user owns become orphaned. Orphaned objects can have serious implications. For example, when a scheduled report or alert becomes orphaned, it ceases to run on its schedule. When this happens, your team can miss important alerts, actions that are tied to affected scheduled reports will cease to function, and any dashboard panels that are associated with affected scheduled reports will stop showing search results. To prevent this from happening, reassign knowledge objects to another user. For more information see Manage orphaned knowledge objects. Manage orphaned knowledge objects When a knowledge object owner leaves a department or company and their Splunk account is deactivated, the knowledge objects that they owned remain in the system. These are orphaned knowledge objects. Knowledge objects without valid owners can cause problems. For example, searches that refer to orphaned lookup definitions may not work. 25 Orphaned scheduled reports can be a particular problem. The search scheduler cannot run a scheduled report on behalf of a nonexistent owner. This happens because the scheduled report is no longer associated with a role. Without that role association, the search scheduler has no way of knowing what search quotas and concurrent search configurations the report is limited by. As a result, it will not run the scheduled report on its schedule at all. This can result in broken dashboards and embedded searches, data collection gaps in summary indexes, and more. The Splunk software provides several methods of detecting orphaned knowledge objects. Once you have found orphaned knowledge objects, you have several options for resolving their orphaned status. Find orphaned knowledge objects There are several ways that you can find out whether or not you have orphaned knowledge objects in your Splunk implementation. Most of these detection methods specialize in orphaned scheduled reports, because they tend to cause the most problems for users. The Reassign Knowledge Objects page in Settings is the only orphaned knowledge object detection method that can find all orphaned knowledge object types. It can only find orphaned knowledge objects that have been shared at the app or global levels. These detection methods have no way of knowing when people are removed using a third-party user authentication system. Review orphaned scheduled search notifications By default your Splunk implementation runs a search to find orphaned scheduled reports on a daily schedule. When it finds orphaned scheduled reports it creates a notification message. If you open that message you can click a link to see a list of the orphaned reports in the Orphaned Scheduled Searches, Reports, and Alerts dashboard. Steps 1. 2. Click Messages when the UI indicates there are messages there. If you find a message indicating that the Splunk software has found orphaned scheduled searches, click the message link to run a search that displays the orphaned scheduled searches. Look at the Orphaned Scheduled Searches, Reports, and Alerts dashboard and report The Orphaned Scheduled Searches, Reports, and Alerts dashboard is delivered with your Splunk platform deployment in the Search & Reporting app. The dashboard loads with the results of the Orphaned Searches, Reports and Alerts report, which is designed to return the names of any orphaned scheduled reports in your system. You can run the Orphaned Searches, Reports And Alerts report directly from the Reports listing page to get the same results. Run the Monitoring Console health check in Splunk Enterprise If you have the Monitoring Console configured for your Splunk Enterprise instance, you can use its health check feature to detect orphaned scheduled searches, reports, and alerts. It will tell you how many of these knowledge objects exist in your system. You have to run a drilldown search to see a list that identifies the orphaned searches by name. Prerequisites 26 Access and customize health check in the Admin Manual. Steps 1. 2. 3. 4. Open the Monitoring Console by selecting Settings > Monitoring Console. Select the Health Check tab. Click the Start button in the upper right corner to run the health check. By default, the health check will search for orphaned scheduled searches, reports, and alerts. If the health check finds orphaned searches, the Monitoring Console marks the Orphaned Scheduled Searches row with an Error notification moves it to the top of the health check list, alongside other rows with Error notifications. Click the Orphaned Scheduled Searches row to launch a drill down search. This search displays the names of the orphaned scheduled searches. Use the Reassign Knowledge Objects page in Settings The Reassign Knowledge Objects page in Settings is the only orphaned knowledge object detection method that detects all knowledge objects (not just searches, reports, and alerts). However, it can only find knowledge objects that have been shared to the app or global levels. Steps 1. 2. 3. Select Settings > All configurations. Click Reassign Knowledge Objects. Click Orphaned to filter out non-orphaned objects from the list. At this point, the list should only contain orphaned objects that have been shared. Now you can determine what you want to do with the items in that list. Reassign one or more shared knowledge objects to a new owner Use the Reassign Knowledge Objects page in Settings to reassign a knowledge object to a new owner. The Reassign Knowledge Objects page can reassign both owned and orphaned knowledge objects. It is designed to work with all Splunk deployments, including those that use search head clustering (SHC). The Reassign Knowledge Objects page cannot reassign knowledge objects that are both orphaned and privately shared. See Reassign private orphaned knowledge objects. Knowledge object ownership changes can have side effects such as giving saved searches access to previously inaccessible data or making previously available knowledge objects unavailable. Review your knowledge objects before you reassign them. Only users with the Admin role can reassign knowledge objects to new owners. Find the knowledge object or objects you want to reassign First, you need to use the filtering options on the Reassign Knowledge Objects page to help you find the knowledge object or objects that you want to reassign. Steps 27 1. 2. 3. Select Settings > All configurations. Click Reassign Knowledge Objects. Find the object or objects that you want to reassign. Objects to find Objects belonging to an owner with an active account. Step to follow Click Filter by Owner and select the owner name from the dropdown. All shared, orphaned objects. Objects belonging to a specific knowledge object type. Objects belonging to a specific app. Click Orphaned. Make a selection from the Object type dropdown. Make a selection from the App dropdown. You can optionally switch All Objects to Objects created in the app to filter out objects created in apps other than the app you have selected. Objects that include a particular text string. Enter the string into the filter field and click Return. Your next steps depend on how many knowledge objects you want to reassign to a different owner. Reassign a single knowledge object to another owner If you are using the Reassign Knowledge Objects page to reassign an individual object to another owner, follow these steps. Prerequisites Find the knowledge object you want to reassign before starting this task. Steps 1. 2. 3. For the knowledge object that you want to reassign, click Reassign in the Action column. Click Select an owner and select the name of the person that you want to reassign the knowledge object to. Click Save to save your changes. Bulk-reassign multiple knowledge objects to another owner If you are using the Reassign Knowledge Objects page to reassign multiple objects to another owner, follow these steps. Prerequisites Find the knowledge objects you want to reassign before starting this task. Steps 1. 2. 3. 4. Select the checkboxes next to the objects that you want to reassign. If you want to reassign all objects in the list to the same owner, click the checkbox at the top of the checkbox column. You can reassign up to 100 objects in one bulk reassignment action. Click Edit Selected Knowledge Objects and select Reassign. (Optional) Remove knowledge objects that you have accidentally selected by clicking the X symbols next to their names. Click Select an owner and select the name of the person that you want to bulk-reassign the selected knowledge objects to. 28 5. Click Save to save your changes. Reassign unshared, orphaned knowledge objects If you want to reassign orphaned knowledge objects that had a Private sharing status when they were orphaned, you cannot reassign them through the UI. There are two ways to reassign unshared (private), orphaned knowledge objects. You can temporarily recreate the invalid owner, or you can copy and paste the knowledge object stanza between the configuration files of the invalid and valid owners. Temporarily recreate the invalid owner The easiest solution for this is to temporarily recreate the invalid owner account, reassign the knowledge object, and then deactivate the invalid owner account. Prerequisites See About users and roles in the Admin Manual to learn how to add and remove users from your Splunk implementation. Steps 1. 2. 3. Add the invalid knowledge object owner as a new user in your Splunk deployment. Use the Reassign Knowledge Objects page to assign the knowledge object to a different active owner. Deactivate the invalid owner account. Perform a knowledge object stanza copy and paste operation between two .conf files If you cannot reactivate invalid owner accounts, you can transfer ownership of unshared and orphaned knowledge objects by performing a .conf file stanza cut and paste operation. You cut the knowledge object stanza out of a .conf file belonging to the invalid owner and paste it into the corresponding .conf file of a valid owner. Prerequisites To use this method you must meet the following requirements: You must be using Splunk Enterprise. You must have filesystem access. You cannot be running SHC on your Splunk deployment. You must be able to restart your Splunk deployment. Steps 1. 2. 3. 4. 5. 6. 7. In the filesystem of your Splunk deployment, open the the .conf file for the invalid owner at etc/users/<name_of_invalid_user>/search/local/<name_of_conf_file>. Locate the stanza for the orphaned knowledge object and cut it out. Save your changes to the file and close it. Open the the corresponding .conf file for the new owner at etc/users/<name_of_valid_user>/search/local/<name_of_conf_file>. Copy the knowledge object stanza that you just cut to the .conf file for the valid owner. Save your changes to the file and close it. Restart your Splunk deployment so the changes take effect. 29 About resolving orphaned scheduled searches The action you take to resolve an orphaned scheduled search, report, or alert depends on what you want it to do going forward. Objective Keep the search running on its schedule Let the search run on an ad-hoc basis. Step to follow Reassign the search to a new owner. Remove the schedule for the search from its definition in Settings > Searches, reports, and alerts. If the search has been shared with other users of an app, users of that app can run it. This can be important if it is used in a dashboard, for example. However, you may need to ensure that other users do not schedule it again. You can do this by limiting the number of roles that have edit access to the search. Alternatively, you can reassign it to a new owner as an unscheduled search. Keep the search from running again under any circumstances. Disable the search, or delete it. Turn off notifications of orphaned searches By default, Splunk software notifies you about orphaned searches. If you would rather not receive these notifications, open limits.conf, look for the [system_checks] stanza, and set orphan_searches to disabled. Disable or delete knowledge objects Your ability to delete knowledge objects in Splunk Web depends on a set of factors: You cannot delete default knowledge objects that were delivered with Splunk software (or with an app). If the knowledge object definition resides in a default directory, it can't be removed through Splunk Web. It can be disabled by clicking Disable for the object in Settings. Only objects that exist in an app's local directory are eligible for deletion. You can always delete knowledge objects that you have created, and which haven't been shared by you or someone with admin-level permissions. Once you share a knowledge object you've created with other users, your ability to delete it is revoked, unless you have write permissions for the app to which they belong. To delete any other knowledge object, your role must have write permissions for the app to which the knowledge object belongs. This applies to knowledge objects that are shared globally as well as those that are only shared within an app. All knowledge objects belong to a specific app, no matter how they are shared. App-level write permissions are usually only granted to users with admin-equivalent roles. If a role does not have write permissions for an app but does have write permissions for knowledge objects belonging to that app, it can disable those knowledge objects. Clicking Disable for a knowledge object has the same function as knowledge object deletion, with the exception that Splunk software does not remove disabled knowledge objects from the system. A role with write permissions for a disabled knowledge object can re-enable it at any time. There are similar rules for data models. To enable a role to create data models and share them with others, the role must 30 be given write access to an app. This means that users who can create and share data models can potentially also delete knowledge objects. For more information, see Manage data models. Grant a role write permissions for an app If your role has admin-level permissions, you can grant a role write permissions for an app in Splunk Web. Once a role has write permissions for an app, users with that role can delete any knowledge object belonging to that app. Users whose roles have write permissions to an app can delete knowledge objects that belong to that app. This is true whether the knowledge object is shared just to the app, or globally to all apps. Even when knowledge objects are shared globally they belong to a specific app. For more information, see Manage knowledge object permissions. Steps 1. 2. 3. 4. 5. From the Splunk Home page, select any app in the Apps Panel to open the app. Click on the Applications menu in the Splunk bar, and select Manage Apps. Find the app that you want to adjust permissions for and open its Permissions settings. Select Write for the roles that should be able to delete knowledge objects for the app. Click Save to save your changes. You can also manage role-based permissions for an app by updating its local.meta file. For more information see Setting access to manager consoles and apps in Securing Splunk Enterprise. Deleting knowledge objects with downstream dependencies You have to be careful about deleting knowledge objects with downstream dependencies, as this can have negative impacts. For example, you could have a tag that looks like the duplicate of another, far more common tag. On the surface, it would seem to be harmless to delete the dup tag. But what you may not realize is that this duplicate tag also happens to be part of a search that a very popular event type is based upon. And that popular event type is used in two important reports--the first is the basis for a well-used dashboard panel, and the other is used to populate a summary index that is used by searches that run several other dashboard panels. So if you delete that tag, the event type breaks, and everything downstream of that event type breaks. This is why it is important to fix poorly named or defined knowledge objects before they become inadvertently hard-wired into the workings of your deployment. The only way to identify the downstream dependencies of a particular knowledge object is to search on it, find out where it is used, and then search on those things to see where they are used--it can take a bit of detective work. There is no "one click" way to bring up a list of knowledge object downstream dependencies at this point. If you really feel that you have to delete a knowledge object, and you're not sure if you've tracked down and fixed all of its downstream dependencies, you could try disabling it first to see what impact that has. If nothing breaks after a day or so, delete it. 31 Deleting knowledge objects in configuration files In Splunk Web, you can only disable or delete one knowledge object at a time. If you need to remove large numbers of objects, the most efficient way to do it is by removing the knowledge object stanzas directly through the configuration files. Keep in mind that several versions of a particular configuration file can exist within your system. In most cases you should only edit the configuration files in $SPLUNK_HOME/etc/system/local/, to make local changes on a site-wide basis, or $SPLUNK_HOME/etc/apps/<App_name>/local/, if you need to make changes that apply only to a specific app. Do not try to edit configuration files until you have read and understood the following topics in the Admin manual: About configuration files Configuration file precedence

Summary: Deleting Users and Managing Orphaned Knowledge Objects in Splunk

When a user is deleted from Splunk, the knowledge objects they own become orphaned, leading to potential issues such as failed scheduled reports, broken dashboards, and disrupted workflows. To avoid these problems, orphaned knowledge objects should be reassigned to new owners. Below are the key aspects of managing orphaned knowledge objects:

Implications of Orphaned Objects

Scheduled reports cease to run, causing missed alerts and incomplete dashboards.
Searches referring to orphaned objects, such as lookup definitions, may fail.
Orphaned objects lose role associations, disabling the search scheduler.

Detection Methods

Reassign Knowledge Objects Page: Detects all orphaned objects shared at app or global levels.
Notifications: Splunk generates daily alerts for orphaned scheduled searches.
Orphaned Searches Dashboard: Displays orphaned reports for review.
Monitoring Console Health Check: Identifies the number of orphaned objects, with drill-down details.

Reassigning Knowledge Objects

Reassign via UI: Admin users can use the "Reassign Knowledge Objects" page to assign objects to active users.
Bulk Reassignment: Reassign up to 100 objects simultaneously using selection filters.
Reassign Private Objects:
- Temporarily recreate the deleted user to reassign objects.
- Alternatively, transfer ownership by editing .conf files directly in the file system.

Handling Orphaned Scheduled Searches

Keep Search Active: Reassign to a new owner.
Run Search Ad-Hoc: Remove its schedule but allow manual execution.
Disable Search: Prevent it from running entirely.

Deleting Knowledge Objects

Default knowledge objects cannot be deleted but can be disabled.
Deletion depends on ownership, sharing permissions, and app-level write access.
Care is required to identify downstream dependencies to avoid breaking dependent dashboards or reports.

Best Practices

Reassign objects before deactivating user accounts.
Review downstream dependencies to prevent disruptions.
Use disabling as a temporary measure to assess impact before deletion.

By proactively managing orphaned knowledge objects, Splunk administrators can ensure the system operates seamlessly even after user deletions.

SUMMARISE: About Splunk regular expressions This primer helps you create valid regular expressions. For a discussion of regular expression syntax and usage, see an online resource such as www.regular-expressions.info or a manual on the subject. Regular expressions match patterns of characters in text and are used for extracting default fields, recognizing binary file types, and automatic assignation of source types. You also use regular expressions when you define custom field extractions, filter events, route data, and correlate searches. Search commands that use regular expressions include rex and regex and evaluation functions such as match and replace. Splunk regular expressions are PCRE (Perl Compatible Regular Expressions) and use the PCRE C library. The Splunk platform includes the license for PCRE2, an improved version of PCRE. However, the Splunk platform does not currently allow access to functions specific to PCRE2, such as key substitution. Regular expressions terminology and syntax Term Description literal The exact text of characters to match using a regular expression. regular expression The metacharacters that define the pattern that Splunk software uses to match against the literal. groups Regular expressions allow groupings indicated by the type of bracket used to enclose the regular expression characters. Groups can define character classes, repetition matches, named capture groups, modular regular expressions, and more. You can apply quantifiers to and use alternation within enclosed groups. character class Characters enclosed in square brackets. Used to match a string. To set up a character class, define a range with a hyphen, such as [A-Z], to match any uppercase letter. Begin the character class with a caret (^) to define a negative match, such as [^A-Z] to match any lowercase letter. character type Similar to a wildcard, character types represent specific literal matches. For example, a period . matches any character, \w matches words or alphanumeric characters including an underscore, and so on. anchor Character types that match text formatting positions, such as return (\r) and newline (\n). alternation Refers to supplying alternate match patterns in the regular expression. Use a vertical bar or pipe character ( | ) to separate the alternate patterns, which can include full regular expressions. For example, grey|gray matches either grey or gray. 32 Term Description quantifiers, or repetitions Use ( *, +, ? ) to define how to match the groups to the literal pattern. For example, * matches 0 or more, + matches 1 or more, and ? matches 0 or 1. back references Literal groups that you can recall for later use. To indicate a back reference to the value, specify a dollar symbol ($) and a number (not zero). lookarounds A way to define a group to determine the position in a string. This definition matches the regular expression in the group but gives up the match to keep the result. For example, use a lookaround to match x that is followed by y without matching y. escape A method of allowing special characters that have meaning in regular expressions, such as periods ( . ), double quotation marks ( " ), and backslash marks ( \ ) to be ignored when they occur in the text that you want to match with the regular expression. The backslash character escapes special characters. Character types Character types are short for literal matches. Term Description Example Explanation \w Match a word character (a letter, number, or underscore character). \w\w\w Matches any three word characters. \W Match a non-word character. \W\W\W Matches any three non-word characters. \d Match a digit character. \d\d\d-\d\d-\d\d\d\d Matches a Social Security number, or a similar 3-2-4 number string. \D Match a non-digit character. \D\D\D Matches any three non-digit characters. \s Match a whitespace character. \d\s\d Matches a sequence of a digit, a whitespace, and then another digit. \S Match a non-whitespace character. \d\S\d Matches a sequence of a digit, a non-whitespace character, and another digit. . Match any character. Use sparingly. \d\d.\d\d.\d\d Matches a date string such as 12/31/14 or 01.01.15, but can also match 99A99B99. Groups, quantifiers, and alternation Regular expressions allow groupings indicated by the type of bracket used to enclose the regular expression characters. You can apply quantifiers ( *, +, ? ) to the enclosed group and use alternation within the group. Term Description Example Explanation * Match zero or more times. \w* Matches zero or more word characters. + Match one or more times. \d+ Match at least one digit. ? Match zero or one time. \d\d\d-?\d\d-?\d\d\d\d Matches a Social Security Number with or without dashes. ( ) Parentheses define match or capture groups, atomic groups, and lookarounds. (H..).(o..) When given the string Hello World, this matches Hel and o W. [ ] Square brackets define character classes. [a-z0-9#] Matches any character that is a through z, 0 through 9, or #. 33 Term { } < > Description Example Explanation Curly brackets define repetitions. Angle brackets define named capture groups. Use the syntax (?P<var> ...) to set up a named field extraction. [[ ]] Double brackets define Splunk-specific modular regular expressions. \d{3,5} Matches a string of 3 to 5 digits in length. (?P<ssn>\d\d\d-\d\d-\d\d\d\d) Pulls out a Social Security Number and assigns it to the ssn field. [[octet]] A simple example of groups, quantifiers, and alternation This example shows two ways to match either to or too. The first regular expression uses the ? quantifier to match up to one more "o" after the first. The second regular expression uses alternation to specify the pattern. to(o)? (to|too) Capture groups in regular expressions A validated 0-255 range integer. A named capture group is a regular expression grouping that extracts a field value when regular expression matches an event. Capture groups include the name of the field. They are notated with angle brackets as follows: matching text (?<field_name>capture pattern) more matching text. For example, you have this event text: 131.253.24.135 fail admin_user Here are two regular expressions that use different syntax in their capturing groups to pull the same set of fields from that event. Expression A: (?<ip>\d+\.\d+\.\d+\.\d+) (?<result>\w+) (?<user>.*) Expression B: (?<ip>\S+) (?<result>\S+) (?<user>\S+) In Expression A, the pattern-matching characters used for the first capture group (ip) are specific. \d means "digit" and + means "one or more." So \d+ means "one or more digits." \. refers to a period. The capture group for ip wants to match one or more digits, followed by a period, followed by one or more digits, followed by a period, followed by one or more digits, followed by a period, followed by one or more digits. This describes the syntax for an ip address. The second capture group in Expression A for the result field has the pattern \w+, which means "one or more alphanumeric characters." The third capture group in Expression A for the user field has the pattern .*, which means "match everything that's left." Expression B uses a common technique called negative matching. With negative matching, the regular expression does not try to define which text to match. Instead it defines what the text is not. In this Expression B, the values that should be extracted from the sample event are "not space" characters (\S). It uses the + to specify "one or more" of the "not space" 34 characters. So Expression B says: 1. 2. 3. 4. 5. Pull out the first string of not-space characters for the ip field value. Ignore the following space. Then pull out the second string of not-space characters for the result field value. Ignore the second space. Pull out the third string of not-space characters for the user field value." Non-capturing group matching Use the syntax (?: ... ) to create groups that are matched but which are not captured. Note that here you do not need to include a field name in angle brackets. The colon character after the ? character is what identifies it as a non-capturing group. For example, (?:Foo|Bar) matches either Foo or Bar, but neither string is captured. Escape special characters You can use the backslash character ( \ ) to escape to escape any special characters that have meaning in regular expressions, such as periods ( . ), double quotation marks ( " ), and backslashes themselves. For details, see: SPL and regular expressions in the Search Reference. Backslashes, in the Search Manual. Modular regular expressions Modular regular expressions refer to small chunks of regular expressions that are defined to be used in longer regular expression definitions. Modular regular expressions are defined in transforms.conf. For example, you can define an integer and then use that regular expression definition to define a float. [int] # matches an integer or a hex number REGEX = 0x[a-fA-F0-9]+|\d+ [float] # matches a float (or an int) REGEX = \d*\.\d+|[[int]] In the regular expression for [float], the modular regular expression for an integer or hex number match is invoked with double square brackets, [[int]]. You can also use the modular regular expression in field extractions. [octet] # this would match only numbers from 0-255 (one octet in an ip) REGEX = (?:2(?:5[0-5]|[0-4][0-9])|[0-1][0-9][0-9]|[0-9][0-9]?) [ipv4] # matches a valid IPv4 optionally followed by :port_num the # octets in the ip would also be validated 0-255 range 35 # Extracts: ip, port REGEX = (?<ip>[[octet]](?:\.[[octet]]){3})(?::[[int:port]])? The [octet] regular expression uses two nested non-capturing groups to do its work. See the subsection in this topic on non-capturing group matching

Summary: Splunk Regular Expressions

Splunk uses Perl Compatible Regular Expressions (PCRE) for matching patterns in text to extract fields, filter events, and perform searches. Regular expressions (regex) are versatile tools for defining patterns using metacharacters, and Splunk enhances their functionality with modular expressions and field extraction features.

Key Concepts:

Regex Usage in Splunk:
- Default field extraction, binary file recognition, and source type assignment.
- Commands like rex and regex or functions like match and replace.
- Modular regex definitions in transforms.conf.
Basic Regex Syntax:
- Literals: Exact text to match.
- Metacharacters: Patterns like . (any character), \d (digit), or \w (word character).
- Anchors: Define positions, e.g., ^ (start of string) or $ (end of string).
- Quantifiers: Define repetitions (*, +, ?, {n,m}).
- Groups: Use parentheses ( ) for grouping, alternation (|), or non-capturing groups (?: ...).
- Character Classes: Enclosed in [ ], e.g., [A-Z] (uppercase letters), [^A-Z] (not uppercase).
- Escape Characters: Use \ to neutralize special characters.

Modular Regular Expressions:

Defined in transforms.conf for reusable patterns.
Example:
- [int] matches integers or hex values.
- [float] builds upon [int] to define floating-point numbers.
- [octet] validates numbers from 0-255 for IP addresses.

Capture Groups:

Extract fields using named groups:
- Syntax: (?<field_name>pattern).
- Example: (?<ip>\d+\.\d+\.\d+\.\d+) extracts an IP address.
Non-capturing groups use (?: ...) for grouping without capturing values.

Key Features:

Lookarounds: Match patterns based on preceding or succeeding text without capturing it.
Negative Matching: Matches what text is not (e.g., \S for non-whitespace characters).
Modular Patterns: Example: [ipv4] uses nested regex to validate IPv4 addresses.

Splunk regex provides a powerful mechanism for parsing and manipulating data with reusable, precise, and modular definitions, enhancing the platform's search and analysis capabilities.

SUMMARISE: Fields and field extractions About fields Fields appear in event data as searchable name-value pairings such as user_name=fred or ip_address=192.168.1.1. Fields are the building blocks of Splunk searches, reports, and data models. When you run a search on your event data, Splunk software looks for fields in that data. Look at the following example search. status=404 This search finds events with status fields that have a value of 404. When you run this search, Splunk Enterprise does not look for events with any other status value. It also does not look for events containing other fields that share 404 as a value. As a result, this search returns a set of results that are more focused than you get if you used 404 in the search string. Fields often appear in events as key=value pairs such as user_name=Fred. But in many events, field values appear in fixed, delimited positions without identifying keys. For example, you might have events where the user_name value always appears by itself after the timestamp and the user_id value. Nov 15 09:32:22 00224 johnz Nov 15 09:39:12 01671 dmehta Nov 15 09:45:23 00043 sting Nov 15 10:02:54 00676 lscott Splunk Enterprise can identify these fields using a custom field extraction. About field extraction As Splunk software processes events, it extracts fields from them. This process is called field extraction. Automatically-extracted fields Splunk software automatically extracts host, source, and sourcetype values, timestamps, and several other default fields when it indexes incoming events. It also extracts fields that appear in your event data as key=value pairs. This process of recognizing and extracting k/v pairs is called field discovery. You can disable field discovery to improve search performance. When fields appear in events without their keys, Splunk software uses pattern-matching rules called regular expressions to extract those fields as complete k/v pairs. With a properly-configured regular expression, Splunk Enterprise can extract user_id=johnz from the previous sample event. Splunk Enterprise comes with several field extraction configurations that use regular expressions to identify and extract fields from event data. For more information about field discovery and an example of automatic field extraction, see When Splunk Enterprise extracts fields. For more information on how Splunk Enterprise uses regular expressions to extract fields, see About Splunk regular expressions. 37 To get all of the fields in your data, create custom field extractions To use the power of Splunk search, create additional field extractions. Custom field extractions allow you to capture and track information that is important to your needs, but which is not automatically discovered and extracted by Splunk software. Any field extraction configuration you provide must include a regular expression that specifies how to find the field that you want to extract. All field extractions, including custom field extractions, are tied to a specific source, sourcetype, or host value. For example, if you create an ip field extraction, you might tie the extraction configuration for ip to sourcetype=access_combined. Custom field extractions should take place at search time, but in certain rare circumstances you can arrange for some custom field extractions to take place at index time. See When Splunk Enterprise extracts fields. Before you create custom field extractions, get to know your data Before you begin to create field extractions, ensure that you are familiar with the formats and patterns of the event data associated with the source, sourcetype, or host that you are working with. One way is to investigate the predominant event patterns in your data with the Patterns tab. See Identify event patterns with the Patterns tab in the Search Manual. Here are two events from the same source type, an apache server web access log. 131.253.24.135 - - [03/Jun/2014:20:49:53 -0700] "GET /wp-content/themes/aurora/style.css HTTP/1.1" 200 7464 "http://www.splunk.com/download" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; Trident/5.0)" 10.1.10.14 - - [03/Jun/2014:20:49:33 -0700] "GET / HTTP/1.1" 200 75017 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine; http://nmap.org/book/nse.html)" While these events contain different strings and characters, they are formatted in a consistent manner. They both present values for fields such as clientIP, status, bytes, method, and so on in a reliable order. Reliable means that the method value is always followed by the URI value, the URI value is always followed by the status value, the status value is always followed by the bytes value, and so on. When your events have consistent and reliable formats, you can create a field extraction that accurately captures multiple field values from them. For contrast, look at this set of Cisco ASA firewall log events: 1 Jul 15 20:10:27 10.11.36.31 %ASA-6-113003: AAA group policy for user AmorAubrey is being set to Acme_techoutbound 2 Jul 15 20:12:42 10.11.36.11 %ASA-7-710006: IGMP request discarded from 10.11.36.36 to outside:87.194.216.51 3 Jul 15 20:13:52 10.11.36.28 %ASA-6-302014: Teardown TCP connection 517934 for Outside:128.241.220.82/1561 to Inside:10.123.124.28/8443 duration 0:05:02 bytes 297 Tunnel has been torn down (AMOSORTILEGIO) 4 Apr 19 11:24:32 PROD-MFS-002 %ASA-4-106103: access-list fmVPN-1300 denied udp for user 'sdewilde7' outside/12.130.60.4(137) -> inside1/10.157.200.154(137) hit-cnt 1 first hit [0x286364c7, 0x0] " While these events contain field values that are always space-delimited, they do not share a reliable format like the preceding two events. In order, these events represent: 1. 2. A group policy change An IGMP request 38 A TCP connection 3. 4. A firewall access denial for a request from a specific IP Because these events differ so widely, it is difficult to create a single field extraction that can apply to each of these event patterns and extract relevant field values. In situations like this, where a specific host, source type, or source contains multiple event patterns, you may want to define field extractions that match each pattern, rather than designing a single extraction that can apply to all of the patterns. Inspect the events to identify text that is common and reliable for each pattern. Using required text in field extractions In the last four events, the string of numbers that follows %ASA-#- have specific meanings. You can find their definitions in the Cisco documentation. When you have unique event identifiers like these in your data, specify them as required text in your field extraction. Required text strings limit the events that can match the regular expression in your field extraction. Specifying required text is optional, but it offers multiple benefits. Because required text reduces the set of events that it scans, it improves field extraction efficiency and decreases the number of false-positive field extractions. The field extractor utility enables you to highlight text in a sample event and specify that it is required text. Methods of custom field extraction As a knowledge manager you oversee the set of custom field extractions created by users of your Splunk deployment, and you might define specialized groups of custom field extractions yourself. The ways that you can do this include: The field extractor utility, which generates regular expressions for your field extractions. Adding field extractions through pages in Settings. You must provide a regular expression. Manual addition of field extraction configurations at the .conf file level. Provides the most flexibility for field extraction. The field extraction methods that are available to Splunk users are described in the following sections. All of these methods enable you to create search-time field extractions. To create an index-time field extraction, choose the third option: Configure field extractions directly in configuration files. Let the field extractor build extractions for you The field extractor utility leads you step-by-step through the field extraction design process. It provides two methods of field extraction: regular expressions and delimiter-based field extraction. The regular expression method is useful for extracting fields from unstructured event data, where events may follow a variety of different event patterns. It is also helpful if you are unfamiliar with regular expression syntax and usage, because it generates regular expressions and lets you validate them. The delimiter-based field extraction method is suited to structured event data. Structured event data comes from sources like SQL databases and CSV files, and produces events where all fields are separated by a common delimiter, such as commas, spaces, or pipe characters. Regular expressions usually are not necessary for structured data events from a common source. With the regular expression method of the field extractor you can: Set up a field extraction by selecting a sample event and highlighting fields to extract from that event. 39 Create individual extractions that capture multiple fields. Improve extraction accuracy by detecting and removing false positive matches. Validate extraction results by using search filters to ensure specific values are being extracted. Specify that fields only be extracted from events that have a specific string of required text. Review stats tables of the field values discovered by your extraction. Manually configure regular expression for the field expression yourself. With the delimiter method of the field extractor you can: Identify a delimiter to extract all of the fields in an event. Rename specific fields as appropriate. Validate extraction results. The field extractor can only build search time field extractions that are associated with specific sources or source types in your data (no hosts). For more information about using the field extractor, see Build field extractions with the field extractor. Define field extractions with the Field extractions and Field transformations pages You can use the Field extractions and Field transformations pages in Settings to define and maintain complex extracted fields in Splunk Web. This method of field extraction creation lets you create a wider range of field extractions than you can generate with the field extractor utility. It requires that you have the following knowledge. Understand how to design regular expressions. Have a basic understanding of how field extractions are configured in props.conf and transforms.conf. If you create a custom field extraction that extracts its fields from _raw and does not require a field transform, use the field extractor utility. The field extractor can generate regular expressions, and it can give you feedback about the accuracy of your field extractions as you define them. Use the Field Extractions page to create basic field extractions, or use it in conjunction with the Field Transformations page to define field extraction configurations that can do the following things. Reuse the same regular expression across multiple sources, source types, or hosts. Apply multiple regular expressions to the same source, source type, or host. Use a regular expression to extract fields from the values of another field. The Field extractions and Field transformations pages define only search time field extractions. See the following topics in this manual: Use the Field extractions page in Splunk Web Use the Field transformations page in Splunk Web. Configure field extractions directly in configuration files To get complete control over your field extractions, add the configurations directly into props.conf and transforms.conf. This method lets you create field extractions with capabilities that extend beyond what you can create with Splunk Web methods such as the field extractor utility or the Settings pages. For example, with the configuration files, you can set up: 40 Delimiter-based field extractions. Extractions for multivalue fields. Extractions of fields with names that begin with numbers or underscores. This action is typically not allowed unless key cleaning is disabled. Formatting of extracted fields. See Create and maintain search-time extractions through configuration files. You can create index-time field extractions only by configuring them in props.conf and transforms.conf. Adding to the default set of indexed fields can result in search performance and indexing problems. But if you must create additional index-time field extractions, see Create custom fields at index time in the Getting Data In manual. Create custom calculated fields and multivalue fields Two kinds of custom fields can be persistently configured with the help of .conf files: calculated fields and multivalue fields. Multivalue fields can appear multiple times in a single event, each time with a different value. To configure custom multivalue fields, make changes to fields.conf as well as to props.conf. See Configure multivalue fields. Calculated fields provide values that are calculated from the values of other fields present in the event, with the help of eval expressions. Configure them in props.conf. See About calculated fields. Build field extractions into search strings The following search commands facilitate the search-time extraction of fields in different ways: rex extract multikv spath xmlkv xpath kvform See Extract fields with search commands in the Search Manual. Alternatively you can look up each of these commands in the Search Reference. Field extractions facilitated by search commands apply only to the results returned by the searches in which you use these commands. You cannot use these search commands to create reusable extractions that persist after the search is completed. For that, use the field extractor utility, configure extractions with the Settings pages, or set up configurations directly in the .conf files.

Summary: Fields and Field Extractions in Splunk

About Fields
Fields in Splunk are key-value pairs (e.g., user_name=fred) that serve as the foundation of searches, reports, and data models. Splunk automatically extracts fields like host, source, and sourcetype during indexing. Fields can be identified in various ways:

As key-value pairs (status=404).
Through fixed positions in delimited data (e.g., logs).

Field Extractions
The process of identifying and capturing fields from events is called field extraction. Splunk uses:

Automatic Field Extractions:
- Default fields (e.g., host, source, sourcetype) are extracted during indexing.
- Fields in key-value pair format are extracted via field discovery.
- Pattern-matching rules (regular expressions) identify non-keyed fields.
Custom Field Extractions:
- Useful for capturing fields not automatically extracted.
- Require regular expressions tied to specific sources, sourcetypes, or hosts.
- Can be created at search time or index time (rarely recommended).

Creating Custom Field Extractions

Field Extractor Utility:
- Guides users in building field extractions using regular expressions or delimiters.
- Suitable for unstructured or structured data.
- Features include highlighting fields, validating results, and specifying required text.
Settings Pages:
- Allows the creation of complex field extractions via the Field Extractions and Field Transformations pages.
- Requires knowledge of regular expressions and configuration files (props.conf and transforms.conf).
Configuration Files:
- Provide advanced control over field extractions.
- Enable delimiter-based extractions, multivalue fields, and index-time extractions.
- Use sparingly to avoid performance issues.

Special Field Types

Calculated Fields: Derived from existing fields using eval expressions.
Multivalue Fields: Contain multiple values for a single field in an event.

Field Extraction in Search Strings
Certain search commands support field extraction during searches:

rex: Extract fields using regular expressions.
extract: Extract fields based on predefined configurations.
Other Commands: spath, xmlkv, xpath, kvform, multikv.

These commands are temporary and only apply to the search in which they are used.

Best Practices

Familiarize yourself with your event data before creating extractions.
Use required text in field extractions to enhance accuracy and performance.
Prefer search-time extractions over index-time extractions for flexibility and efficiency.

SUMMARISE: Use default fields Fields are searchable name-value pairs in event data. When you search, you're matching search terms against segments of your event data; you can search more precisely by using fields. Fields are extracted from event data at either index time or search time. The fields that are extracted automatically at index time are known as default fields. Default fields serve a number of purposes. For example, the default field index identifies the index in which the event is located. The default field linecount describes the number of lines the event contains, and timestamp specifies the time at which the event occurred. Splunk software uses the values in some of the fields, particularly sourcetype, when indexing 41 the data, in order to create events properly. After the data has been indexed, you can use the default fields in your searches. For more information on using default fields in search commands, see About the search language in the Search Manual. For information on configuring default fields, see About default fields in the Getting Data In manual. Type of field Internal fields Default fields Default datetime fields List of fields _raw, _time, _indextime, _cd, _bkt host, index, linecount, punct, source, sourcetype, splunk_server, timestamp date_hour, date_mday, date_minute, date_month, date_second, date_wday, date_year, date_zone Description Contain general information about events. These are fields that contain information about where an event originated, in which index it's located, what type it is, how many lines it contains, and when it occurred. These fields are indexed and added to the Fields menu by default. These are fields that provide additional searchable granularity to event timestamps. Note: Only events that have timestamp information in them as generated by their respective systems will have date_* fields. If an event has a date_* field, it represents the value of time/date directly from the event itself. If you have specified any timezone conversions or changed the value of the time/date at indexing or input time (for example, by setting the timestamp to be the time at index or input time), these fields will not represent that. A field can have more than one value. See Manipulate and evaluate fields with multiple values. You can extract non-default fields with Splunk Web or by using extracting search commands. See About fields. You might also want to change the name of a field, or group it with other similar fields. This is easily done with tags or aliases for the fields and field values. See Tag field value pairs in Search. This topic discusses the internal and other default fields that Splunk software automatically adds when you index data. Internal fields Fields that begin with an underscore are internal fields. Do not override internal fields unless you are absolutely sure you know what you are doing. _raw The _raw field contains the original raw data of an event. The search command uses the data in _raw when performing searches and data extraction. You cannot always search directly on values of _raw, but you can filter on _raw with commands like regex or sort. Example: Return sendmail events that contain an IP address that starts with 10. eventtype=sendmail | regex _raw=*10.\d\d\d\.\d\d\d\.\d\d\d\* 42 _time The _time field contains an event's timestamp expressed in UNIX time. This field is used to create the event timeline in Splunk Web. Note: The _time field is stored internally in UTC format. It is translated to human-readable Unix time format when Splunk software renders the search results (the very last step of search time event processing). Example: Search all sources of type mail for mail addressed to the user strawsky@bigcompany.com. Then sort the search results by timestamp. sourcetype=mail to=strawsky@bigcompany.com | sort _time _indextime The _indextime field contains the time that an event was indexed, expressed in Unix time. You might use this field to focus on or filter out events that were indexed within a specific range of time. Because _indextime is a hidden field, it will not be displayed in search results unless renamed or used with an eval. _cd The _cd field provides an address for an event within the index. It is composed of two numbers, a short number and a long number. The short number indicates the specific index bucket that the event resides in. The long number is an index bucket offset. It provides the exact location of the event within its bucket. Because _cd is a hidden field, it will not be displayed in search results unless renamed or used with an eval. Because _cd is used for internal reference only, we do not recommend that you set up searches that involve it. _bkt The _bkt field contains the id of the bucket that an event is stored in. Because _bkt is a hidden field, it will not be displayed in search results unless renamed or used with an eval. Other default fields host The host field contains the originating hostname or IP address of the network device that generated the event. Use the host field to narrow searches by specifying a host value that events must match. You can use wildcards to specify multiple hosts with a single expression (Example: host=corp*). You can use host to filter results in data-generating commands, or as an argument in data-processing commands. Example 1: Search for events on all corp servers for accesses by the user strawsky. It then reports the 20 most recent events. host=corp* eventtype=access user=strawsky | head 20 Example 2: Search for events containing the term 404, and are from any host that starts with 192. 404 | regex host=*192.\d\d\d\.\d\d\d\.\d\d\d\* 43 index Theindex field contains the name of the index in which a given event is indexed. Specify an index to use in your searches by using: index="name_of_index". By default, all events are indexed in the main index. Example: Search the myweb index for events that have the .php extension. index="myweb" *.php linecount The linecount field contains the number of lines an event contains. This is the number of lines an event contains before it is indexed. Use linecount to search for events that match a certain number of lines, or as an argument in data-processing commands. To specify a matching range, use a greater-than and less-than expression (Example: linecount>10 linecount<20). Example: Search corp1 for events that contain 40 and have 40 lines, and omit events that contain 400. 40 linecount=40 host=corp1 NOT 400 punct The punct field contains a punctuation pattern that is extracted from an event. The punctuation pattern is unique to types of events. Use punct to filter events during a search or as a field argument in data-processing commands. You can use wildcards in the punct field to search for multiple punctuation patterns that share some common characters that you know you want to search for. You must use quotation marks when defining a punctuation pattern in the punct field. Example 1: Search for all punctuation patterns that start and end with : punct=":*:" Example 2: Search the php_error.log for php error events that have the punctuation pattern [--_::]__:___:____/-..-///.___". source="/var/www/log/php_error.log" punct="[--_::]__:___:____''/-..-''///.___" source The source field contains the name of the file, stream, or other input from which the event originates. Use source to filter events during a search, or as an argument in a data-processing command. You can use wildcards to specify multiple sources with a single expression (Example: source=*php.log*). You can use source to filter results in data-generating commands, or as an argument in data-processing commands. Example: Search for events from the source /var/www/log/php_error.log. source="/var/www/log/php_error.log" 44 sourcetype The sourcetype field specifies the format of the data input from which the event originates, such as access_combined or cisco_syslog. Use sourcetype to filter events during a search, or as an argument in a data-processing command. You can use wildcards to specify multiple sources with a single expression (Example: sourcetype=access*). Example: Search for all events that are of the source type access log. sourcetype=access_log splunk_server The splunk_server field contains the name of the Splunk server containing the event. Useful in a distributed Splunk environment. Example: Restrict a search to the main index on a server named remote. splunk_server=remote index=main 404 timestamp The timestamp field contains an event's timestamp value. You can configure the method that is used to extract timestamps. You can use timestamp as a search command argument to filter your search. For example, you can add timestamp=none to your search to filter your search results to include only events that have no recognizable timestamp value. Example: Return the number of events in your data that have no recognizable timestamp. timestamp=none | stats count(_raw) as count Default datetime fields You can use datetime fields to filter events during a search or as a field argument in data-processing commands. If you are located in a different timezone from the Splunk server, time-based searches use the timestamp of the event as specified on the server where the event was indexed. The datetime values are the literal values parsed from the event when it is indexed, regardless of its timezone. So, a string such as 05:22:21 will be parsed into indexed fields: date_hour::5 date_minute::22 date_second::21. date_hour The date_hour field contains the value of the hour in which an event occurred (range: 0-23). This value is extracted from the event's timestamp (the value in _time). Example: Search for events with the string apache that occurred between 10pm and 12am on the current day. apache (date_hour >= 22 AND date_hour <= 24) 45 date_mday The date_mday field contains the value of the day of the month on which an event occurred (range: 1-31). This value is extracted from the event's timestamp (the value in _time). Example: Search for events containing the string apache that occurred between the 1st and 15th day of the current month. apache (date_mday >= 1 AND date_mday <= 15) date_minute The date_minute field contains the value of the minute in which an event occurred (range: 0-59). This value is extracted from the event's timestamp (the value in _time). Example: Search for events containing the string apache that occurred between the 15th and 20th minute of the current hour. apache (date_minute >= 15 AND date_minute <= 20) date_month The date_month field contains the value of the month in which an event occurred. This value is extracted from the event's timestamp (the value in _time). Example: Search for events with the string apache that occurred in January. apache date_month=1 date_second The date_second field contains the value of the seconds portion of an event's timestamp (range: 0-59). This value is extracted from the event's timestamp (the value in _time). Example: Search for events containing the string apache that occurred between the 1st and 15th second of the current minute. apache (date_second >= 1 AND date_second <= 15) date_wday The date_wday field contains the day of the week on which an event occurred (Sunday, Monday, etc.). The date is extracted from the event's timestamp (the value in _time) and determines what day of the week that date translates to. This day of the week value is then placed in the date_wday field. Example: Search for events containing the string apache that occurred on Sunday. apache date_wday="sunday" 46 date_year The date_year field contains the value of the year in which an event occurred. This value is extracted from the event's timestamp (the value in _time). Example: Search for events containing the string apache that occurred in 2008. apache date_year=2008 date_zone The date_zone field contains the value of time for the local timezone of an event, expressed as hours in Unix Time. This value is extracted from the event's timestamp (the value in _time). Use date_zone to offset an event's timezone by specifying an offset in minutes (range: -720 to 720). Example: Search for events containing the string apache that occurred in the current timezone (local). apache date_zone=local

Summary: Default Fields in Splunk

Default fields in Splunk are searchable attributes extracted from event data to provide precise search capabilities. These fields are automatically added during indexing or at search time and can be categorized as internal fields, other default fields, and datetime fields.

1. Internal Fields

Fields starting with an underscore (_) provide foundational event data and should not be overridden unless necessary:

_raw: Contains the original event data used for filtering or extraction.
_time: The event's timestamp in Unix time, used for search timelines.
_indextime: Time when the event was indexed.
_cd: Location of the event within its index.
_bkt: ID of the event’s bucket for internal referencing.

2. Other Default Fields

These fields describe event origins and attributes:

host: Originating hostname or IP of the event.
index: Specifies the index where the event is stored.
linecount: Number of lines in the event.
punct: The event's unique punctuation pattern.
source: Input file, stream, or location of the event.
sourcetype: Format or type of event data (e.g., log types).
splunk_server: Splunk server containing the event.
timestamp: Event’s timestamp, configurable for search filtering.

3. Default Datetime Fields

These fields add granular search capabilities for event timestamps:

date_hour, date_mday, date_minute, date_month, date_second, date_wday, date_year, date_zone: Extract timestamp components like hour, day, month, year, etc.

Usage and Customization

Use default fields in search commands for precise filtering and processing.
Additional fields can be extracted or customized with Splunk Web, search commands, tags, or aliases.
Be cautious when modifying internal fields, as they are critical for event referencing and data integrity.

Default fields ensure efficient event organization and robust search functionalities, facilitating effective data analysis in Splunk.

SUMMARISE: Use the field extractor in Splunk Web Build field extractions with the field extractor Use the field extractor utility to create new fields. The field extractor provides two field extraction methods: regular expression and delimiters. The regular expression method works best with unstructured event data. You select a sample event and highlight one or more fields to extract from that event, and the field extractor generates a regular expression that matches similar events in your dataset and extracts the fields from them. The regular expression method provides several tools for testing and refining the accuracy of the regular expression. It also allows you to manually edit the regular expression. The delimiters method is designed for structured event data: data from files with headers, where all of the fields in the events are separated by a common delimiter, such as a comma or space. You select a sample event, identify the delimiter, and then rename the fields that the field extractor finds. data that resides in a file that has headers and fields separated by specific characters Overview of the field extractor To help you create a new field, the field extractor takes you through a set of steps. The field extractor workflow diverges at the Select Method step, where you select the field extraction method that you want to use. This table gives you an overview of the required steps. For detailed information about a step, click the link in the Step Title column. Step Title Select sample Select method Description Select the source type or source that is tied to the events that have the field (or fields) that you want to extract. Then choose a sample event that has that field (or fields). Select a field extraction method. You can have the field extractor generate a field-extracting regular expression, or you can employ delimiter-based field extraction. The choice you make depends on whether you are trying to extract fields from unstructured or structured event data. Field Extraction Method Both Both Regular expression 50 Step Title Select fields Rename fields Validate fields Save Access the field extractor Description Highlight one or more field values in the event to identify them as fields. The field extractor generates a regular expression that matches the event and extracts the field. Optionally, you can: Provide additional sample events to improve extraction accuracy. Identify required text to focus the field extraction on events that contain this text. Examine field extraction results. Update the underlying regular expression manually. Identify the delimiter that separates all of the fields in the event, and then rename one or more of those fields. Examine the field extraction results. Identify incorrectly extracted fields as counterexamples to improve the accuracy of the field extraction. Name your new field extraction, set its permissions, and save it. Field Extraction Method Delimiters Regular expression Both There are several ways to access the field extractor utility. The access method you use can determine which step of the field extractor workflow you start at. All users can access the field extractor after running a search that returns events. You have three post-search entry points to the field extractor: Bottom of the fields sidebar All Fields dialog box Any event in the search results You can also enter the field extractor: from the Field Extractions page in Settings. when you add data with a fixed source type. from the Splunk Web Home page (if you have Admin role privileges). Access the field extractor from the bottom of the fields sidebar When you use this method to access the field extractor it runs only against the set of events returned by the search that you have run. To get the full set of source types in your Splunk deployment, go to the Field Extractions page in Settings. 1. 2. Run a search that returns events. Scroll down to the bottom of the fields sidebar and click Extract New Fields. The field extractor starts you at the Select Sample step. 51 Access the field extractor from the All Fields dialog box When you use this method to access the field extractor you can only extract fields from the data that has been returned by your search. To get the full set of source types in your Splunk deployment, go to the Field Extractions page in Settings. 1. 2. 3. Run a search that returns events. At the top of the fields sidebar, click All Fields. In the All Fields dialog box, click Extract new fields. The field extractor starts you at the at the Select Sample step. Access the field extractor from a specific event Use this method to select an event in your search results, and create a field extraction that: Extracts one or more fields found in that event. Is tied to the source type of that event. When you use this method to access the field extractor, the field extractor runs against the set of events returned by the search that you have run. 1. 2. 3. Run a search that returns events. Find an event that you want to extract fields from, and click the arrow symbol to the left of the timestamp to open it. Click Event Actions, and select Extract Fields. The field extractor starts you at the Select Method step, in a new browser tab. You have already defined the source type and sample event. 52 Access the field extractor through the Field Extractions page in Settings This entry method is available to all users. 1. 2. Select Settings > Fields > Field extractions. Click the Open field extractor button. The field extractor starts you at the Select Sample step. Access the field extractor through the Home page This entry method is available only to users whose roles have the edit_monitor capability, such as Admin. On the Home page, click the extract fields link under the Add Data icon. The field extractor starts you at the Select Sample step. Access the field extractor after you add data This entry method is available only to users whose roles have the edit_monitor capability, such as Admin. After you add data to Splunk Enterprise, use the field extractor to extract fields from that data, as long as it has a fixed source type. For example: You add a file named vendors.csv to your Splunk deployment and give it the custom source type vendors. After you save this input, you can enter the field extractor and extract fields from the events associated with the vendors source type. Another example: You create a monitor input for the /var/log directory and select Automatic for the source type, meaning that Splunk software automatically determines the source type values of the data from that input on an event by event basis. When you save this input you do not get a prompt to extract fields from this new data input, because the events indexed from that directory can have a variety of source type values. 1. 2. 3. 4. Enter the Add Data page. See "How do you want to add data?" in the Getting Data In manual. Define a data input with a fixed source type. This can be an existing source type or a custom source type that you define. See "View and set source types for event data" in the Getting Data In manual. Save the new data input. Note: Wait 30 seconds before going to the next step. This gives the Splunk software some time to index the data and get it ready for field extraction. In the "File has been uploaded successfully" dialog box, click Extract Fields. The field extractor starts you at the Select Sample step. Field Extractor: Select Sample step In the Select Sample field extractor step, you do two things: First you identify a data type for your field extraction. Your data type selection brings up a list of events that have the selected source or source type value. Then you select an event from the list that has the field or fields that you want to extract. 53 The field extractor bypasses the Select Sample step when you enter the field extractor from a specific event in your search results. When you do this, the field extractor starts you off at the Select Method step. Select a data type and a sample event Note: The field extractor bypasses the first step of this procedure (select a data type) if you choose your source type before you enter the field extractor. The Splunk field extractor is limited to twenty lines on a sample event. This happens when you enter the field extractor: After you run a search where a specific source type is identified in the search string and then click the Extract New Fields link in the fields sidebar or the All Fields dialog box. After you run a search that returns a set of events that all have the same source type, and then click the Extract New Fields link in the fields sidebar or All Fields dialog box. After you add a data input with a fixed source type. Steps 1. Select a Data Type for your field extraction. Each field extraction is associated with a specific source type or source value. If you have entered the field extractor after running a search, the sets of sources and source types that you can choose from are limited to those discovered in the results returned by that search. To see all of the source and source type sets in your Splunk deployment, go to the Field Extractions page in Settings. If you select sourcetype the Source Type list appears. Choose a source type there. If you do not see the source type that you would like to use, try specifying the source type that you want to use in that search and rerunning it. If you select source the Source Name field appears. Enter a source value there. This screenshot is an example of the source type listing you see when you enter the field extractor from the Field Extractions page in Settings

Summary: Field Extraction in Splunk Web

The Field Extractor in Splunk Web allows users to create new fields from event data using two primary methods:

Regular Expression Method: Best for unstructured data.
- Users highlight fields in a sample event to generate a regex for extracting fields.
- Tools are provided for testing, refining, and manually editing regex.
Delimiter Method: Best for structured data (e.g., CSV files with headers).
- Users identify a delimiter (e.g., comma, space) to extract and rename fields.

Steps in the Field Extractor Workflow

Select Sample: Choose the source type or a sample event containing desired fields.
Select Method: Choose between regular expressions or delimiters for field extraction.
Extract Fields: Highlight field values (regex) or identify delimiters (structured data).
Rename Fields: Assign meaningful names to extracted fields.
Validate Fields: Test and improve extraction accuracy by identifying counterexamples.
Save Fields: Name the extraction, set permissions, and save.

Accessing the Field Extractor

The utility can be accessed through various methods:

Post-Search Methods:
- From the fields sidebar or the "All Fields" dialog box after running a search.
- Directly from specific events in search results using the "Event Actions" menu.
Settings Menu:
- Navigate to Settings > Fields > Field Extractions and click "Open field extractor."
Home Page:
- Available for users with admin privileges under the "Add Data" icon.
Add Data Process:
- After adding a new dataset with a fixed source type, users can initiate field extraction.

Key Notes

Field extraction is tied to source types or specific sources.
The field extractor can only display 20 lines per sample event.
Steps may vary depending on how the extractor is accessed (e.g., bypassing the "Select Sample" step for specific events).

This streamlined process ensures accurate field extraction, enhancing the usability of Splunk for analyzing event data.

SUMMARISE: Search macros Use search macros in searches Search macros are reusable chunks of Search Processing Language (SPL) that you can insert into other searches. Search macros can be any part of a search, such as an eval statement or search term and do not need to be a complete command. You can also specify whether the macro field takes any arguments. Insert search macros into search strings When you put a search macro in a search string, place a back tick character (

 ) before and after the macro name. On

most English-language keyboards, this character is located on the same key as the tilde (~). You can reference a search macro within other search macros using this same syntax. For example, if you have a search macro named mymacro it looks like the following when referenced in a search: sourcetype=access_* | mymacro Macros inside of quoted values are not expanded. In the following example, the search macro users is not expanded. "audituserslocal" Don't include macros with hyphens in your searches; the Search app doesn't support hyphens in macro names. For example, use macro_name instead of macro-name in your searches. Preview search macros in search strings Check the contents of your search macro from the Search bar in the Search page using the following keyboard shortcut: Command-Shift-E (Mac OSX) Control-Shift-E (Linux or Windows) The shortcut opens a preview that displays the expanded search string, including all nested search macros and saved searches. If syntax highlighting or line numbering are enabled, those features also appear in the preview. You can copy parts of the expanded search string. You can also click Open in Search to run the expanded search string in a new window. See Preview your search. Search macros that contain generating commands When you use a search macro in a search string, consider whether the macro expands to an SPL string that begins with a Generating command like from, search, metadata, inputlookup, pivot, and tstats. If it does, you need to put a pipe character before the search macro. For example, if you know the search macro mygeneratingmacro starts with the tstats command, you would insert it into your search string as follows: | mygeneratingmacro See Define search macros in Settings. 198 When search macros take arguments If your search macro takes arguments, define those arguments when you insert the macro into the search string. For example, if the search macro argmacro(2) includes two arguments that are integers, you might have inserted the macro into your search string as follows: argmacro(120,300). If your search macro argument includes quotes, escape the quotes when you call the macro in your search. For example, if you pass a quoted string as the argument for your macro, you use: mymacro("He said \"hello!\""). Your search macro definition can include the following: A validation expression that determines whether the arguments you enter are valid. A validation error message that appears when you provide invalid arguments. Additional resources For more information, see the following resources. Define search macros in Settings Search macro examples Generating commands, in the Search Reference. Define search macros in Settings Search macros are reusable chunks of Search Processing Language (SPL) that you can insert into other searches. Search macros can be any part of a search, such as an eval statement or search term, and do not need to be a complete command. You can also specify whether the macro field takes any arguments. Prerequisites See Insert search macros into search strings. See Design a search macro definition. (Optional) If your search macros require the search writer to provide argument variables, you can design validation expressions that tell the search writer when invalid arguments have been submitted. See Validate search macro arguments. Steps 1. 2. 3. 4. 5. 6. Select Settings > Advanced Search > Search macros. Click New to create a search macro. (Optional) Check the Destination app and verify that it is set to the app that you want to restrict your search macro to. Select a different app from the Destination app list if you want to restrict your search macro to a different app. Enter a unique Name for the search macro. If your search macro includes an argument, append the number of arguments to the name. For example, if your search macro mymacro includes two arguments, name it mymacro(2). In Definition, enter the search string that the macro expands to when you reference it in another search. (Optional) Click Use eval-based definition? to indicate that the Definition value is an eval expression that returns a string that the search macro expands to. 199 7. (Optional) Enter any Arguments for your search macro. This is a comma-delimited string of argument names. Argument names may only contain alphanumeric characters (a-Z, A-Z, 0-9), underscores, and dashes. The string cannot contain repetitions of argument names. 8. 9. 10. (Optional) Enter a Validation expression that verifies whether the argument values used to invoke the search macro are acceptable. The validation expression is an eval expression that evaluates to a Boolean or string value. (Optional) Enter a Validation error message if you defined a validation expression. This message appears when the argument values that invoke the search macro fail the validation expression. Click Save to save your search macro. Design a search macro definition The fundamental part of a search macro is its definition, which is the SPL chunk that the macro expands to when you reference it in another search. If your search macro definition has variables, the macro user must input the variables into the definition as tokens with dollar signs on either side of them. For example, $arg1$ might be the first argument in a search macro definition. The SPL in a search macro definition must comply with the syntax requirements of the search command that uses it. For example, eval command syntax requires that any literal string in the expression is surrounded by double quotation marks. When using a search macro with the eval command, a literal string in the search macro definition must be surrounded by double quotation marks. Pipe characters and generating commands in macro definitions When you use generating commands such as search, inputlookup, rest, or tstats in searches, put them at the start of the search, with a leading pipe character. If you want your search macro to use a generating command, remove the leading pipe character from the macro definition. Place it at the start of the search string that you are inserting the search macro into, in front of the search macro reference. For example, you have a search macro named mygeneratingmacro that has the following definition: tstats latest(_time) as latest where index!=filemon by index host source sourcetype The definition of mygeneratingmacro begins with the generating command tstats. Instead of preceding tstats with a pipe character in the macro definition, you put the pipe character in the search string, before the search macro reference. For example: | mygeneratingmacro Validate search macro arguments When you define a search macro that includes arguments that the user must enter, you can define a Validation expression that determines whether the arguments supplied by the user are valid. You can define a Validation error message that appears when search macro arguments fail validation. The validation expression must be an eval expression that evaluates to a Boolean or a string. If the validation expression is boolean, validation succeeds when the validation expression returns true. If it returns false, or returns null, validation fails. 200 If the validation expression is not Boolean, validation succeeds when the validation expression returns null. If it returns a string, validation fails

Summary: Search Macros in SPL

What are Search Macros?

Reusable chunks of Search Processing Language (SPL) used in other searches.
Can represent any part of a search (e.g., eval statement or term) without being a complete command.
Can include arguments for customization.

Inserting Search Macros into Searches

Use backticks () around the macro name (e.g., mymacro`).
Avoid macros with hyphens in names (use macro_name instead of macro-name).
To include generating commands (e.g., tstats), precede the macro reference with a pipe (|).

Previewing Macros

Use shortcuts:
- Mac: Command-Shift-E
- Windows/Linux: Control-Shift-E
Opens a preview showing the expanded search string with syntax highlighting.

Macros with Arguments

Define arguments when inserting macros (e.g., argmacro(120,300)).
Escape quotes within arguments (e.g., mymacro("He said \"hello!\"")).
Define validation expressions to check argument validity and provide error messages.

Defining Search Macros

Go to Settings > Advanced Search > Search Macros.
Click New and set the Destination app.
Enter a unique macro Name. If it takes arguments, append the count (e.g., mymacro(2)).
Define the SPL Definition the macro expands to.
Optionally:
- Specify Arguments (alphanumeric, underscores, and dashes only).
- Add a Validation expression and Validation error message.
Save the macro.

Design Guidelines

Use $arg$ syntax for variables in the macro definition.
For generating commands, remove the pipe (|) in the macro definition and include it in the search string.
- Example: | mygeneratingmacro`` expands to a tstats-based command.
Ensure SPL syntax complies with the search command requirements (e.g., use double quotes for literal strings in eval).

Validation of Arguments

Validation expressions evaluate to:
- Boolean: Validation passes if true; fails if false or null.
- String: Validation fails if a string is returned; passes if null.

Additional Resources

Guides on defining search macros, examples, and generating commands in the SPL Search Reference.

SUMMARISE: Manage and explore datasets Dataset types and usage A dataset is a collection of data. Some datasets you define and maintain for a specific business purpose. Other datasets are generated from ingesting or uploading data. Datasets are represented as a table, with fields for columns and field values for cells. You can view and manage datasets with the Datasets listing page. Dataset types You can work with three dataset types. Two of these dataset types, lookups and data models, are existing knowledge objects that have been part of the Splunk platform for a long time. Table datasets, or tables, are a new dataset type that you can create and maintain in Splunk Cloud Platform and Splunk Enterprise. Use the Datasets listing page to view and manage your datasets. See View and manage datasets. Lookups The Datasets listing page displays two categories of lookup datasets: lookup table files and lookup definitions. It lists lookup table files for .csv lookups and lookup definitions for .csv lookups and KV store lookups. Other types of lookups, such as external lookups and geospatial lookups, are not listed as datasets. You upload lookup table files and create file-based lookup definitions through the Lookups pages in Settings. See About lookups. Data model datasets Data models are made up of one or more data model datasets. When a data model is composed of multiple datasets, those datasets can be arranged hierarchically, with a root dataset at the top and child datasets beneath it. In data model dataset hierarchies, child datasets inherit fields from their parent dataset but can also have additional fields of their own. You create and edit data model dataset definitions with the Data Model Editor. See About data models. Note: In previous versions of the Splunk platform, data model datasets were called data model objects. Table datasets Table datasets, or tables, are focused, curated collections of event data that you design for a specific business purpose. You can derive their initial data from a simple search, a combination of indexes and source types, or an existing dataset of any type. For example, you could create a new table dataset whose initial data comes from a specific data model dataset. After this new dataset is created, you can modify it by updating field names, adding fields, and more. You define and maintain datasets with Table Views, which translates sophisticated search commands into simple UI editor interactions. It is easy to use, even if you have minimal knowledge of Splunk search processing language (SPL). 204 Manage datasets The Datasets listing page shows all of the datasets that you have access to in your Splunk implementation. You can see what types of datasets you have, who owns them, and how they are shared. For information about the table dataset features of the Datasets listing page, see Manage table datasets. Open the Datasets listing page In the Search app, click Datasets in the green Apps bar. View dataset detail information You can expand a dataset row to see details about that dataset, such as the fields contained in the dataset, or the date that the dataset was last modified. When you view the detail information of a table dataset, you can also see the datasets that that table dataset is extended from, if applicable. 1. 2. 3. In the Search & Reporting app, click Datasets to open the Datasets listing page. Find a dataset that you want to review. Click the > symbol in the first column to expand the row of the dataset details. Explore a dataset Use the Explorer view to inspect a dataset and determine whether it contains information you want. The Explorer view provides tools for the exploration and management of individual datasets. Explore datasets with the View Results and Summarize Fields views. Use a time range picker to see what datasets contain for specific time ranges. Manage dataset search jobs. Export dataset contents. Save datasets as scheduled reports. Perform the same dataset management actions that exist on the Datasets listing page. For information on using the Explorer view, see Explore a dataset. Visualize a dataset with Pivot Use Pivot to create a visualization based on your dataset. You can save the visualization as a report or as a dashboard panel. You do not need to know how to use the Splunk Search Processing Language (SPL) to use Pivot. You can open all dataset types in Pivot. Prerequisites Introduction to Pivot in the Pivot Manual. Steps 1. 2. In the Search & Reporting app, click Datasets. Find a dataset that you want to work with in Pivot. 205 3. Select Explore > Visualize with Pivot. You can also access Pivot from the Explorer view. See Explore a dataset. Investigate a dataset in Search You can create a search string that uses the from command to reference the dataset, and optionally add SPL to the search string. You can save the search as a report, alert, or dashboard panel. The saved report, alert, or dashboard panel is extended from the original dataset through a from command reference. An extended child dataset is distinct from, but dependent on, the parent dataset from which it is extended. If you change a parent dataset, that change propagates down to all child datasets that are extended from that parent dataset. Prerequisites Get started with Search in the Search Manual Extend datasets Steps 1. 2. 3. 4. 5. 6. In the Search & Reporting app, click Datasets to open the Datasets listing page. Locate a dataset that you want to explore in Search. Select Explore > Investigate in Search. The search returns results in event list format by default. Switch the results format from List to Table to see the table view of the dataset. (Optional) Update the search string with additional SPL. Do not remove the from reference. (Optional) Click Save as to save your search, and select either Report, Dashboard Panel, or Alert. (Optional) Click New Table to create a new table dataset based on the search string. Edit datasets From the Datasets listing page you can access editing options for the different dataset types. Dataset Type Data Model Lookup Table Lookup Definition Select Manage > Edit Data Model Opens the Data Model Editor. Manage > Edit Lookup Table Files Result Opens the Lookup table files listing page in Settings. Manage > Edit Lookup Definition Manage dataset permissions Opens the detail page for the lookup definition from the Lookup definitions listing page in Settings. More info See Design data models. See About lookups. See About lookups. Change dataset permissions to widen or restrict their availability to other users. You can set up read and write access by role, and you can make datasets globally accessible, restricted to a particular app context, or private to a single user. By default, only the Power and Admin roles can set permissions for datasets. 206 Lookup table files and lookup definitions 1. 2. On the Datasets listing page, identify a lookup table file or lookup definition that requires permission edits. Select Manage > Edit Permissions. For information about setting permissions for these dataset types, see Manage knowledge object permissions. Lookup table files and lookup definitions are interdependent. Every CSV lookup definition includes a reference to a CSV lookup table file, and any CSV lookup table file can potentially be associated with multiple CSV lookup definitions. This means that each lookup table file must have permissions that are wider in scope or equal to the permissions of the lookup definitions that refer to it. For example, if your lookup table file is referenced by a lookup definition that is shared only to users of the Search app, that lookup table file must also be shared with users of the Search app, or it must be shared globally to all users. If the lookup table file is private, the lookup definition cannot connect to it, and the lookup will not work.

Summary: Managing and Exploring Datasets in Splunk

Dataset Types

Lookups:
- Includes lookup table files (e.g., .csv files) and lookup definitions.
- Created and managed via Lookups settings.
Data Models:
- Comprise hierarchical datasets with root and child datasets.
- Managed using the Data Model Editor.
Table Datasets (Tables):
- Curated collections of event data tailored for specific purposes.
- Can be created from searches, indexes, source types, or other datasets.
- Editable using Table Views with a user-friendly UI.

Dataset Management

Datasets Listing Page:
- Central hub to view, explore, and manage all datasets.
- Provides details like dataset fields, last modified dates, and hierarchical extensions.
Dataset Exploration:
- Use the Explorer View for detailed inspection.
- Features include time-range analysis, exporting contents, and scheduling reports.
Visualization with Pivot:
- Create visualizations without requiring SPL knowledge.
- Visualizations can be saved as reports or dashboard panels.
Investigate in Search:
- Reference datasets using the from command in search strings.
- Save searches as reports, alerts, or dashboards while maintaining a parent-child relationship with the dataset.

Editing and Permissions

Edit Options by Dataset Type:
- Data Models: Use the Data Model Editor.
- Lookup Tables: Edit through the Lookup Table Files settings.
- Lookup Definitions: Access the Lookup Definitions settings.
Permissions Management:
- Restrict or expand access based on user roles.
- Ensure lookup table files have permissions equal to or wider than their associated lookup definitions.

Key Features

Time-Range Picker: Analyze datasets for specific time periods.
Save Dataset Actions: Schedule reports or alerts directly from dataset exploration.
Interdependencies: Lookup table files and definitions must have compatible permissions to function correctly.

By utilizing these tools and techniques, you can effectively manage, explore, and derive insights from your datasets in Splunk.

SUMMARISE: Explore a dataset The Explorer view shows the contents of any dataset on the Datasets listing page. You can inspect the contents of any dataset listed on the page, including data model datasets and lookups. The Explorer view provides several dataset exploration and management capabilities: Use two views for dataset exploration: View Results, which renders the dataset in a standard table format. Summarize Fields, which displays statistical information for each of the fields in your table and their values. 208 Set the dataset time range. Manage the dataset search job. Export the contents of the dataset for a given time range. Extend your dataset as a scheduled report. You can perform the same dataset management actions that you have access to through the Datasets listings page. See Manage datasets and Manage table datasets. Open the Explorer view for a dataset Use the Datasets listing page to access the Explorer view for a selected dataset. 1. 2. 3. In the Search & Reporting app, click Datasets to open the Datasets listing page. Find a dataset you want to explore. Click the dataset name to open it in the Explorer view. Ways to view datasets The Explorer view gives you two ways to view your dataset. You can View Results or you can Summarize Fields. View Results View Results is the default Explorer view. It displays your table dataset as a table, with fields as columns, values in cells, and sample events in rows. It displays the results of a search over the time range set by the time range picker. Summarize Fields Click Summarize Fields to see analytical details about the fields in the table. You can see top value distributions, null value percentages, numeric value statistics, and more. These statistics are returned by a search job that runs over the range defined by the time range picker. It is separate from the search job that populates the View Results display. Set the dataset time range The time range picker lets you restrict the data displayed by the view to events that fall within specific ranges of time. It applies to search-based dataset types like data model datasets and table datasets. The time range picker used in the Explorer view does not include options for real-time searches. Lookup table files and lookup definitions get their data from static CSV files and KV store collections, so the time range picker does not apply to them. They display the same rows of data no matter what time range you select. The time range picker is set to Last 24 hours by default. If your dataset has no results from the last 24 hours, this view is empty at first. You can adjust the time range picker to a range where events are present. The time range picker gives you several time range definition options. You can choose a pre-set time range, or you can define a custom time range. For help with the time range picker, see Select time ranges to apply to your search in the Search Manual. 209 Manage the dataset search job When you enter the Explorer view, a search job runs within the time range set by the time range picker. The search results populate the View Results display. After you launch a dataset search, a set of controls at the top right of the dataset view lets you manage the search job without leaving the Explorer view. In the middle of this control set are pause/start and stop icons that you can use while the dataset search is in progress. The Explorer job controls only manage the search job that produces the results displayed in View Results. They do not affect the job that runs when you open the Summarize Fields display of the dataset. Use the Job menu actions The Job menu lets you access the View Results search job, and information about it. You can use it when a search job is running, paused, or finalized. 1. 2. Click Job. Choose from the list options. Action Edit Job Settings... Description Opens the Job Settings dialog box, where you can change the read permissions for the job, extend the job lifespan, and get a URL for the job. You can use the URL to share the job with others or to add a bookmark to the job in your Web browser. Inspect Job Send Job to Background Runs the job on the background. Use this option if the search job is slow to complete. This enables you to work on other activities, including running a new search job. Opens the Search Job Inspector window and displays information and metrics about the search job. You can select this action while the search is running or after the search completes. For more information, see View search job properties in the Search Manual. For more information, see About jobs and job management in the Search Manual. Share a job Click the Share icon to share the View Results search job. When you select this, the lifetime of the job is extended to 7 days and its read permissions are set to Everyone. For more information about jobs, see About jobs and job management in the Search Manual. Export the job results Click the Export icon to export the results of the View Results search job. You can select to output to CSV, XML, or JSON and specify the number of results to export. For information about other export methods, see Export search results in the Search Manual. 210 Extend the dataset as a scheduled report You can extend your dataset to a new scheduled report. The report uses a from command in its base search to reference the dataset that you are viewing. Changes you make to the dataset are passed down to the report. Changes you make to the report are not passed up to the dataset. Select Edit > Schedule Report to extend the dataset as a scheduled report. This opens the Schedule Report dialog box, where you can create the report schedule and define actions that are triggered each time the report runs. For example, you can arrange to have the Splunk software add the report results to a specific CSV file each time the report runs. You can also define scheduled report actions that email the results to a set of people, or that run scripts. For more information about using this dialog box to create the report schedule and define actions for it, see Schedule reports, in the Reporting Manual. For more information about dataset extension, see Extend datasets. Manage your dataset The Explorer view gives you the same dataset management capabilities as the Dataset listing page. If you review the contents of a dataset and decide you want to work with it, you do not need to return to the Dataset listing page. You can apply management actions to it from this view. The Explorer view includes management actions for all dataset types: Visualize a dataset with Pivot Investigate a dataset with Search Edit a dataset Update dataset permissions Delete a dataset For more information about these tasks, see Manage datasets. The Explorer view includes additional table dataset management capabilities: Extend a dataset as a new table dataset Clone a table dataset Edit table dataset descriptions Accelerate table datasets

Summary: Explore a Dataset

The Explorer View in Splunk provides a detailed interface for inspecting and managing datasets listed on the Datasets page. Key functionalities include:

Exploration and Viewing Options:

View Results: Displays datasets as tables, with fields as columns and sample events as rows. Default time range is the last 24 hours.
Summarize Fields: Provides statistical summaries of field values, including distributions, null percentages, and numeric statistics.
Time Range Picker: Filters data based on predefined or custom time ranges. Static datasets like CSV-based lookups ignore this setting.

Dataset Management:

Search Job Management:
- Pause, stop, or manage search jobs via controls.
- Inspect or send jobs to the background for efficiency.
- Share jobs (extending lifetime to 7 days) or export results in CSV, XML, or JSON formats.
Dataset Extension:
- Create scheduled reports using the dataset as a base. Changes to the dataset propagate to reports but not vice versa.
- Define report schedules and trigger actions like emailing results or saving to a CSV file.
Additional Management Actions:
- Visualize with Pivot for easy reporting.
- Investigate datasets via Search for custom SPL queries.
- Edit or clone datasets, update permissions, or delete datasets.
- Accelerate table datasets for faster performance.

The Explorer View mirrors most capabilities of the Dataset Listing Page, allowing comprehensive dataset management without navigating away.

SUMMARISE: Create and edit table datasets Get started with table datasets Table datasets are a type of dataset that you can create, shape, and curate for a specific purpose. You begin by defining the initial data for the table, such as an index, source type, search string, or existing dataset. Then you edit and refine that table until it fits the precise shape that you and your users require for later analysis and reporting work. After you create your table, you can modify it over time and share it with others so they can refine it further. You can also use techniques like dataset cloning and dataset extension to create datasets that are based on datasets you already created. You can manage table datasets alongside other dataset types that are available to all users of Splunk Enterprise and Splunk Cloud Platform, like data model datasets and lookups. All of these dataset types appear in the Datasets listing page. What you can do with datasets This table explains what Splunk platform users can do with datasets. Dataset activity Why this is useful See also View dataset contents Check a dataset to determine whether it contains fields and values that you want to work with. For example, you can view lookup table files directly instead of searching their contents in the Search view. Manage table datasets Open datasets in Pivot Use Pivot to create visualizations and dashboard panels based on your dataset. Pivot can also help you discover data trends and field correlations within a dataset. Introduction to Pivot Extend datasets Extend your dataset as a search, modify its search string as necessary, and save the search as a report, alert, or dashboard panel. You can also create table datasets that are based on lookups and data model datasets and then modify those tables to fit your specific use cases. Dataset extension Use Table Views to create table datasets You can design sophisticated and tightly-focused collections of event data, even with minimal SPL skills. Define initial data for a new table dataset Share table datasets After you create a table dataset you can give other users read or write access to it so they can curate and refine it. You can also extend your dataset and let other people refine the extension without affecting the original dataset. Manage table datasets View field analytics View field analytics in Table Views Summary mode to find more detailed information about your dataset. Define initial data for a new table dataset Clone table datasets You can make exact copies of table datasets and save the copy with a new name. Only table datasets can be cloned. Manage datasets Accelerate table datasets You can accelerate table datasets to return results faster in a manner similar to report and data model acceleration. This can be helpful if you are using a large dataset as the basis for a pivot report or dashboard panel. Accelerate table datasets Manage table datasets You can create and manage table datasets using the Datasets listing page for the Splunk platform. 212 For information about default features of the Datasets listing page, such as accessing Explorer views of datasets, investigating datasets in Search, and visualizing datasets with Pivot, see Manage datasets. Create table datasets To create a table dataset, click Datasets in the green Apps bar, and then click Create Table View. This takes you to the Table Views workflow. After you define initial data, you can also edit the dataset with Table Views. To learn more about creating a new table dataset, see Define initial data for a new table dataset and View and update a table dataset. When you create table datasets, always give them unique names. If you have more than one table dataset with the same name in your system you risk experiencing object name collision issues that are difficult to resolve. For example, say you have two table datasets named Store Sales, and you share one at the global level, but leave the other one private. If you then extend the global Store Sales dataset, the dataset that is created through that extension will display the table from the private Store Sales dataset instead. Delete a table dataset You can delete any table dataset that you have write permissions for. Before deleting a table dataset, verify that it is not extended to one or more child table datasets. Deleting a parent dataset breaks tables and other objects that are extended from it. For example, if table Alpha is extended to table Beta, and table Beta is in turn used to create a Pivot visualization that is used in a dashboard panel, that dashboard panel ceases to function if you delete table Alpha. 1. 2. 3. On the Datasets listing page, find the table dataset that you want to delete. Select Edit > Delete. Click Delete again to confirm. Extend a dataset as a new table dataset You can extend any dataset as a new table dataset. The extended dataset is a new dataset that is bound to the original dataset through its reference to that dataset. If the definition of the parent dataset changes, those changes are passed down to any child datasets that are extended from it. To see from what datasets a table dataset is extended, expand its row in the Datasets listing page by clicking its > symbol in the first column. The parent datasets for that dataset are listed in an Extends line item. If a table dataset does not have an Extends line item, it is not extended from another dataset. Prerequisites To understand why you might want to extend a dataset and what you can with an extended dataset, see Dataset extension. To access your extended dataset, see View and update a table dataset. 213 Steps 1. 2. 3. 4. 5. 6. On the Datasets listing page, find a dataset that you want to extend. For that dataset, select Edit > Extend in Table. (Optional) Use Table Views to modify the new table. Click Save to open the Save As New Table dialog. Enter a Table Title. Click Save to save the table. Clone a table dataset Clone a table dataset to make a new table dataset that is a copy of an existing dataset. Cloning differs from dataset extension in that you can make changes to the original dataset without affecting datasets that are cloned from that dataset. Table datasets are the only dataset type that can be cloned through the Datasets listing page. You can clone lookup definition datasets through the Lookup Definitions page in Settings. 1. 2. 3. 4. 5. 6. 7. On the Datasets listing page, find a table dataset that you want to copy. Select Edit > Clone. Enter a Table Title. (Optional) Enter a Description. Click Clone Dataset. (Optional) Click Edit to edit your cloned dataset. (Optional) Click Pivot to open the cloned dataset in Pivot and create a visualization based on it. Edit a table dataset Use the Datasets listing page to edit selected table datasets. You can edit a table description, or you can edit the table with Table Views. Add or edit a table dataset title or description Table dataset descriptions are visible in two places: The Dataset listing page, when you expand the table dataset row. The Explorer view of the table dataset, under the dataset name. To add or edit a table dataset title or description, follow these steps: 1. 2. 3. 4. On the Datasets listing page, find a table dataset whose description you would like to add or edit. Expand the dataset row by clicking the > symbol in the first column to see its current description. Select Edit > Edit title or description. Add or update the title or description. Click Save. Edit a table dataset in Table Views Use the Datasets listing page to open and edit a dataset in Table Views. 214 1. 2. 3. 4. On the Datasets listing page, find a table dataset that you want to edit. Select Edit > Edit table. Edit the table in Table Views. Click Save. For more information about using Table Views, see View and update a table dataset. Update table dataset permissions New table datasets are private by default and are available only to the users who created them. If you want other users to be able to view or edit a private table dataset you can change its permissions. By default, only the Power and Admin roles can set permissions for table datasets. On the Datasets listing page, select Edit > Edit Permissions for the table whose permissions you want to edit. You can see whether the table is shared with users of a specific app or globally with users of all apps. You can also see which roles have read or write access to the app. For information about setting permissions, see Manage knowledge object permissions. Share private table datasets that you do not own If you want to share a private table dataset that you do not own, you can change its permissions through the Data Models management page in Settings. You cannot see private datasets that you do not own in the Datasets listing page. 1. 2. 3. 4. Select Settings > Data models. Locate the table dataset that you want to share and select Edit > Edit Permissions. Share the dataset at the App or All apps level, and set read/write permissions as necessary. Click Save. When you return to the Datasets listing page, verify that the dataset is visible and has the new permissions that you set for it. Manage table dataset acceleration Table datasets that contain a large amount of data can be accelerated so that their underlying search completes faster when you view the dataset or visualizations that are backed by it. Table acceleration only applies to table datasets when the tstats or pivot commands are applied to the table datasets. On the Datasets listing page, accelerated table datasets and data model datasets have a page, select Edit > Edit Acceleration for the table you want to accelerate. icon. On the Datasets listing For more information about enabling and managing table acceleration, including caveats and restrictions related to table acceleration, see Accelerate tables. Define initial data for a new table dataset When you create a new table dataset with Table Views, you start by defining initial data. You have three options for initial data. 215 An index and source type combination You can populate your new dataset with events associated with a combination of indexes and source types. An existing dataset You can populate your dataset using a dataset that already exists. The dataset can be a table dataset, a data model dataset, a CSV lookup table, or a CSV lookup definition. A search You can base your dataset on the results of any search string, as long as it doesn't include transforming commands. Prerequisites To access the data required to create table datasets, you must have a role with the get_metadata capability. If you use Splunk Analytics for Hadoop and want to create a dataset based on data from a virtual index, you must get your initial data either from a search that references the virtual index or from an existing dataset that already has the virtual index data.

Summary: Create and Edit Table Datasets in Splunk

Introduction to Table Datasets

Table datasets allow users to define, curate, and shape data for analysis and reporting.
Initial data can come from indexes, source types, search strings, or existing datasets.
Datasets can be refined over time, shared, cloned, or extended for specific use cases.

Key Activities with Datasets

Activity	Purpose
View dataset contents	Verify fields and values without performing a search.
Open datasets in Pivot	Create visualizations and identify trends.
Extend datasets	Modify datasets as needed and save as reports, alerts, or dashboards.
Clone datasets	Create exact copies without affecting the original dataset.
Accelerate datasets	Speed up dataset processing for large data sets used in reports or dashboards.
Share datasets	Allow other users to read or refine datasets.
View field analytics	Analyze field details within Table Views.

Creating and Managing Datasets

Creating Table Datasets
- Access via the Datasets listing page and click "Create Table View."
- Define initial data (index and source type, search string, or existing dataset).
- Use unique names to avoid name collisions.
Editing Datasets
- Edit titles, descriptions, or table content directly in Table Views.
- Save changes to update the dataset for future use.
Deleting Datasets
- Verify that the dataset is not extended to child datasets to avoid breaking dependencies.
- Use "Edit > Delete" on the Datasets listing page.
Extending Datasets
- Extend existing datasets to create new ones linked to the original.
- Changes in the parent dataset reflect in the child datasets.
Cloning Datasets
- Clone datasets to create independent copies.
- Changes in the original do not affect the cloned dataset.

Dataset Permissions

Datasets are private by default.
Permissions can be updated to allow app-specific or global access.
Power and Admin roles can edit permissions.

Acceleration of Datasets

Table acceleration enables faster processing for large datasets in reports and dashboards.
Enable acceleration via "Edit > Edit Acceleration" in the Datasets listing page.

Best Practices

Use descriptive and unique names for datasets.
Avoid unnecessary deletions of parent datasets to prevent downstream issues.
Regularly review permissions and access levels to maintain security.

By leveraging these features, Splunk users can efficiently manage data for analysis, reporting, and visualization.

SUMMARISE: Identify an index and source type combination for initial data 1. 2. 3. 4. 5. 6. 7. 8. 9. In the Search & Reporting app, open the Datasets listing page. Click Create Table View to go to the initial data setup screen. Choose an index that you want to use for initial data. If you do not want to select a specific index, select All indexes. Select a source type that you want to use for initial data. If you do not want to select a specific source type, select All source types. If you select both '''All indexes''' and '''All source types''', you risk creating an overly broad dataset that contains all of the events indexed by your Splunk platform implementation, with the exception of events in _internal and other internal indexes, which you must specify by name. In general, avoid creating overly broad datasets. The datasets feature is designed for creating narrow views of data. Click Next. A preview of your dataset appears. Rows are events, columns are fields, and cells are field values. Select existing fields that you want to see in your dataset. (Optional) If you are not seeing a field choice that you are expecting, add the missing field by following these steps: 1. 2. 3. At the top of the field list, click Add a missing existing field. Enter the field and click Add. Select the added field. Use the dataset preview pane to verify that this is the initial data that you want. If you do not find the existing fields or field values that you were expecting, you can remove this selection and select another one. When you are satisfied that your index, source type, and field selections provide the correct initial data for your dataset, click Start Editing to confirm your index, source type, and field selections. Use an existing dataset for initial data The Datasets tab lets you select an existing dataset for your initial data. You can select any dataset that you can otherwise see on the Datasets listing page, including data model datasets, lookup tables, and lookup definitions. When you create a dataset that uses an existing dataset for initial data, you can choose between cloning and extending the existing dataset. 1. 2. In the Search & Reporting app, open the Datasets listing page. For the dataset that you want to clone or extend, select either Edit > Clone or Edit > Extend in Table. 216 Selection Clone Extend 3. Description Creates an identical copy of the original dataset. Only table datasets can be cloned. Creates a dataset that is extended from an existing dataset. Changes made to the original dataset propagate down to the extended dataset. All dataset types can be extended. If you are working with a lookup table file, select the fields that you want to use in your table. The fields you select are the only fields that will make up your dataset, along with _raw and _time, which are required. You can hover over a field to see field statistics, such as the percentage of events in the dataset that have the field and the top values for the field. Table datasets, data model datasets, and lookup definitions have fixed fields. When you create a new dataset by cloning or extending a dataset with fixed fields, you can't choose which of those fields you want to start with in your dataset. 4. 5. 6. 1. (Optional) If you don't see a field choice that you are expecting, add the missing field by following these steps: At the top of the field list, click Add a missing existing field. 2. 3. Enter the field and click Add. Select the added field. Use the dataset preview pane to verify that this is the initial data that you want. If you do not find the existing fields or field values that you were expecting, you can remove this selection and select another one. When you are satisfied that your index, source type, and field selections provide the correct initial data for your dataset, click Start Editing to confirm your index, source type, and field selections. Provide a search string for initial data There are two methods that you can follow to derive the search string for initial data. Once you provide the search string, the other initial data setup steps are the same. The search string you provide must identify the fields that its search commands operate on. For example, a search that only includes commands like sendemail, highlight, or delete is invalid because those commands do not require that you identify the fields that they operate upon. Use a search string that you created in the Search view 1. 2. 3. 4. 5. 6. In the Search view, create a search that returns events that you want in your table. Click Create Table View to use the search as the initial data for a new table dataset. Table Views opens with the search string you designed in the search field. (Optional) Add more Splunk SPL commands until you have a search that returns results that you want to use in a dataset. Click Save to open the Save As New Table box. Enter a Table Title. Click Save to save the table. Start with a search string that extends an existing dataset This method creates a dataset that is extended from an existing dataset. Changes made to the original dataset propagate down to the extended dataset. All dataset types can be extended. 1. 2. In the Search & Reporting app, open the Datasets listing page. Click Edit > Extend in Table. 217 3. (Optional) Select the fields you want to see in your dataset. You can select fields whether or not the original dataset type has fixed fields. 4. 5. 6. 7. (Optional) Add more Splunk SPL commands until you have a search that returns results that you want to use in a dataset. Click Save to open the Save As New Table box. Enter a Table Title. Click Save to save the table. View and update a table dataset After you define the initial data for your table dataset, you can continue to use Table Views to refine it and maintain it. You also use Table Views to make changes to existing table datasets. Table Views includes several table dataset editing tools: Work with your table in two modes: Rows, which renders the dataset in a standard table format. Summary, which displays statistical information for each of the fields in your table and their values. Click directly on your table to make edits to your dataset. Move field columns, change field names, fix field type mismatches, and update field values. Apply actions to the table that filter events, add fields, edit field names and field values, perform statistical data aggregations, and more. You can apply actions through menu selections, or by making edits directly to table elements. Use a command history feature to review, edit, and undo actions that were applied to the table. Click SPL to see the search language generated for each of your commands. Get to Table Views There are three ways to get to Table Views. Method When you define initial data for a new table dataset When you edit an existing table dataset. Details See Define initial data for a new table dataset. See Edit a table dataset in Manage table datasets. When you extend an existing dataset as a new table dataset Table Views modes See Extend a dataset as a new table datasetin Manage table datasets. You can edit your table in two modes: Rows mode and Summary mode. Rows mode Rows mode is the default Table Views mode. It displays your table dataset as a table, with fields as columns, values in cells, and sample events in rows. It displays 50 sample events from your dataset. It does not represent the results from any particular time range. You can edit your table by applying actions to it, either by making menu selections or by making edits directly to the table. 218 In the context of Table Views, the Rows mode is a search tool rather than an editing tool. It does not provide a time range picker. If you want to see a table-formatted set of results from a specific time range, see Explore a dataset. Summary mode Click Summary to see analytical details about the fields in the table. You can see top value distributions, null value percentages, numeric value statistics, and more. You can apply some menu actions and commands to your table while you are in the Summary mode. You can also apply actions through direct edits, such as moving columns, renaming fields, fixing field type mismatches, and editing field values. When you are in the Summary mode, you can view field analytics for a specific range of time using the time range picker. The time range picker shows events from the last 24 hours by default. If your dataset has no events from the last 24 hours, it has no statistics when you open this view. To fix this, adjust the time range picker to a range where events are present. The time range picker gives you a variety of time range definition options. You can choose a preset time range, or you can define a custom time range. For help with the time range picker, see Select time ranges to apply to your search in the Search Manual. Table element selection options Availability of menu actions depends on the table elements that you select. For example, some actions are only available when you select a field column. You have the same selection options in the Rows and Summary views. Element Applies action to Table Column Entire dataset A field Multi-Column Two or more fields Cell Text A field value How to select Click the asterisk header at the top of the leftmost column. Click a column header. To select multiple nonadjacent columns, hold the CTRL or CMD key and click the header row of each column you wish to select. Deselect columns by clicking them while holding CTRL or CMD. To select a range of adjacent columns, click the header row of the first column, hold SHIFT, and click the header row of the last column. Click a cell. A portion of text within a field value. Click and drag to select text. You can select text for text and iPv4 field types

Summary: Identifying an Index and Source Type Combination for Initial Data in Splunk

Initial Data Setup Steps
- Open the Search & Reporting app and navigate to the Datasets listing page.
- Click Create Table View to access the initial data setup screen.
- Choose an index for your data. Options:
  - Specific Index.
  - All indexes (not recommended for broad datasets).
- Select a source type. Options:
  - Specific source type.
  - All source types (avoid overly broad selections).
- Verify the dataset preview to ensure rows, columns, and field values align with your requirements.
Add Missing Fields (Optional)
- Click Add a missing existing field at the top of the field list.
- Enter and add the field, then select it for inclusion in your dataset.
Finalize and Edit
- Once satisfied, click Start Editing to confirm selections.
- Use tools in Table Views for refinement, such as field adjustments and statistical data aggregations.
Alternative Methods for Initial Data
- Use Existing Dataset: Clone or extend datasets, preserving relationships.
  - Clone: Creates an identical copy.
  - Extend: Inherits changes from the original dataset.
- Provide Search String: Use Splunk SPL commands to define initial data.
Table Views Modes
- Rows Mode: Displays sample events (default view).
- Summary Mode: Provides field analytics like value distributions and statistics.
Time Range Picker
- Default: Last 24 hours. Adjust for datasets with no recent events.
- Options: Preset or custom time ranges.
Editing Tools
- Modify tables using actions such as field renaming, column movement, and type corrections.
- Use SPL to view and refine search commands generated by your actions.

By following these steps, you can effectively define and refine initial data for your Splunk dataset, ensuring precision and relevance.

SUMMARISE: Build a data model About data models Data models drive the pivot tool. Data models enable users of Pivot to create compelling reports and dashboards without designing the searches that generate them. Data models can have other uses, especially for Splunk app developers. Splunk knowledge managers design and maintain data models. These knowledge managers understand the format and semantics of their indexed data and are familiar with the Splunk search language. In building a typical data model, knowledge managers use knowledge object types such as lookups, transactions, search-time field extractions, and calculated fields. What is a data model? A data model is a hierarchically structured search-time mapping of semantic knowledge about one or more datasets. It encodes the domain knowledge necessary to build a variety of specialized searches of those datasets. These specialized searches are used by Splunk software to generate reports for Pivot users. When a Pivot user designs a pivot report, they select the data model that represents the category of event data that they want to work with, such as Web Intelligence or Email Logs. Then they select a dataset within that data model that represents the specific dataset on which they want to report. Data models are composed of datasets, which can be arranged in hierarchical structures of parent and child datasets. Each child dataset represents a subset of the dataset covered by its parent dataset. If you are familiar with relational database design, think of data models as analogs to database schemas. When you plug them into the Pivot Editor, they let you generate statistical tables, charts, and visualizations based on column and row configurations that you select. To create an effective data model, you must understand your data sources and your data semantics. This information can affect your data model architecture--the manner in which the datasets that make up the data model are organized. For example, if your dataset is based on the contents of a table-based data format, such as a .csv file, the resulting data model is flat, with a single top-level root dataset that encapsulates the fields represented by the columns of the table. The root dataset may have child dataset beneath it. But these child dataset do not contain additional fields beyond the set of fields that the child datasets inherit from the root dataset. Meanwhile, a data model derived from a heterogeneous system log might have several root datasets (events, searches, and transactions). Each of these root datasets can be the first dataset in a hierarchy of datasets with nested parent and child relationships. Each child dataset in a dataset hierarchy can have new fields in addition to the fields they inherit from ancestor datasets. Data model datasets can get their fields from custom field extractions that you have defined. Data model datasets can get additional fields at search time through regular-expression-based field extractions, lookups, and eval expressions. The fields that data models use are divided into the categories described above (auto-extracted, eval expression, regular expression) and more (lookup, geo IP). See Dataset field types. Data models are a category of knowledge object and are fully permissionable. A data model's permissions cover all of its data model datasets. 227 See Manage data models. Data models generate searches When you consider what data models are and how they work it can also be helpful to think of them as a collection of structured information that generates different kinds of searches. Each dataset within a data model can be used to generate a search that returns a particular dataset. We go into more detail about this relationship between data models, data model datasets, and searches in the following subsections. Dataset constraints determine the first part of the search through: Simple search filters (Root event datasets and all child datasets). Complex search strings (Root search datasets). transaction definitions (Root transaction datasets). When you select a dataset for Pivot, the unhidden fields you define for that dataset comprise the list of fields that you choose from in Pivot when you decide what you want to report on. The fields you select are added to the search that the dataset generates. The fields can include calculated fields, user-defined field extractions, and fields added to your data by lookups. The last parts of the dataset-generated-search are determined by your Pivot Editor selections. They add transforming commands to the search that aggregate the results as a statistical table. This table is then used by Pivot as the basis for charts and other types of visualizations. For more information about how you use the Pivot Editor to create pivot tables, charts, and visualizations that are based on data model datasets, see Introduction to Pivot in the Pivot Manual. Datasets Data models are composed of one or more datasets. Here are some basic facts about data model datasets: Each data model dataset corresponds to a set of data in an index. You can apply data models to different indexes and get different datasets. Datasets break down into four types. These types are: Event datasets, search datasets, transaction datasets, and child datasets. Datasets are hierarchical. Datasets in data models can be arranged hierarchically in parent/child relationships. The top-level event, search, and transaction datasets in data models are collectively referred to as "root datasets." Child datasets have inheritance. Data model datasets are defined by characteristics that mostly break down into constraints and fields. Child datasets inherit constraints and fields from their parent datasets and have additional constraints and fields of their own. We'll dive into more detail about these and other aspects of data model datasets in the following subsections. Child datasets provide a way of filtering events from parent datasets - Because a child dataset always provides an additional constraint on top of the constraints it has inherited from its parent dataset, the dataset it represents is always a subset of the dataset that its parent represents. 228 Root datasets and data model dataset types The top-level datasets in data models are called root datasets. Data models can contain multiple root datasets of various types, and each of these root datasets can be a parent to more child datasets. This association of base and child datasets is a dataset tree. The overall set of data represented by a dataset tree is selected first by its root dataset and then refined and extended by its child datasets. Root datasets can be defined by a search constraint, a search, or a transaction: Root event datasets are the most commonly-used type of root data model dataset. Each root event dataset broadly represents a type of event. For example, an HTTP Access root event dataset could correspond to access log events, while an Error event corresponds to events with error messages. Root event datasets are typically defined by a simple constraint. This constraint is what an experienced Splunk user might think of as the first portion of a search, before the pipe character, commands, and arguments are applied. For example, status > 600 and sourcetype=access_* OR sourcetype=iis* are possible event dataset definitions. See Dataset constraints. Root search datasets use an arbitrary Splunk search to define the dataset that it represents. If you want to define a base dataset that includes one or more fields that aggregate over the entire dataset, you might need to use a root search dataset that has transforming commands in its search. For example: a system security dataset that has various system intrusion events broken out by category over time. The constraint for the root search dataset must be an event index search. For example, datamodel=dmtest_kvstore.test and datamodel=dmtest_csv.test aren't supported. In addition, the root search dataset constraint can't begin with a command other than the search command. Root transaction datasets let you create data models that represent transactions: groups of related events that span time. Transaction dataset definitions utilize fields that have already been added to the model via event or search dataset, which means that you can't create data models that are composed only of transaction datasets and their child datasets. Before you create a transaction dataset you must already have some event or search dataset trees in your model. Child datasets of all three root dataset types--event, transaction, and search--are defined with simple constraints that narrow down the set of data that they inherit from their ancestor datasets. Dataset types and data model acceleration You can optionally use data model acceleration to speed up generation of pivot tables and charts. There are restrictions to this functionality that can have some bearing on how you construct your data model, if you think your users would benefit from data model acceleration. To accelerate a data model, it must contain at least one root event dataset, or one root search dataset that only uses streaming commands. Acceleration only affects these dataset types and datasets that are children of those root datasets. You cannot accelerate root search datasets that use nonstreaming commands (including transforming commands), root transaction datasets, and children of those datasets. Data models can contain a mixture of accelerated and unaccelerated datasets. See Manage data models. 229 See Command types in the Search Reference for more information about streaming commands and other command types. Example of data model dataset hierarchies The following example shows the first several datasets in a "Call Detail Records" data model. Four top-level root datasets are displayed: All Calls, All Switch Records, Conversations, and Outgoing Calls. 230 All Calls and All Switch Records are root event datasets that represent all of the calling records and all of the carrier switch records, respectively. Both of these root event datasets have child datasets that deal with subsets of the data owned by their parents. The All Calls root event dataset has child datasets that break down into different call classifications: Voice, SMS, Data, and Roaming. If you were a Pivot user who only wanted to report on aspects of cellphone data usage, you'd select the Data dataset. But if you wanted to create reports that compare the four call types, you'd choose the All Calls root event dataset instead. Conversations and Outgoing Calls are root transaction datasets. They both represent transactions--groupings of related events that span a range of time. The "Conversations" dataset only contains call records of conversations between two or more people where the maximum pause between conversation call record events is less than five hours and the total length of the conversation is less than one day. For details about defining different data model dataset types, see Design data models. Dataset constraints All data model datasets are defined by sets of constraints. Dataset constraints filter out events that aren't relevant to the dataset. For a root event dataset or a child dataset of any type, the constraint looks like a simple search, without additional pipes and search commands. For example, the constraint for HTTP Request, one of the root event dataset of the Web Intelligence data model, is sourcetype=access_*. For a root search dataset, the constraint is the dataset search string. For a root transaction dataset, the constraint is the transaction definition. Transaction dataset definitions must identify Group Dataset (either one or more event dataset, a search dataset, or a transaction dataset) and one or more Group By fields. They can also optionally include Max Pause and Max Span values. Constraints are inherited by child datasets. Constraint inheritance ensures that each child dataset represents a subset of the data represented by its parent datasets. Your Pivot users can then use these child datasets to design reports with datasets that already have extraneous data prefiltered out. In the following example, we will use a data model called Buttercup Games. Its Successful Purchases dataset is a child of the root event dataset HTTP Requests and is designed to contain only those events that represent successful customer purchase actions. Successful Purchases inherits constraints from HTTP Requests and another parent dataset named Purchases. 1. 2. HTTP Requests starts by setting up a search that only finds webserver access events. sourcetype=access_* The Purchases dataset further narrows the focus down to webserver access events that involve purchase actions. 231 action=purchase 3. And finally, Successful Purchases adds a constraint that reduces the dataset event set to web access events that represent successful purchase events. status=200 When all the constraints are added together, the base search for the Successful Purchases dataset looks like this: sourcetype=access_* action=purchase status=200 A Pivot user might use this dataset for reporting if they know that they only want to report on successful purchase actions. For details about datasets and dataset constraints, see the topic Design data models. Dataset field types There are five types of data model dataset fields. Auto-extracted A field extracted by the Splunk software at index time or search time. You can only add auto-extracted fields to root datasets. Child datasets can inherit them, but they cannot add new auto-extracted fields of their own. Auto-extracted fields divide into three groups. Group Fields added by automatic key value field extraction Definition These are fields that the Splunk software extracts automatically, like uri or version. This group includes fields indexed through structured data inputs, such as fields extracted from the headers of indexed CSV files. See Extract fields from files with structured data in Getting Data In. Fields added by knowledge objects Fields that you have manually added Fields added to search results by field extractions, automatic lookups, and calculated field configurations can all appear in the list of auto-extracted fields. You can manually add fields to the auto-extracted fields list. They might be rare fields that you do not currently see in the dataset, but may appear in it at some point in the future. This set of fields can include fields added to the dataset by generating commands such as inputcsv or dbinspect. Eval Expression A field derived from an eval expression that you enter in the field definition. Eval expressions often involve one or more extracted fields. Lookup A field that is added to the events in the dataset with the help of a lookup that you configure in the field definition. Lookups add fields from external data sources such as CSV files and scripts. When you define a lookup field you can use any lookup object in your system and associate it with any other field that has already been associated with that same dataset. See About lookups. 232 Regular Expression This field type is extracted from the dataset event data using a regular expression that you provide in the field definition. A regular expression field definition can use a regular expression that extracts multiple fields; each field will appear in the dataset field list as a separate regular expression field. Geo IP A specific type of lookup that adds geographical fields, such as latitude, longitude, country, and city to events in the dataset that have valid IP address fields. Useful for map-related visualizations. See Design data models. Field categories The Data Model Editor groups data model dataset fields into three categories. Category Inherited Definition All datasets have at least a few inherited fields. Child fields inherit fields from their parent dataset, and these inherited fields always appear in the Inherited category. Root event, search, and transaction datasets also have default fields that are categorized as inherited. Extracted Calculated Any auto-extracted field that you add to a dataset is listed in the "Extracted" field category. The Splunk software derives calculated fields through a calculation, lookup definition, or field-matching regular expression. When you add Eval Expression, Regular Expression, Lookup, and Geo IP field types to a dataset, they all appear in this field category. The Data Model Editor lets you arrange the order of calculated fields. This is useful when you have a set of fields that must be processed in a specific order. For example, you can define an Eval Expression that adds a set of fields to events within the dataset. Then you can create a Lookup with a definition that uses one of the fields calculated by the eval expression. The lookup uses this definition to add another set of fields to the same events. Field inheritance All data model datasets have inherited fields. A child dataset will automatically have all of the fields that belong to its parent. All of these inherited fields will appear in the child dataset's "Inherited" category, even if the fields were categorized otherwise in the parent dataset. You can add additional fields to a child dataset. The Data Model Editor will categorize these datasets either as extracted fields or calculated fields depending on their field type. You can design a relatively simple data model where all of the necessary fields for a dataset tree are defined in its root dataset, meaning that all of the child datasets in the tree have the exact same set of fields as that root dataset. In such a data model, the child datasets would be differentiated from the root dataset and from each other only by their constraints. Root event, search, and transaction datasets also have inherited fields. These inherited fields are default fields that are extracted from every event, such as _time, host, source, and sourcetype. You cannot delete inherited fields, and you cannot edit their definitions. The only way to edit or remove an inherited field belonging to a child dataset is to delete or edit the field from the parent dataset it originates from as an extracted or calculated field. If the field originates in a root dataset as an inherited field, you won't be able to delete it or edit it. 233 You can hide fields from Pivot users as an alternative to field deletion. You can also determine whether inherited fields are optional for a dataset or required. Field purposes Fields serve several purposes for data models. Their most obvious function is to provide the set of fields that Pivot users use to define and generate a pivot report. The set of fields that a Pivot user has access to is determined by the dataset the user chooses when they enter the Pivot Editor. You might add fields to a child dataset to provide fields to Pivot users that are specific to that dataset. On the other hand, you can also design calculated fields whose only function is to set up the definition of other fields or constraints. This is why field listing order matters: Fields are processed in the order that they are listed in the Data Model Editor. This is why The Data Model Editor allows you to rearrange the listing order of calculated fields. For example, you could design a chained set of three Eval Expression fields. The first two Eval Expression fields would create what are essentially calculated fields. The third Eval Expression field would use those two calculated fields in its eval expression. Fields can be visible or hidden to Pivot users When you define a field you can determine whether it is visible or hidden for Pivot users. This can come in handy if each dataset in your data model has lots of fields but only a few fields per dataset are actually useful for Pivot users. A field can be visible in some datasets and hidden in others. Hiding a field in a parent dataset does not cause it to be hidden in the child datasets that descend from it. Fields are visible by default. Fields that have been hidden for a dataset are marked as such in the dataset's field list. The determination of what fields to include in your model and which fields to expose for a particular dataset is something you do to make your datasets easier to use in Pivot. It's often helpful to your Pivot users if each dataset exposes only the data that is relevant to that dataset, to make it easier to build meaningful reports. This means, for example, that you can add fields to a root dataset that are hidden throughout the model except for a specific dataset elsewhere in the hierarchy, where their visibility makes sense in the context of that dataset and its particular dataset. Consider the example mentioned in the previous subsection, where you have a set of three "chained" Eval Expression fields. You may want to hide the first two Eval Expression fields because they are just there as "inputs" to the third field. You would leave the third field visible because it's the final "output"--the field that matters for Pivot purposes. Fields can be required or optional for a dataset During the field design process you can also determine whether a field is required or optional. This can act as a filter for the event set represented by the dataset. If you say a field is required, you're saying that every event represented by the dataset must have that field. If you define a field as optional, the dataset may have events that do not have that field at all. Note: As with field visibility (see above) a field can be required in some datasets and optional in others. Marking a field as required in a parent dataset will not automatically make that field required in the child datasets that descend from that parent dataset. 234 Fields are optional by default. Fields that have had their status changed to required for a dataset are marked as such in the dataset's field list.

Summary: Building a Data Model in Splunk

1. Overview of Data Models
Data models in Splunk are hierarchical, search-time mappings of semantic knowledge that enable users to create dashboards and reports without directly writing searches. Designed by Splunk knowledge managers, data models rely on indexed data and knowledge objects like lookups, transactions, field extractions, and calculated fields.

2. What is a Data Model?

Structure: Data models consist of datasets organized hierarchically into parent and child relationships.
Purpose: They simplify report creation by representing specific datasets and using semantic knowledge to generate specialized searches.
Analogy: Comparable to database schemas, data models act as templates for generating statistical tables, charts, and visualizations.

3. Creating a Data Model

Key Requirements: Understand data sources, semantics, and architecture.
Examples:
- Flat models (e.g., CSV files) have a single root dataset.
- Hierarchical models (e.g., logs) include multiple root datasets (events, searches, transactions) with nested child datasets.
Field Sources:
- Auto-extracted fields (e.g., from logs or lookups).
- Fields created at search time (e.g., eval expressions, regex extractions).

4. Types of Datasets

Event Datasets: Broadly represent event categories (e.g., HTTP logs).
Search Datasets: Use custom search strings.
Transaction Datasets: Represent groups of related events.
Child Datasets: Subsets of parent datasets, inheriting fields and adding constraints.

5. Dataset Hierarchies

Parent-Child Structure: Child datasets inherit constraints and fields from their parents but add specificity.
Root Datasets: Serve as the top-level datasets and define overall data scope.

6. Dataset Fields
Five types of fields:

Auto-Extracted: Extracted automatically (e.g., headers in CSV files).
Eval Expressions: Derived through calculations.
Lookups: Added from external data sources.
Regular Expressions: Extracted using custom regex patterns.
Geo IP: Adds geographic data for IPs.

7. Dataset Constraints

Definition: Filters events for relevance using simple search filters, search strings, or transaction definitions.
Inheritance: Child datasets automatically inherit and refine constraints from parent datasets.

8. Acceleration

Purpose: Speeds up pivot table and chart generation.
Applicability: Works with root event datasets or root search datasets with streaming commands.

9. Use Cases

Example: A "Call Detail Records" data model organizes datasets hierarchically (e.g., All Calls → Voice, SMS). Users can choose specific datasets (e.g., "Data") for targeted reports.

10. Field Categories and Inheritance

Categories: Fields are grouped as inherited, extracted, or calculated.
Inheritance: Child datasets inherit parent fields but can add new fields.

Data models streamline reporting, enabling effective data analysis and visualization while leveraging Splunk’s advanced search capabilities.

SUMMARISE: Create a new data model Prerequisites You can only create data models if your permissions enable you to do so. Your role must have the ability to write to at least one app. If your role has insufficient permissions the New Data Model button will not appear. See Enable roles to create data models. Steps 1. 2. 3. 4. 5. 6. Navigate to the Data Models management page. Click New Data Model to create a new data model. Enter the data model Title. The Title field can accept any character except asterisks. It can also accept blank spaces between characters. The data model ID field fills in as you enter the title. Do not update it. The data model ID must be a unique identifier for the data model. It can only contain letters, numbers, and underscores. Spaces between characters are also not allowed. After you click Create you cannot change the ID value. (Optional) Enter the data model Description. (Optional) Change the 'App value if you want the data model to belong to a different app context. App displays app context that you are in currently. Click Create to open the new data model in the Data Model Editor, where you can begin adding and defining the datasets that make up the data model. When you first enter the Data Model Editor for a new data model it will not have any datasets. To define the data model's first dataset, click Add Dataset and select a dataset type. For more information about dataset definition, see the following sections on adding field, search, transaction, and child datasets. For all the details on the Data Model Editor and the work of creating data model datasets, see Design data models. Enable roles to create data models By default only users with the admin or power role can create data models. For other users, the ability to create a data model is tied to whether their roles have "write" access to an app. To grant another role write access to an app, follow these steps. Steps 1. 2. 3. Click the App dropdown at the top of the page and select Manage Apps to go to the Apps page. On the Apps page, find the app that you want to grant data model creation permissions for and click Permissions. On the Permissions page for the app, select Write for the roles that should be able to create data models for the app. 236 4. Click Save to save your changes. Giving roles the ability to create data models can have other implications. See Disable or delete knowledge objects. About data model permissions Data models are knowledge objects, and as such the ability to view and edit them is governed by role-based permissions. When you first create a data model it is private to you, which means that no other user can view it on the Select a Data Model page or Data Models management page or update it in any way. If you want to accelerate a data model, you need to share it first. You cannot accelerate private data models. See Enable data model acceleration. Align data model permissions with those of related knowledge objects When you share a data model the knowledge objects associated with that data model (such as lookups or field extractions) must have the same permissions. Otherwise, people may encounter errors when they use the data model. For example, if your data model is shared to all users of the Search app but uses a lookup table and lookup definition that is only shared with users of the Search app that have the Admin role, everything will work fine for Admin role users, but all other users will get errors when they try to use the data model in Pivot. The solution is either to restrict the data model to Admin users or to share the lookup table and lookup definition to all users of the Search app. Edit the permissions for a data model Prerequisites Manage knowledge object permissions Steps 1. 2. Go to the Data Models management page. Locate the data model that you want to edit permissions for. Use one of the following options. Option Select Edit > Edit Permissions. Additional steps for this option None Expand the row for the dataset. 3. Click Edit for permissions. Edit the dataset permissions and click Save to save your changes. This brings up the Edit Permissions dialog, which you can use to share private data models with others, and to determine the access levels that various roles have to the data models.

Summary: Creating a New Data Model

Prerequisites

Your role must have permissions to write to at least one app. Without the necessary permissions, the "New Data Model" button will not appear.
To enable roles to create data models, grant them "write" access to an app.

Steps to Create a Data Model

Navigate to the Data Models management page.
Click New Data Model.
Enter a title for the data model. Titles can include spaces and any characters except asterisks.
The Data Model ID auto-fills based on the title. It must be unique, use only letters, numbers, and underscores, and cannot contain spaces. The ID cannot be changed after creation.
Optionally, add a description and update the app context.
Click Create to open the Data Model Editor.
In the editor, click Add Dataset to define the data model's datasets (e.g., fields, searches, transactions).

For detailed guidance, refer to the "Design data models" documentation.

Enabling Roles to Create Data Models

Go to Manage Apps via the App dropdown.
Locate the target app and click Permissions.
Assign "Write" access to the desired roles.
Save the changes.

Data Model Permissions

Newly created data models are private. Only the creator can view or edit them.
To share or accelerate a data model, adjust permissions.
Ensure related knowledge objects (e.g., lookups) have compatible permissions to avoid errors for users.

Editing Data Model Permissions

Access the Data Models management page.
Locate the data model and choose Edit > Edit Permissions.
Adjust permissions in the dialog box to share or modify access levels.
Save changes.

Align permissions across all related knowledge objects to ensure proper functionality.

SUMMARISE: Some best practices for data model design It can take some trial and error to determine the data model designs that work for you. Here are some tips that can get you off to a running start. 251 Use root event datasets and root search datasets that only use streaming commands. This takes advantage of the benefits of data model acceleration. When you define constraints for a root event dataset or define a search for a root search dataset that you will accelerate, include the index or indexes it is selecting from. Data model acceleration efficiency and accuracy is improved when the data model is directed to search a specific index or set of indexes. If you do not specify indexes, the data model searches over all available indexes. Avoid circular dependencies in your dataset definitions. Circular dependencies occur when a dataset constraint definition includes fields that are not auto-extracted or inherited from a parent dataset. Circular dependencies can lead to a variety of problems including, but not limited to, invalid search results. Minimize dataset hierarchy depth. Constraint-based filtering is less efficient deeper down the tree. Use field flags to selectively expose small groups of fields for each dataset. You can expose and hide different fields for different datasets. A child field can expose an entirely different set of fields than those exposed by its parent. Your Pivot users will benefit from this selection by not having to deal with a bewildering array of fields whenever they set out to make a pivot chart or table. Instead they'll see only those fields that make sense in the context of the dataset they've chosen. Reverse-engineer your existing dashboards and searches into data models. This can be a way to quickly get started with data models. Dashboards built with pivot-derived panels are easier to maintain. When designing a new data model, first try to understand what your Pivot users hope to be able to do with it. Work backwards from there. The structure of your model should be determined by your users' needs and expectations

Summary: Best Practices for Data Model Design

Use Streaming Commands for Root Datasets
- Define root event and search datasets using streaming commands to leverage data model acceleration effectively.
Specify Indexes for Root Datasets
- Include the index or indexes in constraints or searches for root datasets to improve acceleration efficiency and accuracy. Avoid searching across all indexes unless necessary.
Avoid Circular Dependencies
- Ensure dataset constraints only use auto-extracted or parent-inherited fields to prevent issues like invalid search results caused by circular dependencies.
Minimize Dataset Hierarchy Depth
- Keep the hierarchy shallow, as constraint-based filtering becomes less efficient deeper down the tree.
Use Field Flags
- Expose only relevant fields for each dataset using field flags. This simplifies the experience for Pivot users, showing only the fields applicable to their chosen dataset.
Leverage Existing Dashboards and Searches
- Reverse-engineer dashboards and searches into data models to get a head start and simplify maintenance, especially for Pivot-derived panels.
Design with User Needs in Mind
- Understand Pivot users' goals and work backward to structure the data model based on their requirements and expectations.

SUMMARISE: Use data summaries to accelerate searches Overview of summary-based search acceleration Searches over large datasets can take a long time to complete. This isn't a problem if you run such searches on an infrequent basis. But if you are like many users of Splunk Enterprise, you do not have this luxury. Large dataset searches must be run on schedules, made the basis for panels in popular dashboards, or run ad-hoc frequently by large numbers of users. Splunk Enterprise offers several approaches to speeding up searches of large datasets. One of these approaches is summary-based search acceleration. This is where you create a data summary that is populated by background runs of a slow-completing search. The summary is a smaller dataset that contains only data that is relevant to your search. When you run the search against the summary, the search should complete much faster. There are three methods of summary-based search acceleration: Report acceleration - Uses automatically-created summaries to speed up completion times for certain kinds of event searches. Data model acceleration - Uses automatically-created event summaries to speed up completion times for data-model-based searches. Summary indexing - Populates a summary index using a scheduled search that you define. You can create summary indexes of event data, or you can convert your event data into metrics and summarize it in metrics summary indexes. Report and data model acceleration work only with event data. You can create summary indexes for either event data or metric data. Comparing summary-based search acceleration methods Acceleration method Description Location of summary When should you use it? For more information Report acceleration Accelerates qualifying transforming searches of event data that have been saved as reports. Features automatic backfill for data interruptions. Similar saved searches can use the same acceleration summary when they are accelerated. In .tsidx files, stored alongside buckets in indexers. Use for any qualifying saved search that has 100k or more hot bucket events. Not all searches qualify. Manage report acceleration Data model acceleration Accelerates searches run against qualifying data models by running those searches on a summary of the data model rather than the data model itself. Allows you to speed up searches against large and varied datasets. In .tsidx files, stored alongside buckets in indexers. Consider enabling acceleration for any qualifying data model. Data model acceleration can be faster than report acceleration, especially for relatively complicated searches. Accelerate data models Event summary indexing Speeds up slow-completing transforming searches of event data by summarizing the events returned by the search in a separate In a summary index composed of summarized event data. You must predefine the Create an event summary index if you want to speed up a transforming search that does not qualify for report acceleration. You might also Use summary indexing for increased search efficiency 268 Acceleration method Description events index. Metrics summary indexing Speeds up slow-completing transforming searches of event data by converting the events returned by the search into metric data points and summarizing those metric data points in a separate metrics index. Location of summary event summary index if one does not already exist. When should you use it? want to create a summary index to keep certain data in an index with different data retention policies than your other indexes. For more information In a summary index composed of aggregated metric data points. You must predefine the metric summary index if one does not already exist. Batch mode search Use metrics summary indexing over event summary indexing if it makes sense to convert your event data into metrics data. Metrics summary indexes can provide faster search performance and more efficient data storage than event summary indexes. Use summary indexing for increased search efficiency Batch mode search is a feature that improves the performance and reliability of transforming searches. For transforming searches that don't require the events to be time-ordered, running in batch mode means that the search executes bucket-by-bucket (in batches), rather than over time. In certain reporting cases, this means that the transforming search can complete faster. Additionally, batch mode search improves the reliability for long-running distributed searches, which can fail when an indexer goes down while the search is running. In this case, Splunk software attempts to reconnect to the missing peer and retry the search. Transforming searches that meet the criteria for batch mode search include: Generating and transforming searches (stats, chart, etc.) that do not include the localize or transaction commands in the search. Searches that are not real-time and not summarizing searches. Non-distributed searches that are not stateful streaming. (A streamstats search is an example of a stateful streaming search.) Batch mode search is invoked from the configuration file, in the [search] stanza of limits.conf. Use the search inspector to determine whether or not a transforming search is running in batch mode. See Configure batch mode search. Manage report acceleration Report acceleration is the easiest way to speed up transforming searches and reports that take a long time to complete because they have to cover a large volume of data. You enable acceleration for a transforming search when you save it as a report. You can also accelerate report-based dashboard panels that use a transforming search. This topic covers various aspects of report acceleration in more detail. It includes: A quick guide to enabling automatic acceleration for a transforming report. Examples of qualifying and nonqualifying reports (only specific kinds of reports qualify for report acceleration). Details on how report acceleration summaries are created and maintained An overview of the Report acceleration summaries page in Settings, which you can use to review and maintain the data summaries that are used for automatic report acceleration. 269 Restrictions on report acceleration You cannot accelerate a report if: You created it though Pivot. Pivot reports are accelerated via data model acceleration. See Manage data models. Your permissions do not enable you to accelerate searches. You cannot accelerate reports if your role does not have the schedule_search and accelerate_search capabilities. Your role does not have write permissions for the report. The search that the report is based upon is disqualified for acceleration. For more information, see How reports qualify for acceleration. In addition, be careful when accelerating reports whose base searches include tags, event types, search macros, and other knowledge objects whose definitions can change independently of the report after the report is accelerated. If this happens, the accelerated report can return invalid results. If you suspect that your accelerated report is returning invalid results, you can verify its summary to see if the data contained in the summary is consistent. See Verify a summary. Enabling report acceleration You can enable report acceleration when you create a report, or later, after the report is created. For a more thorough description of this procedure, see Create and edit reports, in the Reporting Manual. Enabling report acceleration when you create a report You can enable report acceleration for qualifying reports. Prerequisites How reports qualify for acceleration Steps 1. 2. 3. 4. 5. 6. 7. 8. In Search, run a search that qualifies for report acceleration. Select Save As > Report. Give your report a Name and optionally, a Description. Click Save to save the search as a report. Select Acceleration. You can only accelerate the report if the report qualifies for acceleration and your permissions allow you to accelerate reports. To be able to accelerate reports your role has to have the schedule_search and accelerate_search capabilities. Select Accelerate Report. Select a Summary Range. Base your selection on the range of time over which you plan to run the report. For example, if you only plan to run the report over periods of time within the last seven days, choose 7 Days. Click Save to save your acceleration settings. 270 Enabling report acceleration for an existing report You can enable acceleration for a qualifying existing report. Prerequisites How reports qualify for acceleration Steps 1. 2. 3. 4. 5. 6. On the Reports listing page, find a report that you want to accelerate. Expand the report row by clicking on the > symbol in the first column. The Acceleration line item displays the acceleration status for the report. Its value will be Disabled if it is not accelerated. Click Edit You can only accelerate the report if the report qualifies for acceleration and your permissions allow you to accelerate reports. To be able to accelerate reports your role has to have the schedule_search and accelerate_search capabilities. Select Accelerate Report. Select a Summary Range. Base your selection on the range of time over which you plan to run the report. For example, if you only plan to run the report over periods of time within the last seven days, choose 7 Days. Click Save to save your acceleration settings. Alternatively, you can enable report acceleration for an existing report at Settings > Searches, Reports, and Alerts. After you enable acceleration for a report When you enable acceleration for your report, the Splunk software begins building a report acceleration summary for the report if it determines that the report would benefit from summarization. To find out whether your report summary is being constructed, go to Settings > Report Acceleration Summaries. If the Summary Status is stuck at 0% complete for an extended amount of time, the summary is not being built. See Conditions under which Splunk software cannot build or update a summary. Once the summary is built, future runs of an accelerated report should complete faster than they did before. See the subtopics below for more information on summaries and how they work. Note: Report acceleration only works for reports that have Search Mode set to Smart or Fast. If you select the Verbose search mode for a report that benefits from report acceleration, it will run as slow as it would if no summary existed for it. Search Mode does not affect searches powering dashboard panels. For more information about the Search Mode settings, see Search modes in the Search Manual. How reports qualify for acceleration For a report to qualify for acceleration its search must meet three criteria: The search string must use a transforming command (such as chart, timechart, stats, and top). If the search string has any commands before the first transforming command, they must be streamable. The search cannot use event sampling. 271 Note: You can use non-streaming commands after the first transforming command and still have the report qualify for automatic acceleration. It's just non-streaming commands before the first transforming command that disqualify the report. For more information about event sampling, see Event sampling in the Search Manual. Examples of qualifying search strings Here are examples of search strings that qualify for report acceleration: index=_internal | stats count by sourcetype index=_audit search=* | rex field=search "'(?.*)'" | chart count by user search test foo bar | bin _time span=1d | stats count by _time x y index=_audit | lookup usertogroup user OUTPUT group | top searches by group Examples of nonqualifying search strings And here are examples of search strings that do not qualify for report acceleration: Reason the following search string fails: This is a simple event search, with no transforming command. index=_internal metrics group=per_source_thruput Reason the following search string fails: eventstats is not a transforming command. index=_internal sourcetype=splunkd *thruput | eventstats avg(kb) as avgkb by group Reason the following search string fails: transaction is not a streaming command. Other non-streaming commands include dedup, head, tail, and any other search command that is not on the list of streaming commands. index=_internal | transaction user maxspan=30m | timechart avg(duration) by user Search strings that qualify for report acceleration but won't get much out of it In addition, you can have reports that technically qualify for report acceleration, but which may not be helped much by it. This is often the case with reports with high data cardinality--something you'll find when there are two or more transforming commands in the search string and the first transforming command generates many (50k+) output rows. For example: index=* | stats count by id | stats avg(count) as avg, count as distinct_ids Set report acceleration summary time ranges Report acceleration summaries span an approximate range of time. You determine this time range when you choose a value from the Summary Range list. At times, a report acceleration summary can have a store of data that slightly exceeds its summary range, but the summary never fails to meet that range, except while it is first being created. For example, if you set a summary range of 7 days for an accelerated report, a data summary that approximately covers the past seven days is created. Every ten minutes, a search is run to ensure that the summary always covers the selected range. These maintenance searches add new summary data and and remove older summary data that passes out of the range. 272 When you then run the accelerated report over a range that falls within the past 7 days, the report searches its summary rather than the source index (the index the report originally searched). In most cases the summary has far less data than the source index, and this--along with the fact that the report summary contains precomputed results for portions of the search pipeline--means that the report should complete faster than it did on its initial run. When you run the accelerated report over a period of time that is only partially covered by its summary, the report does not complete quite as fast because the Splunk software has to go to the source index for the portion of the report time range that does not fall within the summary range. If the Summary Range setting for a report is 7 Days and you run the report over the last 9 days, the Splunk software only gets acceleration benefits for the portion of the report that covers the past 7 days. The portion of the report that runs over days 8 and 9 will run at normal speed. Keep this in mind when you set the Summary Range value. If you always plan to run a report over time ranges that exceed the past 7 days, but don't extend further out than 30 days, you should select a Summary Range of 1 month when you set up report acceleration for that report.

Summary: Using Data Summaries for Accelerating Searches in Splunk Enterprise

Splunk Enterprise offers summary-based search acceleration methods to improve search performance over large datasets. This involves creating smaller data summaries from slow-completing searches and using them for faster searches.

Methods of Summary-Based Search Acceleration

Report Acceleration:
- Description: Automatically creates summaries for qualifying transforming searches saved as reports.
- Location: .tsidx files stored alongside buckets on indexers.
- Use Case: Best for reports with 100k+ hot bucket events. Useful for reports created directly from searches.
Data Model Acceleration:
- Description: Speeds up searches against large datasets by summarizing data models.
- Location: .tsidx files stored alongside buckets on indexers.
- Use Case: Recommended for complex searches run against qualifying data models.
Summary Indexing:
- Event Summary Indexing: Summarizes event data into a predefined summary index.
- Metrics Summary Indexing: Converts event data into metric data points and stores them in a metrics index.
- Use Case: Ideal for non-qualifying searches or for data with specific retention policies.

Optimizing Report Acceleration

Enabling Report Acceleration:
- Qualify searches using transforming commands (e.g., stats, chart, timechart) and streamable commands before the transformation.
- Select a Summary Range that matches your intended reporting time (e.g., 7 days, 1 month).
Conditions for Acceleration:
- Reports created using Pivot or disqualified searches (e.g., non-transforming or non-streaming commands) cannot be accelerated.
- Invalid results may occur if dependent objects (e.g., macros) change after acceleration.
High-Cardinality Data Consideration:
- Acceleration may not be effective for reports generating 50k+ output rows after the first transformation.

Batch Mode Search for Transforming Searches

Executes searches in batches, improving speed and reliability for long-running distributed searches.
Criteria:
- Uses transforming commands.
- Excludes real-time or stateful streaming searches (e.g., streamstats).

Key Takeaways for Faster Searches

Choose the appropriate method based on the dataset size, search complexity, and use case.
Optimize search design to ensure eligibility for acceleration methods.
Set appropriate summary ranges to maximize acceleration benefits.
Utilize metrics summary indexing for data that can be converted to metrics for faster performance and efficient storage.

These practices ensure reliable, scalable, and efficient search acceleration in Splunk Enterprise.

WHAT IS PERL

Perl is a high-level, general-purpose, interpreted programming language. It was originally developed by Larry Wall in 1987 as a Unix scripting language to make report processing easier but has since evolved to support a wide range of programming tasks. Here's an overview:

Key Features of Perl:

Text Processing:
- Perl excels in text manipulation tasks such as pattern matching, string parsing, and regular expression processing.
- It is widely used for tasks involving large volumes of textual data.
Cross-Platform Compatibility:
- Perl runs on most operating systems, including Unix, Linux, Windows, and macOS.
Extensive Library Support:
- Perl has a comprehensive repository of reusable modules and libraries, known as CPAN (Comprehensive Perl Archive Network), which covers everything from web development to bioinformatics.
Dynamic Typing and Flexibility:
- Perl is dynamically typed and supports both procedural and object-oriented programming paradigms.
Regular Expressions Integration:
- One of Perl’s standout features is its powerful and integrated support for regular expressions, making it ideal for pattern matching and text manipulation.
"There's More Than One Way to Do It" (TMTOWTDI):
- This philosophy emphasizes Perl's flexibility and the idea that problems can often be solved in multiple ways, allowing programmers to choose their preferred approach.

Common Uses of Perl:

System Administration: Automating administrative tasks such as backups and log analysis.
Web Development: CGI scripting and building dynamic web applications.
Text Processing: Parsing and transforming text files.
Bioinformatics: Analyzing DNA and protein sequences.
Database Interaction: Querying and managing databases.
Networking: Developing network protocols and handling socket programming.

Example Code (Hello World in Perl):

perl
#!/usr/bin/perl
print "Hello, World!\n";

Advantages of Perl:

Simple syntax for regular expression and text processing.
Extensive module library (CPAN).
Mature and well-supported community.

Disadvantages:

Code readability can be challenging, especially for complex scripts.
Slower execution compared to languages like C or Java.

Despite newer languages like Python gaining popularity, Perl remains a versatile tool in areas like text processing, legacy system maintenance, and bioinformatics.

for PG students of forensics : pl discuss: the importance of digital forensics in cyber security and criminal investigation by helping detect mitigate and investigate cyber attacks

The Importance of Digital Forensics in Cybersecurity and Criminal Investigation

Digital forensics plays a critical role in modern cybersecurity and criminal investigations. As cybercrimes grow in sophistication and frequency, digital forensics provides the tools and methodologies to detect, mitigate, and investigate these attacks effectively. Here’s an exploration of its significance in these domains:

1. Detecting Cyber Attacks

Digital forensics helps in identifying and analyzing digital evidence to detect cyber incidents early.

Key Contributions:

Analyzing System Logs: Examining server and system logs to identify unauthorized access or unusual activity.
Malware Analysis: Investigating malicious software to determine its origin, purpose, and scope of damage.
Network Traffic Monitoring: Detecting suspicious patterns in network traffic to identify breaches or data exfiltration.
Forensic Imaging: Creating and analyzing forensic images of compromised systems to preserve evidence while investigating.

2. Mitigating Cyber Attacks

Digital forensics supports real-time and post-incident measures to contain and reduce the impact of cyber attacks.

Key Contributions:

Incident Response: Identifying the root cause of an attack and assisting in containment strategies to prevent further harm.
Vulnerability Assessment: Identifying vulnerabilities exploited during an attack to guide future defense mechanisms.
Threat Intelligence: Gathering insights about the tools, techniques, and procedures (TTPs) used by attackers for proactive defense.
Data Recovery: Restoring systems and retrieving lost or encrypted data, such as in ransomware cases.

3. Investigating Cyber Attacks

Forensic investigations are crucial for understanding the full scope of cyber incidents and holding perpetrators accountable.

Key Contributions:

Evidence Collection and Preservation:
- Extracting digital evidence from devices, networks, and cloud services while maintaining a proper chain of custody.
Attribution:
- Tracing the source of an attack, such as identifying IP addresses, malware signatures, or attack patterns linked to known threat actors.
Timeline Reconstruction:
- Reconstructing events leading up to and following an attack to understand the sequence of actions taken by the attacker.
Legal Admissibility:
- Ensuring digital evidence is collected and handled in a manner compliant with legal standards, making it admissible in court.

4. Application in Criminal Investigations

Digital forensics is pivotal in solving both cyber-specific crimes and traditional crimes involving digital evidence.

Cyber-Specific Crimes:

Hacking: Identifying methods and entry points used by attackers.
Phishing and Fraud: Tracing email headers, IP addresses, and financial transactions.
Cyberstalking and Harassment: Analyzing social media interactions, emails, and other digital communications.

Traditional Crimes with Digital Evidence:

Terrorism and Organized Crime: Investigating communications, financial transactions, and propaganda distribution.
Human Trafficking: Uncovering evidence from devices used for coordination and communication.
White-Collar Crimes: Analyzing financial records, emails, and data manipulation activities.

5. Role in Cybersecurity

Digital forensics also strengthens the broader field of cybersecurity by:

Improving Defense Strategies:
- Insights from forensic investigations inform better firewalls, intrusion detection systems, and security protocols.
Training Cybersecurity Teams:
- Equipping teams with forensic skills to handle incidents effectively.
Developing Legal Frameworks:
- Assisting in creating laws and policies to address emerging threats in cyberspace.

Case Studies Highlighting the Importance

Wannacry Ransomware (2017):
- Digital forensics traced the malware to its creators, helping law enforcement take action.
Sony Pictures Hack (2014):
- Forensic analysis uncovered evidence linking the attack to state-sponsored actors.

Challenges in Digital Forensics

Volume of Data: The sheer volume of data generated in cyber incidents can overwhelm investigators.
Encryption and Anonymity Tools: Cybercriminals increasingly use encryption and tools like VPNs to hide their activities.
Rapidly Evolving Technologies: Forensics must keep pace with new attack techniques, devices, and platforms.

Conclusion

Digital forensics is indispensable in the fight against cybercrime. By enabling the detection, mitigation, and investigation of cyber attacks, it protects individuals, organizations, and nations from significant harm. Its application in both cybersecurity and criminal investigations underscores its critical role in ensuring justice and security in the digital age.

;;discuss for PG students : digital forensic investigation process : identification , preservation, collection, analysis

Digital Forensic Investigation Process

The digital forensic investigation process is a systematic methodology used to uncover, analyze, and preserve digital evidence in a manner that maintains its integrity and ensures its admissibility in court. The process consists of several key phases: identification, preservation, collection, and analysis. Each phase has specific goals and best practices essential for a successful investigation.

1. Identification

This is the initial step in the digital forensic process where potential evidence is identified. The primary goal is to determine what digital assets (devices, systems, or data) are relevant to the investigation.

Objectives:

Recognize Evidence Sources:
- Identify devices (e.g., computers, smartphones, servers, external drives).
- Identify data sources (e.g., cloud storage, emails, social media, logs).
Determine the Scope:
- Establish the extent of the investigation based on the case details.
- Focus on specific files, folders, or devices relevant to the incident.
Assess the Situation:
- Understand the type of incident (e.g., cybercrime, fraud, or hacking).
- Evaluate the environment (networked systems, standalone devices, or mixed).

Challenges:

Increasing diversity of devices (IoT, smartphones, cloud systems).
Identifying encrypted or hidden evidence.

2. Preservation

Once evidence is identified, it must be preserved to maintain its integrity. Preservation ensures the evidence remains unchanged and tamper-proof throughout the investigation process.

Objectives:

Isolate the Evidence:
- Prevent tampering by disconnecting the device from networks or users.
- Use write-blocking tools to protect data during imaging.
Create Forensic Copies:
- Generate bit-by-bit copies (forensic images) of digital evidence.
- Store these copies securely for analysis.
Document the Process:
- Maintain a chain of custody to document every action taken with the evidence.
- Record metadata such as timestamps, device details, and hash values.

Tools and Techniques:

Write blockers, imaging tools (e.g., EnCase, FTK Imager), and hashing algorithms (e.g., MD5, SHA-256).

Challenges:

Live systems require careful handling to avoid altering volatile data (e.g., RAM).
Ensuring cloud data is preserved without altering metadata.

3. Collection

In this phase, digital evidence is gathered from identified sources. The collection must be systematic and compliant with legal standards to ensure its admissibility in court.

Objectives:

Retrieve All Relevant Data:
- Acquire data from devices, logs, cloud storage, emails, and networks.
- Recover deleted or hidden files using forensic tools.
Ensure Completeness:
- Capture volatile data (RAM, running processes) before shutting down systems.
- Collect supporting artifacts (logs, timestamps, geolocation data).
Minimize Contamination:
- Use validated tools and methods to avoid altering evidence during collection.

Tools and Techniques:

Tools like Cellebrite (mobile forensics), X-Ways Forensics, and Wireshark (network traffic analysis).
Techniques like live acquisition (RAM dumps) and recovery of deleted data.

Challenges:

Remote data collection from cloud platforms or distributed systems.
Handling encrypted or password-protected files.

4. Analysis

Analysis involves examining the collected evidence to uncover patterns, identify suspects, and reconstruct events related to the crime.

Objectives:

Data Examination:
- Analyze files, emails, logs, and metadata for relevant information.
- Identify patterns such as IP addresses, timestamps, or access logs.
Reconstruct Events:
- Develop a timeline of activities (e.g., logins, file access, communication).
- Establish links between suspects, devices, and incidents.
Interpret Results:
- Determine the cause, method, and impact of the incident.
- Link evidence to criminal intent (e.g., emails showing premeditation).

Tools and Techniques:

File and Disk Analysis:
- Tools like Autopsy and EnCase.
Log Analysis:
- Examine system and network logs for traces of attacks.
Timeline Analysis:
- Use timestamps to correlate events across devices.
Data Recovery:
- Recover fragmented or deleted data using forensic software.

Challenges:

Analyzing large datasets requires advanced tools and expertise.
Differentiating between legitimate and malicious activity.

Legal and Ethical Considerations

At every stage, investigators must adhere to legal and ethical guidelines:

Legal Compliance:
- Follow laws related to privacy, data access, and evidence handling.
- Ensure proper warrants and permissions are obtained.
Chain of Custody:
- Maintain detailed documentation to ensure evidence is admissible in court.
Ethical Practices:
- Avoid bias and ensure objective analysis.

Conclusion

The digital forensic investigation process—identification, preservation, collection, and analysis—is foundational to solving cybercrimes and presenting credible evidence in court. Mastery of these phases ensures a systematic and legally sound investigation, helping uncover the truth and hold perpetrators accountable.

discuss for PG students : phases of cyber incident response and tolls

Phases of Cyber Incident Response and Tools

Cyber incident response is a structured process designed to identify, contain, and mitigate the effects of cybersecurity incidents. It consists of multiple phases, each with specific objectives and corresponding tools to ensure a swift and effective resolution of the incident.

1. Preparation

The preparation phase focuses on readiness to handle potential incidents through policies, training, and tools.

Objectives:

Develop Incident Response Plans:
- Define roles, responsibilities, and procedures.
Set Up Infrastructure:
- Implement monitoring tools, firewalls, and security protocols.
Train Staff:
- Conduct cybersecurity awareness programs and mock drills.

Tools:

Security Information and Event Management (SIEM):
- Tools like Splunk, QRadar, and ArcSight monitor and log security events.
Threat Intelligence Platforms:
- MISP (Malware Information Sharing Platform), Recorded Future.
Vulnerability Scanners:
- Nessus, OpenVAS for identifying vulnerabilities.

2. Identification

This phase involves detecting and verifying the occurrence of a cybersecurity incident.

Objectives:

Detect Threats:
- Monitor systems for anomalies, unauthorized access, or malicious activities.
Verify Incidents:
- Differentiate between false positives and genuine threats.
Assess Scope and Impact:
- Determine affected systems and data.

Tools:

Intrusion Detection and Prevention Systems (IDPS):
- Snort, Suricata for detecting malicious traffic.
Log Analysis Tools:
- ELK Stack (Elasticsearch, Logstash, Kibana), Graylog.
Endpoint Detection and Response (EDR):
- CrowdStrike Falcon, Carbon Black.

3. Containment

Containment focuses on limiting the spread of the incident to prevent further damage.

Objectives:

Isolate Affected Systems:
- Disconnect compromised systems from the network.
Implement Temporary Fixes:
- Block malicious IPs, disable user accounts, or change passwords.
Preserve Evidence:
- Document the affected systems and capture volatile data.

Tools:

Firewall Configuration Tools:
- pfSense, Cisco ASA.
Network Segmentation Tools:
- VLANs, SDN solutions.
Forensic Imaging Tools:
- FTK Imager, EnCase for capturing system images.

4. Eradication

This phase involves removing the root cause of the incident and ensuring systems are clean.

Objectives:

Eliminate Threats:
- Remove malware, patch vulnerabilities, and close backdoors.
Strengthen Security:
- Apply updates, implement additional security measures, and review configurations.

Tools:

Malware Removal Tools:
- Malwarebytes, Sophos Clean.
Patch Management Tools:
- WSUS, ManageEngine Patch Manager Plus.
Threat Hunting Platforms:
- ThreatHunter, Velociraptor.

5. Recovery

Recovery focuses on restoring affected systems and ensuring normal business operations.

Objectives:

Restore Operations:
- Rebuild systems from backups, test functionality, and re-integrate into the network.
Monitor Systems:
- Ensure no remnants of the threat remain.

Tools:

Backup and Restore Solutions:
- Acronis, Veeam Backup & Replication.
System Monitoring Tools:
- Nagios, Zabbix.
Penetration Testing Tools:
- Metasploit, Nessus for post-recovery validation.

6. Lessons Learned

This phase focuses on analyzing the incident to prevent recurrence and improve response strategies.

Objectives:

Conduct a Post-Mortem Analysis:
- Review the timeline, actions taken, and areas for improvement.
Update Response Plans:
- Integrate findings into incident response policies.
Share Knowledge:
- Inform stakeholders and industry peers (where applicable).

Tools:

Incident Documentation Tools:
- Jira, Confluence for recording details.
Reporting Tools:
- Tableau, Power BI for visualizing trends.
Threat Intelligence Sharing Platforms:
- ISACs, VirusTotal.

Conclusion

The cyber incident response process ensures a methodical approach to managing and mitigating cyber threats. With proper preparation, efficient use of tools, and continuous learning, organizations can significantly reduce the impact of cyber incidents and enhance their cybersecurity posture.

DISCUSS FOR PG STUDENTS : DIGITAL EVIDENCE, SCOPE, NEED, TYPES,; KEU ELEMENTS OF DIGITAL FORENSICS; CATEGORIES OF DIGITAL EVIDENCE; HEAR SAY EVIDENCE ; INTERNATIONAL BEST PRACTICES OF DIGITAL EVIDENCE; CHAIN OF CUSTODY; ANALYSE TAX EVASION CASES; DATA HIDING TECHNIQUES; OWNERSHIP DETERMINATION; FRAUD ANALYSIS USING BENFORD THEOREM

Digital Evidence: Scope, Need, and Types

Digital Evidence

Digital evidence refers to any information or data stored or transmitted in a digital format that can be used in a court of law to support or refute a claim in criminal or civil investigations.

Scope of Digital Evidence

Wide Applicability:
- Used in cybercrimes, financial fraud, terrorism, corporate espionage, and intellectual property theft.
Cross-Border Nature:
- Digital evidence often involves international jurisdictions.
Technological Advancements:
- Emerging fields like IoT and blockchain widen the scope of digital evidence.

Need for Digital Evidence

Growing Cybercrimes:
- Increasing dependence on technology necessitates digital forensics in investigations.
Critical in Modern Crimes:
- Fraud, identity theft, and hacking cases heavily rely on digital evidence.
Legal and Judicial Use:
- Provides reliable and admissible proof in court.

Types of Digital Evidence

Computer Systems Evidence:
- Hard drives, SSDs, files, emails, and logs.
Mobile Device Evidence:
- Texts, call logs, GPS data, and app activity.
Network Evidence:
- Packet captures, server logs, and firewall data.
Cloud Evidence:
- Data stored on Google Drive, Dropbox, or AWS.
Social Media Evidence:
- Posts, messages, and user activities on platforms like Facebook and Instagram.
IoT Evidence:
- Smart home devices, wearables, and connected appliances.

Key Elements of Digital Forensics

Identification:
- Recognize potential sources of digital evidence.
Preservation:
- Ensure integrity and prevent tampering using tools like write-blockers.
Collection:
- Extract data using imaging tools.
Analysis:
- Decode and interpret evidence using specialized software.
Presentation:
- Present findings in a court-admissible format.

Categories of Digital Evidence

Volatile Evidence:
- Data in RAM or cache, which disappears when the device is turned off.
Non-Volatile Evidence:
- Data stored on hard drives, SSDs, or backup systems.
Live Evidence:
- Data captured from a running system (e.g., network traffic).
Metadata Evidence:
- Details like timestamps, file ownership, and geolocation.

Hearsay Evidence

Hearsay evidence refers to statements not made during the trial but presented to prove the truth of the matter asserted.

Digital Hearsay: Social media posts, chat messages, or emails may fall under hearsay unless authenticated.
Challenges in Digital Context: Proving authorship and ensuring data integrity.
Admissibility: Subject to legal provisions like exceptions for records kept in the ordinary course of business.

International Best Practices for Digital Evidence

Adherence to Standards:
- Follow ISO/IEC 27037 for digital evidence handling.
Integrity Maintenance:
- Use hashing algorithms to ensure data authenticity.
Cross-Jurisdictional Collaboration:
- Utilize frameworks like the Budapest Convention on Cybercrime.
Documentation:
- Maintain detailed logs of evidence handling.
Specialized Training:
- Equip investigators with technical expertise in digital forensics.

Chain of Custody

Definition:
- Documentation process that tracks evidence from collection to presentation in court.
Components:
- Details of evidence acquisition, transportation, storage, and analysis.
Importance:
- Ensures admissibility and prevents tampering allegations.
Tools:
- Chain-of-custody forms, barcoding systems, and digital logs.

Analyzing Tax Evasion Cases

Digital Clues:
- Unreported income, altered invoices, or undeclared assets.
Techniques:
- Forensic accounting, document analysis, and financial transaction tracing.
Tools:
- QuickBooks Audit Trail, IDEA, or ACL for anomaly detection.
Case Study:
- Use data visualization to highlight discrepancies in declared vs. actual revenues.

Data Hiding Techniques

Steganography:
- Concealing data within images, videos, or audio files.
Encryption:
- Encoding data to prevent unauthorized access.
Alternate Data Streams (ADS):
- Hiding data in NTFS file systems.
Obfuscation:
- Masking data using complex naming or file structures.

Ownership Determination

Analyzing Metadata:
- Identify file creators and modification history.
User Account Forensics:
- Correlate activities with specific user accounts.
Device Ownership:
- Match hardware identifiers like MAC addresses to users.

Fraud Analysis Using Benford’s Theorem

Benford’s Theorem:
- Statistical principle stating that in naturally occurring datasets, lower digits (1-3) appear more frequently as leading digits.
Applications:
- Detect manipulated financial records or inflated figures.
Steps in Analysis:
- Extract numerical data, apply Benford’s distribution, and identify anomalies.
Tools:
- Excel, MATLAB, or specialized forensic software.

Conclusion

Digital forensics plays a critical role in modern investigations by addressing the complexities of cybercrime, fraud, and digital evidence. A structured approach involving key elements, best practices, and analytical tools ensures accurate and admissible findings, aiding law enforcement and legal professionals effectively.

DISCUSS DIGITAL EVIDENCE AND DIGITAL FORENSICS FOR PG STUDENTS: TESTS LIKE DATA PROFILING TESTS; STEPS FOR SEIZURE BY TAX INVESTIGATORS; BHARATIYA SAKSHYA ADHINIYAM ON DIGITAL EVIDENCE LIKE SEC 2(E), 57,58, INFORMATION TECHNOLOGY ACT SEC 4, SEC 5, SEC 75A, BANKERS BOOK EVIDENCE ACT; IPR PROTECTION; DATA IMPLICATIONS FOR TAX AUTHORITIES ; SHORTCOMINGS OF CURRENT METHODS OF INVESTIGATION BY TAX AUTHORITIES ;

Digital Evidence and Digital Forensics

Digital evidence encompasses any data stored or transmitted in digital form that can establish facts in a legal case. Digital forensics involves the identification, preservation, extraction, and analysis of this evidence.

Tests Like Data Profiling Tests

Data Profiling Tests

Definition:
- Analytical tests to evaluate and understand the structure, quality, and content of data.
Purpose:
- Identify anomalies or patterns in financial records, logs, or transactions.
Key Tests:
- Metadata Analysis: Examine file properties like timestamps and user activity.
- Outlier Detection: Highlight unusual patterns in datasets, such as unexpected spikes in transactions.
- Benford’s Law: Detect irregularities in numeric data distributions.

Steps for Seizure by Tax Investigators

Preparation:
- Obtain legal authorization for search and seizure.
- Plan logistics, including tools and personnel.
On-Site Actions:
- Secure the premises and restrict access.
- Identify devices with potential evidence (e.g., laptops, mobile phones, servers).
Imaging:
- Use write-blockers to create bit-by-bit copies of data.
- Document the imaging process with timestamps and hash values.
Labeling and Packaging:
- Properly tag devices and storage media.
- Seal and transport evidence to a secure facility.
Documentation:
- Maintain a chain-of-custody record.
- Prepare an inventory of seized items.
Analysis:
- Conduct forensic analysis to uncover hidden assets, tax evasion schemes, or fraudulent activities.

Relevant Laws Governing Digital Evidence

Bharatiya Sakshya Adhiniyam (Indian Evidence Act, 1872)

Section 2(e):
- Defines evidence to include electronic records produced for court inspection.
Section 57:
- Courts may take judicial notice of certain facts, including those supported by digital evidence like emails or digital signatures.
Section 58:
- Facts admitted in court do not require further proof, which includes digital records.

Information Technology Act, 2000

Section 4:
- Legal recognition of electronic records.
Section 5:
- Legal recognition of digital signatures for authentication.
Section 75A:
- Extra-territorial application of the IT Act, ensuring offenses with cross-border elements are prosecutable.

Bankers' Books Evidence Act, 1891

Section 2A:
- Recognizes digital formats of bank records as valid evidence.
Admissibility Requirements:
- The record must be authenticated by a responsible bank officer.

Intellectual Property Rights (IPR) Protection in Digital Forensics

Digital Piracy:
- Forensics can trace unauthorized use or duplication of copyrighted material.
Trademark Violations:
- Analysis of online marketplaces and digital media for counterfeit goods.
Copyright Infringement:
- Examination of metadata and logs to establish unauthorized usage.
Patent Disputes:
- Analyze reverse engineering attempts and trade secret theft.

Data Implications for Tax Authorities

Detection of Tax Evasion:
- Analyze financial transactions, digital invoices, and offshore accounts.
Automation and AI:
- Utilize tools to detect anomalies in GST filings, TDS, or income tax declarations.
Big Data Analytics:
- Extract insights from structured and unstructured data.

Shortcomings of Current Investigation Methods by Tax Authorities

Limited Technical Expertise:
- Lack of trained personnel to analyze complex digital evidence.
Inefficient Tools:
- Reliance on outdated technology hampers deep data analysis.
Jurisdictional Challenges:
- Difficulty accessing offshore digital records or cloud-stored evidence.
Volume of Data:
- Managing and analyzing large volumes of financial and transactional data.
Chain of Custody Issues:
- Poor documentation or mishandling of evidence can render it inadmissible.

Recommendations for Improvement

Training and Capacity Building:
- Regular training in digital forensics and advanced data analysis techniques.
Advanced Tools:
- Adoption of forensic tools like EnCase, FTK, and cloud forensics platforms.
Collaborative Frameworks:
- Develop agreements with foreign jurisdictions for seamless access to cross-border data.
Audit Trails:
- Maintain robust documentation for all evidence handling processes.

Conclusion

Digital evidence and forensics are indispensable in modern investigations. Tax authorities, while leveraging digital tools, must also address challenges such as technical expertise and cross-border complexities. Strengthening legal frameworks and adopting international best practices will ensure that digital evidence is handled effectively and remains admissible in court.

CASE SUMMARIES : NCT DELHI V NAVJOT SANDHU [AFSAN GURU] 2005 SC CASE ;

Case Summary: NCT of Delhi v. Navjot Sandhu (Afzal Guru Case), 2005 SC

Citation:

(2005) 11 SCC 600

Facts of the Case:

Incident:
- On December 13, 2001, five heavily armed terrorists attacked the Indian Parliament.
- Security personnel engaged the attackers, leading to the deaths of nine individuals (including the terrorists).
Investigation:
- The Delhi Police traced the attack to a conspiracy involving the terrorist group Jaish-e-Mohammed.
- Key accused:
  - Mohammad Afzal Guru (main conspirator).
  - Navjot Sandhu (Afsan Guru): Wife of co-accused Shaukat Guru.
  - Shaukat Hussain Guru (Afzal's accomplice).
  - SAR Geelani: Lecturer alleged to have facilitated the attack.
Prosecution Allegations:
- Afzal Guru and others provided logistical support, finances, and a base for the terrorists.
- Mobile call records and other electronic evidence linked the accused to the attack.

Issues Before the Court:

Admissibility of Electronic Evidence:
- The trial court relied on intercepted phone call records and emails to establish conspiracy.
Involvement of the Accused:
- Whether each accused was culpable beyond reasonable doubt.

Key Arguments:

Prosecution:
- Evidence (including confessions, phone records, and recoveries) directly linked the accused to the attack.
- Claimed Navjot Sandhu was aware of and complicit in the conspiracy.
Defense:
- Alleged evidence was fabricated and obtained under duress.
- Navjot Sandhu denied knowledge of her husband's involvement.

Supreme Court Judgment:

On Afzal Guru:
- Found guilty of waging war against the state, conspiracy, and terrorism.
- Sentenced to death penalty.
- The Court justified the sentence, citing the "rarest of rare" doctrine.
On Shaukat Guru:
- Convicted for conspiracy but given a reduced sentence of 10 years' rigorous imprisonment (from life imprisonment).
On Navjot Sandhu:
- Acquitted of all charges due to lack of evidence proving her active involvement.
On SAR Geelani:
- Acquitted due to insufficient evidence linking him to the conspiracy.

Significance of the Case:

Electronic Evidence (Call Records and Emails):
- Supreme Court emphasized the need for proper authentication of electronic evidence under the Indian Evidence Act.
Impact on Anti-Terror Laws:
- Highlighted the stringent application of anti-terror laws, ensuring fairness while dealing with suspects.
Judicial Approach:
- Balanced approach: Convicted the guilty while acquitting those with insufficient evidence.
Public Reaction:
- The case garnered widespread media attention and polarized opinions.
- Afzal Guru's execution in 2013 led to debates about the fairness of trials in terrorism-related cases.

Legal Precedents Set:

Admissibility of Digital Evidence:
- Established the principle that electronic evidence must comply with procedural safeguards.
Rarest of Rare Doctrine:
- Reaffirmed the application of the doctrine in awarding the death penalty.

Criticism:

Perceived Bias:
- Concerns about fairness in Afzal Guru’s trial.
Procedural Gaps:
- Allegations of lapses in evidence collection and presentation.

Conclusion:

The NCT of Delhi v. Navjot Sandhu case remains a landmark judgment in the realm of terrorism law, electronic evidence, and criminal justice. It underscores the importance of due process and judicial prudence in cases involving national security.

CASE SUMMARY: MANU SHARMA V NCT OF DELHI 2010

Case Summary: Manu Sharma v. NCT of Delhi, 2010 SC

Citation:

(2010) 6 SCC 1

Facts of the Case:

Incident:
- On the night of April 29-30, 1999, Jessica Lal, a model and part-time bartender, was shot dead at Tamarind Court, a restaurant owned by socialite Bina Ramani in South Delhi.
- Jessica had refused to serve drinks to Manu Sharma (son of a wealthy politician), leading to an altercation, after which he shot her.
Key Witnesses:
- Several people were present, but most turned hostile during the trial.
- The prosecution relied on testimonies, ballistic reports, and circumstantial evidence.
Initial Trial and Acquittal:
- The trial court acquitted Manu Sharma in February 2006 due to lack of evidence and witnesses turning hostile.
- This sparked public outrage and led to significant media and public campaigns demanding justice.
Appeal:
- The High Court overturned the trial court’s decision, convicting Manu Sharma of murder and sentencing him to life imprisonment.
- The case then went to the Supreme Court on appeal.

Issues Before the Supreme Court:

Whether Manu Sharma’s conviction by the High Court was legally justified.
The role of hostile witnesses and circumstantial evidence in the conviction.
The influence of media trials on the judicial process.

Supreme Court Judgment:

The Supreme Court upheld Manu Sharma’s conviction for the murder of Jessica Lal and confirmed the sentence of life imprisonment.

Key Findings and Observations:

Credibility of Witnesses:
- The Court scrutinized the evidence of witnesses who turned hostile and concluded they did so due to external pressure.
- It relied on corroborative evidence and testimonies from reliable witnesses, including Bina Ramani and others.
Ballistic Evidence:
- The ballistic report established that the bullet that killed Jessica matched the firearm used by Manu Sharma.
- This was critical in linking Manu Sharma to the crime.
Motive and Circumstantial Evidence:
- The refusal to serve drinks was established as the motive for the crime.
- Circumstantial evidence strongly pointed to Manu Sharma’s guilt.
Role of Media Trials:
- The Court noted that while media campaigns highlighted the miscarriage of justice, judgments must be based solely on evidence and law.
- It cautioned against undue media influence in the judicial process.
Justice for Victims:
- The Court emphasized the importance of justice for victims and the need to ensure that wealthy or influential individuals are not above the law.

Significance of the Case:

Hostile Witnesses:
- Highlighted the problem of witnesses turning hostile in criminal cases due to intimidation or bribery.
- Reinforced the importance of independent corroborative evidence.
Judicial Independence:
- Reaffirmed the judiciary’s role in ensuring justice, free from public or media pressure.
Victim-Centric Approach:
- Demonstrated the judiciary’s commitment to protecting the rights of victims and ensuring accountability for perpetrators.
Impact on Criminal Justice System:
- The case became a symbol of judicial integrity and public accountability, especially in high-profile cases involving influential individuals.

Criticism:

Delay in Justice:
- The case highlighted systemic delays in the criminal justice system, with the final verdict coming more than a decade after the crime.
Hostile Witness Problem:
- Raised concerns about the vulnerability of the justice system to external pressures, particularly in cases involving influential accused persons.

Conclusion:

The Manu Sharma v. NCT of Delhi case is a landmark in Indian criminal jurisprudence. It reaffirmed the principle that justice cannot be subverted by money or power and emphasized the role of the judiciary in upholding the rule of law. The case also serves as a reminder of the importance of strengthening witness protection mechanisms and addressing systemic issues in the criminal justice process.

CASE SUMMARY: UNNIKRISHNAN V STATE REP BY INSPECTOR OF POLICE, 2011

Case Summary: Unnikrishnan v. State Rep. by Inspector of Police, 2011 SC

Citation:

(2011) 10 SCC 259

Facts of the Case:

Incident:
- The case revolves around the murder of an industrialist (the deceased) who was allegedly kidnapped for ransom and later killed.
- The appellant, Unnikrishnan, was one of the accused in the crime, along with others.
- The prosecution alleged that the motive was financial gain through ransom.
Prosecution's Evidence:
- The case relied heavily on circumstantial evidence, including call records, the recovery of the deceased’s body, and testimonies of witnesses.
- Evidence included the recovery of a ransom note and weapons used in the crime.
Trial Court:
- The trial court convicted Unnikrishnan and the co-accused based on circumstantial evidence and sentenced them to life imprisonment.
High Court:
- The High Court upheld the trial court's conviction, leading to the appeal in the Supreme Court.

Issues Before the Supreme Court:

Whether the chain of circumstantial evidence was complete and pointed unerringly to the guilt of the accused.
Whether the trial court and High Court erred in convicting the accused based on the evidence presented.
The admissibility and reliability of the evidence, including confessions and recoveries.

Supreme Court Judgment:

The Supreme Court dismissed the appeal and upheld the conviction and sentence of life imprisonment for Unnikrishnan.

Key Findings and Observations:

Circumstantial Evidence:
- The Court reiterated that in cases based on circumstantial evidence, the chain of evidence must be complete and exclude every hypothesis except the guilt of the accused.
- In this case, the Court found the chain of evidence to be complete.
Motive:
- The financial motive for the crime was clearly established through evidence such as ransom demands and the financial condition of the accused.
Witness Testimonies:
- The testimonies of witnesses, though circumstantial, were found to be consistent and credible.
- The Court noted that minor contradictions in witness statements do not necessarily weaken the prosecution’s case.
Call Records and Recovery:
- Mobile phone records played a crucial role in tracing the movements of the accused and establishing their involvement in the crime.
- The recovery of the ransom note and weapons corroborated the prosecution’s narrative.
Judicial Principles:
- The Court emphasized that the presumption of innocence does not mean ignoring credible evidence presented by the prosecution.
- The burden of proof lies on the prosecution, and in this case, it was discharged effectively.

Significance of the Case:

Role of Circumstantial Evidence:
- The case underscores the importance of a cohesive and unbroken chain of circumstantial evidence in securing convictions in complex criminal cases.
Technological Evidence:
- It highlighted the growing reliance on technological evidence, such as call records, in modern investigations.
Motive and Intent:
- Reinforced the principle that establishing motive strengthens the prosecution's case, even when direct evidence is lacking.
Judicial Integrity:
- The judgment reaffirmed the judiciary’s role in ensuring that justice is served, especially in cases involving heinous crimes like kidnapping and murder.

Conclusion:

The Unnikrishnan v. State Rep. by Inspector of Police case is significant for its reliance on circumstantial evidence to secure a conviction. It demonstrates the effectiveness of technological evidence and corroborative testimonies in ensuring justice. This case also serves as a precedent for handling cases where direct evidence is unavailable, emphasizing the importance of a complete and consistent chain of circumstances.

1/2

CASE SUMMARY: KONNADAN ABDUL GAFFOOR V STATE FO KERALA 2012

Case Summary: Konnadan Abdul Gaffoor v. State of Kerala, 2012 SC

Citation:

(2012) 8 SCC 500

Facts of the Case:

Background:
- The case involves the appellant, Konnadan Abdul Gaffoor, who was convicted under the Narcotic Drugs and Psychotropic Substances Act, 1985 (NDPS Act) for possessing a significant quantity of cannabis.
- The seizure occurred during a routine check by law enforcement officers in Kerala.
- The appellant was charged under Sections 20(b)(ii)(C) and 55 of the NDPS Act for possession of a commercial quantity of cannabis and failing to comply with procedural safeguards.
Prosecution’s Argument:
- The prosecution argued that the appellant was caught red-handed and that all statutory procedures, including inventory, sampling, and seizure, were complied with.
- The evidence included testimonies of police officers, the recovery report, and forensic confirmation of the substance being cannabis.
Defense Argument:
- The defense claimed that procedural lapses in the investigation violated the appellant’s rights under the NDPS Act.
- They also argued that the prosecution failed to establish an unbroken chain of custody for the seized material.

Issues Before the Supreme Court:

Were the procedural safeguards under the NDPS Act, particularly Sections 42, 50, and 55, followed during the search and seizure?
Was the chain of custody for the seized material properly maintained?
Did the prosecution prove its case beyond a reasonable doubt?

Supreme Court Judgment:

The Supreme Court upheld the conviction of Konnadan Abdul Gaffoor, dismissing his appeal.

Key Observations and Findings:

Procedural Safeguards:
- The Court emphasized the importance of strict compliance with procedural safeguards under the NDPS Act, given the severity of the penalties.
- In this case, the Court found that the authorities had substantially complied with Sections 42 (power of entry, search, seizure, and arrest), 50 (conditions under which search of persons shall be conducted), and 55 (responsibility of the officer in charge of the police station).
Chain of Custody:
- The Court held that the prosecution successfully established an unbroken chain of custody of the seized material, from seizure to forensic examination.
- This was critical in linking the appellant to the crime.
Quantity of Cannabis:
- The quantity of cannabis seized exceeded the threshold for “commercial quantity” under the NDPS Act, leading to a more stringent punishment.
Burden of Proof:
- The Court reiterated that once the prosecution establishes prima facie evidence, the burden shifts to the accused to rebut the presumption of guilt under Section 35 of the NDPS Act.
- The appellant failed to provide any plausible explanation or evidence in his defense.

Significance of the Case:

Strict Compliance with NDPS Act:
- This case underscores the judiciary's insistence on adherence to procedural safeguards in NDPS cases, ensuring fairness despite the stringent nature of the law.
Chain of Custody:
- The judgment highlights the importance of maintaining a clear and documented chain of custody to uphold the integrity of evidence.
Burden on the Accused:
- The case reiterates that once the prosecution presents a prima facie case, the accused must provide a convincing rebuttal under the NDPS Act.

Conclusion:

In Konnadan Abdul Gaffoor v. State of Kerala, the Supreme Court upheld the conviction due to substantial compliance with the NDPS Act's safeguards and a robust chain of custody for the evidence. The judgment reinforces the principle that while procedural compliance is essential, minor deviations that do not prejudice the accused's rights cannot invalidate a conviction. This case serves as a benchmark for NDPS investigations and prosecutions.

CASE SUMMARY: ANVAR PV V PK BASHEER & OTHERS , 2014

Case Summary: Anvar P.V. v. P.K. Basheer & Others, 2014

Citation:

(2014) 10 SCC 473

Facts of the Case:

Background:
- The case arose out of an election dispute under the Representation of the People Act, 1951.
- Anvar P.V., the appellant, contested the election of P.K. Basheer, alleging malpractices and falsifications during the election campaign.
- The appellant relied on electronic records, including CDs containing speeches by the respondent, to substantiate his claims of election misconduct.
Trial Court Findings:
- The trial court admitted the CDs as evidence without requiring a certificate under Section 65B of the Indian Evidence Act, 1872, which governs the admissibility of electronic records.
- Based on this evidence, the election of P.K. Basheer was declared void.
High Court Ruling:
- The Kerala High Court overturned the trial court’s decision, stating that the CDs could not be admitted without a proper Section 65B certificate.
- The appellant challenged this decision in the Supreme Court.

Key Legal Issues:

Admissibility of Electronic Evidence:
- Whether electronic records, such as CDs, are admissible in court without compliance with Section 65B of the Indian Evidence Act.
Section 65B Requirements:
- Whether a certificate under Section 65B is mandatory to prove the authenticity and integrity of electronic records.

Supreme Court Judgment:

The Supreme Court dismissed the appeal, upholding the High Court’s decision.

Key Observations and Findings:

Mandatory Compliance with Section 65B:
- The Court held that Section 65B of the Indian Evidence Act is a mandatory provision for the admissibility of electronic evidence.
- Any electronic record, such as CDs, emails, or data from devices, must be accompanied by a certificate attesting to its authenticity and integrity.
Primary Evidence vs. Secondary Evidence:
- The Court clarified that electronic records are considered secondary evidence unless the original device or data is produced in court.
- To admit such secondary evidence, compliance with Section 65B is non-negotiable.
Certificate under Section 65B:
- The certificate should contain details of the device, the method of production, and confirmation of the data’s integrity.
- It must be signed by a person in control of the device or system producing the electronic record.
Impact of Non-Compliance:
- In the absence of a Section 65B certificate, electronic evidence cannot be admitted.
- The Court emphasized that this procedural safeguard ensures the reliability of electronic records.

Significance of the Case:

Admissibility Standards for Electronic Evidence:
- This case is a landmark judgment that set a precedent for strict compliance with Section 65B, ensuring the integrity and authenticity of electronic evidence in judicial proceedings.
Impact on Future Cases:
- The ruling significantly affects cases involving cybercrime, digital forensics, and electronic communications, requiring meticulous adherence to evidentiary standards.
Role of Technology in Law:
- The judgment acknowledges the increasing reliance on digital evidence and establishes clear procedural safeguards to prevent misuse or tampering.

Criticism and Later Developments:

The judgment has been criticized for its rigidity, as it imposes strict procedural requirements that may delay justice.
Subsequent cases, such as Arjun Panditrao Khotkar v. Kailash Kushanrao Gorantyal (2020), have revisited the principles laid down in this case, offering clarifications on exceptional situations where the certificate may not be mandatory.

Conclusion:

In Anvar P.V. v. P.K. Basheer, the Supreme Court firmly established the mandatory nature of Section 65B for the admissibility of electronic evidence. This landmark judgment ensures the authenticity and reliability of digital records, playing a crucial role in cases where electronic evidence forms the backbone of the prosecution or defense. It remains a cornerstone for interpreting the Indian Evidence Act in the digital era.

CASE SUMMARY: Case Summary: Anvar P.V. v. P.K. Basheer & Others, 2014 Citation: (2014) 10 SCC 473 Facts of the Case: Background: The case arose out of an election dispute under the Representation of the People Act, 1951. Anvar P.V., the appellant, contested the election of P.K. Basheer, alleging malpractices and falsifications during the election campaign. The appellant relied on electronic records, including CDs containing speeches by the respondent, to substantiate his claims of election misconduct. Trial Court Findings: The trial court admitted the CDs as evidence without requiring a certificate under Section 65B of the Indian Evidence Act, 1872, which governs the admissibility of electronic records. Based on this evidence, the election of P.K. Basheer was declared void. High Court Ruling: The Kerala High Court overturned the trial court’s decision, stating that the CDs could not be admitted without a proper Section 65B certificate. The appellant challenged this decision in the Supreme Court. Key Legal Issues: Admissibility of Electronic Evidence: Whether electronic records, such as CDs, are admissible in court without compliance with Section 65B of the Indian Evidence Act. Section 65B Requirements: Whether a certificate under Section 65B is mandatory to prove the authenticity and integrity of electronic records. Supreme Court Judgment: The Supreme Court dismissed the appeal, upholding the High Court’s decision. Key Observations and Findings: Mandatory Compliance with Section 65B: The Court held that Section 65B of the Indian Evidence Act is a mandatory provision for the admissibility of electronic evidence. Any electronic record, such as CDs, emails, or data from devices, must be accompanied by a certificate attesting to its authenticity and integrity. Primary Evidence vs. Secondary Evidence: The Court clarified that electronic records are considered secondary evidence unless the original device or data is produced in court. To admit such secondary evidence, compliance with Section 65B is non-negotiable. Certificate under Section 65B: The certificate should contain details of the device, the method of production, and confirmation of the data’s integrity. It must be signed by a person in control of the device or system producing the electronic record. Impact of Non-Compliance: In the absence of a Section 65B certificate, electronic evidence cannot be admitted. The Court emphasized that this procedural safeguard ensures the reliability of electronic records. Significance of the Case: Admissibility Standards for Electronic Evidence: This case is a landmark judgment that set a precedent for strict compliance with Section 65B, ensuring the integrity and authenticity of electronic evidence in judicial proceedings. Impact on Future Cases: The ruling significantly affects cases involving cybercrime, digital forensics, and electronic communications, requiring meticulous adherence to evidentiary standards. Role of Technology in Law: The judgment acknowledges the increasing reliance on digital evidence and establishes clear procedural safeguards to prevent misuse or tampering. Criticism and Later Developments: The judgment has been criticized for its rigidity, as it imposes strict procedural requirements that may delay justice. Subsequent cases, such as Arjun Panditrao Khotkar v. Kailash Kushanrao Gorantyal (2020), have revisited the principles laid down in this case, offering clarifications on exceptional situations where the certificate may not be mandatory. Conclusion: In Anvar P.V. v. P.K. Basheer, the Supreme Court firmly established the mandatory nature of Section 65B for the admissibility of electronic evidence. This landmark judgment ensures the authenticity and reliability of digital records, playing a crucial role in cases where electronic evidence forms the backbone of the prosecution or defense. It remains a cornerstone for interpreting the Indian Evidence Act in the digital era.

Case Summary: Anvar P.V. v. P.K. Basheer & Others, 2014

Citation:

(2014) 10 SCC 473

Facts of the Case:

Background:
The case arose from an election dispute under the Representation of the People Act, 1951.
Anvar P.V., the appellant, contested the election of P.K. Basheer, alleging election malpractices and falsifications. To support his allegations, the appellant presented electronic records, including CDs containing speeches by the respondent (P.K. Basheer).
Trial Court Findings:
The trial court admitted the CDs as evidence without requiring a certificate under Section 65B of the Indian Evidence Act, 1872. Section 65B deals with the admissibility of electronic records in court. The court accepted the CDs as valid evidence, and based on them, declared the election of P.K. Basheer void.
High Court Ruling:
The Kerala High Court reversed the trial court's decision. The High Court stated that the CDs could not be admitted as evidence without the mandatory Section 65B certificate, which authenticates the integrity of electronic evidence. The appellant then appealed the decision to the Supreme Court.

Key Legal Issues:

Admissibility of Electronic Evidence:
- Whether electronic records, such as CDs, are admissible in court without complying with Section 65B of the Indian Evidence Act, 1872.
Requirement of Section 65B Certificate:
- Whether it is mandatory to produce a certificate under Section 65B to prove the authenticity and integrity of electronic records.

Supreme Court Judgment:

The Supreme Court dismissed the appeal, agreeing with the High Court’s ruling.

Key Observations and Findings:

Mandatory Compliance with Section 65B:
- The Court emphasized that Section 65B is a mandatory requirement for the admissibility of electronic records in judicial proceedings.
- It clarified that any electronic evidence, such as CDs, emails, or data from devices, must be accompanied by a certificate that confirms the authenticity and integrity of the data.
Primary vs. Secondary Evidence:
- The Court distinguished between primary and secondary evidence.
- Electronic records are secondary evidence unless the original device or data itself is presented in court.
- To admit secondary evidence, compliance with Section 65B is non-negotiable.
Certificate under Section 65B:
- The certificate required under Section 65B must contain details of the device, the process of production, and an assurance that the data has not been tampered with.
- This certificate must be signed by a person in control of the device or system producing the electronic record.
Impact of Non-Compliance:
- The Court held that without a Section 65B certificate, electronic evidence cannot be admitted in court.
- This safeguard ensures the reliability and integrity of electronic evidence.

Significance of the Case:

Admissibility Standards for Electronic Evidence:
- The judgment established a precedent for the strict application of Section 65B, ensuring that electronic evidence meets high standards of authenticity before being accepted in court.
Impact on Future Cases:
- The ruling has significant implications for cases involving cybercrime, digital forensics, and electronic communications. It mandates meticulous adherence to evidentiary standards when electronic records are introduced.
Role of Technology in Law:
- The judgment underscores the growing reliance on digital evidence in legal proceedings and establishes procedural safeguards to prevent misuse or tampering of digital data.

Criticism and Later Developments:

Criticism:
- The decision was criticized for its rigidity in mandating the Section 65B certificate, which can create hurdles in the admission of electronic evidence. It has been argued that this might delay justice in cases involving digital records.
Later Developments:
- Subsequent cases, such as Arjun Panditrao Khotkar v. Kailash Kushanrao Gorantyal (2020), have revisited the principles laid down in Anvar P.V. v. P.K. Basheer, offering clarifications about situations where a Section 65B certificate may not be mandatory, especially when it may be impractical to produce one.

Conclusion:

In Anvar P.V. v. P.K. Basheer, the Supreme Court firmly established that Section 65B of the Indian Evidence Act is mandatory for the admissibility of electronic evidence in judicial proceedings. This landmark judgment ensures the integrity and authenticity of digital records and significantly impacts the way digital evidence is handled in the Indian legal system, especially in the context of cybercrime and election disputes. The ruling remains a critical point of reference for interpreting the Indian Evidence Act in an era increasingly dominated by electronic data.

CASE SUMMARY: TOMASO BRUNO AND ANOTHER V STATE OF UP ,2015

Case Summary: Tomaso Bruno and Another v. State of Uttar Pradesh, 2015

Citation:

(2015) 8 SCC 722

Facts of the Case:

Background:
The case arose from a murder and kidnapping investigation. Tomaso Bruno, an Italian national, and his friend, Giampaolo Mazzon, were accused of kidnapping and murdering a local man, Alok Sharma, in Uttar Pradesh, India.
The two Italians were alleged to have entered India as tourists but had subsequently been involved in a criminal conspiracy. They were accused of kidnapping Alok Sharma in order to extort money from his family. The victim was murdered, and his body was later found in a disfigured state.
After their arrest, the accused were charged under Sections 302 (murder), 364 (kidnapping), and other related provisions of the Indian Penal Code (IPC).
Trial Court Findings:
The trial court found both Tomaso Bruno and Giampaolo Mazzon guilty of murder and kidnapping and sentenced them to life imprisonment. The court based its findings on the statements of the prosecution witnesses, circumstantial evidence, and forensic analysis, including the recovery of the victim’s body.
Appeal to the Supreme Court:
The appellants (Tomaso Bruno and Giampaolo Mazzon) challenged their conviction in the Supreme Court of India, asserting that they were innocent and that the evidence against them was circumstantial.

Key Legal Issues:

Admissibility of Circumstantial Evidence:
- Whether the circumstantial evidence and forensic findings were sufficient to uphold the conviction of Tomaso Bruno and Giampaolo Mazzon.
Evaluation of Evidence in Criminal Cases:
- The standards for evaluating circumstantial evidence in a criminal trial, especially when no direct evidence like eyewitness testimony is available.
Interpreting the Role of Foreign Nationals in Criminal Cases in India:
- The legal rights and protections for foreign nationals involved in criminal cases in India, especially in terms of evidence admissibility and judicial processes.

Supreme Court Judgment:

The Supreme Court upheld the conviction of Tomaso Bruno and Giampaolo Mazzon. The Court found that the circumstantial evidence, including the forensic evidence and the recovery of the victim's body, was sufficient to establish their guilt beyond reasonable doubt.

Key Observations and Findings:

Circumstantial Evidence:
- The Supreme Court reaffirmed the principle that circumstantial evidence can be as valid as direct evidence in proving guilt. The Court relied on several pieces of circumstantial evidence, including:
  - The victim’s abduction and subsequent murder.
  - The recovery of the body, which showed signs of violent crime.
  - The testimony of witnesses and the discovery of key forensic evidence.
Test for Conviction Based on Circumstantial Evidence:
- The Court outlined the following test for convicting someone based on circumstantial evidence:
  - The circumstances must be consistent with the guilt of the accused and should exclude any hypothesis of innocence.
  - The evidence must be unequivocal and leave no room for reasonable doubt about the accused’s involvement in the crime.
Forensic Evidence:
- The forensic examination of the victim’s body provided critical information about the nature of the crime. The injuries on the body were consistent with the claims of the prosecution regarding how the victim was murdered.
Foreign Nationals in Criminal Trials:
- The Court emphasized that foreign nationals are subject to Indian law when they commit crimes within the country. The legal procedures and rights afforded to Indian citizens are equally applicable to foreigners involved in criminal cases.

Significance of the Case:

Role of Circumstantial Evidence:
- The judgment reinforced the importance of circumstantial evidence in criminal trials. It demonstrated that, even in the absence of direct evidence, circumstantial evidence can be used to prove guilt, provided it meets the necessary criteria.
Forensic Evidence in Criminal Trials:
- The case highlighted the growing importance of forensic evidence in criminal investigations, particularly in cases involving murder. The use of forensic science to link suspects to crimes or confirm the nature of the crime was pivotal in this case.
Legal Rights of Foreign Nationals:
- The case emphasized that foreign nationals must follow the legal processes in India when involved in criminal matters. While there might be concerns about jurisdiction, the Supreme Court made it clear that foreign nationals are not immune from prosecution for crimes committed in India.

Conclusion:

In Tomaso Bruno and Another v. State of Uttar Pradesh, the Supreme Court upheld the conviction of the two Italian nationals involved in the kidnapping and murder of Alok Sharma. The case was significant in demonstrating how circumstantial evidence and forensic science can be effectively used to prove guilt in a criminal trial, even when there is no direct eyewitness testimony. The ruling also reinforced the principle that foreign nationals are bound by the legal system of the country in which they commit crimes, and they are not exempt from prosecution under Indian law.

CASE SUMMARY: SHAFI MOHAMMED V STATE OF HIMACHAL PRADESH, 2018

Case Summary: Shafi Mohammed v. State of Himachal Pradesh, 2018

Citation:

(2018) 7 SCC 314

Facts of the Case:

Background:
This case involved a conviction for drug trafficking. The appellant, Shafi Mohammed, was accused of being involved in the trafficking of charas (a form of cannabis) in the state of Himachal Pradesh. He was apprehended by the police, who discovered a significant quantity of charas during a search operation.
Trial Court Findings:
The trial court convicted Shafi Mohammed under the Narcotic Drugs and Psychotropic Substances Act, 1985 (NDPS Act) for possessing and trafficking in narcotics. He was sentenced to rigorous imprisonment for the offense.
Appeal:
The appellant challenged his conviction before the Himachal Pradesh High Court, which upheld the trial court's decision. The appellant then appealed to the Supreme Court of India.

Key Legal Issues:

Admissibility of Evidence:
- Whether the evidence collected during the search and seizure operation was admissible, considering that certain procedural safeguards were not followed.
Presumption under the NDPS Act:
- Whether the presumption under Section 35 of the NDPS Act that the accused had knowledge of the narcotics in his possession applied to the case. This presumption creates a reverse burden on the accused in cases under the NDPS Act.
Credibility of Witnesses and Police Procedures:
- The appellant argued that the search was conducted illegally, and the evidence was tampered with. He contested the credibility of the police officers who conducted the search.

Supreme Court Judgment:

The Supreme Court of India upheld the conviction of Shafi Mohammed, rejecting his appeal.

Key Observations and Findings:

Presumption under Section 35 of the NDPS Act:
- The Supreme Court reiterated that under Section 35 of the NDPS Act, once the prosecution proves that the accused was in possession of the contraband, a presumption arises that the accused had knowledge of the nature of the substance.
- This places the burden of proof on the accused to rebut the presumption of knowledge, making it difficult for the defense to challenge the charge successfully unless they provide strong evidence.
Admissibility of Evidence:
- The Court examined the procedures followed during the search and seizure operation and found that the police had adhered to the relevant provisions of the NDPS Act. Therefore, the evidence collected during the operation was admissible.
- The Court also emphasized the importance of following due process under the law when carrying out searches and seizures under the NDPS Act.
Credibility of Police Officers and Witnesses:
- The appellant’s argument about tampered evidence and unreliable police testimony was dismissed. The Court stated that the testimony of the police officers conducting the search and seizure was credible and consistent with the facts of the case.
Strict Interpretation of NDPS Act:
- The judgment highlighted the stringent provisions of the NDPS Act, which were designed to curb the illegal trafficking of narcotics. The Court made it clear that courts cannot lightly interfere with convictions under the Act, especially when strong evidence supports the prosecution’s case.

Significance of the Case:

Reinforcement of the Presumption of Knowledge under NDPS Act:
- The ruling strengthened the application of Section 35 of the NDPS Act, which places a reverse burden of proof on the accused. This is an important tool in prosecuting narcotic crimes, especially where possession of illegal substances is involved.
Adherence to Search Procedures:
- The judgment also highlighted the importance of police procedure in narcotic-related offenses. It emphasized that adherence to the proper legal framework for searches, seizures, and the collection of evidence is crucial to maintaining the integrity of the judicial process.
Impact on Drug-Related Cases:
- This case reaffirmed the strict nature of drug laws in India, where the Narcotic Drugs and Psychotropic Substances Act plays a central role in prosecuting drug offenders. It also sent a clear message about the challenges for defendants trying to challenge charges under the Act.

Conclusion:

In Shafi Mohammed v. State of Himachal Pradesh, the Supreme Court upheld the conviction of the appellant for drug trafficking under the Narcotic Drugs and Psychotropic Substances Act. The Court emphasized the mandatory presumption of knowledge under Section 35 of the NDPS Act, which places the onus on the accused to prove their innocence. The case reinforced the importance of following procedural safeguards in the collection of evidence and police conduct during searches and seizures. The judgment serves as a significant ruling for future cases involving the NDPS Act and drug trafficking offenses in India.

CASE SUMMARY: ARJUN PADITRAO KHOTKAR V KAILASH KISHAN RAO, 2020

Case Summary: Arjun Panditrao Khotkar v. Kailash Kishanrao Gorantyal, 2020

Citation:

(2020) 7 SCC 1

Facts of the Case:

Background:
The case arose out of a dispute related to the admissibility of electronic records in a civil suit. The appellant, Arjun Panditrao Khotkar, filed a suit to challenge the election of Kailash Kishanrao Gorantyal, claiming that certain documents and electronic evidence should be considered to prove his case. The appellant submitted electronic records, including audio and video recordings, as part of his evidence in the trial court.
Issue:
The key issue in the case was whether the electronic evidence, in the form of audio/video recordings, could be admitted into court without complying with the requirements of Section 65B of the Indian Evidence Act, 1872. Section 65B specifically addresses the admissibility of electronic records in legal proceedings, stating that such records must be accompanied by a certificate that attests to their authenticity.

Key Legal Issues:

Admissibility of Electronic Evidence:
- Whether electronic evidence such as audio/video recordings can be admitted in court without a certificate under Section 65B of the Indian Evidence Act.
Section 65B of the Indian Evidence Act:
- Whether the certificate required by Section 65B is mandatory for the admissibility of electronic records, and if any exceptions can be made.

Supreme Court Judgment:

The Supreme Court of India delivered its judgment in favor of Arjun Panditrao Khotkar by clarifying the application of Section 65B of the Indian Evidence Act, providing some relief in certain cases.

Key Observations and Findings:

Clarification on Section 65B:
- The Supreme Court reaffirmed its earlier ruling in the case of Anvar P.V. v. P.K. Basheer (2014) that Section 65B is mandatory for the admissibility of electronic records. However, the Court observed that exceptions can apply in specific situations where the certificate under Section 65B may not be required.
The Scope of Section 65B Certificate:
- Section 65B of the Indian Evidence Act lays down that a certificate is required to confirm that the electronic record is authentic and accurately represents the data stored on the device. This certificate must be signed by a person in control of the computer or device, with details about the device and the process of creating the record.
- The Court clarified that Section 65B applies to all electronic records—emails, audio/video recordings, digital documents, etc. The certificate ensures that the evidence submitted in court is reliable and has not been tampered with.
Exceptions to the Rule of Section 65B Certificate:
- The Court clarified that while the certificate under Section 65B is generally mandatory, in some cases, secondary evidence such as electronic records could be admissible without the certificate if:
  - The original device from which the record was created can be produced, or
  - The authenticity of the record is not disputed, or
  - The contents of the electronic record are not the primary evidence being contested.
Importance of Section 65B in Ensuring Reliability:
- The Court emphasized that Section 65B serves as a protective measure to ensure the integrity and authenticity of electronic evidence in legal proceedings. Without a proper certificate, there is a risk that the evidence may be tampered or manipulated, leading to potential injustice.

Significance of the Case:

Impact on the Admissibility of Electronic Evidence:
- The judgment reaffirmed the strict requirements of Section 65B for the admissibility of electronic records and emphasized that the certificate plays a vital role in ensuring that the records presented are reliable.
Clarification on Exceptions:
- The Court's decision is significant as it offers clarity regarding the exceptions to the mandatory requirement of a Section 65B certificate. This helps to provide a balance between procedural safeguards and the practical realities of submitting electronic evidence in court.
Future Legal Implications:
- The case has important implications for future cases involving cybercrime, fraud, and digital evidence, ensuring that there is uniformity in the application of Section 65B and the process for the admission of electronic records.
Practical Relief to Litigants:
- The ruling provides practical relief to litigants and legal practitioners by clarifying how electronic records should be treated under the law, especially in situations where compliance with Section 65B may not always be feasible due to technical limitations.

Conclusion:

In Arjun Panditrao Khotkar v. Kailash Kishanrao Gorantyal, the Supreme Court offered important clarifications regarding the mandatory compliance with Section 65B of the Indian Evidence Act for the admissibility of electronic evidence. While the Court upheld the principle that a certificate is generally required, it also made allowances for certain exceptions where the certificate may not be necessary. This judgment ensures a more practical approach to the admissibility of digital evidence in India, balancing legal standards with technological realities in the digital age.

DISCUSS FOR THE BENEFIT OF PG STUDENTS : BENEFITS OF DIGITAL EVIDENCE IN LEGAL PROCEEDINGS IN INDIA

Benefits of Digital Evidence in Legal Proceedings in India

Digital evidence has become an integral part of modern legal proceedings in India. With the rise of technology and the internet, digital evidence is increasingly playing a crucial role in investigations, especially in cybercrimes, fraud cases, election disputes, contract disputes, and even traditional crimes where digital footprints exist. Digital evidence includes any data stored in digital form, such as emails, text messages, documents, images, videos, logs, metadata, and more.

The following outlines the key benefits of using digital evidence in legal proceedings in India:

1. Enhances the Accuracy and Reliability of Evidence:

Unbiased and Objective Evidence:
Digital evidence is often considered more reliable and objective than traditional forms of evidence like eyewitness testimony. It can provide a clear, unambiguous record of events or transactions, reducing the chances of misinterpretation or bias.
Preserved Integrity:
Digital evidence is preserved in its original form and can be authenticated using hash values to confirm its integrity. Technologies like cryptographic hashing allow for the verification that data has not been altered or tampered with since its collection.

2. Widely Applicable Across Different Legal Areas:

Cybercrimes and Fraud:
Digital evidence plays a central role in prosecuting cybercrimes, such as hacking, identity theft, data breaches, and online fraud. It helps investigators track online activities, communications, and transactions.
Criminal Cases:
In traditional criminal cases, digital devices like computers, mobile phones, and surveillance systems often contain crucial evidence like call logs, text messages, GPS data, and photos/videos.
Civil and Commercial Disputes:
Digital evidence is used in commercial disputes involving intellectual property theft, fraud, breach of contract, and more. Emails, contracts, and digital communications are increasingly used to settle disagreements in business.

3. Effective in Proving Causation and Establishing a Timeline:

Data Correlation and Sequence of Events:
Digital records help establish a timeline of events, which is often critical in determining the sequence of actions and linking a defendant to a particular crime. In cases like traffic accidents, domestic violence, and murder investigations, digital evidence like GPS data, phone logs, and social media activity can show where a suspect was at the time of the crime.
Reliable Causation:
In fraud or tax evasion cases, digital evidence can be used to trace the source of financial transactions, establish fraud patterns, or prove illicit activities that led to the crime.

4. Cost and Time Efficiency:

Quicker Investigations and Trials:
Digital evidence can speed up investigations by providing quick access to large amounts of data. With the help of digital forensics tools and data analytics, investigators can analyze volumes of data quickly and efficiently.
Reduced Need for Physical Evidence:
Many cases that would traditionally require physical evidence (e.g., documents, physical items) can now be resolved with digital data, which is often easier to store and retrieve.

5. Stronger Evidence in the Courtroom:

Admissibility and Legal Framework:
With the enactment of the Information Technology Act, 2000, and subsequent amendments, India has developed a robust legal framework for the admissibility of digital evidence in courts. Section 65B of the Indian Evidence Act, 1872 ensures that electronic records are recognized as valid evidence, provided they meet certain authentication criteria.
Clear Documentation:
Digital evidence provides a clear, documented chain of events, including timestamps, user activity, and system logs. This makes it easier for the court to follow and understand the facts of the case, especially when dealing with complex financial frauds or online harassment cases.

6. Remote and Cross-Border Legal Use:

Global Cooperation:
Digital evidence allows for cross-border legal proceedings. With the increasing use of cloud computing, online platforms, and social media, digital evidence can be collected from across the world. This is particularly helpful in cybercrime investigations, where the suspect may be operating from another country.
Remote Access to Evidence:
With digital evidence stored on cloud platforms, law enforcement agencies can access critical data remotely, making investigations more efficient and facilitating international cooperation in criminal justice.

7. Facilitates Forensic Analysis and Expertise:

Forensic Tools and Expertise:
The field of digital forensics has evolved to provide specialized tools and methodologies to collect, analyze, and preserve digital evidence. Forensic experts use these tools to retrieve deleted files, encrypted data, and even data from damaged or destroyed devices.
Exponential Growth of Data Sources:
Digital devices today generate vast amounts of data, and digital forensics helps extract and interpret this data. Evidence from computers, mobile phones, surveillance footage, social media activity, and even IoT devices can provide a comprehensive view of the events surrounding a case.

8. Prevents Evidence Tampering:

Chain of Custody:
With the chain of custody in place for digital evidence, it becomes easier to track the handling, storage, and movement of digital data. This reduces the chances of evidence tampering or alteration. The use of digital signatures and encryption also safeguards the authenticity of digital evidence.
Audit Trails:
Many digital platforms, including email systems, financial software, and cloud storage, provide audit trails that record every action or modification performed on a file or record. These trails are valuable in proving that the evidence has not been tampered with.

9. Digital Evidence in Election Disputes:

Election Integrity and Malpractice Cases:
Digital evidence plays a pivotal role in election disputes, where candidates often submit electronic records such as emails, videos, and social media posts to prove malpractice, misconduct, or defamation. In the landmark case of Anvar P.V. v. P.K. Basheer, the Supreme Court clarified the rules for the admissibility of electronic evidence in election disputes, ensuring that digital records are properly authenticated before being used as evidence.

10. Increasing Legal and Social Awareness:

Technological Advancements:
The rise in the use of digital evidence has led to a greater awareness of technology among legal practitioners, judges, and lawmakers. As more cases involve digital evidence, there is a growing demand for legal education on issues such as data protection, privacy, digital signatures, and cyber laws.
Impact on Legal Reforms:
With the increasing importance of digital evidence, legal reforms continue to evolve, ensuring that the justice system remains adaptable to the challenges of the digital age. Laws governing cybercrimes, online privacy, data protection, and electronic evidence are becoming more robust and comprehensive.

Conclusion:

Digital evidence has revolutionized the way legal proceedings are conducted in India, providing numerous benefits in terms of efficiency, accuracy, security, and admissibility. The growing reliance on electronic records and digital footprints underscores the need for specialized knowledge and expertise in digital forensics and cyber laws. For postgraduate students of forensics, understanding the practical application and importance of digital evidence is essential for dealing with modern-day legal challenges and contributing effectively to the justice system.

DISCUSS FOR PG STUDENTS : DATA PROTECTION AND PRIVACY IN THE CONTEXT OF DIGITAL EVIDENCE AS ADMISSIBLE EVIDENCE IN COURT OF LAW

Data Protection and Privacy in the Context of Digital Evidence as Admissible Evidence in Court of Law

In today’s digital age, data protection and privacy have become critical issues, particularly when digital evidence is being used in legal proceedings. The use of electronic records in courtrooms raises concerns about the balance between the need for justice and the right to privacy of individuals. Digital evidence, whether it’s an email, a text message, a social media post, or a file from a hard drive, can provide valuable information in legal cases. However, this type of evidence also poses significant challenges regarding its admissibility, authentication, and its potential invasion of privacy.

Here’s a detailed discussion for PG students of forensic science on data protection and privacy when dealing with digital evidence in legal proceedings.

1. Legal Framework for Data Protection and Privacy in India

a) The Information Technology Act, 2000

Section 65B of the Indian Evidence Act, 1872: This section, as amended by the Information Technology Act, specifically addresses the admissibility of electronic records. To be admissible in court, electronic records must meet certain criteria, and a certificate of authenticity (under Section 65B) is mandatory for the admission of such records as evidence.
Data Protection and Privacy:
While the Information Technology Act focuses on the admissibility of digital evidence, it is also crucial in data protection. The Act outlines provisions for hacking, identity theft, and data breaches, all of which affect the privacy and protection of individuals' digital information.

b) The Personal Data Protection Bill, 2019

Current Framework:
India’s Personal Data Protection Bill, 2019 (pending approval) is the key legislation aimed at protecting personal data and safeguarding the privacy of individuals. This bill proposes a comprehensive legal framework for data protection, including the use of personal data, its collection, processing, and storage.
Relevance to Digital Evidence:
The bill ensures that data is handled responsibly and sets guidelines for lawful data collection and processing. The consent of the data subject is one of the central tenets of the bill, which makes it highly relevant when digital evidence is being gathered and presented in court. Any violation of these principles may render the evidence inadmissible or illegal to use in legal proceedings.

2. Data Protection in Digital Evidence Collection and Preservation

The process of collecting digital evidence must respect the privacy rights of individuals and adhere to data protection laws. Here are key considerations for PG students regarding the lawful collection and preservation of digital evidence:

a) Legal Collection of Data

Search Warrants and Court Orders:
When collecting digital evidence from individuals’ devices (e.g., mobile phones, computers, servers), investigators must follow the legal protocols for obtaining search warrants or court orders. These legal instruments are essential to protect the privacy rights of individuals and ensure that the collection of evidence does not violate constitutional rights, such as Article 21 (right to life and personal liberty).
Minimizing Intrusion:
Investigators should collect only relevant evidence while minimizing the intrusion into the privacy of individuals. The principle of data minimization is key, where only the necessary amount of data is collected, and irrelevant data is avoided.

b) Ensuring Data Integrity and Chain of Custody

Chain of Custody:
To prevent tampering with digital evidence, a chain of custody must be maintained, ensuring that the evidence is handled, transferred, and stored correctly. Incomplete or improper chain-of-custody procedures may result in the exclusion of evidence.
Data Integrity:
Digital evidence is susceptible to alteration or manipulation. To ensure its integrity, investigators use techniques like hashing (e.g., MD5, SHA1) to generate a digital fingerprint of the data. If the hash value matches at the time of collection and presentation in court, the integrity of the evidence is confirmed.

3. Admissibility of Digital Evidence and Privacy Considerations

a) Admissibility of Digital Evidence in Court

Authentication Under Section 65B of the Indian Evidence Act:
Digital evidence must be accompanied by a certificate under Section 65B of the Indian Evidence Act to be admissible in court. The certificate must confirm the authenticity and integrity of the digital record and must be signed by a person who has control over the device from which the evidence was collected.
Consent and Privacy:
In many cases, especially in personal matters or non-criminal cases, the consent of the data owner is essential to access digital evidence. In the absence of consent, the evidence may be contested on privacy grounds. For example, if the data collection involves personal communications (like emails or private messages), the court may scrutinize whether the data was accessed and presented without violating privacy laws.

b) Privacy in the Context of Electronic Evidence

Intrusive Nature of Digital Evidence:
Some digital evidence may be highly intrusive, especially if it involves accessing personal information such as emails, messages, photos, browsing history, and even location data from GPS systems. The right to privacy under Article 21 of the Indian Constitution may be infringed upon if data collection is not conducted lawfully.
Protecting Sensitive Data:
When dealing with sensitive data (e.g., financial records, health information, or personal communication), the court must ensure that the privacy of individuals is respected. In some cases, digital evidence may need to be redacted or anonymized to avoid revealing personal details that are not relevant to the case at hand.

4. Data Protection During Legal Proceedings

a) Ensuring Confidentiality and Integrity

Court's Role in Protecting Privacy:
The court is responsible for ensuring that digital evidence presented during proceedings does not violate privacy laws. Courts must be cautious in handling personal data, ensuring that it is only disclosed to relevant parties and is not unnecessarily exposed to the public.
Security of Digital Evidence:
During legal proceedings, digital evidence must be stored securely. Forensic labs, legal institutions, and courts must implement stringent security measures to prevent unauthorized access or data breaches.

b) Role of Digital Forensic Experts

Forensic Analysis and Data Privacy:
Forensic experts must ensure that their analysis of digital evidence does not breach the privacy rights of individuals. This includes data sanitization (erasing irrelevant or personal data) and ensuring that the tools used for analysis do not alter or tamper with the evidence.

5. International Best Practices and Data Privacy

a) International Legal Standards

GDPR (General Data Protection Regulation):
The European Union’s GDPR provides one of the strictest data protection frameworks in the world. It places emphasis on the consent of individuals for processing personal data and the right to be forgotten. While India has not yet adopted a comprehensive data protection law, GDPR compliance is a best practice for handling digital evidence, particularly in cross-border cases.
Global Cooperation in Cybercrime:
In international cases involving cybercrime or digital evidence, countries often follow global standards to ensure that privacy is not infringed upon during investigations. Cooperation between nations is essential for the lawful transfer of digital evidence and the protection of personal data.

6. Balancing Privacy and Justice

While privacy is a fundamental right, the need for justice often requires the presentation of digital evidence in court. The challenge lies in maintaining a balance between ensuring privacy protection and allowing access to necessary information to pursue justice. Proper laws and safeguards, including obtaining proper consent, ensuring proper certification (Section 65B certificate), and minimizing data access, are essential for achieving this balance.

Conclusion

The use of digital evidence in legal proceedings requires strict adherence to data protection and privacy laws. For PG students of forensic science, it is crucial to understand the legal framework governing the admissibility of digital evidence and the procedural safeguards required to ensure that privacy is not violated. The balance between the right to privacy and the need for justice is delicate, and forensic experts, legal practitioners, and law enforcement agencies must be vigilant in respecting privacy while ensuring the authenticity and integrity of digital evidence. The future of digital evidence in Indian legal proceedings will depend on how effectively these challenges are addressed and balanced within the framework of data protection laws.

DISCUSS FOR PG STUDENTS : ETHICAL ISSUES IN THE CONTEXT OF DIGITAL EVIDENCE

Ethical Issues in the Context of Digital Evidence

For PG students of forensic science and related fields, it is crucial to understand the ethical challenges involved in handling digital evidence. As digital evidence becomes more prevalent in cybercrime investigations, criminal cases, and other legal proceedings, professionals involved in digital forensics and cybersecurity must adhere to high ethical standards to ensure fairness, justice, and privacy protection.

Below is a detailed discussion of the ethical issues that arise when dealing with digital evidence in criminal investigations and legal proceedings.

1. Privacy Concerns in Digital Evidence Collection

a) Right to Privacy vs. Need for Justice

Privacy is a fundamental human right, enshrined in both international and national laws (e.g., Article 21 of the Indian Constitution), and it extends to the protection of digital data.
In digital forensic investigations, the privacy of individuals must be respected, particularly when the evidence collected involves sensitive information such as personal communication, financial records, health data, and location information.

Ethical Issue: Investigators must ensure that data collection is necessary and proportionate to the investigation. They should avoid accessing irrelevant data or private information that does not relate to the case.

Ethical Solution:

Obtain the necessary legal authority (e.g., search warrants, court orders) before accessing digital data.
Minimize the intrusion into private data by collecting only the data essential for the investigation.
Redact sensitive personal information when presenting evidence in court, if it is not directly relevant to the case.

2. Consent and Authorization

a) Obtaining Consent

In cases where digital evidence is voluntarily provided (such as from a personal device), consent from the owner is necessary. This is especially true in civil cases or non-criminal matters.

Ethical Issue: The investigator must ensure that consent is freely given, and the individual is fully aware of the implications of allowing their data to be accessed. Any form of coercion or misleading could be an ethical violation.

Ethical Solution:

Consent must be informed, meaning the person giving consent should understand the scope and purpose of data access.
Consent forms should be clear and specific, detailing what data will be accessed and how it will be used.

b) Unauthorized Access

In cases where digital evidence is seized without consent, such as in criminal investigations, ethical issues arise regarding the scope of data accessed.

Ethical Issue: Accessing private data without proper authorization or exceeding the scope of the investigation is considered unethical and could lead to evidence being inadmissible.

Ethical Solution:

Investigators must adhere strictly to the warrant or court order specifying the data to be accessed.
Avoid searching personal information that is irrelevant to the investigation.

3. Integrity and Accuracy of Digital Evidence

a) Avoiding Tampering or Alteration

Digital evidence is vulnerable to alteration, either through intentional tampering or unintentional handling errors.
The integrity of digital evidence must be maintained to ensure that it remains accurate and reliable in legal proceedings.

Ethical Issue: Tampering with digital evidence or altering the original data (intentionally or through negligence) is an ethical violation that undermines the credibility of the entire investigation.

Ethical Solution:

Implement a chain of custody protocol to track who handled the evidence and when.
Use tools like hashing algorithms to ensure that the evidence has not been altered during collection, storage, or analysis.
Document all procedures and decisions related to evidence handling for transparency.

4. Confidentiality of Digital Evidence

a) Protecting Confidential Information

Digital evidence often contains highly sensitive and confidential information such as personal details, financial transactions, and business secrets.

Ethical Issue: Breaching confidentiality by disclosing or mishandling digital evidence can result in harm to the parties involved, and may also violate data protection laws and ethical standards.

Ethical Solution:

Ensure that digital evidence is stored securely and only accessible by authorized individuals.
Limit access to confidential information, especially if it is irrelevant to the investigation, and redact personal details when appropriate.
Follow best practices in data encryption and secure storage.

5. Bias and Objectivity in Digital Forensics

a) Avoiding Bias in Analysis

Digital forensic investigators must remain neutral and objective when analyzing digital evidence. Bias can influence the way evidence is interpreted or how conclusions are drawn.

Ethical Issue: Investigators might be influenced by their prejudices or by external pressures (e.g., a desire to support a particular narrative or outcome).

Ethical Solution:

Follow standardized protocols for forensic analysis and reporting, ensuring that conclusions are based on facts, not assumptions.
Avoid personal biases or any attempt to manipulate or misinterpret digital evidence.

6. Handling Digital Evidence in Court

a) Presenting Digital Evidence Fairly

Digital evidence often requires a technical understanding to be presented accurately in court. However, misleading or inaccurate presentations of digital evidence can skew the truth and harm the judicial process.

Ethical Issue: Misrepresentation of digital evidence or failure to explain technical aspects clearly can undermine the integrity of the trial.

Ethical Solution:

Forensic experts and digital evidence handlers must testify honestly and clearly explain the methodologies and tools used during the analysis.
Provide clear, concise, and accurate reports and explain complex technical details in layman’s terms when necessary.

7. Data Breaches and Security Concerns

a) Protecting Data from Breaches

Data breaches during the handling of digital evidence can lead to the unauthorized disclosure of personal or sensitive information, violating ethical standards and privacy laws.

Ethical Issue: Failure to implement security measures during the collection, analysis, and storage of digital evidence can result in serious ethical and legal consequences.

Ethical Solution:

Implement strong data protection measures, including encryption and access controls.
Ensure that digital evidence is handled securely throughout the entire process, from collection to presentation in court.

8. Ethical Training and Professional Conduct

a) Code of Ethics

Professionals involved in digital forensics, including investigators, forensic experts, and legal professionals, must follow a code of ethics to maintain the credibility of the process.

Ethical Issue: Failure to adhere to a professional code can compromise the integrity of the digital forensics field and lead to potential legal challenges.

Ethical Solution:

Adhere to established professional standards such as those provided by the International Association of Computer Science and Information Technology (IACSIT), the National Institute of Standards and Technology (NIST), or similar organizations.
Regularly participate in ethical training to stay informed of current best practices and ethical considerations.

Conclusion

Digital evidence plays a crucial role in modern legal proceedings, and professionals handling such evidence must navigate complex ethical challenges to ensure that justice is served. For PG students of forensic science, it is essential to understand that privacy, integrity, consent, bias, and confidentiality are all central to the ethical handling of digital evidence. By adhering to these ethical principles and following legal standards, forensic professionals can help ensure the admissibility and credibility of digital evidence, thus safeguarding both privacy rights and the administration of justice.

DISCUSS FOR PG STUDENTS: FORENSIC CHALLENGES IN DIGITAL EVIDENCE: TECHNICAL, LEGAL, RESOURCE CHALLENGES

Forensic Challenges in Digital Evidence: Technical, Legal, and Resource Challenges

For PG students in forensic science or related fields, understanding the various challenges involved in handling digital evidence is crucial. As digital evidence becomes an increasingly central part of criminal investigations, it is essential to comprehend the obstacles forensic experts face, which fall into technical, legal, and resource categories. Each of these challenges can impact the reliability, integrity, and admissibility of digital evidence in legal proceedings.

Below, we will discuss these challenges in detail to equip students with the knowledge they need to navigate the complexities of digital forensics.

1. Technical Challenges in Digital Forensics

a) Volatility of Digital Evidence

Digital evidence is often highly volatile, meaning it can easily be lost, altered, or destroyed if not handled properly.
Data on devices such as computers, smartphones, and servers may change as soon as they are powered on or off, or when a new connection is made.

Technical Issue: The timing of data collection is crucial. Failure to collect evidence at the right moment can lead to the loss of critical information.

Solution:

Forensic investigators should employ techniques such as write blockers to prevent modification of data during evidence collection.
Use of imaging tools to capture data from a device in its current state and preserve the integrity of the evidence.

b) Encryption and Data Protection

Encryption technologies, such as full disk encryption and file encryption, are common in modern devices. They are designed to protect privacy but pose a significant challenge for investigators when attempting to retrieve evidence.

Technical Issue: Encrypted data is often inaccessible without the decryption key, which can be difficult to obtain.

Solution:

Forensic investigators must be trained in using decryption tools or work closely with experts who specialize in cryptography.
Use legal means, such as obtaining court orders, to compel suspects to provide decryption keys when necessary.

c) Data Volume and Complexity

Digital evidence can be vast in volume, spanning hard drives, cloud storage, mobile devices, and social media platforms. Sorting through massive amounts of data and identifying relevant evidence can be time-consuming and technically challenging.

Technical Issue: The sheer volume of data makes it difficult for investigators to identify and extract pertinent evidence without the risk of overlooking key information.

Solution:

Employ automated tools for data analysis that can help narrow down the search to relevant files.
Use digital forensics software designed to handle large datasets and quickly identify patterns, keywords, or metadata that could aid in the investigation.

d) Proprietary Formats and Operating Systems

Different devices use different file formats, operating systems, and applications, making it difficult to maintain consistency when dealing with evidence from various sources.

Technical Issue: Investigators may struggle to extract data from proprietary or non-standard formats that are not supported by common forensic tools.

Solution:

Forensic professionals need to be familiar with a wide range of operating systems (Windows, macOS, Linux, Android, iOS, etc.) and file formats (NTFS, HFS, FAT, etc.).
Access to updated and specialized forensic tools designed for diverse platforms is essential to ensure proper extraction and analysis.

2. Legal Challenges in Digital Forensics

a) Admissibility of Digital Evidence

The admissibility of digital evidence is a major concern in legal proceedings. Courts require that digital evidence be authentic, reliable, and legally obtained. The absence of proper protocols during the collection or analysis can result in the evidence being ruled inadmissible.

Legal Issue: Ensuring that digital evidence complies with legal standards for admissibility, including Section 65B of the Indian Evidence Act, is essential for its acceptance in court.

Solution:

Ensure strict adherence to chain of custody protocols to prevent tampering or mishandling of digital evidence.
Collect digital evidence under legal authority, such as search warrants or court orders, to avoid challenges to the evidence's authenticity.

b) Privacy Concerns

Digital evidence often contains sensitive personal information, including communications, financial data, and health records. This raises privacy concerns, especially when handling data unrelated to the crime under investigation.

Legal Issue: Invasions of privacy can result in legal repercussions, particularly if personal data is accessed or disclosed without proper authorization or justification.

Solution:

Forensic professionals must be well-versed in privacy laws, such as the Information Technology Act, the Right to Privacy as enshrined in the Indian Constitution, and GDPR in jurisdictions outside India.
Redact or mask sensitive information when it is irrelevant to the case.

c) Jurisdictional Issues

Digital evidence is often stored on devices located in different geographical locations, which can raise questions of jurisdiction. Investigators may face difficulties when dealing with cross-border investigations and obtaining evidence from servers located in foreign countries.

Legal Issue: Differences in national laws and international treaties can complicate the process of obtaining and using digital evidence.

Solution:

Forensic experts must navigate complex international law and cross-border regulations, ensuring compliance with treaties such as the Mutual Legal Assistance Treaties (MLATs) and extradition laws.

d) Chain of Custody

Maintaining an accurate chain of custody is essential to preserve the integrity of digital evidence and ensure its admissibility in court. Any break in the chain could lead to questions about the authenticity of the evidence.

Legal Issue: Any mishandling or alteration of digital evidence during its collection, storage, or transportation could jeopardize the credibility of the case.

Solution:

Document all interactions with digital evidence meticulously, from collection to analysis to presentation in court.
Implement automated systems to track evidence and ensure the integrity of the chain of custody.

3. Resource Challenges in Digital Forensics

a) Lack of Specialized Training

The field of digital forensics is rapidly evolving, and many investigators may not have access to the specialized training required to handle the complex technological aspects of digital evidence.

Resource Issue: Inadequate or outdated training can result in improper handling or analysis of digital evidence.

Solution:

Establish continuous training programs for forensic professionals, focusing on the latest digital forensics tools, techniques, and legal requirements.
Encourage participation in certified courses and workshops offered by organizations like EnCase, Cellebrite, and others.

b) Cost of Digital Forensics Tools

High-quality digital forensics tools (software and hardware) can be expensive, making it difficult for some law enforcement agencies or private investigators to access the tools needed for thorough investigations.

Resource Issue: The financial burden of acquiring and maintaining cutting-edge forensic tools can strain resources, especially in underfunded jurisdictions.

Solution:

Governments and institutions should allocate more funding to digital forensics as part of their cybersecurity initiatives.
Open-source forensics tools and affordable software solutions can be explored as alternatives to expensive proprietary tools.

c) Time Constraints and Backlog

Due to the high volume of digital evidence and the complexity of analyzing it, forensic investigations can be time-consuming. This is compounded by the backlog of cases waiting for analysis, which can delay justice.

Resource Issue: Backlogs in forensic analysis, especially in digital cases, may result in delayed investigations and prolonged judicial proceedings.

Solution:

Implement automated analysis tools to speed up the process of reviewing digital evidence.
Increase investment in forensic laboratories and staffing to handle the growing number of cases requiring digital evidence analysis.

Conclusion

The challenges involved in digital forensics are multidimensional, encompassing technical, legal, and resource-related difficulties. For PG students studying forensic science, it is crucial to understand that effective digital evidence handling requires not only technical expertise but also adherence to legal standards and ethical considerations. By addressing these challenges through training, resources, and legal awareness, forensic professionals can enhance their ability to handle digital evidence in a way that supports the judicial process and ensures justice.

discuss for pg students : NIST 800-61 CYBER INCIDENT RESPONSE

NIST 800-61: Computer Security Incident Handling Guide – Cyber Incident Response

For PG students in cybersecurity and digital forensics, understanding the principles and frameworks for cyber incident response is critical. The National Institute of Standards and Technology (NIST) has developed a comprehensive guide, NIST 800-61, titled Computer Security Incident Handling Guide, which is a widely accepted standard for responding to cybersecurity incidents in both private and public sectors.

This guide outlines a systematic approach to handling cybersecurity incidents, including the necessary steps for preparation, detection, analysis, containment, eradication, recovery, and post-incident activities. Below is a detailed breakdown of NIST 800-61 and its relevance to cyber incident response.

1. Overview of NIST 800-61: Cyber Incident Response

NIST 800-61 provides best practices and guidelines for creating and managing an effective cyber incident response plan (CIRP). The framework emphasizes that an incident response (IR) process is essential for organizations to manage security breaches and mitigate damage caused by cyberattacks.

The guide focuses on three major areas:

Preparation: Setting up the necessary tools, teams, and processes.
Detection and Analysis: Identifying and understanding the nature of the incident.
Containment, Eradication, and Recovery: Controlling the damage, eliminating the threat, and returning to normal operations.
Post-Incident Activity: Reviewing the incident and improving the response plan.

2. Phases of Cyber Incident Response According to NIST 800-61

NIST 800-61 organizes the cyber incident response process into four distinct phases:

A. Preparation

Preparation involves building a strong foundation for incident response before an incident occurs.

Key Components:

Incident Response Policy: Establish clear guidelines and objectives for incident handling.
Incident Response Team (IRT): Form a specialized team responsible for incident management. Team members should include IT staff, legal advisors, communication experts, and management.
Tools and Resources: Equip the team with tools such as network monitoring, log analysis software, and forensic tools for investigation.
Incident Response Plan (IRP): Develop a well-documented, actionable plan that specifies the roles, responsibilities, and procedures for handling incidents.

B. Detection and Analysis

This phase focuses on identifying potential incidents and thoroughly analyzing them to determine the scope, impact, and nature of the attack.

Key Components:

Monitoring: Continuously monitor network traffic, endpoints, and systems for signs of anomalies or attacks.
Incident Identification: Determine whether the event is a legitimate security incident or a false alarm.
Initial Analysis: Analyze logs, alerts, and available data to understand the incident's severity and affected systems.
Incident Classification: Categorize the type of incident (e.g., malware infection, data breach, DDoS attack) to guide the response process.

Tools and Techniques:

Intrusion Detection Systems (IDS)
Security Information and Event Management (SIEM) solutions
Forensic Software for data collection and analysis

C. Containment, Eradication, and Recovery

Once an incident is confirmed, it is crucial to contain the threat to prevent further damage, eradicate the root cause, and then recover systems to normal operation.

Key Components:

Containment: Implement strategies to limit the spread of the attack, such as isolating infected systems or blocking malicious traffic. The goal is to prevent the incident from escalating while maintaining the integrity of other systems.
Short-term containment could involve temporary measures like disconnecting a system from the network, while long-term containment includes preventing attackers from exploiting further vulnerabilities.
Eradication: Remove the cause of the incident. This may involve deleting malicious files, closing vulnerabilities, or applying patches to systems.
Recovery: Restore affected systems to normal operation, ensuring that they are clean from any malicious artifacts. It's important to monitor the systems during recovery to detect any signs of reinfection or residual issues.

D. Post-Incident Activity

After resolving the incident, it's essential to review the handling process and learn from the event to improve future responses and security posture.

Key Components:

Lessons Learned: Conduct a thorough post-mortem analysis of the incident. Identify what worked well and what didn’t in the response process.
Incident Report: Document the entire incident, including detection, containment, eradication, and recovery actions. This report helps in future incident handling and may be required for regulatory compliance.
Improvement: Based on lessons learned, update the incident response plan, improve security controls, and refine the organization’s policies and procedures to prevent similar incidents in the future.
Communication: Ensure that all stakeholders (management, employees, customers, regulatory bodies) are informed about the incident and any corrective actions taken.

3. Key Concepts and Terms in Cyber Incident Response

For PG students, understanding some key concepts used in NIST 800-61 is crucial:

a) Incident Response Team (IRT)

An organized group of individuals responsible for responding to security incidents.
The team typically includes incident handlers, forensic experts, and IT specialists.

b) Incident Classifications

Malware: Programs designed to damage or disrupt systems (e.g., viruses, worms, ransomware).
Data Breach: Unauthorized access to confidential information.
Denial of Service (DoS): Attacks aimed at disrupting or disabling services.

c) Communication During Incidents

Effective communication is crucial during an incident. The team must communicate with internal stakeholders (e.g., management, employees) and external parties (e.g., customers, law enforcement, media).

d) Chain of Custody

Ensuring that digital evidence is properly collected, documented, and preserved to maintain its integrity for legal or regulatory purposes.

4. Cyber Incident Response Tools

There are several tools that assist in the cyber incident response process, particularly during the detection, analysis, and recovery phases:

Security Information and Event Management (SIEM): Tools like Splunk, LogRhythm, and IBM QRadar are used to collect, monitor, and analyze security-related data from various sources.
Forensic Tools: These tools are essential for analyzing compromised devices and recovering evidence (e.g., EnCase, FTK Imager, X1 Social Discovery).
Intrusion Detection Systems (IDS): These systems (e.g., Snort, Suricata) help in the detection of malicious activities in network traffic.
Incident Management Systems: Platforms like ServiceNow and JIRA help streamline the incident response process by allowing teams to track, manage, and resolve incidents.

5. Importance of Cyber Incident Response in Legal Context

For PG students specializing in digital forensics, understanding the importance of incident response in the legal context is critical. A well-handled incident can lead to:

Admissibility of Evidence: Proper incident response procedures ensure that the collected evidence is legally admissible in court by maintaining the chain of custody and following established guidelines.
Regulatory Compliance: Organizations must comply with various legal and regulatory frameworks (e.g., GDPR, IT Act, 2000 in India). Effective incident response ensures compliance with reporting and documentation requirements.
Litigation: Well-documented and efficient incident handling can help organizations defend themselves in court and may also be used as evidence in lawsuits or criminal investigations.

Conclusion

NIST 800-61 serves as a critical framework for cyber incident response, providing organizations with a structured approach to handle cybersecurity incidents effectively. For PG students, particularly those pursuing studies in digital forensics, understanding the various phases, tools, and challenges involved in cyber incident response is essential. This knowledge equips future forensic experts and cybersecurity professionals to respond swiftly, mitigate risks, and ensure that critical digital evidence is properly handled and preserved for legal purposes.

DISCUSS FOR PG STUDENTS : SPLUNK FOR INCIDENT ANALYSIS

Splunk for Incident Analysis: A Guide for PG Students

Splunk is one of the leading platforms for data analysis, particularly in the context of cybersecurity incident analysis and digital forensics. It is a powerful log analysis and security information event management (SIEM) tool used widely across industries for monitoring, investigating, and responding to cybersecurity incidents.

For PG students specializing in digital forensics, cybersecurity, or incident response, learning how to use Splunk for incident analysis is crucial. This tool can help security analysts identify, investigate, and respond to cyber threats quickly and effectively by analyzing large amounts of data in real-time.

Here is an in-depth discussion on how Splunk can be used for incident analysis.

1. What is Splunk?

Splunk is a software platform primarily used for searching, monitoring, and analyzing machine-generated big data through a web-style interface. It collects, indexes, and visualizes data from various sources, making it particularly useful for analyzing logs, network traffic, system events, and security alerts.

Key Features of Splunk:

Data Ingestion: It can collect data from a wide range of sources, including servers, applications, networks, and security devices (e.g., firewalls, intrusion detection systems).
Real-time Search: Splunk enables real-time searching and filtering of large volumes of data.
Visualization: It offers tools for creating dashboards, graphs, and reports to visualize data.
Alerts and Notifications: Splunk can send alerts based on specific triggers (e.g., when an anomaly is detected).
Machine Learning: It incorporates machine learning for anomaly detection and predictive analysis.

2. Role of Splunk in Cyber Incident Analysis

Splunk plays a central role in identifying and investigating cybersecurity incidents. The platform assists in several key stages of incident response:

A. Detection of Cyber Incidents

Log Collection and Indexing: Splunk gathers log files from multiple sources (firewalls, intrusion detection systems, routers, servers) and indexes them for quick searching. Event logs are crucial in detecting suspicious activity.
Real-Time Monitoring: Splunk’s real-time monitoring allows security teams to continuously track system events and identify unusual patterns, such as unauthorized access attempts, malware activity, or DDoS attacks.

B. Incident Identification and Alerting

Search Queries: Splunk’s powerful search capabilities allow users to run queries on indexed data, making it easier to identify potential cyber incidents.
Preconfigured Alerts: Splunk can trigger alerts when certain thresholds are reached (e.g., a spike in traffic or a known signature of an attack), helping analysts spot potential issues early.
Example: An alert could be triggered by repeated failed login attempts from an unusual location, which might indicate a brute-force attack.

C. Incident Investigation

Detailed Log Analysis: Splunk allows incident responders to drill down into detailed logs to understand the nature of the attack. Security analysts can correlate events from different sources to see the complete picture of the attack.
Data Correlation: By correlating logs across multiple sources (firewalls, servers, endpoints), Splunk can help identify the full scope of an incident, helping analysts understand how the attack unfolded.

D. Root Cause Analysis

After detecting an incident, Splunk assists in identifying the root cause by correlating various data sources. For example, Splunk could highlight that an infected email attachment led to the initial compromise, which then spread across the network.

E. Reporting and Documentation

Splunk enables the generation of comprehensive reports that document all the details of an incident, such as timestamps, affected systems, and detected anomalies. These reports are essential for legal purposes, regulatory compliance, and post-incident reviews.

3. Key Splunk Components for Incident Analysis

A. Splunk Forwarder

The Splunk Forwarder is installed on the source machines or systems and is responsible for forwarding data to the central Splunk instance for processing and indexing. It collects data from a variety of sources such as operating systems, application logs, and network devices.

B. Splunk Enterprise

This is the core platform that collects, indexes, and analyzes data. It allows analysts to perform searches, generate reports, and create dashboards. It is highly customizable and can handle large amounts of data in real-time.

C. Splunk Apps

Splunk Apps are specialized add-ons that extend the functionality of Splunk. For incident analysis, Splunk Security Essentials and Splunk Enterprise Security (ES) are particularly useful, as they offer built-in dashboards, correlation searches, and reports specifically designed for security incident analysis.
Example: The Splunk App for Enterprise Security provides predefined use cases for security events, correlates events from multiple data sources, and produces visual dashboards to detect and respond to incidents.

D. Splunk Search Processing Language (SPL)

SPL (Search Processing Language) is the query language used in Splunk to search and analyze data. For incident analysis, security analysts use SPL to perform complex searches, create custom dashboards, and define alerts.
Example: A sample SPL query to detect brute force attacks might look like:
```
spl
index=security_logs sourcetype=auth_logs "Failed login" | stats count by user, src_ip
```

4. Using Splunk for Incident Analysis: A Step-by-Step Approach

Step 1: Log Ingestion and Data Collection

Begin by ensuring that logs from various devices (servers, firewalls, intrusion detection systems, etc.) are being fed into Splunk in real-time.
The data can come from sources like syslog servers, network devices, or endpoint monitoring solutions.

Step 2: Setting Up Real-Time Monitoring and Alerts

Configure alerts to detect suspicious activities such as login failures, unusual traffic patterns, or malware indicators.
Example alert: “Alert when there are more than 10 failed login attempts from a single IP address within 5 minutes.”

Step 3: Search and Investigate the Incident

Use SPL to search for evidence of the attack across different logs. For example, if there’s an alert for failed logins, use SPL to narrow down the search by user or IP address.
Investigate related logs to trace the origins of the attack and its movement within the network.

Step 4: Correlate Events from Different Sources

Use Splunk’s correlation searches to link related events from different data sources, identifying the broader scope of the incident.
Example: If an attack is detected on an endpoint (like a Trojan infection), correlate this event with firewall logs to see if any external communication was made.

Step 5: Incident Reporting and Documentation

Create detailed incident reports using Splunk’s built-in reporting tools. These reports will document the timeline of events, affected systems, and key findings.
Example: An incident report might show how a specific email attachment led to the installation of malware, which then exfiltrated sensitive data.

Step 6: Post-Incident Review and Analysis

After resolving the incident, use Splunk to perform a post-incident analysis. This includes analyzing how the attack was detected, how quickly it was contained, and what steps can be taken to prevent future incidents.
Update Splunk configurations to address gaps in detection and strengthen security measures.

5. Benefits of Using Splunk for Incident Analysis

Real-Time Detection: Splunk provides continuous monitoring and can alert security teams in real time when suspicious activity is detected.
Comprehensive Data Integration: It collects data from various sources, allowing analysts to investigate incidents from multiple angles.
Improved Incident Response Time: By automating alerts and simplifying data analysis, Splunk reduces the time required to identify and respond to incidents.
Scalability: Splunk is scalable, making it suitable for small, medium, and large enterprises.
Compliance and Reporting: Splunk helps organizations meet compliance requirements by generating comprehensive reports that document incident handling and resolution processes.

Conclusion

Splunk is an essential tool for cyber incident analysis and plays a pivotal role in digital forensics. For PG students, mastering Splunk enables the ability to quickly detect, investigate, and respond to cyber incidents. The platform’s ability to collect, analyze, and correlate vast amounts of data in real time makes it an indispensable resource for cybersecurity professionals and digital forensic experts. Understanding its capabilities and how to use it effectively will provide students with the practical skills necessary to handle complex cyber incidents in real-world scenarios.

DISCUSS : SPLUNK ARCHITECTURE

Splunk Architecture: A Guide for PG Students

Splunk is a powerful platform designed for searching, monitoring, and analyzing machine-generated data from various sources, such as logs, network traffic, and application data. Understanding its architecture is crucial for effectively deploying and utilizing Splunk in cybersecurity, digital forensics, and incident response.

The Splunk architecture is scalable, modular, and allows integration with a wide range of data sources, making it a popular choice for security information and event management (SIEM) solutions.

Here’s a breakdown of Splunk’s architecture and how it supports data collection, indexing, search, and visualization:

1. Core Components of Splunk Architecture

Splunk’s architecture is based on a distributed model that consists of several key components, each responsible for specific tasks related to data ingestion, indexing, searching, and visualization.

A. Forwarders

Splunk Forwarders are lightweight agents that collect data from various sources and send it to Splunk indexers for processing and indexing. There are two types of forwarders:
- Universal Forwarder (UF): A minimal and lightweight version that only forwards raw data.
- Heavy Forwarder (HF): A more feature-rich version that can parse, index, and forward data to the indexers.

Role of Forwarders:

Forwarders are crucial in ensuring that data is collected from the various sources (servers, applications, network devices, etc.) and forwarded to Splunk for indexing.
Example: In a network environment, forwarders are installed on the firewalls, routers, and servers to collect logs and send them to the Splunk Indexers.

B. Indexers

Splunk Indexers are responsible for indexing the incoming raw data. This is where the data parsing and indexing happen. Indexers make the data searchable by breaking it down into events and storing it in an efficient, optimized index.

Role of Indexers:

Indexers ensure that data is parsed and indexed correctly, enabling fast search performance.
They store the indexed data on disk in splunkd (Splunk daemon).
The indexed data is divided into buckets (cold, warm, hot, frozen) depending on how recent the data is.

Example: If logs are ingested from a firewall, the indexer parses the log data and indexes it, making it searchable for later analysis.

C. Search Head

Search Heads are the interfaces that allow users to query and analyze indexed data. They enable users to search, visualize, and report on the indexed data through dashboards, reports, and search processing.

Role of Search Heads:

Search heads are responsible for managing search queries and distributing them to the appropriate indexers.
They present the search results and visualizations to the users via a web interface.

Example: A security analyst can use the search head to search through logs from multiple indexers to detect unusual login activity.

D. Cluster Master (in Clustering Architecture)

When you scale Splunk horizontally, Indexers and Search Heads can be configured in a cluster for load balancing and fault tolerance. The Cluster Master is the component that manages the indexer clusters.

Role of Cluster Master:

It manages the indexer cluster by coordinating the replication and data distribution across all indexers.
It ensures that data is replicated across multiple indexers to provide redundancy and improve availability.

E. Deployment Server

The Deployment Server is used to manage and distribute configuration files, apps, and forwarder configurations across multiple Splunk instances.

Role of Deployment Server:

It provides a centralized location for managing Splunk configurations and ensuring consistency across distributed Splunk components.

Example: If you have hundreds of forwarders in a large deployment, the deployment server ensures that all forwarders are configured in the same way.

F. License Master

License Master is responsible for managing and distributing Splunk licenses to all other Splunk components (Forwarders, Indexers, etc.).

Role of License Master:

It enforces license limits, such as indexing volume and the number of users.
It ensures that the Splunk instance remains within its licensing limits.

G. Data Models & Knowledge Objects

Data Models represent a structured and organized way to access and interpret raw data that has been indexed by Splunk.
Knowledge Objects are predefined sets of fields, tags, and event types that provide useful context for data analysis.

Role of Data Models & Knowledge Objects:

They help make raw data easier to query and analyze by providing semantic layers.
They are particularly useful in the context of security analysis and incident investigation, where analysts may need to correlate data from different sources.

2. Splunk’s Data Flow

The data flow within Splunk follows a streamlined process that ensures efficient collection, indexing, searching, and visualization:

A. Data Ingestion (Forwarders)

Data sources such as logs, network traffic, application data, etc., are ingested by the forwarders.
Forwarders send the data to the Splunk Indexers for parsing and indexing.

B. Parsing and Indexing (Indexers)

The indexers parse the incoming data, extracting valuable information (such as timestamps, IP addresses, and event types).
The parsed data is stored in an indexed format that can be queried later.

C. Searching and Analysis (Search Heads)

Search heads allow users to run search queries against indexed data. The search heads are responsible for presenting results in the form of reports, dashboards, and visualizations.
Users (security analysts, system administrators) interact with search heads to run search queries to detect, investigate, and analyze incidents.

D. Reporting and Visualization (Search Heads and Dashboards)

Reports and visualizations are generated based on search queries to summarize data and help in decision-making.
Splunk’s search heads and dashboards provide easy-to-understand visualizations like graphs, pie charts, and tables to make complex data more digestible.

3. Splunk Distributed Deployment and Scaling

Splunk is designed to scale horizontally, meaning it can handle large amounts of data by distributing the load across multiple instances.

Horizontal Scaling: You can deploy additional indexers and search heads to handle increasing data volumes and users.
Replication and Failover: In large deployments, multiple indexers can replicate data to ensure redundancy and fault tolerance.
Load Balancing: The search heads distribute search queries to available indexers for load balancing, ensuring optimal performance.

4. Splunk Architecture Example in a Cybersecurity Scenario

In a typical cybersecurity use case, such as monitoring for malicious activity or responding to an incident, Splunk’s architecture functions as follows:

Forwarders installed on network devices (routers, firewalls, and servers) collect event logs and send them to indexers.
The indexers parse and index the incoming log data (e.g., failed login attempts, network traffic spikes).
Search heads allow cybersecurity analysts to search through the indexed data for patterns or anomalies that might indicate a breach or attack (e.g., brute force attempts, suspicious IP addresses).
The results of the search are displayed as visualizations (graphs, charts) on a dashboard for the analyst to investigate further.

5. Benefits of Splunk Architecture

Scalability: Splunk can scale to handle vast amounts of data across distributed systems.
Flexibility: Splunk supports a wide variety of data sources and formats, allowing it to be adapted to various use cases.
High Availability and Fault Tolerance: The distributed nature of Splunk provides redundancy, making it suitable for large enterprises and critical infrastructure.
Centralized Management: The deployment server allows for centralized configuration and management of multiple Splunk components.

Conclusion

Understanding Splunk architecture is vital for efficiently leveraging the platform for cybersecurity, digital forensics, and incident response. By breaking down its core components — forwarders, indexers, search heads, and data management layers — students can better appreciate how Splunk works to provide real-time analytics and powerful searching capabilities for handling cyber incidents and data breaches. As future cybersecurity professionals, PG students will be well-equipped to deploy and manage Splunk systems, helping organizations detect, investigate, and mitigate cyber threats effectively.

Cyber Threat Hunting Workshop - ITU 19112020 A.pdf

PDF

DISCUSS FOR PG STUDENTS AN OVERVIEW WITH NUANCES AND IMPORTANT CONCEPTS , IMPORTANT TAKEAWAYS AND METHODS OF CYBER THREAT HUNTING

Cyber Threat Hunting: Overview, Key Concepts, and Takeaways for PG Students

Cyber threat hunting is a proactive approach to detecting and mitigating cybersecurity threats that often bypass traditional security measures. Here's an in-depth discussion based on the provided content:

Overview of Cyber Threat Hunting

Definition: It involves proactive and iterative processes of detecting advanced threats in networks, endpoints, and infrastructure.
Goal: Identify and mitigate potential compromises, reducing dwell time—the period between compromise and detection.

Key Concepts and Frameworks

Presumption of Compromise:
- Assume existing security measures may fail.
- Adopting an "assume breach" mentality helps organizations stay vigilant.
Hunting Methodologies:
- IOC-Based: Based on Indicators of Compromise (e.g., IP addresses, domains).
- Hypothesis-Based: Formulates hypotheses using frameworks like MITRE ATT&CK.
- Baseline-Based: Detects deviations from baseline data.
- Anomaly-Based: Identifies irregular patterns, often useful in fraud detection.
Frameworks:
- MITRE ATT&CK: A knowledge base of adversary tactics and techniques.
- Cyber Kill Chain: A step-by-step process that outlines stages of a cyberattack.
- Pyramid of Pain: Highlights the challenges adversaries face as defenders address indicators like TTPs, artifacts, and domains.

Process of Threat Hunting

Creating Hypotheses:
- Focus on critical assets (e.g., crown jewels) and potential threats.
- Questions like "How would an attacker infiltrate?" guide the hypotheses.
Investigation with Tools and Techniques:
- Use SIEM systems, EDR solutions, and threat intelligence platforms.
- Analyze data using statistical and visualization techniques.
Uncover New Patterns:
- Identify unknown tactics and techniques (TTPs).
- Use findings to improve detection mechanisms.
Automation and Analytics:
- Automate repetitive successful hunting processes.
- Provide feedback to machine learning systems for improved analytics.

Tools and Technology

SIEM (Security Information and Event Management): Centralized log collection and real-time analysis.
EDR (Endpoint Detection and Response): Tracks endpoint activities for malicious behavior.
Network Tools:
- Packet capture tools like Wireshark.
- Intrusion detection systems like Snort.

Important Takeaways

Threat Hunter Skillset:
- Analytical mindset, knowledge of OS and network architecture.
- Proficiency in tools like Sysmon, Volatility, and log analyzers.
Integration with Threat Intelligence:
- Threat intelligence complements threat hunting but isn't a replacement.
- Helps narrow down adversary motivations and tactics.
Benefits:
- Faster detection of advanced threats.
- Minimizes business risks and improves cybersecurity posture.

Methods of Implementation

Threat Hunting Playbooks:
- Documenting hypotheses, tools, and processes for standardized hunting.
- Examples: Jupyter Notebooks for structured investigations.
Case Studies:
- Use cases like detecting malicious logins via RDP or identifying malware persistence mechanisms highlight real-world applications.
Continuous Improvement:
- Threat hunting is iterative; findings from hunts inform future detection strategies.

Cyber threat hunting is a critical component of modern cybersecurity. It enhances detection capabilities, fortifies defenses, and ensures a proactive approach to mitigating evolving threats. By combining technical expertise, advanced tools, and structured methodologies, organizations can significantly bolster their threat management strategies.

DISCUSS METHODOLOGIES, FRAMEWORKS AND PROCESS OF THREAT HUNTING IN DETAIL

Detailed Discussion on Threat Hunting Methodologies, Frameworks, and Processes

1. Threat Hunting Methodologies

Threat hunting methodologies define structured approaches for proactively identifying threats. Here are the most widely used methodologies:

IOC-Based Threat Hunting

Definition: Relies on Indicators of Compromise (IOCs) like malicious IP addresses, domain names, file hashes, or registry changes.
Purpose: Identify known malicious activities and artifacts.
Limitations:
- Reactive in nature; focuses only on previously known threats.
- Misses novel or stealthy attacks.

Hypothesis-Driven Threat Hunting

Definition: Centers on creating hypotheses based on threat actor behavior or potential attack vectors.
Example: Hypothesizing that attackers may use PowerShell for persistence on critical servers.
Benefits:
- Encourages proactive discovery of unknown threats.
- Often aligns with frameworks like MITRE ATT&CK.

Baseline and Anomaly Detection

Definition: Establishes a baseline of normal system behavior and identifies deviations.
Steps:
- Collect historical data to define norms (e.g., usual login hours, traffic patterns).
- Investigate anomalies (e.g., off-hours logins, data exfiltration patterns).
Benefits:
- Effective against novel or polymorphic threats.
- Can detect insider threats.

TTP-Based Hunting

Definition: Focuses on threat actor tactics, techniques, and procedures (TTPs) as outlined in frameworks like MITRE ATT&CK.
Example: Searching for persistence mechanisms like scheduled tasks or registry autoruns.
Strengths:
- Provides context on adversary goals and methods.
- Reduces noise by focusing on specific behaviors.

2. Frameworks for Threat Hunting

Frameworks provide guidance for structuring threat hunting efforts and aligning them with adversary behaviors.

MITRE ATT&CK Framework

Description: A knowledge base of adversary tactics and techniques categorized into the stages of an attack lifecycle.
Applications:
- Aligning hypotheses with known adversary behaviors.
- Mapping detection capabilities to identify gaps.
- Example TTPs: Credential dumping (T1078), lateral movement (T1021).

Cyber Kill Chain

Description: Developed by Lockheed Martin, it outlines the stages of a cyberattack, from reconnaissance to execution.
Stages:
1. Reconnaissance
2. Weaponization
3. Delivery
4. Exploitation
5. Installation
6. Command and Control
7. Actions on Objectives
Applications:
- Guides hunters to focus on specific stages (e.g., delivery or lateral movement).
- Helps break the attack chain to thwart adversaries.

Pyramid of Pain

Description: Highlights the difficulty for adversaries when defenders act on certain indicators.
Tiers (from least impactful to most impactful):
- Hash values
- IP addresses
- Domain names
- Network/Host artifacts
- Tools
- TTPs
Key Insight: Targeting TTPs inflicts the most disruption on attackers.

Diamond Model of Intrusion Analysis

Description: Focuses on four key components:
1. Adversary: The threat actor behind the attack.
2. Infrastructure: Systems used for the attack (e.g., C2 servers).
3. Victim: Targeted individuals or organizations.
4. Capability: Malware, exploits, or techniques used.
Applications:
- Helps analysts understand attack patterns.
- Connects isolated events to build a cohesive attack narrative.

3. Threat Hunting Process

A systematic threat hunting process ensures thorough and repeatable investigations.

Step 1: Hypothesis Creation

Objective: Develop a testable hypothesis about potential threats.
Inputs:
- Threat intelligence reports.
- Suspicious behaviors or patterns.
- Known adversary TTPs (e.g., privilege escalation on Active Directory servers).
Example: "An attacker may exploit vulnerable SMB shares to perform lateral movement."

Step 2: Investigation and Data Collection

Activities:
- Leverage SIEM tools, logs, and telemetry for data collection.
- Utilize packet captures, endpoint activity logs, and user behavior analytics.
Tools:
- Splunk, ELK, or QRadar for log analysis.
- Wireshark for network traffic.
- EDR solutions like CrowdStrike or Carbon Black.

Step 3: Pattern Discovery and Analysis

Objective: Identify patterns, anomalies, or deviations that confirm or refute the hypothesis.
Techniques:
- Statistical analysis: Correlate anomalous login attempts with geographic location.
- Behavioral analysis: Detect persistence mechanisms like startup scripts.
- Visualization tools: Graph anomalous connections or data flows.

Step 4: Detection and Reporting

Activities:
- Document findings with actionable insights.
- Develop detection rules for SIEM/EDR platforms based on identified patterns.
- Example: Create a rule for alerting on PowerShell scripts launched from suspicious locations.

Step 5: Response and Remediation

Objective: Mitigate identified threats and improve defenses.
Actions:
- Quarantine infected systems.
- Block malicious domains or IPs.
- Patch vulnerabilities exploited during the attack.

Step 6: Continuous Improvement

Activities:
- Incorporate lessons learned into detection capabilities.
- Share findings with threat intelligence platforms for community defense.

Important Takeaways

Threat Intelligence Integration:
- Use intelligence feeds to guide hunting efforts and contextualize findings.
Automation:
- Automate repetitive processes with machine learning and AI to focus on complex threats.
Metrics and KPIs:
- Measure success using metrics like dwell time reduction, false positive rates, and coverage of MITRE ATT&CK techniques.
Collaboration:
- Collaborate across teams (IT, forensics, incident response) to enrich investigations.

Conclusion

Effective threat hunting combines structured methodologies, robust frameworks, and systematic processes. By aligning efforts with frameworks like MITRE ATT&CK and leveraging tools like SIEMs and EDR solutions, organizations can proactively identify and mitigate sophisticated cyber threats. The iterative nature of threat hunting ensures continuous improvement, safeguarding critical assets from evolving adversaries.