Terminal
Terminal/Skill
Explains how to map fields for Elastic data streams: Elastic Common Schema (ECS) field references, categorization values, custom field types, and fixing mapping validation errors.
By Elastic
Works with
npx skills add elastic/integration-skills --skill ecs-field-mappingsUse this skill when tasks include:
data_stream/<stream>/fields/ecs.yml with ECS field referencesevent.kind, event.category, event.type, and event.outcome valuestype and mapping properties (metric_type, dimension, multi_fields, and related options)elastic-package check, elastic-package lint, or pipeline test schema checksEvery package needs _dev/build/build.yml at the package root. This file pins the ECS schema version used for field resolution.
dependencies:
ecs:
reference: "git@v9.3.0"
This file is required whenever the package has any field file. The scaffold does not generate it — create it manually. If it is missing or uses an outdated version, tests report ECS fields as undefined (e.g., field "destination.ip" is undefined).
A data stream's fields/ directory contains a small set of YAML files with distinct responsibilities:
base-fields.ymlFixed routing constants and @timestamp. All six fields are ECS fields, so each entry uses external: ecs. Override type and value where the data stream needs a constant_keyword with a fixed value — the description is inherited from ECS automatically.
- name: data_stream.type
external: ecs
- name: data_stream.dataset
external: ecs
- name: data_stream.namespace
external: ecs
- name: event.module
external: ecs
type: constant_keyword
value: <package_name>
- name: event.dataset
external: ecs
type: constant_keyword
value: <package_name>.<stream_name>
- name: '@timestamp'
external: ecs
Do not add other fields here. Only these routing constants and @timestamp belong in base-fields.yml.
constant_keyword candidatesFields that hold a single value for every document in a data stream should use constant_keyword. Beyond the routing constants in base-fields.yml, evaluate these:
| Field | Why constant_keyword |
|---|---|
event.dataset | One value per data stream by definition |
event.module | One value per package |
data_stream.type | Fixed per stream (logs/metrics) |
data_stream.dataset | Fixed per stream |
data_stream.namespace | Set at deployment, constant within index |
observer.vendor | Package represents one vendor |
observer.product | Package represents one product |
When a constant_keyword field is also an ECS field (e.g., observer.vendor), use external: ecs with the type override. This inherits the description from ECS and avoids manual duplication. Place the definition in the appropriate field file (ecs.yml for most ECS fields, base-fields.yml for routing constants):
- name: observer.vendor
external: ecs
type: constant_keyword
value: Acme Corp
remove_from_source option: Because constant_keyword stores the value once in index metadata, it does not need to appear in every document's _source. Elasticsearch handles this automatically — no explicit _source.excludes configuration is needed. This saves storage when the value is always the same.
ecs.ymlPopulate this file with the ECS fields the pipeline sets that dynamic mapping cannot infer. Use only name and external: ecs for each entry — no type, no description. The type is resolved from the ECS schema via _dev/build/build.yml.
On packages whose conditions.kibana.version floor is >= 8.13.0, the ecs@mappings component template dynamically maps standard keyword/date and standard-prefix geo ECS fields, so explicit declarations are REQUIRED only for types it cannot infer (geo_point on non-standard parent prefixes, geo_shape, nested, flattened) or where elastic-package validation fails. The gate is the kibana version constraint, not the package-spec version — ecs@mappings is applied by the stack at install time from 8.13 onward, and a spec-3.x package that still admits 8.11/8.12 installs does not get it there, so such packages still need every pipeline-set ECS field declared. Declaring more is harmless for a builder, but is never something a reviewer should demand — and existing declarations are not removable-noise findings either (see review-integration/references/conflict-resolutions.md).
When you DO declare a field whose name exists in ECS, external: ecs must be used (wiki reference). This applies across field files — ecs.yml, base-fields.yml, and any file that defines an ECS field. You may override properties (e.g., type: constant_keyword, value:) while still using external: ecs — the description is inherited from ECS. Do not use external: ecs in fields.yml, agent.yml, or beats.yml — those files define non-ECS fields.
- name: event.kind
external: ecs
- name: event.category
external: ecs
- name: event.type
external: ecs
- name: event.outcome
external: ecs
- name: event.action
external: ecs
- name: source.ip
external: ecs
- name: source.port
external: ecs
- name: destination.ip
external: ecs
- name: user.name
external: ecs
- name: related.ip
external: ecs
- name: related.user
external: ecs
When attaching extra metadata to an ECS field (for example making a field a TSDB dimension or a constant_keyword with a fixed value), combine external: ecs with that metadata. The description is inherited from ECS. Place the definition in ecs.yml (or base-fields.yml for routing constants):
- name: observer.vendor
external: ecs
type: constant_keyword
value: Acme Corp
fields.ymlIntegration-specific custom (non-ECS) fields only. Use a nested group hierarchy for the vendor namespace:
- name: acme.firewall
type: group
fields:
- name: rule_id
type: keyword
- name: policy_name
type: keyword
- name: bytes_in
type: long
unit: byte
metric_type: gauge
Groups do not need to be declared as type: object — defining a group with nested fields is sufficient. The object structure is implicit.
labels.* exceptionlabels is a core ECS object (type: object, object_type: keyword) designed for ad-hoc key-value metadata. Subkeys under labels.* do not require vendor namespacing — this is the one exception to the vendor-prefix rule.
Use labels.* for simple keyword flags or integration-internal markers (e.g., labels.is_ioc_transform_source). Use the vendor namespace for structured or nested data from an upstream source.
Boolean flags and simple tags can live flat under the vendor group:
- name: acme.firewall
type: group
fields:
- name: is_encrypted
type: boolean
- name: policy_name
type: keyword
Structured data from the source should use sub-groups for logical hierarchy:
- name: acme.firewall
type: group
fields:
- name: rule
type: group
fields:
- name: id
type: keyword
- name: name
type: keyword
- name: action
type: keyword
agent.ymlNon-ECS fields populated by the Elastic Agent or Beats framework but not covered by ECS. Include only when the input type emits these fields. Typical fields: cloud.image.id, cloud.instance.id, host.containerized, host.os.build, host.os.codename, input.type, log.offset.
See references/root-and-core-fields.md for full YAML samples.
beats.ymlFilebeat/Beats-specific fields not covered by ECS. Minimal form contains input.type and log.offset. Some inputs also emit log.flags or log.file.* sub-fields.
See references/root-and-core-fields.md for full YAML samples.
Prefer ECS fields whenever semantics match. If no ECS field exists for the data, add it under the package namespace in fields.yml.
| Field | Type | Notes |
|---|---|---|
event.kind | keyword | Highest-level classification. |
event.category | keyword[] | Broad domain buckets — always an array. |
event.type | keyword[] | Sub-buckets within category — always an array. |
event.outcome | keyword | success, failure, unknown; only set when meaningful. |
event.kind: alert, asset, enrichment, event, metric, pipeline_error, signal, stateevent.category: api, authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, library, malware, network, package, process, registry, session, threat, vulnerability, webevent.type: access, admin, allowed, change, connection, creation, deletion, denied, device, end, error, group, indicator, info, installation, protocol, start, userDecision workflow:
event.kind: event for normal logs, metric for measurements, state for periodic categorical observations about a thing (e.g. CDR posture/compliance findings), asset for entity/inventory snapshot records — one document per entity per collection cycle (see entity-mappings skill), pipeline_error in on_failureevent.category: one or more values (array) for the broad domainevent.type: one or more values (array) for operation styleevent.outcome: only when a clear success/failure/unknown applies; omit for informational/metric eventsUse event.action for source-specific verbs (blocked, dropped, authenticated).
See references/categorization-cheatsheet.md for full worked examples.
ECS defines several timestamp fields with distinct semantics. Use them correctly:
| Field | When to use | Set by |
|---|---|---|
@timestamp | The primary event timestamp. Parse from the source event data. Required. | Integration pipeline |
event.created | When the event was first created or recorded by the source system, if different from @timestamp. | Integration pipeline |
event.start | When an activity or period began (e.g., session start, connection start). | Integration pipeline |
event.end | When an activity or period ended (e.g., session end, connection close). | Integration pipeline |
event.ingested | When the event was ingested into Elasticsearch. | Elasticsearch (outside the integration) |
event.ingested must NEVER be set by an integration pipeline. It is managed automatically by Elasticsearch's final pipeline. Do not add a set processor for event.ingested.
When the source data contains multiple timestamps:
@timestamp.event.created.event.start and event.end.type: date in fields.yml.Some ECS field sets must be nested under a parent entity — they are not valid at document root.
geo — must be nested under: client.geo, destination.geo, host.geo, observer.geo, server.geo, source.geo, threat.indicator.geo
Root-level geo.* fields are not recognized and will appear unmapped. Always set target_field on the geoip processor:
- geoip:
field: source.ip
target_field: source.geo
ignore_missing: true
as (Autonomous System) — nested under: client.as, destination.as, server.as, source.as
When using geoip for geolocation, always also perform an ASN lookup using GeoLite2-ASN.mmdb and rename the raw output fields to ECS names. The geoip ASN processor outputs asn and organization_name, which must be renamed to as.number and as.organization.name:
- geoip:
database_file: GeoLite2-ASN.mmdb
field: source.ip
target_field: source.as
properties:
- asn
- organization_name
ignore_missing: true
- rename:
field: source.as.asn
target_field: source.as.number
ignore_missing: true
- rename:
field: source.as.organization_name
target_field: source.as.organization.name
ignore_missing: true
See the ingest-pipelines skill → references/processor-cookbook.md for the full geo+ASN pattern with both source and destination.
os — nested under: host.os, observer.os, user_agent.os
entity (beta; leaf fields require ECS v9.4.0+) — nested under: user.entity, host.entity, service.entity, cloud.entity, orchestrator.entity, root entity, entity.target. Never invent device.entity.* — not a valid parent. For entity data streams (i.e. those with event.kind: asset), see the entity-mappings skill for the full field catalog, pipeline patterns, and ECS availability matrix.
Some ECS fields use type: nested, meaning they hold an array of objects where each object groups related sub-fields together. The pipeline must produce this structure — do not flatten these into parallel scalar arrays.
ECS fields that use nested type:
| Field | Contains |
|---|---|
email.attachments | file.name, file.size, file.extension, file.mime_type, file.hash.* |
threat.enrichments | indicator.*, matched.* |
threat.indicator.file.elf.sections | name, physical_size, virtual_size, etc. |
threat.indicator.file.pe.sections | name, physical_size, virtual_size, etc. |
process.elf.sections | name, physical_size, virtual_size, etc. |
process.pe.sections | name, physical_size, virtual_size, etc. |
Anti-pattern — parallel arrays (WRONG):
{
"email": {
"attachments": {
"file": {
"name": ["a.pdf", "b.pdf"],
"size": [1024, 2048]
}
}
}
}
This loses the association between each attachment's name and size. Queries cannot isolate individual objects.
Correct — array of objects:
{
"email": {
"attachments": [
{ "file": { "name": "a.pdf", "size": 1024 } },
{ "file": { "name": "b.pdf", "size": 2048 } }
]
}
}
ecs.yml declaration: declare only the parent nested field with external: ecs. Child fields (email.attachments.file.name, etc.) inherit their types from the ECS schema — do not redeclare them individually.
- name: email.attachments
external: ecs
Pipeline construction: when source data delivers attachment metadata as separate parallel arrays (e.g., a comma-separated list of filenames and a separate list of sizes), use a script processor to zip them into an array of objects. See ingest-pipelines → references/painless-patterns.md for array construction patterns and references/processor-cookbook.md → Foreach semantics for iterating over array elements.
- script:
tag: build_email_attachments
description: Build email.attachments as array of nested objects from parallel source arrays.
lang: painless
if: ctx.json?.file_names instanceof List && ctx.json?.file_sizes instanceof List
source: |-
def names = ctx.json.file_names;
def sizes = ctx.json.file_sizes;
int len = Math.min(names.size(), sizes.size());
def attachments = new ArrayList(len);
for (int i = 0; i < len; i++) {
def attachment = new HashMap();
def file = new HashMap();
file.put('name', names.get(i));
file.put('size', sizes.get(i));
attachment.put('file', file);
attachments.add(attachment);
}
ctx.email = ctx.email ?: [:];
ctx.email.attachments = attachments;
When source data already delivers each attachment as a separate object (e.g., a JSON array of attachment objects), no zipping is needed — use rename or set with copy_from to place the array at email.attachments directly.
For non-ECS fields in fields.yml:
keyword for identifiers and exact-match stringsconstant_keyword for fixed values (dataset/module constants)long, double, scaled_float for metrics and numeric valuesdate / date_nanos for timestamps (date_nanos only when sub-millisecond precision is truly needed)ip for IP addressesboolean for true/false (avoid string booleans in pipelines)geo_point for lat/lon coordinatesgroup with nested fields for logical structure — no need to separately declare intermediate object nodesflattened for arbitrary key/value blobs with unknown keysnested for arrays of objects requiring per-object query isolation (heavier than group)text / match_only_text for full-text content; add a keyword sub-field via multi_fields when aggregation is also neededUseful properties on numeric fields: metric_type (gauge or counter), unit (e.g., byte, percent, ms), dimension for low-cardinality TSDB fields.
See references/mapping-type-matrix.md for the full type reference.
| Rule | DO | DON'T |
|---|---|---|
| Use snake_case | user_name, request_count | userName, RequestCount |
| Use lowercase | source_ip | Source_IP |
| No asterisks in names | network.bytes | network.* (literal asterisk) |
| Use groups for hierarchy | vendor.module.field as nested group | vendor.module.field as flat dotted name |
Field names must never contain literal * characters. An asterisk in a field name is almost always a copy-paste error from documentation or wildcard patterns. Use a group with known subfields or flattened for dynamic keys instead.
Both styles are valid in field files:
# Dotted (flat) — common for ECS fields in ecs.yml
- name: source.ip
external: ecs
# Nested group — common for custom fields
- name: acme.firewall
type: group
fields:
- name: rule_id
type: keyword
Pipeline expected output (*-expected.json) always uses nested object form regardless of how the source data represented the field. A source "host.name": "myhost" produces {"host": {"name": "myhost"}} in the output.
When source data contains literal dotted keys that Elasticsearch would otherwise expand, use dot_expander:
- dot_expander:
field: "*"
override: true
In pipeline test expected outputs, geo_point fields appear as objects with lat and lon keys:
"source": {
"geo": {
"location": { "lat": 51.5142, "lon": -0.0931 },
"city_name": "London",
"country_iso_code": "GB"
}
}
These sub-fields do not need entries in fields.yml — they are part of the geo_point type mapping. Only the *.geo.location field (type geo_point) needs to be in ecs.yml for non-standard parent prefixes where ecs@mappings (applied by stacks >= 8.13 at install time; guaranteed only when conditions.kibana.version floor is >= 8.13.0) does not cover it.
- set:
field: event.kind
value: event
- append:
field: event.category
value: web
- append:
field: event.type
value: access
- set:
field: event.outcome
value: success
if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code < 400"
- set:
field: event.outcome
value: failure
if: "ctx?.http?.response?.status_code != null && ctx.http.response.status_code >= 400"
on_failure:
- set:
field: event.kind
value: pipeline_error
When tests report field "destination.ip" is undefined for standard ECS fields:
_dev/build/build.yml exists at the package rootdependencies.ecs.reference is set (use git@v9.3.0)ecs.yml with external: ecsFix the root cause. Do not work around it by:
fields.yml without external: ecsexternal: ecs and defining ECS field types/descriptions manuallyException: Custom (non-ECS) fields reported as undefined must be defined in fields.yml.
_dev/build/build.yml — all ECS fields reported undefined; create with dependencies.ecs.referencebuild.yml — fields from newer ECS versions undefined; update reference to git@v9.3.0ecs.yml — field is undefined in test schema validation; add it to ecs.ymlexternal: ecs — descriptions and types diverge from ECS; always use external: ecs for ECS fields, with overrides as neededmetric_type on non-numeric field — lint errorgeo.* at document root — unmapped; always nest under a parent entityevent.category or event.type set as scalar — must use append processor, not setnested ECS field mapped as parallel arrays — email.attachments, threat.enrichments, and similar nested fields must be arrays of objects, not objects with parallel scalar arrays; see the Nested (array-of-objects) ECS fields section aboveentity.attributes.* / entity.relationships.* with external: ecs at git@v9.3.0 — these leaf fields do not exist at ECS v9.3.0 (attributes is a bare object; entity_reference fieldset is absent). They first appear at v9.4.0. Using external: ecs on them at git@v9.3.0 causes field is undefined errors. Packages with entity data streams must pin git@v9.5.0 in _dev/build/build.yml and match ecs.version: 9.5.0 in the pipeline. See the entity-mappings skill.elastic-package lint
elastic-package check
elastic-package test pipeline --data-streams <stream>
references/mapping-type-matrix.mdreferences/categorization-cheatsheet.mdreferences/root-and-core-fields.mdreferences/fieldset-links.mdreferences/cdr-field-requirements.md — cloud security / CDR integrations only (cloudsecurity_cdr category)entity-mappings/references/entity-field-catalog.md — ECS entity.* fields; entity/inventory data streams only