Terminal
Terminal/Skill
Researches a vendor product's data sources, fields and ECS mappings, and writes a research brief for building an Elastic integration.
By Elastic
Works with
npx skills add elastic/integration-skills --skill research-integrationYou are the research orchestrator. Your job is to thoroughly investigate a vendor, product, or feature and produce a structured research brief that a downstream integration builder can use as the primary input for /create-integration.
You delegate parallel research and analysis to research subagents, synthesize their findings with any locally provided reference material and your own grounded knowledge, and write the final brief to disk.
Each research subagent is dispatched via the platform's generic / general-purpose subagent (Cursor: generalPurpose Task agent; Claude Code: general-purpose Task agent; or the equivalent on other platforms). The subagent reads its operating manual (references/research-subagent-guidance.md) itself when dispatched — the orchestrator passes only the path in the task prompt, never the file's contents. See "Before you start" below.
Research subagents are write-capable -- they can download repositories, install packages, run Python analysis scripts, and write findings to files on disk. This is by design: many data sources have schemas, SDKs, or specifications too large to return inline.
Include any combination of the following when you invoke this command.
Use @-mentions for files/folders and paste links inline.
| Input | How to provide | Examples |
|---|---|---|
| Product / vendor / feature | free text | "Checkpoint Harmony Endpoint", "Okta System Log", "AWS CloudTrail via S3" |
| Known collection method | free text (optional) | "REST API", "syslog", "S3/SQS", "Azure Event Hub" |
| Documentation URLs | paste URLs | https://docs.vendor.com/api/v2, https://docs.vendor.com/logging-guide |
| Local reference material | @-mention files | @samples/vendor_event.json, @notes/vendor-api-notes.md |
| Scope constraints | free text | "only the alerts API", "focus on firewall logs", "audit events only" |
| Output name override | free text | "checkpoint_harmony" (defaults to sanitized product name) |
Anything typed after /research-integration is your research goal.
/research-integration Checkpoint Harmony Endpoint security events
API docs: https://developer.checkpoint.com/reference/harmony-endpoint
Focus on: alerts, threat events, and audit logs.
Known method: REST API with pagination.
/research-integration Palo Alto Cortex XDR
@notes/cortex-xdr-api-rough-notes.md
Need to investigate both the Incidents API and Alerts API.
/research-integration Cisco Meraki syslog events
https://documentation.meraki.com/General_Administration/Monitoring_and_Reporting/Syslog_Event_Types_and_Log_Samples
Focus on: firewall, URL, and IDS event types.
Known method: syslog over UDP/TCP.
/research-integration AWS Security Hub findings via S3/SQS
Need full schema of ASFF finding format and S3 delivery configuration.
Read these reference files from this skill's directory to guide your research strategy:
references/data-collection-methods.md -- understand input types and what to investigate for eachreferences/research-output-template.md -- the structure your final brief must followreferences/api-research-checklist.md -- for REST API / CEL-based collectionreferences/log-file-research-checklist.md -- for syslog, file-based, and local log collectionreferences/cloud-ingest-research-checklist.md -- for S3/SQS, Event Hub, Pub/Sub, and similar cloud deliveryreferences/test-api-script-spec.md -- specification for the API test script generated in Phase 7If the collection method is unknown at invocation time, read all three checklists -- part of your job is to determine the method.
Also load:
ecs-field-mappings skill -- for ECS field mapping guidance during the analysis phaseentity-mappings/references/entity-datastream-classification.md -- read this yourself so you can classify each proposed data stream as event vs entity (apply the research-time rule in that file). Do NOT load the rest of the entity-mappings skill — the field catalog and pipeline patterns are implementation material and must not be loaded during research.references/market-coverage-checklist.md -- read this yourself so you know what to pass through; when dispatching the Track E subagent, point it at this file by path (do NOT paste its contents into the task prompt). The Track E subagent will read it in its own fresh context.references/research-subagent-guidance.md -- the operating manual every research subagent needs. Do NOT read this file yourself unless you specifically need to debug a subagent's behaviour. Instead, point every research subagent at this file by path in its task prompt and instruct it to read the file end-to-end before doing any other work. Embedding the file verbatim doubles its context cost.Do not load other integration-building skills (CEL, pipelines, ecs-field-mappings implementation details, entity-mappings field catalog, etc.). Those are for implementation, not research.
Downstream tooling note: The research workflow itself has no CEL tooling requirements. However, if the recommended collection method turns out to be API-based (CEL input), the downstream /create-integration run will need mito, celfmt, ceplx, and stream installed. Surface this in the Phase 8 summary so the user can verify their toolchain before switching to build mode.
Write all research output to:
research_results/<product_slug>/
Where <product_slug> is a lowercase, underscore-separated identifier derived from the product name (e.g., checkpoint_harmony_endpoint, palo_alto_cortex_xdr, cisco_meraki). The user may override this with the "Output name override" input.
Create this directory structure:
research_results/<product_slug>/
research-brief.md # the main structured research brief
test-api.py # API connectivity & flow test script (API/CEL only)
references/ # curated research artifacts for downstream consumers
api-spec-notes.md # API endpoint details, request/response examples (if API)
log-format-notes.md # log format details, sample lines (if log-based)
field-schema-analysis.md # detailed field inventories written by subagents
market-coverage.md # detailed market coverage analysis (always created)
entity-coverage.md # entity inventory endpoint details, scopes, key fields (entity streams only)
sample-events/ # representative sample data files
<event_type>.json # one file per event type or data format variant
<event_type>.log
temp/ # downloaded raw artifacts (repos, SDKs, schemas, scripts)
<descriptive-subfolder>/ # e.g., vendor-sdk/, schema-files/, openapi-spec/
ecs-mapping-analysis.md # initial ECS field mapping analysis
entity-mapping-analysis.md # entity.* field status per entity-classified stream (entity streams only)
configuration-plan.md # planned integration configuration variables
Not all files are required -- create only what applies to the product's collection method.
Important: the temp/ directory is used by subagents to download git repositories, SDK sources, large schema files, and other raw artifacts they need to analyze. Do not delete temp/ after research completes -- it serves as a reference for the human and may be useful for follow-up work.
@-mentioned local files.Launch multiple research subagents in parallel using the platform's generic / general-purpose subagent (see the dispatch description at the top of this skill). Each subagent focuses on a specific research track. You should launch as many parallel subagents as makes sense for the product -- typically 2-4 subagents, plus the always-on Track E, and Track F when any data stream was classified as entity in Before you start.
IMPORTANT -- subagent context and capabilities:
@-mentioned files directly. Include any relevant content from local reference files and fetched URLs in the task prompt.research_results/<product_slug>/) so it can write to temp/ and references/ within it.temp/ under the working directory.references/ or temp/ and return a concise summary with file paths rather than returning everything inline. This keeps context manageable.Required structure for every research subagent task prompt:
references/research-subagent-guidance.md (relative to the research-integration skill) end-to-end before doing any other work. That file is the subagent's operating manual — methodology, temp/ usage, Python analysis idiom, result delivery contract, quality standards, and anonymization conventions. Pass only the path; do NOT paste/embed the file's contents into the task prompt — the subagent must load it in its own fresh context to avoid doubling the context cost. Track E follows the same pattern for the market coverage checklist.Working directory: research_results/<product_slug>/
- Download raw artifacts to: research_results/<product_slug>/temp/
- Write curated findings to: research_results/<product_slug>/references/
@-mentions (the subagent cannot see your conversation).Instruct the subagent to investigate:
Provide: product name, vendor, any known collection method, any documentation URLs.
Instruct the subagent to investigate the specifics of the data source based on the most likely collection method:
For APIs:
api-research-checklist.md for the detailed OAuth2 investigation checklist.temp/ and use Python to extract endpoint details, request/response schemas, and parameter definitionsFor logs/syslog:
For cloud ingest (S3/SQS, Event Hub, Pub/Sub, etc.):
temp/ and analyze the schemas programmaticallyProvide: product name, likely collection method, any documentation URLs, any local reference material content.
Instruct the subagent to investigate:
For data sources with large schemas: Instruct the subagent to download the schema source (git repo, SDK package, JSON schema file) into temp/ and use Python to programmatically extract field inventories, type information, and enum values. The subagent should write the complete field analysis to references/field-schema-analysis.md (or multiple files if per-event-type breakdowns are needed) and return a summary.
Provide: product name, any documentation URLs, any sample data content from local files.
Instruct the subagent to investigate:
Provide: product name, collection method, any documentation URLs.
Always launch this track in parallel with the other tracks. It is not conditional on collection method.
Instruct the subagent to check whether IBM QRadar, Splunk, and Sumo Logic have an existing integration or app for the product being researched, and to document what each covers and how it collects data.
The subagent must follow references/market-coverage-checklist.md end-to-end. Point the subagent at that file by path and instruct it to read the entire file before doing any other work. Do NOT paste the checklist contents into the task prompt — the subagent will load it in its own fresh context. (This is in addition to the read-references/research-subagent-guidance.md-by-path directive from Phase 2.)
Platform catalog starting points to include in the prompt:
https://www.ibm.com/products/qradar-siem/integrationshttps://splunkbase.splunk.com/appshttps://www.sumologic.com/help/docs/integrations/For each platform, the subagent must determine:
Output: write all findings to references/market-coverage.md using the structure defined in the checklist (summary table → per-platform H2 sections → comparison notes). Return a concise inline summary with which platforms have integrations, the dominant collection method found, and the path to the written file.
Provide: product name, vendor name, common aliases or abbreviations for the product, and the path to references/market-coverage-checklist.md (so the subagent reads it itself — do not paste the checklist content into the prompt).
Launch this track only when you classified at least one proposed data stream as an entity stream during ## Before you start. Run it in parallel with the other tracks.
Instruct the subagent to investigate each entity-classified stream:
var list and the research brief's section 6 (Configuration plan).id, login, email). If none exists, document the composite key.entity.attributes.* (e.g. hasTwoFactorEnabled, siteAdmin, managedBy, roles, permissions).since / updated_after / modified_since parameter? If yes, document the parameter and its semantics. If no, document that the endpoint must be re-called in full each cycle.Output rules for the Track F subagent:
entity.* field names. Only fields that exist in ECS at the pinned version are valid. For source fields with no ECS equivalent, flag them as custom field candidates under <integration_name>.<datastream_name>.<field_name> — do not place them under any ECS prefix.references/entity-coverage.md with one section per entity-classified stream.Task prompt must include (in addition to the research-subagent-guidance.md read-by-path directive):
entity-mappings/references/entity-datastream-classification.md so the subagent can apply the research-time classification signals when assessing each endpoint. Pass the path only; do not embed the file contents.After all subagents return:
references/ or temp/ and returned only summaries. Read the files they reference to get the full picture. The subagent summaries will tell you which files to read and when.[UNVERIFIED] marker.sample-events/ subdirectory.temp/. You can inspect these directly if you need more detail than what the subagent summaries and reference files provide.references/market-coverage.md (written by Track E). Extract the summary table and overall comparison notes — these are used directly in section 1.5 of the research brief.Using the ECS reference skill loaded earlier, perform an initial field mapping analysis:
For each identified field from the product's data, determine:
Identify which event.kind, event.category, event.type, and event.outcome values apply to each event type.
Note any fields that are strong candidates for related.ip, related.user, related.hosts, or related.hash enrichment.
Write the analysis to ecs-mapping-analysis.md.
For each entity-classified stream (those with event.kind: asset, identified in ## Before you start): using the field data from Track F, produce:
The represented entity.type (from the 12 allowed values listed in entity-mappings/references/entity-datastream-classification.md — already loaded during ## Before you start).
The source field that maps to entity.id (the stable primary key from Track F).
A per-field status table for each ECS entity.* field the Track F findings touch:
| ECS field | Status | Source field | Notes |
|---|---|---|---|
entity.id | ✅ direct | id | stable across cycles |
user.entity.attributes.mfa_enabled | 🔄 derived | hasTwoFactorEnabled | boolean conversion needed |
entity.relationships.* | 🔍 investigate | — | vendor docs ambiguous |
entity.attributes.permissions | ⛔ N/A | — | no per-user permission list in API |
Status key: ✅ direct mapping · 🔄 derived/transformed · 🔍 needs investigation · ⛔ not available.
Write this to entity-mapping-analysis.md alongside ecs-mapping-analysis.md. Findings only — no processors, no field YAML, no external: ecs declarations. Those belong in the build phase.
Based on the identified collection method, plan the integration configuration:
references/data-collection-methods.md for the standard variables per input type.configuration-plan.md.Compile the full research brief following the template in references/research-output-template.md. Write it to research_results/<product_slug>/research-brief.md.
The brief must be self-contained -- a reader should be able to use it as the sole input to /create-integration and have everything they need.
When populating section 1.5 (Market Coverage), use the summary table extracted from references/market-coverage.md in Phase 3 step 8. Include the one-line summary paragraph and the three-row platform table inline, then add a reference pointer: See references/market-coverage.md for full per-platform analysis.
Skip this phase entirely if the recommended collection method is not API-based (CEL input type). This phase only applies when the research has identified a REST API as the collection method.
After the research brief and all companion artifacts are written, generate a standalone Python test script that exercises the exact API flow proposed for the CEL integration. This lets a human validate connectivity, authentication, pagination, and response structure against a real (or mock) API before any Elastic Agent work begins.
Read the specification: Load references/test-api-script-spec.md from this skill's directory. It defines every requirement for the script in detail — file structure, CLI arguments, output files, error handling, and the relationship to the proposed CEL program.
Gather inputs from earlier phases. The script is synthesized from research already completed:
configuration-plan.mdWrite the script to research_results/<product_slug>/test-api.py. Key requirements (see spec for full detail):
urllib.request, json, logging, argparse, ssl, etc. No third-party dependencies.argparse with default=os.environ.get(...).https://...), even if the vendor has a single static URL. This enables pointing at mock servers.--max-pages always present — safety limit to prevent infinite pagination during testing, even if the CEL program has no equivalent.test-api.log — verbose log (superset of stdout, written via Python logging)trace.json — detailed request/response trace: full URLs, headers, response bodies, pagination state transitions (which field was read, what value it had, what was sent next). Auth values redacted..tar.gz and print the path with instructions to share it with integration maintainers.KeyboardInterrupt handled gracefully; exit 0 on success, 1 on failure.Mirror the proposed CEL flow. The script's request sequence, pagination logic, and termination conditions must match what was described in the research brief for the CEL program. This is the core value of the script — if it works, the CEL program should work too.
test-api.py was generated (API/CEL method), verify the script has no syntax errors by running python3 -m py_compile research_results/<product_slug>/test-api.py.test-api.py was generated: remind the user to run it against the real API (with credentials) and share the resulting archive back for developmentmito, celfmt, ceplx, and stream are installed before running /create-integration (see scaffold-commands.md Preconditions in the create-integration skill)/create-integration with the brief)[CONSTRUCTED EXAMPLE].[UNVERIFIED] for any detail that could not be confirmed from official sources.next_cursor field in the response body; pass it as the cursor query parameter" is useful.rate_limit() usage, state management approaches, or error handling strategies for the CEL program. The CEL builder agent has its own skills with authoritative patterns. Research output that prescribes CEL implementation details will be ignored or — worse — followed incorrectly, overriding the CEL skill's patterns.
next_cursor with more_to_read boolean. Terminate when more_to_read is false."want_more: body.more_to_read and store the cursor in state.?cursor.next_cursor."X-Ratelimit-Limit, X-Ratelimit-Remaining, X-Ratelimit-Reset."rate_limit() CEL function to parse these headers and propagate the result on every branch."fields/*.yml, or manifest.yml should be authored. The pipeline builder and reviewer skills (ingest-pipelines, ecs-field-mappings, package-spec, review-integration) are the authoritative source for those decisions. Recommendations about processor choice, error-handling structure, or pipeline-level configurability will be ignored or followed incorrectly.
preserve_duplicate_custom_fields flag (legacy pipeline anti-pattern, prohibited by ingest-pipelines/SKILL.md), event.ingested (managed by Elasticsearch), trailing event.original removal toggles, and the preserve_duplicate_custom_fields manifest variable / tag / conditional. Never include these as configuration variables, recommended pipeline behaviors, or "consider supporting…" suggestions, even if they appear in legacy integrations you examined for reference patterns. The only preserve_* config var that is valid is preserve_original_event (file/syslog inputs only); see the standard-var tables in references/data-collection-methods.md.srcip and source.ip for the same value; the latter is already ECS-compliant."preserve_duplicate_custom_fields manifest var so users can keep both srcip and source.ip populated."date processor with target_field: event.start and a fallback to @timestamp via on_failure."references/data-collection-methods.md. Do not propose additional configuration variables outside that authoritative set unless the vendor's API genuinely requires a new product-specific variable (e.g., a tenant ID for a multi-tenant API). Even then, the variable must be tied to a documented vendor-side requirement, not a pipeline behavior toggle.<integration_name>.<datastream_name>.<field_name> (e.g., github.members.is_employee for a GitHub members stream). Never place a non-ECS field under entity.attributes.*, user.*, host.*, or any other ECS prefix as if it were a real ECS field — if ECS adds a conflicting field later, the integration breaks silently.
isEmployee has no ECS equivalent → github.members.is_employee (custom field)."isEmployee → entity.attributes.is_employee."siteAdmin has no ECS equivalent → github.members.site_admin (custom field). Consider user.roles if the value can be expressed as a named role."siteAdmin → entity.attributes.site_admin."entity-mappings and ingest-pipelines skills are the authority on those decisions.
membersWithRole returns hasTwoFactorEnabled per member — a candidate for user.entity.attributes.mfa_enabled."convert processor with type: boolean targeting user.entity.attributes.mfa_enabled."role as a string — a candidate for user.roles (an array of user role objects, may need splitting)."foreach processor over role.split(',') to populate user.roles."httpjson as an input type. The httpjson input is deprecated in favour of CEL. For any REST/HTTP API, recommend cel regardless of how simple the API is or what input type existing streams in the same package use. An existing httpjson stream in the same package is a legacy pattern, not a template. Do not reference it as justification for using httpjson in new streams.After this command completes, continue with:
test-api.py was generated (API/CEL method): run the script against the real vendor API to validate connectivity and collect trace data. Share the resulting .tar.gz archive back — the trace file is valuable input for CEL program development and pipeline testing./create-integration @research_results/<product_slug>/research-brief.md to build the integration using the research brief as input.research_results/<product_slug>/references/sample-events/ via @-mentions.