Terminal
Terminal/Skill
Replaces sensitive values in customer log files with placeholders and keeps the original format, so logs can be shared safely.
By Elastic
Works with
npx skills add elastic/integration-skills --skill anonymize-logsSanitize customer-provided logs so they are safe to share or commit. Replace identity only; keep the original structure, delimiters, quoting, field order, and line breaks.
Input may be a file or a single syslog / KV / NDJSON line. Output must match that shape — do not pretty-print, wrap, truncate, or “fix” values as part of sanitization.
| Input | How to provide |
|---|---|
| Log file(s) to sanitize | @-mention files or paste inline |
| Output location (optional) | free text path, defaults to same directory with .sanitized suffix |
| In-place override (optional) | say "in place" to overwrite the original |
Only replace sensitive values. Do not touch anything else.
Parsers and anyone reviewing the log depend on exact whitespace, delimiters, quoting, and line structure.
<134> (or any PRI) “to look more like syslog.”Do not replace as you read. Distinct originals that collapse to one placeholder, or a first-occurrence-only replace that leaves later copies, are both failures.
original value → placeholder for every sensitive token (including nested ones: URL query params, hex dwords inside compound IDs, path segments).Cover at minimum:
token_prefix, password_hash_prefix, hashed_token) — partial exposure still identifies the credentialuser@tenant.onmicrosoft.com), email subjects and body texthome_tenant_id, resource_tenant_id, aad_tenant_id variants), account IDs, subscription IDs, billing account IDs, org slugs embedded in paths or JSON fields, org unit paths (e.g. orgunit_path, org_unit_path), department names and IDs, cost center IDs, Windows SIDs (Security Identifiers) in pipe names, task names, or registry pathsdatabase.host, database.name, database_principal_name), Windows domain topology fields (domain controller hostnames, NT domain names, domain_controller_object_guid, domain_controller_object_sid)device_id / serial / similar fieldsC:\Users\alice\, /home/bob/, HKLM\...\S-1-5-21-...)Also cover nested and compound identifiers (below).
Apply placeholder conventions and shape rules (see below) as you go.
These are commonly missed. Include them in the map, including tokens nested inside URLs, compound IDs, and path segments.
IPv4 hidden in hex compound IDs. Some vendor IDs look like {0x00000001,0x1,0xcb007114,0x00000002}. A hex dword may be an IPv4 address (try both endiannesses). If you replace origin / src / dst IPs, decode each dword as IPv4. If either decoding is a plausible public/private IP and does not match the already-substituted origin, replace that dword with the hex encoding of the same documentation IP you used for origin (keep 0x + 8 hex digits). Keep the surrounding {0x…,0x…} shape; only the IP-bearing dword changes. Example: 0xcb007114 → 203.0.113.20 (big-endian); if origin was replaced with 198.51.100.10, use 0xc633640a.
Topology-encoded hostnames. Not a DNS name: SW-BLDG-01, CHI-FW-01, DC1-VPN-MGMT. In hostname / syslog-host / device-name fields, replace site/role/building codes. Do not replace vendor syslog message IDs like %NAME-6-302018 — those are product mnemonics, not hostnames. Placeholder: host-1.example.local or example-fw-01 matching original length class (short token vs FQDN).
Policy / ACL / access-group names. In policy, ACL, or access-group values, replace \d{8,14} date-like runs → 20240101000000 (same digit length); vs\d+ → vs01 (or drop vs/date suffix: example-policy); VLAN numbers in names → vlan100. Keep the token class (still looks like a policy name). Example: vlan200-20240115103000 → vlan100-20240101000000.
Interface names. ethN-MM.VLAN → eth1-01.100 (preserve eth + dotted subunit shape). IF-<org>-… / location words → IF-Example (keep an IF- prefix if present). Generic inside / outside / lan / wan may stay if they are interface roles, not site names. Prefer replacing when the token is a customer site or org.
Windows / SMB paths with org units. Replace directory components above the last two levels with generic segments (example-department, E01, EXAMPLE-UNIT). Keep depth, backslashes, and the final filename (Thumbs.db) if it is a generic OS file. Year folders like \2026\ can stay (not identifying).
Prefix-preserving opaque IDs. SaaS and identity-platform IDs often have a short type prefix plus a long mixed-case suffix. Replace the suffix, keep the prefix. Same original → same placeholder. Also replace base64 blobs in URL paths with same-length synthetic base64. Replace real-looking app/role display names with generic equivalents (Example Application Role Management).
Device serials and MDM IDs. Long hex/alnum values in device_id / serial / DeviceId (and similar) → synthetic same charset/length (SN000000000001, SEC123ABCDEFG45H). Do not leave a real device or MDM serial.
Identity-store names and group hierarchies. <NAME>-AD / identity store → EXAMPLE-AD. Hash-delimited hierarchies: keep structural prefixes such as Location#All Locations# / Device Type#All Device Types#; replace the leaf if it is a real city/site (Example-Site). Real-looking store names even after username sanitization must still be replaced.
IPs nested in URLs and query strings. After replacing top-level src_ip / dst_ip, scan URL query and path for ip=, src=, dst=, and dotted quads. Use the same RFC 5737 IP already used for that host if it appeared elsewhere; else 198.51.100.10. Do this even when the surrounding URL is malformed.
OCSP / AIA / CRL certificate paths. http://ocsp.digicert.com/AAAAAAAAAAAAAAAAAAAAAAAAAAAA — host may stay (public CA), path is cert serial/issuer material. If hostname matches ocsp. / crl. / cacerts. (public or not), replace the path with a same-length synthetic base64/hex string (zeros or A). Keep / count and charset class.
JNDI / interpolation probes in usernames. If a field contains ${jndi: or ${lower: or ${::-, replace the entire field value with a synthetic username (alice.johnson, or Lab1's User if the original quoting/apostrophe should be preserved). Do not leave probe URLs. Preserve surrounding quoting.
Public destination hostnames in customer telemetry. Public hosts (docs.google.com, api.github.com) can stay when they are the log-source vendor API. In firewall, proxy, DNS, UTM, and content-filter events, destination hostnames and destination IPs are customer telemetry. Replace them even if they are well-known consumer sites (youtube.com) or public unicast IPs. Keep product hosts that are the log source vendor API (api.github.com in a GitHub audit log).
Confirm after sanitization:
ip= / src= / dst= / dotted quads, hex dwords inside compound IDs, host-like topology tokens, device serials, prefix-preserving opaque IDspython3 -c "
import json, sys
with open('FILE') as f:
for i, line in enumerate(f, 1):
line = line.strip()
if line:
try: json.loads(line)
except Exception as e: print(f'Line {i}: {e}')
"
Use consistent, realistic-looking replacements — not REDACTED strings, which break format-sensitive parsers.
| Type | Replacement |
|---|---|
user@example.com, admin@example.org | |
| IPv4 | RFC 5737 ranges: 198.51.100.10, 203.0.113.20, 192.0.2.30 |
| IPv6 | 2001:db8::10 |
| Hostname / FQDN | host-1.example.local, srv-web-01.example.internal |
| Topology hostname | host-1.example.local |
| Domain | example.com, example.org, example.net |
| UUID | 89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b |
| API key / token | sk_test_example_key_1234567890, dGVzdC10b2tlbi0xMjM0NTY3ODk= |
| Username | alice.johnson, bob.smith |
| Display name | Alice Johnson, Bob Smith |
| Org / company name | Example Corp, Acme Inc |
| Account / tenant ID | 000000000000, example-tenant-id |
| Cloud resource ID | arn:aws:iam::000000000000:user/example-user |
| S3 bucket name | example-bucket |
| MAC address | 00-00-5E-00-53-23 (RFC 7042 documentation range) |
| Serial number | SN000000000001 |
| Device / machine ID | use a synthetic UUID or device-id-example-000001 |
| MDM / vendor-prefixed device ID | same charset and length (SEC123ABCDEFG45H) |
| Windows SID | S-1-5-21-000000000-000000000-000000000-1000 |
| File path (Windows) | C:\Users\example-user\AppData\... |
| File path (Unix) | /home/example-user/... or use ~ |
| Kubernetes cluster | example-cluster, example-node-1 |
| Phone number | 734-555-0100 (555 range is reserved for fiction) |
| Database host / name | db-host.example.local, example_database |
| Department / org unit | example-department, /example-org/example-unit |
| Hashed / partial token | replace with full synthetic token of same format |
| DHCP fingerprint | example-dhcp-fingerprint-000001 |
| JA4 fingerprint | replace with same-length hex string |
| Transaction / sequence / event-instance ID (numeric) | synthetic integer matching the original digit count — e.g. 48273915 → 10000002 |
| Session / request / correlation ID | same-length synthetic string (preserve length and charset), not a descriptive name |
| Hex dword in a compound ID | 8 hex digits, 0x prefix; encode the same RFC 5737 IP used for origin / src |
| Policy / ACL with date | example-policy, vlan100-20240101000000 |
Interface ethN-MM.VLAN | eth1-01.100 |
| Opaque ID with type prefix | keep prefix, synthetic suffix |
| Identity store | EXAMPLE-AD |
| Location hierarchy leaf | Example-Site |
| OCSP / CRL / AIA path | same-length A/0 base64 |
| JNDI username | alice.johnson (preserve quotes) |
Consistency rule: map identical original values to identical placeholders throughout the file. If the same IP appears 10 times, it must become the same replacement IP all 10 times — so cross-event correlations remain testable. Build the full map first, then apply it globally. Distinct originals must get distinct placeholders — do not collapse 30 usernames into alice.johnson. Nested copies (the same IP in src, in a URL ip=, and in a hex compound ID) must all use that original’s placeholder.
Every replacement must have the same shape as the original value. Parsers and downstream processing depend on value format, not just field presence.
/d/123/edit → /d/456/edit, not /d/example-document-id/editcn1=48273915 cn2=3061847 → cn1=10000002 cn2=1000002ip= / src= / dst=) — preserve the scheme, path structure, query string shape, and any illegal characters
https://docs.google.com/drawings/d/123/edit → https://docs.google.com/drawings/d/000000000000/edit (replace the ID, not the host — docs.google.com is a public service name, not an org identifier; this applies to vendor API logs)https://internal.corp.com/api/v1/resource → https://host-redacted.example.local/api/v1/resource (replace the internal hostname, keep the path)https://example.com/path or /some/apidocs.google.com, api.github.com, s3.amazonaws.com) identify a service, not an organization, in SaaS/vendor API logs and do not need to be replaced thereMalformed or garbage values must not be replaced. If a value looks broken, synthetic, or contains no real identifying information (e.g. http://1=Y +z\\, 00/00/0000, N/A, empty strings, placeholder-looking values), leave it exactly as-is. Replacing a malformed value with a well-formed placeholder changes the shape. You may still replace embedded identity inside a malformed token (real IP, hostname, user) as long as the malformation survives — extra quotes, unencoded { }, spaces in URLs, truncated fields, U+FFFD.
If you are unsure what shape to use, look at neighbouring values of the same field type in the same file and match their format.
Do not replace:
ALLOW, DENY, INFO, ERROR)%NAME-6-302018) — product mnemonics, not hostnamesJun 27 14:06:33.189 JST to Jul 24 00:47:24.618 CETJST, EDT, EST, Eastern, (GMT-3:00)Buenos Aires, Georgetown with another zone. Date parsers key off the abbreviation; swapping JST→CET both breaks parse and still encodes a geography (the wrong one). Privacy for timezone abbreviations is out of scope — leave them as-isdocs.google.com, api.github.com, etc.) when they are the log-source vendor API — replace the path ID if it is sensitive, not the host.
In firewall, proxy, DNS, UTM, and content-filter events, destination hostnames and destination IPs are customer telemetry. Replace them even if they are well-known consumer sites (youtube.com) or public unicast IPs. Keep product hosts that are the log source vendor API (api.github.com in a GitHub audit log).Mozilla/5.0 ...) — these reveal browser/OS type but not identity; safe to keep