Processing pipelines

Open Usage and settings and the Processing pipelines editor. An admin can create an ordered pipeline for logs, traces or metrics, choose a filter, edit its typed processors and preview sample or recent stored data before saving. There are at most 50 pipelines per signal and 20 processors per pipeline. Changes affect new input after configuration publication; they do not reparse historical records.

Safe workflow

1. Start with a small service filter and representative samples, including malformed and secret-shaped values.
2. Use the processor forms for parsing, attribute changes, redaction, filtering, retention and supported metric derivation. Unknown processor types/fields are rejected on save.
3. Run the preview. Inspect each processor’s diagnostics, match counts and resulting records. A preview does not retain or bill a new sample.
4. Check typed fields, service/environment identity, trace IDs and metric labels. Keep low-cardinality labels for log-derived metrics.
5. Save, send a fresh record, then inspect the explorer and ingest outcomes. Disabled pipelines do not run. Changes require a published configuration revision, so do not assume an immediate global switch.

Default redaction runs before customer processing on bodies, string attributes and exceptions. Card, authorization, bearer, AWS-secret-shaped and password-key defaults protect common credentials. Email and IP policies are configurable. No pattern library guarantees removal of all personal information; avoid collecting it and test your own payloads. Do not copy real secrets into a parser sample.

API and processor reference

• GET /telemetry/pipelines: list.
• POST /telemetry/pipelines: create.
• PUT /telemetry/pipelines/{id}: replace configuration.
• DELETE /telemetry/pipelines/{id}: remove, 204 with no response body.
• POST /telemetry/pipelines/test: preview a draft against bounded samples or a recent range.

A configuration contains signal, name, enabled, nonnegative position, a query-language filter and typed processors. The test request contains pipeline, samples and optional recent: {from,to}. Use the editor’s JSON representation and generated API schemas for the exact processor variant and required properties rather than a legacy parser payload.

Supported parser choices include JSON, regex with named captures, grok, common access-log templates, syslog, CSV and key/value formats where offered by the editor. Processors can also set/rename/delete attributes, remap severity or service, parse time, redact, exclude/sample, set supported trace-retention policies and derive supported log metrics. A restrictive retention or exclusion rule is a data-loss decision. Test before enabling it and keep an export when required.

Legacy /log_parsers is not a pipeline endpoint, and parsers created there are not converted to pipelines automatically. Recreate each one in the pipelines editor, and do not POST a legacy rule body to /telemetry/pipelines.