Skip to content
Docs

Job actions

In jobs, various actions are available to perform simple and advanced data transformation and manipulation. The actions configured in a Job are executed sequentially, after Job input, and before Job output.

Terminology

Section titled “Terminology”

Data which enters and flows through Jobs are referred to as event data. For example, when reading multiple lines from a log file, each line’s text enters the Job as a single event. Each input determines how input data is structured into discreet events.

Events are typically represented as JSON within Jobs. Actions typically operate on fields within an event’s JSON representation. For example, the basic add, remove and rename actions are field-level transformations.

Concept Mappings

Section titled “Concept Mappings”

E.g. ‘I want to use regular expressions’:

  • extract: extract fields using a regular expression with groups
  • extract: substitute text using output-pattern
  • filter: pass through events if field values match (how:patterns) or not (how:exclude)

Some people might be aware of extract as grok in other systems.

Can arrive at filter from more direct questions like ‘Only pass through certain events?’, ‘Enforce schema?’.

Multiple ways of accessing this information. For instance, a FAQ is a good old-fashioned way, very searchable.

‘I want to reshape/manipulate events’

  • add: new fields - can overwrite. The new values may contain variable expansions.
  • copy: new fields copied from other parts of the event, using JSONPath notation.
  • script: new fields initialized using Lua expressions.
  • remove: existing fields. May choose to complain if not present.
  • flatten: remove any ‘nested’ objects in the event, force it to be flat.

So there’s a number of words like ‘reshape’ or ‘manipulate’ which point to the same need.

‘I want to convert text into numbers’

  • convert: specify fields and the desired type, e.g. ‘num’. Can just set auto to get a reasonable guess.
  • csv: can specify ‘num’ etc. when defining columns. There is autoconvert.
  • extract: can optionally specify this with convert parameter.

Conversion is a very broad concept, including time! convert is more UI discoverable than before, since conversions presents a drop-down of the conversion options, rather than having to remember the magic words. (There is also units similarly for the question ‘How do I convert units?’ )