S3

S3 (s3)

Write events to a S3 bucket.

Contents

• Fields
• Authentication
• Location
• Object Properties

Fields

Field	Type	Required	Description
retry	Retry		How to retry failed requests.
batch	Batch		Batching input events together.
input-field	event-field (string)		Use the specified field as the content for the object.
bucket-name	string	✅	Bucket Name.
object-name	s3_output:object-name	✅	Name of object, either as text or from a field.
mode	Mode		Put or delete the object? Default is Put.
preprocessors	Preprocessors		Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified.
track-schema	boolean (bool)		Check the schema of the written data and update __SCHEMA_NUMBER (written data must be JSON).
scuba	Scuba		If this Job is sending data to Scuba Lite, optionally have the Job configure the Scuba Lite instance.

Authentication

Authentication

Field	Type	Required	Description
access-key	string		Access Key ID.
secret-key	string		Secret Key ID.
security-token	string		Security Token.
session-token	string		Session Token.
role-arn	string		A Role ARN for assuming role using above credentials.

Location

Location

Field	Type	Required	Description
region	string		S3 Region.
endpoint	string		S3 Endpoint, if needed.

Object Properties

Object Properties

Field	Type	Required	Description
disable-object-name-guid	boolean (bool)		Disable the GUID prefix if you want object name to be treated literally (off for deletes).
guid-prefix	string		GUID Prefix, will be prepended to the GUID, the default value is ”/”.
guid-suffix	string		GUID Suffix, will be appended to the GUID if specified.

Retry Fields

Field	Type	Required	Description
count	integer		How to retry? Either forever or for a limited number of times.
pause	string		How long to pause before re-trying.

Batch Fields

Field	Type	Required	Description
fixed-size	integer		maximum number of events in an output batch.
mode	symbol	✅	If ‘document’ send on end of document generated by input. If ‘fixed’, use fixed_size.
timeout	string	✅	interval after which the batch is sent, to keep throughput going (default 100ms).
header	string		put a header line before the batch.
footer	string		put a header line after the last line of the batch.
use-document-marker	bool		Enrich the job metadata with a document marker (for document handling in batch mode).
wrap-as-json	bool		Format the output batch as a JSON array.

Scuba Fields

Field	Type	Required	Description
host	string	✅	The Scuba Lite instance (for example: https://scuba-lite.example.com).
insecure	bool	✅	Allow self-signed certificates for Scuba Lite connections.
token	string	✅	The Scuba Lite API token.
table-name	string	✅	The Scuba Lite table name.
file-pattern	string	✅	The Scuba Lite file search pattern.
columns	object	✅	The columns specified by name and type.

Mode Options

Value	Name	Description
put	put	Put Objects
delete	delete	Delete Objects

Preprocessors Options

Value	Name	Description
gzip	gzip	Gzip the output data
parquet	parquet	Extract the received data as JSON rows from a parquet file
base64	base64	Decode base64 as binary