S3

S3 (s3)

Stream data from a S3 Object.

Contents

• Fields
• Authentication
• Location
• Object Properties

Fields

Field	Type	Required	Description
trigger	trigger		How often to run the command.
mode	Mode		The operating mode for this input.
ignore-linebreaks	boolean (bool)		Treat object as one event.
preprocessors	Preprocessors		Preprocessors (process downloaded data before making it available to the job) these processors will be run in the order they are specified.
timestamp-mode	Timestamp Mode		Derive a timestamp for this object for filtering purposes based on the selected strategy.
maximum-age	string		Remove any objects older than this many seconds from the candidate list.
fingerprinting	boolean (bool)		Enable object fingerprinting, which will cause an object to only be downloaded once.
maximum-fingerprint-age	duration (string)		Remove any object fingerprints older than this from the tracker.
include-regex	string		Include objects matching the specified regular expressions.
exclude-regex	string		Exclude objects matching the specified regular expressions.
retry	Retry		Timeout and Retry.

Authentication

Authentication

Field	Type	Required	Description
access-key	string		Access Key ID.
secret-key	string		Secret Key ID.
security-token	string		Security Token.
session-token	string		Session Token.
role-arn	string		A Role ARN for assuming role using above credentials.

Location

Location

Field	Type	Required	Description
bucket-name	string	✅	The storage service container for created objects.
object-names	string	✅	The object names. When using list modes these are treated as search prefixes.
region	string		S3 Region.
endpoint	string		S3 Endpoint.

Object Properties

Object Properties

Field	Type	Required	Description
object-name-field	event-field (string)		The field that the object name from an operation should be stored in.
creation-time-field	event-field (string)		The field that the object creation time should be stored in.
last-modified-field	event-field (string)		The field that the object last modified time should be stored in.
content-length-field	event-field (string)		The field that the object content length information should be stored in.
content-type-field	event-field (string)		The field that the object content type information should be stored in.
etag-field	event-field (string)		The field that the object ETag should be stored in.
data-field	event-field (string)		A field that the object data should be nested in.

Retry Fields

Field	Type	Required	Description
count	integer		How to retry? Either forever or for a limited number of times.
pause	string		How long to pause before re-trying.

Mode Options

Value	Name	Description
list-and-download-objects	list-and-download-objects	List Objects and Download
list-objects	list-objects	List Objects
download-objects	download-objects	Download Given Objects

Preprocessors Options

Value	Name	Description
extension	extension	Preprocess the object or blob based on the extension of the object or blob name (.gz, .parquet)
gzip	gzip	UnGzip the received data
parquet	parquet	Extract the received data as JSON rows from a parquet file
base64	base64	Encode the binary data as base64

Timestamp Mode Options

Value	Name	Description
none	none	The default mode, do not filter based on timestamps
last-modified	last-modified	Filter object on the last-modified timestamp reported by the service
blob-name-pattern	blob-name-pattern	Filter blobs on the timestamp derived from the object name for example: relevant-name-pattern: =(?P<Y>[\\d]{4,4})-(?P<m>[\\d]{2,2})-(?P<d>[\\d]{2,2})/