AWS ALB logs from S3

Use Settings → Sources → Add S3 source to import AWS Application Load Balancer access logs. Two ALBs may share one source when they write to the same bucket and region. Give each ALB a separate, non-overlapping prefix.

Before you start

You need owner or admin access, a plan with native sources, a destination stream, and an S3 bucket where ALB access logging is already enabled. Fluxtail reads .log and .log.gz files. Other filenames are ignored.

Create a dedicated IAM identity restricted to the chosen bucket and prefixes. Avoid root keys and administrator policies. The permissions are s3:ListBucket, s3:GetObject, and, for a versioned bucket, s3:GetObjectVersion. A bucket policy can also restrict access. Do not grant Fluxtail permission to upload or delete files.

For example, replace YOUR_BUCKET, alb-one/, and alb-two/ in this IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR_BUCKET",
      "Condition": { "StringLike": { "s3:prefix": ["alb-one/*", "alb-two/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": ["arn:aws:s3:::YOUR_BUCKET/alb-one/*", "arn:aws:s3:::YOUR_BUCKET/alb-two/*"]
    }
  ]
}

IAM treats * and ? as wildcards, including inside bucket paths. Review the policy if your prefixes contain these characters. Fluxtail does not audit all permissions held by a key or reject keys for having additional permissions.

Set up the source

1. Enter a display name, bucket name, and AWS region, such as us-west-2.
2. Enter folder paths from the bucket root, one per line. A trailing / is optional: alb-one means alb-one/, not alb-one-other/. Folders cannot overlap, including across paused sources in the same account and bucket.
3. Select the destination stream, or choose New stream to create one here. A created stream is saved immediately, even if you cancel source setup.
4. Choose the import start date and time in the picker. The time is UTC.
5. Enter the access key ID, secret access key, and optional session token.
6. Select Create source, then Resume when you want collection to begin.

Only Test connection contacts AWS during setup. New sources are saved paused. After you resume, collection status shows any access problems. An inactive draft can be saved without credentials; add them in the browser before resuming.

The start selects files by S3 modification time, including files exactly at the boundary. It does not change the request times inside them. Fluxtail scans repeatedly, so delayed uploads and keys behind a previous listing position are still discovered. The bucket may remain private.

Credentials are encrypted and never returned by the source API. Replace them in the browser when they expire. After collection begins, the bucket, region, prefixes, format, start boundary and destination are locked. The display name and credentials remain editable.

Verify a request

Send a request to your ALB and wait for AWS to deliver its access-log file. Open the destination stream and set the time picker around the request time. For older imports, use historical time bounds rather than the default live window.

Open the record to check its complete original message and parsed aws_alb body. The event time is request_creation_time, falling back to the response time when the request time is absent. The body keeps the original timestamp precision, full URL, client and target addresses, separate ALB and target status codes, timings, trace header and diagnostic fields. Missing values and negative timing sentinels stay unchanged.

Exact labels come from the same parsed fields, for example:

Label	Example
aws_alb_request_method	GET
aws_alb_elb_status_code	502
aws_alb_target_status_code	200
aws_alb_request_url	https://example.com/health
aws_alb_client_address	2001:db8::1

Exact filter values support up to 2 KiB of UTF-8 text. Longer values remain in the body and message but have no exact-match label. Numeric comparisons such as “duration greater than one second” are not supported yet.

S3 uses the existing Live Tail and search behavior. Requests keep their original timestamps; old imports are not treated as current events. Use historical queries to verify import completeness.

Progress and recovery

The source page shows completed and blocked file counts, safe error reasons, last completed scan, and batches awaiting confirmation. A successful file does not clear another file’s blocked status.

• Pause stops new collection work. Already accepted batches can finish.
• Rescan restarts discovery while keeping completed identities and pending batches. It does not reimport completed revisions.
• Definite temporary failures retry automatically. Expired credentials must be replaced; quota exhaustion waits for available capacity.
• Malformed lines, corrupt gzip, changed/deleted revisions and size limits leave the file blocked. Other files continue. Earlier accepted batches are retained.
• Uncertain publication or storage waits for positive evidence. Contact support if it remains unresolved; keep the source and its collection history.

Collection retries deduplicate each object revision within its source. Setup rejects overlapping folders across sources in the same account and bucket. Duplicate records inside AWS files, copied files, and deleting then recreating a source can still produce duplicates.

An agent can prepare an inactive source through hosted MCP. Complete credentials in the browser; never paste AWS keys into an MCP request. Then use filters and Live Tail to investigate requests.

Test connection and status

Choose the import start date and time in the UTC picker. Use Test connection in the source form to run a read-only check with the entered settings. It does not save, resume or import logs. When editing, blank credential fields show masked dots and keep the saved key. Prefix edits reuse it; changing the bucket or region requires re-entering credentials.

The test checks listing and the metadata of one ALB file per prefix. It examines up to 100 entries per prefix and has a time limit. Each folder shows the number of entries checked and matching .log or .log.gz files, or confirms it is empty. Partial checks are labelled when more entries exist. If no sample is found, listing is verified but file access is not. File contents, encryption permissions and access to every file or version are not tested. The result explains access denial, invalid or expired credentials, a wrong region, and incomplete checks.

Saved sources show collector status, the last successful check, the last full scan, and blocked-file explanations. A successful connection test does not mean files have been imported. Resume collection and check its progress separately.

Client addresses

The Client IP column shows the request peer recorded by ALB, including IPv6. That peer can be a proxy. Source IP is the log sender; S3 imports have no network sender. Host is the reported hostname. Missing values show -.

Use the footer to show or hide Client IP. Click an address to apply the exact client_address label filter across supported formats. Existing ALB records use their aws_alb_client_address value; other formats can supply the canonical client_address label. Addresses are matched exactly, including IPv6 spelling. Smart view omits the duplicate client address from the message when this column is visible. Original fields remain available in log details.