Pick what you're trying to do. Each journey collects only the inputs it needs and dispatches the
GitHub workflows (pilot-onboarding.yml for Journeys 1–4, upload-gpg-key.yml for keys) — you just don't have to know which mode or flag maps to which intent.
fetch() calls to
api.github.com. Classic PAT with repo + workflow, or fine-grained with
Actions, Contents and Secrets read/write.Shown in the run title and kept in the run record. One line.
Do this once per pilot/environment. Journeys 2 and 3 reuse the spec you produce here.
Skip if the pilot's key already lives in Secrets Manager for this environment. Otherwise paste it
here: it's encrypted in your browser (libsodium sealed-box) and staged as GPG_KEY_UPLOAD_PAYLOAD.
Choose whether to test it on sample data first (Journey 2 will sync it only after a real decrypt works) or
sync immediately.
“Test-first” stages only; the next sample run (Journey 2) tests this key and persists it on success. “Sync now” needs the spec in step 2 to know pilot / environment / secret path — fill that in first.
If this pilot's files arrive on an SFTP server instead of S3, enter its credentials here. They are encrypted in your
browser (libsodium sealed-box), staged as SFTP_KEY_UPLOAD_PAYLOAD, and upload-sftp-key.yml writes them to
Secrets Manager at /<namespace>/<env>/sftp_keys.<pilot_id> (the spec in step 2 names pilot / environment / secret path).
Then set source.type to sftp in step 2: runs copy the files to the S3 location first, then carry on as usual.
Fill the form (generates YAML live), paste an existing spec, or load one already committed in the repo.
With type sftp, source.bucket/source.prefix above are the S3 landing location the files are copied into before the run reads them.
Validate sends the YAML inline (preflight_only) — checks shape, mandatory fields and that the S3 source exists; nothing is committed.
Open PR commits it to a new branch and opens a pull request — a historical run (Journey 4) only accepts a spec merged into the protected branch.
Fast iteration loop: try a fresh sample drop without touching the rest of the spec.
This spec's source is SFTP — no S3 location needed. The run connects, copies a few sample files of each file type into this run's own folder (nothing is moved on the server), disconnects, writes an audit, and samples from there.
Overrides source.bucket/source.prefix of the active spec for this test only.
Nothing is committed: the spec is sent inline and stored write-once on S3 in its own folder
(onboarding_specs/pilot_<id>/samples/<timestamp>_<hash>/).
Or paste a full URI:
Takes the current sample iteration through the real Spark pipeline — ingestion then data quality. Two phases: prepare (stores the spec, stages headers, builds the Spark steps — nothing launches), then launch, where you dispatch the datalake-ops workflow that creates the cluster and runs both steps. No approval gate; output stays in this iteration's S3 folder.
Whichever you pick is copied into a new iteration folder under
onboarding_specs/pilot_<id>/samples/; the original is never modified, and the new folder's manifest records it as the parent.
Full pipeline run on a new EMR cluster, created through datalake-ops. Phase 1 prepares the ingestion and data-quality Spark steps (approval-gated); phase 2 launches them through datalake-ops. Quality reports land in the run's S3 folder.
A historical run only uses a spec that is committed on the protected branch (reviewed & merged via Journey 1's PR).
The historical S3 location, schema and table kind come from that spec. The run's job also waits on the
pilot-historical-submit environment's reviewers.
The spec is stored and the Spark steps are prepared. This dispatches etl_retry_workflow_emr.yml in
bidgely/datalake-ops with your token — it creates the cluster and runs both steps. Review before launching.
Uses the token from the Connection bar, so it needs Actions read/write on bidgely/datalake-ops.
Prefer the GitHub UI? Copy these values into that workflow's “Run workflow” form — the same values are in the prepare run's summary.