How to Check CI Artifacts Against the Exact Code Under Review

01Scope

Write down the expected subject before reading the result.

Identify the repository, immutable candidate revision, expected workflow and event, run identifier and attempt. Record the checks that must complete and the source files they cover. A branch name helps explain intent, but it can move; a branch name alone cannot identify the checked code.

Some pipelines run trusted control code from one revision while checking a candidate from another. Keep those identities separate. Inspect the checkout and trigger records to establish the candidate actually tested; the workflow’s displayed revision may describe its controls. A run selected because it started just after a security scan is only a candidate match until its recorded trigger connects the two.

02Originals

Retain the artifact before interpreting its verdict.

Read artifact metadata through the authenticated service connection used for the review. Match the repository, run, attempt and expected artifact name, and refuse an expired or ambiguous selection. If the service supplies an artifact digest, compare it with the downloaded archive bytes. Save the original archive and the metadata that supplied the expected digest.

A locally calculated hash is useful for detecting later changes. By itself, it does not establish who produced a file. Even matching an authenticated service digest does not prove the workflow was trustworthy or that its checks covered the required behavior. Those are separate parts of the review.

Before extraction, restrict archive size, member count and allowed paths. Reject duplicate or unexpected members and unsafe paths. Retain only the intended report files using bounded reads and ordinary files; arbitrary archive contents should never become executable review instructions.

03Replay

Check what the report actually says about the candidate.

Compare each retained report with its recorded digest and byte count. Verify the report’s source inventory against the pinned candidate: file paths, file sizes and content digests must describe the intended scope. A report with one omitted source file should fail a completeness check even if someone recalculates the report’s own digest.

Check the execution identifiers, scanner versions, findings and required job steps. Preserve skipped, failed and unavailable checks. A successful JavaScript analysis does not establish Python analysis coverage. Likewise, a browser fixture check does not prove a real customer task was accepted. State the scope beside the verdict.

04Worked example

A successful run can expose a stale verifier.

Consider this illustrative record. The approved scope names branch review/current and immutable candidate C. Run R, attempt 1, reports success for candidate C on review/current. The local capture script instead expects review/previous because it was copied from an earlier qualification. The branch comparison refuses the capture before an artifact is imported.

The refusal establishes a mismatch between the script and the observed run. It does not establish that the workflow failed or that the candidate is safe. First compare the approved scope, exact candidate, workflow and attempt with the service records. If those establish review/current as the intended branch, preserve the failed capture and correct that expectation in a new script version.

Download and verify the original artifacts from the same established run. Retain the corrected script digest and its result, explicitly linked to the earlier refusal. If the source revision or execution identity does not match, stop: changing the expectation to whatever a passing run happens to contain would conceal the mismatch. C and R are illustrative identifiers, not real commit or run values.

05Failure checks

Make the verifier demonstrate the mismatches it rejects.

Use copies of the retained evidence for deliberate alterations. Change one archive byte, change a report byte, substitute the execution identifier, and remove a source record while recalculating the report’s internal digest. Each relevant check should refuse the changed copy. Leave the originals untouched.

These checks provide evidence that the verifier rejects those specific alterations. They do not establish resistance to every attack, independent security approval or complete runtime provenance. Keep the altered-case results and the limitation with the successful replay.

• Changed archive bytes fail the retained archive digest check.
• Changed report bytes fail the member comparison.
• A substituted run fails the execution binding.
• An omitted source record fails inventory completeness.

06Decision

Record the result and the next unresolved requirement.

A useful handoff names the candidate, trusted controls, run and attempt, original artifact digests, checks performed, coverage limits and retained failures. Say whether the evidence was only captured, successfully replayed, imported into a candidate, or used in a separate release decision. These are distinct events.

Importing authenticated reports can change a repository’s evidence files. Record that new revision, verify the permitted differences and keep the checked source inventory explicit. Subsequent source edits need their own applicable qualification; a previous passing report cannot silently follow them.

This guide describes an artifact-review method informed by retained qualification work. It does not establish production readiness, a public execution service or independent acceptance of a model-produced change.