You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Task retries after approval expiry: reconciling a committed export without authorising a new one
#76421
I'm testing a task where permission to perform a data export expires, but a retry may still need to confirm an export that already completed. I'd appreciate a sanity check on the Airflow lifecycle assumptions.
The local case is deliberately small: a TaskFlow task validates a signed approval for fixed fictional inputs, writes the export and its operation record in one SQLite transaction, then raises an exception after commit. On retry, it checks for that operation first. An identical completed operation returns RECONCILED_EXISTING; if nothing was committed, the approval must still be valid before a new write. The operation ID stays fixed across attempts.
On Airflow 3.1.0, dag.test() ran the first attempt, marked it up for retry, then completed on the second attempt with one export row. Separate tests cover expiry before a new write and reconciliation after expiry. This only demonstrates a local transactional sink, not exactly-once delivery to an external system or a distributed executor test.
Is keeping that distinction entirely inside the task callable the appropriate pattern, particularly when a failed task is manually cleared? My intended behaviour is that clearing the task does not itself grant permission for a new export: the retained operation record still governs reconciliation, and a new logical export needs a newly approved operation ID. I'm not using try_number as the idempotency key; it is only used to inject the first-attempt failure in the demo.
The runnable example and test results are at https://github.com/AffixIO/airflow-approved-export. I maintain AffixIO, whose local SDK supplies the signature check in this example. There's no request to add it to Airflow, and no real data or external export endpoint is involved. If there's an existing example addressing expiring authorisation versus completed-side-effect reconciliation, I'd rather build on that than duplicate it.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I'm testing a task where permission to perform a data export expires, but a retry may still need to confirm an export that already completed. I'd appreciate a sanity check on the Airflow lifecycle assumptions.
The local case is deliberately small: a TaskFlow task validates a signed approval for fixed fictional inputs, writes the export and its operation record in one SQLite transaction, then raises an exception after commit. On retry, it checks for that operation first. An identical completed operation returns
RECONCILED_EXISTING; if nothing was committed, the approval must still be valid before a new write. The operation ID stays fixed across attempts.On Airflow 3.1.0,
dag.test()ran the first attempt, marked it up for retry, then completed on the second attempt with one export row. Separate tests cover expiry before a new write and reconciliation after expiry. This only demonstrates a local transactional sink, not exactly-once delivery to an external system or a distributed executor test.Is keeping that distinction entirely inside the task callable the appropriate pattern, particularly when a failed task is manually cleared? My intended behaviour is that clearing the task does not itself grant permission for a new export: the retained operation record still governs reconciliation, and a new logical export needs a newly approved operation ID. I'm not using
try_numberas the idempotency key; it is only used to inject the first-attempt failure in the demo.The runnable example and test results are at https://github.com/AffixIO/airflow-approved-export. I maintain AffixIO, whose local SDK supplies the signature check in this example. There's no request to add it to Airflow, and no real data or external export endpoint is involved. If there's an existing example addressing expiring authorisation versus completed-side-effect reconciliation, I'd rather build on that than duplicate it.
All reactions