Skip to main content
codingstairs

Scheduler result contracts and safe failure observability

The absence of an exception does not mean that a scheduled job succeeded. When jobs catch errors and return None, error, counts, or partial states, monitoring cannot distinguish normal empty work from a failure.

4 viewsAbout 2 min read
Share on X opens in a new window
Table of contents

The absence of an exception does not mean that a scheduled job succeeded. When jobs catch errors and return None, error, counts, or partial states, monitoring cannot distinguish normal empty work from a failure.

From execution result to an operational decision

Run the job

Concurrency limits and idempotency keys fix the processing scope.

Normalize the result

Converge on completed, empty, skipped, partial, retrying, or failed.

Record bounded evidence

Store status codes and failed scope, not raw payloads.

Choose the next action

Stop, resume, or ask for review according to status and retryability.

Fix the state vocabulary

Every job should converge on at least one of completed, empty, skipped, partial, retrying, or failed. Counts are supporting evidence; the state comes first. For example, a batch that generates twelve items is partial or failed when only some items were generated, and only a complete batch is eligible for skip.

Do not persist raw failure text

Provider responses, URLs, DSNs, and user input can appear in exception strings. Store bounded codes such as status:partial, status:failed, or exception:TimeoutError in failure tables and logs instead of the original text. Those codes are enough for retry decisions and operational filters.

State transitions and actions

State Meaning Automatic retry Operator action
completed Entire contracted scope finished None Sample the result
empty A valid query returned zero rows Usually none Check the input period
skipped A complete result exists None Inspect the skip reason
partial Only part was committed Only with proven idempotency Replay missing scope
retrying Bounded backoff is active Within the cap Watch repeated causes
failed Cap or non-retryable error None Fix and resume 
{"status":"partial","processed":9,"failed":3,"errorCode":"PROVIDER_TIMEOUT","retryable":true}

Treating processed > 0 as success makes omissions permanent. Treating empty as failure wastes external API traffic and database writes. Keep status, scope, and retryability separate.

Completion criteria

  • Exceptions, structured failures, and partial results use one scheduler failure path.
  • A job with a success count and a failed scope is not observed as success only.
  • A failure-store outage does not stop the next scheduled job.
  • Operators can query the job ID, status code, and retry scope without seeing sensitive source text.

Related course: Close partial failure, recovery, and platform boundaries

Terms in this content

More in backend

All in this category →

Related posts

Was this article helpful?