Просмотр новости
Найдите то, что Вас интересует
Спутники
РН
SpaceX
Роскосмос
NASA
СпутникиРНSpaceXРоскосмосNASAIntroductionModern applications are usually deployed across multiple pods for scalability and availability. That works well for APIs, but scheduled jobs create a different problem: every pod may have the same scheduler, so every pod can decide to run the same job. This can lead to duplicate processing, inconsistent data, duplicate notifications, or a job being left half-finished when the pod running it crashes.The natural first question is: how do we make sure only one pod runs the job? The answer is distributed locking. But once we solve that problem, another, less obvious problem appears : what happens if a pod that used to own the lock wakes up later and still tries to update shared data?This blog walks through those problems one by one and explains when an advisory lock, Redis lock, fencing token, retry, and idempotency are useful.The Problem 1: Multiple Pods, One Scheduled JobImagine three pods running the same Go service. The scheduler runs every morning at 10 AM. Because the scheduler exists inside every pod, all three pods can trigger at almost the same time.10:00 AMPod 1 → Scheduler fires → Process jobPod 2 → Scheduler fires → Process jobPod 3 → Scheduler fires → Process jobIf the job inserts records, sends notifications, updates statuses, generates files, or calls another service, the same work may happen three times.We don't actually want three pods doing the work. We want one pod to take responsibility for the run, while the others step aside.Solution 1: Distributed LockingA local mutex such as sync.Mutex cannot solve this because the mutex exists only inside one Go process. Each pod has its own process and therefore its own mutex.Pod 1 → Mutex 1Pod 2 → Mutex 2Pod 3 → Mutex 3What we need is a lock that all pods can see. This is where distributed locking comes in.1.PostgreSQL Advisory LockIf PostgreSQL is already part of the system, an advisory lock is a simple way to coordinate the pods.SELECT pg_try_advisory_lock(12345);One pod gets the lock. The other pods get a failure and skip the current execution.Pod 1 → Lock acquired → ExecutePod 2 → Lock unavailable → SkipPod 3 → Lock unavailable → SkipA session-level PostgreSQL advisory lock is tied to the database connection. If that connection disappears, PostgreSQL releases the lock automatically.2.Redis LockRedis can solve the same coordination problem when Redis is already part of the architecture. A common pattern uses NX and an expiry time.SET job_lock <pod_id> NX PX 30000NX means the key is created only if it doesn't already exist, while PX gives the lock a time-to-live.What Did the Lock Actually Solve?At this point, we have solved one specific problemProblem:Multiple pods can execute the same scheduled job. ↓Distributed Lock ↓Solution:Only one pod executes the job at a time.This is the important thing to understand: an advisory lock or Redis lock solves coordination. It answers the question, "Who is allowed to execute this job right now?"It does not automatically guarantee that an old or stalled process can never write to shared data later.Problem 2: Stale WritesNow consider a slightly more difficult scenario. Pod 1 acquires the lock and starts processing. During the job, the pod gets stuck because of a long garbage-collection pause, a slow network call, CPU starvation, or some other problem.Pod 1 → Acquires lock → Starts processing | Pod becomes slow | Lock expires/releases |Pod 2 → Acquires lock → Starts processingIf Pod 1 eventually wakes up, it may still believe that it owns the job. If it continues and writes to shared data, Pod 1 and Pod 2 can now both write to the same data.Pod 2 → Acquires lock → Writes latest dataPod 1 → Wakes up → Writes old/stale data ↓ Data can be overwrittenThis is the subtle problem that a plain distributed lock does not solve. The lock was doing its job: Pod 2 was allowed to take over after Pod 1 stopped making progress. The problem is that Pod 1 was still alive and capable of writing.Solution 2: Fencing TokensThis is why fencing tokens are introduced. Instead of only giving a pod a lock, the coordination layer also gives every lock acquisition a monotonically increasing token.Pod 1 → Lock acquired → Token 5Pod 2 → Lock acquired → Token 6Pod 3 → Lock acquired → Token 7Every write to the shared data carries the token. The data store accepts a write only if its token is newer than the token it has already accepted.Pod 1 → Token 5 → Starts processing → StallsPod 2 → Token 6 → Writes data → ACCEPTEDPod 1 → Wakes up → Writes with Token 5 → REJECTED (stale token)Now the shared data store itself protects against the stale writer. The lock answers "who can execute?" The fencing token answers "is this writer still the latest owner?"This distinction is especially important when the job updates shared, mutable data such as inventory, balances, status records, or other state where an old write could overwrite a newer one.Problem 3: Failed ExecutionSuppose Pod 1 acquires the lock and processes 60% of the job before crashing.The lock can be released, but that doesn't mean the job completed.The other pods had skipped the execution because Pod 1 already held the lock. Once the lock is released, a new scheduler execution can start, but it needs to know whether the previous execution actually completed or failed.Without tracking the job status, the system cannot reliably determine whether the job needs to be executed again.Solution 3: Job Status and RetryMaintain an explicit status for the job execution.PENDING → RUNNING → COMPLETED | ↓ FAILED | ↓ RETRYWhen Pod 1 starts: PENDING → RUNNINGIf it completes: RUNNING → COMPLETEDIf it crashes: RUNNING → FAILEDThe lock is released when Pod 1 stops, but the job remains marked as FAILED.A subsequent scheduler execution can acquire the lock and check whether the failed job is eligible for retry.Scheduler ↓Acquire Lock ↓Find FAILED Job ↓Check Retry Time ↓Retry JobRetries should normally use backoff rather than immediately trying again on every scheduler tick.Attempt 1 → immediatelyAttempt 2 → after 5 minutesAttempt 3 → after 10 minutesProblem 4: Duplicate ProcessingRetries introduce another question: what if the first attempt completed some of the work before failing?For example, a job needs to process 100 records and successfully processes 60 before crashing. When the retry starts, those first 60 records must not be processed again in a way that creates duplicate side effects.Solution 4: Idempotent ProcessingThis is where idempotency becomes important. The job should be safe to run again without producing incorrect duplicate effects.First attempt:100 records → 60 processed → CRASHRetry:60 already processed → skip40 remaining → processDepending on the use case, this can be achieved through upserts, unique constraints, processed flags, idempotency keys, or checks before performing the operation.Now, Putting Everything Together Distributed Lock ↓Who is allowed to run?Fencing Token ↓Is this writer still the latest owner?Job Status + Retry ↓Did the job actually complete?Idempotency ↓Is it safe to run the job again?Together, these mechanisms cover different failure modes rather than trying to make one mechanism solve everything.Reference Architecture Scheduler (all pods) | Acquire Lock | +---------+-------------+ | | Lock acquired Lock unavailable + token issued | | Skip v Execute Job | Write with token | Data store validates token freshness | +---------+---------+ | | COMPLETED FAILED | Retry + BackoffThe scheduler triggers the work. The distributed lock decides which pod gets to execute. The fencing token protects shared data from stale writers. Job status tells us whether the run completed. Retry handles failures. Idempotent processing makes a retry safe.When Should You Use What? Business ValueData integrity — prevents duplicate execution and protects shared data from stale writesResilience — a crashed pod does not silently lose the scheduled jobOperational visibility — job status makes failures and retries visibleSafer scaling — increasing the number of pods does not mean increasing the number of job executionsReduced manual intervention — retry and backoff can recover from transient failures automatically
| # | Наименование новости | Тональность | Информативность | Дата публикации |
|---|---|---|---|---|
| 1 | Очередь задач на Postgres: SKIP LOCKED + lease/heartbeat + backpressure (практический опыт) | 0 | 12.42 | 13-01-2026 |
| 2 | Postgresso #5 (90) | 1 | 11.73 | 08-07-2026 |
| 3 | Семь раз подумай, один раз пошардируй: как мы начали горизонтально масштабировать метаданные чатов Телемоста | 0 | 7 | 29-06-2026 |
| 4 | Postgresso #5 (90) | 0 | 6 | 08-07-2026 |
| 5 | A quick installation of PostgreSQL on Fedora | 0 | 5.15 | 26-01-2023 |
| 6 | PostgreSQL для бэкендера: 10 фич, которыми мало пользуются, а зря | 5 | 7 | 30-06-2026 |
| 7 | Silos are the Bane of Value Delivery | -5 | 7 | 29-06-2026 |
| 8 | Как я придумывал замену Redis и что из этого получилось | 1 | 7 | 05-08-2026 |
| 9 | Каждые 5 минут транзакции в PostgreSQL замирают на 3-7 секунд. ... | -1 | 8.77 | 21-09-2026 |