[wasimxyz]

What changes

Watcher 1.4.3 fixes two things.

A watch folder that disappears while the watcher runs. Before, a watcher whose folder was deleted or unmounted kept reporting watching, because nothing looked at the folder once watching had started. Now:

• Every heartbeat tick, watchdog's report that the watch folder itself was deleted, and an upload thread whose look finds the folder itself unavailable ask the supervisor to check the folder. The check runs on the supervisor's poll thread, not on the heartbeat, watchdog or upload thread. A delete inside the folder is still ignored.
• The look at the folder runs on its own daemon thread, and the poll thread waits for it for 10 seconds. A stat on a hung share can block for a minute or for good, and the poll thread is the one that sees a stop. A look that takes longer counts as a problem, watch_directory not responding: <path>. A look that is still running is waited for by the next ones and not started again. The waiting state's check and the start of a runtime use the same look. A caller that is stopping doesn't wait for it: FolderProbe.look takes a cancel callable and waits in steps of 0.1 s, and the supervisor's checks and the uploader pass their stop signal. The look that starts a runtime keeps its 10 seconds.
• One failed look is not enough. A second look 3 seconds later has to fail too, and its reason is the one reported. A share that drops for a moment costs no upload and no rescan, and a folder that is still being deleted on Windows is reported once, as "not found", and not first as "not readable".
• On a confirmed problem the runtime is torn down as for a config rebuild and the watcher enters the waiting state with the reason, as it does when the folder is missing at start. It logs one warning, queues the watch_directory_unavailable error, and queues no second watcher_started event, because the watcher has not stopped.
• The move is bounded where a dead share could hold it. Uploads in progress get 10 seconds (not the 120 seconds of a config rebuild). The observer's stop, the wait for a heartbeat tick and the event flush get 5 seconds each. An upload thread that is blocked reading the share can't be cut off, so it is left behind with its runtime's state database open, and the next runtime starts only once it has exited. Every wait of the move ends on a stop, and the runtime is then left in place for stop, which cuts its uploads off within its own budget (the stop is also checked after each look and before the teardown). The first heartbeat of the waiting state, and a request already in flight, still have only the HTTP timeouts. The log says what was left behind, the upload threads or only a heartbeat tick.
• Between the folder disappearing and the check, every file in it used to look missing, so the upload worker dropped each pending_uploads row for good and counted "Queued file missing" attempts toward the cancel. The uploader now asks whether the whole folder is gone first. If it is, the row or queue entry stays, nothing is counted, and the check is requested at once.
• A folder that is there can still leave the watcher blind. The supervisor now records the folder's device and inode number when it starts watching, and compares it at each check, which catches a folder that was renamed and made again, or one that something was mounted on. An inode number of 0 is ignored. A runtime whose monitor stopped watching is rebuilt as before. A rebuilt runtime that goes blind again within 2 minutes is not rebuilt again. When watchdog has stopped watching, the watcher waits with watch_directory can't be watched: <path> and retries on the waiting state's backoff, so an emitter that dies at once doesn't cause a rescan at every check. When watchdog is still watching and only the folder's identity differs (FUSE mounts without use_ino, SMB or virtual drives without persistent file IDs report a new number at every look), the watcher keeps the runtime, logs both identities once, and stops comparing the identity until the next config change or restart, so live uploads don't stop. The rebuild of a blind runtime is bounded like the move, because the identity can change while upload threads are blocked on the old filesystem.
• A rebuild whose start fails, or whose folder is gone when it starts, no longer queues a watcher_started event that says the watcher resumed.
• When the folder is back, the existing waiting-state check starts a new runtime and a watcher_started event says it resumed. A runtime does not start next to threads that the move left behind on the same folder. While they run, the waiting reason says uploads stuck on <folder> have not ended, in place of the stale "not found". A config change that moves the watch folder doesn't wait for them (they can't touch the new folder), and one that keeps the folder waits up to 2 minutes, logs every minute, and ends on a stop. Asking whether those threads have exited logs nothing, where it wrote a line about twice a second.
• During a stop, an upload thread doesn't look at the folder when a file can't be found: a stop is not a missing file.
• One "not responding" while waiting no longer ends the backoff of a start that keeps failing. A "not found", or two problems in a row, still do.
• The Windows service no longer logs "is running" or "is waiting" and then "was asked to stop while it was starting". It logs only the second. The end-to-end drivers still see the end of the start.

A stop during watch start-up. watch installed its Ctrl+C and SIGTERM handlers only after the supervisor had started, so a SIGTERM during the config sync, the first heartbeat or the start-up scan ended the process and Ctrl+C raised KeyboardInterrupt, with no stopped heartbeat and no watcher_stopped event. The handlers are now installed before the start. They call WatcherSupervisor.request_stop, the supervisor's one stop signal, and so does the Windows service (from a thread that starts before supervisor.start()). The start-up scan reads it between entries and ends early, so a stop during a long scan is acted on within seconds, and the normal clean stop follows, exit code 0. The files the scan hadn't reached are scanned at the next start: the scan writes nothing to the state database for them, and a baseline scan on the first start of a new-only environment records nothing at all when it stops early, because a partial baseline would make the next start upload the rest of the backlog as new. The upload worker isn't started for a watcher that is already stopping, and poll leaves a rebuild or a folder check alone once a stop is requested. The config sync and the first heartbeat are not cut short: each has its HTTP timeout.

The release is watcher 1.4.3 (version, lock file and docs snapshots). There is no changelog entry, because it is a watcher-only release.

Docs

• User docs: the waiting state in concepts.mdx, manage-watchers.mdx and troubleshoot-a-watcher.mdx now says the watcher also goes back to waiting when its folder disappears while it runs, and resumes when it is back. windows-service.mdx says a stop ends the scan at its next file from 1.4.3.
• docs/developer/watcher.md describes the stop signal and the scan stop, the bounded and cancelable look and the second look, what the move to the waiting state includes, which of its waits are bounded and end on a stop and what a stop still waits for, the folder identity check and its fallback, the wait for a runtime that keeps going blind, and the waits for threads that a move left behind. The known-bug examples in the end-to-end docs use a placeholder that is not a real issue number.

How it was tested

• make check: clean, and it rewrote nothing.
• make py-test-unit: 2486 passed. The changed unit test files also pass on Python 3.12 in a separate environment (452 passed).
• make py-test-integration (with its own test database): 95 passed.
• End-to-end on macOS under the watch command: watcher/e2e/test_setup_startup.py and test_shutdown.py, 32 passed. The Windows service variants were not run here.
• uv run pytest watcher/e2e --collect-only collects 208 tests, and watcher/tests/test_e2e_catalog.py passes (56). make docs-catalogs changes no snapshot.
• New unit tests cover each point above. A stop during a hung look, during the second look and during the move (each returns within about a second, and the stop that follows is clean), during a blind rebuild, during the wait for left-behind threads, and during an upload thread's look; the identity that changes at every look (the runtime is kept, both identities are logged once, and a config change compares again); the threads left behind (per folder, bounded, with a reason, and no log flood); the backoff reset rule; and the uploader's stop. The timing tests with short probe timeouts now use 2 second timeouts and a longer hang. The timing-sensitive test files were run 5 times in a row, and once more as three copies in parallel, with no failure.
• Each new behavior was reverted once, and the tests for it fail. Where one guard was redundant with another (a stop check in the folder check and one at the top of the teardown), both were removed.
• A stop in the middle of a long scan, or during a move on a dead share, is not an end-to-end test, because it can't be hit from outside. The unit tests drive it.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
data-hub	Ready	Preview	Nov 13, 2026 11:58pm UTC
data-hub-docs	Ready	Preview	Nov 13, 2026 11:58pm UTC