Go back to waiting when the watch folder disappears, and stop cleanly during start-up (#390)

When the watch folder disappears while the watcher runs (deleted, or a
network share that drops), the watcher now goes back to waiting with the
reason, as it does when the folder is missing at start, and resumes when
the folder is back. Each heartbeat asks for a folder check, run on its own
thread with a 10 second limit so a hung share can't freeze the watcher,
and confirmed by a second look 3 seconds later. Uploads stuck on a dead
share are given up on after about 15 seconds, and pending uploads keep
their place while the folder is gone. A folder that can't be watched
waits with a growing delay, and a renamed or replaced folder is noticed by
its identity. A stop during `watch` start-up now ends cleanly, the
start-up scan ends early when a stop arrives (CLI and Windows service),
and folder checks end on a stop. Bumps the watcher to 1.4.3; two
end-to-end tests now pass instead of being expected failures.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Author: wasimxyz

docs/developer/testing.md

@@ -568,13 +568,13 @@ bug:
 @pytest.mark.xfail(
     strict=True,
     raises=KnownBugStillPresent,
-    reason="ENG-2118: a stop during start-up ends with no stopped heartbeat",
+    reason="bug-example: what goes wrong today",
 )
-@pytest.mark.covers("process.stop->clean_stop_during_scan")
-def test_a_stop_during_start_up_still_ends_with_a_clean_stop(...):
-    ...  # The precondition: the watcher is held up in its first heartbeat, and a stop was requested.
+@pytest.mark.covers("machine.thing->intended_outcome")
+def test_the_intended_behavior(...):
+    ...  # The precondition: everything that must hold before the bug can show.
     with known_bug():
-        assert pc.wait_exit(30) == 0
+        assert the_intended_behavior()
 ```
 
 `known_bug()` turns an `AssertionError` raised inside it (a `WaitTimeout` too) into
@@ -583,7 +583,7 @@ block is a different failure, so the test fails and doesn't count as the bug. Pu
 assertion that shows the intended behavior inside. The check enforces the pattern: the xfail is
 strict and has `raises=KnownBugStillPresent`, the test has a `with known_bug():` block and names
 only `xfail_bug` transitions, its reason starts with the bug of one of them (its Linear issue, such
-as `ENG-2118:`, or a short slug until the issue is filed), and
+as `ENG-0000:`, or a short slug until the issue is filed), and
 no other test names an `xfail_bug` transition. To see that a test fails inside the block, run it
 with `--runxfail`.
 

docs/developer/watcher.md

@@ -45,7 +45,7 @@ What the **Watchers** page shows for each watcher, derived from its most recent
 | **Online**       | A heartbeat arrived in the last 5 minutes.                                                        |
 | **Unresponsive** | No heartbeat for more than 5 minutes. The process crashed, the PC is off, or the network is blocking the API. A watcher that stopped for a Windows restart or shutdown, or for an automatic update, shows **Unresponsive** after 15 minutes instead. |
 | **Stopped**      | The watcher shut down cleanly and said so in its last heartbeat. After a stop you chose, such as `data-hub-watcher service stop`, it keeps showing **Stopped**, however long ago that was, until it starts again. After a Windows restart or shutdown, or the stop of an automatic update, it shows **Stopped** for 15 minutes, then **Unresponsive** if it hasn’t returned. |
-| **Waiting: folder not found** | The watcher is running and sending heartbeats, but its watch folder is missing, so it isn’t watching anything. Watcher 1.4.0 and later. |
+| **Waiting: folder not found** | The watcher is running and sending heartbeats, but its watch folder is missing, so it isn’t watching anything. A watcher goes into this state when it starts without the folder (watcher 1.4.0 and later), and when the folder disappears while it runs (watcher 1.4.3 and later). It starts watching again once the folder is back. |
 | **Registered**   | The watcher registered but hasn’t sent its first heartbeat yet.                                   |
 
 A watcher that stopped because Windows is restarting or shutting down (watcher 1.4.1 and later), or because it is installing an automatic update (watcher 1.4.2 and later), is expected back. It shows **Stopped**, and hovering over the badge tells you when it starts showing **Unresponsive** if it hasn’t returned: 15 minutes after its last heartbeat. Any other stop, such as `data-hub-watcher service stop` or Ctrl+C, shows **Stopped** until the watcher starts again.

docs/site/content/docs/concepts.mdx

@@ -12,7 +12,7 @@ The **Watchers** area is the fleet view: every registered watcher, its health, i
 
 Each watcher sends a [heartbeat](/docs/concepts#heartbeat) every 60 seconds with version, instrument, watch directory, upload mode, activity counters, and uptime. The fleet list surfaces:
 
-- **Status**: **Online**, **Waiting: folder not found** (running, but its watch folder is missing), **Unresponsive** (no heartbeat for more than 5 minutes, or more than 15 for a watcher that stopped for a Windows restart, a shutdown or an automatic update), **Stopped** (shut down cleanly. A stop you chose stays **Stopped** until the watcher starts again, and a stop for a Windows restart, a shutdown or an automatic update turns **Unresponsive** after 15 minutes), or **Registered** (no first heartbeat yet). See [Watcher status](/docs/concepts#watcher-status).
+- **Status**: **Online**, **Waiting: folder not found** (running, but its watch folder is missing, or went missing while it ran), **Unresponsive** (no heartbeat for more than 5 minutes, or more than 15 for a watcher that stopped for a Windows restart, a shutdown or an automatic update), **Stopped** (shut down cleanly. A stop you chose stays **Stopped** until the watcher starts again, and a stop for a Windows restart, a shutdown or an automatic update turns **Unresponsive** after 15 minutes), or **Registered** (no first heartbeat yet). See [Watcher status](/docs/concepts#watcher-status).
 - **Change pending** or **Change rejected**: the latest config change from the web app hasn’t reached the watcher yet, or the watcher refused it. See [Change a watcher’s config](#change-a-watchers-config).
 - **Edited on PC, restart pending**: someone changed `config.yaml` on the PC while the watcher was running, and the watcher hasn’t loaded it yet. See [Edits made on the PC](#edits-on-the-pc).
 - **Last heartbeat**: a stale value is the first sign of a stopped, disconnected, or crashed watcher

docs/site/content/docs/manage-watchers.mdx

@@ -43,6 +43,8 @@ The watcher checks for the folder again once a minute and starts watching as soo
 - Create or reconnect the folder on the PC, or
 - Point the watcher at the right folder from the web app; see [Change a watcher’s config](/docs/manage-watchers#change-a-watchers-config)
 
+Watcher 1.4.3 and later also go back to this state when the folder disappears while they run, for example when a network drive drops or the folder is deleted. The watcher notices within a minute, looks again a few seconds later so that a drop of a moment doesn’t count, stops watching, and shows **Waiting: folder not found**. It gives a file that was uploading at that moment up to 10 seconds to finish, cuts it off if it isn’t done, and uploads it again once the folder is back. When the folder returns, the watcher starts watching it again without a restart, and queues a `watcher_started` event that says it resumed.
+
 Older watchers exit at startup instead, with an error that the watch directory doesn’t exist.
 
 ## A config change shows “Change rejected” [#change-rejected]

docs/site/content/docs/troubleshoot-a-watcher.mdx

@@ -46,7 +46,7 @@ The Windows auto-update path requires that `service install` (or `service reinst
 
 `service reinstall` is the right command after an out-of-band wheel swap (for example a manual `uv tool install -U` from an Administrator shell). Stop and uninstall are best-effort, so it works even if the service is already gone.
 
-With watcher 1.4.2 and later, `service stop` finishes within about 25 seconds once the watcher is running, even when Data Hub doesn’t answer. It can take longer while the watcher is still starting, because it syncs its config, sends its first heartbeat and scans the watch folder before it looks for a stop. It can also take longer while the watcher applies a config change from Data Hub, which waits up to 2 minutes for uploads to finish. Once the stop begins, the watcher starts no new file or part. A part already on its way gets up to 15 seconds to arrive. In the last 10 seconds, the watcher sends its `watcher_stopped` event and then its final heartbeat, so it shows as **Stopped** when Data Hub answers.
+With watcher 1.4.2 and later, `service stop` finishes within about 25 seconds once the watcher is running, even when Data Hub doesn’t answer. It can take longer while the watcher is still starting. Before watcher 1.4.3 it scanned the whole watch folder first, which takes minutes on a large folder. From 1.4.3 a stop ends the scan at its next file, and the files it hadn’t reached are scanned at the next start. The watcher still syncs its config and sends its first heartbeat before it looks for a stop, so a stop that arrives then waits for Data Hub to answer each request, or for each request to time out after 35 seconds. It can also take longer while the watcher applies a config change from Data Hub, which waits up to 2 minutes for uploads to finish. Once the stop begins, the watcher starts no new file or part. A part already on its way gets up to 15 seconds to arrive. In the last 10 seconds, the watcher sends its `watcher_stopped` event and then its final heartbeat, so it shows as **Stopped** when Data Hub answers.
 
 A file that the watcher had to cut off isn’t lost. When the instrument uploads automatically, the watcher uploads that file again the next time it starts. A file that someone requested with **Upload** is different: `service stop` cancels the requests, so request the file again. A restart of the PC and the stop of an automatic update keep them queued, as [When the PC restarts or shuts down](#when-the-pc-restarts-or-shuts-down) describes. See also [Request uploads](/docs/manage-watchers#request-uploads).
 

docs/site/content/docs/windows-service.mdx

@@ -450,5 +450,5 @@
     ]
   },
   "prog": "data-hub-watcher",
-  "version": "1.4.2"
+  "version": "1.4.3"
 }

docs/site/src/lib/cli-catalog.snapshot.json

@@ -21,7 +21,7 @@
 * a known-bug test follows the pattern: ``xfail(strict=True, raises=KnownBugStillPresent,
   reason="<bug>: ...")``, a ``with known_bug():`` block around the intended-behavior
   assertion, and only ``xfail_bug`` transitions in ``covers``. The reason starts with the bug
-  of one of those transitions (its Linear issue, such as ``ENG-2118``, or a short slug until
+  of one of those transitions (its Linear issue, such as ``ENG-0000``, or a short slug until
   the issue is filed). A test that isn't an xfail never names an ``xfail_bug`` transition.
 
 What counts as a marker: ``@pytest.mark.covers("id", ...)``, ``@pytest.mark.platforms("cli")``

uv.lock

@@ -56,7 +56,7 @@ class Transition:
       failing loudly (XPASS) the day the bug is fixed, which is when the mark and the
       ``known_bug()`` block come off and the status becomes ``covered``. A broken
       precondition raises something else, so it fails the test instead of hiding as an
-      expected failure. ``bug`` is the Linear issue (``ENG-2118``), or a short slug until the
+      expected failure. ``bug`` is the Linear issue (``ENG-0000``), or a short slug until the
       issue is filed, and the ``reason`` of the test starts with it.
     * ``excluded``: deliberately not tested end to end. ``reason`` says why and where it is
       tested instead. A test must not name it.
@@ -80,7 +80,7 @@ def __post_init__(self) -> None:
         if self.status == "xfail_bug" and not self.bug:
             raise ValueError(
                 f"{self.id}: an xfail_bug transition needs a bug: its Linear issue "
-                "(such as ENG-2118) or a short slug"
+                "(such as ENG-0000) or a short slug"
             )
         if self.status != "xfail_bug" and self.bug:
             raise ValueError(f"{self.id}: only an xfail_bug transition has a bug")