Go back to waiting when the watch folder disappears, and stop cleanly during start-up

docs/developer/testing.md

@@ -568,13 +568,13 @@ bug:
 @pytest.mark.xfail(
     strict=True,
     raises=KnownBugStillPresent,
-    reason="ENG-2118: a stop during start-up ends with no stopped heartbeat",
+    reason="bug-example: what goes wrong today",
 )
-@pytest.mark.covers("process.stop->clean_stop_during_scan")
-def test_a_stop_during_start_up_still_ends_with_a_clean_stop(...):
-    ...  # The precondition: the watcher is held up in its first heartbeat, and a stop was requested.
+@pytest.mark.covers("machine.thing->intended_outcome")
+def test_the_intended_behavior(...):
+    ...  # The precondition: everything that must hold before the bug can show.
     with known_bug():
-        assert pc.wait_exit(30) == 0
+        assert the_intended_behavior()
 ```
 
 `known_bug()` turns an `AssertionError` raised inside it (a `WaitTimeout` too) into
@@ -583,7 +583,7 @@ block is a different failure, so the test fails and doesn't count as the bug. Pu
 assertion that shows the intended behavior inside. The check enforces the pattern: the xfail is
 strict and has `raises=KnownBugStillPresent`, the test has a `with known_bug():` block and names
 only `xfail_bug` transitions, its reason starts with the bug of one of them (its Linear issue, such
-as `ENG-2118:`, or a short slug until the issue is filed), and
+as `ENG-0000:`, or a short slug until the issue is filed), and
 no other test names an `xfail_bug` transition. To see that a test fails inside the block, run it
 with `--runxfail`.
 

docs/site/content/docs/concepts.mdx

@@ -45,7 +45,7 @@ What the **Watchers** page shows for each watcher, derived from its most recent
 | **Online**       | A heartbeat arrived in the last 5 minutes.                                                        |
 | **Unresponsive** | No heartbeat for more than 5 minutes. The process crashed, the PC is off, or the network is blocking the API. A watcher that stopped for a Windows restart or shutdown, or for an automatic update, shows **Unresponsive** after 15 minutes instead. |
 | **Stopped**      | The watcher shut down cleanly and said so in its last heartbeat. After a stop you chose, such as `data-hub-watcher service stop`, it keeps showing **Stopped**, however long ago that was, until it starts again. After a Windows restart or shutdown, or the stop of an automatic update, it shows **Stopped** for 15 minutes, then **Unresponsive** if it hasn’t returned. |
-| **Waiting: folder not found** | The watcher is running and sending heartbeats, but its watch folder is missing, so it isn’t watching anything. Watcher 1.4.0 and later. |
+| **Waiting: folder not found** | The watcher is running and sending heartbeats, but its watch folder is missing, so it isn’t watching anything. A watcher goes into this state when it starts without the folder (watcher 1.4.0 and later), and when the folder disappears while it runs (watcher 1.4.3 and later). It starts watching again once the folder is back. |
 | **Registered**   | The watcher registered but hasn’t sent its first heartbeat yet.                                   |
 
 A watcher that stopped because Windows is restarting or shutting down (watcher 1.4.1 and later), or because it is installing an automatic update (watcher 1.4.2 and later), is expected back. It shows **Stopped**, and hovering over the badge tells you when it starts showing **Unresponsive** if it hasn’t returned: 15 minutes after its last heartbeat. Any other stop, such as `data-hub-watcher service stop` or Ctrl+C, shows **Stopped** until the watcher starts again.

docs/site/content/docs/manage-watchers.mdx

@@ -12,7 +12,7 @@ The **Watchers** area is the fleet view: every registered watcher, its health, i
 
 Each watcher sends a [heartbeat](/docs/concepts#heartbeat) every 60 seconds with version, instrument, watch directory, upload mode, activity counters, and uptime. The fleet list surfaces:
 
-- **Status**: **Online**, **Waiting: folder not found** (running, but its watch folder is missing), **Unresponsive** (no heartbeat for more than 5 minutes, or more than 15 for a watcher that stopped for a Windows restart, a shutdown or an automatic update), **Stopped** (shut down cleanly. A stop you chose stays **Stopped** until the watcher starts again, and a stop for a Windows restart, a shutdown or an automatic update turns **Unresponsive** after 15 minutes), or **Registered** (no first heartbeat yet). See [Watcher status](/docs/concepts#watcher-status).
+- **Status**: **Online**, **Waiting: folder not found** (running, but its watch folder is missing, or went missing while it ran), **Unresponsive** (no heartbeat for more than 5 minutes, or more than 15 for a watcher that stopped for a Windows restart, a shutdown or an automatic update), **Stopped** (shut down cleanly. A stop you chose stays **Stopped** until the watcher starts again, and a stop for a Windows restart, a shutdown or an automatic update turns **Unresponsive** after 15 minutes), or **Registered** (no first heartbeat yet). See [Watcher status](/docs/concepts#watcher-status).
 - **Change pending** or **Change rejected**: the latest config change from the web app hasn’t reached the watcher yet, or the watcher refused it. See [Change a watcher’s config](#change-a-watchers-config).
 - **Edited on PC, restart pending**: someone changed `config.yaml` on the PC while the watcher was running, and the watcher hasn’t loaded it yet. See [Edits made on the PC](#edits-on-the-pc).
 - **Last heartbeat**: a stale value is the first sign of a stopped, disconnected, or crashed watcher

docs/site/content/docs/troubleshoot-a-watcher.mdx

@@ -43,6 +43,8 @@ The watcher checks for the folder again once a minute and starts watching as soo
 - Create or reconnect the folder on the PC, or
 - Point the watcher at the right folder from the web app; see [Change a watcher’s config](/docs/manage-watchers#change-a-watchers-config)
 
+Watcher 1.4.3 and later also go back to this state when the folder disappears while they run, for example when a network drive drops or the folder is deleted. The watcher notices within a minute, looks again a few seconds later so that a drop of a moment doesn’t count, stops watching, and shows **Waiting: folder not found**. It gives a file that was uploading at that moment up to 10 seconds to finish, cuts it off if it isn’t done, and uploads it again once the folder is back. When the folder returns, the watcher starts watching it again without a restart, and queues a `watcher_started` event that says it resumed.
+
 Older watchers exit at startup instead, with an error that the watch directory doesn’t exist.
 
 ## A config change shows “Change rejected” [#change-rejected]

docs/site/content/docs/windows-service.mdx

@@ -46,7 +46,7 @@ The Windows auto-update path requires that `service install` (or `service reinst
 
 `service reinstall` is the right command after an out-of-band wheel swap (for example a manual `uv tool install -U` from an Administrator shell). Stop and uninstall are best-effort, so it works even if the service is already gone.
 
-With watcher 1.4.2 and later, `service stop` finishes within about 25 seconds once the watcher is running, even when Data Hub doesn’t answer. It can take longer while the watcher is still starting, because it syncs its config, sends its first heartbeat and scans the watch folder before it looks for a stop. It can also take longer while the watcher applies a config change from Data Hub, which waits up to 2 minutes for uploads to finish. Once the stop begins, the watcher starts no new file or part. A part already on its way gets up to 15 seconds to arrive. In the last 10 seconds, the watcher sends its `watcher_stopped` event and then its final heartbeat, so it shows as **Stopped** when Data Hub answers.
+With watcher 1.4.2 and later, `service stop` finishes within about 25 seconds once the watcher is running, even when Data Hub doesn’t answer. It can take longer while the watcher is still starting. Before watcher 1.4.3 it scanned the whole watch folder first, which takes minutes on a large folder. From 1.4.3 a stop ends the scan at its next file, and the files it hadn’t reached are scanned at the next start. The watcher still syncs its config and sends its first heartbeat before it looks for a stop, so a stop that arrives then waits for Data Hub to answer each request, or for each request to time out after 35 seconds. It can also take longer while the watcher applies a config change from Data Hub, which waits up to 2 minutes for uploads to finish. Once the stop begins, the watcher starts no new file or part. A part already on its way gets up to 15 seconds to arrive. In the last 10 seconds, the watcher sends its `watcher_stopped` event and then its final heartbeat, so it shows as **Stopped** when Data Hub answers.
 
 A file that the watcher had to cut off isn’t lost. When the instrument uploads automatically, the watcher uploads that file again the next time it starts. A file that someone requested with **Upload** is different: `service stop` cancels the requests, so request the file again. A restart of the PC and the stop of an automatic update keep them queued, as [When the PC restarts or shuts down](#when-the-pc-restarts-or-shuts-down) describes. See also [Request uploads](/docs/manage-watchers#request-uploads).
 

docs/site/src/lib/cli-catalog.snapshot.json

@@ -450,5 +450,5 @@
     ]
   },
   "prog": "data-hub-watcher",
-  "version": "1.4.2"
+  "version": "1.4.3"
 }

watcher/e2e/e2e_support/catalog/check.py

@@ -21,7 +21,7 @@
 * a known-bug test follows the pattern: ``xfail(strict=True, raises=KnownBugStillPresent,
   reason="<bug>: ...")``, a ``with known_bug():`` block around the intended-behavior
   assertion, and only ``xfail_bug`` transitions in ``covers``. The reason starts with the bug
-  of one of those transitions (its Linear issue, such as ``ENG-2118``, or a short slug until
+  of one of those transitions (its Linear issue, such as ``ENG-0000``, or a short slug until
   the issue is filed). A test that isn't an xfail never names an ``xfail_bug`` transition.
 
 What counts as a marker: ``@pytest.mark.covers("id", ...)``, ``@pytest.mark.platforms("cli")``

watcher/e2e/e2e_support/catalog/model.py

@@ -56,7 +56,7 @@ class Transition:
       failing loudly (XPASS) the day the bug is fixed, which is when the mark and the
       ``known_bug()`` block come off and the status becomes ``covered``. A broken
       precondition raises something else, so it fails the test instead of hiding as an
-      expected failure. ``bug`` is the Linear issue (``ENG-2118``), or a short slug until the
+      expected failure. ``bug`` is the Linear issue (``ENG-0000``), or a short slug until the
       issue is filed, and the ``reason`` of the test starts with it.
     * ``excluded``: deliberately not tested end to end. ``reason`` says why and where it is
       tested instead. A test must not name it.
@@ -80,7 +80,7 @@ def __post_init__(self) -> None:
         if self.status == "xfail_bug" and not self.bug:
             raise ValueError(
                 f"{self.id}: an xfail_bug transition needs a bug: its Linear issue "
-                "(such as ENG-2118) or a short slug"
+                "(such as ENG-0000) or a short slug"
             )
         if self.status != "xfail_bug" and self.bug:
             raise ValueError(f"{self.id}: only an xfail_bug transition has a bug")

watcher/e2e/e2e_support/catalog/setup_startup.py

@@ -6,7 +6,7 @@
 
 from __future__ import annotations
 
-from e2e_support.catalog.model import CLI, Transition, covered, excluded, xfail_bug
+from e2e_support.catalog.model import CLI, Transition, covered, excluded
 
 MACHINE = "setup_startup"
 
@@ -152,12 +152,14 @@
         "`could not start watching` reason, retries after 1, 2, 4 and 8 heartbeat "
         "intervals, and queues events only when the reason changes.",
     ),
-    xfail_bug(
+    covered(
         MACHINE,
         "waiting.folder_vanished->waiting_then_resumed",
         "When the watch folder disappears while the watcher runs, it moves to `waiting` "
-        "with a reason, and starts watching again when the folder returns.",
-        "ENG-2110",
+        "with a `watch_directory not found` reason, queues one `watch_directory_unavailable` "
+        "event and no second `watcher_started`. When the folder returns, the watcher starts "
+        "watching it again without a restart (a `watcher_started` event says it resumed), and "
+        "a file written to it is uploaded.",
     ),
     # --- left to unit tests -----------------------------------------------------------------
     excluded(

watcher/e2e/e2e_support/catalog/shutdown.py

@@ -8,7 +8,7 @@
 
 from __future__ import annotations
 
-from e2e_support.catalog.model import CLI, Transition, covered, xfail_bug
+from e2e_support.catalog.model import CLI, Transition, covered
 
 MACHINE = "shutdown"
 
@@ -82,18 +82,20 @@
         "every heartbeat and event request for longer than the client waits and asserts that "
         "the process exits within 30 seconds of the stop request.",
     ),
-    xfail_bug(
+    covered(
         MACHINE,
         "process.stop->clean_stop_during_scan",
-        "CLI only. A graceful stop that arrives while the watcher is still starting (the "
-        "config sync, the first heartbeat and the startup scan all run before `watch` "
-        "installs its signal handlers) still sends a `stopped` heartbeat and a "
-        "`watcher_stopped` event. The test holds up the first heartbeat to keep the process "
-        "in that window, because the scan itself is too quick to hit from outside. The "
-        "Windows service has no such window: `SvcStop` sets an event that its loop reads "
-        "when the start has finished, so the stop is not lost. The service counterpart is "
-        "`service.stop->clean_stop_during_start` in `test_windows_service.py`.",
-        "ENG-2118",
+        "CLI only. A graceful stop that arrives while the watcher is still starting (during the "
+        "config sync, the first heartbeat or the start-up scan) is not lost: `watch` installs "
+        "its signal handlers before it starts the supervisor. They call the supervisor's "
+        "`request_stop`, which ends the start-up scan early, so the start returns without "
+        "scanning and the watcher then stops with a `stopped` heartbeat and a `watcher_stopped` "
+        "event. The test holds up the first heartbeat to keep the process in that window, "
+        "because the scan itself is too quick to hit from outside, and checks the log for the "
+        "scan that ended early. (A stop in the middle of a long scan is a unit test: "
+        "`test_cli_watch_stop.py`.) The Windows service hands its stop to the same call from "
+        "before its start; its counterpart is `service.stop->clean_stop_during_start` in "
+        "`test_windows_service.py`.",
         platforms=CLI,
     ),
 )

watcher/e2e/e2e_support/catalog/windows_service.py

@@ -65,9 +65,11 @@
         MACHINE,
         "service.stop->clean_stop_during_start",
         "A `service stop` that arrives while the service is still starting (its first "
-        "heartbeat is held up) is not lost: when the start has finished the service stops "
-        "cleanly, with a `stopped` heartbeat and a `watcher_stopped` event. (The CLI has "
-        "this window and loses the stop, which is `process.stop->clean_stop_during_scan`.)",
+        "heartbeat is held up) is not lost: a thread hands it to the supervisor's "
+        "`request_stop` at once, which ends the start-up scan, and when the first heartbeat "
+        "has returned the service stops cleanly, with a `stopped` heartbeat and a "
+        "`watcher_stopped` event. (The CLI has the same window, which is "
+        "`process.stop->clean_stop_during_scan`.)",
         platforms=SERVICE,
     ),
     covered(

watcher/e2e/e2e_support/waiting.py

@@ -47,12 +47,12 @@ def known_bug() -> Iterator[None]:
     the server holds what it should, the file was written. If any of that fails the test
     fails, and not as an expected failure. Only the intended-behavior assertion goes inside::
 
-        @pytest.mark.xfail(strict=True, raises=KnownBugStillPresent, reason="ENG-2118: ...")
-        @pytest.mark.covers("process.stop->clean_stop_during_scan")
-        def test_a_stop_during_start_up_still_ends_with_a_clean_stop(...):
+        @pytest.mark.xfail(strict=True, raises=KnownBugStillPresent, reason="bug-example: ...")
+        @pytest.mark.covers("machine.thing->intended_outcome")
+        def test_the_intended_behavior(...):
             ...  # precondition
             with known_bug():
-                assert pc.wait_exit(30) == 0
+                assert the_intended_behavior()
 
     An ``AssertionError`` raised inside the block, a :class:`WaitTimeout` included, is
     re-raised as :class:`KnownBugStillPresent` with the original as its cause. When the bug

watcher/e2e/test_setup_startup.py

@@ -25,7 +25,7 @@
 from e2e_support.factory import WatcherFactory
 from e2e_support.files import SMALL, write_run
 from e2e_support.server import ServerView
-from e2e_support.waiting import KnownBugStillPresent, known_bug, stays_true
+from e2e_support.waiting import stays_true
 
 pytestmark = pytest.mark.e2e
 
@@ -677,14 +677,6 @@ def test_a_start_that_keeps_failing_backs_off_and_then_succeeds(
     assert resumed[-1].message.startswith(pc.resumed_prefix)
 
 
-@pytest.mark.xfail(
-    strict=True,
-    raises=KnownBugStillPresent,
-    reason=(
-        "ENG-2110: a watch folder that disappears while the watcher runs "
-        "leaves it reporting `watching` instead of `waiting`"
-    ),
-)
 @pytest.mark.covers("waiting.folder_vanished->waiting_then_resumed")
 def test_a_watch_folder_that_vanishes_puts_the_watcher_back_to_waiting(
     watcher: WatcherFactory, server: ServerView, watch_dir: Path, instrument_id: str
@@ -697,13 +689,20 @@ def test_a_watch_folder_that_vanishes_puts_the_watcher_back_to_waiting(
     # The folder goes away while the watcher runs, as when a network drive drops.
     shutil.rmtree(folder)
 
-    with known_bug():
-        row = server.wait_for_watcher_status(pc.watcher_id, "waiting", timeout=12)
-        assert row.status_reason is not None
-        assert row.status_reason.startswith("watch_directory not found")
+    row = server.wait_for_watcher_status(pc.watcher_id, "waiting", timeout=12)
+    assert row.status_reason is not None
+    assert row.status_reason.startswith("watch_directory not found")
+    # The move reports why, once, and the watcher did not start again: it never stopped.
+    unavailable = server.wait_for_events(
+        pc.watcher_id, event_type="error", kind="watch_directory_unavailable"
+    )
+    assert [e.details["reason"] for e in unavailable] == [row.status_reason]
+    assert count_events(server, pc.watcher_id, "watcher_started") == 1
 
     # And it picks up again when the folder comes back.
     folder.mkdir()
     server.wait_for_watcher_status(pc.watcher_id, "watching")
+    resumed = server.wait_for_events(pc.watcher_id, count=2, event_type="watcher_started")
+    assert resumed[-1].message.startswith(pc.resumed_prefix)
     written = write_run(folder, "RUN001", {"a.csv": SMALL})
     server.wait_for_uploaded(instrument_id, "RUN001", "RUN001/a.csv", content=written["a.csv"])