Skip to content

[IO-591] Tailer stops reading after rotation to an empty file twice - #941

Closed
atharv1205 wants to merge 1 commit into
apache:masterfrom
atharv1205:IO-591-Tailer-rotation
Closed

atharv1205 wants to merge 1 commit into
apache:masterfrom
atharv1205:IO-591-Tailer-rotation

Conversation

@atharv1205

Copy link
Copy Markdown

Jira: https://issues.apache.org/jira/browse/IO-591

Problem

Tailer silently stops delivering lines when the tailed file is rotated to an empty file twice without any write in between (for example, mv app.log app.log.1; touch app.log twice), and the newest file is then written to.

Reproduction on master (844a19b) with a real file: tail app.log, write first, rotate (move, create empty) twice without writing, then write after-two-rotations to the new app.log.

  • Actual: the listener receives [first] only, and nothing more is ever read.
  • Expected: [first, after-two-rotations].

Root cause

In Tailer.run(), a rotation is detected only when length < position. After the first rotation, the position is reset to 0 on the new, empty file. The second empty replacement also has length 0, so the second rotation is never detected and the reader stays open on the renamed file. When data is later written to the path, length > position holds and readLines(reader) is called on the stale reader, which is at end of file forever.

Fix

In the length > position branch, the open reader is probed for one byte at the current position. Only if it is at end of file, a new reader is opened on the path to check for data at the same position. If the new reader has data, the path names another file, so the existing rotation handling runs: the old reader is drained and closed, and the replacement is read from the beginning.

  • Normal tailing is unchanged apart from a one byte read and a seek per poll that has new data. The extra open happens only when the open reader is at end of file while the path is longer.
  • A size that is ahead of the readable data (for example on a caching network file system, or a custom Tailable) is not treated as a rotation, because the new reader is also at end of file. This avoids replaying the whole file or reopening in a tight loop.
  • No public API change.

Tests

Added to TailerRotationTest, deterministic with in-memory Tailable implementations in the style of the existing tests:

  • testRotationToEmptyFileTwiceWithoutWrites: the IO-591 scenario. Fails on master with expected: <[first, after]> but was: <[first]>, passes with the fix.
  • testPartialLineIsNotRotation: a trailing partial line is not mistaken for a rotation.
  • testSizeAheadOfDataIsNotRotation: a size ahead of the readable data is not mistaken for a rotation.

mvn (default goal) passes: 7664 tests, 0 failures; Checkstyle, PMD, SpotBugs, japicmp and RAT are clean.

Before you push a pull request, review this list:

  • Read the contribution guidelines for this project.
  • Read the ASF Generative Tooling Guidance if you use Artificial Intelligence (AI).
  • I used AI for assistance.
  • Run a successful build using the default Maven goal with mvn; that's mvn on the command line by itself.
  • Write unit tests that match behavioral changes, where the tests fail if the changes to the runtime are not applied. This may not always be possible, but it is a best practice.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Each commit in the pull request should have a meaningful subject line and body. Note that a maintainer may squash commits during the merge process.

When the tailed file was rotated to an empty file twice without a write in
between, Tailer kept reading the renamed file and lost all later lines.
Detect this case when the open reader is at end of file but a new reader on
the path has data at the current position.
@garydgregory garydgregory changed the title IO-591: Tailer stops reading after rotation to an empty file twice [IO-591] Tailer stops reading after rotation to an empty file twice Nov 20, 2026
@garydgregory
garydgregory marked this pull request as draft November 21, 2026 22:37
@garydgregory

garydgregory commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

@atharv1205

The PR reproduces the reported failure. However, I don’t think this implementation is ready to merge.

The main problem is that different read results do not establish that the file was replaced. An existing reader can return EOF while a newly opened reader sees an append to the same file. The PR treats this as rotation, drains the original reader, and resets the replacement reader to zero.

A deterministic test reproduces the resulting duplicate delivery:

  • Expected: [first, after]
  • Actual: [first, after, first, after]
    A persistently stale original reader can also cause previously delivered lines to be replayed. Rechecking that reader does not reliably solve this: another visibility change can occur during the recheck.

There are additional concerns:

  • The size-ahead test misses the problematic transition. Its content remains permanently empty, so it never exercises data becoming visible between operations.
  • The FileNotFoundException catch is too broad. It covers seeking, reading, and closing the probe, as well as opening it. Failures after a successful open can therefore be silently treated as an unavailable path.
  • Recovery opens the replacement twice. The probe reader is discarded, and another reader is opened for consumption. This adds I/O and another opportunity for the path to change between checking and reading.
    PR [IO-591] Fix Tailer after consecutive rotations to empty files #942 instead refreshes a stalled reader at its existing offset. For IO-591, that offset is already zero, so this recovers all content written after the empty-file rotations. At nonzero offsets, it avoids replaying earlier lines merely because readers have different views of the data. A refresh does not generate a rotation notification; existing size/path-based rotation handling remains in place.

It also retains the original reader when opening the fresh reader fails, uses the successfully opened reader directly, and limits the optional-open exception handling to the open itself.

The expanded tests cover delayed visibility, persistently stale readers, nonzero offsets, repeated open failures, seek/read/close failures, shutdown, and resource cleanup, alongside real-file rotation tests. The rotation regressions fail against unmodified main, and the replay tests expose the unsafe implementations.

Closing; you are credited in changes.xml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants