Repository navigation
[IO-591] Tailer stops reading after rotation to an empty file twice - #941
atharv1205 wants to merge 1 commit into
Conversation
When the tailed file was rotated to an empty file twice without a write in between, Tailer kept reading the renamed file and lost all later lines. Detect this case when the open reader is at end of file but a new reader on the path has data at the current position.
|
The PR reproduces the reported failure. However, I don’t think this implementation is ready to merge. The main problem is that different read results do not establish that the file was replaced. An existing reader can return EOF while a newly opened reader sees an append to the same file. The PR treats this as rotation, drains the original reader, and resets the replacement reader to zero. A deterministic test reproduces the resulting duplicate delivery:
There are additional concerns:
It also retains the original reader when opening the fresh reader fails, uses the successfully opened reader directly, and limits the optional-open exception handling to the open itself. The expanded tests cover delayed visibility, persistently stale readers, nonzero offsets, repeated open failures, seek/read/close failures, shutdown, and resource cleanup, alongside real-file rotation tests. The rotation regressions fail against unmodified main, and the replay tests expose the unsafe implementations. Closing; you are credited in |
Jira: https://issues.apache.org/jira/browse/IO-591
Problem
Tailersilently stops delivering lines when the tailed file is rotated to an empty file twice without any write in between (for example,mv app.log app.log.1; touch app.logtwice), and the newest file is then written to.Reproduction on master (844a19b) with a real file: tail
app.log, writefirst, rotate (move, create empty) twice without writing, then writeafter-two-rotationsto the newapp.log.[first]only, and nothing more is ever read.[first, after-two-rotations].Root cause
In
Tailer.run(), a rotation is detected only whenlength < position. After the first rotation, the position is reset to 0 on the new, empty file. The second empty replacement also has length 0, so the second rotation is never detected and the reader stays open on the renamed file. When data is later written to the path,length > positionholds andreadLines(reader)is called on the stale reader, which is at end of file forever.Fix
In the
length > positionbranch, the open reader is probed for one byte at the current position. Only if it is at end of file, a new reader is opened on the path to check for data at the same position. If the new reader has data, the path names another file, so the existing rotation handling runs: the old reader is drained and closed, and the replacement is read from the beginning.Tailable) is not treated as a rotation, because the new reader is also at end of file. This avoids replaying the whole file or reopening in a tight loop.Tests
Added to
TailerRotationTest, deterministic with in-memoryTailableimplementations in the style of the existing tests:testRotationToEmptyFileTwiceWithoutWrites: the IO-591 scenario. Fails on master withexpected: <[first, after]> but was: <[first]>, passes with the fix.testPartialLineIsNotRotation: a trailing partial line is not mistaken for a rotation.testSizeAheadOfDataIsNotRotation: a size ahead of the readable data is not mistaken for a rotation.mvn(default goal) passes: 7664 tests, 0 failures; Checkstyle, PMD, SpotBugs, japicmp and RAT are clean.Before you push a pull request, review this list:
mvn; that'smvnon the command line by itself.