Skip to content

[fs] Expose storage class and Jindo's directory listing iterator - #11247

Open
tonymtu wants to merge 2 commits into
apache:masterfrom
tonymtu:feat/iterative-directory-file-status
Open

tonymtu wants to merge 2 commits into
apache:masterfrom
tonymtu:feat/iterative-directory-file-status

Conversation

@tonymtu

@tonymtu tonymtu commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Jindo returns the OSS storage class in directory listings. Paimon currently drops it when wrapping each entry in FileStatus. This PR passes it through FileStatus.getStorageClass() for both array and iterator listings.

It also exposes Jindo's listStatusIterator through FileIO.listStatusIterative(Path) and forwards the call through the FileIO wrappers.

Tests

  • Built the Jindo plugin and its dependencies. All 115 selected FileIO and Jindo tests passed, including wrapper delegation, page errors and the single-file case.
  • Tested on OSS with Jindo 6.9.1, 100 keys per page, 205 immediate files across five storage classes, and one subdirectory. All three methods returned the same files over six traversals each. Both directory listing methods returned the subdirectory and the correct class for every file. Also checked a single Archive object and an empty directory.

OSS access logs gave the following request counts per traversal. All metadata HEAD requests targeted the directory.

Method Data page LIST Directory probe LIST Directory HEAD Total
listFilesIterative(path, false) 3 3 6 12
listStatusIterative(path) 3 3 6 12
listStatus(path) 3 1 2 6

@tonymtu
tonymtu marked this pull request as draft November 18, 2026 10:18
@tonymtu
tonymtu marked this pull request as ready for review November 18, 2026 10:25
@tonymtu tonymtu changed the title [fs] Add iterative directory listings with optional storage class [fs] Expose storage class and Jindo's directory listing iterator Nov 18, 2026

@JingsongLi JingsongLi left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directory pagination and storage-class metadata are useful for OSS lifecycle tools. The 67 selected FileIO/wrapper tests passed on macOS JDK 11. All 7 Jindo tests passed separately on Linux amd64 JDK 11, where the Jindo native library is supported. The SDK path normalization case below remains uncovered.

if (listedDirectory != null
&& status instanceof JindoHadoopFileStatus
&& status.isFile()
&& listedDirectory.equals(status.getPath().getParent())) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Normalize the listed directory before comparing its child path

new Path(URI.create("oss://bucket/dir/")) preserves the trailing slash through HadoopCompliantFileIO.path and Hadoop makeQualified. Jindo 6.9.1 canonicalizes the directory and returns oss://bucket/dir as the child's parent. That makes this equality false in both listStatus and listStatusIterative, so an Archive file is reported with no storage class. I reproduced it with a Jindo SDK Archive file status fed to this exact wrapper. The wrapper returned null with the URI trailing slash and Archive with the normalized path. Please normalize the requested URI with the same Path rules as the SDK and cover both listing entry points.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. Fixed in 1606936 by normalizing the directory path the same way Jindo does before comparing it with the child’s parent. I added regression coverage for both listStatus and listStatusIterative and verified on OSS with Jindo 6.9.1 that both methods report Archive with or without the trailing slash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants