Repository navigation
Conversation
There was a problem hiding this comment.
Directory pagination and storage-class metadata are useful for OSS lifecycle tools. The 67 selected FileIO/wrapper tests passed on macOS JDK 11. All 7 Jindo tests passed separately on Linux amd64 JDK 11, where the Jindo native library is supported. The SDK path normalization case below remains uncovered.
| if (listedDirectory != null | ||
| && status instanceof JindoHadoopFileStatus | ||
| && status.isFile() | ||
| && listedDirectory.equals(status.getPath().getParent())) { |
There was a problem hiding this comment.
[P2] Normalize the listed directory before comparing its child path
new Path(URI.create("oss://bucket/dir/")) preserves the trailing slash through HadoopCompliantFileIO.path and Hadoop makeQualified. Jindo 6.9.1 canonicalizes the directory and returns oss://bucket/dir as the child's parent. That makes this equality false in both listStatus and listStatusIterative, so an Archive file is reported with no storage class. I reproduced it with a Jindo SDK Archive file status fed to this exact wrapper. The wrapper returned null with the URI trailing slash and Archive with the normalized path. Please normalize the requested URI with the same Path rules as the SDK and cover both listing entry points.
There was a problem hiding this comment.
Thanks for catching this. Fixed in 1606936 by normalizing the directory path the same way Jindo does before comparing it with the child’s parent. I added regression coverage for both listStatus and listStatusIterative and verified on OSS with Jindo 6.9.1 that both methods report Archive with or without the trailing slash.
Purpose
Jindo returns the OSS storage class in directory listings. Paimon currently drops it when wrapping each entry in
FileStatus. This PR passes it throughFileStatus.getStorageClass()for both array and iterator listings.It also exposes Jindo's
listStatusIteratorthroughFileIO.listStatusIterative(Path)and forwards the call through the FileIO wrappers.Tests
OSS access logs gave the following request counts per traversal. All metadata HEAD requests targeted the directory.
listFilesIterative(path, false)listStatusIterative(path)listStatus(path)