Repository navigation
In HadoopAttributeStore, get absolute path for attributePath - #2123
Merged
Merged
Conversation
pomadchin
reviewed
Apr 4, 2017
| val ap = new Path(rootPath, "_attributes") | ||
| val fs = ap.getFileSystem(hadoopConfiguration) | ||
| // Get the absolute path to attributes | ||
| (fs, fs.getFileStatus(ap).getPath) |
Member
There was a problem hiding this comment.
getFileStatus throws FileNotFoundException when the path does not exist
Member
There was a problem hiding this comment.
A straightforward solution would be to use try catch, and to create dirs on exception.
Another solution would be to getFileStatus of a root path and to create only _attributes folder, but that would require manual catalog creation, and still would throw runtime exceptions.
val (fs, attributePath) = {
val ap = new Path(rootPath, "_attributes")
val fs = ap.getFileSystem(hadoopConfiguration)
// Get the absolute path to attributes
try {
fs.getFileStatus(ap)
} catch {
case _: FileNotFoundException => fs.mkdirs(ap)
}
(fs, fs.getFileStatus(ap).getPath)
}
Member
There was a problem hiding this comment.
Oh, try catch way would be ok, here is exists method implementation:
public boolean exists(Path f) throws IOException {
try {
return getFileStatus(f) != null;
} catch (FileNotFoundException e) {
return false;
}
}The result smth like:
val (fs, attributePath) = {
val ap = new Path(rootPath, "_attributes")
val fs = ap.getFileSystem(hadoopConfiguration)
// Create directory if it doesn't exist
if(!fs.exists(ap)) fs.mkdirs(ap)
// Get the absolute path to attributes
(fs, fs.getFileStatus(ap).getPath)
}
Signed-off-by: Grigory Pomadchin <[email protected]>
pomadchin
force-pushed
the
fix/hadoop-layer-exists
branch
from
April 4, 2017 06:10
c4ee998 to
5beed5c
Compare
lossyrob
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2113
What was happening is, because the attributePath was being stored as a local path, the
attributePathmethod returned the relative path as well; when being compared to listed paths inlayerExists, it would always fail the match. This didn't surface itself beforehand because we would always use absolute paths with HadoopLayer types; this prevents a problem if a user decides to user a relative path when the default filesystem is local.This was a bit tough to capture in a unit test, but I did test this manually against geotrellis-landsat-tutorial.