Repository navigation
Scan assets from disk - #16413
Scan assets from disk#16413
Conversation
984450d to
95ad2f7
Compare
|
@tvolkert I'd appreciate help with the windows failed test, I'm stuck at it without access to windows machine. Can't reproduce on my OS X. Stack trace is not helpful. Should I add loads of print() to debug AppVeyor? BTW I wrote the failed unit test. |
|
@fmatosqg actually, see the discussion in #14246. The fix to google/file.dart#68 just landed (and as of #16442 is rolled into Flutter), so the right answer is actually to update the asset tests to use /cc @goderbauer |
|
@tvolkert I'm going on a rabbit hole, I'd appreciate some guidance.
|
|
Sorry - I started reviewing the PR, and before we dive too deeply in fixing the tests, let's take a step back on the design. Rather than having a
|
|
@tvolkert Thanks for review, some feedback on feedback: That makes sense, though I'm not sure how to interpret some entries with globstar on variants. I'm not familiar with globstar so I'll draft some unit tests right here. Consider these files:
Another example:
Can you show some places where globstar is supported? If they also deal with variants I can try to see how they behave. |
|
I wrote up a sample of how this could work -- https://gist.github.com/tvolkert/40b0bcad883dd843f5829a881ddf82b1 Basically, it takes a raw listing of asset files (split out by package) and deduces which are the base assets versus the variants. |
|
@tvolkert thx for gist, I'll definitely be having a look. First things first, it's hard for me to keep up the pace for lots of days and while globstar and subdir scanning is definitely a good thing, I got rejected on CI because of windows/posix, so I have to aim at there first. Hopefully me or somebody else can pick up from there. The good news is that I got a way of making it work with I had the same old "globals make test impossible" problem. I had to pass fs as parameter everywhere, and as far as I remember it was mostly limited to assets.dart. Is there any smarter way of replacing/overriding the Well, almost there. |
|
@fmatosqg thanks for all your work on this! Check out the following test: It uses |
|
Amazing, extremely simplifies things :) I can think of some OO tricks with wrapping everything in a class and calling main twice. Or extending |
|
I'd create a void runTests(FileSystemStyle style) {
group('...', () {
MemoryFileSystem fs;
setUp(() {
fs = new MemoryFileSystem(style: style);
});
testUsingContext('...', () {
...
}, overrides: <Type, Generator>{
FileSystem: () => fs,
});
});
}... then just call runTests(FileSystemStyle.posix);
runTests(FileSystemStyle.windows); |
|
@tvolkert I made some progress on windows testing under OS X. I tried to make copies from one file system to another as part of the test preparation but it's another rabbit hole I'm into. I didn't try doing it with setupAll, that will be tomorrow. And now that I think, I'd expect the posix test to fail as well, since the schema is coming from the disk and the test rig is reading/writing on MemoryFileSystem. But to be honest I didn't read its code or docs yet. |
|
Hi @tvolkert
At best I get If modifying makes everything work, but I'm afraid that io.File may not be using the right file system. also works. Oh, and flutter_manifest seems to be sending |
|
TL;DR for above post, what is correct, is it right to have the following unit test? where flutter_manifest has: Currently that unit test will make sure we're compatible with |
|
Freezing work on MemFs for a while, making no progress for quite some time. Last almost stable version is on https://github.com/fmatosqg/flutter/tree/asset_scan_memfs fbda4f1 Going back to original work on scanning paths according to agreed specs. |
95ad2f7 to
05d457a
Compare
|
Checklist:
And my goal: Use Dart poet to generate class from bundled assets. |
05d457a to
b7112e1
Compare
|
@fmatosqg that you for your hard work on this! FYI, I'm probably not going to have time to review until at least Wednesday this week. Just didn't want to give the impression I'm ignoring the review or ungrateful 😄 |
|
Wed is probably before I'll be ready. I'm starting to think it's unlikely to pass through windows tests without modifying other files such as I can write tests for memory_filesystem and others to expose my problems, but I'm not sure I'd know what's the expected behaviour. Things like who uses |
|
Not ready for review. Needs extensive cleaning. |
|
@tvolkert I'd appreciate your feedback on the latest commit b3e83a1 for specifically 2 files that are part of the rabbit hole - One thing that I can't figure out is how to detect it's a windows filesystem when on LocalFileSystem. I need that to feed Ideally I'd like to work on making those files close to "LGTM" before code reviewing |
|
Suggestion from gitter dart: use Platform when fs is LocalFileSystem. Then I'd use style when StyleableFileSystem. |
|
@tvolkert this is it, it's ready for code review. I'll push the squash with the first review change. I'd be more comfortable wrapping this work without globstar for now, there's a couple of things I'd consider more critical such as adding new file to disk and re-generating bundle for hot reloads, or adding docs. Also because I'd like to take a break from this code. Don't feel pressured, but I'd appreciate a signal in case you're busy but are aware of this state. Thanks for the guidance on the critical steps, I totally understand that this is not a priority ticket and everybody has other responsibilities. |
|
Hi, thanks for your contribution! @tvolkert will probably continue reviewing your PR next week but I was wondering if you could briefly describe what the rules you've implemented are since it wasn't quite clear to me from the patch what the intent was. Is the current rule just "if you specify a directory, we will include all the assets in that directory"? |
|
@Hixie thx for quick response. I'll probably have docs edited in next few days along with an example app, I hope they help understand what it is about. |
Updates doc to reflect changes on flutter/flutter#16413
|
Is Option B + globstar necessary to make this PR unanimously desirable? I'd be willing to move forward quickly if the answer is yes. I particularly think this would be skewing the sdk further away from native mobile practices but I'm ok if it helps to move on. The biggest problem for me is that I have little control of my free time to work on this and in a few weeks I won't have any - like most of the year. |
|
Given the following files: And a pubspec of: I'd expect there to be an error where we say we can't find |
|
@HansMuller may know what the current intent is and the history behind it, I think he was the last one to work on this. |
|
As of #10901, it's not necessary to include the main asset, when resolution-specific variants are defined. For example `./asssets/2.png' isn't strictly needed in #16413 (comment) |
|
@tvolkert Note that Option B.2 forces dev to reference the asset as |
|
Also I'm assuming that mdpi files (1x) are less important given that they are rarely used - few phones or devices will pick them if a higher-res version is available. While xxhdpi (3x) are mandatory for all practical purposes: an app without it will suffer from low-res images in most phones. |
|
I also found a very curious case when fiddling with pubspec and checking why it didn't display as the size I expected: #17241. It's important to this ticket because |
tvolkert
left a comment
There was a problem hiding this comment.
This looks great! Almost all of my comments are just style nits.
| for (_Asset main in assetVariants.keys) { | ||
|
|
||
| // necessary for making unit tests deterministic | ||
| final List<_Asset> sortedKeys = assetVariants |
There was a problem hiding this comment.
Nit: you can extract the comparator into a top-level private method, like so:
int _byBasename(_Asset a, _Asset b) {
return a.assetFile.basename.compareTo(b.assetFile.basename);
}
...
final List<_Asset> sortedKeys = assetVariants
.keys()
.toList()
..sort(_byBasename);| if (_cache[directory] == null) { | ||
| final List<String> paths = <String>[]; | ||
| for (FileSystemEntity entity in fs.directory(directory).listSync(recursive: true)) { | ||
| for (FileSystemEntity entity in fs.directory(directory).listSync( |
| /// assets/bar | ||
| /// | ||
| /// returns | ||
| /// {asset: packages/test_package/assets/foo: |
There was a problem hiding this comment.
Can you format this with one entry per line to make it easier to read:
{
asset: packages/test_package/assets/foo: [
asset: packages/test_package/assets/foo,
asset: packages/test_package/assets/var1/foo,
asset: packages/test_package/assets/var2/foo,
],
asset: packages/test_package/assets/bar: [
asset: packages/test_package/assets/bar
],
}
| entryUri: entryUri, | ||
| relativeUri: relativeUri, | ||
| )); | ||
| if ( assetUri.toString().endsWith('/')){ |
There was a problem hiding this comment.
Nit: no space after open parens, and yes space after close parens (before the curly brace)
There was a problem hiding this comment.
good catch, I used default dart formatter and it did as you say
| /// | ||
| /// Returns null on missing assets. | ||
| /// | ||
| /// Given package: 'test_package' and an assets directory like this: |
There was a problem hiding this comment.
Alas, it looks like the json-schema dart package doesn't support the $comment keyword, so we can't add comments to the schema to document the syntax of assets in the manifest there.
However, there is https://flutter.io/assets-and-images/ -- can you also send a PR to update the documentation on the website that'll go in tandem with this?
There was a problem hiding this comment.
Done 2 weeks ago: flutter/website#992.
Not sure how to coordinate the merge, would you like me to remind you when this merge lands on beta so you can merge the flutter/website?
There was a problem hiding this comment.
Yeah, that'd be great. thanks!
| establishFlutterRoot(); | ||
| writeSchema(schema, windowsFs); | ||
| await testMethod(); | ||
| }, |
There was a problem hiding this comment.
}, overrides: <Type, Generator>{
...
});| establishFlutterRoot(); | ||
| writeSchema(schema, posixFs); | ||
| await testMethod(); | ||
| }, |
There was a problem hiding this comment.
}, overrides: <Type, Generator>{
...
});| FileSystem: () => windowsFs, | ||
| Platform: () => | ||
| new FakePlatform(environment: <String, String>{ | ||
| 'FLUTTER_ROOT': _kFlutterRoot, |
There was a problem hiding this comment.
Just put the environment map on one line to make the indentation easier to read.
| FileSystem: () => posixFs, | ||
| Platform: () => | ||
| new FakePlatform(environment: <String, String>{ | ||
| 'FLUTTER_ROOT': _kFlutterRoot, |
There was a problem hiding this comment.
Just put the environment map on one line to make the indentation easier to read.
| testUsingContextAndFs( | ||
| 'Expect error generating manifest, wrong non-existing directory is listed', () async { | ||
| establishFlutterRoot(); | ||
|
|
There was a problem hiding this comment.
Just one newline (same below)
|
I did my best to follow all suggestions but by the number of them it was hard to make sure nothing slipped through. Should I squash both commits and append comment |
|
Thanks. Don't worry about squashing commits - I'll squash when it's merged. |
|
@fmatosqg Thank you very much for your hard work on this. It is very much appreciated. For my use case, I have a lot of html files that I download from firebase based on user's choice, during app setup. Then I create a local jaguar server then serve these files. Without this feature, I'd have to create my own script that goes and finds all the files in a specific folder recursively then add those into pubspec.yaml. I hope this gets wrapped up in a couple of days! |
|
@MaskyS timeframes for this PR are better measured in weeks lol. That said I have figured out the last of @tvolkert's comments, but I'm quite short on time to clean it and roll it for all the places that need it. Just a warning, the lastest version of the PR doesn't cover recursive scanning. That can be quite tricky because of some magic on how image with different resolutions are treated. That said, I find it very unorthodox to use more than a couple of htmls in the whole app, and even less ortohodox to have them bundled with the app during build. |
|
@fmatosqg if you like, I can merge this PR and try to make the changes I'm suggesting in a follow-on change since they get quite deep in the weeds of Dart. Would you like to proceed with that plan? |
|
@tvolkert sounds awesome to me :) let me send you a commit that has everything figured out for |
|
@tvolkert I appreciate the help. My latest code is here. https://github.com/fmatosqg/flutter/tree/scan_assets_from_disk_needs_cleaning
Reminder, pending checking/doing something on |
|
Thanks again for the contribution and your patience with the long code review! 😄 |
|
likewise 🥇 |
|
Thanks so much @fmatosqg for this very important improvement! |
Updates doc to reflect changes on flutter/flutter#16413

Opt in parameter will scan for all assets under given folder.
To do: Scenario when app is built, then user adds one more asset without touching pubspec.yaml. It's currently not refreshing bundle, but comparing timestamp for both pubspec and list of files under scanned path will be enough.
Covers part of #4890