Skip to content

Scan assets from disk - #16413

Merged
tvolkert merged 3 commits into
flutter:masterfrom
fmatosqg:scan_assets_from_disk
May 21, 2018
Merged

tvolkert merged 3 commits into
flutter:masterfrom
fmatosqg:scan_assets_from_disk

Conversation

@fmatosqg

Copy link
Copy Markdown
Contributor

Opt in parameter will scan for all assets under given folder.

To do: Scenario when app is built, then user adds one more asset without touching pubspec.yaml. It's currently not refreshing bundle, but comparing timestamp for both pubspec and list of files under scanned path will be enough.

Covers part of #4890

@fmatosqg
fmatosqg force-pushed the scan_assets_from_disk branch from 984450d to 95ad2f7 Compare April 10, 2018 15:19
@fmatosqg

fmatosqg commented Apr 10, 2018 •

Copy link
Copy Markdown
Contributor Author

@tvolkert I'd appreciate help with the windows failed test, I'm stuck at it without access to windows machine.

Can't reproduce on my OS X. Stack trace is not helpful. Should I add loads of print() to debug AppVeyor?

BTW I wrote the failed unit test.

@tvolkert

Copy link
Copy Markdown
Contributor

@fmatosqg actually, see the discussion in #14246. The fix to google/file.dart#68 just landed (and as of #16442 is rolled into Flutter), so the right answer is actually to update the asset tests to use MemoryFileSystem, with a style of FileSystemStyle.windows for the windows tests.

/cc @goderbauer

@fmatosqg

fmatosqg commented Apr 11, 2018 •

Copy link
Copy Markdown
Contributor Author

@tvolkert I'm going on a rabbit hole, I'd appreciate some guidance.

  • Should I replace all instances I find of fs.file(), fs.isFileSync() etc with something coming from MemoryFileSystem ? If yes, removing import 'base/file_system.dart'; from all related files is the way to get started and find all offending uses?
  • Should I use FileSystemStyle.windows for all build systems, and if not how would I do the check?
  • can I still use getReplaySource('', 'a/b/c') on windows or do I need to go through URI conversion, and how?

@tvolkert

Copy link
Copy Markdown
Contributor

Sorry - I started reviewing the PR, and before we dive too deeply in fixing the tests, let's take a step back on the design.

Rather than having a enable-asset-scan attribute in the pubspec, I think we should just respect asset entries that follow a certain pattern, such as:

  1. Any entry that ends with a slash is treated as a directory, and we include all files within the directory (recursively)
  2. We support globstar-style syntax (the pattern ‘**’ used in a filename expansion context will match all files and zero or more directories and subdirectories.)

@fmatosqg

fmatosqg commented Apr 11, 2018 •

Copy link
Copy Markdown
Contributor Author

@tvolkert Thanks for review, some feedback on feedback:

That makes sense, though I'm not sure how to interpret some entries with globstar on variants.

I'm not familiar with globstar so I'll draft some unit tests right here.

Consider these files:

a/file
a/b/file
a/b/c1/file
a/b/c2/file
  • If I aim at a/ then we should get asset a/file with variant a/b/file
  • If I aim at a/b/**/ it's still easy to tell but tricky to implement, asset is a/b/file and it has 2 variants a/b/c1/file and a/b/c2/file
  • Now if we do a/**/ things get messy and I don't know how to tell variants from main assets.

Another example:
disk is

d/e/file
  • Importing d/ will get asset file without main asset d/file but with variant d/e/file
  • Importing d/e/ is still unambiguous, asset d/e/file without a variant
  • But d/**/ could mean either.

Can you show some places where globstar is supported? If they also deal with variants I can try to see how they behave.
Pls advise.

@tvolkert

Copy link
Copy Markdown
Contributor

I wrote up a sample of how this could work -- https://gist.github.com/tvolkert/40b0bcad883dd843f5829a881ddf82b1

Basically, it takes a raw listing of asset files (split out by package) and deduces which are the base assets versus the variants.

@fmatosqg

Copy link
Copy Markdown
Contributor Author

@tvolkert thx for gist, I'll definitely be having a look.

First things first, it's hard for me to keep up the pace for lots of days and while globstar and subdir scanning is definitely a good thing, I got rejected on CI because of windows/posix, so I have to aim at there first. Hopefully me or somebody else can pick up from there.

The good news is that I got a way of making it work with .posix but .windows is breaking unit test everywhere. I guess having tests is good...

I had the same old "globals make test impossible" problem. I had to pass fs as parameter everywhere, and as far as I remember it was mostly limited to assets.dart. Is there any smarter way of replacing/overriding the fs global coming from base/file__system.dart with something being injected in the test, but still reverting to default value?

Well, almost there.

@tvolkert

Copy link
Copy Markdown
Contributor

@fmatosqg thanks for all your work on this!

Check out the following test:

}, overrides: <Type, Generator>{

It uses testUsingContext() and overrides the FileSystem injected value with a MemoryFileSystem

@fmatosqg

fmatosqg commented Apr 12, 2018 •

Copy link
Copy Markdown
Contributor Author

Amazing, extremely simplifies things :)
Last MemoryFileSystem question (for today lol),
How do I make everything run with both .posix and .windows on my laptop? So I know it won't fail on neither CIs.

I can think of some OO tricks with wrapping everything in a class and calling main twice. Or extending testUsingContext so it runs with 2 possible values for fs.

@tvolkert

tvolkert commented Apr 12, 2018 •

Copy link
Copy Markdown
Contributor

I'd create a runTests(FileSystemStyle style) method, like so:

void runTests(FileSystemStyle style) {
  group('...', () {
    MemoryFileSystem fs;

    setUp(() {
      fs = new MemoryFileSystem(style: style);
    });

    testUsingContext('...', () {
      ...
    }, overrides: <Type, Generator>{
      FileSystem: () => fs,
    });
  });
}

... then just call runTests() twice:

runTests(FileSystemStyle.posix);
runTests(FileSystemStyle.windows);

@fmatosqg

fmatosqg commented Apr 12, 2018 •

Copy link
Copy Markdown
Contributor Author

@tvolkert I made some progress on windows testing under OS X.
Everything is getting redirected to the MemoryFileSystem as expected, but there's one nice surprise:when validating the manifest (flutter_manifest.dart on _validate method) it needs to load flutter_tools/schema/pubspec_yaml.json from a real file system.

I tried to make copies from one file system to another as part of the test preparation but it's another rabbit hole I'm into. I didn't try doing it with setupAll, that will be tomorrow.

And now that I think, I'd expect the posix test to fail as well, since the schema is coming from the disk and the test rig is reading/writing on MemoryFileSystem. But to be honest I didn't read its code or docs yet.

@fmatosqg

fmatosqg commented Apr 15, 2018 •

Copy link
Copy Markdown
Contributor Author

Hi @tvolkert
I'm stuck at trying to load the schema file when switching to MemoryFileSystem with window style. I can't figure out a number of things, like

  • what's the expected format for the string on createSchemaFromUrl, if starting with file://C:/ or /C:/ (first one seems to work better when hardcoded
  • how to generate such string. The beggining parts seem to be easy to find for POSIX but I can't figure a safe way to build it
  • how to safely generate the file at the expected place createSchemaFromUrl will try to read when called from flutter_manifest.dart _validate. I can generate the "expected" folder and file either with dart:io or from the MemoryFileSystem, and I can successfully read from that file, although the contents are not what I expect. Though skipping either folder or file generation results in an error.

At best I get FileSystemException: Cannot open file, path = '/C:/Users/fmatos/code/android/flutter/fwk_flutter/packages/flutter_tools/schema/pubspec_yaml.json' (OS Error: No such file or directory, errno = 2) in many of those situations, coming from
return new File(filePath).readAsString() on createSchemaFromUrl

If modifying createSchemaFromUrl to instead

final hardcodedSchema = 'file:///Users/fmatos/code/android/flutter/fwk_flutter/packages/flutter_tools/schema/pubspec_yaml.json';
schemaUrl=hardcodedSchema;

makes everything work, but I'm afraid that io.File may not be using the right file system.

var editedSchemaPath = 'file://C:/Users/fmatos/code/android/flutter/fwk_flutter/packages/flutter_tools/schema/pubspec_yaml.json';
schemaUrl=editedSchemaPath;

also works.

Oh, and flutter_manifest seems to be sending file:///C:/Users/fmatos/code/android/flutter/fwk_flutter/packages/flutter_tools/schema/pubspec_yaml.json which has an extra slash after the expected //

@fmatosqg

fmatosqg commented Apr 15, 2018 •

Copy link
Copy Markdown
Contributor Author

TL;DR for above post, what is correct, file:///C:/ or file://C:/ ?

is it right to have the following unit test?

final uri = fs.path.toUri(schemaPath).toString();
assert(!uri.startsWith('file:///C:/'),'Uri for Windows should be file://C:/');

where flutter_manifest has:

String get schemaPath => fs.path.join(
    fs.path.absolute(Cache.flutterRoot), 'packages', 'flutter_tools', 'schema',
    'pubspec_yaml.json',
  );

Currently that unit test will make sure we're compatible with schema.dart, but it's failing according to (I guess) something written in MemoryFileSystem.dart.

@fmatosqg

Copy link
Copy Markdown
Contributor Author

Freezing work on MemFs for a while, making no progress for quite some time. Last almost stable version is on https://github.com/fmatosqg/flutter/tree/asset_scan_memfs fbda4f1

Going back to original work on scanning paths according to agreed specs.

@fmatosqg
fmatosqg force-pushed the scan_assets_from_disk branch from 95ad2f7 to 05d457a Compare April 16, 2018 14:07
@fmatosqg

fmatosqg commented Apr 16, 2018 •

Copy link
Copy Markdown
Contributor Author

Checklist:

  • implement non-recursive directory scan, write unit tests
  • get appveyor success
  • squash commits
  • code review
  • check for timestamp of pubspec, scanned files
  • code review
  • write documentation (Asset documentation should have a simple intro. #4927), migrate one sample app (gallery?)
  • implement globstar, write unit tests
  • write specs for fonts
  • write non-recursive directory scan for fonts, write unit tests

And my goal: Use Dart poet to generate class from bundled assets.

@fmatosqg
fmatosqg force-pushed the scan_assets_from_disk branch from 05d457a to b7112e1 Compare April 16, 2018 14:15
@tvolkert

Copy link
Copy Markdown
Contributor

@fmatosqg that you for your hard work on this! FYI, I'm probably not going to have time to review until at least Wednesday this week.

Just didn't want to give the impression I'm ignoring the review or ungrateful 😄

@fmatosqg

Copy link
Copy Markdown
Contributor Author

Wed is probably before I'll be ready.

I'm starting to think it's unlikely to pass through windows tests without modifying other files such as flutter_manifest.dart or package_map.dart, they show very odd behaviour when used with memory_filesystem.dart.

I can write tests for memory_filesystem and others to expose my problems, but I'm not sure I'd know what's the expected behaviour. Things like who uses file:///Users or /Users are not clear to me, I'd appreciate if you can direct me some documentation. On windows there seems to be even more variations: C:\ /C:/ file:///C:/ and weird combos of / and \ in the same string and sometimes still working.

@fmatosqg

Copy link
Copy Markdown
Contributor Author

Not ready for review. Needs extensive cleaning.

@fmatosqg

Copy link
Copy Markdown
Contributor Author

@tvolkert I'd appreciate your feedback on the latest commit b3e83a1 for specifically 2 files that are part of the rabbit hole - package_map.dart and flutter_manifest.dart. My cleaning is close to finished in those 2 files, excepted obvious comments and print. But they have odd code that I have no idea how to make better.

One thing that I can't figure out is how to detect it's a windows filesystem when on LocalFileSystem. I need that to feed new Url.fromFile(...,windows:true) in a couple of places.

Ideally I'd like to work on making those files close to "LGTM" before code reviewing assets.dart so we will have the chance to get a focused conversation when going there.

@fmatosqg

Copy link
Copy Markdown
Contributor Author

Suggestion from gitter dart: use Platform when fs is LocalFileSystem. Then I'd use style when StyleableFileSystem.

@fmatosqg

Copy link
Copy Markdown
Contributor Author

@tvolkert this is it, it's ready for code review. I'll push the squash with the first review change.

I'd be more comfortable wrapping this work without globstar for now, there's a couple of things I'd consider more critical such as adding new file to disk and re-generating bundle for hot reloads, or adding docs. Also because I'd like to take a break from this code.

Don't feel pressured, but I'd appreciate a signal in case you're busy but are aware of this state.

Thanks for the guidance on the critical steps, I totally understand that this is not a priority ticket and everybody has other responsibilities.

@Hixie

Hixie commented Apr 28, 2018

Copy link
Copy Markdown
Contributor

Hi, thanks for your contribution! @tvolkert will probably continue reviewing your PR next week but I was wondering if you could briefly describe what the rules you've implemented are since it wasn't quite clear to me from the patch what the intent was. Is the current rule just "if you specify a directory, we will include all the assets in that directory"?

@fmatosqg

Copy link
Copy Markdown
Contributor Author

@Hixie thx for quick response.
That's correct, if an asset listed in pubspec.yaml ends in / it will be treated as a directory and all files included directly under it will be added as if listed directly. Variants are scanned accordingly. No support for recursive folders, globstar or wildcards.

I'll probably have docs edited in next few days along with an example app, I hope they help understand what it is about.

fmatosqg added a commit to fmatosqg/website that referenced this pull request Apr 30, 2018
@fmatosqg

fmatosqg commented May 2, 2018

Copy link
Copy Markdown
Contributor Author

Is Option B + globstar necessary to make this PR unanimously desirable? I'd be willing to move forward quickly if the answer is yes.

I particularly think this would be skewing the sdk further away from native mobile practices but I'm ok if it helps to move on. The biggest problem for me is that I have little control of my free time to work on this and in a few weeks I won't have any - like most of the year.

@tvolkert

tvolkert commented May 2, 2018

Copy link
Copy Markdown
Contributor

Given the following files:

./assets/1.png
./assets/2.0x/1.png
./assets/3.0x/1.png
./assets/3.0x/2.png

And a pubspec of:

assets:
  - assets/1.png
  - assets/2.png

I'd expect there to be an error where we say we can't find assets/2.png. Put another way, I'd expect that you need to have a base asset in order for the variant to be relevant. As I say this though, I realize that I'm not sure if that's what happens today -- @fmatosqg do you know?

@Hixie

Hixie commented May 2, 2018

Copy link
Copy Markdown
Contributor

@HansMuller may know what the current intent is and the history behind it, I think he was the last one to work on this.

@HansMuller

Copy link
Copy Markdown
Contributor

As of #10901, it's not necessary to include the main asset, when resolution-specific variants are defined. For example `./asssets/2.png' isn't strictly needed in #16413 (comment)

@fmatosqg

fmatosqg commented May 3, 2018

Copy link
Copy Markdown
Contributor Author

@tvolkert
This is an example of what's supported today and how it may look like
https://github.com/fmatosqg/flutter_problem/tree/variants_PR_16413

Note that Option B.2 forces dev to reference the asset as assets/3.0x/<filename> in dart code.

image

@fmatosqg

fmatosqg commented May 3, 2018

Copy link
Copy Markdown
Contributor Author

Also I'm assuming that mdpi files (1x) are less important given that they are rarely used - few phones or devices will pick them if a higher-res version is available. While xxhdpi (3x) are mandatory for all practical purposes: an app without it will suffer from low-res images in most phones.

@fmatosqg

fmatosqg commented May 3, 2018

Copy link
Copy Markdown
Contributor Author

I also found a very curious case when fiddling with pubspec and checking why it didn't display as the size I expected: #17241. It's important to this ticket because new Image.asset disagrees with AssetManifest.json.

@tvolkert tvolkert left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Almost all of my comments are just style nits.

for (_Asset main in assetVariants.keys) {

// necessary for making unit tests deterministic
final List<_Asset> sortedKeys = assetVariants

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: you can extract the comparator into a top-level private method, like so:

int _byBasename(_Asset a, _Asset b) {
  return a.assetFile.basename.compareTo(b.assetFile.basename);
}

...

  final List<_Asset> sortedKeys = assetVariants
    .keys()
    .toList()
    ..sort(_byBasename);

if (_cache[directory] == null) {
final List<String> paths = <String>[];
for (FileSystemEntity entity in fs.directory(directory).listSync(recursive: true)) {
for (FileSystemEntity entity in fs.directory(directory).listSync(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't wrap here

/// assets/bar
///
/// returns
/// {asset: packages/test_package/assets/foo:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you format this with one entry per line to make it easier to read:

{
  asset: packages/test_package/assets/foo: [
    asset: packages/test_package/assets/foo,
    asset: packages/test_package/assets/var1/foo,
    asset: packages/test_package/assets/var2/foo,
  ],
  asset: packages/test_package/assets/bar: [
    asset: packages/test_package/assets/bar
  ],
}

entryUri: entryUri,
relativeUri: relativeUri,
));
if ( assetUri.toString().endsWith('/')){

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: no space after open parens, and yes space after close parens (before the curly brace)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, I used default dart formatter and it did as you say

///
/// Returns null on missing assets.
///
/// Given package: 'test_package' and an assets directory like this:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alas, it looks like the json-schema dart package doesn't support the $comment keyword, so we can't add comments to the schema to document the syntax of assets in the manifest there.

However, there is https://flutter.io/assets-and-images/ -- can you also send a PR to update the documentation on the website that'll go in tandem with this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 2 weeks ago: flutter/website#992.
Not sure how to coordinate the merge, would you like me to remind you when this merge lands on beta so you can merge the flutter/website?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that'd be great. thanks!

establishFlutterRoot();
writeSchema(schema, windowsFs);
await testMethod();
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}, overrides: <Type, Generator>{
  ...
});

establishFlutterRoot();
writeSchema(schema, posixFs);
await testMethod();
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}, overrides: <Type, Generator>{
  ...
});

FileSystem: () => windowsFs,
Platform: () =>
new FakePlatform(environment: <String, String>{
'FLUTTER_ROOT': _kFlutterRoot,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just put the environment map on one line to make the indentation easier to read.

FileSystem: () => posixFs,
Platform: () =>
new FakePlatform(environment: <String, String>{
'FLUTTER_ROOT': _kFlutterRoot,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just put the environment map on one line to make the indentation easier to read.

testUsingContextAndFs(
'Expect error generating manifest, wrong non-existing directory is listed', () async {
establishFlutterRoot();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one newline (same below)

@fmatosqg

Copy link
Copy Markdown
Contributor Author

I did my best to follow all suggestions but by the number of them it was hard to make sure nothing slipped through.

Should I squash both commits and append comment #4890 to reference issue number?

@tvolkert

Copy link
Copy Markdown
Contributor

Thanks. Don't worry about squashing commits - I'll squash when it's merged.

@MaskyS

MaskyS commented May 19, 2018

Copy link
Copy Markdown

@fmatosqg Thank you very much for your hard work on this. It is very much appreciated. For my use case, I have a lot of html files that I download from firebase based on user's choice, during app setup. Then I create a local jaguar server then serve these files. Without this feature, I'd have to create my own script that goes and finds all the files in a specific folder recursively then add those into pubspec.yaml. I hope this gets wrapped up in a couple of days!

@fmatosqg

Copy link
Copy Markdown
Contributor Author

@MaskyS timeframes for this PR are better measured in weeks lol. That said I have figured out the last of @tvolkert's comments, but I'm quite short on time to clean it and roll it for all the places that need it.

Just a warning, the lastest version of the PR doesn't cover recursive scanning. That can be quite tricky because of some magic on how image with different resolutions are treated.

That said, I find it very unorthodox to use more than a couple of htmls in the whole app, and even less ortohodox to have them bundled with the app during build.

@tvolkert

Copy link
Copy Markdown
Contributor

@fmatosqg if you like, I can merge this PR and try to make the changes I'm suggesting in a follow-on change since they get quite deep in the weeds of Dart. Would you like to proceed with that plan?

@tvolkert tvolkert left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fmatosqg

Copy link
Copy Markdown
Contributor Author

@tvolkert sounds awesome to me :)

let me send you a commit that has everything figured out for flutter_manifest_test.dart, it's only pending applying the same "generate pubspec.yaml on the correct filesystem" for the other test file that covers asset.dart

@fmatosqg

Copy link
Copy Markdown
Contributor Author

@tvolkert I appreciate the help. My latest code is here. https://github.com/fmatosqg/flutter/tree/scan_assets_from_disk_needs_cleaning

  • code will fail on analyzer
  • flutter_manifest_test.dart should pass any unit tests
  • asset_bundle_package_test.dart should fail only on tests that involve MemoryFileSystem, the fix I was thinking is make testUsingContextAndFs public (from flutter_manifest_test.dart or a new file) and simply reference them.

Reminder, pending checking/doing something on asset.dart needsBuild() for the situation where
1.1 app is running with debug
1.2 developer adds/removes file to a scanned path (I'm almost sure it works but not sure on bad side effects)
2.1 full build
2.2 developer adds/removes file to scanned path
2.3. full build again, no clean. (observing AssetManifest.json and files copied to android intermediate folder will probably tell everything - I think it will work easier for adding files but removing them may be tricky)

@tvolkert
tvolkert merged commit 197d431 into flutter:master May 21, 2018
@tvolkert

Copy link
Copy Markdown
Contributor

Thanks again for the contribution and your patience with the long code review! 😄

@fmatosqg

Copy link
Copy Markdown
Contributor Author

likewise 🥇

@mit-mit

mit-mit commented May 23, 2018

Copy link
Copy Markdown
Member

Thanks so much @fmatosqg for this very important improvement!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants