Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement size analyzer to unzip & parse APK and AOT size snapshot to generate analysis json #62495

Merged
merged 7 commits into from Jul 31, 2020

Conversation

@peterdjlee
Copy link
Contributor

@peterdjlee peterdjlee commented Jul 29, 2020

Description

Implemented SizeAnalyzer where an APK file and an AOT size snapshot can be passed in through analyzeApkSize(apk, aotSizeJson) which returns a Map that contains the breakdown of APK data with AOT snapshot data included under libapp.so. `analyzeApkSize(apk, aotSizeJson) also prints out a top level summary to the terminal as such:
flutter_tools_apk_summary

Related Issues

#51594

Tests

I added a test file analyze_size_test.dart to test that this tool

  • builds APK analysis correctly
  • outputs summary to the command line correctly

Checklist

Before you create this PR, confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I signed the CLA.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I updated/added relevant documentation (doc comments with ///).
  • All existing and new tests are passing.
  • The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
  • I am willing to follow-up on review comments in a timely manner.

Breaking Change

Did any tests fail when you ran them? Please read Handling breaking changes.

  • No, no existing tests failed, so this is not a breaking change.
  • Yes, this is a breaking change.
json.decode(aotSizeJson.readAsStringSync()) as List<dynamic>,
);

// Rearrange children to ensure the type key shows at the top.
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

why does the order of the keys in the output map matter?

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

It's to make sure that the type field isn't just hidden at the bottom of the json because of the giant AOT data. I'm not sure when the user would need to check the type manually by opening up the json, but I was thinking it'd be better to make the type field more visible.


static const int tableWidth = 80;

Future<Map<String, dynamic>> analyzeApkSize({
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

add dart doc describing what this method does. Add a note saying that aotSizeJson can be from instruction sizes or v8 snapshot (if that is correct)

Copy link
Member

@xster xster Jul 29, 2020

From other PR: if this function now does 2 things, reflect it in the name.

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

@@ -34,6 +34,7 @@ dependencies:
native_stack_traces: 0.3.7
build_daemon: 2.1.4
shelf: 0.7.5
vm_snapshot_analysis: ^0.5.0+1
Copy link
Member

@jonahwilliams jonahwilliams Jul 29, 2020

this should be an exact version, then run flutter update-packages --force-upgrade to update the checksums

BufferLogger logger;
FakeProcessManager processManager;

const FakeCommand unzipCommmand = FakeCommand(
Copy link
Member

@jonahwilliams jonahwilliams Jul 29, 2020

for the consts here, I would hoist them out of main at put them at the top level

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

currentParent = currentParent.childByName(pathPart) ??
currentParent.addChild(SymbolNode(pathPart));
}
// TODO: this shouldn't be necessary, https://github.com/dart-lang/sdk/issues/41137
Copy link
Member

@jonahwilliams jonahwilliams Jul 29, 2020

this issue is marked as fixed, is this still necessary

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Removed

return child;
}

List<SymbolNode> get ancestors {
Copy link
Member

@jonahwilliams jonahwilliams Jul 29, 2020

Neither ancestors nor siblings are used in the test, so they currently have 0 coverage

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Removed since those are not used (the entire SymbolNode code was copied from here: #51594 (comment))

return rootNode;
}

/// Prints the paths from currentNode all leaf nodes.
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

make this documentation more specific. This method seems like it is specific to the lib/ directory in an APK. note that here

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done


if (currentNode.children.isNotEmpty && !currentNode.name.contains('libapp.so')) {
for (final SymbolNode child in currentNode.children) {
_printLibDetails(child, totalPath + '/', aotSizeJson);
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

use string concatenation '$totalPath/'

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

int numBytes,
int level, {
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

make these two named and required parameters

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

_printLibDetails(child, totalPath + '/', aotSizeJson);
}
} else {
// Print total path and size if currentPath does not have any chilren.
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

nit: if currentNode does not have any children

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

if (libraryUri != null) ...libraryUri.split('/') else '@stubs',
if (className?.isNotEmpty ?? false) className,
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

i'm surprised that we are checking that className.isNotEmpty and checking that libraryUri != null. I would expect these two checks to be the same. Why are we checking for null on one and empty on the other?

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Removed

} else if (path.endsWith('libflutter.so')) {
childWithPathAsName.name += ' (Flutter Engine)';
}
currentNode.addChild(childWithPathAsName);
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

does childWithPathAsName.addValue(pathsToSize[paths]); need to be called before currentNode.addChild(childWithPathAsName); so that the childWithPathAsName's value is summed into currentNode's value when calling addChild?

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

childWithPathAsName's value should already have been added to currentNode on the previous iteration of the loop.


final List<SymbolNode> sortedSymbols = root.children.toList()
..sort((SymbolNode a, SymbolNode b) => b.value.compareTo(a.value));
for (final SymbolNode node in sortedSymbols.take(10)) {
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

why 10? put in a const with a descriptive name

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

final List<SymbolNode> sortedSymbols = root.children.toList()
..sort((SymbolNode a, SymbolNode b) => b.value.compareTo(a.value));
for (final SymbolNode node in sortedSymbols.take(10)) {
_printEntitySize(node.name, node.value, 4);
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

same comment here. why 4?

Map<String, dynamic> apkAnalysisJson,
List<String> path,
List<dynamic> aotSizeJson,
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

make these required named parameters

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

}

/// Adds breakdown of aot size data as the children of the node at the given path.
Map<String, dynamic> addAotSizeDataToJson(
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 29, 2020

rename to addAotSizeDataToApkAnalysis

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

@xster
Copy link
Member

@xster xster commented Jul 29, 2020

Note the analysis test https://github.com/flutter/flutter/pull/62495/checks?check_run_id=923824683

Also, high level UX suggestion: I think we should list all the files in devtools, but for CLI where space is limited, the various small files aren't that useful to breakdown. I suggest bundling every single file at the root that's <1kB into just one line that says Others. If users want to know, they can click into the devtool.

Copy link
Member

@xster xster left a comment

Great start!

import 'logger.dart';

class SizeAnalyzer {
SizeAnalyzer(this._fileSystem, this._logger, this._processUtils);
Copy link
Member

@xster xster Jul 29, 2020

Using required named parameters will be clearer than positional parameters here.

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done


static const int tableWidth = 80;

Future<Map<String, dynamic>> analyzeApkSize({
Copy link
Member

@xster xster Jul 29, 2020

From other PR: if this function now does 2 things, reflect it in the name.

// TODO: implement Windows.
String unzipOut;
try {
unzipOut = (await _processUtils.run(<String>[
Copy link
Member

@xster xster Jul 29, 2020

From other PR: using zipinfo rather than unzip could be an optimization (feel free to do it in a separate PR)

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Added a TODO

// TODO(peterdjlee): Add aot size for all platforms.
apkAnalysisJson = addAotSizeDataToJson(
apkAnalysisJson,
'lib/arm64-v8a/libapp.so (Dart AOT)'.split('/'),
Copy link
Member

@xster xster Jul 29, 2020

add code comments here for what this is doing

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 29, 2020

Done

}
}

class SymbolNode {
Copy link
Member

@xster xster Jul 29, 2020

This can probably be private. Also add comments

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 30, 2020

Done

''';

setUp(() {
fileSystem = MemoryFileSystem.test();
Copy link
Member

@xster xster Jul 29, 2020

ah is this why those parameters were in the constructor? There is already a dependency injection mechanism. See other tests using testUsingContext. Then you can remove those construction params.

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 30, 2020

@jonahwilliams helped me put in those constructors explaining we might not want to use the global keyword. Would removing the construction params force us to use the global keyword?

Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

Yes, new code should not use the global zone injection - it makes testing the code very brittle as we've found that new contributors frequently do not understand which interfaces need to be overriden.

Copy link
Member

@xster xster Jul 30, 2020

So the new consensus is to do constructor injection through the entire graph? Is there a tracking bug to migrate existing code?

Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

#47161 , but I haven't done a good job of keeping it updated

@peterdjlee
Copy link
Contributor Author

@peterdjlee peterdjlee commented Jul 30, 2020

Analysis test is failing with trailing U+0020 space character for a few lines. What does this mean? Couldn't find any documentation on it.

@@ -0,0 +1,166 @@
// Copyright 2014 The Flutter Authors. All rights reserved.
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 30, 2020

2020

Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

confusingly it should be 2014

}
return json;
}
}
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 30, 2020

new line at end of file

);
logger.printStatus('━' * tableWidth);
final Directory tempApkContent = fileSystem.systemTempDirectory.createTempSync('flutter_tools.');
// TODO: implement Windows.
Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

This needs a bug assigned to it

Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

tempApkContent.path
])).stdout;
} on Exception catch (e) {
print(e);
Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

this should be a printError and not a regular print

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 30, 2020

Done

// 'path/to/file' where file = 1500 => pathsToSize[['path', 'to', 'file']] = 1500
for (final String line in const LineSplitter().convert(unzipOut)) {
// Expression to match 'Size' column to group 1 and 'Name' column to group 2.
final RegExp parseUnzipOutput = RegExp(r'^\s*\d+\s+[\w|:]+\s+(\d+)\s+.* (.+)$');
Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

Prefer hoisting this into a private static, to avoid recompiling the regexp every time it is created

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 30, 2020

Done

/// A pretty printer for an entity with a size.
void _printEntitySize(
String entityName, {
@required int byteSize,
Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

formatting is off a bit here, indent the required params by 2 spaces

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 30, 2020

Done

// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
import 'package:file/memory.dart';
Copy link
Member

@jonahwilliams jonahwilliams Jul 30, 2020

you need a newline here to make analysis pass

@jonahwilliams
Copy link
Member

@jonahwilliams jonahwilliams commented Jul 30, 2020

You should be able to sync to the latest master to get a correctly formatted pubspec

import 'package:flutter_tools/src/base/process.dart';
import 'package:flutter_tools/src/base/terminal.dart';
Copy link
Member

@kenzieschmoll kenzieschmoll Jul 31, 2020

Do these need to be relative imports to fix the analyze bot?

Copy link
Contributor Author

@peterdjlee peterdjlee Jul 31, 2020

Looks like other files have relative imports! Changed to relative imports.

// TODO(peterdjlee): Implement a way to unzip the APK for Windows. See issue #62603.
String unzipOut;
try {
// TODO(peterdjlee): Use zipinfo instead of unzip.
Copy link
Member

@jonahwilliams jonahwilliams Jul 31, 2020

FYI, as far as I know we do not document requiring zipinfo to be installed, so switching may not be straightforward

Copy link
Member

@xster xster Jul 31, 2020

both zipinfo and unzip are built-in on macOS and neither are built-in on Ubuntu so we're more or less in the same position either way.

Copy link
Member

@jonahwilliams jonahwilliams Jul 31, 2020

No, we're not in the same state - we require unzip and warn if it is not installed and do no such thing for zipinfo

Copy link
Member

@xster xster Jul 31, 2020

ah I see, SG, let's leave it then. This makes it simpler.

Copy link
Member

@jonahwilliams jonahwilliams left a comment

LGTM

@jonahwilliams
Copy link
Member

@jonahwilliams jonahwilliams commented Jul 31, 2020

As soon as https://cirrus-ci.com/task/5692958575951872

Cirrus CI Remove

01:09 clone
00:27 pub
00:02 flutter_pkg
00:15 artifacts
05:14 setup

finishes this should be good to land. The g3 breakage is expected because we've added a new dependency (not in this PR but a prior one)

@peterdjlee peterdjlee merged commit 8b39af2 into flutter:master Jul 31, 2020
32 of 33 checks passed
Pragya007 added a commit to Pragya007/flutter that referenced this issue Aug 11, 2020
… generate analysis json (flutter#62495)

* Implement size analyzer to unzip & parse APK and AOT size snapshot to generate analysis json
mingwandroid added a commit to mingwandroid/flutter that referenced this issue Sep 6, 2020
… generate analysis json (flutter#62495)

* Implement size analyzer to unzip & parse APK and AOT size snapshot to generate analysis json
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants