Commit graph

50567 commits

Author SHA1 Message Date
Kiyoung Kim
ebe7c9c92f Create common.LoadDictionaryFromFile
There are some places defining same file open function and use
common.LoadDictionaryFromLines. This commit creates
LoadDictionaryFromFile to reduce some code redundancy.

Test: m -j & atest passed
Change-Id: I6a3fa48693095937f8c79ce6f3c110b6862a1967
2019-06-26 13:09:52 +09:00
Tao Bao
4daa755fb2 Merge "releasetools: Make blockimgdiff Python 3 compatible." 2019-06-21 14:23:09 +00:00
Treehugger Robot
4ba3984b4c Merge "Remove experimental pass manager for fuzzer builds." 2019-06-21 00:51:26 +00:00
Dan Willemsen
27c05b2405 Merge "Revert "Deprecate bash m* commands and use the soong's modules-in* build commands instead."" 2019-06-20 22:35:06 +00:00
Patrice Arruda
c4dae4d7ab Revert "Deprecate bash m* commands and use the soong's modules-in* build commands instead."
This reverts commit 610b6302cf.

Reason for revert: m mlibc from root dir works. Create a tmp dir and then cd in to tmp. Run the same command and it fails.

Change-Id: I0f0c056d7e668da621ae99d347a055ecdf2dec07
2019-06-20 22:32:46 +00:00
Tao Bao
b813120af1 releasetools: Make blockimgdiff Python 3 compatible.
Bug: 131631303
Test: `python -m unittest test_blockimgdiff`
Test: `python3 -m unittest test_blockimgdiff`
Change-Id: I8cf072ca4af6a525bc675cba0be998ff7be7f4f1
2019-06-20 14:30:30 -07:00
Dan Shi
3771cff834 Merge "Create individual build artifact for test suite build output" 2019-06-20 20:30:34 +00:00
Dan Shi
51b789e422 Create individual build artifact for test suite build output
This change creates following build artifacts that are currently part of
device-tests.zip and general-tests.zip. The smaller build artifacts are
used to reduce the download size required to run TradeFed suite.
Detailed design can be found in the bug.

From device-tests
device-tests_configs: include all test config files in device-tests, zip
  file is less than 1MB.

From general-tests
general-tests_configs: include all test config files in general-tests,
  zip file is less than 1MB.
general-tests_host-shared-libs: include tools and shared libraries
  required to run host side test, zip file is about 60MB

Bug: 73786521
Test: make device-tests and general-tests, verify output.
Change-Id: I34c9aa3c1c672d5d58eab6de92198a7b8e4a681c
2019-06-20 10:02:36 -07:00
Tao Bao
1f1aafa30d Merge "releasetools: Make rangelib Python 3 compatibile." 2019-06-20 14:51:21 +00:00
Treehugger Robot
fde688ebc8 Merge "Fixes a typo in a comment." 2019-06-20 04:20:36 +00:00
Treehugger Robot
345ed53d98 Merge "Enforce property split in GSI" 2019-06-20 01:53:15 +00:00
Treehugger Robot
9bcf0376ab Merge "Deprecate bash m* commands and use the soong's modules-in* build commands instead." 2019-06-20 00:20:23 +00:00
Patrice Arruda
610b6302cf Deprecate bash m* commands and use the soong's modules-in* build commands instead.
The functionality of m* envsetup.sh commands has been implemented in
soong_ui. The bash functionality of m* commands can be replaced by
soong's modules-in* build commands.

Fixes: b/130049705
Test: * Loaded the envsetup.sh on a new environment
      * Executed each m* command.
      * Verified with "type <build command>" to make sure
        the function generation is correct.
      * Ran on Darwin environment.

Change-Id: I8792e4f1f1727aba2ca6ee4bdfe657f6ac42302a
2019-06-19 14:16:38 -07:00
Tao Bao
9521c802ec Merge "releasetools: Make common.ZipWriteStr Python 3 compatible." 2019-06-19 21:09:01 +00:00
Tao Bao
d660c8d8f3 releasetools: Make rangelib Python 3 compatibile.
Bug: 131631303
Test: `python -m unittest test_rangelib`
Test: `python3 -m unittest test_rangelib`
Change-Id: Ia8e26623d5967f2eea003252ee56b861350d626b
2019-06-19 10:30:34 -07:00
Treehugger Robot
435dcbb845 Merge "Disable apex update for GSI" 2019-06-19 10:38:42 +00:00
Treehugger Robot
9871d36024 Merge "Add more OWNERS" 2019-06-19 09:34:14 +00:00
Tao Bao
c1a1ec30df releasetools: Make common.ZipWriteStr Python 3 compatible.
Python 2 and 3 behave differently when calling ZipFile.writestr() with
zinfo.external_attr being 0. Python 3 uses `0o600 << 16` as the value
for such a case (since
18ee29d0b8),
which seems to make more sense. Otherwise the entry will end up with
0o000 as the permission bits. This CL updates common.ZipWriteStr to
follow the logic in Python 3, in order to get consistent behavior
between using the two versions.

Bug: 131631303
Test: `python -m unittest test_common.CommonZipTest`
Test: `python3 -m unittest test_common.CommonZipTest`
Change-Id: If8429855d922ef1ad76591f703215a0ce5089f0f
2019-06-18 23:53:53 -07:00
Treehugger Robot
e346d8c25e Merge changes from topics "rbe-deps-dexpreopt-images", "rbe-deps-system-modules"
* changes:
  Use dexpreopt image deps from Soong
  Depend on all the files from system modules
2019-06-18 22:48:22 +00:00
Tao Bao
aadc9b9159 Merge "releasetools: Update partitions in deterministic order." 2019-06-18 22:20:26 +00:00
Treehugger Robot
0405f780fc Merge "Update fs_config documentation" 2019-06-18 22:08:46 +00:00
Tao Bao
f1113e97ae releasetools: Update partitions in deterministic order.
Previously it was using regular dict.

Test: python -m unittest test_common.DynamicPartitionsDifferenceTest
Change-Id: If108a4512aeaf9d3c8775c030cad6e44342b9d3d
2019-06-18 12:14:32 -07:00
Neil Fuller
c4924c616c Merge "Remove a host simulation of the runtime module" 2019-06-18 15:33:11 +00:00
SzuWei Lin
79e0d25c33 Enforce property split in GSI
Pure GSI build targets has no vendor partition, such as
aosp_$arch_ab and gsi_$arch. The system properties defined by
PRODUCT_PROPERTY_OVERRIDES will be in /system/build.prop.

The patch defined a fake BOARD_VENDORIMAGE_FILE_SYSTEM_TYPE to
let these system properties flow to vendor and won't pollute the
system.img.

The bug also move some properties to /product/build.prop.

Bug: 135508595
Bug: 131162245
Bug: 134781120
Test: check the /system/build.prop do not have "ro.carrier=unknown"
Test: adb remount on GSI Q on P
Change-Id: Ib200d66cf98fea572c26338e058bce29eb5e0cd7
2019-06-18 18:23:18 +08:00
Isaac Chen
fae280264e Disable apex update for GSI
Currently, GSI does not include apex support and TARGET_FLATTEN_APEX is true.
This will cause issues for vendors with apex support (e.g. Cuttlefish).
This change set ro.apex.updatable to false in product to override that sets
in vendor.

Bug: 135411972
Bug: 134673003
Test: $ lunch aosp_x86-userdebug; m -j; emulator
      $ lunch aosp_cf_x86_phone-userdebug
      # Replace system.img in super.img with GSI
      # The resulted CF could boot and browse the web successfully.

Change-Id: I08fd7a1b254aac276926329e064c35b714764936
2019-06-18 15:57:51 +08:00
Dan Willemsen
cb4c357661 Use dexpreopt image deps from Soong
Instead of just depending on the single image file, depend on the entire
image that gets loaded -- the .oat / .vdex files, and the modules that
are part of the image that have their own files.

This is necessary for RBE, where the rule only gets the files that it
depends upon.

Test: treehugger
Test: build a system image with RBE
Change-Id: I93a580b42cf4ef1824ee1397e551c065f083348a
2019-06-18 01:19:04 +00:00
Dan Willemsen
70b934ee64 Depend on all the files from system modules
Test: treehugger
Change-Id: Ie84dda3d05a84658ad9386a3034d861138f25b2c
2019-06-18 01:19:04 +00:00
Colin Cross
01585fc876 Add more OWNERS
Test: none
Change-Id: I0d70871bbdd6e3a15e60d10e95f7630f5da3ccdb
2019-06-17 15:22:33 -07:00
Tom Cherry
de18e892d3 Update fs_config documentation
While searching for some fs_config related symbols, I ran across this
being out of date, so update it.

The new documentation defers to the help section of the underlying
script more to hopefully avoid this in the future.

Test: n/a
Change-Id: I94893cbb29b8d7e81963a12e3f8316ac3f41b7ae
2019-06-17 13:31:28 -07:00
Mitch Phillips
ee2dcac47a Remove experimental pass manager for fuzzer builds.
Sanitizer coverage is currently broken with the experimental pass
manager. See b/133876586 for more information. The patch is currently
being worked on upstream (https://reviews.llvm.org/D62888), but is not
ready yet. Hence, we disable it here (similar to the LTO bug).

Bug: 133876586
Test: Build anything with SANITIZE_TARGET='fuzzer', verify that it has
'sancov' symbols.

Change-Id: I74d02e52ccbe2a3d96b6b8e30230197b554e60a0
2019-06-17 10:37:55 -07:00
Jaewoong Jung
e6f57e05ec Fixes a typo in a comment.
Test: TreeHugger
Change-Id: Ie124090280eb0c1235e50baa4a869399bb440abb
2019-06-16 21:48:42 -07:00
Patrick Tu
58982618e1 Merge "AIDEGen: Collect the classes.jar of the prebuilt modules into module-info.json" 2019-06-17 02:51:56 +00:00
Dan Shi
fde58bccbc Merge "Update auto-gen test config template with EXTRA_CONFIGS placeholder" 2019-06-14 22:21:39 +00:00
Treehugger Robot
87ebb3013f Merge "Allows skipping building the super images for partial builds." 2019-06-14 21:42:33 +00:00
Treehugger Robot
2e1516dc44 Merge "Start the deprecation of BUILD_HOST_STATIC_LIBRARY" 2019-06-14 20:28:46 +00:00
Treehugger Robot
acfbc544e6 Merge "Fail build when sum of sizes of logical partitions exceeds OEM-set limit" 2019-06-14 20:23:08 +00:00
Daniel Norman
0bf940cace Allows skipping building the super images for partial builds.
Previously, setting PRODUCT_BUILD_SUPER_PARTITION to false for a partial
build (with PRODUCT_USE_DYNAMIC_PARTITIONS == true) would fail to
include necessary keys in misc_info.txt that are required when merging
two partial builds to create a dynamic-partition-enabled mixed build.

This change ensures these necessary keys are included even when
PRODUCT_BUILD_SUPER_PARTITION is false. Setting
PRODUCT_BUILD_SUPER_PARTITION to false causes partial builds to skip
building super.img and super_empty.img, instead relying on these images
to come from the final merged build.

Bug: 134764140
Test: Building & booting a dynamic-partition-enabled mixed build, and
inspecting partial builds' logs / out folder to ensure that
super.img/super_empty.img were not created.

Change-Id: I99431a9a342e9b0617510e250597f3024ef39322
2019-06-14 11:42:45 -07:00
Dan Willemsen
22f69e1094 Start the deprecation of BUILD_HOST_STATIC_LIBRARY
There aren't any common users of this left on AOSP, and only a few
internally.

Test: treehugger
Change-Id: I37277dbf10aa253ef64ca748f82765d16ae1cf80
2019-06-14 09:37:42 -07:00
Hridya Valsaraju
219b6294dc Fail build when sum of sizes of logical partitions exceeds OEM-set limit
BOARD_SUPER_PARTITION_ERROR_LIMIT can be set by OEMs to cause a
build failure when the sum of sizes of logical partitions exceeds the same.
It is set to BOARD_SUPER_PARTITION_SIZE by default.

Bug: 133329143
Test: mmm -j32

Change-Id: Ief85c9ab3e3b9d071cd22b70e13b5c64bb784b9c
2019-06-14 09:34:46 -07:00
Treehugger Robot
4934e72933 Merge "emulator: add option to disable avb" 2019-06-14 11:08:31 +00:00
Dan Shi
4c55da09db Update auto-gen test config template with EXTRA_CONFIGS placeholder
Bug: 134509111
Test: add "require_root: true" to init_benchmarks and libpower_test
build the modules, confirm the extra target preparer is added in the
test config.

Change-Id: I2fdae79d45fd1e5866ee94d1f0e59df106be2a87
2019-06-13 20:32:22 -07:00
patricktu
379f7f0ef6 AIDEGen: Collect the classes.jar of the prebuilt modules into module-info.json
Current information is not enough if the prebuilt modules are depend on
AAR libs Therefore, AIDEGen has to collect the classes.jar of the
prebuilt modules from build system.

Build module-info.json without this change:
Build time: 1m55.001s
File size: 14,918,354 Bytes

Build module-info.json with this change:
Build time: 1m56.292s
File size: 15,577,031 Bytes

Bug: 132768299
Test: 1. Checkout the internal master branch
      2. Patch this CL
      3. m -j out/target/product/generic_x86_64/module-info.json
      4. Open the module-info.json
      5. For verifying soong_java_prebuilt.mk, check the classes.jar of
         module prebuilt-google-play-* exists.
         e.g.
         "prebuilt-google-play-...": {
              ...
              "classes_jar": [
	      	"out/target/common/obj/JAVA_LIBRARIES/
		 prebuilt-google-..._intermediates/classes.jar"
	      ]
         }
      5. For verifying java_prebuilt_internal.mk, check the classes.jar
         of module ink exists.
         e.g.
	 "ink": {
	     ...
	     "classes_jar": [
	     	"out/target/common/obj/JAVA_LIBRARIES/ink_intermediates/
		 classes.jar"
	     ]
	 }

Change-Id: I09518c92260db47d2686493fa13951f316159d13
2019-06-14 10:47:18 +08:00
Treehugger Robot
491ae5b2d1 Merge changes I2183ac9f,I2c76701c,Ibbe4e11a,I0706faa2,Ic5b7c4fb, ...
* changes:
  Fix dependency on BOARD_AVB_SYSTEM_KEY_PATH
  Fix some java library dependencies
  Depend on NDK libraries
  Call clean-path in more places
  Add missing dependecy on SOONG_ZIP
  Add dependencies on the toolchain (clang, etc)
2019-06-13 17:13:22 +00:00
Treehugger Robot
b7035aef21 Merge "Make kernel modules into normal installed files" 2019-06-13 10:51:23 +00:00
Dan Willemsen
6b72c73a50 Fix dependency on BOARD_AVB_SYSTEM_KEY_PATH
Bug: 130111713
Test: treehugger
Change-Id: I2183ac9f8c45eec178876768dfcc83ae62445293
2019-06-12 21:36:57 +00:00
Dan Willemsen
f14179b3ff Fix some java library dependencies
Bug: 130111713
Test: treehugger
Test: build system image w/RBE
Change-Id: I2c76701c1ad396c1b13f873297785c736fba544d
2019-06-12 21:36:29 +00:00
Dan Willemsen
b441630067 Depend on NDK libraries
Bug: 130111713
Test: treehugger
Test: build system image with RBE
Change-Id: Ibbe4e11a7681c52d2197ce35ae47d4939ca246b6
2019-06-12 21:35:46 +00:00
Dan Willemsen
bbe4e11a76 Call clean-path in more places
When an Android.mk specifies a local path that is outside the current
directory, it uses '..', since we prepend $(LOCAL_PATH)/ to the given
path.

This path ('a/b/../c') gets inserted into command lines and
dependencies. For dependencies, when it gets to Ninja, Ninja calls
CanonicalizePath and removes those references ('a/c'), but the command
line is preserved.

So we've got a command line that references 'a/b', but no dependency on
it. Usually that's not a big problem, but it's an issue if we're trying
to only expose dependencies to the rule, like with RBE only sending the
dependencies to the remote workers. So use our 'clean-path' macro to
collapse the '..' references in Kati so that they don't show up in the
command line either.

Test: treehugger
Test: build a system image with RBE
Change-Id: I0706faa2ac15e9c12c720f43c7bb8171c5efd97c
2019-06-12 21:35:33 +00:00
Dan Willemsen
0706faa2ac Add missing dependecy on SOONG_ZIP
Bug: 130111713
Test: treehugger
Change-Id: Ic5b7c4fbefa83cf002c5f30c1b4646f35e718ca2
2019-06-12 21:35:24 +00:00
Dan Willemsen
c5b7c4fbef Add dependencies on the toolchain (clang, etc)
Historically, we've relied on the path to the clang compiler changing in
order to trigger rebuilds instead of direct dependencies on the compiler
itself. That's somewhat more reliable, since the actual $(CLANG) file is
often a script that doesn't actually change during every update.

In Soong, we've added the dependency anyway, so do it here as well. This
makes it easier for my RBE experiments to identify when we intend to use
clang and to send the entire compiler.

Test: treehugger
Test: build a system image with RBE, find that every command that needed
the compiler gets it.
Change-Id: I3838b5d77884394a8c0f7a1ba133142102602084
2019-06-12 21:34:52 +00:00