A few more misc improvements that I found while analyzing the
performance of base_rules.mk.
This brings an aosp-master/aosp_crosshatch-userdebug kati run from 33.3s
to 28.6s
Bug: 158488548
Test: build-aosp_crosshatch.ninja is the same before/after
Change-Id: If99c31cc7b5d7133d70eb644c6095f19060b71e5
Merged-In: If99c31cc7b5d7133d70eb644c6095f19060b71e5
(cherry picked from commit 8447b8116a)
It turns out we have a few tests with thousands of test data files,
which this logic does not handle very well. Just switching from `x :=
$(x) ...` to `x += ...` accomplishes the majority of these gains, but I
rewrote the logic to be less function-call+eval based to hopefully make
it easier to understand.
For an aosp-master/aosp_coral-eng build, this brings the time spent in
this logic from 7.6s to 0.3s (total kati runtime of 54s to 46s). Kati's
peak memory use (maxrss) also shrunk from 6.1GB to 3.8GB.
Bug: 158488548
Test: build-aosp_coral.ninja is the same before/after
Change-Id: Ia2ec8b29b5c1dd12fa7ea10b4f80fe802c692725
Merged-In: Ia2ec8b29b5c1dd12fa7ea10b4f80fe802c692725
(cherry picked from commit c5ec8e95bc)
This reverts commit 6e1fa4452b.
Reason for revert: relanding a reverted changed
Bug: 153073816
Test: m
Exempt-From-Owner-Approval: cherry-pick from AOSP
Merged-In: I3735866776cf6eb0320c9fb8c6be8ac0c48baa1c
(cherry picked from commit 403c255b6f)
Change-Id: I3735866776cf6eb0320c9fb8c6be8ac0c48baa1c
Don't copy tests or test data to the test suite directories if the
module is marked LOCAL_UNINSTALLABLE_MODULE := true.
Fixes: 153758132
Test: m checkbuild
Change-Id: Ibdf18807a2ffe9bb352d74880a802ad55b57f154
Merged-In: Ibdf18807a2ffe9bb352d74880a802ad55b57f154
(cherry picked from commit fca369e4e3)
Propagate partition tag data to apexkeys.txt and
apkcerts.txt so that merge_target_files.py can build
merged versions of these files by filtering the
framework files for framework partitions and filtering
the vendor files for vendor partitions.
Bug: 138942268
Change-Id: Ic3226728e97dae84d38ec230ccc86d1b124bea94
Merged-In: Ic3226728e97dae84d38ec230ccc86d1b124bea94
Similar to native:vendor, native:product can use VNDK libs but not
vndk_private.
It is activated when PRODUCT_PRODUCT_VNDK_VERSION is set.
This restores the reverted commit
4e7e76fe5a (aosp/1197274).
The problem of the original CL was assuming no modules have both
LOCAL_PRODUCT_MODULE and LOCAL_USE_VNDK in the old implementations.
But many vendor modules in the targets without setting
PRODUCT_PRODUCT_VNDK_VERSION in old branches had both flags that
caused link failures.
To make it no-op without PRODUCT_PRODUCT_VNDK_VERSION, I defined
LOCAL_USE_VNDK_PRODUCT that is set to true if
PRODUCT_PRODUCT_VNDK_VERSION is defined and LOCAL_PRODUCT_MODULE is
true.
Bug: 146620523
Test: build with PRODUCT_PRODUCT_VNDK_VERSION := current
Change-Id: I344c7dc1c47f08706c101e486ff07c3f10aff8ac
This reverts commit 4e7e76fe5a.
Reason for revert: build breaks (bid: 6147225) and postsubmit test fails.
There are some modules that have both "product_specific: true" and "vendor_available: true", which tags the module as "native:product" unintentionally.
We need to clean up these cases first and revisit this CL.
Bug: 146620523
Bug: 147987741
Change-Id: Ib07543235d72a135b6b732aaa909c147d2df832b
Similar to native:vendor, native:product can use VNDK libs but not
vndk_private.
It is activated when PRODUCT_PRODUCT_VNDK_VERSION is set.
Bug: 146620523
Test: build with PRODUCT_PRODUCT_VNDK_VERSION := current
Change-Id: Icfd94dfc30e77581991799d9e2f408f57da22cea
Adds the backend makefile parser for the LOCAL_FUZZ_DATA vars generated
by the cc_fuzz target.
Bug: 141026328
Test: m example_fuzzer, fuzz target should have adjacent corpus/dict
files.
Change-Id: Ide2c34318f11be132992650ce5cc1fd27895915c
TARGET_TRANSLATE_2ND_ARCH is never set to true any more,
TARGET_NATIVE_BRIDGE_ARCH := arm is always used instead.
Bug: 141242600
Test: m checkbuild
Change-Id: Id3e0614a1d8a088d37929655b74fecd812606db2
This reverts commit 068d101077.
Reason for revert: Caused a regression
Test: Added data to a sh_test and compared before vs after
Fixes: 140761783
Change-Id: Ied6427cb2dc3093fb88632eb8f337c2b76e86b95
Support for rust modules built in Soong.
Bug: 136189233
Test: Test module builds.
Test: crosvm builds.
Change-Id: Idf2e4a4f760cbcee1d35fe6567ef8e12d4855094
This saves another 5-15% of makefile eval time (depending on
target/branch).
I'm not sure if deps-license is still used at all, but we'll only save
the information for it if it's actually going to run now.
Since the default module tag is "optional", we really don't need to save
a list of most modules in another variable. I don't see anything that
really needs to see the list of optional modules.
Test: treehugger
Change-Id: Iaf54e9932fbe8e61f10bd0c9336fc4fdefea27ab
It turns out that this single line was responsible for 28-44% of the
time we spent parsing makefiles. So instead of filtering across
$(ALL_MODULES), check to see if a variable that we always would set is
set.
Test: treehugger
Test: check kati-build eval time before/after
Change-Id: I946c2efd060181a4ae9edd3409c53d9b49a58285
In the case of building a multilib library that defines LOCAL_VINTF_FRAGMENTS,
the previous code would create duplicate rules for the fragment. This change
uses the my_vintf_new_installed variable (which may be empty) to avoid the
duplication.
Bug: 139268165
Test: Verify that build warnings are gone, yet VINTF fragment still installed.
Change-Id: I579ec650019a29a6a105118ad5e5ae9376e8d9bc
my_installed_test_data (LOCAL_TEST_DATA) no present in test
suite zip files for device tests
Bug: 135957483
Test: 1. Remove the patch aosp/1008195
2. m general-tests
3. unzip general-tests.zip
4. data files should exist in target/testcases/toybox-tests
Change-Id: I3f0a4d87e284c809625843d55dc5d35ef77e79bb
When an Android.mk specifies a local path that is outside the current
directory, it uses '..', since we prepend $(LOCAL_PATH)/ to the given
path.
This path ('a/b/../c') gets inserted into command lines and
dependencies. For dependencies, when it gets to Ninja, Ninja calls
CanonicalizePath and removes those references ('a/c'), but the command
line is preserved.
So we've got a command line that references 'a/b', but no dependency on
it. Usually that's not a big problem, but it's an issue if we're trying
to only expose dependencies to the rule, like with RBE only sending the
dependencies to the remote workers. So use our 'clean-path' macro to
collapse the '..' references in Kati so that they don't show up in the
command line either.
Test: treehugger
Test: build a system image with RBE
Change-Id: I0706faa2ac15e9c12c720f43c7bb8171c5efd97c
All users have been removed.
Test: treehugger
Change-Id: I2cfe720bef88cc5485e90746d85a7e0a5d068ef7
Merged-In: I2cfe720bef88cc5485e90746d85a7e0a5d068ef7
So that we actually respect different LOCAL_REQUIRED_MODULES for the
host and device versions instead of unioning them. That got particularly
problematic when LOCAL_SHARED_LIBRARIES is implicitly added to
LOCAL_REQUIRED_MODULES. We also used to walk through device-only modules
when filling out the list of required modules, which triggered even more
extra installations.
This also changes the requirements for PRODUCT_HOST_PACKAGES so that it
no longer accepts target-only phony modules (since we can now
differentiate them). They were all removed in previous patches.
Test: treehugger; diff resulting builds
Test: diff list of product_target_FILES and product_host_FILES
Change-Id: I2ed8950320d31f5693323ad8cef6ec5b6780b7d4
We have developed a vendor module and set LOCAL_VENDOR_MODULE := true<space> (Added a space character at the end)
The android build system then can't installed it in the right partition until we removed the extra space character.
bug: 129725067
Change-Id: I081ffe7f39a9c850007ba304c815436500be694c
This is similar to module-built-files, except that it only returns
files built for the target, not the host.
Bug: 119423884
Test: Build with the no-vendor-variant VNDK change that uses this
function.
Change-Id: I2a3d99003b05999eae01c0b90bb62b5263d65592
After migrating the primary install location of test modules into a
generic testcase folder. All test modules will install to
out/target/product/<product>/testcase/<module_name>/ if they don't
specify their LOCAL_MODULE_PATH.
But the dependent test module should also be copied to testcase
folder even its LOCAL_MODULE_PATH be set.
BUG: 128815093
Test: 1. vi cts/tests/signature/api-check/Android.mk
add LOCAL_MODULE_PATH for cts-hiddenapi_flags-csv
2. m CtsHiddenApiBlacklistDebugClassTestCases
Then, hiddenapi_flags.csv should also be copied to testcase
folder.
Change-Id: Iff872447348e74b2728e0913d04e46ccbaa4e972
This reverts commit 224e103308.
Reason for revert: Build Breakage in git_pi-dev-plus-aosp/docs @5366136
make -j110 docs showcommands dist DIST_DIR=/buildbot/dist_dirs/git_pi-dev-plus-aosp-linux-docs/5366136 checkbuild
FAILED:
Dependencies in out found with no rule to create them:
out/target/product/generic/data/app/CtsVerifierTester/CtsVerifierTester.apk
out/target/product/generic/data/app/TradeFedTestApp/TradeFedTestApp.apk
out/target/product/generic/data/app/TradeFedUiTestApp/TradeFedUiTestApp.apk
15:53:46 stopping
and
make -j50 showcommands dist TARGET_PRODUCT=cf_x86_phone DIST_DIR=/buildbot/dist_dirs/git_master-linux-ndk_translation_all/5366149 ndk_translation_all
FAILED: ninja: 'out/target/product/vsoc_x86/data/nativetest/arm/arm_insn_tests_arm_static/arm_insn_tests_arm_static', needed by 'out/target/product/vsoc_x86/obj/PACKAGING/ndk_translation_tests_intermediates/arm_insn_tests_arm_static_result.xml', missing and no known rule to make it
15:55:38 ninja failed with: exit status 1
make: *** [run_soong_ui] Error 1
Return Code: 2
Change-Id: Idf95ef2e06526a0a31690420c923207db627605f
1. A test can add a runtime dependent test module by just setting
LOCAL_REQUIRED_MODULES or LOCAL_TARGET_REQUIRED_MODULES. Then the dependent test
module will be copied to testcase folder.
2. Do not install to $(TARGET_OUT_DATA) for testcase
BUG: 117224272
Test: 1. (a) vi cts/tests/tests/text/Android.mk
(b) add LOCAL_REQUIRED_MODULES := CtsPrintTestCases
(c) m -j CtsTextTestCases
(d) Then, CtsPrintTestCases should also be built to testcase folder like below.
./target/product/generic_arm64/testcases/CtsPrintTestCases
Change-Id: I24ea3783486c54a05cfa9d3d0375b977afc230f8
Some devices opt out of the warning with `BUILD_BROKEN_ENG_DEBUG_TAGS :=
true` still, but the rest have no more warnings left.
Test: build_test in treehugger
Change-Id: I1534a0db006d72c7921a20ab793586ef1c7bdb21
Merged-In: I1534a0db006d72c7921a20ab793586ef1c7bdb21
Adding an intermediate file to LOCAL_ADDITIONAL_CHECKED_MODULE will
cause the file to be built for m <module name>, mma, or m checkbuild.
Test: m checkbuild
Change-Id: I2b7c3c8f01960ce2f9176c8a2108f759fd8fc704
Uses a new BUILD_BROKEN_ENG_DEBUG_TAGS variable that defaults to true
until we mark all of the broken targets, then we'll switch it to false.
These are fairly trivial to fix, so I don't expect to keep this flag
around for long.
Test: build_test on aosp and internal master; check logs
Change-Id: I6176fec265e78aecff965b7cf1636f831f68a140
Define non_system_module in base_rules.mk when the module is product, vendor,
product_services or proprietary module.
Bug: 74699609
Test: m -j
Change-Id: Iebdd3ec8c643be3d9877b7a96cbdd83465fa3878
Soong is already checking for reasonable paths, and it supports
generated test data as well, which this does not.
Test: OUT_DIR=$PWD/out m nothing
Change-Id: I62dd09b74f0f3347930fd51c213e2f66c7dfeb2d
- add "dependencies" and "srcs" to collect in module-info.mk.
- add "Srcs" in core/base_rules.mk
Bug: 112523202
Test: make out/target/product/generic_x86_64/module-info.json and
generate out/target/product/generic_x86_64/module-info.json
Change-Id: I0669377b2e5e6b4ee225f1930bda208eff092dea
Add soong_cc_prebuilt.mk for Soong modules to use so they can avoid
going through all of prebuilt_internal.mk, dynamic_binary.mk and
binary.mk. Also moves stripping support into Soong.
Relands I8b37dda2b449b6b5d48d5b983f43de452f494a1e with fixes for
builds with GENERATE_BREAKPAD_SYMBOLS=true
Bug: 113936524
Test: m checkbuild
Change-Id: Id41cd76ccb1b5bb6ac43b41ae3e8f1ce46d7ad7e
Add soong_cc_prebuilt.mk for Soong modules to use so they can avoid
going through all of prebuilt_internal.mk, dynamic_binary.mk and
binary.mk. Also moves stripping support into Soong.
Bug: 113936524
Test: m checkbuild
Change-Id: I8b37dda2b449b6b5d48d5b983f43de452f494a1e
This particular invocation of 'sort' is expensive (it may amount to 30%
of the total ckati execution time for the clean build).
Test: verify that the result of running 'm nothing' is unchanged,
verify that the result of running 'm -j deps-license
PROJ_PATH=packages/app/% DEP_PATH=external/%' is unchanged.
Change-Id: If1cfddd4dee24559a26ecceebdf03cf49cc5a367
This CL introduces the product variable
PRODUCT_FORCE_PRODUCT_MODULES_TO_SYSTEM_PARTITION, which can contain a
list of module names to be "unmarked" from being LOCAL_PRODUCT_MODULE.
If a module name is included in this variable and if that module is
declared as LOCAL_PRODUCT_MODULE, the setting will be overriden and the
module will be installed in /system. This is useful to control this
behavior on a device-specific level.
Test: m; and checked resulting output.
Bug: 110072687
Change-Id: Iaab663a7ed757c1dd4f16169b0ac3d71650014d2
