The output directory is deleted, so when I want to
see the files there, it can really be helpful to
see more.
Bug: N/A
Test: N/A
Change-Id: Ie40c412fe1bebbf87db00a4dbce107696ab2805e
Context
- from-text android.jar files are built using Metalava, and these can be
utilized in `decodeSdkDep` so that any modules that depends on APIs
can be compiled using from-text android.jars
- This change removes dependency on source java files when compiling
stub android.jar files
Implementation
- Modify java_api_library module to create system modules using the
generated android.jar
- Replace modules in decodeSdkDep to link against java_api_library
modules
- Add --build-from-text-stub flag to hide the feature behind a flag
Test: m --build-from-text-stub
Bug: 271154441
Change-Id: I104df595edc65c0006820d5ae5b15f1fb167e190
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.
Bug: 269457150
Change-Id: I1eec5d7b6a268cae4c633d8d89ed485598ebca45
If this option is set, Soong generates ninja weight list including
modules in HugeModulesList in allowlists.go
Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Id92b7f9f9e8152c1c46ae071c5821a479cf47bce
Add skip-sdk-check to skip checking the SDK version when extracting an
APK/APEX from an App Set Bundle. This can be used when the platform SDK
version is not defined and the APEXs/APKs use SHA based SDK versions.
This check should not be set to true for non Beta dessert releases
Bug: 274518686
Test: # Add SHA targeting modules to platform
m SOONG_SKIP_APPSET_SDK_CHECK=true #Build Passes
m SOONG_SKIP_APPSET_SDK_CHECK=false #Build Fails
m #No config supplied, build fails
Change-Id: I1919437d3410f09c991e1de39031bd88e1f8246a
Add a flag that enables logic specific to multitree-build builds.
Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
To improve build efficiency, the metrics for critical path and
parallelism ratio is necessary. That information has been included in
soong.log, so added it into metrics as well.
Bug: 271526845
Test: build and check if metrics pb has critical path info
Change-Id: I14e1a78c13d400b792d3b05df18604da48759ade
workspace.
This is necessary to solve bazel package boundary issues where the api
file might exist in a different directory
Test: m api_bp2build && build/bazel/bin/bazel build --config=android
--config=api_bp2build //build/orchestrator/apis:system
Test: multitree_build system/nothing (in multitree)
Change-Id: Id64085d65a1943bdb394ea80c875db96ca373839
These are not necessary for queryview and can lead to inaccurate query results.
Test: m queryview; grep bp2build_all_srcs
Change-Id: Ifaf65b4ec5828718c2becb39b4507cf033ea4546
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).
Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
RBE has signifigant performance overhead during
analysis time, in theory to reduce execution time.
But multiproduct_kati only tests analysis, and does
not run any execution, so RBE is just slowing us
down.
Bug: 229863781
Bug: 265603584
Test: time ./build/soong/build_test.bash --products aosp_arm64 on stage_aosp_master
Change-Id: Icbb2f001fee03a75bad37c14d08a203be20ac6cf
So that it can be used in the bazel implementation.
Bug: 268006095
Test: m nothing, check that it's in out/soong/soong_injection
Change-Id: I1520dd874076dee7fa083648d0790b060d658e5a
Test: `m --bazel-mode-dev` and intentionally break the build.
We can see soong_metrics in the output directory and metrics_uploader
executed.
Bug: 268228148
Change-Id: I79e3b11468fca446673ebf752697a99eaff01b0f
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.
On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.
Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.
Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
This signifigantly improves the startup time of soong-built
python binaries. For example, running
`m apexer && time out/host/linux-x86/bin/apexer` gives
0.734s before this cl, and 0.094s after.
Fixes: 259718110
Test: Presubmits
Change-Id: Ib19e83e2c60c39a849525be117279c318de3afa7
multiproduct_kati assumed each build would take ~30GB of RAM,
but it seems it's closer to 40GB now.
The current build_test builds use 7 jobs, so this would bring
us down to 5, which should still keep us under the 6 hour
build time limit as currently these tests are at ~3 hours.
Bug: 262629589
Test: Presubmits
Change-Id: I51af364cc602a3bf7ba85fd8f92c4dd57585abbd
This test builds an "empty" filegroup. Running it can be useful to
verify that the api_bp2build workspace has been setup correctly.
Test: TH
Change-Id: I78639a2dcdeb659b2d6fdf6abc85d85c4fb697bf
- This is a wrapper function for all files in soong_injection directory.
This should prevent an error in the workspace generated for
api_bp2build.
- Rename the existing CreateSoongInjectionFiles fn and make it package
private to prevent confusion
(The subsequent CL in this stack should contain a smoke test for
api_bp2build)
Test: b build //:empty --config=api_bp2build --config=android
Test: TH
Change-Id: Iddb0aa1aff2f709826edd587aa99fccddf80f08f
Create a
build/bazel/product_config/generated/products/<product_name>/BUILD
file that contains the platform definitions needed for
a particular product. Currently we just create it for the
current lunch target, but the idea is that eventually when
all product config is in starlark, all the products will
have their platform definitions in the tree at once.
Bug: 249685973
Test: Presubmits
Change-Id: I08c82ff28dcf62f09d3b1d2e3186a6b961e12f6e
This involved refactoring much of the main method in soong_ui/main.
Test: b build libcore:all
Test: use the build-time started flag and verify via printf that
it's the same across upload.go and metrics.SetBuildDateTimestamp()
Change-Id: Id7fe256337e8ee6c40542eba662c0eadb38e9674
Test: Run `m --bazle-mode-dev nothing` again after `m --bazel-mode-dev nothing`.
bp2build_metrics.pb no longer exists, the rest metrics files
have the timestamp of the latest creation time.
Bug: 256617990
Change-Id: I869d1b9c3abcb517744e66b21e5c12d6b5b1e97b
* `android.Context` holds `android.Config`, so provide an accessor to it
and avoid passing them separately to a lot of functions.
* factor common code in `readBazelPaths`/`getExistingBazelRelatedFiles`
into `readFileLines`
* refactor check-error-and-quit into `maybeQuit`
* use switch instead of if-elseif-... where appropriate
* rearrange the code in `runApiBp2build`, `runSymlinkForestCreation`
Test: treehugger
Change-Id: I639929c21ec4999cdfd737c07136f32df2d488bc
Define a structure android.CmdArgs and keep all soong_build command line
arguments used to build a configuration in it. Some of them related to BazelMode
were previously kept in Blueprint's bootstrap.Args where they do not belong.
This simplifies the code and lays the gound work for future simplifying changes.
Test: treehugger
Change-Id: I995871f4e6b0e95cc9df65696ad2020df221d8c8
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified soong_build_metrics is available in local metrics tables from builds that include this CL
Bug: 257537925
Change-Id: Ic5419ecb203c9b62457e1c553d592ba593a9bfff
The synthetic soong_injection workspace is shared between api_bp2build
(used in API export) and bp2build (used in API domain analysis which
runs mixed builds by default). The former creates them in ReadOnly mode,
but the latter creates them in ReadWrite mode (to allow users to
edit/experiment). To prevent errors in the end-to-end builds in
multittree, make api_bp2build create these files in ReadWrite mode as
well.
Test: mulittree_build locally
Change-Id: I8766d88951a3b625739e4dc12d468450ad7322ea
Users can use this feature by
1. Setting PRODUCT_INCLUDE_TAGS += <val> in their product mk files
2. Set
```
blueprint_packge_includes {
match_al: ["<val>"],
}
other_module_type {name: foo}
other_module_type {name: bar}
```
bar and foo will be included if and only if <val> is set
Test: Unit tests in blueprint
Test: TH
Change-Id: I32eed4e3b5ac47fb565c62d13d8881fa984c86f4
Background: toolchain/llvm_android/test_compiler.py generates clang-dev, a directory
symlink pointing to out/install/linux-x86/clang-dev, which itself
contains a BUILD file. This BUILD file should be excluded from the
symlink forest because it's not an allowlisted BUILD file.
Problem: However, our current symlink forest logic and finder logic doesn't
handle this situation. It's not in bazel.list:
```
$ less out/.module_paths/bazel.list | grep prebuilts/clang
prebuilts/clang/host/linux-x86/BUILD.bazel
prebuilts/clang/host/linux-x86/cc_toolchain_config.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_constants.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_features.bzl
prebuilts/clang/host/linux-x86/clang-r450784d/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r450784e/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r458507/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909b/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r475365/BUILD.bazel
```
The symlink forest logic uses this bazel.list file and matches it
against the keepExistingBuildFile list to exclude BUILD files from the
symlink forest. Since clang-dev/BUILD.bazel is not in bazel.list, it's
symlinked into the forest.
All of these gymnastics could be avoided if we added custom BUILD file
name support to Bazel, and only symlinked (e.g.) Android.bazel and
ignored all other BUILD files in the tree. It would be very clear which
checked-in BUILD files we want to use in the symlink forest.
I think we should pursue custom BUILD file names instead of adding more
complexities, to say, the finder logic, to support such a use case. It's
also why I decided to add this hardcoded exclusion to keep the
workaround simple.
Bug: 260809113
Test: presubmits
Test: $ DIST_DIR=/tmp/dist prebuilts/python/linux-x86/bin/python3 toolchain/llvm_android/test_compiler.py --build-only --target aosp_raven-userdebug --no-clean-built-target --module dist --module droid --module tidy-soong_subset --with-tidy ./
Change-Id: I415371543585c1c5e8e00e6958105f65ea5978ee
Unlike bp2build, api_bp2build depends on a well-defined subset of checked-in
BUILD/bzl files, and we create symlinks for that subsset in
out/soong/api_bp2build. Since we now cd into the workspace after
aosp/2308141, we need to add the sylimks of the external repositories as
well
Test: bazel build --config=api_bp2build //:empty
Change-Id: Ia7ecc5543226893a3878bc0fd35e025a37c80413
They are still read when using bazel with queryview, so they
need to be in the workspace so that they can be found.
Bug: 234449134
Test: m queryview && ./build/bazel/bin/bazel query --config=queryview //...
Change-Id: I1f2187dffd5a0e6cfb44ebe0c6738fa9bbef509d
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified bp2build_metrics is available in local metrics tables from builds that include this CL
Bug: 257537925
Change-Id: I09617a7850f1471f18192a0418396126fd4c98ef
Prior to this change, the bundletool and extract_apks tools require that
all ABIs that an APEX or APK provides must be compatible with the
TargetConfig. Instead, this change allows an APK to be selected if it
has at least one compatible ABI with the TargetConfig.
Bug: 260115309
Test: go test .
Change-Id: If67ce8128099611257a834862295a2bf5fa427d3
Add a new argument in soong_ui that will be used to gate this behavior.
This approach is expected to solve incrementality issues in multi-tree.
As part of soong's bootstrap process, all source dirs are added to deps
of out/soong/build.ninja (via globs). Since multitree_build writes to the
"source" api_surfaces directory, it changes its mtime and causes a
recompilation of out/soong/build.ninja in the subsequent invocation.
Test: TH (for single-tree)
Test: Inspected ninja files (for multi-tree)
(Run a full build)
touch out/api_surfaces/vendorapi/libc/.../math.h
orchestrator/prebuilts/build-tools/linux-x86/bin/nsjail --config
out/trees/vendor_aosp_cf_arm64_phone/nsjail.cfg --
prebuilts/build-tools/linux-x86/bin/ninja -f out/soong/bootstrap.ninja
-d explain -n out/soong/build.ninja
(ninja: no work to do)
Change-Id: Ib823163ec1153344a2f593daa8d7156c24ff5bc3
Previously, the NewTextContext created a context that always called
NewNameResolver with a export filter that always returned true.
This change fixes that by:
1. Changing NewNameResolver to take a Config parameter instead of a
filter parameter and pushing the code to create the filter from the
Config from newNameResolver() in cmd/soong_build/main.go into the
NewNameResolver function.
2. Extracting a newTestContextForFixture that does not create a
NameResolver or set it on the context. That avoids creating a
NameResolver before the test has prepared the config.
3. Modify the fixture to create and set the NameResolver in the Context
after the config has been prepared by the test.
4. Added test to verify that it all works.
Bug: 234825639
Test: m nothing
Change-Id: Ie4b13f18093dc01a0ab65a9ecfd143585d843b55
Each build mode has to provide an output file, which also has a
corresponding dependency file (<output file>.d). Previously, some of
the build mode functions returned that file, others just returned and
relied on the doChosenActivity() method to return the file.
That was confusing as the same file name was referenced from two
separate places. This change makes all the build mode functions return
their output file explicitly and the doChosenActivity()just returns
what it receives. It also moves the writeDepsFile call to adjacent to
the return.
Bug: 257650737
Test: m nothing
Change-Id: I35f96ab404d9ffb1da747a3ecb8fc4e1feb9c026
Previously, the Context object created in the main() method was
initialized (ctx.Register()) for the main soong build. Build modes
that did not want that build modes that did not want that (symlink,
bp2build, apiBp2build) had to create their own context, or in the
case of runSymlinkForestCreation its own EventHandler. That was
very confusing.
This change avoids that by simply pushing the call to ctx.Register()
into the build modes that require it and that allows the other build
modes to use the main context and its EventHandler.
One point to note is that to ensure the exact same initialization as
before the runApiBp2build has to explicitly call the following:
ctx.SetAllowMissingDependencies(false)
That is because the main context could have that set to true.
Bug: 257650737
Test: m nothing
Change-Id: Iee239fb87edf443fed65156fa14b8a30c89a2328
Previously, running `m nothing` would write the soong_build_metrics.pb
file 3 times:
1. For bp2build - it contains a single `soong_build` event as the rest
is written to the `bp2build_metrics.pb` file.
2. For Bazel symlink forrest - it too contains a single `soong_build`
event as the rest is appended to the `bp2build_metrics.pb` file.
3. For soong_build proper (including mixed builds) - it contains useful
information.
This change avoids that by pushing the code to write the
soong_build_metrics.pb file from the top level where it applies to all
build modes into the build mode specific function that needs it.
Bug: 257590265
Test: m nothing
# Run with some logging to make sure the file is only created once.
Change-Id: Iee239fb87edf443fed65156fa14b8a30c89a2328
Change-Id: I09724f6143352de6b7d6fb29eaf3fbed5a1abd21
Mixed mode is a special form of the soong build. They both share common
initialization and metrics so they should be grouped together to avoid
duplicating that common behavior.
Bug: 257650737
Test: m nothing
Change-Id: I939b6264f94b69825735dce895132f33401238b6
Retrieving an environment var through Config.Getenv() automatically
adds it to a soong.environment.used* file along with its current value.
Changes to its current value will trigger a rebuild of the appropriate
target. The LOG_DIR is a sub directory of DIST_DIR and on build servers
DIST_DIR encodes the build number making it unique every time. So,
using Config.Getenv("LOG_DIR") will mean the CI builds will never
re-use the previous output.
Previously, bp2build and symlink forrest both added LOG_DIR to their
respective soong.environment.used* file. This change avoids that by
passing through the value from the main() function which retrieves it
using availableEnv["LOG_DIR"] which avoids it being added to that
file.
Bug: 257590265
Test: grep LOG_DIR out/soong/soong.environment.used*
# Before this change it includes LOG_DIR, after it does not.
Change-Id: Iac76aca2d5dc04f47b0924ab5afedef5014bc8b6
Previously, writeDepFile was passed a copy of the main EventHandler to
which it then added an event. Adding an event requires appending
information to slices in the EventHandler structure. As the slices are
a copy the addition only affects the copy and are not present in the
EventHandler which is written out as part of the metrics.
This change fixes that by passing a pointer to an EventHandler into
writeDepFile. For consistency it also replaces a few other uses of
EventHandler with *EventHandler instead.
Bug: 257590265
Test: m nothing
# Check generated text metrics to make sure they don't contain ninja_deps.
# Apply this change.
m nothing
# Check generated text metrics to make sure they do contain ninja_deps.
Change-Id: I0fa0f2b08844cb7c7b4e99b37602703deb43ad20
Previously, it stored an android.Context (no pointer) which while it
worked (because the current contents are themselves pointers) it is
inconsistent with how the rest of the code references it and could
cause weird behavior if additional fields were added to the Context.
Test: m nothing
m bp2build
Change-Id: I9c70f925dde85e5b15431cc232b0c2eb6371ec6d
The internals of CodegenMetrics was updated in aosp/2276671. Use the
new API to initialize an empty object to prevent a null pointer
exception in api_bp2uild.
Test: m api_bp2build
Change-Id: Iad9c38b9881da896171f1c6d4e49d4875acfaab8
It was assigning --bazel-mode-staging to cmdlineArgs.BazelMode, which
looks wrong.
Test: presubmits
Change-Id: I24b087977fc6861eef3a1c30e58d9bf42a344c52
Bug: b/256212479
Test: Prior to thi CL bp2build.symlink_forest event was missing in bp2build_metrics.pb after a clean mixed build
Change-Id: I53bfc4114a383c0d1f9c4c7945e7d4c69bc50b0c
These are needed to add staging-mode to some throttled builds. See go/roboleaf-launch-tests for details.
Test: m nothing
Test: build/soong/build_test.bash --bazel-mode-staging
Change-Id: I456ef80fb9c50579a28fba3dbd184d4e4653dde7
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
- the generating rule has `restat=true`, so its output need not have a newer timestamp to prevent it from rerunning on subsequent ninja invocations
- its dependents (so far only bootstrap.ninja's "build build.ninja") already have proper dependencies in the depfile, namely
1. Android.bp.list,
2. entries in Android.bp.list,
3. bazel.list and
4. entries in bazel.list
This change prevents symlink related changes from spuriously retriggering build.ninja
Change-Id: I93f1fea7054dfbfc7c13ece34d2d1f07a81bbe07
Test: manually verifying bazel analysis is triggered only under the right set of CUJs using build/bazel/ci/incremental_build.py
Bug: b/239044236
This is okay because Blueprint already adds the module files to the
.d file of its output, so until now, they were duplicated.
Test: presubmits
Change-Id: I1bb75c5124aadd1703f3d150948dd7540ba772be
Bug: b/255309129
Test: `lunch aosp_x86-userdebug` and `export NINJA_ARGS="-d explain"`
Then do the following:
1) rm -rf out && m bp2build
2) touch bionic/libc && m bp2build
3) ALLOW_MISSING_DEPENDENCIES=true m bp2build
After each step above observe the following two
a)`stat out/soong/soong.environment.used.bp2build | grep Modify`
b)`stat out/soong/bp2build_workspace_marker | grep Modify`
Verified:
step 2 changes only (b) - prior to this CL, it'd change (a) as well
step 3 still changes both (as expected)
Change-Id: I98a94878a14e19043f448b0904fc67d5d1dc9733
Improves some error messages that would have given little/no information
Bug: 254650145
Test: relevant unit tests
Test: Induce errors and check metrics proto
Change-Id: Ife6116af74af6e62c2f8ae8774e53c28178fb8d0
update and create bundle config pb python library
update config.pb file to the source of truth:
google3/third_party/java_src/android_appbundle/bundletool/schemas/proto/config.proto
current config.pb is stale and needs update.
Also, add a python library host soong module config_proto, so that it
can be used by other python code.
BUG: b/240288941
TETS: tested along with primary_train_build_worker that reuse
BundleConfig.pb
Change-Id: I426a5cb7ec42573ed2110494c50813abc0a96296
This is to use bazel to build targets that are being prepared for an
incipient release to the prod mode allowlist.
Bug: 254265047
Test: m nothing
Test: m nothing --bazel-mode-dev
Test: m nothing --bazel-mode-staging
Change-Id: Ic78a59cf51dba83ef1ac26483586560ea9b24aaf
Bug: b/239044089
Test: run build/bazel/ci/incremental_build --bazel-mode com.android.adbd and see the additional metrics in summary.csv
Change-Id: I2c3b9cead2ee43c700a9ad52e669a64aab1499ce
The generated Bazel workspace will only contain api specific targets.
This is feasible since these targets do not have any cross dependencies
with the targets in the bp2build workspace
The advantages of a new mode are
1. Does not pollute bp2build workspace with api targets
2. Does not block api targets with the current allowlist conversion
mechansims in bp2build
(In the future we might want to combine these two workspaces)
A Soong module type will generate a Bazel target if it implements
ApiProvider interface
Test: m apigen
Test: m nothing
Change-Id: I69c57ca6539f932e0ad554ce84a87fb7936fdba0
The logic here has diverged from the logic in bundletool and resulted in
the wrong APEX variant being chosen for a 64bit-only product.
Bug: 246476965
Test: go test .
Change-Id: Ic3b067e98a65146cfa399e7c9b231f397e51c23e
Prior to this `getToc` could return (nil, nil) which caused a crash
later.
Test: m extract_apks && out/host/linux-x86/bin/extract_apks \
-o foo.apks -sdk-version=33 \
-abis=X86_64,X86,ARM64_V8A,ARMEABI_V7A \
-screen-densities=all -extract-single test.apks
where test.apks comes from ag/19673938
Test: vendor/google/modules/ArtGoogle/build-with-prebuilt-art-module.sh droid
with prebuilts in place for cf_x86_64_phone_prebuilt_art-userdebug
chained build on git_master_throttled
Bug: 246562532
Change-Id: Id849371b4dea04f08c07b201c8a466735b5b0994
This environment variable has been out of use for some time. It used to
control Bazel-as-executor, which has been removed from Bazel for several
releases.
Bug: 243077098
Test: Treehugger
Change-Id: I4edfb5f3574c369a16547ea06ba780293c37ecb3
Previous behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Use handcrafted build file even if there were allowlisted bp2build
modules in the same package.
- Package listed in bp2buildKeepExistingBuildFile and a soong module has
a bp2build: { label } attribute:
- Merge the handcrafted and bp2build generated build files
New behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Merge with bp2build generated build file.
Bug: 234167862
Test: ./build/bazel/ci/bp2build.sh
Change-Id: Ifbaf4f8f0f5158b5b2bd6d534eb2311e2e5f399b
This allows "bazel mixed builds prod mode", in additional to reworking
the mechanism in which mixed builds dev mode is enabled.
As a followup, CI scripts will be migrated to use the new flags, as
USE_BAZEL_ANALYSIS=1 is deprecated.
Test: Manually ran --bazel-mode with an allowlist verifying that the
module alone was enabled
Test: Manually verified --bazel-mode and --bazel-mode-dev cause a build
failure
Change-Id: If0d34360e60452f428b05828f4ec7596b7cb619a
This refactoring prepares for introduction of bazel prod mode, an
alternative mechanism for mixed builds allowlist handling.
* Decide bazel-mode as close to soong_build main as possible
* BazelContext itself decides whether a module is allowlisted
* Separate bp2build and mixed build allowlist
Test: m nothing, manually verified all modules are mixed build disabled
(via metrics)
Test: USE_BAZEL_ANALYSIS=1 m nothing, manually verified that mixed build
disabled/enabled modules are identical before and after change.
Change-Id: I0f55d8b85000cb4a871a099edc6d7d868d7df509
deprecated flag that simply adds 5s to a build
the `ctx.Done()` signal handling is unnecessary because of clean-up hooks already in place
Test: m libc
Bug: N/A
Change-Id: Ica12d727264af1e61e97393bc8cbeba9f78cbc83
- Top-level trace for all soong_build runs
- Includes adding Peek() to OncePer because not all soong_build
invocations have GenerateBuildActions run.
- A trace per mutator invocation
Test: m && build/bazel/scripts/print_analysis_metrics.py
Change-Id: Ief5c04630484fb38ec7e3757de45c7dc294d3b3c
This is to generally warn Google devs about lack of LOAS creds.
Given the switch to RBE, lack of LOAS creds will start resulting in build
failures, so adding a warning message ahead of time to all builds. Will
convert this to a build failure after a week.
We need LOAS creds to be able to even fetch the CDPush config file to
determine whether to use RBE in a build a not, which is why I'm making
this a broad error message across builds.
Test: ran "m nothing" build with / without gcert and with / without
"stubby" in $PATH and confirmed appropriate error message.
Bug: b/235985591
Change-Id: I15cbaf372b0a7b79f868a06c7d5ede19b49ae687
On the build servers, LOG_DIR is a subset of DIST_DIR, which encodes the
build number. This means that LOG_DIR is guaranteed to be different for
every build, and we'll never re-use previous soong runs on CI builds.
The current use case (writing out soong metrics) shouldn't care if the
value changes, so pipe the environment variable through, bypassing the
env dep checks. It's not clear if this is the only issue triggering
soong_build re-runs on the build servers, but it's a start.
Test: Change DIST_DIR, soong_build does not re-run.
Change-Id: I59bf117f3fbc0e7f10f0506d3d1ac7564c6a00dc
This was accidentally removed in aosp/2094705. Among other possible
issues, this caused the "used environment variable" file to be omitted
from the depfile of soong_build during mixed builds, which would result
in soong_build not being subsequently retriggered during env var
changes.
Sendng this out as a quickfix, regression test to follow.
Bug: 233774464
Test: Manual `m nothing && USE_BAZEL_ANALYSIS=1 m nothing && m nothing`,
verifying that a rebuild was retriggered in each invocation.
Change-Id: I1d9dbd1bc4696df760ebb9be262bc165c161bb21
This large refactoring has both immense performance implications and
improves mixed builds complexity / usability. Summary:
1. Queueing calls to Bazel is done in a new mutator instead of a full
soong_build pass. Normal soong_build flow is interrupted (via a
functional hook in blueprint) to invoke bazel and parse its response.
2. Implementing mixed build support for additional modules is as simple
as implementing MixedBuildsBuildable. In this interface, define the
request that must be queued to Bazel, and then subsequently define
how to handle the returned bazel cquery metadata.
3. Mixed builds consists of only a single pass. This greatly
improves mixed build performance.
Result:
A 33% runtime improvement on soong analysis phase with mixed builds.
Caveats:
C++ BazelHandler handling still remains a bit of a mess; I did what
I could within this CL's scope, but this may require additional cleanup.
Test: Treehugger
Test: Verified that aosp_arm ninja file is bit-for-bit identical with or
without this change.
Change-Id: I412d9c94d429105f4ebfafc84100d546069e6621
A typo caused one of the two files to be compared to itself if they
both had the same size.
Bug: 228496289
Test: Test_filesHaveSameContents
Change-Id: Ie44f32e3c9b8ef725f0d9933c7e701340036710a
Some of the prebuilt ELF files used for bionic heads cause an
unexpected EOF error, ignore unexpected EOF the same way we do
for EOF.
Test: not yet
Change-Id: I267d11b4d12b83ecebedc72a565e148c5e53af6d
There are cc_binary_prebuilt modules in the tree that are shell
scripts, and attempting to extract an elf ID from them results
in an error. Ignore too short files and files missing the elf
magic header the same way we do for elf files without an elf ID.
Bug: 218888599
Test: Test_elfIdentifierFromReaderAt_BadElfFile
Change-Id: If7117925ca2371a8ee61ba3616372f6e9b0fab0e
The command line run inside the sbox sandbox can be very long and
printing it obscures the error message. It is already written to
a script for execution, so point to the script instead of printing
it.
Fixes: 220592170
Test: manual
Change-Id: Icf883344af0f707802b090b4cfb0ef6dcfe0e5c1
Add support to sbox for only writing output files if they are changed
to support enabling restat. Move the clearing of the output directory
into sbox, and when write if changed is enabled delete obsolete output
files without deleting output files that will be replaced.
Enable restat for local metalava rules. Restat for metalava rules
run remotely is still blocked on b/183630617.
Bug: 222095735
Test: TestRuleBuilder_Build
Test: m SystemUI
Change-Id: If8fc47e711f4a6f43947ab69f17cccc8ba6ae61a
Add a symbols_map tool that can extract an identifiying hash from
and elf file or an r8 dictionary. The tool writes the hash to a
textproto, and also supports a merge mode to combine textprotos into
a output file for inclusion in the build artifacts.
Bug: 218888599
Test: m dist
Test: symbols_map_test.go
Change-Id: Icd3ed6e5510e058c92d97c78759e7a4cfcdbb6ca
Metrics are added to a materialized protobuf soong_build_metrics.pb
Recorded events have scoped names and recorded runtimes.
For example, mixedbuilds includes the following events:
mixedbuilds.prepare
mixedbuilds.analyze.write_files
mixedbuild (this is a total of all mixedbuild events)
Test: Manually verified materialized protos for mixedbuild, bp2build,
and legacy build.
Change-Id: I2e72d4502c0373bee0279bb1320b2e4dd586129c
Dumping of RBE metrics should only happen when reproxy was started.
Reproxy is started only when Soong is run in make / build mode and not
in dumpvars mode.
Doc: https://docs.google.com/document/d/1Iqhu5QgozB_rA9fucMHO6X4U6A8e8qFtZyub9j83ICQ/edit#
Test: Ran a build with / without RBE. No metrics dumped during lunch
without RBE and metrics dumping worked as expected with RBE.
Bug: b/197213143
Change-Id: Ic3403331e1f4ac2c7c72fd9169aa7b82a7545c75
I0e293944c5c804b4512fb834f8f18e1d6ecd053c accidentally used
Println instead of Fprintln, causing it to print the current
contents of the buffer instead of adding to it.
Bug: 216192129
Test: manual
Change-Id: I1ddeeef3443c1091cf729328cb8fd74c217432ec
Discarding PT_INTERP in the linker script generated for the embedded
linker allows linking without the --no-dynamic-linker argument.
Bug: 216192129
Test: manual
Change-Id: I0e293944c5c804b4512fb834f8f18e1d6ecd053c
When the environment variable config file is loaded after config object
is initialized, we end up loading the USE_RBE variable in the config
file after the code to start reproxy process is run. This causes a
problem when USE_RBE variable itself is set in the config files. To
avoid this, I've moved the config file loading code to be within the
config package, this makes the main.go file cleaner too.
Test:
Ran a build with USE_RBE:true set in the config json file. The build
fails without this change and succeeds with this change.
Bug: b/209486170
Change-Id: Iab3957b64f4b5456a861057d16ad318f4f78e0cb
This is useful for external users to be able to specify their own config files that can be loaded by soong during startup. In addition, we need this for upcoming changes to incorporate an experiments framework in Soong since the config file will be fetched from CDPush and put into the OUT_DIR folder by the config file fetcher binary.
Note: Once this is merged into internal branch, I'll fully get rid of
the vendor/google/ path from Soong in aosp.
Test:
1. Ran a build in aosp with these changes and no config file was loaded.
2. Ran a build in internal master with these changes and the current
config file inside vendor/google was loaded as expected.
Bug: b/214035335
Change-Id: I9af83687d4eaeee1ffb0f88a750cfeb7c6d2bafb
Tie verbosity of bp2build to env var BP2BUILD_VERBOSE
Test: m bp2build
Test: BP2BUILD_VERBOSE=true m bp2build
Change-Id: Ib8f10ad73ed06c935126a77d18c1d9febc25348f
Extract and install the primary apk normally, and then unzip the rest
of them as a post install command.
Bug: 204136549
Test: app_set_test.go
Change-Id: I17437ff27f49df6bc91bdbbea6173b46c7d3ec4e
Linux has limit of 32 pages for a single argument. Sbox's method of
handling commands as a single large string can run into this limitation.
Write the large command to a script file and then execute that file.
This better accommodates the large commands, and leaves a script in the
temporary directory useful for manual inspection if necessary.
Bug: 177070955
Bug: 174232579
Fixes: 202297224
Test: m
Test: make particular target with very long command exceeding limit.
Change-Id: Ia298fdfd7a759821c37f540deaf800026041e511
This is necessary because GenerateAndroidBuildActions() is effectively a
mutator and therefore can change the data stored in each module.
Sometimes one wants to dump that data in the JSON module graph, and in
order to do so, actions need to be generated.
I briefly entertained removing StopBeforePrepareBuildActions(), but it
incurs a 10-second penalty on "m soong_docs" and "m bp2build" so I
decided to keep it.
Test: Presubmits.
Change-Id: I438aaf981e989dc39a09e8a2ba73f7a08ee55386
Device builds are no longer supported on Mac, but we do support building
various host tools, including the SDK build-tools and platform-tools
packages. These have dependencies on [java] device modules, so we don't
completely disable device modules, only hide them from Make (which makes
them more difficult to trigger from the command line).
Also fix the mac build of multiproduct_kati, so that `m blueprint_tools`
works on Mac.
Bug: 187222815
Test: `m`, `m dist`, etc on Mac
Change-Id: I92f16605d5cd173d431cbcb79081234d45cc6e2e
Soong writes out text protos for the sbox manifests, and if those change
every time soong changes, we trigger rebuilds of significant portions of
the tree. This is very annoying when attempting to iterate on Soong, so
disable this functionality at the soong_build level.
Bug: 200727888
Change-Id: Icf21e1775a4c4200e983c1b8ac66ddd780d7c90c
Note that this is an initial, rather crude attempt at this,
that unblocks generating BUILD files needed to build AOSP android apps.
In particular, note that
- Jetifier step currently ignored
- Visibility set to public for all output targets
- All dependencies are gathered in 'deps', though they are split under various categories in bp files
- Host, host-only notions are ignored for now
Test: Copy first 2 lines of, e.g. prebuilts/sdk/current/support/Android.bp to a BUILD.bazel file and run pom2bp -pom2build -regen BUILD.bazel
Bug: 198433754
Change-Id: I5efa93517e6c319fd828da5b16ba7efd2d90bca8
If SOONG_UI_ANSI_OUTPUT is set to a true value force the simple status
output to keep ANSI codes. This will allow buildbots to collect logs
with ANSI codes in them. Smart status output is not affected as it
always keeps ANSI codes.
Bug: 147310922
Test: manual
Change-Id: I0b78ceebb65125b8e8dafb4787816fb679d3eb3e
Under load sometimes the 1 millisecond timeouts are not run before
a 1 second sleep finishes, or `echo foo` is not run before a 1
second timeoutt. Increase the long sleeps and timeouts to 10 seconds.
This doesn't affect normal test runtime as the command should timeout
after 1 millisecond.
Bug: 199207577
Test: Test_runWithTimeout
Change-Id: I31166850a473b9cde37906a62d026580e7a7bd3f
Support three options for converting modules with unconverted
dependencies
1. (default) Warn when converting a module if it has unconverted deps.
2. Error when encountering a module with unconverted deps. (not hooked
up yet)
Test: build/bazel/ci/bp2build.sh
Test: build/bazel/ci/mixed_libc.sh
Test: BP2BUILD_ERROR_UNCONVERTED=1 build/bazel/ci/bp2build.sh with
unconverted deps -- get appropriate error
Bug: 181155349
Change-Id: Ifaabf0cd2e43e963366dc137159c705294165c3d
This field was missing for Property objects that have non-empty inner
Property objects
Bug: 172797653
Test: m soong_docs
Change-Id: Iee9c66f8b85d68a6b5bf18fd9787143191c4f002
They were a bit repetitive.
In addition, make the environment checks more correct; they could
probably use an integration test or two to make sure that when the
environment changes, exactly those outputs are rebuilt that need to be,
but for now, this is an improvement already.
Test: Presubmits.
Change-Id: Idd79b81ca6975d57d00e5bf4699d266152505ff8
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.
It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.
The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.
Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.
Test: Presubmits.
Change-Id: If928b34de075969fd42932212ce9187808cbdf86
The breakage was because:
- aosp/1818236 removed the "rm -fr" invocation from the Ninja file that
removed the old queryview tree
- queryview files were created with permissions 440 to discourage manual
editing
These two together made incremental queryview invocations not be able to
overwrite its output files.
(Of course, it also made them incorrect since old files were lft in place)
Test: Presubmits.
Change-Id: Idaa3e1db82c9eb8325c48e37aca5146e69af7333
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .
Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
They both used to return out/soong. Unfortunately, the tests seem to
assume the old behavior, so I had to keep tests inconsistent and I was
scared of dexpreopt, too.
Test: Presubmits.
Change-Id: Ib00c7956fd2cdb28cc11d481c17e5a3ff5ad563a
It was confusing because bootstrapping uses two configurations: the
"global" config and the special-cased bootstrap one.
This change merges them.
Test: Presubmits.
Change-Id: I82b482cbe28a343ab6991374b2a28667e1a06b48
I erroneously assumed that the continuous builds don't set DIST_DIR,
whereas they do.
The variable naming is quite questionable, but for now, the important
thing is that the code works and I wanted to do that with the least
amount of change possible.
Also add some more logging:
- Shell environment
- Command line arguments
- Location of logs.zip
- The logs directory
Test: Presubmits.
Change-Id: I651b1045eeef568aa860fd2457fa7799ad68ba38
These are: TopFile and GeneratingPrimaryBuilder.
Also re-shuffle the list of flags to make a bit more sense and finish
the rename of BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively.
Test: Presubmits.
Change-Id: I103ff5f09f1c0d16f695a7da5dea13b55028e33e
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.
Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.
Side cleanup: write .d files in a slightly cleaner way.
Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
In particular, the writing of build-globs.ninja and the collection of
glob list files for build.ninja.d is now done outside of the Blueprint
invocation.
This is useful because RunBlueprint() is sometimes called multiple times
in a single soong_build invocation and because now glob processing is
not hidden by the machinery of Blueprint.
Test: Presubmits.
Change-Id: I6eb19b357901e452e6d1e7ce055f8a3bab4a74f4
They used to be defined in Blueprint but since it doesn't have a
separate existence anymore, we can consolidate all command line
arguments here.
Note that Blueprint is called directly from soong_ui, but that happens
by a simple Go function call and not by invoking a separate binary, so
command line arguments are not needed there.
Test: Presubmits.
Change-Id: Ifa2c101ddbe424b76fc5a508d3d41c329e7353ca
The following files are forked:
- build-globs.ninja
- The glob list files
- .d files for the glob list files
- The output file (build.ninja or the bp2build marker file)
This makes bp2build and soong_build not overwrite each other's files
they need for proper incrementality.
Test: Presubmits.
Change-Id: I69d192cbd36ecd9677f46f3fa095dfce6f872227
This fixes null bp2build builds.
When running bp2build, out/soong/.bootstrap/build-globs.ninja wasn't
generated, therefore none of the globs in out/soong/.primary/globs have
actions to generate them. This makes bp2build always re-run on a null
build, because it depends on those non-existent globs in its depfile.
Fix the above by generating a build-globs.ninja file with the globs that
bp2build encounters during conversion, and add tests to verify that null
builds and bp2build -> soong workflows correctly regenerates the
build-globs.ninja file.
Test: soong integration tests for null soong->soong, bp2build->bp2build and bp2build->soong builds.
Fixes: 193650250
Change-Id: Icfbad351dfa85541d1ea451f73e65cac87097926
This is done by moving SetupSignals() to its own little package.
There are a number of tiny little utility packages for soong_ui we might
be better of merging, but that's for another change (maybe)
Test: Presubmits.
Change-Id: I07b0ca98bfb8884ef4223d665e632183b9896a0d
This mostly cuts the dependency from multiproduct_kati to the build
package; the remaining logic is SetupSignals which will be dealt with in
a followup change.
This requires some code duplication (e.g. ensureFileExists() and
detectTotalRAM()).
This removes a Finder invocation added in aosp/452598; it looks like
it's not a measurable performance hit. The quoted half a second does not
matter much in build_test{1,2,3,4} which take an amount of time measured
in hours.
It also makes ANDROID_QUIET_BUILD unsupported for the UI of
multiproduct_kati; the function of that variable seems to be to suppress
the output from build actions, which don't exist in multiproduct_kati so
it should not matter. In addition, I checked whether anyone sets this
flag for multiproduct_kati and it doesn't appear to be the case.
Test: Presubmits.
Change-Id: I1c2901e1ea69c3a35429e6c967a7bba845676ef7
This serves to not link parts of soong_ui (and eventually soong_build)
into a separate, weird binary. This is in turn good because they contain
any number of global variables and no one really thought about what
happens when two instances are executing at the same time in the same
address space.
This comes with a slight performance hit: 5 aosp_* projects build 152
seconds instead of 146. I suppose this is a price worth paying for a
clean design?
Test: presubmits.
Change-Id: I5623dcab2290f0fc392dd2ede597b9794a3d2a4e
Print more messages in run_with_timeout to make it easier to
determine which log statements came before the timeout and the
on_timeout command.
Test: run_with_timeout_test.go
Change-Id: I92ad663529fa03d91185ba382654a73f0eba1ab7
