Background: toolchain/llvm_android/test_compiler.py generates clang-dev, a directory
symlink pointing to out/install/linux-x86/clang-dev, which itself
contains a BUILD file. This BUILD file should be excluded from the
symlink forest because it's not an allowlisted BUILD file.
Problem: However, our current symlink forest logic and finder logic doesn't
handle this situation. It's not in bazel.list:
```
$ less out/.module_paths/bazel.list | grep prebuilts/clang
prebuilts/clang/host/linux-x86/BUILD.bazel
prebuilts/clang/host/linux-x86/cc_toolchain_config.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_constants.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_features.bzl
prebuilts/clang/host/linux-x86/clang-r450784d/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r450784e/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r458507/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909b/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r475365/BUILD.bazel
```
The symlink forest logic uses this bazel.list file and matches it
against the keepExistingBuildFile list to exclude BUILD files from the
symlink forest. Since clang-dev/BUILD.bazel is not in bazel.list, it's
symlinked into the forest.
All of these gymnastics could be avoided if we added custom BUILD file
name support to Bazel, and only symlinked (e.g.) Android.bazel and
ignored all other BUILD files in the tree. It would be very clear which
checked-in BUILD files we want to use in the symlink forest.
I think we should pursue custom BUILD file names instead of adding more
complexities, to say, the finder logic, to support such a use case. It's
also why I decided to add this hardcoded exclusion to keep the
workaround simple.
Bug: 260809113
Test: presubmits
Test: $ DIST_DIR=/tmp/dist prebuilts/python/linux-x86/bin/python3 toolchain/llvm_android/test_compiler.py --build-only --target aosp_raven-userdebug --no-clean-built-target --module dist --module droid --module tidy-soong_subset --with-tidy ./
Change-Id: I415371543585c1c5e8e00e6958105f65ea5978ee
Unlike bp2build, api_bp2build depends on a well-defined subset of checked-in
BUILD/bzl files, and we create symlinks for that subsset in
out/soong/api_bp2build. Since we now cd into the workspace after
aosp/2308141, we need to add the sylimks of the external repositories as
well
Test: bazel build --config=api_bp2build //:empty
Change-Id: Ia7ecc5543226893a3878bc0fd35e025a37c80413
They are still read when using bazel with queryview, so they
need to be in the workspace so that they can be found.
Bug: 234449134
Test: m queryview && ./build/bazel/bin/bazel query --config=queryview //...
Change-Id: I1f2187dffd5a0e6cfb44ebe0c6738fa9bbef509d
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified bp2build_metrics is available in local metrics tables from builds that include this CL
Bug: 257537925
Change-Id: I09617a7850f1471f18192a0418396126fd4c98ef
Prior to this change, the bundletool and extract_apks tools require that
all ABIs that an APEX or APK provides must be compatible with the
TargetConfig. Instead, this change allows an APK to be selected if it
has at least one compatible ABI with the TargetConfig.
Bug: 260115309
Test: go test .
Change-Id: If67ce8128099611257a834862295a2bf5fa427d3
Add a new argument in soong_ui that will be used to gate this behavior.
This approach is expected to solve incrementality issues in multi-tree.
As part of soong's bootstrap process, all source dirs are added to deps
of out/soong/build.ninja (via globs). Since multitree_build writes to the
"source" api_surfaces directory, it changes its mtime and causes a
recompilation of out/soong/build.ninja in the subsequent invocation.
Test: TH (for single-tree)
Test: Inspected ninja files (for multi-tree)
(Run a full build)
touch out/api_surfaces/vendorapi/libc/.../math.h
orchestrator/prebuilts/build-tools/linux-x86/bin/nsjail --config
out/trees/vendor_aosp_cf_arm64_phone/nsjail.cfg --
prebuilts/build-tools/linux-x86/bin/ninja -f out/soong/bootstrap.ninja
-d explain -n out/soong/build.ninja
(ninja: no work to do)
Change-Id: Ib823163ec1153344a2f593daa8d7156c24ff5bc3
Previously, the NewTextContext created a context that always called
NewNameResolver with a export filter that always returned true.
This change fixes that by:
1. Changing NewNameResolver to take a Config parameter instead of a
filter parameter and pushing the code to create the filter from the
Config from newNameResolver() in cmd/soong_build/main.go into the
NewNameResolver function.
2. Extracting a newTestContextForFixture that does not create a
NameResolver or set it on the context. That avoids creating a
NameResolver before the test has prepared the config.
3. Modify the fixture to create and set the NameResolver in the Context
after the config has been prepared by the test.
4. Added test to verify that it all works.
Bug: 234825639
Test: m nothing
Change-Id: Ie4b13f18093dc01a0ab65a9ecfd143585d843b55
Each build mode has to provide an output file, which also has a
corresponding dependency file (<output file>.d). Previously, some of
the build mode functions returned that file, others just returned and
relied on the doChosenActivity() method to return the file.
That was confusing as the same file name was referenced from two
separate places. This change makes all the build mode functions return
their output file explicitly and the doChosenActivity()just returns
what it receives. It also moves the writeDepsFile call to adjacent to
the return.
Bug: 257650737
Test: m nothing
Change-Id: I35f96ab404d9ffb1da747a3ecb8fc4e1feb9c026
Previously, the Context object created in the main() method was
initialized (ctx.Register()) for the main soong build. Build modes
that did not want that build modes that did not want that (symlink,
bp2build, apiBp2build) had to create their own context, or in the
case of runSymlinkForestCreation its own EventHandler. That was
very confusing.
This change avoids that by simply pushing the call to ctx.Register()
into the build modes that require it and that allows the other build
modes to use the main context and its EventHandler.
One point to note is that to ensure the exact same initialization as
before the runApiBp2build has to explicitly call the following:
ctx.SetAllowMissingDependencies(false)
That is because the main context could have that set to true.
Bug: 257650737
Test: m nothing
Change-Id: Iee239fb87edf443fed65156fa14b8a30c89a2328
Previously, running `m nothing` would write the soong_build_metrics.pb
file 3 times:
1. For bp2build - it contains a single `soong_build` event as the rest
is written to the `bp2build_metrics.pb` file.
2. For Bazel symlink forrest - it too contains a single `soong_build`
event as the rest is appended to the `bp2build_metrics.pb` file.
3. For soong_build proper (including mixed builds) - it contains useful
information.
This change avoids that by pushing the code to write the
soong_build_metrics.pb file from the top level where it applies to all
build modes into the build mode specific function that needs it.
Bug: 257590265
Test: m nothing
# Run with some logging to make sure the file is only created once.
Change-Id: Iee239fb87edf443fed65156fa14b8a30c89a2328
Change-Id: I09724f6143352de6b7d6fb29eaf3fbed5a1abd21
Mixed mode is a special form of the soong build. They both share common
initialization and metrics so they should be grouped together to avoid
duplicating that common behavior.
Bug: 257650737
Test: m nothing
Change-Id: I939b6264f94b69825735dce895132f33401238b6
Retrieving an environment var through Config.Getenv() automatically
adds it to a soong.environment.used* file along with its current value.
Changes to its current value will trigger a rebuild of the appropriate
target. The LOG_DIR is a sub directory of DIST_DIR and on build servers
DIST_DIR encodes the build number making it unique every time. So,
using Config.Getenv("LOG_DIR") will mean the CI builds will never
re-use the previous output.
Previously, bp2build and symlink forrest both added LOG_DIR to their
respective soong.environment.used* file. This change avoids that by
passing through the value from the main() function which retrieves it
using availableEnv["LOG_DIR"] which avoids it being added to that
file.
Bug: 257590265
Test: grep LOG_DIR out/soong/soong.environment.used*
# Before this change it includes LOG_DIR, after it does not.
Change-Id: Iac76aca2d5dc04f47b0924ab5afedef5014bc8b6
Previously, writeDepFile was passed a copy of the main EventHandler to
which it then added an event. Adding an event requires appending
information to slices in the EventHandler structure. As the slices are
a copy the addition only affects the copy and are not present in the
EventHandler which is written out as part of the metrics.
This change fixes that by passing a pointer to an EventHandler into
writeDepFile. For consistency it also replaces a few other uses of
EventHandler with *EventHandler instead.
Bug: 257590265
Test: m nothing
# Check generated text metrics to make sure they don't contain ninja_deps.
# Apply this change.
m nothing
# Check generated text metrics to make sure they do contain ninja_deps.
Change-Id: I0fa0f2b08844cb7c7b4e99b37602703deb43ad20
Previously, it stored an android.Context (no pointer) which while it
worked (because the current contents are themselves pointers) it is
inconsistent with how the rest of the code references it and could
cause weird behavior if additional fields were added to the Context.
Test: m nothing
m bp2build
Change-Id: I9c70f925dde85e5b15431cc232b0c2eb6371ec6d
The internals of CodegenMetrics was updated in aosp/2276671. Use the
new API to initialize an empty object to prevent a null pointer
exception in api_bp2uild.
Test: m api_bp2build
Change-Id: Iad9c38b9881da896171f1c6d4e49d4875acfaab8
It was assigning --bazel-mode-staging to cmdlineArgs.BazelMode, which
looks wrong.
Test: presubmits
Change-Id: I24b087977fc6861eef3a1c30e58d9bf42a344c52
Bug: b/256212479
Test: Prior to thi CL bp2build.symlink_forest event was missing in bp2build_metrics.pb after a clean mixed build
Change-Id: I53bfc4114a383c0d1f9c4c7945e7d4c69bc50b0c
These are needed to add staging-mode to some throttled builds. See go/roboleaf-launch-tests for details.
Test: m nothing
Test: build/soong/build_test.bash --bazel-mode-staging
Change-Id: I456ef80fb9c50579a28fba3dbd184d4e4653dde7
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
- the generating rule has `restat=true`, so its output need not have a newer timestamp to prevent it from rerunning on subsequent ninja invocations
- its dependents (so far only bootstrap.ninja's "build build.ninja") already have proper dependencies in the depfile, namely
1. Android.bp.list,
2. entries in Android.bp.list,
3. bazel.list and
4. entries in bazel.list
This change prevents symlink related changes from spuriously retriggering build.ninja
Change-Id: I93f1fea7054dfbfc7c13ece34d2d1f07a81bbe07
Test: manually verifying bazel analysis is triggered only under the right set of CUJs using build/bazel/ci/incremental_build.py
Bug: b/239044236
This is okay because Blueprint already adds the module files to the
.d file of its output, so until now, they were duplicated.
Test: presubmits
Change-Id: I1bb75c5124aadd1703f3d150948dd7540ba772be
Bug: b/255309129
Test: `lunch aosp_x86-userdebug` and `export NINJA_ARGS="-d explain"`
Then do the following:
1) rm -rf out && m bp2build
2) touch bionic/libc && m bp2build
3) ALLOW_MISSING_DEPENDENCIES=true m bp2build
After each step above observe the following two
a)`stat out/soong/soong.environment.used.bp2build | grep Modify`
b)`stat out/soong/bp2build_workspace_marker | grep Modify`
Verified:
step 2 changes only (b) - prior to this CL, it'd change (a) as well
step 3 still changes both (as expected)
Change-Id: I98a94878a14e19043f448b0904fc67d5d1dc9733
Improves some error messages that would have given little/no information
Bug: 254650145
Test: relevant unit tests
Test: Induce errors and check metrics proto
Change-Id: Ife6116af74af6e62c2f8ae8774e53c28178fb8d0
update and create bundle config pb python library
update config.pb file to the source of truth:
google3/third_party/java_src/android_appbundle/bundletool/schemas/proto/config.proto
current config.pb is stale and needs update.
Also, add a python library host soong module config_proto, so that it
can be used by other python code.
BUG: b/240288941
TETS: tested along with primary_train_build_worker that reuse
BundleConfig.pb
Change-Id: I426a5cb7ec42573ed2110494c50813abc0a96296
This is to use bazel to build targets that are being prepared for an
incipient release to the prod mode allowlist.
Bug: 254265047
Test: m nothing
Test: m nothing --bazel-mode-dev
Test: m nothing --bazel-mode-staging
Change-Id: Ic78a59cf51dba83ef1ac26483586560ea9b24aaf
Bug: b/239044089
Test: run build/bazel/ci/incremental_build --bazel-mode com.android.adbd and see the additional metrics in summary.csv
Change-Id: I2c3b9cead2ee43c700a9ad52e669a64aab1499ce
The generated Bazel workspace will only contain api specific targets.
This is feasible since these targets do not have any cross dependencies
with the targets in the bp2build workspace
The advantages of a new mode are
1. Does not pollute bp2build workspace with api targets
2. Does not block api targets with the current allowlist conversion
mechansims in bp2build
(In the future we might want to combine these two workspaces)
A Soong module type will generate a Bazel target if it implements
ApiProvider interface
Test: m apigen
Test: m nothing
Change-Id: I69c57ca6539f932e0ad554ce84a87fb7936fdba0
