Start a background goroutine at the beginning of soong_build that
captures the CPU usage, heap size, and total system memory every
second. Propagate the values through soong_build_metrics.pb back
to soong_ui, and then into build.trace.gz.
Test: m nothing, examine build.trace.gz
Change-Id: Iad99f8f1f088f4f7f7d5f76566a38c0c4f4d0daa
This reverts commit 182b56b870.
Reason for revert: prime suspect for breaking boot tests b/307495247, b/307411752
Bug:307495247
Change-Id: Iea05703b767d2699ca3cf69377eb44b1d21697ad
This change defaults Java stubs to be generated from API text files
during build. Using the `--build-from-source-stubs` flag, users can
toggle between the feature.
Test: m nothing && verify ninja path exists between android_stubs_current and android_stubs_current.from-text, and does not exist between android_stubs_current.from-source, m nothing --build-from-source-stub && verify the opposite
Bug: 274805756
Change-Id: I28834f92c1b1311e3fe0a71a6ea9e8ec2e278d7e
This involves some minor changes to testing infrastructure.
This is a rollforward of aosp/2628496 with a couple of minor changes:
- In ParseBuildFiles, filter out all build files that are kept due
to ShouldKeepExistingBuildFileForDir
- Add some minor test infrastructure for StubbedBuildDefinitions, with a
couple of proof of concept tests used to demonstrate its usage. This
pattern will become immensely more common as we implement allowlist v2
(as we will need to update all tests which today simulate build
definitions that have missing deps)
Bug: 285631638
Fixes: 286545783
Test: bp2build.sh
Test: m nothing
Change-Id: I7c3a03b02098e39dd8e51d327482b440f294478f
This feature is obsolete.
This makes a large number of codepaths "dead code" (such as
module-specific implementations of ApiBp2build functionality). These
will be deleted in a followup CL.
Bug: 284029211
Test: Presubmits
Change-Id: Ib53b99f1fe8c24380d219caf44e9bb3b96724fa0
Revert submission 2651299-bazel_sandwich
Reason for revert: b/293883239, checking if this breaks the build.
Reverted changes: /q/submissionid:2651299-bazel_sandwich
Change-Id: I5f1d4bb13d21e6599f5c353dcaba2375f5ec234d
The "bazel sandwich" is a mechanism for bazel to depend on make/soong
outputs. The name comes from the fact that bazel is now at the top
and bottom of the build graph. This is intended to allow us to work
on converting the partition builds to bazel while not all of the
dependencies of the partition have been converted.
It works by adding the bazel_sandwich_import_file rule, which emits a
dangling symlink that starts with bazel_sandwich:, and includes
information that the aquery handler in soong reads. The aquery handler
rewrites the symlink so that it points to a file generated by
make/soong, and adds a ninja dependency from the symlink to the file
it's targeting.
This allows us to depend on make-built files from bazel, but notably
it doesn't allow us to depend on analysis-time information from make.
This shouldn't be a problem for the partitions, but limits the use of
the bazel sandwich to similar, less complicated types of builds.
go/roboleaf-bazel-sandwich
Bug: 265127181
Test: m bazel_sandwich
Change-Id: Ic41bae7be0b55f251d04a6a95f846c50ce897adc
This allows us to set product variables as build settings instead
of loading them from a target's provider, which further allows us
to read product config variables in transitions.
Bug: 287539062
Bug: 269577299
Test: Presubmits
Change-Id: I8497703f706162572ceb3486240e1eb02a37f5f6
This reverts commit bcfadce598.
Because now test_compiler.py copies clang-dev instead of creating a
symlink.
Bug: 288327832
Test: run test_compiler.py
Change-Id: Ib676a41c152046d710d73fdaac678d8a98de5af2
From the previous change, build.ninja is generated as a symlink of
build.<product>.ninja for any missing use case of build.ninja. As this
was a temporary solution while introducing new changes, removing this
feature and move all build.ninja usage into build.<product>.ninja.
Bug: 277029044
Test: aosp_cf build succeeded
Change-Id: I6e8d2840491654fa4253b1febfaecf0ee9455a2d
This paves the way for some feature work for bp2build, and also shaves
0.25-0.5s from the runtime of bp2build (as tested on my machine).
Test: Treehugger
Bug: 285631638
Change-Id: I841f3f904731d02468a6a68a9535d45e0ea9d7c1
The parsed BUILD files will be scanned for obvious definitions of BUILD
targets which have Android.bp counterparts. In such cases, bp2build will
automatically omit conversion of these defined modules (to prevent
collisions). With this change, we no longer need one-off denylisting of
modules which have BUILD file definitions.
This has a 0.03s to 0.2s slowdown for bp2build with current state. This
impact is identical on a heavier test branch, as well. I also ran an
experiment that applied BUILD scanning to all source BUILD files
(regardless of allowlisting), and this had a 2 second slowdown.
We may want to look into parallelizing or improving the performance of
this evaluation, but it's probably not worth the effort at this time,
since the current performance hit is small.
Test: New integration test
Test: Removed libprotobuf-python from denylist and tested building the
package
Test: Treehugger
Change-Id: Ibde3bab12cd4a8fed642ad46e5344a56953bec91
In non-incremental build, there is no ninja_log. For this case, use
HINT_FROM_SOONG as an alternative solution.
Bug: 273947040
Test: 1.m after removing out/.ninja_log
2.check if non-incremental CI build uses HINT_FROM_SOOONG
3.check if incremental CI build uses NINJA_LOG
4.check if there is no regression in CUJ
Change-Id: I00cd216df096cb2288eeab233729acefb0d1b73c
This allows USE_BAZEL_VERSION to be set for m builds, which will use
Bazelisk for any Bazel invocations during those builds.
This should be used only for manual debugging, typically to either test
new Bazel features, verify Bazel compatibility with Android, or culprit
find new Bazel breakages.
Test: Manually run builds with USE_BAZEL_VERSION, toggled off and on to
ensure the build was rerun. Tested with a broken commit, a working
commit, and 'last_green' special term
Change-Id: I8b475dca5c8d4bd849ee3724a8c3aca9b631bcb8
Current build.ninja does not contain any product name, while other ninja
files (such as combined ninja) do. This change adds product name to the
build.ninja so it can be separated over multiple lunch targets
Bug: 277029044
Test: build succeeded and checked if out/soong/build.ninja has been
renamed
Change-Id: I16dc71f829fd76f01b98da0d509a8e0ef6f62fa9
This fixes an incrementality bug that resulted in superfluous
build.ninja regeneration after a new glob definition is added.
Fixes: 279674820
Test: Treehugger
Test; New integration test in this CL
Change-Id: Ifefe66a0eb1c125e9ad5373d60437a1cb1e6fdec
Currently, uninstantiated, the field will be set to [""]
instead of an empty slice.
Test: m nothing --ensure-allowlist-integrity
Test: m nothing --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (this
fails, as expected)
Change-Id: Ib4cd26f0cb0d40714b8c3f263ee2d22093ee15ef
We now ensure that the module is disabled and _not_ enabled before
failing the build.
Bug: 280439299
Test: m nothing --ensure-allowlist-integrity --bazel-mode-staging (on
master and aosp)
Change-Id: I59c969a27b064f07913a511cb29bc06d23daa371
There are a number of instances where we are exporting information
from soong to bazel via soong_injection. This could be more bazel-centric
if the information was instead held in bzl files, and both bazel and
soong read it from there.
Add a starlark package that will run
//build/bazel/constants_exported_to_soong.bzl at initialization time,
and then results can be retreived with GetStarlarkValue.
Since changes to the starlark files mean that soong has to rerun,
add them as ninja deps.
Unfortunately, the starlark code has to be run at runtime rather than
pregenerating their results, because tests run from intellij wouldn't
go through any pregeneration steps. This means that starlark is run
multiple times during the build, once per test package and once per
primary builder invocation. (currently 3, could be reduced to 2 if we
made the symlink forest generation into its own standalone tool) The
starlark code we have so far in this cl is very fast, roughly half a
millisecond, so it's not a big deal for now, but something to keep an
eye on as we add more starlark constants.
Bug: 279095899
Test: go test
Change-Id: I1e7ca1df1d8d67333cbfc46e8396e229820e4476
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.
Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh
Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
Instead of listing up module names, use another information from soong
such as module type, or size of inputs.
I focused on recall, not precision or accuracy, because false negative
module can slow down whole build
Bug: 273282046
Test: m --ninja_weight_source=ninja_log
Change-Id: I8fcc26f1312fda36197c787d240bc3d7b5346481
Context
- from-text android.jar files are built using Metalava, and these can be
utilized in `decodeSdkDep` so that any modules that depends on APIs
can be compiled using from-text android.jars
- This change removes dependency on source java files when compiling
stub android.jar files
Implementation
- Modify java_api_library module to create system modules using the
generated android.jar
- Replace modules in decodeSdkDep to link against java_api_library
modules
- Add --build-from-text-stub flag to hide the feature behind a flag
Test: m --build-from-text-stub
Bug: 271154441
Change-Id: I104df595edc65c0006820d5ae5b15f1fb167e190
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.
Bug: 269457150
Change-Id: I1eec5d7b6a268cae4c633d8d89ed485598ebca45
If this option is set, Soong generates ninja weight list including
modules in HugeModulesList in allowlists.go
Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Id92b7f9f9e8152c1c46ae071c5821a479cf47bce
Add a flag that enables logic specific to multitree-build builds.
Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
workspace.
This is necessary to solve bazel package boundary issues where the api
file might exist in a different directory
Test: m api_bp2build && build/bazel/bin/bazel build --config=android
--config=api_bp2build //build/orchestrator/apis:system
Test: multitree_build system/nothing (in multitree)
Change-Id: Id64085d65a1943bdb394ea80c875db96ca373839
These are not necessary for queryview and can lead to inaccurate query results.
Test: m queryview; grep bp2build_all_srcs
Change-Id: Ifaf65b4ec5828718c2becb39b4507cf033ea4546
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).
Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
So that it can be used in the bazel implementation.
Bug: 268006095
Test: m nothing, check that it's in out/soong/soong_injection
Change-Id: I1520dd874076dee7fa083648d0790b060d658e5a
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.
On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.
Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.
Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
This test builds an "empty" filegroup. Running it can be useful to
verify that the api_bp2build workspace has been setup correctly.
Test: TH
Change-Id: I78639a2dcdeb659b2d6fdf6abc85d85c4fb697bf
- This is a wrapper function for all files in soong_injection directory.
This should prevent an error in the workspace generated for
api_bp2build.
- Rename the existing CreateSoongInjectionFiles fn and make it package
private to prevent confusion
(The subsequent CL in this stack should contain a smoke test for
api_bp2build)
Test: b build //:empty --config=api_bp2build --config=android
Test: TH
Change-Id: Iddb0aa1aff2f709826edd587aa99fccddf80f08f
Create a
build/bazel/product_config/generated/products/<product_name>/BUILD
file that contains the platform definitions needed for
a particular product. Currently we just create it for the
current lunch target, but the idea is that eventually when
all product config is in starlark, all the products will
have their platform definitions in the tree at once.
Bug: 249685973
Test: Presubmits
Change-Id: I08c82ff28dcf62f09d3b1d2e3186a6b961e12f6e
