The parsed BUILD files will be scanned for obvious definitions of BUILD
targets which have Android.bp counterparts. In such cases, bp2build will
automatically omit conversion of these defined modules (to prevent
collisions). With this change, we no longer need one-off denylisting of
modules which have BUILD file definitions.
This has a 0.03s to 0.2s slowdown for bp2build with current state. This
impact is identical on a heavier test branch, as well. I also ran an
experiment that applied BUILD scanning to all source BUILD files
(regardless of allowlisting), and this had a 2 second slowdown.
We may want to look into parallelizing or improving the performance of
this evaluation, but it's probably not worth the effort at this time,
since the current performance hit is small.
Test: New integration test
Test: Removed libprotobuf-python from denylist and tested building the
package
Test: Treehugger
Change-Id: Ibde3bab12cd4a8fed642ad46e5344a56953bec91
Unfortunately, genrules are not always available with `m`, instead we
need to know their output paths in order to build them and diff them.
Rewriting in Python lets us store module:output path maps more easily.
Test: ./genrule_sandbox_test.py gen_fstab.gs201 \
libbt_topshim_bridge_header \
android-support-multidex-instrumentation-version
Change-Id: If74130e5a4381cc0e1fab396ebb90dfd5a595a1c
This is a followup to aosp/2606989.
This flag is not necessary now that we're using one platform name
for all of mixed builds.
Also rename current_product to mixed_builds_product so that it's clear
that that this platform should only be used for mixed builds.
In addition, make the bazelrc files point to the named products again
instead of the mixed build product so that b builds will still have
qualified outputs, but mixed builds won't.
Test: Presubmit and kernel build tools abtd run
Change-Id: I7f764cf42cd1323f4b495d1320931f59a076ac63
This is a followup to aosp/2606989.
This flag is not necessary now that we're using one platform name
for all of mixed builds.
Also rename current_product to mixed_builds_product so that it's clear
that that this platform should only be used for mixed builds.
In addition, make the bazelrc files point to the named products again
instead of the mixed build product so that b builds will still have
qualified outputs, but mixed builds won't.
Test: Presubmit
Change-Id: I5acbd18869589f67efb0ad3ebfda2c4d5a6a3850
The CFI transition being applied creates a situation where Bazel
clears its convenience symlinks. This change points the script
directly to the location of the deapexer artifact instead of the
symlink.
Bug: 261733820
Test: Ran the tests
Change-Id: I23951f0fe75ce1fbc3c0dd969244a797ac1ff983
`rm -rf out` used to fail because bazel would not set write permission on some files. Now it's been fixed and thus there is little reason to prefer `m clean` (also users are accustomed to `rm -rf out`)
Bug: NA
Test: verified `m clean` works;interestingly soong_metrics file is created at the end
Change-Id: I000d16508613045811fc7792e5798f7c150dcc05
This change adds missing dependency of Soong modules as symlink
directory in create_mock_soong so that a correct ninja graph can be
created when invoking androidmk_test.sh.
Test: build/soong/test/run_integration_tests.sh
Bug: 283905797
Change-Id: I7dff0094fdc23b7c438c917dbc00a5143a4dc5ae
This fixes an incrementality bug that resulted in superfluous
build.ninja regeneration after a new glob definition is added.
Fixes: 279674820
Test: Treehugger
Test; New integration test in this CL
Change-Id: Ifefe66a0eb1c125e9ad5373d60437a1cb1e6fdec
There are a number of instances where we are exporting information
from soong to bazel via soong_injection. This could be more bazel-centric
if the information was instead held in bzl files, and both bazel and
soong read it from there.
Add a starlark package that will run
//build/bazel/constants_exported_to_soong.bzl at initialization time,
and then results can be retreived with GetStarlarkValue.
Since changes to the starlark files mean that soong has to rerun,
add them as ninja deps.
Unfortunately, the starlark code has to be run at runtime rather than
pregenerating their results, because tests run from intellij wouldn't
go through any pregeneration steps. This means that starlark is run
multiple times during the build, once per test package and once per
primary builder invocation. (currently 3, could be reduced to 2 if we
made the symlink forest generation into its own standalone tool) The
starlark code we have so far in this cl is very fast, roughly half a
millisecond, so it's not a big deal for now, but something to keep an
eye on as we add more starlark constants.
Bug: 279095899
Test: go test
Change-Id: I1e7ca1df1d8d67333cbfc46e8396e229820e4476
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.
Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh
Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
Currently, the expected output shows as the main text for
unrelated presubmit failures.
Bug: 278748024
Test: build/soong/tests/mixed_mode_test.sh
Change-Id: I553a82597c7c7791a822e330eb9b2663f35567a1
These must be added in the soong_injection code as opposed to just
defined loosely in checked-in bzl files because the product_vars
select statement must be updated to support the new platforms.
Bug: 269577299
Test: b test --config=android //build/bazel/...
Change-Id: I7bba9af214896dd3b5938bae70b7c0cea4f75e41
Mostly exporting variables to Bazel, but also allowlisting a BUILD
file.
Bug: 251217226
Test: Unit tests
Change-Id: Id87015a3cd5d970700c4058ec989bb0c14c36bcb
This module uses librypto from cronet which is different from the
default one, so the test would always fail.
Bug: 274473434
Test: run the script locally.
Change-Id: I9aac4da0990953db312e2163ee8af05e692f6f3f
This reverts commit 1e75eccb18.
Reason for revert: The SBOM generation change has been submitted in aosp/2485540, so its integration test in this change should be ok to submit now.
Change-Id: Idb8c84a871b2a5633bed998eb7483fc91d30e97a
This revert was created by Android Culprit Assistant. The culprit was identified in the following culprit search session (http://go/aca-get/5b65d203-d364-4ade-aabb-1330fe45236a).
Change-Id: I10231bc624a15a2ba477712b3a5950f5fc9113e8
This test ensures that API export works ok if the api file exists in a
different directory (precisely, package) than the *_api_contribution
target.
Test: tests/run_integration_tests.sh
Change-Id: I1ff171b93773b514a9a081f962606f4c28abe42e
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).
Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
With this CL, we avoid deleting the bp2build directory and regenerating
all BUILD files. Instead, we regenerate BUILD files which have changes
on the filesystem, and delete old BUILD files which should no longer
exist.
This improves incremental Bazel server performance by about ~5 seconds.
Previously, Bazel would have cache misses for regenerated packages, and
would thus need to analyze them even if no meaningful changes took
place.
For ease of implementation, we avoid removing stale (empty) directories
from the bp2build workspace, but this should have no effect on the
build. (These will be removed on next clean)
Test: New integration tests
Test: Manual benchmarking in conjunction with persistent bazel server
Change-Id: I3e489ff403be34040122876012329060a2506366
since bp2build runs concurrently, errors can occur in any order
Bug: 265845583
Change-Id: Iea6d26bc73112dc89c963545b3e0fc7e1b7fa349
Test: run test localy multiple times
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.
On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.
Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.
Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
This signifigantly improves the startup time of soong-built
python binaries. For example, running
`m apexer && time out/host/linux-x86/bin/apexer` gives
0.734s before this cl, and 0.094s after.
Fixes: 259718110
Test: Presubmits
Change-Id: Ib19e83e2c60c39a849525be117279c318de3afa7
Brings down time taken to fail from 18s to 13s (on aosp branch)
Test: temporarily coded random failure
Bug: N/A
Change-Id: Ib694eec977293f4dd7054e779d1b82b8cace93f6
This keeps this test compatible with a Bazel release change which
affects which files are present in `bazel cquery --output=files` for
py_binary targets. Since the executable will be present in a predictable
location under bazel-bin, we can hardcode that, instead.
We could have alternatively used --output_groups=python_zip_file for
'hermetic' python, but the bp2build-built deapexer target is missing
some dependencies on protobuf.
Bug: 265176532
Test: Manual runs of apex_comparison_tests.sh with the new Bazel release
Change-Id: Ib7021e7e81d1a7bf5ff4d222e2a7947a06d16791
This test builds an "empty" filegroup. Running it can be useful to
verify that the api_bp2build workspace has been setup correctly.
Test: TH
Change-Id: I78639a2dcdeb659b2d6fdf6abc85d85c4fb697bf
Resend aosp/2350557, with minor change on line 65 of apex_cc_module_arch_variant_tests.sh.
The pattern was:
'.*android_target.*: ([0-9]+)$'
and is changed to:
'.*opt-ST.*: ([0-9]+)$'
since the Bazel output directory name has been changed.
Bug: 258422889
Test: build/soong/tests/run_integration_tests.sh
Change-Id: Ib114bc0084d8ea2cd232d6478cc81a039b138bc4
Create a
build/bazel/product_config/generated/products/<product_name>/BUILD
file that contains the platform definitions needed for
a particular product. Currently we just create it for the
current lunch target, but the idea is that eventually when
all product config is in starlark, all the products will
have their platform definitions in the tree at once.
Bug: 249685973
Test: Presubmits
Change-Id: I08c82ff28dcf62f09d3b1d2e3186a6b961e12f6e
Needed because we import absl-py in our bazel.workspace file.
It only worked before in b, not mixed builds, because b was
run from the real android tree instead of out/soong/workspace.
Bug: 234449134
Bug: 249685973
Test: Presubmits
Change-Id: Ie6985582c55a4a95338fb3a8ff7b13ddb522c391
Revert submission 2308141-colefaust_run_b_from_out_folder
Reason for revert: Breaks CI due to more instances of the bazel shutdown issue
Reverted Changes:
I51c28e232:Allowlist build files in external/python/absl-py
I9e6530d95:Cd into the bazel workspace instead of using --pac...
Change-Id: Ice47ef2baaef9976546f9562dc7dd9b19b383d59
Needed because we import absl-py in our bazel.workspace file.
It only worked before in b, not mixed builds, because b was
run from the real android tree instead of out/soong/workspace.
Bug: 234449134
Bug: 249685973
Test: Presubmits
Change-Id: I51c28e23260bed7ca7d4968fd823f52f0d26a0ab
Make PackageVarContext implement PathGlobContext by implementing
GlobWithDeps. This will allow calls to ExistentPathForSource
inside a VariableFunc to use optimized glob dependencies instead of
falling back to AddNinjaFileDeps, which is resulting in extra
dependencies from soong_build on top level directories, triggering
extra Soong regenerations.
Remove the fallback path in ExistentPathForSource by making it take
a PathGlobContext, which is now a superset of PathContext.
Rewrite TestNinjaDeps to not rely on the unoptimized glob dependencies
in VariableFuncs and instead call ctx.Config().AddNinjaFileDeps
directly.
Bug: 257079828
Test: test_create_global_include_directory
Change-Id: I48cf189157d78b9252d339dbc9baeb27e4694807
This will make it easy to change a "false" boolean to "true" to push
`--bazel-mode` by default for all users.
Users may disable bazel-by-default with BUILD_BROKEN_DISABLE_BAZEL, and
bazel is disabled by default on Darwin hosts (due to lack of test
coverage).
Bug: 254628592
Bug: 254084584
Test: Presubmits
Test: Presubmits for aosp/2263623
Change-Id: Iec712119e06ca8ec93028207d88277f879184cc2
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
test_bp2build_generates_all_buildfiles uses the same directory for
--output_base and --sandbox_tmpfs_path. This is required on the build
servers, but causes issues locally. This commit adds a more helpful
error message when this test fails.
Test: build/soong/tests/bp2build_bazel_test.sh
Change-Id: Ib29a7edf456f8ae39bd429ebb367c8115e02c284
* We are interested mostly in the conversion the `default_applicable_licenses`
attribute, as `default_visibility` cannot be handled until every module's
`visibility` is handled.
* Several referenced license modules had to be manually enabled for
conversion, and likewise a few trivial Android.bp containing only
package and license modules.
* As Bazel allows only a single `package` rule, the package rules in a
manually converted BUILD.bazel files were removed (in
external/protobuf and prebuilts/clang/host/linux-x86 trees).
* The converted package rule is emitted before the `load` statements per
Bazel documentation.
Bug: 190817312
Test: treehugger
Change-Id: If8bf6fee1580177de3bb402923615bcd48923ed2
