Given that we can map them directly to disabling the
corresponding Bazel features: `-{link_,use_lib}crt`
Test: Existing (adapted) Unit Tests
Test: bp2build.sh
Change-Id: Ib502f6fb929ace8e86a1001e3cc21f399317500c
Only public, system, test, module_lib and system_server are
converted in order to generate api_fingerprint.txt in Bazel.
Test: java_sdk_library_conversion_test.go and TH
Bug: 266973526
Change-Id: I67a00806165e5afad3876b6cd5cdbc6b0dd65d8b
This changes bp2build codegen, symlink forest generation, and
soong_build so that they do not rewrite output files if the contents are
unchanged.
Bug: 266983462
Test: m droid
Test: canonical_perf.sh benchmarking
Test: Manually verified that rerunning analysis did not regenerate
out/soong/workspace/prebuilts/sdk/BUILD.bazel unless contents changed
Change-Id: I5ec227df7a32b53c7fa0d741fb1403a51931024b
So that it can be used in the bazel implementation.
Bug: 268006095
Test: m nothing, check that it's in out/soong/soong_injection
Change-Id: I1520dd874076dee7fa083648d0790b060d658e5a
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.
On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.
Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.
Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
Previously, Bazel was invoked during mixed builds with a stripped down
environment. This CL adds an allowlist of environment variables that are
passed to Bazel.
Test: WITH_TIDY=1 DISABLE_ARTIFACT_PATH_REQUIREMENTS=true mss tidy-packages-modules-NeuralNetworks --bazel-mode-dev
Change-Id: I23147bec59f6522953cf623e7bcaa0f1f99a75a3
Test: bp2build tests and locally built MusicKotlin
(cannot allowlist yet due to soong/bazel custom_package differences)
Change-Id: I1cbd0fe2af0e6eeed65cf8d57dc85c1f54646c50
This fixes two main sources of nondeterminism:
1. Fix a bug in the ConfigurationAxis comparator (which caused
ConfigurationAxis sorting to be nondeterministic)
2. Process C++ dependencies using the sorted ConfigurationAxis order. In
theory, the order in which dependencies are processed shouldn't
matter (as they should end up in different select stanzas). However,
in the case of InApex stubs, this is not the case; we now ensure
that lists are concatenated in a predictable order.
Added bonus: Some cleanup with SortConfigurationAxes which
makes use of go generics (this made it easier to debug this issue).
Will follow-up with regression tests.
Test: Manually verified that build.ninja checksum and BUILD.bazel checksums do not change after running `m nothing` 6 times in AOSP (with comment-only Android.bp changes in between each run)
Change-Id: I81168e45bdbbcd61ea95ff665cf6c4bc180aa4e0
This API surface is provided by platform and mainline modules to other
mainline modules. Rename it to module-lib API surface to align it with
the terminology in go/android-api-types
Test: go test ./bp2build
Test: go test ./cc
Change-Id: Ieb9f3214e66366fc2ceb6f47e2d6623acb537827
We were looping over a map, and thus processing files
in a non-deterministic order. Some tests check for errors
produced during the symlink forest creation and they would
flake due to this nondeterminism.
Test: m nothing
Change-Id: Ie1a4cf98e341e4a951f110b7d5611bb69c4ddd2b
Brings down time taken to fail from 18s to 13s (on aosp branch)
Test: temporarily coded random failure
Bug: N/A
Change-Id: Ib694eec977293f4dd7054e779d1b82b8cace93f6
Previously if there was a value set outside of product variables and one
inside a conditions default soong config var, we would ignore the
conditions default. Now, we maintain the conditions default and use
the root value in cases where there is not a value for a product
variable.
Test: go test soong tests
Change-Id: Ic991e3aebe5bb6039353f4e3d25625e7c5190f96
- This is a wrapper function for all files in soong_injection directory.
This should prevent an error in the workspace generated for
api_bp2build.
- Rename the existing CreateSoongInjectionFiles fn and make it package
private to prevent confusion
(The subsequent CL in this stack should contain a smoke test for
api_bp2build)
Test: b build //:empty --config=api_bp2build --config=android
Test: TH
Change-Id: Iddb0aa1aff2f709826edd587aa99fccddf80f08f
adds support for *-> android_library and android_library->* edges
Change-Id: I41d4e1d1b8106a17e67951d47e67b59ef3170d17
Test: manually inspected build files for libWallpaperPicker & android-suppor-v4
Create a
build/bazel/product_config/generated/products/<product_name>/BUILD
file that contains the platform definitions needed for
a particular product. Currently we just create it for the
current lunch target, but the idea is that eventually when
all product config is in starlark, all the products will
have their platform definitions in the tree at once.
Bug: 249685973
Test: Presubmits
Change-Id: I08c82ff28dcf62f09d3b1d2e3186a6b961e12f6e
This is a follow up of aosp/2336916. All the properties that have
variant_prepend tag are supported in bp2build.
Bug: 261644490
Test: TH and modified tests with updated expected behaviors
Change-Id: I13bb5d4d8fcd38bf153fa9083c9c945aec2dfa19
When a cc library has stubs, its static variant shouldn't be included in an apex to ensure the library with stubs isn't statically linked. See https://cs.android.com/android/platform/superproject/+/master:build/soong/apex/apex.go;l=2804;drc=89b01aeaa9e19377ff547baab791277719b8aaf3
This CL modifies bp2build so that it excludes apex_available tag from the static target if the library has stubs. This way, we can ensure that no apex can depend on the static target which results to static linking.
Test: go test
Bug: 255589949
Change-Id: Iedf248994b808436f2440570b094fd06d6284ae9
and Sanitize.Misc_undefined.
This is a redo of aosp/2277186 with changes to account for issues
in mixed builds. Specifically, for now we're disabling mixed builds
for modules that use ubsan. This bug tracks enabling it:
b/261058727.
Bug: 253428057
Test: Unit tests
Change-Id: Ia1277a7fa9a82f40998d7f3d2c40ea90f38201e7
- The converter runs for api_bp2build, not bp2build workspace
- Since droidstubs is an internal module created by java_sdk_library,
the conversion encompasses the latter as well
- Since droidstubs do not have an api_surface attribute, this conversion
uses naming convention to infer the api_surface represented by the api
file e.g. *stubs.source -> publicapi, *stubs.source.system -> systemapi)
- Also adds an SdkIntraCore enum to represent the API surface provided
by one core module to another
There is also ongoing work to check in java_api_contribution modules in
Soong. Once we have that, we can update this converter to operate on
that module type instead
Test: go test ./bp2build
Change-Id: Ia85828e04c738d9ffcc524856d7c3034ee29bbf9
The comments in aosp/2336916 are also addressed here
Test: TH and TestCcLibraryVariantPrependPropOrder
Change-Id: If1472658fcd4b5544dec2e2691049a180520c84e
osLinux is implicitly glibc, use //build/bazel/platforms/os:linux_glibc
for the configurability select key for it.
Bug: 259266326
Test: build/bazel/ci/mixed_libc.sh
Test: USE_HOST_MUSL=true build/bazel/ci/mixed_libc.sh
Change-Id: Ifbd121ab3d943f3fdfbd3f6c3e982d1eaaaa9ec1
Because when stubs_symbol_file is defined on cc_library_shared, has_stubs is already implied to be true.
Bug: 260771171
Test: go test
Change-Id: I3b9a2cfb9000411d1b5f22da87dc339267775799
The BUILD file will now have the needed attributes to run the tradefed
test rules.
Test: b test //packages/modules/adb:adbd_test --platforms=//build/bazel/platforms:android_target
Change-Id: I5cb84ccdde27a7fadbef479269df594344a29187
So that they the allowlists can be read from build_mainline_modules_bundles.sh.
Bug: 262302253
Test: m out/soong/bazel_prod_mixed_builds_enabled_list.txt out/soong/bazel_staging_mixed_builds_enabled_list.txt && cat out/soong/bazel_prod_mixed_builds_enabled_list.txt && echo && cat out/soong/bazel_staging_mixed_builds_enabled_list.txt
Change-Id: Id5f89ebb651a52dc5368681a20d6183703e05bd4
* changes:
Change out path to android_target-opt in bazel integration tests
Add check for handcrafted BUILD file in bp2build
Add bp2build conversion to reference to fdo_profile targets
ag/20469925 added a handcrafted BUILD file (allowlisted in ag/20597986)
to internal master. ag/c/platform/build/soong/+/20583593/12/cc/bp2build.go#818 generates references to the targets in the BUILD file. However in tm-dev, the BUILD file doesn't exist while the .afdo files do.
One solution is to cherry-pick the BUILD file to tm-dev. However, tm-dev doesn't have vendor/google/build/soong/bp2build_allowlist.go to check in the BUILD file in bp2build.
This CL adds a check that the BUILD file exists to avoid the failure as in https://android-build.googleplex.com/builds/pending/P45724500/aosp_cf_x86_64_phone-userdebug/latest/view/logs/build_error.log in tm-qpr-dev-plus-aosp branch.
Bug: 253540178
Test: go test
Change-Id: I47fb853015ca230afe3cefe1d37728bf714624be
Merged-In: I47fb853015ca230afe3cefe1d37728bf714624be
The fdo_profile targets are handcrafted in ag/20469925. Bp2build creates fdo_profile attribute on cc_library_shared and references to the fdo_profile target if the profile exists. This works under assumption that all afdo profiles have an associated Bazel fdo_profile target declared in the same folder/package.
As noted in b/253540178, this won't work when we check in the BUILD files (which is likely 6+ months away) because some profiles might only exist internally but not in AOSP. We will implement a long-term solution once we figure out all the requirements we need to support afdo both in Soong and Bazel.
Bug: 253540178
Test: go tests
Change-Id: Iebd5dd7a76583b4b2bf0d9ee56d58f247d313a54
Merged-In: Iebd5dd7a76583b4b2bf0d9ee56d58f247d313a54
Some properties in soong are marked "variant_prepend", which means
in bp2build output, variant properties(select ...) should come before
general properties.
Prepend property is added in StringListAttribute
stuct so that bp2build can handlle variant_prepend properly, if
Prepend is properly set in different module logics.
This change also coordinates cc library with Prepend.
Coordination with other module types will be in separate CLs.
Bug: 261644490
Test: TestCcLibraryHeadersSimple,
TestCcLibraryHeadersArchAndTargetExportSystemIncludes and TH
Change-Id: I3b61c491c54bbe91ba9aa2af9b4c84193c91ae7f
The libc_musl_crt* cc_object modules use header_libs to add
headers to the search path. Propagate static_libs, shared_libs
and header_libs to includes_deps.
Bug: 259266326
Test: TestCcObjectHeaderLib
Change-Id: I8db4d6886761426d3ece38c43ac868d3248f7a9f
Add bp2build support for cc_prebuilt_object to allow buiiding
musl crt objects that use prebuilt clang_rt.crtbegin and
clang_rt.crtend.
Bug: 259266326
Test: //extenral/musl:libc_musl builds
Change-Id: Ic2b0375bc6e7336d31c9e2cf51bdc2ea894185aa
and Sanitize.Misc_undefined.
This is a redo of aosp/2277186 with changes to account for issues
in mixed builds. Specifically, for now we're disabling mixed builds
for modules that use ubsan. This bug tracks enabling it:
b/261058727.
Bug: 253428057
Test: Unit tests
Change-Id: I9dbec1d99353f8cbe6751d5241283ab19592dc27
Previously, the fixture test infrastructure was hard coded to
initialize itself in preparation for invoking the PrepareBuildActions()
method. That meant it could not be used for testing the bp2build and
apiBp2build modes which required different initialization and called a
different method.
This change extracts that behavior into a FixtureTestRunner and adds an
implementation that allows it to test the above two modes. It then uses
that to implement the runBp2BuildTestCaseWithSetup method.
The TestPrebuiltLibraryAdditionalAttrs was the only test which broke as
it supplied an invalid path to the MockFS (it does not support using a
trailing / to represent an empty build directory). So, it was modified
to add a couple of files inside those directories instead.
Test: m nothing
Change-Id: I6798a4f761160af8d1bfed81d46de9628bda3eb9
The tags will then be read by an aspect applied from apex attributes to
validate that the deps are explicitly opt-in to the appropriate apex.
The semantics of this is the same as the Soong apex_available property.
This also adds the "base_apex_name" attr to override_apex conversion so
that apex_available checks continue to work on the non-Google and non-Go
apex names.
Test: presubmits
Fixes: 218841706
Change-Id: I19b3637da9aa47573b252d74f62b601fbdc3784d
Revert submission 2277186-ubsan_bp2build
Reason for revert: b/260630869 broke mixed_droid
Reverted Changes:
Idd9b33e53:bp2build conversion for Sanitize.Integer_overflow
I0d636e439:Add Sanitizer libs up the dependency tree
If113de7bc:Introduce toolchain features for sanitizers
Change-Id: Idf7e7b62f6fb6d4ff92341cc4c9c4b7426b70951
The axis for OS and in APEX was previously ignoring the excludes from
OS, which would result in excludes for non-Android OSes being
incorrectly ignored.
Test: go test Soong tests
Test: b build //hardware/libhardware:all
Bug: 260131489
Change-Id: Ie77f763bea0f473ac69a0c6b6bc3216e7359ad12
Bazel needs to recreate the neverallow behavior for absolute includes,
so export the list of include dirs to soong_injection.
Bug: 250931909
Test: aosp/2302925
Change-Id: I3bd345ea4cfb3181f07e4570466f2fc8c5fe32da
java_library modules with .kt srcs or common_srcs will be converted into
module type kt_jvm_library
Test: m bp2build, manually inspected build files for allowlisted modules
Bug: 258688914
Change-Id: I8293a11c8247b4b76358d0991f82c6b61b58adc3
Previously, it stored an android.Context (no pointer) which while it
worked (because the current contents are themselves pointers) it is
inconsistent with how the rest of the code references it and could
cause weird behavior if additional fields were added to the Context.
Test: m nothing
m bp2build
Change-Id: I9c70f925dde85e5b15431cc232b0c2eb6371ec6d
This is a second attempt at aosp/2273288 which got rolled back because
it turned out that:
1. We make ~120K symlinks in AOSP (!), all of which need to be deleted
2. System calls are sometimes slow
3. Golang spawns a new OS-level thread for each blocking system calls to
keep cores busy
All this together means that we sometimes had 10K system calls in
flight, which meant 10K OS-level threads, which is when Go gives up and
says "I created too many threads, please help".
The fix is to move the system calls into a pool of goroutines, which
soon end up on a pool of threads (since they mostly do blocking system
calls)
Test: Presubmits.
Change-Id: Ia9aefff3b0ed373f09bb6c8b2ec1d8b0f00b213b
Bug: b/256212479
Test: Prior to thi CL bp2build.symlink_forest event was missing in bp2build_metrics.pb after a clean mixed build
Change-Id: I53bfc4114a383c0d1f9c4c7945e7d4c69bc50b0c
Since Bazel's java_import requires a jars attribute to be specified,
the generated neverlink-duplicated module is of type java_library
Change-Id: I14a866dfc583507a9462add50d95060cbfe540c5
Bug: 244210934
Test: m bp2build, go test ./bp2build, manual inspection of generated Build and jar files
This makes symlink forest creation ca. 2x faster again, taking 2-3
seconds instead of 5.
Who would have thought that os.RemoveAll() is slow.
Test: Presubmits.
Change-Id: I91e41319c972dbf1113cf723e383c785433c18b9
This makes it take ~5 seconds on AOSP instead of ~10. Frankly, the
speedup is somewhat disappointing but at least the code is not
complicated.
Test: Presubmits.
Change-Id: Icf94d7ca8bd80c458d014f4cf4cc1be7138deaa6
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
bp2build (i.e. bootstrap.ninja#bp2build_workspace_marker) generate $OUT/soong/bp2build/**/BUILD.bazel files
Having them as a dependency would thus make bp2build_workspace_marker stale upon incremental builds (because we don't re-touch the marker file if it already exists)
See Also: https://android-review.googlesource.com/c/platform/build/soong/+/2263423
Bug: b/253450880
Test: NINJA_ARGS='-d explain' m --bazel-mode adbd
repeat and see if bp2build is rerun
prior to this CL, one would see the input /usr/local/google/home/usta/aosp/out/soong/bp2build/build/make/tools/BUILD.bazel trigger bp2build over and over again
Change-Id: I904cd333a5d6ef506fc4019eda7623ef96a1daa3
Soong supports string properties, but they are overloaded, and can mean
one of three things:
* path reference
* module reference
* string literal
Bazel has different types: label and string attributes. Thus there needs
to be a way to categorize them correctly in bp2build.
This CL introduces a new function to be used on properties like
apex_key.private_key / apex_key.public_key, as well as
android_app.certificate / apex.certificate.
It is important to disambiguate the prop betenn a string literal
attribute or file/rule target label attribute, so this functions does
just that. The new attributes are then further handled by their
respective macros (apex_key, android_binary, apex).
Bug: 253557437
Fixes: 253557437
Test: presubmits, new tests
Change-Id: Id8111cdd60d3aabcae7d17fe9da84d0ee3966023
Also disallow arch variant of genrule.out.
This is to be consistent with bazel where we are migrating to.
Bug: 254114674
Test: Manual
Change-Id: I685a2e64102b7bb68128b39931f0bc85878bc6de
In Soong, decodeMultilib, used to get multilib to determine the dep variations, return "first" if defaultMultilib is set to "common". apex sets defaultMultilib to "common" which means equivalent compileMultilib in bp2build for apex should be "first" (See new Soong unit tests for more context).
This CL fixes bp2build for apex to be more correct.
Bug: 251559512
Test: go tests
Change-Id: Id1cb4407980fc1fab91822c81326f37fb4adfa0a
1) Bp2build convert python_binary_host main attribute as LabelAttribute. Currently "main" attribute in python_binary_host is handled as string but for some modules (e.g certify_bootimg) the "main" attribute points to a file in its subpackage like "subpackage/file.py" and should be converted to "//.../subpackage:file.py".
2) Filter out duplicated labels in the merged list of "required" attributes of python_binary_host and its defaults.
Test: b build //system/tools/mkbootimg:certify_bootimg
Test: b build //build/make/tools/releasetools:check_target_files_signatures
Bug: 253081249
Bug: 253101186
Change-Id: Ic2cb4cadec2c1348da70af9f0730da9914d3a8ca
The module types in scope of this conversion are
1. cc_library and cc_library_shared (non-null llndk or stubs prop)
2. cc_library_headers (all)
For (2), we need some postprocessing on the results of the parser
bp2BuildParseBaseProps. This is necessary because arch and os specific
API exports need to be flattened and added to the generateed API headers
target along NoConfigAxis
e.g.
```
The api equivalent of
cc_library_headers (
name = "lifoo",
deps = select({
"//build/bazel/platforms/arch:arm": ["arm_deps"],
"//build/bazel/platforms/arch:arm64": ["arm64_deps"],
}),
)
should be
cc_api_library_headers (
name = "lifoo",
deps = ["arm_deps", "arm64_deps"],
)
```
For (1), we also need to generate 1:many header api targets so that
arch-specific deps can propagate arch metadata to the top-level
api_domain rule
Test: go test ./bp2build
Test: go test ./cc
Change-Id: Ie40cba1ac8e89f290b3d926c190d5e93abd52859
The merged build files should be added as ninja dependencies,
so we rerun when they're changed.
Fixes: 246552590
Test: m bp2build, m bp2build again and observe it didn't rerun. Then add a comment in external/protobuf/BUILD.bazel, run m bp2build again, and observe it reruns
Change-Id: I26ed035cc0a894500a192f9aa3371fb46519689b
(and remove unused keyName field.)
The private_key and public_key props of an apex_key can point to either
a module or a string. If it's a module, then respect it. If it's a
string, there's additional product variable lookup to find the apex_key
files in product_vars's DefaultAppCertificate parent dir.
This is similar to android_app_certificate.
Test: presubmits
Change-Id: Ib258da14cb0c2df8b5f817fcbc46afebcf225db8
The module types in scope are
1. ndk_library
2. ndk_headers, versioned_ndk_headers
3. api_domain
These modules will no longer generate Bazel targets in bp2build
workspace. A new soong mode `api_bp2build` will be used to generate API
specific targets in a separate Bazel workspace at out/soong/api_bp2build
Test: go test ./bp2build
Change-Id: I01d06b6d8b8364c0d56d6d3f07f8f8bf21db31d7
The generated Bazel workspace will only contain api specific targets.
This is feasible since these targets do not have any cross dependencies
with the targets in the bp2build workspace
The advantages of a new mode are
1. Does not pollute bp2build workspace with api targets
2. Does not block api targets with the current allowlist conversion
mechansims in bp2build
(In the future we might want to combine these two workspaces)
A Soong module type will generate a Bazel target if it implements
ApiProvider interface
Test: m apigen
Test: m nothing
Change-Id: I69c57ca6539f932e0ad554ce84a87fb7936fdba0
But do the splitting before adding $(location...) flags.
Fixes: 247830974
Test: Unit tests
Test: m nothing
Change-Id: I4a7e8e20f720a1074a5ffa17da4fe9b96ca6ed58
Add barebones riscv64-linux-android support. This should be enough
to add riscv64-specific entries to Android.bp files, but can't
actually compile anything until there are riscv64 toolchains.
Test: arch_test.go
Change-Id: I0dcc7e797d9352dd38243be908a7f19004ff3db1
The certificate module is handled the same in Soong between android apps
and apexes, so share the bp2build code as well.
There are a few changes in this CL:
- If override_apex.certificate is unset, the generated apex also unsets
it. This prevents the generated apex from using the base apex's
certificate, which is most likely incorrect (e.g. google variant using
the cert for the aosp variant). Instead, rely on the default
certificate handling in the macro.
- If the certificate prop is a string, then it gets generated into
certificate_name in order to disambiguate. This behavior is identical
to android_app.
Test: added various unit tests.
Bug: 249089160
Fixes: 249089160
Change-Id: I99e18964ff546429a985d0f64dc21e2c69d35d9d
Add it to the `CommonAttributes` struct and handle it in fillCommonBp2BuildModuleAttrs
Bug: 190817312
Test: treehugger
Change-Id: I7b21056e680384d4046372ee844512029721026e
bazel_path conversion would treat "./" as a package boundary
which would be incorrect
Bug: b/243072036
Test: unit test build_conversion_test.go and `m libc`
Change-Id: I0fc44d2fd7a4270442c8755219a11fd585244c5c
