Bazel overlay is an experimental feature. This renames the feature to 'queryview' to better describe its purpose, and also move away from the already overloaded 'overlay' term in Android.
Test: m queryview && bazel query --package_path=out/soong/queryview //...
Change-Id: I8b5068c7db46cb61a03a8e87af9c7c9077ebeff9
A recent change to Blueprint [1] added the feature to infer the type of
list elements to more than just "list of strings", but this also changed
the text to "list of string" (without s). This CL fixes the bazel
overlay, which uses "list of strings".
Test: m bazel_overlay && bazel query //... --package_path=out/soong/bazel_overlay
[1]: 764a771917%5E%21/#F1
Change-Id: I7522d15b1ce751807e1881ce1e94f068d226b316
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.
Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
This keeps the top level directory clean, and leverages the
new "build/bazel" repo location / package path.
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Bug: 171180724
Fixes: 171180724
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I151b8f3d2177167711e5974e5eabad8d6e910276
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.
source build/soong/bazelenv.sh to enter bazel-enabled mode.
Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.
Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
surface module properties as attributes.
This CL maps int, bool, string and string_list module props from Soong
modules into their respective Bazel targets.
With this CL, you can now query modules based on their properties. For
example:
$ bazel query 'attr(nocrt, 1, //...)'
$ bazel query 'attr(apex_available, //apex_available:platform, //...)'
$ bazel query //art/dalvikvm:dalvikvm--linux_glibc_x86_64 --output=build | grep compile_multilib
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery 'attr(apex_available, com.android.runtime, //...)'
Test: soong_build tests
Fixes: 162720644
Fixes: 164320355
Change-Id: Iea8e594b952feccac3281f36dd6bdee8e7d62c3a
This patchset changes bazel_overlay to generate soong_module as a macro,
instead of a rule, and generate module properties in the BUILD files as
kwargs to the soong_module macro.
Here's a sample of the new BUILD files with module properties:
bionic/libdl/BUILD.bazel:
https://paste.googleplex.com/6484466996346880?raw
art/build/apex/BUILD.bazel:
https://paste.googleplex.com/5461276001042432?raw
bionic/apex/BUILD.bazel:
https://paste.googleplex.com/4932795173437440?raw
soong_module is now a macro that conditionally expands to underlying
soong_<module type> rules with statically defined attributes. In this
CL, we are starting with a hardcoded filegroup rule definition to
demonstrate the conditional rule loading within the soong_module macro.
If the module_type matches an existing Bazel rule, soong_module forwards
the entire **kwargs into the rule, which Bazel typechecks.
Non-filegroup module types will be expanded into generic_soong_module,
but with the kwargs dropped.
This approach allows us to:
1) Programmtically generate soong_<module type> rules for all module
types available in Soong, together with the statically defined attribute
types in `attrs`.
2) Incrementally migrate and test individual module types from
generic_soong_module to their module rule shims.
3) Swap out the module rule shims to the actual Bazel rules (e.g
cc_library, java_library) and perform attribute manipulation in Starlark
itself.
Example of querying against the 'srcs' attribute in soong_filegroup:
```
$ bazel cquery 'kind(soong_filegroup, //...)' | wc -l
590
$ bazel cquery --output=build 'attr(srcs, "linker.cpp",
kind(soong_filegroup, //bionic/...))'
INFO: Analyzed 3907 targets (0 packages loaded, 0 targets configured).
INFO: Found 3907 targets...
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/bionic/linker/BUILD.bazel:4144:13
soong_filegroup(
name = "linker_sources",
generator_name = "linker_sources",
generator_function = "soong_module",
generator_location = "bionic/linker/BUILD.bazel:4144:13",
srcs = ["dlfcn.cpp", "linker.cpp", "linker_block_allocator.cpp",
"linker_dlwarning.cpp", "linker_cfi.cpp", "linker_config.cpp",
"linker_debug.cpp", "linker_gdb_support.cpp", "linker_globals.cpp",
"linker_libc_support.c", "linker_libcxx_support.cpp",
"linker_namespaces.cpp", "linker_logger.cpp",
"linker_mapped_file_fragment.cpp", "linker_phdr.cpp",
"linker_relocate.cpp", "linker_sdk_versions.cpp", "linker_soinfo.cpp",
"linker_tls.cpp", "linker_utils.cpp", "rt.cpp"],
deps = [],
)
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/soong_module.bzl:32:23
in <toplevel>
```
This CL is known to be lacking the following features, and will be looked at in follow up CLs:
1) Pretty printing reflect.Interface properties, like arch, multilib and
dists.
2) Generating module Bazel rule shims for all module types, instead of
hardcoding them like `soong_filegroup`.
Bug: 162720644
Test: bazel_overlay_test.go (soong build test)
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ic1e448887eb540ed15a55bc4090cf75a4d832d41
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.
To create this overlay, run `m bazel_overlay`.
A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.
This enables users to use bazel query as a way to introspect the Soong
module graph. For example,
- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:
$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids
- Why does com.android.runtime depend on lzma?
$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime
- What does the dep graph of //bionic/libc:crtbegin_so look like?
$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png
https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8
Test: croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.
Fixes: 160119159
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
For an end user hitting these errors, the most likely reason is that
they have mixed up linker and crtbegin_dynamic.o from different builds
in their out tree.
Test: Build a linux_bionic executable in a tree exhibiting b/157549171.
Test: m nothing
Bug: 157549171
Change-Id: I64bee0848ed61226da6654caf3bafb51cf46d923
apex_available/min_sdk_version are required to be available for mainline
modules(APK/APEX).
- java_import/java_library_static
- android_library/android_library_import
Bug: 156996905
Test: prebuilts/sdk/update_prebuilts.sh
Change-Id: I201ebabaa533c5ee19e394ab30ca2fa0e3d7ce0e
"dlv attach":
- Spams the terminal
- Requires sysctl -w kernel.yama.ptrace_scope=0
- Apparently, does not allow the debugger to inspect variables
Test: Manual.
Change-Id: I625369effaf5abda2b5d884c8ce5bde7247774eb
apex_set takes an .apks file that contains a set of prebuilt apexes with
different configurations. It uses extract_apks to select and install the
best matching one for the current target.
Bug: 153456259
Test: apex_test.go
Test: com.android.media.apks
Change-Id: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.
Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
Golang's FileInfo() implementation recognizes FAT filesystem directories
correctly, but doesn't actually add executable bits, which essentially
makes them inaccessible. Use 0777 when we know we're dealing with
directories.
(This is a cherry-pick change.)
Fixes: 153207404
Test: manually tested with an aar file with FAT permission bits.
Change-Id: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Merged-In: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.
Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.
Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
