This shaves off ~20s, which is ~11%, of `m nothing` with mixed builds.
Test: Manual verification with mixed builds of aosp_flame.
Change-Id: Ief1bc56c92329e899c829fc1374eea9ee9b94fb7
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.
Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
Bazel overlay is an experimental feature. This renames the feature to 'queryview' to better describe its purpose, and also move away from the already overloaded 'overlay' term in Android.
Test: m queryview && bazel query --package_path=out/soong/queryview //...
Change-Id: I8b5068c7db46cb61a03a8e87af9c7c9077ebeff9
A recent change to Blueprint [1] added the feature to infer the type of
list elements to more than just "list of strings", but this also changed
the text to "list of string" (without s). This CL fixes the bazel
overlay, which uses "list of strings".
Test: m bazel_overlay && bazel query //... --package_path=out/soong/bazel_overlay
[1]: 764a771917%5E%21/#F1
Change-Id: I7522d15b1ce751807e1881ce1e94f068d226b316
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.
Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
This keeps the top level directory clean, and leverages the
new "build/bazel" repo location / package path.
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Bug: 171180724
Fixes: 171180724
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I151b8f3d2177167711e5974e5eabad8d6e910276
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.
source build/soong/bazelenv.sh to enter bazel-enabled mode.
Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.
Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
surface module properties as attributes.
This CL maps int, bool, string and string_list module props from Soong
modules into their respective Bazel targets.
With this CL, you can now query modules based on their properties. For
example:
$ bazel query 'attr(nocrt, 1, //...)'
$ bazel query 'attr(apex_available, //apex_available:platform, //...)'
$ bazel query //art/dalvikvm:dalvikvm--linux_glibc_x86_64 --output=build | grep compile_multilib
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery 'attr(apex_available, com.android.runtime, //...)'
Test: soong_build tests
Fixes: 162720644
Fixes: 164320355
Change-Id: Iea8e594b952feccac3281f36dd6bdee8e7d62c3a
This patchset changes bazel_overlay to generate soong_module as a macro,
instead of a rule, and generate module properties in the BUILD files as
kwargs to the soong_module macro.
Here's a sample of the new BUILD files with module properties:
bionic/libdl/BUILD.bazel:
https://paste.googleplex.com/6484466996346880?raw
art/build/apex/BUILD.bazel:
https://paste.googleplex.com/5461276001042432?raw
bionic/apex/BUILD.bazel:
https://paste.googleplex.com/4932795173437440?raw
soong_module is now a macro that conditionally expands to underlying
soong_<module type> rules with statically defined attributes. In this
CL, we are starting with a hardcoded filegroup rule definition to
demonstrate the conditional rule loading within the soong_module macro.
If the module_type matches an existing Bazel rule, soong_module forwards
the entire **kwargs into the rule, which Bazel typechecks.
Non-filegroup module types will be expanded into generic_soong_module,
but with the kwargs dropped.
This approach allows us to:
1) Programmtically generate soong_<module type> rules for all module
types available in Soong, together with the statically defined attribute
types in `attrs`.
2) Incrementally migrate and test individual module types from
generic_soong_module to their module rule shims.
3) Swap out the module rule shims to the actual Bazel rules (e.g
cc_library, java_library) and perform attribute manipulation in Starlark
itself.
Example of querying against the 'srcs' attribute in soong_filegroup:
```
$ bazel cquery 'kind(soong_filegroup, //...)' | wc -l
590
$ bazel cquery --output=build 'attr(srcs, "linker.cpp",
kind(soong_filegroup, //bionic/...))'
INFO: Analyzed 3907 targets (0 packages loaded, 0 targets configured).
INFO: Found 3907 targets...
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/bionic/linker/BUILD.bazel:4144:13
soong_filegroup(
name = "linker_sources",
generator_name = "linker_sources",
generator_function = "soong_module",
generator_location = "bionic/linker/BUILD.bazel:4144:13",
srcs = ["dlfcn.cpp", "linker.cpp", "linker_block_allocator.cpp",
"linker_dlwarning.cpp", "linker_cfi.cpp", "linker_config.cpp",
"linker_debug.cpp", "linker_gdb_support.cpp", "linker_globals.cpp",
"linker_libc_support.c", "linker_libcxx_support.cpp",
"linker_namespaces.cpp", "linker_logger.cpp",
"linker_mapped_file_fragment.cpp", "linker_phdr.cpp",
"linker_relocate.cpp", "linker_sdk_versions.cpp", "linker_soinfo.cpp",
"linker_tls.cpp", "linker_utils.cpp", "rt.cpp"],
deps = [],
)
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/soong_module.bzl:32:23
in <toplevel>
```
This CL is known to be lacking the following features, and will be looked at in follow up CLs:
1) Pretty printing reflect.Interface properties, like arch, multilib and
dists.
2) Generating module Bazel rule shims for all module types, instead of
hardcoding them like `soong_filegroup`.
Bug: 162720644
Test: bazel_overlay_test.go (soong build test)
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ic1e448887eb540ed15a55bc4090cf75a4d832d41
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.
To create this overlay, run `m bazel_overlay`.
A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.
This enables users to use bazel query as a way to introspect the Soong
module graph. For example,
- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:
$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids
- Why does com.android.runtime depend on lzma?
$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime
- What does the dep graph of //bionic/libc:crtbegin_so look like?
$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png
https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8
Test: croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.
Fixes: 160119159
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
For an end user hitting these errors, the most likely reason is that
they have mixed up linker and crtbegin_dynamic.o from different builds
in their out tree.
Test: Build a linux_bionic executable in a tree exhibiting b/157549171.
Test: m nothing
Bug: 157549171
Change-Id: I64bee0848ed61226da6654caf3bafb51cf46d923
apex_available/min_sdk_version are required to be available for mainline
modules(APK/APEX).
- java_import/java_library_static
- android_library/android_library_import
Bug: 156996905
Test: prebuilts/sdk/update_prebuilts.sh
Change-Id: I201ebabaa533c5ee19e394ab30ca2fa0e3d7ce0e
"dlv attach":
- Spams the terminal
- Requires sysctl -w kernel.yama.ptrace_scope=0
- Apparently, does not allow the debugger to inspect variables
Test: Manual.
Change-Id: I625369effaf5abda2b5d884c8ce5bde7247774eb
apex_set takes an .apks file that contains a set of prebuilt apexes with
different configurations. It uses extract_apks to select and install the
best matching one for the current target.
Bug: 153456259
Test: apex_test.go
Test: com.android.media.apks
Change-Id: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.
Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
Golang's FileInfo() implementation recognizes FAT filesystem directories
correctly, but doesn't actually add executable bits, which essentially
makes them inaccessible. Use 0777 when we know we're dealing with
directories.
(This is a cherry-pick change.)
Fixes: 153207404
Test: manually tested with an aar file with FAT permission bits.
Change-Id: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Merged-In: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.
Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.
Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
This change automates one of the best practices for
custom tools. The best practice says "Prefer to list
input files on the command line, otherwise we may not
know to re-run your command when a new input file is
added." [1]
Normally you'd reference these inputs with $(in) or
one of the forms of the $(location*) substituions on
the command line for the custom tool. However, if the
custom tool does not accept the list of inputs on the
command line, the build system could fail to re-run
the custom tool if the list changes.
This change adds a hash of the list of input names to
the sbox command that wraps the custom tool. If the
list of inputs change, the hash will change, and
therefore the sbox command will change, causing ninja
to re-run the custom tool. The hash is visible to
(but ignored by) the sbox command, and hidden from
your custom tool.
[1] https://android.googlesource.com/platform/build/soong/+/refs/heads/master/docs/best_practices.md#custom-build-tools
Test: TestGenruleHashInputs
Bug: 149397658
Change-Id: I18b547ea3c4296ee15bd6150a4778a8f376d80b7
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
Test: touch empty.zip; merge_zips out.zip empty.zip
(output should contain `empty.zip: zip: not a valid zip file`)
Change-Id: If707876af2301d7ad7d065c004cffb0c2491c7df
Apparently we have some instances of Soong taking 16GB of RAM. While
that's a problem to solve on it's own, make multiproduct_kati's
auto-parallelism detection smarter by taking into account the total RAM
on the machine instead of just the number of CPUs.
Bug: 146925549
Test: check soong.log for autodetected parallelism values
Change-Id: Ice34c2cf73ea4f8f235521cbefc8654a79a04eef
This is so that we can split the build_test build on large branches into
multiple builds, each testing a subset of the products.
Test: m blueprint_tools (runs the go unit tests)
Test: build/soong/build_test.bash --only-config
vs
Test: build/soong/build_test.bash --only-config --shard-count=2
Test: build/soong/build_test.bash --only-config --shard-count=2 --shard=2
Change-Id: I40ccc1aa477bc0ffa74ff564d155068509be18f0
soong_javac_wrapper is silencing a useless warning:
warning: [options] bootstrap class path not set in conjunction with -source 1.9
but recent versions of javac have started also printing:
1 warning
Read the warning count, subtract the number of silenced warnings, and
reprint it if the non-silenced warning count is nonzero.
Fixes: 144118634
Test: javac_wrapper_test.go
Change-Id: Ie1d0a978188ab7b1c41027f718a1274608628123
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table. Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.
Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.
So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".
Also fix up rule_builder to actually tell sbox about the depfile.
Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d
Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
Android build generates 180K+ zip files that are to be merged and the
result passed to the indexing pipeline to generate the source code
cross-reference. Their names cannot be passed on the command line,
hence the need for this change. That required extensive changes to
the code as it no longer can be assumed that all the input files can
be kept open.
Bug: 121267023
Test: internal + treehugger + combine index files
Change-Id: I0a829f96ff7187ad967fb6b8cede387501ec57cc
mm is now doing the same thing as mma. Now print a deprecation message
for other tools that may be setting ONE_SHOT_MAKEFILE directly (or users
that are still using a very old envsetup.sh).
Test: mm # does not print
Test: ONE_SHOT_MAKEFILE=bionic m
Change-Id: I98221657024d824a61c0d721d5c7dd1a2ea2fb1e
1. Update pom2bp to support "host_support" device module.
2. Allow other module type in support library
(The intention is to support the room annotation processor)
Test: m
Bug: 137918883
Change-Id: I9a8fbcff903cc395d5dc38430774202ce389d6bc
Merged-In: I9a8fbcff903cc395d5dc38430774202ce389d6bc
(cherry picked from commit bd7c2f9a1d)
This is the first part of deprecating the mmm command. mmm will simply
be redirected to mmma by building the list of modules with their
dependencies.
Bug: b/135187558
Test: Ran mmm external/libusb:libusb.
Change-Id: I50cf0d6a07197556ab6130e612561c6d5a3fb31e
This is the first part of deprecating the mm command. mm will simply
be redirected to mma by building the modules and their dependencies.
Bug: b/135187558
Test: Ran mm in external/protobuf directory.
Change-Id: Idad40f89e3768aeb459172f632a44fd7b0a201bc
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.
Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
Keep a running map of the critical path to each edge as it
finishes. At the end of the build, find the edge with the
longest critical path and print the path to out/soong.log.
Test: critical_path_test.go
Change-Id: Ie01d26b068b768156b166bff00cdc3273e8124ca
Soong_ui produces a build_error raw file to $(OUT_DIR) where
the file contains a list of build action errors. Each build action
error represents an error raised by a command. The build_error file
is populated if there was a build error.
Bug: b/132969697
Test: Ran m for successful build. Introduced a broken build change,
ran m and verified using printproto that build_error was
generated successfully.
Change-Id: I690ca1778b4e56f144a3173ba1d16d8494c32c15
There was one case that did not work:
1) Create dir in root source tree
2) Run "m libc"
The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.
Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
of the source tree.
Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
ctx.Writer is recommended stream to use for standard io.
Fixes: b/135487952
Test: Lunch and executed ./out/soong_ui --dumpvar-mode and
--dumpvars-mode to check the usage is output to the console.
Change-Id: I38351ebdb05f49f92d6b4bfb0423092299d1c5ea
The --build-mode flag is a new flag in soong_ui that accepts a build
action (m, mm, mmm, mma, mmma), the directory where the build action
is occuring and an optional list of build arguments and invokes the
build option. This is to deprecate the build actions behavior in
envsetup.sh and start using the one defined in soong_ui.
Bug: b/130049705
Test: Ran lunch and ran all the build action commands in soong_ui.
Ran a bunch of invalid commands such as no Android.mk file in
a specific directory, invalid target name, etc...
Change-Id: I88e9e59f47f8c5cdc872fb83a0f3f0a0919885b6
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.
Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
by using printproto command.
Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
Write log output through StatusOutput so that the status implementation
can synchronize it with its own output.
Test: status_test.go
Change-Id: I917bdeeea4759a12b6b4aa6d6d86ee18a2771723
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality. Replace it with stdio.Stdout() as an
io.Writer.
Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
Instead of extracting the aar, and passing the res directory to another rule
(along with a dependency to a different known file), don't keep the
extracted res directory, and pass the aar directly to the resource
compliation rule, which will extract just the res directory.
I need this for my RBE experiments, where non-listed output files won't
exist in other rules.
Test: m
Change-Id: I99074381052cbcebb6a402484abae9ab2e40284f
Currently, the command line argments is being processed in multiple
places. In the main soong_ui arguments, there are several if statements
that checks if a specific argument was specified in order to execute
the requested operation. This does not scale well when adding or
removing a command in the near future. In order to support the build
commands (m, mma, etc...) from the envsetup.sh in soong_ui, a refactor
was required in order to add a command rather quickly and to have the
flexibiity to unit test the command. The soong_ui arguments format is
as follows:
soong_ui <command> [<arg 1> <arg 2> ... <arg n>]
The <command> is a specific operation to be executed. The arguments
after the command are processed by the command itself.
Below is the list of changes made in this commit:
* Created a new command infrastructure that allows adding
or deprecating a command easily.
* Fixed a bug when running ./soong_ui.bash directly would cause
a panic due to index being out of range for args list.
Bug: b/130049705
Test: Below is the list of testing done on this commit:
* Ran lunch and verified that the output is the same as the the output
without the modifications. lunch indirectly runs soong_ui passing
in the --dumpvar-mode (to read makefile variables such as
TARGET_PRODUCT) and --dumpvars-mode (to build the build cache).
* Ran ./soong_ui.bash directly (with unsupported flags and no flags)
and the proper message appeared that soong native UI is not yet
available.
* Ran m, mm, mmm, mma, mmma commands.
* Ran the make installclean command:
make -j80 TARGET_PRODUCT=aosp_arm64 TARGET_BUILD_VARIANT=eng dist DIST_DIR=/tmp/helloworld installclean
* Ran ./out/soong_ui -j80 --make-mode PRODUCT-aosp_arm64-eng dist checkbuild tests
Change-Id: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
Merged-In: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
This changes the default behavior of pom2bp to match pom2mk, and not
require pre-extraction of all of the manifests.
Test: convert an internal pom2mk user to pom2bp
Change-Id: I08120cce1f923ef28519563ffd772032b2e500ed
Allow callers of pom2bp to add dependencies on runtime libraries as
well as static libraries. Renames -extra-deps to --extra-static-libs
to make it clearer what it does.
Bug: 130287656
Test: prebuilts/sdk/update_prebuilts.py --stx 5280039
prebuilts/misc/common/androidx-test/update-from-gmaven.py
Change-Id: I8a3ee8056aa4ed9935a4f47d563ec423672cf38b
This will prevent it from being affected by future css location changes.
Test: m soong_docs
Fixes: 130567020
Change-Id: I80fd7b8a12a32bcea7b270423b8d8f8dc0976177
This essentially allows you to declare that everything in a directory
will be created by the rule, and we'll ensure that your command actually
writes out all of the claimed outputs, and remove any other files that
previously existed in that directory.
Test: built-in tests
Change-Id: I990dce2b3a0d89ebd2736ac1a0cadfb5864c6e73
The dependency type in the pom file is not always accurate, just
add all jar and aar dependencies to both the -nodeps and final
module.
Test: cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp && mma
Change-Id: I1784b8ebf3485db19405cb353c9bb5fb3ae837d8
android/soong/common was renamed to android/soong/android long
ago, but the pctx package path was still "android/soong/common".
This required all users of rules defined in android/soong/android
to import "android/soong/android" and then
pctx.Import("android/soong/common").
Test: m checkbuild
Change-Id: I20d096522760538f7cfc2bec3d4bfeba99b275d4
* changes:
Add a whitelist for services.art changing between builds
Add diff_target_files tool
Import files from compare_target_files for use in diff_target_files
Some tools used by genrules set the timestamp of the generated file
to an old value. For example, when using tar to extract files it
will set the timestamp of the extracted file to the value stored in
the tar file. This can cause unnecessary rebuilds due to output files
that are older than their inputs. Touch output files in sbox before
moving them into the output location.
Test: m checkbuild && m checkbuild
Change-Id: Idb0e71d2bf0ac3618338371132cc71be74bf9d9d
Add a tool that lists files that have changed between two builds by
comparing the target-files.zip artifacts.
Bug: 121158314
Test: unit tests
Change-Id: I3ca5dd1fdf652dffef923ad7307fd71e49e2e95a
Allow rules built with RuleBuilder to use depfiles. Ninja only
supports a single depfile with single output. If there are
multiple outputs in a rule, move all but the first to implicit
outputs. If multiple depfiles are specified, use new support
in dep_fixer to combine additional depfiles into the first depfile.
Test: rule_builder_test.go
Change-Id: I7dd4ba7fdf9feaf89b3dd2b7abb0e79006e06018
Bring property type strings to the start of property comments. This is
how source.android.com shows variables and paratemeters, and it is also
easier to read, especially when a comment ends with a code example.
Test: m soong_docs
Change-Id: Ic3e9db3a3711495998919ee95bbef6c7cb33b215
To add code format (<pre>...</pre>) support to module type texts.
Fixes: 124332708
Test: m soong_docs
Change-Id: Ibe9d3845a8f952fdef5007893d4af5836809ce1f
It appears that there's a bug in repo where when a linkfile is removed
from the manifest, the symlink remains in the source tree.
Cleaning this up needs to happen before we start scanning the source
tree for Android.bp/Android.mk files, so this has to be done very early
-- way before CleanSpec processing could handle it.
Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=10583
Test: ln -s missing hardware/qcom/sdm710/Android.bp; m
Change-Id: Ib68f2507ffe58ccdd9fbc88925f8a4f6150f2f7d
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings. Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.
Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
The href base added by I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
doesn't really play nice with anchor links. This fixes the issue.
Fixes: 126550408
Test: m soong_docs
Change-Id: If62411a6391fd8151fa40cb2ecf32c5006797d6c
This adds a script to copy the current page's base url to hrefs in the
doc.
Fixes: 123473548
Test: m soong_docs
Change-Id: I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
Previously the build doc was a gigantic list of modules and properties,
which can be overwhelming to new users. This change breaks it up by
packages, so that it is easier to look up and feels more coherent.
Bug: 123521276
Test: m soong_docs
Change-Id: I1a1a331f4b4deb8782d698e9076098c5a27b9566
Uses more python rather than C++, and skips less of Py_Main.
Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
It now has module list on the left and the main panel containing
a section for each module. Each section contains the navigable list
of the module's attributes, grouped by attribute's origin.
Test: Manual
Bug: 120512870
Change-Id: Ib62ef58c61daa88950f0934a70ba8ed16f433bf2
bootstrap.ModuleTypeDocs needs a mapping of module types to factories
to support factories that are wrapped in ModuleFactoryAdapter closures.
It also returns ModuleType objects grouped into Package objects.
Bug: 67909957
Test: m soong_docs
Change-Id: I70eac9f0f0e13075580da92d4219792ca0b18fbf
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990
Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
The zip file format does support scripts/programs prefixed to the
archive, though many of the offsets are supposed to start from the
beginning of the file. Some tools (python and zipinfo) are able to
read zip files with arbitrary prefixes, but others (libziparchive and
zipdetails) fail.
So pass the file to prefix onto the zip file into merge_zips so that we
can set the offsets from the real start of the file.
Test: m sepolicy_tests (runs the embedded python interpreter)
Test: zipinfo out/host/linux-x86/bin/sepolicy_tests
Test: zipdetails out/host/linux-x86/bin/sepolicy_tests
Change-Id: If73d4c2465581f7de5aa47959284ecf2059df091
Gold uses two PT_LOAD programs: RX, RW
LLD is using three PT_LOAD programs: R, RX, RW
All non-static host bionic executables are linked with LLD, as it's
required in order to embed the linker. The dynamic linker is considered
a static executable for this definition.
When the linker had been built with Gold, we inserted the the RX&RW
loads in the middle of the executable's readonly load, splitting the R
load into two. When this happened, all of the linker's loads existed 1:1
in the final executable.
Now that the linker is built with LLD, we're inserting the loads into
the same place, but LLD merges the first part of the executable's R load
with the linker's R load, so there's not a 1:1 matching.
So just make sure that all of the linker's loads are contained within
the executable's loads (and with the appropriate flags).
Test: m blueprint_tools (runs new tests)
Test: build host bionic binaries
Change-Id: If7868d2ab8980d27ba5ab8994fefe0d6142f215a
Patterns containing multiple globs or a recurisve glob may match
files with the same name in multiple directories. Keep the relative
directories of matches after the path entry containing a glob.
Bug: 117295826
Test: zip2zip_test.go
Change-Id: I5d663e546953af374175837551d23f484d568377
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.
Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
In order to simplify the wrapper function, and stop using a linker
script, generate a set of flags to pass to LLD. Then run
host_bionic_inject on the linked binary in order to verify the
embedding, and give the wrapper function the address of the original
entry point (_start).
Bug: 31559095
Test: build host bionic with prebuilts/build-tools/build-prebuilts.sh
Change-Id: I53e326050e0f9caa562c6cf6f76c4d0337bb6faf
Don't warn on duplicate files in merge_zips if they have identical
CRC32 and size values.
Test: m checkbuild
Test: merge_zips_test.go
Change-Id: I61336ca4d4d3b7402c24a7abd337bd350fe10930
Make needs to exclude all files that match a filespec except for
those specified in a list. Add a -X argument that overrides the
-x argument to allow files be included that were previously
excluded.
Bug: 69500920
Test: zip2zip_test.go
Change-Id: Icc8aebc788b53bcdde73a39c9c7cf3107a049251
Add -0 flag to convert files in a zip to stored instead of deflated.
Bug: 69500920
Test: zip2zip_test.go
Change-Id: I6c2b10f3b200a53a3339e3c97a78f65192b309ca
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.
Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.
Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.
Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
Add test that cover basic command line usage of soong_zip. -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.
Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
Move the actual per-product builds out of the main function, combining
the product configuration and build sections. This makes it a lot more
readable.
This ends up saving a lot of FDs -- a -only-config build on aosp with 77
products used to require ~500 FDs, it succeeds with a limit of 150 FDs
now. I'm leaving the code to bump our FD limit though, since 2x the
number of internal products is too close to the 1024 limit for comfort.
Bug: 70370883
Test: prlimit -n150:150 build/soong/build_test.bash -only-config
Change-Id: Ia559beadc19deb8a5b9d50af6e0276e846fd8608
This uses a very large amount of intermediates, but can speed up
subsequent multiproduct_kati runs by not re-reading all the makefiles
unless necessary.
Since we'll no longer have the full build output on every run,
explicitly save away a std_full.log of the log from the last time we ran
Kati.
Test: build/soong/build_test.bash -products aosp_arm -incremental
Change-Id: Iaae580c0e49a2024528617d39b2b477e29d99250
Move some of the more complicated conditional logic out to helper
functions. Use the {{- }} syntax to strip previous whitespace to
allow spacing out the remaning conditionals in the templates.
Test: (cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp)
Change-Id: I766bd0e1837aa04375f322fbe796d923cd99ecde
Allows specifying make goals when using multiproduct_kati, for example:
$ multiproduct_kati -only-config dump-products
This makes dumping products with multiproduct_kati easier than before.
Test: multiproduct_kati -only-config -products=full,aosp_arm64 -out=/tmp/p dump-products
Change-Id: If526bd2ba1b7bbadcb230c020202ef9e421e1852
merge_zips -stripFile was only considering the name of the file and
ignoring the path. Make it more useful by supporting blueprint style
globs. The previous behavior can be recreated by prefixing with **/.
Bug: 111389216
Test: m checkbuild
Change-Id: I25760fe3f1f77704dd9da9d107d9a38a415d681f
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.
For inputs:
Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.
Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.
multiproduct_kati loses its custom status routines, and uses the common
one instead.
For outputs:
The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.
The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.
A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.
Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.
Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
manifest property should only be added to aar imports and not jar
imports.
Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ifaab81582c87f0af4419f90c3928d0d5d52cec33
When manifest merging in Soong is turned on the modules created
by pom2bp will need to have min_sdk_version specified in order
to prevent manifest_fixer from setting it to the current SDK
level. Read the minSdkVersion attribute out of the
AndroidManifest.xml file inside each .aar file.
Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ieaf37ba6eccaf32dc5d3411566335830b1a327ff
This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
When the config loading was changed to using AndroidProducts.mk lists
from soong_ui instead of finding them itself, multiproduct_kati stopped
being able to find all the products. Move the finder initialization
earlier so that we can dump the full list.
Test: build/soong/build_test.bash -only-config
Test: forrest runs of every downstream build_test
Change-Id: I085d40f9008a47b850783499365977a0ff70ac66
protoc dependency files, at least for C++ outputs, uses the form of:
a/b.c \
a/b.h: <dep1> <dep2>...
Ninja will fail the command when it parses a dep file and there's more
than one output file (even though it doesn't care what the output file
name is). So this tool will parse the original file, and output a
version with only a single output file.
Bug: 67329638
Test: NINJA_ARGS="-t deps ...pb.c" m
Test: NINJA_ARGS="-t deps ...srcjar" m
Test: NINJA_ARGS="-t deps ...srcszip" m
Test: Run dep_fixer across all of taimen's dep files, no failures.
Test: Run dep_fixer against the processed files, no changes.
Test: Run androidmk across all of our Android.mk files, inspect the diffs
Change-Id: I4263b7d5faea37285afa6b24dedf5964aa7d19dc
If a static android library lib1 has static_libs: ["lib2"] then the
R.class files for packages in lib2 will be merged into the jar for
lib1. If an app has lib1 in its static_libs it will get the R.class
files from lib2 through lib1, instead of regenerating the R.java
files with numbering that matches the resource table of the app.
Pass transtive static android library dependencies on the aapt2
command line so that aapt2 will always regenerate the R.java
constants for those packages.
Also extract the packages that have R.java files after each aapt2
invocation. This is not necessary for Soong, but is passed to
make to let it force regenerating the packages using
--extra-packages.
Bug: 78300023
Test: m checkbuild
Change-Id: I0f3444af44d2a9f370d1f156c908972f8cc3a1ee
Convert pom2mk to pom2bp that writes out Android.bp files instead
of Android.mk files. pom2mk stays for now until the last users
of it are cleaned up.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.mk
Change-Id: I584d63c4228bad32f9e1914b06bde807078d6a55
Add an option to pom2mk to exclude modules by name.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk -exclude androidx.car_car
Change-Id: I083907ef364384aace524ced81820567f5075d76
pom2mk -regen Android.mk will read the previous arguments out of
Android.mk, rerun pom2mk with those arguments, and write the result
to Android.mk.
Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk
Change-Id: Ie8e0f97ae5b87c8078cf8e753489326d80da1896
Specifically, the use case is android-support-car requires some stubs that don't have a Maven pom.
So those stubs can't easily be listed as a dependency in the pom.
However, we do want the dependency to exist when importing to Make and detected by extract_deps.py
Bug: 76692459
Test: m -j blueprint_tools \
&& update_prebuilts.py -s <build_id> \
&& git diff HEAD^ -- fix_dependencies.mk \
&& m -j checkbuild
Change-Id: If546ecf7416580a01bde7b906976ada4b1145253
An upcoming change will stop exporting ProductVariables from Config, so
switch to using existing accessor functions, and add more when they're
missing.
Bug: 76168832
Test: out/soong/build.ninja is identical
Change-Id: Ie0135bdbd2df3258ef3ddb53e5f8fc00aa9b97f7
For easier debugging if there are multiple duplicates
Bug: 76692459
Test: ./update_current.py -x <build-id>
Change-Id: I3079cb4ed1cfd365d792a7b41c9cdb4e7a1e148f
The make javac rule uses [ -s srcjar-list ] to decide whether or
not to run javac. zipsync was putting a trailing "\n" in the
list file, so the file was never empty. There was one case of a
package that contained no source files (so java-source-list was
empty) and had a res directory with an xml file that declared no
resources (so aapt.jar was created but empty). The 1-byte
srcjar-list file caused javac to run and error with:
javac: no source files
Test: m checkbuild
Change-Id: I26b394b66bf81e4f5abbd27e4dc06bee1d9420a8
extract_srcjars.sh uses zipinfo and unzip, which fail with an
error on an empty zip file. Instead of trying to hack around
this (which is hard to make guarantees for since they are
non-hermetic host tools), replace them with a go tool to unzip
a set of zip files into a directory.
Bug: 73885582
Test: m checkbuild
Change-Id: I151fed347ed5196726e36866ffc27bc831799afb
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.
Test: m -j checkbuild && real examples:
Bug: b/70568913
Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
Allow native modules to specify use_version_lib, which will make
an android::build::GetBuildNumber() function available. For host
builds, the function will return the build number at the time that
the module was linked. For device modules it will return the
value of the ro.build.version.incremental property.
Bug: 71719137
Test: build_version_test
Test: m build_version_test && touch build/make/core/Makefile build/soong/cc/libbuildversion/tests/build_version_test.cpp && m build_version_test shows different build numbers for binary and library tests.
Change-Id: I6f7d40b7574bb8206866c4e39bad9c710c796e32
Parse the machine-specific symbol tables into a custom symbol table
object and use a single function to find the offset and size of the
symbol in the custom table. Make the functions to convert a
machine-specific symbol table into the custom table also take mocks,
and provide functions to dump mocks from real files. Add tests
that take the mocks and verify they produce the right offset and
size.
Test: cmd/symbol_inject/*_test.go
Change-Id: I25654032b5017dd13a9a1fe29f8b8826ce5bc82b
Some exe files have a .data symbol at the same address as the
soong_build_number symbol. If the .data symbol is after
soong_build_number in the symbol list, symbol_inject would think
the end address was the same as the start address, and use
uint32(-1) as the size.
Use sort.Search to find the first symbol whose section number is
the same as the target symbol, but whose address is higher than
the target symbol.
Test: manual
Change-Id: I51d6e53c6b906222ba68c5cf93be944843e23550
macho symbol table entries are not always in order, which breaks
finding the next symbol to find the size of the target symbol.
Test: build_version_test
Change-Id: I41d1c3c3ff9929694e9ec2b034553d6b7ddef937
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.
Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.
Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
Only include dependencies for scope "compile."
Bug: 73263586
Test: ./update_current.py -s -t <build-id>
Change-Id: I2ac6055cb4c1ad1f8d7924869f54f50e5e662742
Move __init__.py files population to merge_zips, and add options: -p, and
-e. Since par needs entry_point.txt to search the program entry point, so
add -e option to support it.
Test: ./merge_zips -p -e entry_point.txt out in1.zip in2.zip
Bug: 70568913
Change-Id: Ida10faf125e1b94dffc98b30240db3b90fd75b25
Add a rule in soong that re-executes soong_build in order to
generate build documentation. This allows Soong to customize
the documentation.
Bug: 70516282
Test: m soong_docs
Change-Id: If143cfacd6ac20274cd7bb8d8fab0c07025a5553
On large branches (250+ configurations), the open file count can go
over the default 1024 soft limit on Ubuntu. Many systems have increased
that default, but for the ones that haven't, at least opt into the
hard limit until this can be refactored to use fewer open files.
Bug: 70370883
Test: prlimit -n256:4096 build/soong/build_test.bash -only-config
Test: ulimit -Sn 256; build/soong/build_test.bash -only-config (darwin)
Change-Id: I7a952ffc89a0149ab65b04db1523d348daa6ab3e
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
which should make debugging faster.
Bug: 66921582
Test: ./out/soong/host/linux-x86/bin/sbox --output-root outs --sandbox-path out/.temp/sbox-work -c "cd __SBOX_OUT_DIR__ && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p" outs/a outs/b outs/z # and observe the below output:
mismatch between declared and actual outputs
in sbox command(cd out/.temp/sbox-work/sbox343858828 && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p)
in sandbox out/.temp/sbox-work/sbox343858828,
failed to create 2 files:
b: not a file
z: does not exist
did create 16 files:
a
asubdir/child
c
d
e
f
g
h
i
j
...6 more
Change-Id: I75e37834c44d4279dec874701d67ce8bb01b872c
This reverts commit 63a250a336.
Reason for revert: Some failures:
namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file
such as New Build Breakage: aosp-master/build_test @ 4475274
Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
This reverts commit fb941913a3.
Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.
Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
Add a --ignore_duplicates option to cause merge_zips to
silently take the first entry in the case of duplicate entries
in the zips to be merged. This will be used for soong Jacoco
support to combine the instrumented jar with the original jar
to pick up any classes that were excluded.
Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: I49ede3f07ce0ed7701b4db7058da2e4f11b8043e
Jacoco support will use zip2zip to create a jar that is a subset
of another jar, and will need exclusion filters and recursive
globs. Switch zip2zip from filepath.Match to pathtools.Match,
and check each included file against the exclusion list.
Bug: 69629238
Test: zip2zip_test.go
Change-Id: Ibe961b0775987f52f1efa357e1201c3ebb81ca9c
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).
Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
For unbundled builds, LOCAL_SDK_VERSION needs to be set, otherwise we'll
try to use frameworks-res, which isn't available.
Bug: 64723465
Test: Use with prebuilts/sdk/update_current.py
Change-Id: If5e1233b4bcb09b3d0e2251e8dd1a8ecfc87eb19
Instead of deleting artifacts/logs from successful build (unless -keep
is set), and keeping unsuccessful artifacts, keep all logs and remove
all artifacts (unless -keep is set, then we'll compress the artifacts).
If -dist is passed in, we'll put an archive of the logs into the
DIST_DIR. Even compressed, the rest of the artifacts are still a bit too
large to dist (~5.6GB on AOSP).
Test: build/soong/build_test.bash
Test: build/soong/build_test.bash -keep
Test: build/soong/build_test.bash -dist
Change-Id: I87f55978c18c8ff2e517b8271554ba383003742f
Android Support Library 26+ does not specify dependency types, so parse
all pom files first, then propagate missing dependency types if we've
got the information.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 26.0.0-beta2 com/android/support
Change-Id: Ieaff757ff198c9a7b4b006623340b382728c1fd4
Also removeTempDir -> !keepOutDir
Also shorten the help comment
Bug: 68336760
Test: rm out -rf && m -j
Test: sbox -c # and observe that the output does not have a stacktrace
Change-Id: I6fbb385132128ab1273c8398ff2425191049751e
For repositories that contain multiple versions of the same artifact,
we'd only like to create makefiles for one of the versions of the
artifact.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 25.3.1 com/android/support
Change-Id: I5bdbba8d2143a47610d56d679381eb01b3cf176d
Most notably, the sandbox depfile path should be passed into the tool.
Bug: 68336760
Test: m -j checkbuild
Change-Id: I22f944a3f57d613fda26de0ea777a915cafcd020
ALLOW_MISSING_DEPENDENCIES=true can cause missing dependencies to be
hidden from GenerateAndroidBuildActions, which is triggering the
len(jars) == 0 check in compileJavaHeader. For the
ALLOW_MISSING_DEPENDENCIES=true case the check is unnecessary, as the
build rules will be replaced with runtime errors. On the off chance
the check would have been hit in a real build, allow merge_zips to
create an empty zip file if it is asked to.
Test: m ALLOW_MISSING_DEPENDENCIES=true
Change-Id: I3171f921d51229ddf38a1a647d32566658c673fa
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)
Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
javac warns whenever -source is used with a version that does not
match javac, because it wants an rt.jar provided for the given
source level. We are not using -source to prevent usage of APIs
that don't exist in the newer rt.jar, so hide the messages.
Test: javac_wrapper_test.go
Change-Id: Ic94b61767436848fc164e377dc3cc5a63726f1cf
build/soong/soong_ui.bash --dumpvars-mode \
--vars="..." \
--abs-vars="..." \
--var-prefix="..." \
--abs-var-prefix="..."
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dump-many-vars \
DUMP_MANY_VARS="..." \
DUMP_MANY_ABS_VARS="..." \
DUMP_VAR_PREFIX="..." \
DUMP_ABS_VAR_PREFIX="..."
and
build/soong/soong_ui.bash --dumpvar-mode [--abs] VAR
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dumpvar-[abs-]-VAR
But uses soong_ui and ckati, so that we use a consistent make parser and
sandboxing configurations.
One major output difference between the pure make implementation and
this one is that report_config in Go is implemented using embedded
newlines in single quotes, while the make implementation uses `` with
embedded echo commands. This seems to work fine for both bash and zsh,
and report_config isn't meant to be machine-parsed anyways.
Test: build/soong/soong_ui.bash --dumpvar-mode report_config
Test: build/soong/soong_ui.bash --dumpvar-mode TARGET_DEVICE
Test: build/soong/soong_ui.bash --dumpvar-mode --abs PRODUCT_OUT
Test: build/soong/soong_ui.bash --dumpvar-mode --abs ALL_PRODUCTS
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="report_config TARGET_DEVICE" --abs-vars="ALL_PRODUCTS"
Test: build/soong/soong_ui.bash --dumpvars-mode --vars=TARGET_DEVICE --abs-vars=PRODUCT_OUT --var-prefix=v_ --abs-var-prefix=a_
Change-Id: I0fbd0732bbf6fcfcd24084cf3c830a91a4b6bfc2
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).
envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.
Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
This can make it easier to add build/soong/build_test.sh to continuous
build branches to catch new breaks, but where some products are known to
be broken.
Test: build/soong/build_test.bash --skip-products aosp_arm,aosp_arm64
Change-Id: Ib240570b68957077047ca75b78658429cb8c6a49
Putting resources in dex jars is going to need to pull files from
classpath jars that do not match *.class.
Test: m -j checkbuild
Change-Id: Ia37b8b9387e5f5636769afc6937b9aa3c9ec1ced
* changes:
Extract the linker and embed it into host bionic binaries
Add prefix_symbols support to cc_object
Fix ToolchainClangCflags for host bionic
Don't add a rpath to the linker
The linux kernel requires that the ELF interpreter (runtime linker)
that's referenced by PT_INTERP be either an absolute path, or a relative
path from the current working directory. We'd prefer a relative path
from the binary, similarly to how we handle looking up shared libraries,
but that's not supported.
Instead, extract the load sections from the runtime linker ELF binary
and embed them into each host bionic binary, omitting the PT_INTERP
declaration. The kernel will treat it as a static binary, and we'll use
a special entry point (linker_wrapper) to fix up the arguments passed by
the kernel before jumping to the embedded linker. From the linker's
point of view, it looks like the kernel loaded the linker like normal.
Bug: 31559095
Test: Enable host bionic,
out/soong/host/linux_bionic-x86/nativetest64/libdemangle_test/libdemangle_test
Change-Id: I8d0aea9790b5e86fcc3ea6e2d00cfa33907e2853
