Commit graph

157 commits

Author SHA1 Message Date
Colin Cross
70c4741215 Add explicit rspfile argument to RuleBuilderCommand.FlagWithRspFileInputList
Using $out.rsp as the rsp file adds extra complexity around keeping
the $ unescaped.  Make callers to FlagWithRspFileInputList provide
an explicit path for the rsp file instead.

Bug: 182612695
Test: rule_builder_test.go
Change-Id: I3f531d80c1efa8a9d09aac0a63790c5b11a9f0c6
2021-03-16 16:52:56 -07:00
Liz Kammer
a5f86249a4 Merge changes from topic "mixed-bp2build"
* changes:
  Support autoconverted modules in mixed builds
  Use handcrafted build targets in bp2build
2021-03-16 21:40:04 +00:00
Christopher Parsons
bc3f7e0276 Merge "Refactor and cleanup of cquery processing" 2021-03-16 21:12:01 +00:00
Liz Kammer
bdc609972c Support autoconverted modules in mixed builds
modules converted with bp2build_available are will also be available to
be used in mixed builds.

Test: build/bazel/scripts/milestone-2/demo.sh full
Test: go tests
Change-Id: I49f16ec3ba5bb11dfed8066af069c27eb04371fb
2021-03-16 15:11:49 -04:00
Chris Parsons
944e7d01aa Refactor and cleanup of cquery processing
Test: USE_BAZEL_ANALYSIS=1 m libc
Change-Id: Iaf9a92e84d39c132e2444a8aaafd79505a12b8ec
2021-03-16 14:39:16 -04:00
Paul Duffin
e66946b43e Migrate genrule package to a per test build directory
Bug: 182885307
Test: m nothing
Change-Id: I668f083ba914569198eaa90642821340321d6b4c
2021-03-16 12:46:52 +00:00
Paul Duffin
e84b1338c5 Migrate from result methods to function Asserts
Bug: 181070625
Test: m nothing
Change-Id: Iadb85270290acb52c55f2ad70c9f145f1c34b611
2021-03-12 22:24:51 +00:00
Paul Duffin
672cb9fbc7 Convert genrule tests to use test fixtures
Bug: 181070625
Test: m nothing
Change-Id: I7dae0aa0e493e2073286360038ddaafa2d90093c
2021-03-09 21:15:40 +00:00
Chris Parsons
8d6e433c1e Support cc_object modules in mixed builds
Test: With a handwritten conversion of crtbegin_so1, USE_BAZEL_ANALYSIS=1 m crtbegin_so1
Change-Id: I7c777d7f46b37aa1827cc04205e2014f9293bf35
2021-03-01 18:22:34 -05:00
Treehugger Robot
bbfd5ab606 Merge "Move bazel.Properties to a BazelModuleBase" 2021-02-24 05:53:25 +00:00
Liz Kammer
ea6666fba9 Move bazel.Properties to a BazelModuleBase
This allows more direct access to the bazel label and whether the module
is bp2build available.

Test: go test *
Change-Id: I23375835d20fa53d7d94127b2dc2d5bb20487bfb
2021-02-23 10:58:57 -05:00
Liz Kammer
fc46bc1ee4 Refactor BazelTargetModule
This eliminates the need to remove quotes, delete attributes, and
re-checking that name has correct prefix. Additionally, this allows
assignment directly to the BazelTargetModuleProperties struct, which
allows defaulting unused fields and clarity of which field is being set.

Test: go test soong tests
Test: ran ./build/bazel/scripts/milestone-2/demo.sh
Change-Id: Ia9bfcce76234c793a4ddd5f29a661150f83341c9
2021-02-23 08:29:40 -05:00
Treehugger Robot
b68036f44b Merge "Add LOCAL_LICENSE_KINDS to build/soong" 2021-02-10 07:18:29 +00:00
Jingwen Chen
fb4692a743 bp2build: refactor BazelTargetModule naming boilerplate.
This CL replaces the "__bp2build__" name prefix boilerplate with a props
creation function, and centralizes the prefixing in there.

Test: TH
Test: soong tests
Change-Id: Ic963199ab60dcce0d3361abff111cfa9acd4c21b
2021-02-08 04:10:18 -05:00
Bob Badour
02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Jingwen Chen
77e8b7b6d2 bp2build: add bazel_module: { bp2build_available } prop.
This CL adds a per-target allowlist to instruct bp2build on which modules it should generate Bazel targets for.

Test: soong tests
Change-Id: I869e66fce405c2c6689b381569b8cc0118cbcf76
2021-02-05 09:26:22 -05:00
Jingwen Chen
1fd14691dd bp2build: Refactor CreateBazelTargetModule API.
This CL refactors the CreateBazelTargetModule API to minimize boilerplate, and to establish a well defined function signature for the expected metadata about a BazelTargetModule.

Test: soong tests

Test: TH
Change-Id: I474ff5a2b0db8deeed49ba4ca73b416ccb494fdd
2021-02-05 09:24:55 -05:00
Liz Kammer
356f7d45c1 bp2build: convert paths/module refs to Bazel label
This currently expands all globs, still need to support converting glob
syntax.

Test: go build_conversion_test
Test: GENERATE_BAZEL_FILES=true m nothing
Test: m nothing
Bug: 165114590
Change-Id: If7b26e8e663d17566fad9614ca87a8da1f095284
2021-02-04 13:45:56 -05:00
Jingwen Chen
a42d6417b3 Make bp2buildMutators registration local to TestContext.
The previous implementation relied on the implicit registration of Bp2Build mutators, resulting in test non-hermeticity. Refactor bp2build tests to explicitly specify the bp2build mutators under test.

Test: Soong tests

Test: TH
Change-Id: I9b9674bad1ea533b3bd31b07077a9e02c99b4c1d
2021-01-26 21:57:27 -05:00
Jingwen Chen
885ee7ad86 bp2build: support genrule $(location) and $(locations)
Soong genrules support $(location) and $(locations) cmd variable
shortcuts without labels. The shortcut is for the location of the first
tool module from the concatenation of the tools and tool_files
properties.

Bazel doesn't support this shortcut, so the bp2build mapping needs to
support it.

Documentation:
https://cs.android.com/android/platform/superproject/+/master:build/soong/genrule/genrule.go;l=95-96;drc=316e07c593ab66599c74725cf482dedbb32b2875
Code: https://cs.android.com/android/platform/superproject/+/master:build/soong/genrule/genrule.go;l=236-246;drc=316e07c593ab66599c74725cf482dedbb32b2875

Test: build_conversion_test.go
Test: TH

Change-Id: I8aa98ae460af3a3fbb0a7835572518680dc7ade1
2021-01-26 09:15:01 -05:00
Jingwen Chen
316e07c593 bp2build: automatically convert all genrules.
Results of `cquery //bionic/...`: https://paste.googleplex.com/5608885982330880?raw

Note that this CL does not contain the global module name mapping to
fully qualified label lookup mechanism, so while the BUILD targets are
queryable, they are not buildable (yet).

Test: GENERATE_BAZEL_FILES=true m nothing && ./build/bazel/scripts/bp2build-sync.sh write && bazel cquery //bionic/...

Change-Id: I6f2316222723d8b612d9b041ca82a188cd05bac9
2021-01-24 20:51:18 -05:00
Colin Cross
97a1be6e7b Merge changes from topic "sbox_tools"
* changes:
  Sandbox genrule tools
  Call ctx.InstallFile for uninstallable cc modules
  Don't copy uninstallable variants of NDK libraries to sysroot
2020-12-17 22:01:06 +00:00
Colin Cross
ba9e403703 Sandbox genrule tools
This relands I38393900677c5dbe8e72fe06a7dd7d71f3c09f82 after
I8038ed5c6f05c989ac21ec06c4552fb3136b9a7a, which makes the ASAN
variants of libraries have PackagingSpecs so they can be copied
into sandboxed genrules.

Copy tools used by genrules into the sandbox directory.  This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.

Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I57c0d5fc8bba216fac4deb972d0d2098593e8963
2020-12-17 10:02:56 -08:00
Treehugger Robot
1a74be780b Merge "Assert android.ApexModule interface for types having ApexModuleBase" 2020-12-17 04:14:35 +00:00
Chris Parsons
dbcb1ff469 Use aquery to declare bazel actions in the ninja file.
This effectively moves execution of Bazel actions outside of soong_build
and alongside ninja execution of the actual ninja files, whether that be
by ninja or by Bazel itself.

This almost allows for mixed builds and Bazel-as-Ninja-executor to
coexist, but requires hacks explained in b/175307058.

Test: Treehugger
Test: lunch aosp_flame && USE_BAZEL_ANALYSIS=1 m libc
Test: lunch aosp_flame && USE_BAZEL=1 USE_BAZEL_ANALYSIS=1 m libc,
though this requires a hack of the main BUILD file. See b/175307058

Change-Id: Ia2f6b0f1057e8cea3809de66d8287f13d84b510c
2020-12-15 17:39:23 -05:00
Jiyong Park
45bf82e953 Assert android.ApexModule interface for types having ApexModuleBase
Bug: 173472337
Test: m nothing
Change-Id: Idf1c6cb9fff6c18e34c4636e38a662ba4ff7d538
2020-12-15 14:31:27 +00:00
Colin Cross
897a5ad507 Merge "Pass pctx and ctx to NewRuleBuilder" 2020-12-03 20:07:30 +00:00
Evgenii Stepanov
f47c90db24 Reduce default shard size.
Current HWASan build on master fails if OUT_DIR is ~17 characters,
passes with 3. Halve defaultShardSize to avoid this.

Bug: 170327170
Test: build with long OUT_DIR

Change-Id: I04e3cae91118891c0e28d895108670dd0d373a3f
2020-12-02 18:56:57 -08:00
Colin Cross
f1a035e6be Pass pctx and ctx to NewRuleBuilder
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.

Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
2020-12-01 16:22:16 -08:00
Alex Humesky
29e3bbe8b8 Update comments in genrule.go.
Test: Noop
Change-Id: Ib341899ea0f04e059d5471ee6e5bff8ebba82583
2020-11-30 12:30:20 -05:00
Colin Cross
3ea4eb8d35 Use a unique depfile for each gensrcs command
gensrcs modules run the same command once for each input file to
produce each output file.  Each command needs its own depfile instead
of reusing one per shard.  The depfiles will be merged together
by RuleBuilder into one depfile per shard.

Test: TestGenSrcs
Change-Id: Iaf4f2cf9f5592c20e32944ddf34e0a61aff17ba8
2020-11-24 15:05:28 -08:00
Colin Cross
38214f5ac5 Merge changes I8af00c8c,Ia526ae89
* changes:
  Reland: Rewrite sbox to use a textproto manifest
  Fix gensrcs subdirs
2020-11-24 19:27:35 +00:00
Colin Cross
e5218b6be6 Merge "Annotate dependency tags for dependencies of installed files" 2020-11-24 16:01:35 +00:00
Colin Cross
e9fe2949b8 Annotate dependency tags for dependencies of installed files
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).

Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.

Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.

This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.

Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
2020-11-23 18:06:08 -08:00
Jingwen Chen
30f5aaaa77 Move bazel_module property to a common file, and add it to filegroup.
This enables prototyping against aosp/1441774 to demonstrate mixed
builds with converted BUILD files.

Test: TH Presubmit
Test: USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 build actions
Bug: 171263886

Change-Id: I7b5cd0449d043ba26a339a0ef98b562fc62e13c8
2020-11-22 22:01:44 -05:00
Colin Cross
e16ce36818 Reland: Rewrite sbox to use a textproto manifest
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.

In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
2020-11-20 15:45:04 -08:00
Colin Cross
f18859626c Fix gensrcs subdirs
The depfile was intended to be placed alongside the per-shard
output file, but was in the final directory instead.  Move
it into the per-shard directory.

Also clean up and comment how shards and shard directories
work.

Test: m framework-cppstream-protos
Test: TestGenSrcs
Change-Id: Ia526ae8997314998814ed8e35598cc2a5bc4aabd
2020-11-20 15:37:04 -08:00
Colin Cross
619b9ab260 Revert "Rewrite sbox to use a textproto manifest"
This reverts commit 151b9ff0cf.

Reason for revert: broke builds

Change-Id: I69b3b8795d5a36b4fa0debb1af2d433be3c15d6c
2020-11-20 18:44:31 +00:00
Colin Cross
151b9ff0cf Rewrite sbox to use a textproto manifest
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
2020-11-18 13:02:22 -08:00
Colin Cross
3d68051218 Move genrule on top of RuleBuilder
In preparation for more complicated sandboxing support in sbox, use
a single implementation of the sbox sandboxing by moving genrule to
use RuleBuilder's sbox support instead of creating an sbox rule
directly.

Also move genrule's input list hash support into RuleBuilder.

Test: genrule_test.go
Test: rule_builder_test.go
Change-Id: I292184d02743c7e6887ebbcd232ba565db2ab0cc
2020-11-17 11:23:45 -08:00
Colin Cross
b5ae193b8f Revert "Annotate dependency tags for dependencies of installed files"
This reverts commit 62a0cfd054.

Reason for revert: b/173475545

Change-Id: I4e834200c8e68dfa1b8144dfd1fa95ca68554980
2020-11-17 06:32:06 +00:00
Colin Cross
62a0cfd054 Annotate dependency tags for dependencies of installed files
Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.

Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.

This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.

Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: Ic22603a5c0718b5a21686672a7471f952b4d1017
2020-11-14 16:24:10 -08:00
Colin Cross
ae8600b507 Pass Config to NewTestContext instead of ctx.Register
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.

Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
2020-11-12 10:07:49 -08:00
Anton Hansson
72f1849c71 Make genrules uninstallable ETC instead of PHONY
Makes it possible for make modules to depend on genrules via
constructs like $(call intermediates-dir-for,ETC,foo)/foo.

Test: presubmit
Change-Id: Ic9a014f55ea503f1c0c8167f351ecd1407083562
2020-11-03 12:34:18 +00:00
Yifan Hong
60e0cfb5cb Add vendor-ramdisk image to Soong.
Add vendor_ramdisk_available and vendor_ramdisk attribute to
various rules. When a vendor_ramdisk variant of a module is
generated, it is installed to $OUT/vendor-ramdisk.

It is similar to a ramdisk image.
Test: m nothing -j

Change-Id: Ib2d16459f3094dbe21c3bdb7c016cb4b2bf62765
2020-10-22 10:26:26 -07:00
Chris Parsons
aa8be0515b Change bazel_module property to be of struct type instead of string
Test: Manual verification, and update to genrule_test.go
Change-Id: Ic507a35438be5261342a958f3a462a73739e2482
2020-10-14 16:22:37 -04:00
Colin Cross
405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Christopher Parsons
c8f84809ad Merge "Mixed bazel/soong build prototype for genrule" 2020-10-09 14:27:04 +00:00
Chris Parsons
f3c96efea4 Mixed bazel/soong build prototype for genrule
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.

source build/soong/bazelenv.sh to enter bazel-enabled mode.

Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
2020-10-08 22:46:23 -04:00
Colin Cross
323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00