Commit graph

74 commits

Author SHA1 Message Date
Inseob Kim
c7c4746151 Add buildinfo_prop module
buildinfo_prop module is a replacement for build/make/tools/buildinfo.sh
so other images like microdroid can refer to build.prop.

For now, buildinfo_prop only supports a few build.prop properties, and
it's only used in microdroid.

Bug: 189164487
Test: build
Change-Id: I120654ca23a68de414df8da2051c6677afbab441
Merged-In: I120654ca23a68de414df8da2051c6677afbab441
(cherry picked from commit 4f1f3d97ca)
2022-05-02 09:51:45 +09:00
Sam Delmerico
24c5603815 refactor bp2build module allowlists into struct
To make testing easier, refactor existing module-global variables into a
struct that can be mocked.

Test: build/bazel/bp2build.go
Change-Id: I9d177677644ea743641a745b1839a3a8b29f902a
2022-04-14 13:31:05 +00:00
Sam Delmerico
7f88956c16 refactor Bazel variable export
Most of the variable export code for cc modules can be re-used for
exporting variables for java modules. Refactor this code into a more
composable structure for reuse.

Test: build/bazel/bp2build.sh
Test: manual comparison of
  out/soong/soong_injection/cc_toolchain/constants.bzl
  with previous output
Change-Id: Ie5a6fee08cc888b7dc69c3e324e5c3f8aa269a8f
2022-04-06 14:48:32 +00:00
Chris Parsons
715b08fd82 Add metrics collection to soong_build
Metrics are added to a materialized protobuf soong_build_metrics.pb

Recorded events have scoped names and recorded runtimes.
For example, mixedbuilds includes the following events:

mixedbuilds.prepare
mixedbuilds.analyze.write_files
mixedbuild     (this is a total of all mixedbuild events)

Test: Manually verified materialized protos for mixedbuild, bp2build,
and legacy build.

Change-Id: I2e72d4502c0373bee0279bb1320b2e4dd586129c
2022-03-25 17:19:44 -04:00
Liz Kammer
72beb34609 Add package for printing starlark formatted data
Bug: 216168792
Test: build/bazel/ci/bp2build.sh
Change-Id: I3a06b19396f7ffe1c638042cda7e731dd840f1d6
2022-02-08 17:32:28 -05:00
Colin Cross
4acaea9cbf Revert^2 "Build license metadata files in Soong"
b3264f2090

Change-Id: I5ea5f7e29eb1dbc72b8b30f010006a25f5f72fac
2021-12-10 23:05:02 +00:00
Colin Cross
b3264f2090 Revert "Build license metadata files in Soong"
Revert submission 1899703

Reason for revert: b/210158272
Reverted Changes:
I142c19284:Reuse license metadata files from Soong
Ife89b8f23:Add dependency license annotations
I8e7410837:Build license metadata files in Soong

Change-Id: I48f2bf2e2797ac9d38f7ece35ba616b823035607
2021-12-10 23:02:57 +00:00
Colin Cross
3dd2ff28ed Build license metadata files in Soong
Soong has enough information to build the license metadata files
without resorting to the fixups required in Make.

Bug: 207445310
Test: m checkbuild
Change-Id: I8e74108376162b8fdb87ba098ebe94350aa1f7c4
2021-12-08 15:08:59 -08:00
satayev
0ee2f913ef Test SdkSpecForm.
Bug: 190818041
Test: presubmit
Change-Id: Ib8cd891f03537712d709ed063dd76dee55221118
2021-12-01 17:43:00 +00:00
Colin Cross
6ac9576eef Add tests for ctx.InstallFile
Add tests that cover Soong-only installation as well as installation with
InstallBypassMake.

Bug: 204136549
Test: TestInstall
Test: TestInstallBypassMake
Change-Id: Iac22c9fdf99994e06b419623ee5fa399ef6957fb
2021-11-09 17:25:57 -08:00
Paul Duffin
581f2e5f79 Detect duplicates in sdkRegistry
Bug: 195754365
Test: m nothing
Change-Id: I67c5022b7cc61891fd6b90365f8271d97d7bcd98
2021-09-23 11:37:57 +01:00
Lukacs T. Berki
c6012f36e1 Invoke soong_docs from the bootstrap Ninja file.
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.

It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.

The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.

Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
2021-09-08 09:31:52 +02:00
Lukacs T. Berki
3a82169a55 Invoke queryview from the bootstrap Ninja file.
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .

Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
2021-09-06 17:10:49 +02:00
Dan Willemsen
4591b6496d Upgrade to golang protobuf api v2
Major operations:
 * Fix the go_package entries to be consistent, as the new tool
   complains when it doesn't have a '/'.
 * Regenerate with the new protoc-gen-go tool.
 * github.com/golang/protobuf -> google.golang.org/protobuf
 * proto.[Un]MarshalText -> prototext.[Un]Marshal

Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
2021-07-23 11:12:05 -07:00
Paul Duffin
b0bb376efa Add license modules to the sdk
Adds initial support for adding license modules to the sdk, along with
any referenced license text files. There is a number of minor
improvements to be made but the core of the support is there and it
works for ART sdks.

Basically, this change will automatically add license modules
referenced from any sdk member module as an internal sdk member. An
internal module has an sdk snapshot specific name that will not
conflict with the source module and which cannot be referenced from
outside the sdk snapshot.

Bug: 181569894
Test: m art-module-sdk art-module-host-exports art-module-test-exports
      - diff output before and after this change
        made sure that every prebuilt had a licenses field
        and that the license modules were added
Change-Id: I0c5ccabf58f4ef487e42ef8e61a5b2a74c0e81af
2021-05-11 08:28:49 +01:00
Rupert Shuttleworth
b7e30760af Add various test files as testSrcs, so that they run on CI.
Test: go test
Test: TH (now that these are hooked up to TH)

Change-Id: Ia199794ef1a64074f3f9159f8c160ba3e45fd181
2021-04-21 11:23:57 -04:00
Liz Kammer
620dea6720 Split bazel -path functions and clarify docs
Test: go test
Change-Id: I62f58998fc7d52c67ed5acfdb8230d404b2a5472
2021-04-15 13:00:42 -04:00
Lukacs T. Berki
53b2f36bd6 Delete android/env.go .
Properly plumb the available environment to the configuration instead of
going through the global variable originalEnv.

Test: Presubmit.
Change-Id: Ia1ea753d5e72c02a0dcaa4c0b43bd6e24fc47cec
2021-04-13 08:10:01 +02:00
Jiyong Park
f1691d2a2c Move java.sdkSpec to the android package
... in preparation for making the handling of sdk versions consistent
across java and cc modules.

Bug: 175678607
Test: m
Change-Id: I598f0454bce9b7320621022115412fbe97403945
2021-04-03 08:25:12 +09:00
Colin Cross
e55bd423df Pass rsp files into sbox and rewrapper
The current implementation causes inputs listed in an rsp file used with
sbox to be duplicated 3 times in the build.ninja file; once as a
dependency of the rule, once in the rspfile_content field of the rule
with the paths rewritten to be relative to the sandbox, and once in the
rule to write the sbox manifest.  When RBE is enabled it also gets a
fourth copy in the list of files to be treated as inputs by rewrapper.

Reduce this to a single copy by using "$in" for the rspfile_content so
that the files only have to be listed in the input dependencies of the
rule, and then add support to sbox to rewrite the rsp file while copying
it into the sandbox, and pass it to rewrapper as well.

Test: m lint-check
Change-Id: I3f46f61119508d39a8bb231c99fc130153fb6f04
2021-03-25 11:06:45 -07:00
Jingwen Chen
12b4c2706d bp2build: add allowlist for package-level conversions.
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.

It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.

Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.

Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
2021-03-24 02:27:19 -04:00
Colin Cross
77cdcfdeaf Move android package on top of remotexec
Remove the references to the android package in remotexec so that
the android package can reference the remoteexec package.  This
will allow RuleBuilder to integrate directly with remoteexec.

Bug: 182612695
Test: m checkbuild
Change-Id: I15be5ef126d8aacbd605518638f341daf6f31bb3
2021-03-18 16:17:34 -07:00
Christopher Parsons
bc3f7e0276 Merge "Refactor and cleanup of cquery processing" 2021-03-16 21:12:01 +00:00
Chris Parsons
944e7d01aa Refactor and cleanup of cquery processing
Test: USE_BAZEL_ANALYSIS=1 m libc
Change-Id: Iaf9a92e84d39c132e2444a8aaafd79505a12b8ec
2021-03-16 14:39:16 -04:00
Paul Duffin
d7aea1dbe5 Remove TestHelper
Bug: 181070625
Test: m nothing
Change-Id: Idc39c8621424e25c1b4310c72ed2943d34a66bbc
2021-03-13 16:02:30 +00:00
Paul Duffin
9dcf253197 Add new function Assert methods
Bug: 181070625
Test: m nothing
Change-Id: If7977493d4f7380e746df1b9b8b4cb13c39abeac
2021-03-12 14:25:33 +00:00
Paul Duffin
0550055007 Separate TestHelper from fixtures
Although they are intended to be used together they are separate pieces
of functionality so should be separate.

Bug: 181070625
Test: m nothing
Change-Id: I3b2ccf66748a3c53b63c112256a8eb20e0448179
2021-03-12 14:25:33 +00:00
Lukács T. Berki
b821b9b852 Merge "Move environment staleness check to soong_ui." 2021-03-02 09:44:18 +00:00
Lukacs T. Berki
3243aa51eb Move environment staleness check to soong_ui.
Also delete the now-unnecessary soong_env binary.

Test: Manually. Also checked what happens when a used environment
variable changes.

Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
2021-03-02 09:32:49 +01:00
Paul Duffin
358161232c Add test fixture support
Adds the test fixture support and converts a few tests to exercise the
code and show how it works.

Bug: 181070625
Test: m nothing
Change-Id: I0a2b40fff93b6041f9aa8c4ef0aba91da1bc8bf3
2021-02-26 23:01:50 +00:00
Liz Kammer
ea6666fba9 Move bazel.Properties to a BazelModuleBase
This allows more direct access to the bazel label and whether the module
is bp2build available.

Test: go test *
Change-Id: I23375835d20fa53d7d94127b2dc2d5bb20487bfb
2021-02-23 10:58:57 -05:00
Bob Badour
02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Paul Duffin
064b70c918 Export dex implementation jars from prebuilt_apex
Dexpreopt and boot jars package check all require access to dex
implementation jars created for java_library and java_sdk_library. They
were available when building from source but not when building from
prebuilts, even though they are embedded within the .apex files that
are referenced from prebuilt_apex.

This changes adds support to prebuilt_apex to export the dex
implementation jars and updates java_import to use those exported dex
implementation jars.

In a source build dexpreopt/boot jars package check access the apex (or
platform) specific variant of a java_library, e.g. core-oj, from which
it retrieves the dex implementation jar path.

After this change in a prebuilt build dexpreopt/boot jars package check
behave in the same way except in this case they retrieve the dex
implementation jar path from the apex (or platform) specific variant of
the java_import, e.g. core-oj.

The work to export files from a `.apex` file for use by other modules
is performed by a new `deapexer` module type. It is not used directly
in an `Android.bp` file but instead is created implicitly by
`prebuilt_apex`,

In order to do that this contains the following changes:
* Adds a new `dexapexer` module type to handle the exporting of files
  from the `.apex` file.
* Adds an exported_java_libs property to prebuilt_apex to specify the
  set of libraries whose dex implementation jars need exporting.
* Creates apex specific variants of the libraries listed in the
  exported_java_libs property.
* Adds the set of exported files to the ApexInfo to make them available
  to the apex specific variants.
* Prevents the prebuilt_apex variants from being merged together as
  they will not be compatible.
* Modifies java_import to use the exported file for variants of a
  prebuilt_apex.
* Adds a ninja rule to unpack (using deapexer) the contents of the
  prebuilt_apex's apex file, verify that the required files are present
  and make them available as outputs for other rules to use.
* Some minor refactorings to support these changes.
* Adds tests to cover prebuilt only, prebuilt with source preferred,
  and prebuilt preferred with source.

Test: m nothing
Bug: 171061220
Change-Id: Ic9bed81fb65b92f0d59f64c0bce168a9ed44cfac
2021-01-15 18:14:10 +00:00
Colin Cross
9aed5bc715 Add a new SingletonModule type
A SingletonModule is halfway between a Singleton and a Module.  It has
access to visiting other modules via its GenerateSingletonBuildActions
method, but must be defined in an Android.bp file and can also be
depended on like a module.

Bug: 176904285
Test: singleton_module_test.go
Change-Id: I1b2bfdfb3927c1eabf431c53213cb7c581e33ca4
2021-01-07 17:19:27 -08:00
Bob Badour
37af046002 Revert^2 "Add ability to declare licenses in soong."
df98d3e4a5

Change-Id: Ie7e1695ecb8cd943124426a0cad2c0d1db2b46ed
2021-01-06 20:48:24 -08:00
Jerome Gaillard
df98d3e4a5 Revert "Add ability to declare licenses in soong."
Revert submission 1377717-metalics

Reason for revert: This has broken renderscript_mac target for aosp-master, see b/176909442

Reverted Changes:
I26ac54ca9:Define the standard license_kind rules.
I656486070:Export soong license data to make.
If9d661dfc:Export soong license data to make.
I97943de53:Add ability to declare licenses in soong.
Icaff40171:Rough-in license metadata support to make.
Ib8e538bd0:Add variables for notice deps, license kinds etc.

Change-Id: Iddf5468e0175831ddb642011f2768f87a5e4fcf2
2021-01-06 19:00:05 +00:00
Bob Badour
8a36d94714 Add ability to declare licenses in soong.
See: http://go/android-license-checking-in-soong-v2-design

Bug: 151953481
Bug: 151177513
Bug: 67772237

Change-Id: I97943de53b071cd9918679c17015ed3397c172e9
2021-01-05 08:42:48 -08:00
Colin Cross
96c44127d1 Reimplement DepSet as a wrapper around a generic implementation
Implement depSet as a generic depsets implementation using reflection,
and then make DepSet a type-safe wrapper around it.  This will allow
additional wrappers for depsets that work with other types.  All of
this can be replaced with generics once Go supports them.

Test: depset_test.go
Change-Id: Id9df17bcc76f6c1545e7eb498f298066cf8a7679
2020-12-07 12:27:50 -08:00
Colin Cross
38214f5ac5 Merge changes I8af00c8c,Ia526ae89
* changes:
  Reland: Rewrite sbox to use a textproto manifest
  Fix gensrcs subdirs
2020-11-24 19:27:35 +00:00
Colin Cross
e5218b6be6 Merge "Annotate dependency tags for dependencies of installed files" 2020-11-24 16:01:35 +00:00
Colin Cross
e9fe2949b8 Annotate dependency tags for dependencies of installed files
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).

Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.

Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.

This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.

Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
2020-11-23 18:06:08 -08:00
Jingwen Chen
30f5aaaa77 Move bazel_module property to a common file, and add it to filegroup.
This enables prototyping against aosp/1441774 to demonstrate mixed
builds with converted BUILD files.

Test: TH Presubmit
Test: USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 build actions
Bug: 171263886

Change-Id: I7b5cd0449d043ba26a339a0ef98b562fc62e13c8
2020-11-22 22:01:44 -05:00
Colin Cross
e16ce36818 Reland: Rewrite sbox to use a textproto manifest
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.

In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
2020-11-20 15:45:04 -08:00
Colin Cross
b893f8766c Merge "Revert "Rewrite sbox to use a textproto manifest"" 2020-11-20 18:45:35 +00:00
Colin Cross
619b9ab260 Revert "Rewrite sbox to use a textproto manifest"
This reverts commit 151b9ff0cf.

Reason for revert: broke builds

Change-Id: I69b3b8795d5a36b4fa0debb1af2d433be3c15d6c
2020-11-20 18:44:31 +00:00
Colin Cross
60d06cf8df Merge "Rewrite sbox to use a textproto manifest" 2020-11-20 17:43:29 +00:00
Colin Cross
a684540945 Add more comments to arch.go
Make sure every exported function or type has a godoc comment.

Also makes minor changes like unexporting functions that are not used
outside the package and fixing minor style warnings.

Bug: 173449605
Test: m checkbuild
Change-Id: I533a595d02035aae8b2b603590be639826d2d4c8
2020-11-19 11:20:05 -08:00
Treehugger Robot
7a64f7e5b6 Merge changes from topic "package"
* changes:
  add android_filesystem
  Introduce PackagingBase
2020-11-19 07:49:00 +00:00
Jiyong Park
dda8f69e43 Introduce PackagingBase
PackagingBase is the base struct that implements the basic packaging
functionalities. Here, packaging means grouping dependencies into an
output file where the built artifacts from the dependencies are placed
in a directory-like structure in the output file. The exact format of
the output file is irrelevant; it could be a filesystem image, tar.gz,
zip, or whatever.

PackagingBase is responsible for traversing the dependencies and copying
their outputs under the package root directory, which is expected to be
provided by the module type that includes PackagingBase. Then the
concrete module type is expected to do final step of converting the
package root directory into the output file of specific format.

Bug: 159685774
Bug: 172414391
Test: m dist out/dist/cvd-host_package.tar.gz
Change-Id: I5446eee4834ce3b6f0f5843d93bb330a26d42fe3
2020-11-19 08:58:06 +09:00
Colin Cross
151b9ff0cf Rewrite sbox to use a textproto manifest
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
2020-11-18 13:02:22 -08:00