Commit graph

129 commits

Author SHA1 Message Date
Liz Kammer
59c0eaedca Remove arch-variant from some python properties
Version and package should not vary based on architecture.

Test: m nothing
Change-Id: Id9cb655d9c25caee3b6068a261b3db36e1654467
2021-09-20 12:56:17 -04:00
Rob Seymour
f6c37cc87d Merge "Add module based host-tools snapshot" 2021-09-14 14:47:03 +00:00
Rob Seymour
925aa09ea1 Add module based host-tools snapshot
Add new module:
host_snapshot {
    name: "host-snapshot"
    deps: [
      (list of host tools)
    ],
    ...
}

Package host tools using android.PackagingBase to capture host tools and transitive packaging data.
Add JSON meta data to snapshot that allows snapshot to be installed via development/vendor_snapshot/update.py

Add support to generate a fake host snapshot of all host modules that is used to detect required
modules via development/vendor_snapshot/update.py.

Bug: 192556798
Bug: 194799048
Bug: 192896149
Test: m HOST_FAKE_SNAPSHOT_ENABLE=true host-fake-snapshot dist -- check snapshot exists in dist

Change-Id: I849c4db801cd858408f6fe6a3ce69262a23a5be9
2021-09-13 21:29:14 +00:00
Alex Márquez Pérez Muñíz Díaz Púras Thaureaux
560cb6685c python_{binary,library}{,_host} handle lib property
Have binary and library bp2build convert `libs` to
`deps` for bazel

Bug: 196083751
Test: python_{binary,library}_conversion_test.go
Test: build/bazel/ci/mixed_{libc,droid}.sh
Change-Id: I2d5f6ef2e83dd608910edb7adb2eb9a56d25293c
2021-08-27 14:10:33 +00:00
Alex Márquez Pérez Muñíz Díaz Púras Thaureaux
dc212c0b24 Add python_library_host -> py_library bp2build support
Bug: 196081770
Test: bp2build/python_library_conversion_test.go
Test: build/bazel/ci/mixed_{libc,droid}.sh
Change-Id: I4da9938eb0b039f97b83badd2269af153c7edbcc
2021-08-26 16:03:34 +00:00
Alex Márquez Pérez Muñíz Díaz Púras Thaureaux
0fc781c7c2 Add python_library -> py_library bp2build support
Bug: 196091681
Test: bp2build/python_library_conversion_test.go
Test: build/bazel/ci/mixed_{libc,droid}.sh
Change-Id: Ice87d75533c97fd9c139dc59de09a039e2713a01
2021-08-25 15:57:01 +00:00
Alex Márquez Pérez Muñíz Díaz Púras Thaureaux
d75507fff7 Fix inverted error message
The code checks to see if the `child` is not a Python library,
but then when reporting the error it uses the module name as
the dependency name and the child (dependency) as the module name

Test: Existing tests pass
Change-Id: Ied606342291312d8485a8fd2ddcc580ad24ae82f
2021-08-20 21:02:43 +00:00
Liz Kammer
2ada09a546 Don't create a new module for bp2build conversion.
A performance improvement for bp2build as Blueprint/Soong no longer have
the overhead of additional modules. The creation of these modules
results in:
* traversal of additional modules for each subsequent mutator
* synchronization over a go channel to collect newly created modules:
https://cs.android.com/android/platform/superproject/+/master:build/blueprint/context.go;l=2594,2600;drc=1602226f23181b8c3fbfcaf3358f0297e839d7d3

We avoid both of these by storing the information directly in the
underlying module.

Also as a fringe benefit, removes some necessary boilerplate for
conversion.

For benchmarks, reduces runtime ~1% for 1% converted, ~24% for 100%
converted. See more: go/benchmarks-for-https:-r.android.com-1792714

Test: ran benchmarks/tests in bp2build
Test: build/bazel/ci/bp2build.sh
Change-Id: Ie9273b8cbab5bc6edac1728067ce184382feb211
2021-08-17 15:57:09 -04:00
Julien Desprez
8620b65bf1 Merge "Switch python_test_host default to unit_test:true" 2021-05-27 21:31:37 +00:00
Julien Desprez
66534a0144 Switch python_test_host default to unit_test:true
Test: presubmit
Bug: 183730987
Change-Id: Ib54c30c751731aaae8a46af96e809c46469659d1
2021-05-27 20:49:13 +00:00
Colin Cross
0e446159bc Support blueprint_go_binary in PathForModuleSrc
Allow blueprint_go_binary to be used as test data by supporting it in
PathForModuleSrc.

Also make python use GetDirectDepsWithTag instead of VisitDirectDeps
so it doesn't fail on the dependency to a Blueprint (non-Android)
module.

Fixes: 186528269
Test: manual
Change-Id: Ibc1d7caf7bf1fa5db805c3b885a95df4baee0ecf
2021-05-03 13:35:32 -07:00
Julien Desprez
d3dbf07c37 Mark par_test as not unit tests
They seem to be used as Soong validation. They are not implemented against the python
unit tests framework, so we can execute them as such in CI automatically.

Change-Id: I8fe662d5d3399d60c8a05acc79b4dd51b5d138a6
Test: presubmit
Bug: 183730987
2021-04-20 15:56:09 +00:00
yangbill
3aa297570b Change par file generate rules.
Make sure the shebang of the par file is consistent with the main python
script rather than use /usr/bin/python arbitrarily.

Bug: 185399398
Bug: 185094069
Test: m atest ; atest-dev --help
      m atest-py2 ; atest-py2-dev --help
Change-Id: Ice4590b3d173e6d11e608c2ca29b297df0dc4089
2021-04-19 08:08:34 +00:00
Jingwen Chen
0702791a99 bp2build: arch-configurable selects for label list attrs.
This CL adds the configurable LabelListAttribute support to bp2build.

Test: go test
Change-Id: I2ef9e385d9cf1b1845988128eca1d8cda1ecb5e8
2021-03-30 13:22:48 +00:00
Jingwen Chen
12b4c2706d bp2build: add allowlist for package-level conversions.
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.

It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.

Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.

Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
2021-03-24 02:27:19 -04:00
Paul Duffin
89648f98fa Remove usages of FixtureFactory from misc packages
These packages have already been migrated to use per test build
directory so have no need for a FixtureFactory.

Bug: 183235980
Test: m nothing
Change-Id: I667d1d992caaf0f615de91f89efdae11c44986c2
2021-03-22 18:31:53 +00:00
Paul Duffin
803876aaa6 Convert python package to fixtures
Bug: 182885307
Test: m nothing
Change-Id: I78506a10f0e6ba8ad4952e8305f47ffd5a33b359
2021-03-17 23:56:56 +00:00
Paul Duffin
d0890452e3 Add preparers for python
Bug: 181070625
Test: m nothing
Change-Id: I590ad62db6fd62719c83286780563cf6d55e3e17
2021-03-17 23:56:29 +00:00
Paul Duffin
c60dd805ad Fix hole in python testing code that ignored a broken test
The test running code would only check errors if there were any
reported. That ignored the "module with duplicate runfile path" test
which was not reporting the expected error.

This change corrects that mistake, and then fixes the test which had
started to fail.

It is possible that the failing test highlights a bug in the python
code but if so that will need to be fixed in a follow up bug.

Bug: 183046274
Test: m nothing
Change-Id: I961b15a452725e7b15ba9d21b4a4a672ba3d004a
2021-03-17 23:56:23 +00:00
Jingwen Chen
13b9b42f8f bp2build: add python_binary_host converter.
This CL adds a simple python_binary_host to native py_binary converter
for standalone modules that don't depend on libs. It also adds support
for the conditional py2/py3 build target based on the version prop.

Test: milestone-2/demo.sh full
Test: bazel query 'kind("py_binary rule", //bionic/...)'
Test: bazel run //bionic/libc/tools:genfunctosyscallnrs
Test: bazel run //bionic/libc/tools:genseccomp
Test: go tests
Fixes: 182236395
Change-Id: Ibe5ec6cd0dc12a61b3a449a8c723d80b891fae42
2021-03-09 18:58:42 -05:00
Treehugger Robot
277303f042 Merge "Add ctx to AndroidMkExtraEntriesFunc" 2021-02-22 22:40:31 +00:00
Colin Cross
aa2555387d Add ctx to AndroidMkExtraEntriesFunc
Add a ctx parameter to AndroidMkExtraEntriesFunc to allow them to
access providers.

Test: m checkbuild
Change-Id: Id6becc1e425c3c3d8519248f8c0ce80777fac7cc
Merged-In: Id6becc1e425c3c3d8519248f8c0ce80777fac7cc
2021-02-19 23:05:40 +00:00
Bob Badour
02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Treehugger Robot
2859d737f0 Merge "Add comments/refactor for python.go" 2020-12-04 03:45:21 +00:00
Colin Cross
f1a035e6be Pass pctx and ctx to NewRuleBuilder
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.

Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
2020-12-01 16:22:16 -08:00
Liz Kammer
d737d02f16 Add comments/refactor for python.go
Test: all soong tests
Test: m nothing
Change-Id: Ib3b95d7c2831b97026e76a39af515fd51c6cb2c7
2020-12-01 13:18:47 -08:00
Liz Kammer
57f5b33ad3 Add test suite handling to central androidmk code
MTS is introducing partial MTS test suites that are per-module, with
names of the format: mts-${MODULE}. By centralizing the code for test
suites, we can automatically add "mts" test suite when an
"mts-${MODULE}" test suite is specified, reducing duplication.

Test: m mts
Bug: 170318013
Change-Id: I8ce9d3c252fcc0a937bb5f2826d21cb6c6932d82
2020-12-01 12:23:56 -08:00
Liz Kammer
c55dcfcc2b Merge "Refactor python to use AndroidMkEntries." 2020-11-26 00:05:52 +00:00
Liz Kammer
d8dceb0f2d Refactor python to use AndroidMkEntries.
This is to enable consolidating test compatibility suite handling in
Soong's AndroidMk files.

Test: go test soong tests
Test: m nothing and compare soong/Android mk files -- only diffs is that
      lines matching "^LOCAL_SHARED_LIBRARIES := $" are absent
Change-Id: I4922a1c8b0231cd98ebd2bc7257956eb9c0d240c
2020-11-24 10:10:03 -08:00
Colin Cross
e5218b6be6 Merge "Annotate dependency tags for dependencies of installed files" 2020-11-24 16:01:35 +00:00
Colin Cross
e9fe2949b8 Annotate dependency tags for dependencies of installed files
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).

Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.

Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.

This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.

Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
2020-11-23 18:06:08 -08:00
Colin Cross
1bc63938f0 Add java_data parameter to python modules
csuite has a python module that wants to embed the outputs of java
modules.  This has caused issues with mismatched variants bewteen
the arch-specific python module and the common java modules.  Add
a java_data property that is similar to the data property but
uses the common arch variant.

Bug: 173977903
Test: m checkbuild
Change-Id: I0f2f0e4159650cd5d42b510d5177678e7ee91b4d
2020-11-23 18:35:55 +00:00
Colin Cross
e20113d8ab Use local variations for python version splits
Use a local variation for python version splits.  This causes
dependencies from python modules not to look for variations
that match the version split, which will simplify a python module
depending on a java module (for example for test data) because
Soong won't look for a python version variant of the java module.

Bug: 173977903
Test: go test ./python
Change-Id: Ib034140c478ffbc7467ab830d3cfa5683c31d05c
2020-11-23 18:35:49 +00:00
Dan Shi
ebb51503a4 Merge "Add unit_test test option in test configs" 2020-11-18 04:40:37 +00:00
Treehugger Robot
bc1f77e590 Merge "Fix bug in python module hasExtention." 2020-11-16 22:56:52 +00:00
Liz Kammer
3dfd3ce478 Fix bug in python module hasExtention.
Test: m nothing
Test: soong go tests
Change-Id: I1944a3bbb871bc1021c841d944e765bb0cadba86
2020-11-16 11:30:55 -08:00
Dan Shi
d79572f73e Add unit_test test option in test configs
This change allows a test (native, java, rust or python) to be included
in host-unit-tests suite when test option `unit_test` is set to true.

Bug: 172006742
Test: m host-unit-tests
Change-Id: I69d3eb5b51198c549e2e6914ceac3a4fc33c3cf2
2020-11-16 11:05:00 -08:00
Colin Cross
ae8600b507 Pass Config to NewTestContext instead of ctx.Register
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.

Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
2020-11-12 10:07:49 -08:00
Liz Kammer
7e93e5b001 Consolidate python version properties in mutator
Test: m
Bug: 165114590
Change-Id: Iaf874f7098317a52827b113167e3dc1ed044e907
2020-11-02 11:30:39 -08:00
Colin Cross
405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Colin Cross
323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Treehugger Robot
025ae97b0e Merge "Add support of test data to python_host_test" 2020-09-29 21:56:19 +00:00
Jiyong Park
1613e5541f HostCross is an attribute of a Target, not OsType
A host target is considered as being cross-compiled when the target
can't run natively on the build machine. For example, linux_glibc/x86_64
is a non-cross target on a standard x86/Linux machine, but is a cross
host on Mac. Previously, whether cross or not was a static attribute of
an OsType. For example, Windows was always considered as cross host,
while linux_bionic was not. This becomes a problem when we support more
host targets like linux_bionic/arm64 which should be cross-host on
standard x86/Linux machines.

This change removes HostCross from the OsClass type and instead adds a
property HostCross to the Target type. When a target is being added, it
is initialized to true when the target can't run natively on the current
build machine.

Bug: 168086242
Test: m
Change-Id: Ic37c8db918873ddf324c86b12b5412952b0f2be2
2020-09-24 14:14:31 +09:00
Dan Shi
3194912150 Add support of test data to python_host_test
Bug: none
Test: m -j vts_vndk_abi_test
confirm abi_dump.zip found next to test binary

Change-Id: I834dddfc13eb9e7addd234307b085b16124db234
2020-09-21 14:36:43 -07:00
Liz Kammer
dd849a81f3 Add data_native_bins property to java_test_host
When multiple os/arch variants are supported, java_test_host could not
find a matching arch due to java having arch:common, whereas native
binaries support a specific architecture. This change adds the property
`data_native_bins` in order to support binaries with the appropriate
os/arch variants.

Test: m FirmwareDtboVerification with data_native_bins
Test: forrest
Bug: 153848038
Change-Id: I45adebff0fde2811d5ef5620c697b97b768c951f
2020-07-27 09:14:06 -07:00
Liz Kammer
e0070ee313 Implement OutputFiles for python modules.
This enables python binaries to be included in data properties with the
format ":module".

Test: m FirmwareDtboVerification with local changes
Bug: 153848038
Change-Id: I2a02dc382fcbb142d8e4fbdbb6ca461986627d25
2020-06-22 15:51:39 -07:00
Colin Cross
a6384821f9 Fix using generated test configs for sh_test, python_test and rust_test modules
Annotate the test config properties with android:"path", and always
use LOCAL_FULL_TEST_CONFIG to pass the path to Make.

Bug: 157621988
Test: manual
Change-Id: I1c020e642f83d994d63c9e9cb56e686086d5acaa
2020-06-10 22:48:16 -07:00
Jaewoong Jung
4b79e98a6e Soong package structure refactoring
Give prebuilt_etc and sh_binary their own packages and split the
gigantic main Android.bp up to small, per-package ones.

Test: m nothing, TreeHugger
Bug: 156980228
Change-Id: I7b00cd344b9f16861f1ff39edf0029f016b853d0
2020-06-01 13:44:48 -07:00
Dan Willemsen
bc32a50ff4 Don't check elf properties of python prebuilts
We're not doing a great job of copying over all the library dependencies (especially when using sanitizer libraries). Just omit these from the checks, which shouldn't really matter.

Bug: 151093572
Change-Id: I8470bbfbf36a51739e8a7e699ad090a66c9db8b2
Test: treehugger
2020-03-11 20:55:17 +00:00
Dan Willemsen
7fa7646ad7 Add tests for py(2|3)-cmd
Test: m par_test{,3} py{2,3}-cmd; ./runtest.sh
Change-Id: I4aadd9fc85be4381bd9a0843064ceaa87150b5ee
2020-02-19 19:40:38 -08:00