Commit graph

211 commits

Author SHA1 Message Date
Paul Duffin
c60dd805ad Fix hole in python testing code that ignored a broken test
The test running code would only check errors if there were any
reported. That ignored the "module with duplicate runfile path" test
which was not reporting the expected error.

This change corrects that mistake, and then fixes the test which had
started to fail.

It is possible that the failing test highlights a bug in the python
code but if so that will need to be fixed in a follow up bug.

Bug: 183046274
Test: m nothing
Change-Id: I961b15a452725e7b15ba9d21b4a4a672ba3d004a
2021-03-17 23:56:23 +00:00
Jingwen Chen
13b9b42f8f bp2build: add python_binary_host converter.
This CL adds a simple python_binary_host to native py_binary converter
for standalone modules that don't depend on libs. It also adds support
for the conditional py2/py3 build target based on the version prop.

Test: milestone-2/demo.sh full
Test: bazel query 'kind("py_binary rule", //bionic/...)'
Test: bazel run //bionic/libc/tools:genfunctosyscallnrs
Test: bazel run //bionic/libc/tools:genseccomp
Test: go tests
Fixes: 182236395
Change-Id: Ibe5ec6cd0dc12a61b3a449a8c723d80b891fae42
2021-03-09 18:58:42 -05:00
Treehugger Robot
277303f042 Merge "Add ctx to AndroidMkExtraEntriesFunc" 2021-02-22 22:40:31 +00:00
Colin Cross
aa2555387d Add ctx to AndroidMkExtraEntriesFunc
Add a ctx parameter to AndroidMkExtraEntriesFunc to allow them to
access providers.

Test: m checkbuild
Change-Id: Id6becc1e425c3c3d8519248f8c0ce80777fac7cc
Merged-In: Id6becc1e425c3c3d8519248f8c0ce80777fac7cc
2021-02-19 23:05:40 +00:00
Bob Badour
02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Treehugger Robot
2859d737f0 Merge "Add comments/refactor for python.go" 2020-12-04 03:45:21 +00:00
Colin Cross
f1a035e6be Pass pctx and ctx to NewRuleBuilder
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.

Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
2020-12-01 16:22:16 -08:00
Liz Kammer
d737d02f16 Add comments/refactor for python.go
Test: all soong tests
Test: m nothing
Change-Id: Ib3b95d7c2831b97026e76a39af515fd51c6cb2c7
2020-12-01 13:18:47 -08:00
Liz Kammer
57f5b33ad3 Add test suite handling to central androidmk code
MTS is introducing partial MTS test suites that are per-module, with
names of the format: mts-${MODULE}. By centralizing the code for test
suites, we can automatically add "mts" test suite when an
"mts-${MODULE}" test suite is specified, reducing duplication.

Test: m mts
Bug: 170318013
Change-Id: I8ce9d3c252fcc0a937bb5f2826d21cb6c6932d82
2020-12-01 12:23:56 -08:00
Liz Kammer
c55dcfcc2b Merge "Refactor python to use AndroidMkEntries." 2020-11-26 00:05:52 +00:00
Liz Kammer
d8dceb0f2d Refactor python to use AndroidMkEntries.
This is to enable consolidating test compatibility suite handling in
Soong's AndroidMk files.

Test: go test soong tests
Test: m nothing and compare soong/Android mk files -- only diffs is that
      lines matching "^LOCAL_SHARED_LIBRARIES := $" are absent
Change-Id: I4922a1c8b0231cd98ebd2bc7257956eb9c0d240c
2020-11-24 10:10:03 -08:00
Colin Cross
e5218b6be6 Merge "Annotate dependency tags for dependencies of installed files" 2020-11-24 16:01:35 +00:00
Colin Cross
e9fe2949b8 Annotate dependency tags for dependencies of installed files
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).

Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.

Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.

This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.

Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
2020-11-23 18:06:08 -08:00
Colin Cross
1bc63938f0 Add java_data parameter to python modules
csuite has a python module that wants to embed the outputs of java
modules.  This has caused issues with mismatched variants bewteen
the arch-specific python module and the common java modules.  Add
a java_data property that is similar to the data property but
uses the common arch variant.

Bug: 173977903
Test: m checkbuild
Change-Id: I0f2f0e4159650cd5d42b510d5177678e7ee91b4d
2020-11-23 18:35:55 +00:00
Colin Cross
e20113d8ab Use local variations for python version splits
Use a local variation for python version splits.  This causes
dependencies from python modules not to look for variations
that match the version split, which will simplify a python module
depending on a java module (for example for test data) because
Soong won't look for a python version variant of the java module.

Bug: 173977903
Test: go test ./python
Change-Id: Ib034140c478ffbc7467ab830d3cfa5683c31d05c
2020-11-23 18:35:49 +00:00
Dan Shi
ebb51503a4 Merge "Add unit_test test option in test configs" 2020-11-18 04:40:37 +00:00
Treehugger Robot
bc1f77e590 Merge "Fix bug in python module hasExtention." 2020-11-16 22:56:52 +00:00
Liz Kammer
3dfd3ce478 Fix bug in python module hasExtention.
Test: m nothing
Test: soong go tests
Change-Id: I1944a3bbb871bc1021c841d944e765bb0cadba86
2020-11-16 11:30:55 -08:00
Dan Shi
d79572f73e Add unit_test test option in test configs
This change allows a test (native, java, rust or python) to be included
in host-unit-tests suite when test option `unit_test` is set to true.

Bug: 172006742
Test: m host-unit-tests
Change-Id: I69d3eb5b51198c549e2e6914ceac3a4fc33c3cf2
2020-11-16 11:05:00 -08:00
Colin Cross
ae8600b507 Pass Config to NewTestContext instead of ctx.Register
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.

Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
2020-11-12 10:07:49 -08:00
Liz Kammer
7e93e5b001 Consolidate python version properties in mutator
Test: m
Bug: 165114590
Change-Id: Iaf874f7098317a52827b113167e3dc1ed044e907
2020-11-02 11:30:39 -08:00
Colin Cross
405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Colin Cross
323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Treehugger Robot
025ae97b0e Merge "Add support of test data to python_host_test" 2020-09-29 21:56:19 +00:00
Jiyong Park
1613e5541f HostCross is an attribute of a Target, not OsType
A host target is considered as being cross-compiled when the target
can't run natively on the build machine. For example, linux_glibc/x86_64
is a non-cross target on a standard x86/Linux machine, but is a cross
host on Mac. Previously, whether cross or not was a static attribute of
an OsType. For example, Windows was always considered as cross host,
while linux_bionic was not. This becomes a problem when we support more
host targets like linux_bionic/arm64 which should be cross-host on
standard x86/Linux machines.

This change removes HostCross from the OsClass type and instead adds a
property HostCross to the Target type. When a target is being added, it
is initialized to true when the target can't run natively on the current
build machine.

Bug: 168086242
Test: m
Change-Id: Ic37c8db918873ddf324c86b12b5412952b0f2be2
2020-09-24 14:14:31 +09:00
Dan Shi
3194912150 Add support of test data to python_host_test
Bug: none
Test: m -j vts_vndk_abi_test
confirm abi_dump.zip found next to test binary

Change-Id: I834dddfc13eb9e7addd234307b085b16124db234
2020-09-21 14:36:43 -07:00
Liz Kammer
dd849a81f3 Add data_native_bins property to java_test_host
When multiple os/arch variants are supported, java_test_host could not
find a matching arch due to java having arch:common, whereas native
binaries support a specific architecture. This change adds the property
`data_native_bins` in order to support binaries with the appropriate
os/arch variants.

Test: m FirmwareDtboVerification with data_native_bins
Test: forrest
Bug: 153848038
Change-Id: I45adebff0fde2811d5ef5620c697b97b768c951f
2020-07-27 09:14:06 -07:00
Liz Kammer
e0070ee313 Implement OutputFiles for python modules.
This enables python binaries to be included in data properties with the
format ":module".

Test: m FirmwareDtboVerification with local changes
Bug: 153848038
Change-Id: I2a02dc382fcbb142d8e4fbdbb6ca461986627d25
2020-06-22 15:51:39 -07:00
Colin Cross
a6384821f9 Fix using generated test configs for sh_test, python_test and rust_test modules
Annotate the test config properties with android:"path", and always
use LOCAL_FULL_TEST_CONFIG to pass the path to Make.

Bug: 157621988
Test: manual
Change-Id: I1c020e642f83d994d63c9e9cb56e686086d5acaa
2020-06-10 22:48:16 -07:00
Jaewoong Jung
4b79e98a6e Soong package structure refactoring
Give prebuilt_etc and sh_binary their own packages and split the
gigantic main Android.bp up to small, per-package ones.

Test: m nothing, TreeHugger
Bug: 156980228
Change-Id: I7b00cd344b9f16861f1ff39edf0029f016b853d0
2020-06-01 13:44:48 -07:00
Dan Willemsen
bc32a50ff4 Don't check elf properties of python prebuilts
We're not doing a great job of copying over all the library dependencies (especially when using sanitizer libraries). Just omit these from the checks, which shouldn't really matter.

Bug: 151093572
Change-Id: I8470bbfbf36a51739e8a7e699ad090a66c9db8b2
Test: treehugger
2020-03-11 20:55:17 +00:00
Dan Willemsen
7fa7646ad7 Add tests for py(2|3)-cmd
Test: m par_test{,3} py{2,3}-cmd; ./runtest.sh
Change-Id: I4aadd9fc85be4381bd9a0843064ceaa87150b5ee
2020-02-19 19:40:38 -08:00
Dan Willemsen
d7a1dee1c1 Python3 on device links against liblog
Fixes: 148037229
Test: cherry-pick https://r.android.com/1212194; m try_python_binary_test
Change-Id: I6e26ba232d7698a4889a6e0be1aeb09d54cc4dc5
2020-01-20 22:14:45 -08:00
Dan Shi
2468d015fe Pass auto_gen_config setting in Android.bp to makefile
Bug: none
Test: local test with fuzzy_fastboot
Change-Id: I2c62f6fb58737538ef9e1e9fcbe63e03dfa7ee91
2020-01-06 16:21:10 -08:00
Colin Cross
98be1bb00f Move filesystem into Config
The filesystem object was available through ModuleContext.Fs(), but
gives too much access to the filesystem without enforicing correct
dependencies.  In order to support sandboxing the soong_build
process move the filesystem into the Config.  The next change will
make it private.

Bug: 146437378
Test: all Soong tests
Change-Id: I5d3ae9108f120fd335b21efd612aefa078378813
2019-12-18 08:19:10 -08:00
Dan Willemsen
67d4258786 Test par file argument handling
This had been broken with my python3 launcher.

Test: build/soong/python/tests/runtest.sh
Change-Id: I0524e5a80244bf1080cb28237b23ef0d023e03dc
2019-12-03 23:54:04 -08:00
Colin Cross
4b49b768a2 Make TestContext.RegisterModuleType take an android.ModuleFactory
Avoid having to pass ModuleFactoryAdaptor to every call to
RegisterModuleType in a test by wrapping RegisterModuleType.

Test: all soong tests
Change-Id: If8847d16487de0479cc3020b728256922b3cadba
2019-11-25 10:59:44 -08:00
Dan Willemsen
8d4d7bee6d Add python3 embedded launcher support
Test: m par_test{,3}; build/soong/python/tests/runtest.sh
Change-Id: I9c0fac9e2947616fdeedbfc55026dc3065966e71
2019-11-04 19:48:19 -08:00
Colin Cross
0f7d2ef3ac Add method to determine variations from a Target
The arch variants are hardcoded in every module type.  Refactor
them out into a Target.Variations() method in preparation for
splitting the arch mutator into two, which will require using
different variations.

Test: m checkbuild
Change-Id: I28ef7cd5168095ac888fe77f04e27f9ad81978c0
2019-10-16 14:52:30 -07:00
Treehugger Robot
c6136c9d65 Merge "Remove old-style support for translated second architectures" 2019-10-07 20:48:07 +00:00
Colin Cross
ff6c33d885 Replace RelPathString() with ToMakePath()
Add a ToMakePath() method that returns a new path that points out
out/ instead of out/soong/, and replace the
"$(OUT_DIR)/" + path.RelPathString()
pattern with
path.ToMakePath().String()

Bug: 141877526
Test: m checkbuild
Change-Id: I391b9f2ed78c83a58d905d48355ce9b01d610d16
2019-10-03 10:07:53 -07:00
Colin Cross
70dda7e3da Separate InstallPath from OutputPath
Create a new type InstallPath that is similar to OutputPath to
differentiate intermediates output paths from installed output
paths.

RelPathString is a poorly defined, undocumented function that is
primarily used to get an install path relative to out/soong to
generate an equivalent install path for Make relative to $(OUT_DIR).
Move it to InstallPath for now, and fix the one remaining user on
OutputPath.

Add a method to create an NDK install path so that ndk_sysroot.go
doesn't have to do it manually with PathForOutput.

Bug: 141877526
Test: m checkbuild
Change-Id: I83c5a0bd1fd6c3dba8d3b6d20d039f64f353ddd5
2019-10-03 10:07:53 -07:00
Dan Shi
6ffaaa830d Allow test to specify whether or not to auto-generate test config
Attribute `auto_gen_config` is added to test modules.
Test config will be generated if:
the attribute is not set and AndroidTest.xml doesn't exists
or
the attribute is set to true, whether or not AndroidTest.xml exists.

Test config will NOT be auto-generated if:
the attribute is not set and AndroidTest.xml exists
or
the attribute is set to false, whether or not AndroidTest.xml exists.

Bug: 141684102
Test: build test module with auto_gen_config set to true
Change-Id: I64fb003a83d8c32a967835e5f8d12fe4476043be
2019-09-26 13:29:28 -07:00
Colin Cross
3b19f5d71d Remove old-style support for translated second architectures
Translated second architectures now go in NativeBridgeArch instead
of DeviceSecondaryArch.

This reapplies I568046330abc002d4eed582cb999b62a5eaba790 with
ctx.Config().HasMulitlibConflict() added to fix the NDK build,
which has arm64, arm, x86_64, and x86 architectures enabled.

Test: m checkbuild
Test: OUT_DIR=out_ndk build/soong/scripts/build-ndk-prebuilts.sh
Test: no change to build.ninja or Android-aosp_cf_x86_phone.mk
Change-Id: Iadcafbd64bfb9579ae7c86914927c43a062b0c8e
2019-09-19 12:54:32 -07:00
Colin Cross
402be41aa8 Revert "Remove old-style support for translated second architectures"
This reverts commit fe6330c7d2.

Reason for revert: this was accidentally propping up NDK builds
Bug: 141242600
Change-Id: Ie9f8e31479a74e8e9e14d720a5936da6b1d5d113
2019-09-18 21:12:31 +00:00
Colin Cross
fe6330c7d2 Remove old-style support for translated second architectures
Translated second architectures now go in NativeBridgeArch instead
of DeviceSecondaryArch.

Bug: 141242600
Test: m checkbuild
Change-Id: I568046330abc002d4eed582cb999b62a5eaba790
2019-09-18 16:23:18 +00:00
Colin Cross
ee94d6ab14 Add RuleBuilder helper functions for built and prebuilt tools
Replace the common pattern of:
cmd.Tool(ctx.Config().HostToolPath(ctx, "tool"))
with:
cmd.BuiltTool("tool")

And similarly for PrebuiltBuildTool.

Test: m checkbuild
Change-Id: I7d63188505362c7df6a3b3e7330b4a2cca5a2409
2019-07-11 13:05:19 -07:00
Colin Cross
09364fd955 Fix python proto srczip argument order
-P has to come before -D to have an effect on the files read from the
directory.

Fixes: 130160833
Test: atest acloud_test
Change-Id: I62a998f1ad1e3b45f590babbf39330955d368373
2019-04-08 20:04:23 +00:00
Colin Cross
cc0ce80ed5 Fix package path of android/soong/android pctx
android/soong/common was renamed to android/soong/android long
ago, but the pctx package path was still "android/soong/common".
This required all users of rules defined in android/soong/android
to import "android/soong/android" and then
pctx.Import("android/soong/common").

Test: m checkbuild
Change-Id: I20d096522760538f7cfc2bec3d4bfeba99b275d4
2019-04-04 03:59:34 +00:00
Colin Cross
fe17f6f0e8 Add support for protoc plugins
Add a proto.plugin property to allow specifying a custom protoc
plugin to generate the code.

Fixes: 70706119
Test: m am StreamingProtoTest
Change-Id: I1ecdd346284b42bbcc8297019d98d2cd564eb94c
2019-04-02 16:38:55 +00:00
Colin Cross
19878da6a0 Move proto compilation to RuleBuilder
Using blueprint.Rule for protoc commands was causing code duplication
because there was no good way to run the same protoc for cc, java and
python but then run custom source packaging steps for java and python.
Move most of the code into a common function that returns a
RuleBuilder, and then let java and python add their own commands at
the end of the rule.

Bug: 70706119
Test: All Soong tests
Test: m checkbuild
Change-Id: Ic692136775d273bcc4f4de99620ab4878667c83a
2019-04-02 16:38:47 +00:00
Colin Cross
8a49795df1 Replace ctx.ExpandSources with android.PathsForModuleSrc
Move the logic from ctx.ExpandSources into android.PathsForModuleSrc
and ctx.ExpandSource into android.PathForModuleSrc, and deprecate
them.  When combined with the pathDepsMutator this will let all
properties that take source paths also take filegroups or genrule
outputs, as long as they are tagged with `android:"path"`.

Test: All soong tests
Change-Id: I01625e76b5da19240e9649bf26a014eeeafcab8f
2019-03-20 19:36:13 +00:00
Colin Cross
27b922f53e Annotate paths and deprecate ExtractSource(s)Deps
Add `android:"path"` to all properties that take paths to source
files, and remove the calls to ExtractSource(s)Deps, the
pathsDepsMutator will add the necessary SourceDepTag dependency.

Test: All soong tests
Change-Id: I488ba1a5d680aaa50b04fc38acf693e23c6d4d6d
2019-03-07 18:36:35 +00:00
yangbill
4f41bc2bed [SOONG] Not auto-generate test config if test_suite is cts
Current soong use source code's under cts as the rule for judgement.
Should change to use test_suites define instead.

Bug: 124313692
Test: m hello_world_test, make sure test config be auog-enerated.
      Modified platform_testing/tests/example/native/Android.bp
      m hello_world_test, make sure test config not be auto-generatetd.

Change-Id: I1bc5216f73329d2a82d9ff29ccbede436dd2976c
2019-02-19 22:25:50 +08:00
Dan Willemsen
6ca390f0b4 Support building a par file that does not automatically run
Mainly so that we can build a `python` prebuilt that acts like the
normal python install, though you could also build different ones with
more packages pre-installed.

Bug: 117811537
Test: move built py2-cmd into prebuilts/build-tools/path/linux-x86/python and build
Change-Id: I21215f6fd3754d89f8c65e1dfeb3f2deea23239f
2019-02-17 12:14:23 -08:00
Dan Willemsen
707542f298 Simplify python launcher, use __main__.py
Uses more python rather than C++, and skips less of Py_Main.

Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
2019-02-15 14:33:59 -08:00
Dan Willemsen
22e9244b16 Add par_test
This test adds a number of checks for the visible python state inside a
hermetic par file.

Test: build/soong/python/tests/runtests.sh
Change-Id: I7188d862a309a09623169e1f967bde86707d41af
2019-02-14 02:18:05 +00:00
Colin Cross
5f692ec219 Remove empty DepsMutator methods
Add an empty DepsMutator to ModuleBase so it doesn't have to be
implemented on every module that doesn't need it.

Test: all soong tests
Change-Id: I545a832a0dbf27386d3080377a75ea482cd9ce59
2019-02-01 17:17:58 -08:00
Logan Chien
02880e4196 Add missing dependencies for python_test
This commit adds missing shared lib dependencies for `python_test`
modules with embedded launcher.

Bug: 119086738
Test: CHECK_ELF_FIELS=true make check-elf-files
Change-Id: I26f8e1eb9086930093f60c7daa54469850fab32d
2019-01-29 22:26:57 +08:00
yelinhsieh
80880a3460 Autogen python test config.
Feature request from developer, autogen python test config.

Bug:118442443

Test: $source build/envsetup.sh ; lunch
      $sed -i '/test_config/d' tools/acloud/Android.bp
      $make acloud_test
      $[ -f $ANDROID_HOST_OUT_TESTCASES/acloud_test/acloud_test.config ] && echo "Found" || echo "Not found"

Change-Id: I196230581b456afe03e67f9cd84c028d18ba08d9
2019-01-14 03:30:06 +00:00
yangbill
7265e5f9c1 Pass -S option to wrapper for python binary.
Prevent problem for some users using PAR file built by android's
build system which cause by the different site package of python
installed by user themselves.

Bug: 120469915
Test: 1. m atest
         atest --help
      2. apply aosp/861470
         full build

Change-Id: I72c83f9397f0e88eecfcd32806f1746520299447
2019-01-04 18:10:32 +08:00
Dan Willemsen
83f97aed95 Fix mac build w/echo -n
Soong doesn't wrap everything in bash, but relies on the ninja default
of /bin/sh. We should probably improve that in the future, but for now,
just fix the build.

On Mac, /bin/sh and /bin/bash are both bash, but /bin/sh implies `set -o
posix`, and ignores the `-n` flag.

Test: treehugger
Test: build on a mac
Change-Id: Icf41b1c5a1ce9eb0a56e39e68c433fc80b53620f
2018-12-04 13:06:02 -08:00
Dan Willemsen
54c5b61f0a python par: trim and convert the entry point at build time
Instead of doing these string operations in the C++ code at every
startup of an embedded par, trim and convert the entry point file to a
python module path at build time.

Test: m
Change-Id: I04a6459a80c84a704b8a25a144ab1e19622d6cc3
2018-12-03 15:58:45 -08:00
Dan Willemsen
263dde779e Fix par file zip offsets
The zip file format does support scripts/programs prefixed to the
archive, though many of the offsets are supposed to start from the
beginning of the file. Some tools (python and zipinfo) are able to
read zip files with arbitrary prefixes, but others (libziparchive and
zipdetails) fail.

So pass the file to prefix onto the zip file into merge_zips so that we
can set the offsets from the real start of the file.

Test: m sepolicy_tests  (runs the embedded python interpreter)
Test: zipinfo out/host/linux-x86/bin/sepolicy_tests
Test: zipdetails out/host/linux-x86/bin/sepolicy_tests
Change-Id: If73d4c2465581f7de5aa47959284ecf2059df091
2018-12-03 15:55:15 -08:00
Nicolas Geoffray
6e1bf2b760 Define runfiles_path before try so that finally can use it.
Otherwise, python will complain the variable has not been initialized,
in case ExtractRunFiles throws an exception.

Test: m
Change-Id: I3d9e3cda5e558436ee4182c44e4345151c031e41
2018-10-11 15:18:42 +01:00
Nan Zhang
cba97e69ab Fix invalid memory error for python binary build
The root cause is we didn't check if the optionalpath is valid or not,
the registerbuildation function directly invoke the path var and
cause the invalid memory error. We just return if the launcher doesn't
exist.

The ctx.VisitDirectDepsWithTag() also handles allowmissingdependency so
we are ok if launcher doesn't exist.

Test: N/A
Bug: b/116698229, b/67510844
Change-Id: I40941079a64d7797ab879fc5edaa29e835b493a0
2018-09-26 16:57:24 -07:00
Julien Desprez
e146e39fa6 Support 'test_config' into soong modules
Test: make general-tests
Bug: 110982517
Change-Id: Ib2eab2653fdfce6f699b85c9fbc64558b6d40363
2018-08-07 15:51:21 -07:00
Dan Willemsen
59339a29e1 Fix go vet issues
Test: go vet ./...
Change-Id: Ifb936ccc5e2b5a2c3fcbbbcb54f680e2973ea1b3
2018-07-22 21:18:45 -07:00
Colin Cross
6b75360cbc Use WalkDeps instead of VisitDepsDepthFirst
VisitDepsDepthFirst is almost never correct, as it can't query
dependency tags of multiple dependencies between the same two modules.
Replace VisitDepsDepthFirst with WalkDeps in sanitize.go and
python.go.  Also verify the dependency tag before continuing to
recurse to ensure that they don't recurse through genrules and into
unrelated modules.

Test: m checkbuild
Change-Id: I2f7560126f56b51a40ec39dfbdcc18b5891489f7
2018-06-22 12:44:35 -07:00
Nan Zhang
bea0975f18 Fix embedded_launcher can't find files
The problem came from Python libraries doesn't know the information that
we enabled embedded_launcher (only Python binary knows about that). And
we can't simply remove runfiles dir for Python libraries since host
Python mode need this.

Bug: b/80441699
Test: m perf_profo_flames
Change-Id: I73ffc4d7504f95a708ae7cca47bc6c15a673aa31
2018-06-04 12:32:55 -07:00
Dan Willemsen
184901135c Revert "Revert "Revert "Revert "Add path interposer""""
This reverts commit 09f4540d66.

Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.

Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.

Also fixes some potential nil pointer dereferences.

Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
2018-05-31 14:59:33 -07:00
Dan Willemsen
09f4540d66 Revert "Revert "Revert "Add path interposer"""
This reverts commit c59a92cb1a.

Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.

Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
2018-05-25 23:26:00 +00:00
Dan Willemsen
c59a92cb1a Revert "Revert "Add path interposer""
This reverts commit 96c957ae20.

Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.

Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.

Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
2018-05-25 14:07:00 -07:00
Treehugger Robot
c039ab084d Merge "Allow "-" to be part of Python file names" 2018-05-24 17:34:22 +00:00
Nan Zhang
ccf636e36a Revert "Revert "Add libc++.so as install dependencies for hermetic Python.""
This reverts commit bfb0aed2dd.

Reason for revert: <LOCAL_REQUIRED_MODULES overrides the "required:" property in Android.bp>
Test: m -j sepolicy_tests
Bug: b/80140289

Change-Id: I4e51007a7d4dda69cbf4f31253e9a56d3f5e35c7
2018-05-23 18:09:34 -07:00
Nan Zhang
1a479b6ecd Allow "-" to be part of Python file names
Test: m -j py-setuptools
Bug: b/79751992

Change-Id: I4cd0cf8671a9bb912a8ba819265274ab0c9e0074
2018-05-23 15:15:39 -07:00
Treehugger Robot
78a01965d0 Merge "Fix Protobuf importing issue for embedded Python" 2018-05-23 20:24:42 +00:00
Artem Iglikov
bfb0aed2dd Revert "Add libc++.so as install dependencies for hermetic Python."
This reverts commit ff7d397218.

Reason for revert: breaks a few targets

Change-Id: I3f26bfda9e276730accff558333f910a1dc017e8
2018-05-23 08:49:38 +00:00
Nan Zhang
f0c4e4339a Fix Protobuf importing issue for embedded Python
As I mentioned in b/79751992 #4, when launcher loads __main__ as entry point,
the module dict got set using absolute pkg path which
was runfiles.google.protobuf which caused the type checking failed
(expected: google.protobuf for type()). After then, starting with non entrypoint
python files, the importing module dict would be set with from...import...

So remove dir:runfiles/ from zip section so that during entrypoint Python file
importing the proto files using
from...import..., the type checking can be passed.

Bug: b/79751992
Test: m clean && m out/host/linux-x86/bin/perf_proto_stack;
And execute it:
usage: perf_proto_stack [-h] [--syms SYMS] [--json-out JSON_OUT]
                        [--print-samples] [--skip-kernel-syms]
                        [--print-pid-histogram] [--print-sym-histogram]
                        [--print-dso-histogram] [--parallel PARALLEL]
                        file [file ...]
perf_proto_stack: error: too few arguments

Change-Id: I95a4a9ee95ce7f90d893ea0529c384d7b944abe4
2018-05-23 03:33:36 +00:00
Nan Zhang
ff7d397218 Add libc++.so as install dependencies for hermetic Python.
Bug: b/80140289
Test: m clean && m test_me
Change-Id: I3acbf1bac3b866b37f9ec69584dc9be7a5ad1d89
2018-05-22 14:18:30 -07:00
Dan Willemsen
4339853a20 Add a dependency fixer for proto deps
protoc dependency files, at least for C++ outputs, uses the form of:

  a/b.c \
  a/b.h: <dep1> <dep2>...

Ninja will fail the command when it parses a dep file and there's more
than one output file (even though it doesn't care what the output file
name is). So this tool will parse the original file, and output a
version with only a single output file.

Bug: 67329638
Test: NINJA_ARGS="-t deps ...pb.c" m
Test: NINJA_ARGS="-t deps ...srcjar" m
Test: NINJA_ARGS="-t deps ...srcszip" m
Test: Run dep_fixer across all of taimen's dep files, no failures.
Test: Run dep_fixer against the processed files, no changes.
Test: Run androidmk across all of our Android.mk files, inspect the diffs
Change-Id: I4263b7d5faea37285afa6b24dedf5964aa7d19dc
2018-05-07 16:21:59 -07:00
Colin Cross
ff3ae9da74 Consistently use Bool instead of proptools.Bool
Use Bool instead of proptools.Bool and String instead of proptools.String.

Test: m checkbuild
Change-Id: I32d84add9f27128c7a65413e9612fd920613584f
2018-04-10 22:50:34 -07:00
Nan Zhang
b8fa197878 Add Python protobuf support.
Python protobuf std libs will be wrapped in final binary/test par file.

Bug: b/70568913
Test: manually create real examples.
Change-Id: I7376ec9175f3e03d1adbd20858a7f74e826387ad
2018-03-22 15:56:51 -07:00
Logan Chien
4203971351 Extract failIfErrored() to android/testing.go
Bug: 74506774
Test: lunch aosp_walleye-userdebug && make  # runs unit tests
Change-Id: I1c09412d5988dca2cc1c5f041893b313ab1c163a
2018-03-15 11:02:38 +08:00
Nan Zhang
1db85407aa Remove timestamp based filelist file for tracking Python dependencies
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.

Test: m -j checkbuild && real examples:
Bug: b/70568913

Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
2018-03-09 14:12:48 -08:00
Nan Zhang
27e284d2b0 Revert "Revert "Support filegroup in exclude_srcs""
This reverts commit 606e9de344.

Reason for revert: <try to fix the broken build yesterday>

Change-Id: I2963b9af63c7c7398159e5e9a1e448266e1c81d5
Test: unittest
2018-02-15 14:38:40 -08:00
Tobias Thierer
606e9de344 Revert "Support filegroup in exclude_srcs"
This reverts commit f36a3d9b6d.

Reason for revert: Broke several builds. I'm acting build cop, reverting.

Bug: 70351683

Change-Id: I775ada4e9cb6473519d51420b41b818af163da44
2018-02-09 09:46:11 +00:00
Nan Zhang
f36a3d9b6d Support filegroup in exclude_srcs
Test: add unit-test, m -j32
Bug: b/70351683
Change-Id: Iff83c56d45dd668d9df6131c7df2e23e5c73a21b
2018-02-07 10:18:41 -08:00
Colin Cross
ee6143cde2 Add VisitDirectDepsWithTag
Add a method on ModuleContext and TopDownMutatorContext to visit
direct dependencies that have a given dependency tag.

Test: m checkbuild
Change-Id: Ib875563091dcae6b7282b3e3427d0eb07d8c8af5
2018-01-02 18:23:43 -08:00
Nan Zhang
b8bdacfd0b Add HostToolPath() for Python binary
So in the genrule, Python binary module can be used as tool.

Test: manually write a test genrule.
Change-Id: Idfd3af4c1002dd608e1bdffa203e01c802f1bf21
2017-12-06 15:13:10 -08:00
Nan Zhang
d9ec5e7dea Revert "Revert "Change Python in Soong to support device side build.""
This reverts commit 0a32e5936b.

Reason for revert: <libsqlite was missing for darwin_x86_64 before>

Change-Id: I2e13e849a503a705ffad425df292380f2f73954e
2017-12-01 20:00:31 +00:00
Nan Zhang
0a32e5936b Revert "Change Python in Soong to support device side build."
This reverts commit 3bba05962d.

Reason for revert: <revert mac builds>

Change-Id: I25f1380573d69f7e111287624a2d4a3d95fdd86b
2017-12-01 06:19:27 +00:00
Nan Zhang
3bba05962d Change Python in Soong to support device side build.
Add python_test to generate executables for both host and device side.
And change python_library_host to python_library.

installation for python device test will be under
target/product/generic_arm[64]/data/nativetest[64].

Bug: b/69114465
Test: m clean && m -j
Device Test:
adb root; adb push nan_devicetest /data;
adb shell; ./nan_devicetest
print(sys.path)
['/data/nan_devicetest/runfiles', '/data/nan_devicetest', '/data/nan_devicetest/internal', '/data/nan_devicetest/internal/stdlib']

Change-Id: If8317070a3aa1b6dab2e84b8df2d037f495d7247
2017-11-30 13:55:49 -08:00
Nan Zhang
ea568a4a24 Change bool, and string properties to *bool, and *string for java,
python, and genrule.

Test: m -j checkbuild
Bug: b/68853585
Change-Id: Ic9a8083818e920dc399a4b00841e2aa496f70faa
2017-11-08 21:25:40 -08:00
Nan Zhang
c9c6cb77d8 Add support for test_suites for python_binary_host and python_test_host
Test: manually add this property, and then check Android-aosp_arm.mk
Bug: b/35394669
Change-Id: Iddaf677839714daf4f8913ff4bf9e944d38c18db
2017-11-07 13:13:59 -08:00
Colin Cross
d11fcda940 Convert Visit*Deps from blueprint.Module to android.Module
Also adds checks that the dependencies are android.Modules and
are not disabled.

Test: m checkbuild
Change-Id: I05e945f38915d49cd3c0ab72a86576949bc7eff2
2017-10-24 13:01:03 -07:00
Colin Cross
ae88703df5 Move ModuleContext.ModuleBuild to ModuleContext.Build
Now that android.ModuleContext does not include blueprint.ModuleContext
we can rename android.ModuleContext.ModuleBuild to
android.ModuleContext.Build without colliding with
blueprint.ModuleContext.Build.  Leave ModuleBuild as a wrapper around
Build for now to avoid having to update all the users outside
build/soong simultaneously.

Test: m checkbuild
Change-Id: I18eb8cc04faf002049a11d9aac97e9732ff5d638
2017-10-24 10:59:00 -07:00
Colin Cross
6ccbc913e4 Let tests override environment
Pass an environment to TestConfig that will be used for
all Config.Getenv calls.

Test: none
Change-Id: I683cd9c9e0db61c9bfd2adb27fca78f558f225c4
2017-10-10 23:28:43 -07:00
Dan Willemsen
7d1681a174 Use python2.7 instead of python2
Ubuntu has both aliases, but Mac only declares python2.7, not python2.

Test: `m sepolicy_tests` on internal master
Change-Id: Ia2b5d2d95195f307befbc1893a465752f14b8a7b
2017-09-25 21:48:34 +00:00
Nan Zhang
a3fc4ba733 Add support for python_defaults modules
Test: python_test.go
Change-Id: I2077414a5b06da5e660a1b48bfdb2eb926fb702f
2017-09-05 17:31:24 -07:00
Nan Zhang
d4e641b6e9 <Hermetic> Replace Soong Python bootstrap process with embedded
launcher.

For Python2, we bundle embedded launcher as bootstrapper within every
.par file. This feature is only enabled for linux_x86_64 for now. We
provide a user flag: hermetic_enabled within bp file. By default, Pyhon2
still use classic bootstrapping way to construct .par file and relys on
host interpreter. Once embedded_launcher is enabled, launcher will be
used to bootstrap .par file and execute user program.

For Python3, the launcher will be ready soon, and for now it still relys
on classic bootstrapping.

Test: Real example is used to test.

Bug: b/63018041

Change-Id: I28deba413d8ad3af407595e46f77d663e79a3705
2017-09-05 17:26:57 -07:00
Colin Cross
5c51792926 Make binaries executable
Split InstallFileName into InstallExecutable that does chmod +x
after copying the file.  Also remove InstallFile and rename
InstallFileName to InstallFile.

Test: m -j checkbuild
Change-Id: Id41ad4eafe521f6cd5d8cc250b7747ecb3da8dfc
2017-08-31 13:48:52 -07:00
Colin Cross
a18e9cfa29 Remove error from AndroidMkDataProvider.AndroidMk
It's never anything except nil, and it unnecessarily complicates
the implementations.

Test: m -j checkbuild
Change-Id: I3e3b7251f32ffa84dbdfd0448faf248c306ca808
2017-08-11 15:24:11 -07:00
Colin Cross
27a4b05441 Remove error from AndroidMkData.Extra
It's never anything except nil, and it unnecessarily complicates
the implementations.

Test: m -j checkbuild
Change-Id: I8a117a86aa39aeb07d9d8d0686ef869c52784f19
2017-08-11 15:24:11 -07:00
Colin Cross
cec8171420 Add integration testing infrastructure
Fix mutator registration for tests to allow different tests
in the same package to register different mutators.

Allow tests to track the resulting ModuleBuildParams objects
to use in assertions, and provide helpers for getting them.
For example:
    config := android.TestConfig(buildDir)
    ctx := android.NewTestContext()
    ctx.RegisterModuleType(...)
    ctx.MockFileSystem(...)
    ctx.ParseBlueprintsFile("Android.bp")
    ctx.PrepareBuildActions(config)
    ctx.Register()
    // Get the Inputs value passed to the javac rule for the foo module
    inputs := ctx.ModuleForTests("foo".Rule("javac").Inputs

Test: java_test.go
Change-Id: I10c82967f5f3586d2c176f169906b571ed82fc73
2017-07-14 14:19:51 -07:00
Colin Cross
36242850fd Refactor factories
Change module factories from returning a blueprint.Module and a list
of property structs to returning an android.Module, which holds the
list of property structs.

Test: build.ninja identical except for Factory: comment lines
Change-Id: Ica1d823f009db812c518f271a386fbff39c9766f
2017-06-30 21:08:36 +00:00
Nan Zhang
aac67d38b2 Fix the issue that /tmp/soong_python_test* isn't being removed.
Bug: 62499851
Test: go test
Change-Id: I84bd66cd1410dd9fc6add3ba5626d8ec4fe702d7
2017-06-12 10:49:42 -07:00
Nan Zhang
5323f8e32f Add python_test_host module.
bug: 31676493
Test: created py_test modules in real folder. and ran 'mma'.
Change-Id: I22aa2fad74b11e4a31ea7a4c4a4f0ea64cd3fc94
2017-05-17 11:26:00 -07:00
Nan Zhang
270051119a Fix the issue "\n" didn't get interpreted properly.
Added params to WriteFile rule that allow users to turn on the
specific 'echo' options, such as "enable the interpretation of backslash
escapes".

Bug: b/38261000
Test: tested with real data files.
Change-Id: I5145056b5c2522a3a72b9436055ec95d2fbd94c0
2017-05-16 15:10:29 -07:00
Colin Cross
67a5c132c5 Prettify soong ninja build descriptions
Descriptions currently look like:
[  0% 4/29328] cc out-soong/.intermediates/external/clang/lib/Sema/libclangSema/android_arm_armv7-a-neon_denver_static_core/obj/external/clang/lib/Sema/SemaCodeComplete.o

This is not very helpful - most of the characters are used to show the
output path, which contains useful information like target architecture,
but also contains most of the path to the source files twice, and less
useful information like the exact variant name used by soong.
Make the descriptions look like:
[  0% 3/29329] //external/clang/lib/Sema:libclangSema clang++ SemaTemplateInstantiate.cpp

This is //path/to/module:modulename tool relative/path/to/source/file

Test: builds, looks pretty
Change-Id: I3087aa7d4eb1860ef6239d77407b8b35445616d7
2017-05-10 11:04:16 -07:00
Nan Zhang
db0b9a3cf3 Supported python build in host side.
The base module handles all the common functionalites, such as version
compatibilty check, version variations split, source file format check,
source/data file duplicate check.

The library/binary module focuses on how to generate binary build actions,
such as setting up stub script, zipping, filling in __init__.py in
runfiles dir tree.

Bug: b/31676493
Test: go test under python package

Change-Id: I06608369f350f7195873d459e1c8d1bdb811e77e
2017-05-05 13:27:56 -07:00