This no-op refactoring facilitates some upcoming functional changes for
"bp2build allowlist v2". The work requires that the bp2build conversion
mutator be changed from a TopDown mutator to a BottomUp mutator.
Refactoring all bp2build-related methods so that they use Bp2buildMutatorContext
makes it easier to make this functional change without touching tens of
files and multiple projects.
Bug: 285631638
Test: m bp2build
Change-Id: I3d1ef3064146e959c6f0dc315350fc9764bf2bd2
This reverts commit b1094d602b.
Reason for revert: prebuilts updated in aosp/2726579
Bug: 278602456
Change-Id: I750b0f7e4dd94329a4455499f869a1823e660d77
* changes:
Handle proto.include_dirs in bp2build for py
Add proto.local_include_dirs support in bp2build
Add manual tag to proto_library at the root package
Followup to aosp/2711114 which added proto.include_dirs support for cc
Test: go test ./bp2build
Test: Added an integration test in build/bazel/examples
Bug: 285140726
Change-Id: Ic78dacbe46d9ea67e9441948852a3f94dabbb35e
If a python_library uses a pkg_path foo/bar, then the proto srcs in
that libray need to import the dep .proto as foo/bar/proto.proto.
This behavior is restricted to python modules. To implement this is in
bp2build, this CL creates a new interface with a single method
`PkgPath`. Only python module structs implement this interface, and
this method is only available during bp2build
Test: Added a bp2build unit test
Test: TH
Change-Id: If8d207c0b321f75337a053795826b283a5eaaf46
The precompilation logic assumes that the source tree matches the python
prebuilts, which is not necessarily true. This very much breaks during
major version upgrades, so just disable precompilation temporarily while
we generate new prebuilts.
Bug: 278602456
Test: treehugger
Change-Id: Idb206bb1d971e8c0fa556c75419623c3bc457898
Due to upstream changes in python path calculations, I'm simplifying the
python launcher to work more like a standard python distribution. This
is effectively changing the stdlib path from `internal/stdlib` to
`internal/python3.10` (or `3.11`, etc). This allows us to specify the
zip file as PYTHONHOME, set PYTHONPLATLIBDIR to `internal` and use the
default detection after that.
That does mean during upgrades that the stdlib pkg path will change, so
move the source vs prebuilt calculation from the Android.bp into Soong
to choose which stdlib module to pick up (with the corresponding
`pkg_path`)
Bug: 278602456
Test: treehugger with python3.10
Test: a python3.11 source + 3.10 prebuilt build
Test: a python3.11 source+prebuilt build
Change-Id: I8b02e7b22a1f1d1e02819ae1a31a99cdc985542c
We encountered a duplicate sources bug in python proto generation
because we're only using the base name. This CL makes it use the
entire path to avoid this.
Fixes: 295333879
Test: Presubmits
Change-Id: I4ab821508cd5979bd595e327c1ccd3d68110fba6
* changes:
Handle .proto files in different package for filegroups
Handle .proto files that end up in a different package
Allow creation of BazelTargets in a different directory
Bazel poses a strict requirement that .proto files and proto_library
must be in the same package. This CL handles this automatically by
creating the proto_library in a separate dir/package if necessary
Implementation details
- Partition the `srcs` by package. `srcs` has been computed using
`transformSubpackagePath`, so the information about packages is
available at this point
- Create a proto_library in each package by using
`CommonAttributes.Dir`. Collect all these additional libraries
and put them in `info.Proto_libraries` so that they get added as deps
of (cc|python|...)_proto_library
- Add an import_prefix to the proto_library in subpackages relative to
the current directory. This relies on the assumption that every src is
beneath the current directory (Soong will complain if a path in
Android.bp contains ../)
filegroup module type uses a separate code-path to create proto_library.
This will be handled in the next CL in stack.
Test: bp2build unit tests
Test: TH
Test: Built the failing internal module mentioned in
b/292583584#comment1
Bug: 292583584
Change-Id: I437fc89092321b26c5f0511387cde9e84084d6f9
Ic37c8db918873ddf324c86b12b5412952b0f2be2 converted python_binary_host
and python_library_host from HostSupportedNoCross to HostSupported,
but left python_test_host. Do the same for python_test_host.
Fixes dependencies between java_test_host modules that create
host cross variants and python_test_host modules that were missing
host cross variants.
Bug: 282918027
Test: builds
Change-Id: Iae687aa3aa0f0b005f2dd27469f631145247a008
There are no remaining Python 2 binaries using an embedded
launcher in Android.
Bug: 245854393
Test: m par_test py2-cmd py3-cmd && build/soong/python/tests/runtest.sh
Change-Id: I241bbaa417060b51b4d2883011ccb43e22ace4c8
There was a request for using b with python tests. bp2build python
tests exactly the same way as python binaries so that they can be
used with `b`.
Bug: None
Test: go test
Change-Id: Id68a6a73572745a4885b3e5bb1b8452e36baa982
Except if BUILD_BROKEN_USES_SOONG_PYTHON2_MODULES is set, and except for
some core py2 modules that can't be removed until python2 is fully gone.
Bug: 203436762
Test: m nothing
Change-Id: I62ccb6f5687eab1e79c372ffc234a90ca5b566ac
Mobly is a Python-based test framework that specializes in supporting
test cases that require multiple devices. There are some Mobly based
Android tests, but these tests cannot be added to xTS since they are
not using TF. Currently TF provides a 2-device Mobly test runner.
This change is a short-term solution to help teams (eg. bluetooth) to
add their multi-device Mobly tests to xTS without writing TF configs.
For long-term, device infra team and xTS infra team will provide a
solution for more complex multi-device tests in xTS and deprecate this
TF template.
Test: manually check the generated mobly xml config
Bug: 272239365
Change-Id: I252cc774094eebbd9264243d30023ace4839c2cd
Still fall back to the hardcoded list for what to put in LOCAL_SHARED_LIBRARIES,
because I couldn't figure out a good way to get those modules at the moment.
Bug: 259718110
Test: Presubmits
Change-Id: Ia992ffea8819d8c67280b08bc45505f77af09de0
This signifigantly improves the startup time of soong-built
python binaries. For example, running
`m apexer && time out/host/linux-x86/bin/apexer` gives
0.734s before this cl, and 0.094s after.
Fixes: 259718110
Test: Presubmits
Change-Id: Ib19e83e2c60c39a849525be117279c318de3afa7
The goal of this cl is to simplify the python rules,
mostly by removing the "decorator" pattern that they
currently use, and instead making separate module
types for libraries, binaries, and tests that inherit
from each other.
Bug: 259718110
Test: Verified ninja files are unchanged (they only change in the list of soong sources because I added/deleted files)
Change-Id: I1e836e2cc4782c7818f91db7df7895de3b8db7ca
Some tests need to add custom tradefed options, but still want to
keep most of the soong autogenerated tradefed xml file.
Expose a test_options: { tradefed_options: [...] } property that
will allow tests to add more options to the autogenerated xml file.
Fixes: 184895128
Test: go test, and verified that the ninja files did not change for aosp_arm64
Change-Id: I50d4ad139322e9e207202f1e1a50f5bbb424aa6f
This reverts commit 8ec823cba1.
Reason for revert: DroidMonitor: Potential culprit for Bug b/262965953 - verifying through ABTD before revert submission. This is part of the standard investigation process, and does not mean your CL will be reverted.
Change-Id: I236cc36981d8b30527ca286632727f8ca267e969
Some tests need to add custom tradefed options, but still want to
keep most of the soong autogenerated tradefed xml file.
Expose a test_options: { tradefed_options: [...] } property that
will allow tests to add more options to the autogenerated xml file.
Fixes: 184895128
Test: go test, and verified that the ninja files did not change for aosp_arm64
Change-Id: I75f7eb002c8325ce7cdc76e12e76e16195320620
The new behavior has been enabled by default, and these
flags aren't necessary anymore.
Fixes: 245583294
Test: m py_dont_import_folder_of_entrypoint_test && /ssd/aosp-master/out/host/linux-x86/testcases/py_dont_import_folder_of_entrypoint_test/x86_64/py_dont_import_folder_of_entrypoint_test
Change-Id: I5b6f98da51791bc5d28662ef799a10c1bb6a35a0
1) Bp2build convert python_binary_host main attribute as LabelAttribute. Currently "main" attribute in python_binary_host is handled as string but for some modules (e.g certify_bootimg) the "main" attribute points to a file in its subpackage like "subpackage/file.py" and should be converted to "//.../subpackage:file.py".
2) Filter out duplicated labels in the merged list of "required" attributes of python_binary_host and its defaults.
Test: b build //system/tools/mkbootimg:certify_bootimg
Test: b build //build/make/tools/releasetools:check_target_files_signatures
Bug: 253081249
Bug: 253101186
Change-Id: Ic2cb4cadec2c1348da70af9f0730da9914d3a8ca
This reverts commit 4d5cc709fa.
Reason for revert: Acloud issue was fixed in aosp/2243150, make sure the host unit tests are run on this cl via go/abtd before submitting
Change-Id: Ie148430d331a01a9877adaf3cb88f1c57af136a0
Bug: 245583294
Bug: 247578564
Bug: 251688241
This reverts commit 5fe655d1c7.
Reason for revert: DroidMonitor: Potential culprit for Bug b/251688241 - verifying through ABTD before revert submission. This is part of the standard investigation process, and does not mean your CL will be reverted.
Change-Id: I1ea4df27557fc891d8f0156ec5681f9b0c601c98
Now that usages have been updated, enable the new
behavior by default. If everything goes well with
this, we can then remove the flags.
Bug: 245583294
Bug: 247578564
Test: Presubmits
Change-Id: Idac8d2dc0f96ae750dba4a99b32525676d0d6f25
So that we can more easily enable these new flags on several
modules.
It seems you can still apply a python_defaults to a python_library
even with this change, so I didn't break it out into a separate
python_binary_defaults.
Bug: 245583294
Test: m py_dont_import_folder_of_entrypoint_test && out/host/linux-x86/testcases/py_dont_import_folder_of_entrypoint_test/x86_64/py_dont_import_folder_of_entrypoint_test
Change-Id: Iecf9c1845df4630bafdea9957eb2450c15f8596b
The python interpreter will by default add the directory of
the entrypoint script to the beginning of sys.path. This
can be disabled in python 3.11+ (which is not released yet)
using the PYTHON_SAFE_PATH environment variable or the -P flag.
As a workaround to have this behavior in older python versions,
we can make an __soong_entrypoint_redirector__.py file at the
root of the zip file that is the entrypoint, and then that
file will redirect to the real entrypoint.
This brings non-embedded-launcher python modules closer to
the embedded launcher version. The embedded launcher binaries
already act like this because they start at an __main__.py file
at the root of the zip file.
Bug: 245583294
Test: m py_dont_import_folder_of_entrypoint_test && out/host/linux-x86/nativetest64/py_dont_import_folder_of_entrypoint_test/py_dont_import_folder_of_entrypoint_test
Change-Id: I39aaf04fb19c3ba7f5c9d98220872d6d08abf736
stub_template_host.txt added all the top-level modules to the
PYTHONPATH, which isn't correct, and caused absl.logging to
override the built-in logging module.
Removing this also makes it more consistent with python binaries
built with embedded_launcher: true. embedded_launcher: true
binaries don't add the top-level modules.
Fixes: 245583294
Test: m py_dont_add_top_level_dirs_test && out/host/linux-x86/testcases/py_dont_add_top_level_dirs_test/x86_64/py_dont_add_top_level_dirs_test
Change-Id: Id3069565d2b2c4b2bda0ff5301e757a7b4201751
Added SPDX-license-identifier-Apache-2.0 to:
python/tests/proto_pkg_path/Android.bp
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Change-Id: I17e9945a265d0ede3e291ed6f5d410dde035fbe5
Currently, python protobuf sources are generated as if
pkg_path didn't exist, but then are moved into the pkg_path
directory after being generated. This means they're generated
with import statements in them that don't include the pkg_path.
These import statements won't work at all when pkg_path is at
least 2 levels deep, but currently erroneously work with a 1
level deep pkg_path because we mistakenly add the top-level
modules in a soong-built python zip to the PYTHONPATH. We want
to remove those modules from the PYTHONPATH, so the generated
protobuf source files have to use the correct imports.
Since there are existing cases of code that needs to be updated,
guard this new behavior behind a flag, protos_respect_pkg_path.
We will set this to true on modules individually as we update
them, and then eventually change the default to true and remove
this flag.
Bug: 247578564
Test: m py_proto_pkg_path_test && out/host/linux-x86/nativetest64/py_proto_pkg_path_test/py_proto_pkg_path_test
Change-Id: I3695cf5521837da087592f2ad5350201035b7b0e
Python programs run via a wrapper script. That wrapper script
can be started with open file descriptors, for example by using
the <(echo foo) syntax in bash. The shouldn't be closed when
starting the real script.
Bug: 203436762
Test: Presubmits
Change-Id: Ib20450f713b15c8a5527d1e433bfd9635cead85f
