Introduce `commonAttributes` & `fillCommonBp2BuildModuleAttrs used in
CreateBazelTargetModule
Adapt `bp2BuildInfo` to use `commonAttrs` instead of `Name`.
And thus also all downstream users of `CreateBazelTargetModule`.
As initial user, the Soong `required` property will be
translated to Bazel's `data`.
Bug: 198146582, 196091467
Test: build_converstion_test.go:TestCommonBp2BuildModuleAttrs
Test: go test
Test: mixed_{libc,droid}.sh
Change-Id: Ib500e40f7e2cb48c459f1ebe3188962fc41ec124
Add new module:
host_snapshot {
name: "host-snapshot"
deps: [
(list of host tools)
],
...
}
Package host tools using android.PackagingBase to capture host tools and transitive packaging data.
Add JSON meta data to snapshot that allows snapshot to be installed via development/vendor_snapshot/update.py
Add support to generate a fake host snapshot of all host modules that is used to detect required
modules via development/vendor_snapshot/update.py.
Bug: 192556798
Bug: 194799048
Bug: 192896149
Test: m HOST_FAKE_SNAPSHOT_ENABLE=true host-fake-snapshot dist -- check snapshot exists in dist
Change-Id: I849c4db801cd858408f6fe6a3ce69262a23a5be9
Have binary and library bp2build convert `libs` to
`deps` for bazel
Bug: 196083751
Test: python_{binary,library}_conversion_test.go
Test: build/bazel/ci/mixed_{libc,droid}.sh
Change-Id: I2d5f6ef2e83dd608910edb7adb2eb9a56d25293c
The code checks to see if the `child` is not a Python library,
but then when reporting the error it uses the module name as
the dependency name and the child (dependency) as the module name
Test: Existing tests pass
Change-Id: Ied606342291312d8485a8fd2ddcc580ad24ae82f
A performance improvement for bp2build as Blueprint/Soong no longer have
the overhead of additional modules. The creation of these modules
results in:
* traversal of additional modules for each subsequent mutator
* synchronization over a go channel to collect newly created modules:
https://cs.android.com/android/platform/superproject/+/master:build/blueprint/context.go;l=2594,2600;drc=1602226f23181b8c3fbfcaf3358f0297e839d7d3
We avoid both of these by storing the information directly in the
underlying module.
Also as a fringe benefit, removes some necessary boilerplate for
conversion.
For benchmarks, reduces runtime ~1% for 1% converted, ~24% for 100%
converted. See more: go/benchmarks-for-https:-r.android.com-1792714
Test: ran benchmarks/tests in bp2build
Test: build/bazel/ci/bp2build.sh
Change-Id: Ie9273b8cbab5bc6edac1728067ce184382feb211
Allow blueprint_go_binary to be used as test data by supporting it in
PathForModuleSrc.
Also make python use GetDirectDepsWithTag instead of VisitDirectDeps
so it doesn't fail on the dependency to a Blueprint (non-Android)
module.
Fixes: 186528269
Test: manual
Change-Id: Ibc1d7caf7bf1fa5db805c3b885a95df4baee0ecf
They seem to be used as Soong validation. They are not implemented against the python
unit tests framework, so we can execute them as such in CI automatically.
Change-Id: I8fe662d5d3399d60c8a05acc79b4dd51b5d138a6
Test: presubmit
Bug: 183730987
Make sure the shebang of the par file is consistent with the main python
script rather than use /usr/bin/python arbitrarily.
Bug: 185399398
Bug: 185094069
Test: m atest ; atest-dev --help
m atest-py2 ; atest-py2-dev --help
Change-Id: Ice4590b3d173e6d11e608c2ca29b297df0dc4089
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.
It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.
Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.
Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
These packages have already been migrated to use per test build
directory so have no need for a FixtureFactory.
Bug: 183235980
Test: m nothing
Change-Id: I667d1d992caaf0f615de91f89efdae11c44986c2
The test running code would only check errors if there were any
reported. That ignored the "module with duplicate runfile path" test
which was not reporting the expected error.
This change corrects that mistake, and then fixes the test which had
started to fail.
It is possible that the failing test highlights a bug in the python
code but if so that will need to be fixed in a follow up bug.
Bug: 183046274
Test: m nothing
Change-Id: I961b15a452725e7b15ba9d21b4a4a672ba3d004a
This CL adds a simple python_binary_host to native py_binary converter
for standalone modules that don't depend on libs. It also adds support
for the conditional py2/py3 build target based on the version prop.
Test: milestone-2/demo.sh full
Test: bazel query 'kind("py_binary rule", //bionic/...)'
Test: bazel run //bionic/libc/tools:genfunctosyscallnrs
Test: bazel run //bionic/libc/tools:genseccomp
Test: go tests
Fixes: 182236395
Change-Id: Ibe5ec6cd0dc12a61b3a449a8c723d80b891fae42
Add a ctx parameter to AndroidMkExtraEntriesFunc to allow them to
access providers.
Test: m checkbuild
Change-Id: Id6becc1e425c3c3d8519248f8c0ce80777fac7cc
Merged-In: Id6becc1e425c3c3d8519248f8c0ce80777fac7cc
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
MTS is introducing partial MTS test suites that are per-module, with
names of the format: mts-${MODULE}. By centralizing the code for test
suites, we can automatically add "mts" test suite when an
"mts-${MODULE}" test suite is specified, reducing duplication.
Test: m mts
Bug: 170318013
Change-Id: I8ce9d3c252fcc0a937bb5f2826d21cb6c6932d82
This is to enable consolidating test compatibility suite handling in
Soong's AndroidMk files.
Test: go test soong tests
Test: m nothing and compare soong/Android mk files -- only diffs is that
lines matching "^LOCAL_SHARED_LIBRARIES := $" are absent
Change-Id: I4922a1c8b0231cd98ebd2bc7257956eb9c0d240c
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).
Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.
Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.
This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.
Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
csuite has a python module that wants to embed the outputs of java
modules. This has caused issues with mismatched variants bewteen
the arch-specific python module and the common java modules. Add
a java_data property that is similar to the data property but
uses the common arch variant.
Bug: 173977903
Test: m checkbuild
Change-Id: I0f2f0e4159650cd5d42b510d5177678e7ee91b4d
Use a local variation for python version splits. This causes
dependencies from python modules not to look for variations
that match the version split, which will simplify a python module
depending on a java module (for example for test data) because
Soong won't look for a python version variant of the java module.
Bug: 173977903
Test: go test ./python
Change-Id: Ib034140c478ffbc7467ab830d3cfa5683c31d05c
This change allows a test (native, java, rust or python) to be included
in host-unit-tests suite when test option `unit_test` is set to true.
Bug: 172006742
Test: m host-unit-tests
Change-Id: I69d3eb5b51198c549e2e6914ceac3a4fc33c3cf2
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.
Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
A host target is considered as being cross-compiled when the target
can't run natively on the build machine. For example, linux_glibc/x86_64
is a non-cross target on a standard x86/Linux machine, but is a cross
host on Mac. Previously, whether cross or not was a static attribute of
an OsType. For example, Windows was always considered as cross host,
while linux_bionic was not. This becomes a problem when we support more
host targets like linux_bionic/arm64 which should be cross-host on
standard x86/Linux machines.
This change removes HostCross from the OsClass type and instead adds a
property HostCross to the Target type. When a target is being added, it
is initialized to true when the target can't run natively on the current
build machine.
Bug: 168086242
Test: m
Change-Id: Ic37c8db918873ddf324c86b12b5412952b0f2be2
When multiple os/arch variants are supported, java_test_host could not
find a matching arch due to java having arch:common, whereas native
binaries support a specific architecture. This change adds the property
`data_native_bins` in order to support binaries with the appropriate
os/arch variants.
Test: m FirmwareDtboVerification with data_native_bins
Test: forrest
Bug: 153848038
Change-Id: I45adebff0fde2811d5ef5620c697b97b768c951f
This enables python binaries to be included in data properties with the
format ":module".
Test: m FirmwareDtboVerification with local changes
Bug: 153848038
Change-Id: I2a02dc382fcbb142d8e4fbdbb6ca461986627d25
Annotate the test config properties with android:"path", and always
use LOCAL_FULL_TEST_CONFIG to pass the path to Make.
Bug: 157621988
Test: manual
Change-Id: I1c020e642f83d994d63c9e9cb56e686086d5acaa
Give prebuilt_etc and sh_binary their own packages and split the
gigantic main Android.bp up to small, per-package ones.
Test: m nothing, TreeHugger
Bug: 156980228
Change-Id: I7b00cd344b9f16861f1ff39edf0029f016b853d0
We're not doing a great job of copying over all the library dependencies (especially when using sanitizer libraries). Just omit these from the checks, which shouldn't really matter.
Bug: 151093572
Change-Id: I8470bbfbf36a51739e8a7e699ad090a66c9db8b2
Test: treehugger
The filesystem object was available through ModuleContext.Fs(), but
gives too much access to the filesystem without enforicing correct
dependencies. In order to support sandboxing the soong_build
process move the filesystem into the Config. The next change will
make it private.
Bug: 146437378
Test: all Soong tests
Change-Id: I5d3ae9108f120fd335b21efd612aefa078378813
Avoid having to pass ModuleFactoryAdaptor to every call to
RegisterModuleType in a test by wrapping RegisterModuleType.
Test: all soong tests
Change-Id: If8847d16487de0479cc3020b728256922b3cadba
The arch variants are hardcoded in every module type. Refactor
them out into a Target.Variations() method in preparation for
splitting the arch mutator into two, which will require using
different variations.
Test: m checkbuild
Change-Id: I28ef7cd5168095ac888fe77f04e27f9ad81978c0
Add a ToMakePath() method that returns a new path that points out
out/ instead of out/soong/, and replace the
"$(OUT_DIR)/" + path.RelPathString()
pattern with
path.ToMakePath().String()
Bug: 141877526
Test: m checkbuild
Change-Id: I391b9f2ed78c83a58d905d48355ce9b01d610d16
Create a new type InstallPath that is similar to OutputPath to
differentiate intermediates output paths from installed output
paths.
RelPathString is a poorly defined, undocumented function that is
primarily used to get an install path relative to out/soong to
generate an equivalent install path for Make relative to $(OUT_DIR).
Move it to InstallPath for now, and fix the one remaining user on
OutputPath.
Add a method to create an NDK install path so that ndk_sysroot.go
doesn't have to do it manually with PathForOutput.
Bug: 141877526
Test: m checkbuild
Change-Id: I83c5a0bd1fd6c3dba8d3b6d20d039f64f353ddd5
Attribute `auto_gen_config` is added to test modules.
Test config will be generated if:
the attribute is not set and AndroidTest.xml doesn't exists
or
the attribute is set to true, whether or not AndroidTest.xml exists.
Test config will NOT be auto-generated if:
the attribute is not set and AndroidTest.xml exists
or
the attribute is set to false, whether or not AndroidTest.xml exists.
Bug: 141684102
Test: build test module with auto_gen_config set to true
Change-Id: I64fb003a83d8c32a967835e5f8d12fe4476043be
Translated second architectures now go in NativeBridgeArch instead
of DeviceSecondaryArch.
This reapplies I568046330abc002d4eed582cb999b62a5eaba790 with
ctx.Config().HasMulitlibConflict() added to fix the NDK build,
which has arm64, arm, x86_64, and x86 architectures enabled.
Test: m checkbuild
Test: OUT_DIR=out_ndk build/soong/scripts/build-ndk-prebuilts.sh
Test: no change to build.ninja or Android-aosp_cf_x86_phone.mk
Change-Id: Iadcafbd64bfb9579ae7c86914927c43a062b0c8e
This reverts commit fe6330c7d2.
Reason for revert: this was accidentally propping up NDK builds
Bug: 141242600
Change-Id: Ie9f8e31479a74e8e9e14d720a5936da6b1d5d113
Translated second architectures now go in NativeBridgeArch instead
of DeviceSecondaryArch.
Bug: 141242600
Test: m checkbuild
Change-Id: I568046330abc002d4eed582cb999b62a5eaba790
Replace the common pattern of:
cmd.Tool(ctx.Config().HostToolPath(ctx, "tool"))
with:
cmd.BuiltTool("tool")
And similarly for PrebuiltBuildTool.
Test: m checkbuild
Change-Id: I7d63188505362c7df6a3b3e7330b4a2cca5a2409
-P has to come before -D to have an effect on the files read from the
directory.
Fixes: 130160833
Test: atest acloud_test
Change-Id: I62a998f1ad1e3b45f590babbf39330955d368373
android/soong/common was renamed to android/soong/android long
ago, but the pctx package path was still "android/soong/common".
This required all users of rules defined in android/soong/android
to import "android/soong/android" and then
pctx.Import("android/soong/common").
Test: m checkbuild
Change-Id: I20d096522760538f7cfc2bec3d4bfeba99b275d4
Add a proto.plugin property to allow specifying a custom protoc
plugin to generate the code.
Fixes: 70706119
Test: m am StreamingProtoTest
Change-Id: I1ecdd346284b42bbcc8297019d98d2cd564eb94c
Using blueprint.Rule for protoc commands was causing code duplication
because there was no good way to run the same protoc for cc, java and
python but then run custom source packaging steps for java and python.
Move most of the code into a common function that returns a
RuleBuilder, and then let java and python add their own commands at
the end of the rule.
Bug: 70706119
Test: All Soong tests
Test: m checkbuild
Change-Id: Ic692136775d273bcc4f4de99620ab4878667c83a
Move the logic from ctx.ExpandSources into android.PathsForModuleSrc
and ctx.ExpandSource into android.PathForModuleSrc, and deprecate
them. When combined with the pathDepsMutator this will let all
properties that take source paths also take filegroups or genrule
outputs, as long as they are tagged with `android:"path"`.
Test: All soong tests
Change-Id: I01625e76b5da19240e9649bf26a014eeeafcab8f
Add `android:"path"` to all properties that take paths to source
files, and remove the calls to ExtractSource(s)Deps, the
pathsDepsMutator will add the necessary SourceDepTag dependency.
Test: All soong tests
Change-Id: I488ba1a5d680aaa50b04fc38acf693e23c6d4d6d
Current soong use source code's under cts as the rule for judgement.
Should change to use test_suites define instead.
Bug: 124313692
Test: m hello_world_test, make sure test config be auog-enerated.
Modified platform_testing/tests/example/native/Android.bp
m hello_world_test, make sure test config not be auto-generatetd.
Change-Id: I1bc5216f73329d2a82d9ff29ccbede436dd2976c
Mainly so that we can build a `python` prebuilt that acts like the
normal python install, though you could also build different ones with
more packages pre-installed.
Bug: 117811537
Test: move built py2-cmd into prebuilts/build-tools/path/linux-x86/python and build
Change-Id: I21215f6fd3754d89f8c65e1dfeb3f2deea23239f
Uses more python rather than C++, and skips less of Py_Main.
Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
This test adds a number of checks for the visible python state inside a
hermetic par file.
Test: build/soong/python/tests/runtests.sh
Change-Id: I7188d862a309a09623169e1f967bde86707d41af
Add an empty DepsMutator to ModuleBase so it doesn't have to be
implemented on every module that doesn't need it.
Test: all soong tests
Change-Id: I545a832a0dbf27386d3080377a75ea482cd9ce59
Prevent problem for some users using PAR file built by android's
build system which cause by the different site package of python
installed by user themselves.
Bug: 120469915
Test: 1. m atest
atest --help
2. apply aosp/861470
full build
Change-Id: I72c83f9397f0e88eecfcd32806f1746520299447
Soong doesn't wrap everything in bash, but relies on the ninja default
of /bin/sh. We should probably improve that in the future, but for now,
just fix the build.
On Mac, /bin/sh and /bin/bash are both bash, but /bin/sh implies `set -o
posix`, and ignores the `-n` flag.
Test: treehugger
Test: build on a mac
Change-Id: Icf41b1c5a1ce9eb0a56e39e68c433fc80b53620f
Instead of doing these string operations in the C++ code at every
startup of an embedded par, trim and convert the entry point file to a
python module path at build time.
Test: m
Change-Id: I04a6459a80c84a704b8a25a144ab1e19622d6cc3
The zip file format does support scripts/programs prefixed to the
archive, though many of the offsets are supposed to start from the
beginning of the file. Some tools (python and zipinfo) are able to
read zip files with arbitrary prefixes, but others (libziparchive and
zipdetails) fail.
So pass the file to prefix onto the zip file into merge_zips so that we
can set the offsets from the real start of the file.
Test: m sepolicy_tests (runs the embedded python interpreter)
Test: zipinfo out/host/linux-x86/bin/sepolicy_tests
Test: zipdetails out/host/linux-x86/bin/sepolicy_tests
Change-Id: If73d4c2465581f7de5aa47959284ecf2059df091
Otherwise, python will complain the variable has not been initialized,
in case ExtractRunFiles throws an exception.
Test: m
Change-Id: I3d9e3cda5e558436ee4182c44e4345151c031e41
The root cause is we didn't check if the optionalpath is valid or not,
the registerbuildation function directly invoke the path var and
cause the invalid memory error. We just return if the launcher doesn't
exist.
The ctx.VisitDirectDepsWithTag() also handles allowmissingdependency so
we are ok if launcher doesn't exist.
Test: N/A
Bug: b/116698229, b/67510844
Change-Id: I40941079a64d7797ab879fc5edaa29e835b493a0
VisitDepsDepthFirst is almost never correct, as it can't query
dependency tags of multiple dependencies between the same two modules.
Replace VisitDepsDepthFirst with WalkDeps in sanitize.go and
python.go. Also verify the dependency tag before continuing to
recurse to ensure that they don't recurse through genrules and into
unrelated modules.
Test: m checkbuild
Change-Id: I2f7560126f56b51a40ec39dfbdcc18b5891489f7
The problem came from Python libraries doesn't know the information that
we enabled embedded_launcher (only Python binary knows about that). And
we can't simply remove runfiles dir for Python libraries since host
Python mode need this.
Bug: b/80441699
Test: m perf_profo_flames
Change-Id: I73ffc4d7504f95a708ae7cca47bc6c15a673aa31
This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This reverts commit bfb0aed2dd.
Reason for revert: <LOCAL_REQUIRED_MODULES overrides the "required:" property in Android.bp>
Test: m -j sepolicy_tests
Bug: b/80140289
Change-Id: I4e51007a7d4dda69cbf4f31253e9a56d3f5e35c7
