Sandboxing it generates this diff:
38,39c38,39
< #ifndef YY_YY_OUT_SOONG_TEMP_SBOX_794A09CEE4E110D9FF38139A8943928FFD7288A5_OUT_AWKGRAM_TAB_H_INCLUDED
< # define YY_YY_OUT_SOONG_TEMP_SBOX_794A09CEE4E110D9FF38139A8943928FFD7288A5_OUT_AWKGRAM_TAB_H_INCLUDED
---
> #ifndef YY_YY_OUT_AWKGRAM_TAB_H_INCLUDED
> # define YY_YY_OUT_AWKGRAM_TAB_H_INCLUDED
280c280
< #endif /* !YY_YY_OUT_SOONG_TEMP_SBOX_794A09CEE4E110D9FF38139A8943928FFD7288A5_OUT_AWKGRAM_TAB_H_INCLUDED */
---
> #endif /* !YY_YY_OUT_AWKGRAM_TAB_H_INCLUDED */
Which is acceptable, the ifdef is based on the path to the file and
just there to prevent duplicate imports.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py awkgram.tab.h
Change-Id: I85c5e0f65e97d18f1aa8b36fa6b19402d2da6c8c
Sandboxing it generates this diff:
1c1
< #define ANGLE_COMMIT_HASH "1f7a2ce0bf57"
---
> #define ANGLE_COMMIT_HASH "unknown hash"
3,4c3,4
< #define ANGLE_COMMIT_DATE "2023-11-17 17:33:59 +0000"
< #define ANGLE_COMMIT_POSITION 26027
---
> #define ANGLE_COMMIT_DATE "unknown date"
> #define ANGLE_COMMIT_POSITION 0
These constants appear to be unused, and we don't really want the build
to inspect the git history, so just let them be unkown.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py angle_commit_id
Change-Id: I3e35af14d13142927ded9477e975576d7324c6b7
These genrules already work with sandboxing, but bison emits #line
directives that reference filepaths. These paths differ between
sandboxed and non-sandboxed builds, which caused
genrule_sandboxing_test.py to think that they didn't work with
sandboxing.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py emp_ematch.yacc.c emp_ematch.yacc.h
Change-Id: I85ed0f80dee7997af6b08a37b12e9c0ad0bd8386
These genrules already work with sandboxing, I'm not sure why they
were added to the list.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py c2hal_test_genc++_headers c2hal_test_genc++
Change-Id: I697c9cff1db0bf71b3608684fde73535a72f71b2
These genrules already work with sandboxing, but they write the path
to a tool into a comment in their outputs, which differs between
sandboxing and non-sandboxing builds. This made genrule_sandbox_test.py
think that they didn't work with sandboxing.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py libchrome-include libchrome-crypto-include
Change-Id: Ibcad839e9d374a2f992d051805548c58303cf7ef
r8retrace-run-retrace appears to already work with sandboxing.
r8retrace-dexdump-sample-app gets a diff in the output files due to
differing paths in sandboxed/non-sandboxed environments, but the
meaningful content is still the same.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py r8retrace-dexdump-sample-app r8retrace-run-retrace
Change-Id: Ice0eeb762d35a4b32e8fb6d480ecf6d38959d0cc
These version files appear to be unused, and we don't really want
to support having the build access the git history.
Bug: 307824623
Test: Presubmits
Change-Id: Id5700bf4a56955bdf6edd4c50ceefa4184f54555
The jdepsGeneratorSingleton can get the module path directly, it doesn't
need to be collected by each module type that implements IDEInfo. Fixes
module types (like android_library) that didn't reach the code that
collected the path.
Bug: 309835196
Test: out/soong/module_bp_java_deps.json contains path for ExtServices.core
Change-Id: If8cb81b4f708e0367f156ade164bee253bf53492
These work fine with sandboxing, I'm not sure why they were added
to the list.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py egl_extensions_functions_hdr egl_functions_hdr gles1_core_functions_hdr gles1_extensions_functions_hdr gles2_core_functions_hdr gles2_extensions_functions_hdr gles31_only_functions_hdr gles3_only_functions_hdr
Change-Id: Ib179322f5b828dc85fccf9c0d1bacaad3fd359bf
These genrules are nondeterministic even without sandboxing, which
caused genrule_sandbox_test.py to think that they didn't work with
sandboxing.
Bug: 307824623
Test: m aidl_camera_build_version apexer_test_host_tools authfs_test_apk_assets common-profile-text-protos core-tests-smali-dex futility_cmds gen_corrupt_superblock_apex gen_manifest_mismatch_apex_no_hashtree generate_hash_v1 lib-test-profile-text-protos libmojo_jni_headers measure_io_as_jar PackageManagerServiceServerTests_apks_as_resources pandora-python-gen-src sample-profile-text-protos services.core.protologsrc statsd-config-protos temp_layoutlib vm-tests-tf-lib vts_vndk_abi_dump_zip wm_shell_protolog_src wmtests.protologsrc
Change-Id: I289decc2ac85d45b4c0f930171145553e477b1dd
These genrules have non-deterministic outputs even without sandboxing.
Though I couldn't easily figure out where the nondeterminism is coming
from.
Bug: 307824623
Test: Presubmits
Change-Id: Ia60c6fae164a1f66cad2e19ccaab7d178905f2a4
Like in aosp/2825629, these genrules right the command used to
generate their files to the files themselves, which differs between
sandboxed/non-sandboxed invocations and caused genrule_sandbox_test.py
to think that sandboxing caused meaningful differences.
Bug: 307824623
Test: Presubmits
Change-Id: I248de8a45ab03a498297a026250d6f0b7c838e25
These genrules work with sandboxing already, they just had
non-determinism that lead genrule_sandboxing.py to think that they
didn't.
Non-determinism sources include:
- HeadlessBuildTimestamp literally just generates a header file with
a timestamp, not much we can do here other than re-architect their
code.
- Pdlc has a non-determinism issue I sent out a cl to fix:
https://github.com/google/pdl/pull/85
- Python tools write the command they were ran with to generated files,
and non-embedded-launcher python scripts have non-deterministic
paths. Switched to embedded_launcher: true: aosp/2825231
- In addition, the path to the genrule sandbox is not
non-deterministic, but it is a hash of the inputs to the sandbox.
When running genrule_sandbox.py, it compares a "partial" sandbox
(which only sandboxes tools) to a "full" sandbox, and these two
runs have different sandbox hashes.
Bug: 307824623
Test: Presubmits
Change-Id: Ib966262dc1aac99a0798f26d8a03966cc97fe1bf
go/roboleaf-busy-beavers-sandboxing
Most of these genrules work out of the box with sandboxing, I'm not sure why they were added.
However hidl_hash_test_gen needs a fix, and hidl2aidl_translate_cpp_test_gen_src produces
nondeterministic results even without sandboxing.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py hidl2aidl_test_gen_aidl hidl2aidl_translate_cpp_test_gen_headers hidl2aidl_translate_cpp_test_gen_src hidl2aidl_translate_java_test_gen_src hidl2aidl_translate_ndk_test_gen_headers hidl2aidl_translate_ndk_test_gen_src hidl_cpp_impl_test_gen-headers hidl_cpp_impl_test_gen-sources hidl_error_test_gen hidl_export_test_gen-headers hidl_format_test_diff hidl_hash_test_gen hidl_hash_version_gen hidl_java_impl_test_gen
Change-Id: Ia865ba3ed9b1ede21b440c3b640fcdb5d7661c9d
gen_ltp_config used to read files from the source tree,
but this cl refactors it to package all the files it
needs into the tool itself, so it's more hermetic and
all the genrules don't need to explicitly list the files
the tool needs.
go/roboleaf-busy-beavers-sandboxing
Bug: 307824623
Test: diff'd the files produced by the ltp_config_* genrules before/after this change
Change-Id: Ia72084965dcb8659394068b7c6877adb1c882dc2
With the other cls in this topic, the modules build with sandboxing.
They still fail the genrule_sandobox_test.py because their builds are
non-deterministic though. I wasn't able to find the cause of the
non-determinism, so relying on presubmits to make sure nothing breaks.
Bug: 307824623
Test: m com.android.apex.test.bar_stripped com.android.apex.test.baz_stripped com.android.apex.test.foo_stripped com.android.apex.test.pony_stripped com.android.apex.test.sharedlibs_generated com.android.apex.test.sharedlibs_secondary_generated with sandboxing enabled
Change-Id: I4006732bf1ba08e846dee2e18d89dbf45f2cd7ba
These modules actually work fine with sandboxing, they were just added
to the list because they have non-deterministic outputs, which caused
the genrule sandboxing script to think that sandboxing affected their
outputs.
Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py UpdatableSystemFontTest_NotoColorEmojiV0.sig UpdatableSystemFontTest_NotoColorEmojiV0.ttf UpdatableSystemFontTest_NotoColorEmojiVPlus1.sig UpdatableSystemFontTest_NotoColorEmojiVPlus1.ttf UpdatableSystemFontTest_NotoColorEmojiVPlus2.sig UpdatableSystemFontTest_NotoColorEmojiVPlus2.ttf with aosp/2816385
Change-Id: I07b1df779c8d47ad875a4fc2d3af5e46945cff83
The denylist was built by running `genrule_sandbox_test.py` on all
modules in the tree. `genrule_sandbox_test.py` checks that the sandboxed
genrules build, but also that they get the same results as the
unsandboxed version.
In this case, the genrule actually builds with sandboxing just fine,
but they have non-deterministic results, which caused
`genrule_sandbox_test.py` to think that they didn't work with
sandboxing.
Test: m com.android.apex.apkrollback.test.pem com.android.apex.apkrollback.test.pubkey com.android.apex.cts.shim.debug.pem com.android.apex.cts.shim.debug.pubkey com.android.apex.cts.shim.pem com.android.apex.cts.shim.pubkey com.android.apex.cts.shim.v2_no_pb com.android.apex.cts.shim.v2_signed_bob com.android.apex.cts.shim.v2_signed_bob_rot com.android.apex.cts.shim.v2_signed_bob_rot_rollback com.android.apex.cts.shim.v2_unsigned_apk_container com.android.apex.cts.shim.v3_signed_bob com.android.apex.cts.shim.v3_signed_bob_rot com.android.apex.cts.shim_not_pre_installed.pem com.android.apex.cts.shim_not_pre_installed.pubkey com.android.apex.rotation.key.bob.pem com.android.apex.rotation.key.bob.pk8 com.android.apex.rotation.key.bob.rot com.android.apex.rotation.key.bob.rot.rollback com.android.apex.rotation.key.bob.x509.pem com.android.overlaytest.overlaid.pem com.android.overlaytest.overlaid.pubkey com.android.overlaytest.overlay.pem com.android.overlaytest.overlay.pubkey
Change-Id: I950767449025163d8c71bb5a7b2e2f15a1ce4a84
Contingent on aosp/2788322
Test: GENRULE_SANDBOXING=true m libperfetto_client_experimental # bit
identical
Change-Id: I885f420850a165f042b94685e7cf1215d5620716
I'm not sure how this was missed earlier.
Bug: 290816499
Test: build/soong/tests/genrule_sandbox_test.py -t sdk_phone_x86_64 all with aosp/2666142
Change-Id: I8d21a34e3b13ac568fa6153a31c43ba3e4d516bd
These were added with the android 14 release.
Bug: 290816499
Test: run genrule_sandbox_test.py with a local change to check all genrules in the tree
Change-Id: Icf3627c245638ab3f73b83e24ef04c916d7ab58b
Bug: 290816499
Test: run genrule_sandbox_test.py with a local change to check all genrules in the tree
Change-Id: Id18b801c2306dd59b5b593b004b513b578ce3705
This no-op refactoring facilitates some upcoming functional changes for
"bp2build allowlist v2". The work requires that the bp2build conversion
mutator be changed from a TopDown mutator to a BottomUp mutator.
Refactoring all bp2build-related methods so that they use Bp2buildMutatorContext
makes it easier to make this functional change without touching tens of
files and multiple projects.
Bug: 285631638
Test: m bp2build
Change-Id: I3d1ef3064146e959c6f0dc315350fc9764bf2bd2
Instead, we return an error. This allows us to access some product
variable information earlier when it will not be used as an attribute
without panicing
Test: m nothing
Change-Id: Id094b2b9e1364a8d174d99b3824fa149fb235b3e
According to go/roboleaf-busy-beavers-sandboxing, this field should just take in any data files needed by the tool. The existing description makes it sound like this property should only contain a single file that will be used as the tool itself.
Change-Id: I3ef3b8ceb52f7a7e6de9e0a897d5cc05c9c2d336
So that users can use soong config variables / product variables
to adjust a genrule's command.
Bug: 295910468
Test: m nothing
Change-Id: I9fedf8d5d52e515c3fdb913411ce1b3fecb7ba81
If `srcs` contains a gensrcs/genrule module, the current bp2build module
will put it in the catch-all `srcs` attribute. This is reserved for .cpp
sources, so if the genrule produces a .proto/.aidl/... file, this will
fail.
This handles genrules that produce .proto files. To implement this, this
creates an additional partition that detects if the other module is a
genrule/gensrc that produces .proto files. If true, it will append it to
the proto partition.
This CL does not handle
- genrule that produce .c/.aidl/.yacc/.... files. They will continue to
be partitioned into the catch-all partition
- java modules
Test: unit tests
Test: TH
Bug: 293205700
Change-Id: Ib720fdf3a053ff5dd256e6faa632e3fa7776966d
Embedding multiple includes from a genrule may be difficult to read,
instead we generate a header library that contains the headers and all
include dirs that Soong generates.
Test: go test bp2build tests
Change-Id: I590c74c133f015f27cccf5a2fd916153ad9c125e
Bug: 290816499
Test: run genrule_sandobx_test.py with a local change to check all genrules in the tree
Change-Id: I258fe11640c71d532ef48ed88270dec72bd69814
Previously, genrule export_include_dirs always added ModuleDir to
exported include dirs when export_include_dirs is set but not when
export_include_dirs is not set. Now when export_include_dirs is set, we
also export the directory without the additional ModuleDir subdir.
Test: genrule go tests
Test: set export_include_dirs and test
Change-Id: I46e860b2c20c1a96bddd14367d7fa737d901994d
When "write_if_changed: true" is set, it will call restat for ninja.
With this option the output file will be copied only if it is changed.
Bug: 290130959
Test: ninja rule include "--write-if-changed"
Change-Id: I8bd77b43b22eb0115e0bdc73718b2d6997d92652
With the other change in this topic,
`OpenwrtControlServerProto_h` and
`OpenwrtControlServerProto_cc` both successfully build with sandboxing
on.
Test: build/soong/tests/genrule_sandbox_test.py --show-diff OpenwrtControlServerProto_h OpenwrtControlServerProto_cc
Test: GENRULE_SANDBOXING=true m OpenwrtControlServerProto_h OpenwrtControlServerProto_cc
Test: GENRULE_SANDBOXING=true m droid
Change-Id: Id56824ed935e1d16d2333e5a1aeac248cdfcaeb6
This enables sandboxing for inputs that are necessary but do not need to
have a source file generated.
Test: GENRULE_SANDBOXING=true m framework-javastream-protos \
framework-cppstream-protos
Change-Id: Id5ca1dab5799c25fa96b564a7d2008c2e7b5382b
Soong does not enforce apex_available on the contents of test apex. To
prevent special-casing test apexes in the apex validation aspect in
Bazel, drop the test apexes from the tags altogether.
( The core problem I am trying to solve is making sure that stub
libraries in Bazel have a single apex available. apex validation happens
to be a nice side benefit)
Bug: 277651159
Test: go test ./bp2build
Change-Id: Ibb3cfedb5c0f2cda0464bf3758c70b67cb5885d1
Also disallow arch variant of genrule.out.
This is to be consistent with bazel where we are migrating to.
Bug: 254114674
Test: Manual
Change-Id: I685a2e64102b7bb68128b39931f0bc85878bc6de
When gensrcs's srcs prop is set with filegroup from external package and especially when gensrcs generates cpp headers, it's not trivial to know what the output paths and their correponding include paths look like.
In mixed build, there are a few bugs in gensrcs's include paths for cpp headers (b/248555356 and b/248554581) that we'll eventually need to fix. These tests serve as an documentation of the existing behavior when gensrcs generate cpp headers.
Test: go test
Bug: 248555356
Change-Id: I10168dd4229be8f110a31955d214ef792c8050de
`statslog.cpp` with cmd `"$(location stats-log-api-gen) --cpp $(genDir)/statslog.cpp"` produces `out/.intermediates/frameworks/proto_logging/stats/stats_log_api_gen/statslog.cpp/gen/statslog.cpp`. Hence, $(genDir) is equivalent to `<package-dir>/<module-name>/gen`.
Currently, bp2buld converts `$(genDir)` to `$(GENDIR)`. In Bazel, `$(GENDIR)` is only the base of the generated code (e.g. `bazel-bin`).
```
genrule(
name = "statslog.cpp",
cmd = "$(location :stats-log-api-gen) --cpp $(GENDIR)/statslog.cpp",
outs = ["statslog.cpp"],
tools = [":stats-log-api-gen"],
)
```
produces `bazel-bin/statslog.cpp` but expects to have `bazel-bin/frameworks/proto_logging/stats/stats_log_api_gen/statslog.cpp`.
By converting to `$(RULEDIR)` which is `bazel-bin/frameworks/proto_logging/stats/stats_log_api_gen`.
There had not been any genrule module allowlisted with genDir
yet. So this should not cause any issue with modules converted before
this CL.
Bug: 247536535
Test: go tests
Test: presubmit builds and tests
Change-Id: I65c6aafadab6b180b7ef700427e041547ae7e98a
'$' cannot be included in the genrule source file name as it is an
invalid character.
One workaround to include a file with '$' in the filename is to use
glob(*) and match pattern.
However, shell currently evaluates the '$' sign and leads to unexpected behavior.
This change fixes the issue by shell-escaping the filepath in generating
build actions.
Test: m
Bug: b/194980152
Change-Id: I6fd919c568b5b6526e4de5155104a08ecadab307
All existing gensrsc modulePartners who use the property can use BUILD_BROKEN_DEP_FILE to bypass the error
Test: CI
Bug: 179452413
Fix: 179452413
Change-Id: I7cd39484b43eba693d79188b9a374f192198f90f
This also introduces a workaround for the fact that
apexer depends on aapt2, but aapt2 doesn't build
with bp2build yet. Aapt2 is removed from apexer's
requirements during bp2build.
Bug: 204244290
Test: ./build/bazel/ci/bp2build.sh
Change-Id: I837597ce035c7d5c06e1a3957166583a7a94b5c7
This large refactoring has both immense performance implications and
improves mixed builds complexity / usability. Summary:
1. Queueing calls to Bazel is done in a new mutator instead of a full
soong_build pass. Normal soong_build flow is interrupted (via a
functional hook in blueprint) to invoke bazel and parse its response.
2. Implementing mixed build support for additional modules is as simple
as implementing MixedBuildsBuildable. In this interface, define the
request that must be queued to Bazel, and then subsequently define
how to handle the returned bazel cquery metadata.
3. Mixed builds consists of only a single pass. This greatly
improves mixed build performance.
Result:
A 33% runtime improvement on soong analysis phase with mixed builds.
Caveats:
C++ BazelHandler handling still remains a bit of a mess; I did what
I could within this CL's scope, but this may require additional cleanup.
Test: Treehugger
Test: Verified that aosp_arm ninja file is bit-for-bit identical with or
without this change.
Change-Id: I412d9c94d429105f4ebfafc84100d546069e6621
While Bazel genrules will allow genrules to accept a directory as input,
the results can be unexpected to a user as changes to the contents of
the directory may not trigger a rebuild as expected. Restricting this
in Soong ensures that converted targets will behave as expected.
Test: CI
Change-Id: I8616f58d1df267005e6c0ca3f4730d06de52c0d9
