Test: Built AFL fuzzers individually and built all using haiku command
and built libfuzzers individually and also by using haiku command. Ran
selected fuzzers manually to ensure fuzzing still worked.
Description: Previously we needed to add cc_afl_fuzz to build an afl fuzz binary,
however, to turn current libFuzzers into AFL fuzzers this would required
an update to each Android.bp file which is a lot of work, and would also
require an approval from each Android.bp file owner, which is even more
work.
To get around this (and also to match how AFL fuzzers are built in G3)
we will build AFL++ fuzz binaries by command line option FUZZ_FRAMEWORK.
When FUZZ_FRAMEWORK=AFL is set, all cc_fuzz modules will be built
for AFL rather than libFuzzer. Devs can also specify if a cc_fuzz module
is only for libFuzzer or AFL by using fuzzing_frameworks. If
fuzzing_frameworks is left blank then it will be assumed that the
cc_fuzz module can be built for all available fuzzing frameworks.
Change-Id: Ia7a8224627f2de61606b410d1d1a56d7bdc0955f
Each conversion required defining a separate mutator, which will each
operate on _all_ modules and requires each to repeat checks whether the
mutator should operator. Instead, we introduce a single mutator and
modules can define a ConvertWithBp2build to implement bp2build
conversion for that module.
Test: bp2build.sh
Bug: 183079158
Change-Id: I99d4b51f441c2903879092c5b56313d606d4338d
libc's native_bridge shared variant needs to be installable: false,
make the installable property arch-variant and allow the static and
shared clauses to override it.
This relands I8e0eaed2169e89520b07bb0326bb97ac0b4a70ac with fixes to
allow only the shared variant to be marked installable: false.
Bug: 204136549
Test: m checkbuild
Change-Id: Ie579b5ba56ca21f4d9243088afd5078d8c122a68
Rust fuzzers were not packaging up their CC shared dependencies.
This would lead to fuzzers using the shared libraries included on
system, which may not be sanitized, leading to incorrect behavior.
This refactors the relevant code from CC and calls it from the Rust
build logic.
Bug: 202282599
Test: output rust fuzzer zip file includes shared dependencies.
Change-Id: I92196eb0141733797a67eae24f8e9aedea94c3bc
Without this, any libraries with vendor: true set won't be included.
Bug: 197917632
Test: make haiku, run several fuzz targets on a device
Change-Id: Icab8d3f17d1f07f49940ee0ea52d9312a925affa
Rust and cc fuzz packaging have common. This change is to put this
common methods and fields into fuzz_common.
Test: make haiku and make haiku-rust ran locally and verify corpus,
dict and config are packaged along with the executable.
Change-Id: I04bf535cd2597ed8b2bd3d8e6ea8c28f38bdc18b
Dedups the many repetitions of the code to obtain a *Prebuilt from a
module.
Bug: 177892522
Test: m nothing
Change-Id: I1ededbe9ee79e89ea6dd8882dfee4be0bf0b51b7
Using $out.rsp as the rsp file adds extra complexity around keeping
the $ unescaped. Make callers to FlagWithRspFileInputList provide
an explicit path for the rsp file instead.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I3f531d80c1efa8a9d09aac0a63790c5b11a9f0c6
libicu_* fuzzer depends on installable cc modules and non-installable
cc modules with the same name, because the non-installable modules are
stubs used in the compile-time, and shouldn't be used in runtime.
Bug: 175583039
Test: make haiku dist
Test: Run 2 arbitrary binaries from out/dist/fuzz-target-x86_64.zip on device
Test: Run 2 arbitrary binaries from out/dist/fuzz-host-x86_64.zip on host
Change-Id: I0ecf24bb83e3985da3b8f393adab8c87ab7400c6
Adds a PlatformSanitizable interface which both CC and Rust can
implement so that the sanitizer mutators in CC can sanitize Rust
shared/static libraries appropriately.
Bug: 147140513
Test: m nothing
Change-Id: Ib31103b6c4902a4d5df2565c0d7c981298d100a3
Bug: 176998722
Test: built fuzz target with these options specified, checked resulting
config.json
Change-Id: Id38668dd11e76eead4ec87bd225ba67e3b30b80d
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
The android.WriteFile rule takes careful escaping to produce the
right contents. Wrap it in an android.WriteFileRule that handles
the escaping.
Test: compare all android.WriteFile outputs
Change-Id: If71a5843af47a37ca61714e1a1ebb32d08536c31
The stubs methods are very specific to cc for now, move them out
of LinkableInterface so they are not shared with rust. Instead,
create a cc.Module.library field that contains the libraryInterface
to simplify calling libraryInterface methods on cc modules.
Test: all Soong tests
Test: no change to Soong outputs
Change-Id: I0289d866ce1f7a765631fe3101a62b1b4988ba1c
Add vendor_ramdisk_available and vendor_ramdisk attribute to
various rules. When a vendor_ramdisk variant of a module is
generated, it is installed to $OUT/vendor-ramdisk.
It is similar to a ramdisk image.
Test: m nothing -j
Change-Id: Ib2d16459f3094dbe21c3bdb7c016cb4b2bf62765
This will be used to track who should be recognized for any CVEs
resulting from bugs found by a fuzz target.
Bug: 145745999
Test: n/a
Change-Id: I21e065ab8013d013b1d9f42981c729b996163387
Add a -r argument to soong_zip that reads a list of files from a file
like the -l argument but treats it as a Ninja rsp file with escaping.
Replace the -l arguments in Soong that are using rsp files with -r.
Fixes: 162435077
Test: TestReadRespFile, TestZip
Change-Id: I4605312e99406ab1bd0c37af9c5ad212393f0403
cc_fuzz relies on an invariant that's not exactly true. We assume that
for each fuzz target, we'll only have a dependency on a single sanitized
variant of a shared library. In a few instances, this is proven not to
be true, as we end up with a transitive dependency on a shared library
with sanitizer coverage instrumentation, and one without sancov.
This results in breaking the packaging for some fuzz targets. This then
goes on to break `make haiku` in some scenarios.
While this isn't a completely technically correct solution (as we
basically resolve one of the sanitized variants pseduorandomly), it does
resolve the issue for now. Realistically, we should select *both* of
them, and set the DT_RUNPATHS on the shared libraries to point to the
dependencies that have the sanitization that they're expecting. In
practice - this shouldn't break sancov (we might just silently drop some
coverage) or hwasan (we might just silently drop some hwasanification).
I believe that the walk order of VisitDirectDeps is deterministic, and
as such this shouldn't affect the reproducability of fuzz target builds
(and thus won't blow up the Soong rebuilds). ccross@ or dwillemsen@ can
speak better to this than I can though.
Bug: 148306195
Bug: 151102177
Bug: 155123587
Test: lunch flame_hwasan-userdebug && make haiku
Change-Id: I8d4001d93da33e4e5d21f740beb88a20fcc26e2a
`make haiku` shouldn't build fuzz targets that are disabled on the
continuous fuzzing infra. This should reduce pressure on the repackager.
Bug: N/A
Test: make haiku && verify that example_fuzzer isn't there.
Change-Id: I6a687a3ab8cbb1eb75113e96aba2c4f9f8058010
cc_fuzz reexports transient dependeny DSO's to /data/fuzz through Make.
We intentionally export the non-stripped variants so that ClusterFuzz
can use the symbolized variant, but we don't re-export to
/symbols/data/fuzz. This means that tools like `stack` and
`hwasan_symbolize` can't pick up the symbols and don't know what to do.
Fix this by re-exporting to /symbols/ as well.
Bug: N/A
Test: make example_fuzzer && ls
$ANDROID_PRODUCT_OUT/symbols/data/fuzz/arm64/lib
Change-Id: Id0343c95a0a83e16e6f67f29ff6361fb4d757c05
This change is to add granularity for disabling
of the fuzzer from running on host, device or both.
Bug: 145824604
Test: make haiku successfully completes after change
Change-Id: I686b439e36c622108f7b5b3e3f4738b8295320a7
"data" field specifies data dependencies that will be installed in
fuzzer's output directory.
"data" behaves similar to "corpus", except "data" preserves directory
structure, e.g.
data: ["foo/bar.txt"]
is installed into
$OUT/data/fuzz/<arch>/<target>/data/foo/bar.txt
Test: build a fuzzer with data depenency, check data is installed
correctly
Change-Id: Ia1255026278435181b6d93f91f8f9ad39c96d07f
cc_fuzz ignored soong namespaces, and built all cc_fuzz modules into the
architecture package. This caused naming conflicts where two modules had
the same name but were in different namespaces.
Now, cc_fuzz will only build modules that are in the same namespace as
the product definition - resolving the duplicate definitions error.
Bug: 145240569
Test: m haiku
Change-Id: I0a0acd139426301586ac5e3aba8ce5527cb7a1b5
Use lists instead of maps to make the order and contents of fuzz
packaging rules reproducible.
Test: m nothing; touch Android.bp; m nothing
Change-Id: Id11323f7528e594370c391bfe82eb7367bc1bcd5
RPATH needs to be set to traverse the local directory's lib folder for
fuzz target packages.
Bug: N/A
Test: m fuzz
Change-Id: Idea4f04ce4c55315df7e808de5b2b4f06a88d82a
Currently, prebuilt shared libraries fail the linkable.CcLibrary()
check, as the module returned by NewPrebuiltSharedLibrary uses a
prebuiltLibraryLinker as the module.linker, rather than a
libraryDecorator (which is the check by CcLibrary()).
We also need to ensure that we discard LLNDK stubs libraries, so we
manually categorise and discard them as well. They unfortunately are are
cc.Modules that aren't CcLibraries, as they use a custom linker object
as well (stubDecorator).
Fixes: 144415986
Test: m fuzz
Change-Id: I3b85ef66d1602cb8c035a0a90bddf30674e2eb71
Instead of writing all fuzz target files into a huge zipfile (by number
of entries), we instead write each fuzz target and its files to a
zipfile, then coalesce them into a big zipfile.
This should simplify the repackager significantly, as no unzipping and
repackaging should be needed (just renaming).
This also solves b/144418402, where we are very close to hitting the
POSIX args limit on the soong_zip command for the device fuzz package,
due to the line length generated by all the entries in the huge zipfile.
Fixes: 144418402
Test: make fuzz
Change-Id: I89797e1bea2691ff4a33a227306f2f9083d8af6e
Change the ClusterFuzz package built by 'make fuzz' to now package
shared libraries per-fuzz target.
Changes documented in go/proposed-clusterfuzz-package
This has the impact of:
- This package is much easier to post-process in the Haiku export
cronjob.
- Shared libraries are now per-fuzz-target. This means that we will
have some level of duplication (libdl, libc, libc++, libclang_rt.*,
etc.) as these libraries will be in multiple fuzz target's dirs.
This *only* affects the fuzz-$target-$arch.zip file, not the fuzz targets in
$ANDROID_HOST_OUT/fuzz or $ANDROID_PRODUCT_OUT/data/fuzz. These two
install directories still use the global shared libraries in order to
preserve space on devices.
Bug: 144360679
Test: m fuzz
Change-Id: I5f19a13070e01d2d39598f9f215a983b398a9f37
Some fuzz target dependencies are generated at Soong-time in a
nondeterministic fashion. When these are interpreted by Make, they may
cause rebuilds due to being in a different order.
Ensure that the dependencies are always sorted, which should avoid
triggering unnecessary rebuilds.
Bug: 144250431
Test: make <module> && make <module>, ensure no rebuild.
Change-Id: I44edc1a3bb3f57be3e3f093f765241b41928fa4d
Additional context (for Googlers): go/android-fuzzing-shared
This patch adds the Soong frontend for shared library fuzzing. We
traverse dependencies at soong install time to find all transient shared
libraries that $module depends on. We then ask the Make backend to
depend on the shared library.
We also create the source:destination mappings between where the shared
libraries are built to where they should be installed to for fuzzing.
This is then depended on by the Make backend.
Bug: N/A
Test: m fuzz, note the contents of $ANDROID_PRODUCT_OUT/data/fuzz/lib,
and out/soong/fuzz-target-*.zip now has shared libraries.
Change-Id: Id7afbd34bc9c055110af96cd3c668b730d404aee
This is causing some build errors because the config.json file isn't
associated with a rule (because the rule looks for .txt).
Test: ran locally, checked output directory
Bug: 142551000
Change-Id: Idcdb226fa2d23055a8850b2a0545202921edc3b1
Unstripped binaries allow for online symbolization, and offline
symbolization using the debug info in the binary. Debug information
isn't stripped for host binaries, but is for target. Target should also
contain debugging info.
Bug: N/A
Test: m fuzz && # Check that fuzz targets in the fuzz-*.zip package have
debug info.
Change-Id: Ibd3002674d519e927340a50dfdfbf44f2d809d58
Adds the ability for rust modules to be compiled as C libraries, and
allows cc modules to depend on these rust-generated modules. This also
means that soong-rust should not have any dependencies on soong-cc aside
from what's required for testing.
There's a couple small fixes included as well:
- A bug in libNameFromFilePath that caused issues when library's had
"lib" in their name.
- VariantName is removed from rust library MutatedProperties since this
was unused.
Bug: 140726209
Test: Soong tests pass.
Test: Example cc_binary can include a rust shared library as a dep.
Test: m crosvm.experimental
Change-Id: Ia7deed1345d2423001089014cc65ce7934123da4
Specifically, this adds:
* Owner
* Disable (stop fuzzer from running in Haiku)
* Bug Component
* Bug Hotlist
The fields are all inside a new 'options' struct.
The values from these fields (if any) are written into a config file as
json.
Bug: 142551000
Test: ran locally with a modified build file and verified output in .zip
Change-Id: I86edf74c2cebe9912ac0ad203f99028be4062c8b
Re-plumb the fuzz corpora through an intermediates directory.
Previously, due to the way that LOCAL_TEST_DATA is implemented, we would
break when corpora were included:
a) outside of a corpus/ directory, or
b) not directly under the module directory (i.e. my_module/subdir/corpus)
Corpora are now written into an intermediates directory so that they
play nice with LOCAL_TEST_DATA, because unfortunately the key-value pair
required there is not as simple as source:destination.
Test: m example_fuzzer && ls \
$ANDROID_PRODUCT_OUT/data/fuzz/arm64/example_fuzzer/corpus
Bug: N/A
Change-Id: I5888b06ed349781c6712f1ae77cc3582283a9552
If fuzzer coverage sanitized shared libraries are not deployed - we shouldn't
break users during dynamic link time. I need to think more about how we
should solve this problem in earnest, but for now let's just disable
static linkage of libc++ on device.
Bug: 142671952
Test: Cherrypick ag/9550833 and ag/9544155, make and run the fuzz target
there.
Change-Id: I39dfd6ba314e7a610ef3f0b30f35383a5e3a1e4b
Corpus files called 'FOO' incorrectly go into a subdirectory in the
exported zipfile, e.g.:
my_fuzzer/corpus/FOO/FOO
This patch removes that extra, unnecessary intermediate directory,
e.g.:
my_fuzzer/corpus/FOO
Bug: 142432347
Test: m fuzz && unzip -l out/soong/fuzz-host-x86_64.zip
Change-Id: Id23d4dd2d8322a8ad0234f8acb6878ea4dc3c6e4
Add the fuzz packages to dist. This requires that the phony for 'make'
be define in make itself, rather than in Soong. See comments on other
patch in topic for more information.
Bug: 141026328
Test: m fuzz dist
Change-Id: I4cd476adcfa42e4b40911c4a32427a7b00c67cc3
Adds the `make fuzz` build rule. This offers a few features on top of
`make haiku`:
1. Identifies all fuzz targets through the build system, rather than
having to maintain a golden file.
2. Packages now exist in out/soong/ and are packaged on a per-architecture
basis. Packages also now contain fuzzer seed corpus && dictionary.
3. This gives us further extension options:
- Packaging shared libraries for shared library fuzzing.
- Presubmit tests for fuzzing. Each fuzz target could be made into a
regression test by adding the crash testcase to the corpus. We can
also ensure that example_fuzzer finds a bug with an ASan report as a
presubmit smoke test.
Bug: 141026328
Test: m fuzz && unzip -l out/soong/fuzz-$ARCH.zip
Change-Id: I7aaad616d6b194a3beaf908241a9817df9dfdce1
