Commit graph

29 commits

Author SHA1 Message Date
Bob Badour
986a839161 Improve flags for compliance tools.
Test: m droid dist reportmissinglicenses

Change-Id: I4090dae3d5d33d1908d67dff31aeee92d2b261da
2022-06-03 12:44:22 -07:00
Bob Badour
c778e4cba4 Support OUT_DIR for license graph.
Bug: 226066987

Test: OUT_DIR=/tmp/outdir m all listshare checkshare htmlnotice rtrace dumpgraph dumpresolutions compliancenotice_bom compliancenotice_shippedlibs
Test: OUT_DIR=/tmp/outdir m /tmp/outdir/target/product/bonito/obj/NOTICE.xml.gz
Test: m all listshare checkshare htmlnotice rtrace dumpgraph dumpresolutions compliancenotice_bom compliancenotice_shippedlibs
Test: m out/target/product/bonito/obj/NOTICE.xml.gz

Change-Id: I6282c647a389c5935d5ce7c79193f86d32c76365
2022-03-23 14:12:12 -07:00
Bob Badour
77570059cd Phony target to report all library names.
Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m nothing reportallnoticelibrarynames

Change-Id: I0f791e2d92b8135d6af3a6c93f0db348fc69f5a1
2022-03-02 22:32:37 -08:00
Bob Badour
e9b38c175a Greater-than zero is more idiomatic.
Test: m all
Change-Id: I6000d937b98c84226a347b69c69b333a15beb355
2022-02-09 15:56:59 -08:00
Bob Badour
49dd4f75ff Add support for a product name as well as title.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m htmlnotice; out/soong/host/linux-x85/htmlnotice ...
Test: m textnotice; out/soong/host/linux-x85/textnotice ...
Test: m xmlnotice; out/soong/host/linux-x85/xmlnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Id5b2ebe3752081e3a89bba1d717cf103f7846043
2022-02-04 14:50:59 -08:00
Bob Badour
682e1bae57 Regularize command-line flags.
All the notice binaries have -title

All the binaries that can -stripPrefix can strip multiple.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m bom; out/soong/host/linux-x85/bom ...
Test: m dumpgraph; out/soong/host/linux-x85/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x85/dumpresolutions ...
Test: m htmlnotice; out/soong/host/linux-x85/htmlnotice ...
Test: m rtrace; out/soong/host/linux-x85/rtrace ...
Test: m textnotice; out/soong/host/linux-x85/textnotice ...
Test: m xmlnotice; out/soong/host/linux-x85/xmlnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I08357bf1adb048abba6563cf3cea6ee6d60405e0
2022-02-02 15:32:50 -08:00
Bob Badour
f87922450e license metadata xml notice files
Introduce the below command-line tool:

xmlnotice outputs a NOTICE.xml file constructed from the license
texts of the transitive closure of dependencies.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m xmlnotice; out/soong/host/linux-x85/xmlnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I267effd3a1c1b981bcdc5e058dce561a2d4a7948
2022-02-02 15:32:50 -08:00
Bob Badour
608bdff0a7 Add support for gzipping html output.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m htmlnotice; out/soong/host/linux-x85/htmlnotice ...

where ... is -o=html.gz followed by the path to the .meta_lic file for
the system image. In my case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

the rest of ... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I7a42d5186876609a401956754e3dcff64211fb15
2022-02-01 15:35:40 -08:00
Treehugger Robot
6ebf02914c Merge changes I40a05866,Id3d6e701
* changes:
  license metadata reverse trace
  Fix copy+paste error.
2022-02-01 01:25:59 +00:00
Bob Badour
c817845ea5 license metadata reverse trace
Introduce the below command-line tool:

rtrace outputs a list of targets and conditions causing one or more
projects or target nodes to require sharing to resolve a restricted
condition.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m rtrace; out/soong/host/linux-x85/rtrace -rtrace=...

where ... is a project or license metadata file followed by the path to
the .meta_lic file for the system image. In my case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

system/core ${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
or
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic ${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I40a0586699d9b8a8dd2bd4ba26756c9649ebf964
2022-01-31 14:16:23 -08:00
Bob Badour
91af68b1e1 Fix copy+paste error.
The tool under test is dumpresolutions not listshare.

Test: m dumpresolutions; out/soong/host/linux-x85/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Id3d6e701ebd3dab03f407116d85a5e2aab8c5c59
2022-01-31 14:14:27 -08:00
Colin Cross
bb45f8c74d Track dependencies when reading notice files
Track which files are read by the notice file indexer and add a flag
to textnotice and htmlnotice to support writing them out.

Bug: 207445310
Test: textnotice_test
Test: htmlnotice_test
Change-Id: Ib74706b8a87a5ed9268a0fe37982ecf89f4e227d
2022-01-31 10:15:10 -08:00
Colin Cross
d0f05c9eda Move compliance commands into subdirectories
Move each compliance command into its own directory to avoid Go tooling
considering all the files to be a single package and complaining about
conflicting global names.

Test: go test ./...
Change-Id: I3db6310e7368bcc7fe6a3704b40a84078ed44209
2022-01-27 17:14:03 -08:00
Colin Cross
38a6193180 Give compliance a pkgPath
Change the pkgPath of compliance to android/soong/tools/compliance
to distinguish it from the standard library.

Also add a go.mod file so that the Go tools can find the package.

Test: go test ./...
Change-Id: I061be2b91378db6508e9c88be2548648af8be4e7
2022-01-27 17:14:03 -08:00
Colin Cross
179ec3e55e Fix errors caught by go vet in compliance package
`go test` implicitly runs `go vet` and fails the tests if vet
errors are found.  Fix all the issues found by vet.

Test: go test build/make/tools/compliance/...
Change-Id: If0684cf124ece4931af440008cd44a61c22de475
2022-01-27 17:14:02 -08:00
Colin Cross
35f79c37aa Run gofmt on compliance
Test: builds
Change-Id: I404b138faac0db9a92201be6163b19d9e6bff810
2022-01-27 17:14:02 -08:00
Bob Badour
2546febca7 license metadata bill of materials list
Introduce the below command-line tool:

bom outputs a text file listing 1 installed path per line.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m bom; out/soong/host/linux-x85/bom ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I73975ca7b161945a62ff83888527ce01fb47d75a
2022-01-26 20:58:24 -08:00
Bob Badour
00c8a38714 license metadata shipped libraries list
Introduce the below command-line tool:

shippedlibs outputs a text file listing 1 library per line containing
the libraries the shipped image is derived from.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m shippedlibs; out/soong/host/linux-x85/shippedlibs ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I98e2c1eec94ad7878e911eee2458a26e12ee2b19
2022-01-26 17:21:39 -08:00
Bob Badour
6ea1457c9b license metadata html notice files
Introduce the below command-line tool:

htmlnotice outputs a NOTICE.html file constructed from the license
texts of the transitive closure of dependencies.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m htmlnotice; out/soong/host/linux-x85/htmlnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Idbbeb2939d8cbf497237516fe468004fcd2d72a1
2022-01-26 16:02:17 -08:00
Bob Badour
e6fdd1403b license metadata text notice files
Introduce the below command-line tool:

textnotice outputs a NOTICE text file constructed from the license
texts of the transitive closure of dependencies.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m textnotice; out/soong/host/linux-x85/textnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Ia691869fd8e58ef008024f48c23b1a4b4435677a
2022-01-26 16:02:11 -08:00
Treehugger Robot
3a0656307c Merge "Regression testcase for shipped vs nonshipped." 2022-01-12 23:35:17 +00:00
Treehugger Robot
710813e871 Merge "Fix typo in doc: code segment for container dot" 2022-01-12 22:31:40 +00:00
Bob Badour
00bf3d9b29 Regression testcase for shipped vs nonshipped.
Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I17221ec8fe429b5f745c0733f8bb1ab0ea2f4567
2022-01-11 16:38:55 -08:00
Bob Badour
99dba55b4b Fix typo in doc: code segment for container dot
Test: not applicable
Change-Id: Ibaadc6e8fd4a267b50a46feb2dd6278ff1f5ace1
2022-01-11 14:46:22 -08:00
Bob Badour
103eb0f9bc Performance and scale.
Defer edge creation.

Don't create edges until the count is known to avoid repeated allocate+
copy operatios.

Limit resolutions.

Allow only a single resolution condition set per target, and overwrite
intermediate results. Reduces memory and obviates allocations.

Propagate fewer conditions.

Instead of propagating notice conditions to parents in graph during
initial resolve, leave them on leaf node, and attach to ancestors in
the final walk. Reduces copies.

Parallelize resolutions.

Use goroutines, mutexes, and waitgroups to resolve branches of the
graph in parallel. Makes better use of available cores.

Don't accumulate resolutions inside non-containers.

During the final resolution walk, only attach actions to ancestors from
the root down until the 1st non-aggregate. Prevents an explosion of
copies in the lower levels of the graph.

Drop origin for scale.

Tracking the origin of every potential origin for every restricted
condition does not scale. By dropping origin, propagating from top
to bottom can prune many redundant paths avoiding an exponential
explosion.

Conditions as bitmask.

Use bit masks for license conditions and condition sets. Reduces maps
and allocations.

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Ia2ec1b818de6122c239fbd0824754f1d65daffd3
2022-01-11 10:40:50 -08:00
Bob Badour
b285515ca1 license metadata remove path on top-down walk
Performance optimization means not every path will be traversed.

Instead of updating parents via the path, perform a 2nd bottom-up walk
after the top-down walk to propagate the new resolutions to parents.

Note: the 2nd walk method will add resolutions to statically linked
libraries etc. at deeper levels, but those do not affect what gets
reported. In particular, note that test data for dumpresolutions
changes, but none of the test data for listshare, checkshare etc.
changes.

Test: m all systemlicense listshare checkshare dumpgraph dumpresolutions

Bug: 68860345
Bug: 151177513
Bug: 151953481
Change-Id: I76361c4e33bbadbbea38cbec260430e8f9407628
2021-12-08 12:52:59 -08:00
Bob Badour
6dd0035e04 compliance package: listshare and checkshare
package to read, consume, and analyze license metadata and dependency
graph.

Includes the below command-line tools:

listshare outputs csv of projects to share to meet restricted and
reciprocal license requirements with one project per line. The first
field is the path to the project, and subsequent fields identify the
license resolutions as colon-separated target:annotations tuples.

checkshare outputs error messages to stderr for any targets where
policy dictates both sharing and not sharing the source-code, and PASS
or FAIL to stdout. exit status indicates success 0 or conflict found 1

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I4ff3f98848f7e6d03a35734300d763ef5f245d53
2021-12-06 16:39:35 -08:00
Bob Badour
1ded0a1b18 compliance package: dumpresolutions
package to read, consume, and analyze license metadata and dependency
graph.

Includes the below command-line tool:

dumpresolutions outputs the resulting set of resolutions after the
bottom-up and top-down resolves, or after joining 1 or more condition
walks.

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I9869400126cd7ad4b7376b0bab31b46aad732f5d
2021-12-06 16:38:08 -08:00
Bob Badour
fa739da020 compliance package for license metadata: dumpgraph
package to read, consume, and analyze license metadata and dependency
graph.

Includes testdata/ and the the below command-line tool:

dumpgraph outputs edges of the graph as "target dependency annotations"

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all
Test: m systemlicense
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I5fe57d361da5155dbcb2c0d369626e9200c9d664
2021-12-06 16:37:17 -08:00