Commit graph

75 commits

Author SHA1 Message Date
Ibrahim Kanouche
397a9cf430 Update generateSPDXNamespace to generate a unique spdx doc namespace
Test: m compliance_sbom
Change-Id: Icc1e5aec0e1758ee32c43cfa79b5b49b3d1d9d30
2023-04-20 17:43:12 +00:00
Ibrahim Kanouche
8c745a436b Added validation to SBOM generator
Test: m compliance_sbom
Change-Id: I1ff2dfbc48361cfb785c64306112bc687ca51057
2023-04-13 16:10:12 +00:00
Ibrahim Kanouche
f89fc4aa9b Added Document Fields to SBOM generator
Added functions to generate unique spdx doc namespace and generate a
clean document name

Test: m compliance_sbom

Bug: 265472710
Change-Id: I86ea9ddf50d066e139b757e8a093f98b8df8c81f
2023-04-04 18:53:10 +00:00
Ibrahim Kanouche
91f2f9d851 Revert "Revert "Updated SBOM generator module to generate JSON spdx utility bill of""
This reverts commit 928ee9d9f7.

Reason for revert: Fixed the initial cause of the revert. Added spdx-tools to the missing branches. See b/276427351

Change-Id: I7bd0b3f194b27dc9a255ccadeb2a9a12a3d59f66
2023-04-01 21:24:30 +00:00
Bob Badour
928ee9d9f7 Revert "Updated SBOM generator module to generate JSON spdx utility bill of"
This reverts commit e97adc5db9.

Reason for revert: Roll back until the kernel manifests are updated with the spdx-tools library.

Change-Id: Ib2327862acd928ebcea0f328daecc162b46d2c78
2023-03-31 14:51:36 +00:00
Ibrahim Kanouche
e97adc5db9 Updated SBOM generator module to generate JSON spdx utility bill of
material

Test: m compliance_sbom

Bug: 265472710
Change-Id: Iad9ddbd2abf17ff0b034f1410c55dd99051f7127
2023-03-30 23:34:32 +00:00
Ibrahim Kanouche
aef5110fbe Fixed typos and naming for SPDX validation
Test: m compliance_sbom
Change-Id: I39edb73e11afc9050a2481a899b19f8f10eb7e72
2023-01-18 22:49:17 +00:00
Ibrahim Kanouche
cf066f01f3 Merge "Add deps tracking for SBOMs" 2023-01-18 16:11:49 +00:00
Ibrahim Kanouche
649b4d741c Add deps tracking for SBOMs
Test: m compliance_sbom

Bug: 242220547
Change-Id: Ife93e0e1cdc43490818798922d574ae1b1d441d1
2023-01-16 16:44:22 +00:00
Bob Badour
f9d23a9f51 Fix concurrency bug in bottom-up resolution walk.
Concurrency is hard, and now that we no longer track origin it doesn't
improve performance in any meaningful way.

Bug: 261787132

Test: m droid dist compliance_dumpgraph compliance_dumpresolutions \
        compliance_sbom compliance_listshare compliance_rtrace \
        compliance_checkshare xmlnotice textnotice htmlnotice \
        compliancenotice_shippedlibs compliancenotice_bom

Test: m compliance_checkshare cts && \
        out/host/linux-x86/bin/compliance_checkshare out/host/linux-x86/gen/META/lic_intermediates/out/host/linux-x86/cts/android-cts.zip.meta_lic

Test: similar command as above for gts on internal

Test: m compliance_checksare droid dist && \
        out/host/linux-x86/bin/compliance_checkshare out/target/product/sunfish/gen/META/lic_intermediates/out/target/product/sunfish/obj/PACKAGING/systemimage_intermediates/system.img.meta_lic

Change-Id: I57a75927bf879c3ce6603049d8d583211dc0ce29
2023-01-04 00:02:02 +00:00
Bob Badour
3fe369c271 An ActionSet doesn't need to walk the graph.
Resolutions map back to the root, but actions do not. An iteration
works just fine.

Simplify TargetNodeSet so that it is directly iterable.

Bug: 261787132

Test: m droid dist compliance_dumpgraph compliance_dumpresolutions \
        compliance_sbom compliance_listshare compliance_rtrace \
        compliance_checkshare xmlnotice textnotice htmlnotice \
        compliancenotice_shippedlibs compliancenotice_bom

Test: m compliance_checkshare cts && \
        out/host/linux-x86/bin/compliance_checkshare out/host/linux-x86/gen/META/lic_intermediates/out/host/linux-x86/cts/android-cts.zip.meta_lic

Change-Id: Ic5a2d809b5a9a47b5d85f61e3a4a790dbe8f5fd2
2023-01-04 00:01:49 +00:00
Bob Badour
42b02efd05 Create regression test to catch concurrency error
The bottom-up walk to resolve conditions treats "already started" as
"alreay finished".

Bug: 261787132

Test: m compliance_checkshare
Change-Id: Ibb548bd4f5464b47682633878d475dfe011df3e1
2023-01-04 00:01:36 +00:00
Bob Badour
36f4a72aa7 Create regression test to catch WalkAction error.
Bug: 261787132

Test: m compliance_checkshare

Change-Id: I9f6a39a7f1cf50f9a0134b16f68d33f171cf7f13
2023-01-04 00:01:26 +00:00
Bob Badour
a6ee6d5511 Fix tests so they can fail.
Expected values were overwriting actual values.

Origin no longer tracked.

Bug: 261787132

Test: m droid dist

Test: m compliance_checkshare

Change-Id: Ie1fbc3d596cb08c6f0935a79441d8b00a4d1eb97
2023-01-04 00:01:03 +00:00
Bob Badour
cac8a3cf09 Change condition to match go/thirdpartylicenses
Policy uses restricted_if_statically_linked at
https://opensource.google/documentation/reference/thirdparty/licenses#RestrictedIfStaticallyLinked

Test: m droid
Change-Id: I9b18d3133ae72de1bede99f46b530298313bc2e6
2022-11-30 10:52:41 -08:00
Ibrahim Kanouche
f0c6cf501c Merge "Optimized project selection for getProjectMetadata" 2022-11-10 15:28:12 +00:00
Ibrahim Kanouche
a68ed086ab Optimized project selection for getProjectMetadata
Test: m compliance_sbom

Change-Id: I9e60b7e138a4fca74170954d8bb83862677e750f
2022-11-03 16:01:13 +00:00
Bob Badour
3afc3004f2 Fix final nits from noticeindex change.
Test: m droid dist cts alllicensemetadata
Change-Id: Ie446e5693a09f4ae1c936cdae6f1e668105541f0
2022-11-02 10:24:04 -07:00
Bob Badour
88b02afbf0 Merge "Switch noticeindex from regex to proto." 2022-11-02 15:46:38 +00:00
Maciej Żenczykowski
458f90272c sbom_test - use UTC
Test: TreeHugger, m compliance_sbom
Signed-off-by: Maciej Żenczykowski <maze@google.com>
Change-Id: Id2abec8537a1dfa32d6816da4c34c86f46313086
2022-11-01 19:49:05 -07:00
Ibrahim Kanouche
1bb27903e8 Merge "Added SBOM generator module to implement the spdx utility bill of material" 2022-11-01 23:36:22 +00:00
Ibrahim Kanouche
bedf1a8a86 Added SBOM generator module to implement the spdx
utility bill of material

Added methods to graph.go to extract the annotation types

Test: m compliance_sbom

Test: tested against real targets, see go/paste/4932131502424064

Bug: 241591242
Change-Id: I344c15d64cd034d72076f9d9a677e593e288e3a6
2022-11-01 20:42:01 +00:00
Bob Badour
5c12c66769 Replace nil-able *sync.Waitgroup with sync.Once
Simplifies synchronization and eliminates lock for nil waitroup.

Test: m droid

Test: m out/soong/.intermediates/packages/modules/StatsD/apex/com.android.os.statsd/android_common_com.android.os.statsd_image/NOTICE.html.gz

Change-Id: I381ee79e142214e7331241071f076db2f7960ba6
2022-10-29 22:48:13 -07:00
Bob Badour
d2c28ba897 Fix top-down resolve re-walking graph too much.
Firing off multiple concurrent walks of the same sub-tree with the same
conditions. Data race meant every walk would proceed. Move the logic to
entry of walk function and compare under lock.

Bug: 255526010

Test: m droid

Test: m out/soong/.intermediates/packages/modules/StatsD/apex/com.android.os.statsd/android_common_com.android.os.statsd_image/NOTICE.html.gz

Change-Id: Ie30edbb2ac9eaa9aa55badfc518d51eaadbb6be6
2022-10-29 22:27:49 -07:00
Bob Badour
43daade15f Add some documentation and simplify data model.
Bug: 245562496

Test: m droid
Change-Id: Iae757a5767522e0734abbe3840ea0939620197e7
2022-10-28 12:06:08 -07:00
Bob Badour
ab5cfbd5b7 Switch noticeindex from regex to proto.
Bug: 254534552

Test: m droid dist cts alllicensemetadata

Test: repo forall -c 'echo -n "$REPO_PATH  " && $ANDROID_BUILD_TOP/out/host/linux-x86/bin/compliance_checkmetadata . 2>&1' | fgrep -v PASS

Change-Id: I34dfe143cdc88432d6e33abee096409a37ce3b7e
2022-10-27 18:13:24 -07:00
Bob Badour
d6574e5c97 Add AllMetadataFiles() method to track deps.
Improve METADATA parsing error message.

Example error message (for an unqoted string url starting with ssh:):

============
  error in project "<redacted>" METADATA "<redacted>/METADATA": proto: (line 4:12): invalid value for string type: ssh

METADATA and METADATA.android files must parse as text protobufs
defined by
   build/soong/compliance/project_metadata_proto/project_metadata.proto

* unknown fields don't matter
* check invalid ENUM names
* check quoting
* check unescaped nested quotes
* check the comment marker for protobuf is '#' not '//'

if importing a library that uses a different sort of METADATA file, add
a METADATA.android file beside it to parse instead
============

Bug: 254534552

Test: m droid dist cts alllicensemetadata

Change-Id: Ie5c0adc7362941e455dd522baaa31f2913d7db5f
2022-10-27 18:12:30 -07:00
Ibrahim Kanouche
776ad80e68 Added functions to projectmetadata to retrieve
additional project info

Test: m droid

Bug: 254901942
Change-Id: I3de4bc528bd321c76900d277295bb10709035a9c
2022-10-22 01:22:53 +00:00
Bob Badour
dc62de4760 Refactor projectmetadata into separate package.
Replace regular expressions to extract fields from a text proto with
and actual parsed protobuf.

Refactor TestFS into its own package, and implement StatFS.

Test: m droid dist cts alllicensemetadata

Test: repo forall -c 'echo -n "$REPO_PATH  " && $ANDROID_BUILD_TOP/out/host/linux-x86/bin/compliance_checkmetadata . 2>&1' | fgrep -v PASS

Change-Id: Icd17a6a2b6a4e2b6ffded48e964b9c9d6e4d64d6
2022-10-18 16:55:47 -07:00
Colin Cross
4b54525b2b Fix nondeterminisim in xmlnotice
SafePathPrefixes contains "prebuilts/" which is a prefix of another
entry "prebuilts/module_sdk" which can both match the same path.
SafePathPrefixes is a map, so the iteration order is nondeterminisitic.
Move both SafePathPrefixes and SafePrebuiltPrefixes into lists that
will always have a deterministic iteration order.

Bug: 230357391
Test: build NOTICE.xml.gz multiple times
Change-Id: Ibfcd6715b70f26164e0ef4d59f73b240f47f8db7
2022-09-28 15:40:20 -07:00
Ibrahim Kanouche
e7c33de179 Added a Breadth-first top down walk function
to policy_walk.

Test: m droid dist
Change-Id: I678d2a2402c7c3ab446e8533c9f862cd8f54f889
2022-09-26 19:04:51 +00:00
Bob Badour
113c92b0a3 Make notice order more deterministic.
Bug: 230357391

Test: m droid dist
Change-Id: Ib3f771f9fd16743fca3f6c80c2ff85c9e42772a9
2022-09-23 11:27:24 -07:00
Bob Badour
085a2c23e7 Policy clarified: No need to share a "distribution medium"
Including code built from restricted sources in a distribution medium
does not require sharing the code for building the distribution medium.

Test: m cts dist

Test: m cts dist gts (requires cherry-pick to internal)

Change-Id: I7fcd889b11a97f8deaf4de9d72fdadd09deebe30
2022-09-22 03:23:23 +00:00
Bob Badour
10f5c48b23 Policy change: GPL+CE is permissive.
Bug: 210546823

Test: m cts compliance_checkshare compliancenotice_bom \
    compliancenotice_shippedlibs compliance_listshare \
    compliance_dumpgraph compliance_dumpresolutions htmlnotice \
    compliance_rtrace textnotice xmlnotice

Change-Id: I7da36972ffbb1494e33fd63db8e5ec851d47704c
2022-09-20 21:50:08 -07:00
Sasha Smundak
ded9acd86d Fix go.mod for tools/compliance
Now it can be built with `go build`, and can be cross-referenced.

Test: treehugger
Change-Id: Ic35631ff6a915c8cf58db14aa83c849549e64aba
2022-08-01 14:24:21 -07:00
Bob Badour
986a839161 Improve flags for compliance tools.
Test: m droid dist reportmissinglicenses

Change-Id: I4090dae3d5d33d1908d67dff31aeee92d2b261da
2022-06-03 12:44:22 -07:00
Bob Badour
213095a919 Rename listshare and checkshare.
Bug: 151177513
Bug: 213388645
Bug: 210912771

Test: m droid dist reportmissinglicenses
Change-Id: I223c4496d776a1bbd4fdce31ed79a638ed3491d8
2022-04-13 11:04:43 -07:00
Bob Badour
c778e4cba4 Support OUT_DIR for license graph.
Bug: 226066987

Test: OUT_DIR=/tmp/outdir m all listshare checkshare htmlnotice rtrace dumpgraph dumpresolutions compliancenotice_bom compliancenotice_shippedlibs
Test: OUT_DIR=/tmp/outdir m /tmp/outdir/target/product/bonito/obj/NOTICE.xml.gz
Test: m all listshare checkshare htmlnotice rtrace dumpgraph dumpresolutions compliancenotice_bom compliancenotice_shippedlibs
Test: m out/target/product/bonito/obj/NOTICE.xml.gz

Change-Id: I6282c647a389c5935d5ce7c79193f86d32c76365
2022-03-23 14:12:12 -07:00
Bob Badour
77570059cd Phony target to report all library names.
Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m nothing reportallnoticelibrarynames

Change-Id: I0f791e2d92b8135d6af3a6c93f0db348fc69f5a1
2022-03-02 22:32:37 -08:00
Treehugger Robot
5796849ac8 Merge "Support multiple library names per target." 2022-02-11 22:17:17 +00:00
Bob Badour
e9b38c175a Greater-than zero is more idiomatic.
Test: m all
Change-Id: I6000d937b98c84226a347b69c69b333a15beb355
2022-02-09 15:56:59 -08:00
Bob Badour
5028abccb5 Support multiple library names per target.
The prior interface to make supported only a single package name per
target; although, a target might have multiple licenses each with its
own package name.

Bug: 151177513
Bug: 210912771

Test: m all dist
Test: flash; About Phone -> Legal Information -> Third-party licenses
Change-Id: I5cf1964ec18af91ec7e258a1c6722527e3596cf3
2022-02-09 11:56:58 -08:00
Bob Badour
49dd4f75ff Add support for a product name as well as title.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m htmlnotice; out/soong/host/linux-x85/htmlnotice ...
Test: m textnotice; out/soong/host/linux-x85/textnotice ...
Test: m xmlnotice; out/soong/host/linux-x85/xmlnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Id5b2ebe3752081e3a89bba1d717cf103f7846043
2022-02-04 14:50:59 -08:00
Bob Badour
682e1bae57 Regularize command-line flags.
All the notice binaries have -title

All the binaries that can -stripPrefix can strip multiple.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m bom; out/soong/host/linux-x85/bom ...
Test: m dumpgraph; out/soong/host/linux-x85/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x85/dumpresolutions ...
Test: m htmlnotice; out/soong/host/linux-x85/htmlnotice ...
Test: m rtrace; out/soong/host/linux-x85/rtrace ...
Test: m textnotice; out/soong/host/linux-x85/textnotice ...
Test: m xmlnotice; out/soong/host/linux-x85/xmlnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I08357bf1adb048abba6563cf3cea6ee6d60405e0
2022-02-02 15:32:50 -08:00
Bob Badour
f87922450e license metadata xml notice files
Introduce the below command-line tool:

xmlnotice outputs a NOTICE.xml file constructed from the license
texts of the transitive closure of dependencies.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m xmlnotice; out/soong/host/linux-x85/xmlnotice ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I267effd3a1c1b981bcdc5e058dce561a2d4a7948
2022-02-02 15:32:50 -08:00
Bob Badour
608bdff0a7 Add support for gzipping html output.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m htmlnotice; out/soong/host/linux-x85/htmlnotice ...

where ... is -o=html.gz followed by the path to the .meta_lic file for
the system image. In my case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

the rest of ... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I7a42d5186876609a401956754e3dcff64211fb15
2022-02-01 15:35:40 -08:00
Treehugger Robot
6ebf02914c Merge changes I40a05866,Id3d6e701
* changes:
  license metadata reverse trace
  Fix copy+paste error.
2022-02-01 01:25:59 +00:00
Bob Badour
c817845ea5 license metadata reverse trace
Introduce the below command-line tool:

rtrace outputs a list of targets and conditions causing one or more
projects or target nodes to require sharing to resolve a restricted
condition.

Bug: 68860345
Bug: 151177513
Bug: 151953481
Bug: 213388645
Bug: 210912771

Test: m all
Test: m systemlicense
Test: m rtrace; out/soong/host/linux-x85/rtrace -rtrace=...

where ... is a project or license metadata file followed by the path to
the .meta_lic file for the system image. In my case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

system/core ${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
or
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic ${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: I40a0586699d9b8a8dd2bd4ba26756c9649ebf964
2022-01-31 14:16:23 -08:00
Bob Badour
91af68b1e1 Fix copy+paste error.
The tool under test is dumpresolutions not listshare.

Test: m dumpresolutions; out/soong/host/linux-x85/dumpresolutions ...

where ... is the path to the .meta_lic file for the system image. In my
case if

$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)

... can be expressed as:

${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic

Change-Id: Id3d6e701ebd3dab03f407116d85a5e2aab8c5c59
2022-01-31 14:14:27 -08:00
Colin Cross
bb45f8c74d Track dependencies when reading notice files
Track which files are read by the notice file indexer and add a flag
to textnotice and htmlnotice to support writing them out.

Bug: 207445310
Test: textnotice_test
Test: htmlnotice_test
Change-Id: Ib74706b8a87a5ed9268a0fe37982ecf89f4e227d
2022-01-31 10:15:10 -08:00