Commit graph

7896 commits

Author SHA1 Message Date
Ibrahim Kanouche
bedf1a8a86 Added SBOM generator module to implement the spdx
utility bill of material

Added methods to graph.go to extract the annotation types

Test: m compliance_sbom

Test: tested against real targets, see go/paste/4932131502424064

Bug: 241591242
Change-Id: I344c15d64cd034d72076f9d9a677e593e288e3a6
2022-11-01 20:42:01 +00:00
Bob Badour
43daade15f Add some documentation and simplify data model.
Bug: 245562496

Test: m droid
Change-Id: Iae757a5767522e0734abbe3840ea0939620197e7
2022-10-28 12:06:08 -07:00
Bob Badour
d6574e5c97 Add AllMetadataFiles() method to track deps.
Improve METADATA parsing error message.

Example error message (for an unqoted string url starting with ssh:):

============
  error in project "<redacted>" METADATA "<redacted>/METADATA": proto: (line 4:12): invalid value for string type: ssh

METADATA and METADATA.android files must parse as text protobufs
defined by
   build/soong/compliance/project_metadata_proto/project_metadata.proto

* unknown fields don't matter
* check invalid ENUM names
* check quoting
* check unescaped nested quotes
* check the comment marker for protobuf is '#' not '//'

if importing a library that uses a different sort of METADATA file, add
a METADATA.android file beside it to parse instead
============

Bug: 254534552

Test: m droid dist cts alllicensemetadata

Change-Id: Ie5c0adc7362941e455dd522baaa31f2913d7db5f
2022-10-27 18:12:30 -07:00
Treehugger Robot
a2faaa8c45 Merge "Remove all ZIP64LIMIT hack" 2022-10-27 22:44:22 +00:00
Treehugger Robot
1fca7c5e7f Merge "Enable zip64 support in build image script" 2022-10-27 22:31:00 +00:00
Jooyung Han
e3862eacb6 Merge "Fix boot fail due to missing /apex in images" 2022-10-27 21:54:13 +00:00
Spandan Das
03b846ff37 Merge "Migrate check_elf_file.py to python3" 2022-10-27 16:46:18 +00:00
Spandan Das
60b8195c83 Migrate check_elf_file.py to python3
Also, create a python_binary_host so that it runs using the hermetic
python toolchain.

Test: m check-elf-files
Bug: 203436762
Change-Id: I964342a27f6b6c9dcdbbe910d4dc6c9ec00c2213
2022-10-27 02:08:23 +00:00
Kelvin Zhang
37a4290909 Remove all ZIP64LIMIT hack
In the old days, we hacked values of ZIP64LIMIT to get around size
limitations of non-zip64 supported zip files. Now that we switched to
python3 + zip64, there's no point in keeping those hacks.

Test: th
Bug: 255683436
Change-Id: I913db33dad5503736c68a7a1f1321aa952019f60
2022-10-26 13:22:52 -07:00
Kelvin Zhang
1de1788dda Enable zip64 support in build image script
Images are getting bigger each year, we need zip64 support to store >4GB
files in .zip archive.

Test: th
Bug: 255683436
Change-Id: I9adbefa5ec72e357b02e46d6e629c054d466ddd7
2022-10-26 12:30:29 -07:00
Po Hu
4d2f64b6f0 Fix boot fail due to missing /apex in images
Do not move or rename any files in merge_target_files.py's temporary directory. They will be used to generate images further.

Bug: 254760332
Change-Id: I973a49b29234d49f5254a1403a0e236e69425d74
Test: check SYSTEM/apex in target-files.zip generated by merget_target_files.py
2022-10-26 02:34:12 +00:00
Treehugger Robot
1a6c0c5d86 Merge "Delete compare_fileslist.py" 2022-10-25 21:22:19 +00:00
Treehugger Robot
fe7277fdde Merge "The --why parameter in whichgit should be optional" 2022-10-25 20:08:59 +00:00
Cole Faust
29c032d260 Delete compare_fileslist.py
It was written in python 2, and is unused.

Bug: 203436762
Test: Presubmits
Change-Id: I3298ec06727ba1b72337753f52e3d09519fe0b39
2022-10-25 11:34:42 -07:00
Ibrahim Kanouche
bfa6c8e553 Merge "Added functions to projectmetadata to retrieve additional project info" 2022-10-25 17:25:11 +00:00
Cole Faust
3b9af315d2 Merge "Delete java-layers.py" 2022-10-25 17:00:52 +00:00
Cole Faust
c22b90c5d4 Merge "Delete parsedeps.py" 2022-10-25 17:00:11 +00:00
Treehugger Robot
f99edd9f67 Merge "Convert fileslist_util.py to python 3" 2022-10-25 15:58:19 +00:00
Treehugger Robot
aa3b322971 Merge "Convert findleaves.py to python 3" 2022-10-25 09:17:27 +00:00
Cole Faust
e0fab08a33 Convert fileslist_util.py to python 3
Also update it from getopt to argparse.

Bug: 203436762
Test: m $OUT/installed-files-root.txt
Change-Id: I006a36feab01cbbd76dc7d7c1c0675197848212a
2022-10-24 18:46:03 -07:00
Cole Faust
6a75c4f782 Delete java-layers.py
There are no usages of LOCAL_JAVA_LAYERS_FILE,
and java-layers.py was written in python 2.

Bug: 203436762
Test: Presubmits
Change-Id: I1be0f2547757a853e20811f28529479487037e94
2022-10-24 18:15:54 -07:00
Cole Faust
e995769d44 Delete parsedeps.py
This script is only ever manually run, and it's written
in python 2. Just delete it instead of updating to python
3 because it's unlikely anyone uses it anymore.

Bug: 203436762
Test: Presubmits
Change-Id: Ib55b3bf43a8066822308f49d1a948d27c520c5b2
2022-10-24 18:02:23 -07:00
Cole Faust
407d214b0d Convert findleaves.py to python 3
Kati special-cases findleaves.py to an internal
C++ implementation, so I don't think this python
script is even used. Update it anyways, and
test by manually running findleaves.py.

Bug: 203436762
Test: ./build/make/tools/findleaves.py --mindepth=2 --dir=build/make/target Android.mk
Change-Id: If27cc37d8f673388c0e31b9491e792b98ca59da4
2022-10-24 17:57:39 -07:00
Spandan Das
ce69e9c812 Convert check_radio_versions to python3
Additionaly,
1. Run it as a python_binary_host so that it uses the hermetic python
toolchain
2. Fix an existing bug where if `board_info_txt` was empty, $^ would
   evaluate to its own path, and it would try to parse the relevant
   configuration from the wrong file

Bug: 196300985
Test: m out/target/product/vsoc_arm64/android-info.txt
Change-Id: Ie1e51a93041591829da4351dd57f37106e15d764
2022-10-24 21:02:00 +00:00
Joe Onorato
8e908b221f The --why parameter in whichgit should be optional
Test: ./build/make/tools/whichgit --modules out/target/product/coral/system.img
Change-Id: Ibc6e208d75c8e68c8130dc9ba3fc7af51390691b
2022-10-24 08:03:20 -07:00
Ibrahim Kanouche
776ad80e68 Added functions to projectmetadata to retrieve
additional project info

Test: m droid

Bug: 254901942
Change-Id: I3de4bc528bd321c76900d277295bb10709035a9c
2022-10-22 01:22:53 +00:00
Treehugger Robot
e9d0a7f5d9 Merge "Convert fs_config_generator.py to python 3" 2022-10-21 21:17:23 +00:00
Treehugger Robot
775187f921 Merge "Fix ota build error when using boot variable files" 2022-10-21 20:43:21 +00:00
Cole Faust
216978e7c4 Convert fs_config_generator.py to python 3
Bug: 203436762
Test: ./test_fs_config_generator.py and m libc
Change-Id: I16ce5f5e273ade064acce8889a9eb3f2707910ae
2022-10-21 11:16:18 -07:00
Kelvin Zhang
eb147e051e Fix ota build error when using boot variable files
When performing a deep copy of info dict, input_file is serialized as
filename. Make ExtractFromInputFile support reading from filename of a
ZipFile.

Test: build OTA with partner provided target_files
Bug: 253549364
Change-Id: I366a076c32d638d61b83a1df9bf864807cf5257c
2022-10-21 10:53:21 -07:00
Treehugger Robot
e7bf5958b9 Merge "Fix broken fs_config_generator tests" 2022-10-21 01:33:41 +00:00
Kelvin Zhang
08b7e3aacb Merge "add a test case to validate deepcopy on PartitionBuildProps" 2022-10-20 21:46:07 +00:00
Cole Faust
f08ab3d098 Fix broken fs_config_generator tests
These tests were failing at TOT. Also add a
python_test_host target so the tests will be
run on CI in the future.

Bug: 203436762
Test: m libc, prebuilts/build-tools/linux-x86/bin/py2-cmd build/make/tools/fs_config/test_fs_config_generator.py
Change-Id: I6dda841023a2f5e76d59360d08626fc1a8842ffe
2022-10-20 13:40:07 -07:00
Treehugger Robot
de764d7d82 Merge "Update normalize_path.py to Python 3." 2022-10-20 20:27:06 +00:00
GeQi
3fa9e32563 add a test case to validate deepcopy on PartitionBuildProps
Test: atest --host releasetools_test
Bug: 253549364
Change-Id: I187554ae33041ed090ee352167ef8deb51d754bc
2022-10-20 18:39:08 +08:00
Lukacs T. Berki
740641571c Update normalize_path.py to Python 3.
Test: presubmits.
Change-Id: I2db4fedcbb658a7ca4efa3a4557a216b8682735d
2022-10-20 08:11:18 +00:00
Treehugger Robot
eb5f9e2bef Merge "check_target_files_vintf: Support flattened apexes" 2022-10-20 03:19:24 +00:00
Kelvin Zhang
5ef25191f7 Fix picle error on ota_from_target_files
When trying to make a deep copy of PartitionBuildProps, execution fails
with picle error because PartitionBuildProps stores a ZipFile object,
which stores a lock object, which doesn't support pickling.

To fix it, pickle ZipFile object as a str to its path.

Test: generate ota with boot variable files
Bug: 253549364
Change-Id: I92c5d78ce65045495203c85862922d33d886e9ea
2022-10-19 11:25:22 -07:00
Bob Badour
e790bc6f0a Merge "Refactor projectmetadata into separate package." 2022-10-19 02:04:50 +00:00
Bob Badour
dc62de4760 Refactor projectmetadata into separate package.
Replace regular expressions to extract fields from a text proto with
and actual parsed protobuf.

Refactor TestFS into its own package, and implement StatFS.

Test: m droid dist cts alllicensemetadata

Test: repo forall -c 'echo -n "$REPO_PATH  " && $ANDROID_BUILD_TOP/out/host/linux-x86/bin/compliance_checkmetadata . 2>&1' | fgrep -v PASS

Change-Id: Icd17a6a2b6a4e2b6ffded48e964b9c9d6e4d64d6
2022-10-18 16:55:47 -07:00
Yi-yo Chiang
163ece5b55 Merge "Remove deprecated BOARD_BUILD_SYSTEM_ROOT_IMAGE" 2022-10-18 06:57:24 +00:00
Jooyung Han
eb1182115b check_target_files_vintf: Support flattened apexes
Flattened apexes are extracted and copytree()'d to /apex directory to
simulate "activation" on device.

Bug: 242314000
Bug: 239055765
Test: m otapackage # target with "flattened" apexe
Change-Id: I90cc37b5f64ebad1e15f6fc5fa245d0f5955095d

Change-Id: I7e8c3fdf8e4620344d23cee0de509e574206ad9b
2022-10-18 07:46:14 +09:00
Kelvin Zhang
2e1ff6e262 Remove unnecesasry use of ZipClose()
ZipClose() was introduced to work with large zip without switching to
ZIP64. It is a hacky patch. Now we have moved to python3 and zip64
completely, no need to keep using it.

Test: th
Bug: 246504616
Change-Id: I1ff15171bb26887b819f655e32817627ecad1132
2022-10-17 10:04:45 -07:00
Jooyung Han
62283b9649 Skip flattened apexes while check_target_files_vintf
Flattened apexes should be handled differently. For now just skip
flattened apexes to fix the build breakage.

Bug: 253669548
Bug: 253700117
Bug: 239055765
Test: m otapackage for targets with "flattened apexes"
Change-Id: I102a961fc1365f42881ca9ee6546e71246792d2a
2022-10-17 10:38:41 +09:00
Wei Li
ddfc1ebf14 Merge "Change function ZipDelete to use Python module zipinfo instead of command 'zip'." 2022-10-16 05:55:00 +00:00
Wei Li
8895f9ecea Change function ZipDelete to use Python module zipinfo instead of command 'zip'.
'zip' is a host command, and it can not be provided as a dependency when some release tools are used in Bazel rules.

Test: atest --host releasetools_test
Bug: 243748589

Change-Id: Ie5f42eadbd316ccd018b19194c466a908971af82
2022-10-14 18:31:56 -07:00
Rob Seymour
0a00cd27d2 Merge "Add APEX data to checkvintf for compatibility check." 2022-10-14 18:04:46 +00:00
Cole Faust
43d2d682a8 Remove extra semicolons from event log tag generated files
To avoid an errorprone error.

Test: Presubmits
Change-Id: I5d5eefd5e54e6e862755894dd0b2f34e9e7fad89
2022-10-13 17:47:42 -07:00
Yi-Yo Chiang
06c6273073 Remove deprecated BOARD_BUILD_SYSTEM_ROOT_IMAGE
This variable is always false or empty now, so clean up remaining
references of the variable.

Remove "buildvariant=" kernel commandline as it was used by
system-as-root kernels to determine the system build variant. It is
unsupported now.

Bug: 241346584
Test: Presubmit
Change-Id: I0f97b6e0bf8b03429f65366119ff78f9c7983516
2022-10-13 16:08:53 +08:00
Treehugger Robot
1c5f813bf0 Merge "Zipalign: Don't align directory entries" 2022-10-12 23:28:34 +00:00