Commit graph

159 commits

Author SHA1 Message Date
Jeff Gaston
2cc7d9fc78 pom2mk to include extra-deps as top-level deps too
Specifically, the use case is android-support-car requires some stubs that don't have a Maven pom.
So those stubs can't easily be listed as a dependency in the pom.
However, we do want the dependency to exist when importing to Make and detected by extract_deps.py

Bug: 76692459
Test: m -j blueprint_tools \
      && update_prebuilts.py -s <build_id> \
      && git diff HEAD^ -- fix_dependencies.mk \
      && m -j checkbuild

Change-Id: If546ecf7416580a01bde7b906976ada4b1145253
Merged-In: If546ecf7416580a01bde7b906976ada4b1145253
(manually cherry-picked from commit 358f25e97d)
2018-04-23 18:41:59 -04:00
TreeHugger Robot
ce3a4bb268 Merge "Have pom2mk list all duplicates modules" into pi-dev 2018-04-23 18:57:35 +00:00
TreeHugger Robot
c86f03385e Merge "Have pom2mk include runtime deps too" into pi-dev 2018-04-23 18:57:17 +00:00
Dan Willemsen
050ca73dbf Use Config/DeviceConfig functions to access ProductVariables
An upcoming change will stop exporting ProductVariables from Config, so
switch to using existing accessor functions, and add more when they're
missing.

Bug: 76168832
Test: out/soong/build.ninja is identical
Change-Id: Ie0135bdbd2df3258ef3ddb53e5f8fc00aa9b97f7
Merged-In: Ie0135bdbd2df3258ef3ddb53e5f8fc00aa9b97f7
(cherry picked from commit 3fb1faeeb9)
2018-04-17 10:34:23 -07:00
Jeff Gaston
4447ff7d38 Have pom2mk list all duplicates modules
For easier debugging if there are multiple duplicates

Bug: 76692459
Test: ./update_current.py -x <build-id>

Change-Id: I3079cb4ed1cfd365d792a7b41c9cdb4e7a1e148f
Merged-In: I3079cb4ed1cfd365d792a7b41c9cdb4e7a1e148f
2018-04-09 12:44:13 -04:00
Jeff Gaston
15a7f83104 Have pom2mk include runtime deps too
It still excludes test deps though

Bug: 76692459
Test: ./update_current.py -x <build-id>

Change-Id: I55a786e9feab31f99b95b7cebe84869757e46270
Merged-In: I55a786e9feab31f99b95b7cebe84869757e46270
2018-04-09 12:44:07 -04:00
Alan Viverette
d95159cb33 Always include JAR dependencies (not just with --static-deps) in Makefile
Bug: 76692459
Test: ./update_current.py -x <build-id>
Change-Id: Ie0f77e52089adbe153d668fe39902545f469f2d2
Merged-In: Ie0f77e52089adbe153d668fe39902545f469f2d2
2018-04-06 13:20:44 -04:00
Colin Cross
0a7d85a896 Replace extract-srcjars.sh with zipsync tool
extract_srcjars.sh uses zipinfo and unzip, which fail with an
error on an empty zip file.  Instead of trying to hack around
this (which is hard to make guarantees for since they are
non-hermetic host tools), replace them with a go tool to unzip
a set of zip files into a directory.

Bug: 73885582
Test: m checkbuild
Change-Id: I151fed347ed5196726e36866ffc27bc831799afb
Merged-In: I151fed347ed5196726e36866ffc27bc831799afb
(cherry picked from commit 436b76564d)
2018-03-20 11:21:42 -07:00
Colin Cross
86803cfe6e add a library to report build numbers without causing rebuilds
Allow native modules to specify use_version_lib, which will make
an android::build::GetBuildNumber() function available.  For host
builds, the function will return the build number at the time that
the module was linked.  For device modules it will return the
value of the ro.build.version.incremental property.

Bug: 71719137
Test: build_version_test
Test: m build_version_test && touch build/make/core/Makefile build/soong/cc/libbuildversion/tests/build_version_test.cpp && m build_version_test shows different build numbers for binary and library tests.
Change-Id: I6f7d40b7574bb8206866c4e39bad9c710c796e32
2018-03-02 16:55:51 -08:00
Colin Cross
8673b5b959 Rewrite symbol_inject to be testable
Parse the machine-specific symbol tables into a custom symbol table
object and use a single function to find the offset and size of the
symbol in the custom table.  Make the functions to convert a
machine-specific symbol table into the custom table also take mocks,
and provide functions to dump mocks from real files.  Add tests
that take the mocks and verify they produce the right offset and
size.

Test: cmd/symbol_inject/*_test.go
Change-Id: I25654032b5017dd13a9a1fe29f8b8826ce5bc82b
2018-03-02 16:47:13 -08:00
Colin Cross
dfce764476 Fix finding next symbol when multiple symbols have the same address
Some exe files have a .data symbol at the same address as the
soong_build_number symbol.  If the .data symbol is after
soong_build_number in the symbol list, symbol_inject would think
the end address was the same as the start address, and use
uint32(-1) as the size.

Use sort.Search to find the first symbol whose section number is
the same as the target symbol, but whose address is higher than
the target symbol.

Test: manual
Change-Id: I51d6e53c6b906222ba68c5cf93be944843e23550
2018-02-28 15:04:59 -08:00
Colin Cross
c4a18e0291 Sort macho symbol table entries
macho symbol table entries are not always in order, which breaks
finding the next symbol to find the size of the target symbol.

Test: build_version_test
Change-Id: I41d1c3c3ff9929694e9ec2b034553d6b7ddef937
2018-02-23 22:45:55 -08:00
Colin Cross
64c6d4bf14 Fix symbol_inject on win32 exes
Win32 exes seem to prefix each symbol with an underscore like
macho.

Test: build_version_test
Change-Id: I9de0a0f5f25f93ad17d34b7c1b993511c77a57eb
2018-02-23 18:24:48 -08:00
Colin Cross
baccf5b984 Use __SBOX_OUT_DIR__ in sbox output file list
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.

Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.

Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
2018-02-21 14:55:34 -08:00
Colin Cross
52226ad920 Fix injecting data into a .o file
Use section.Offset instead of Addr.

Test: symbol_inject -i test.o -o test2.o -s symbol -v value
Change-Id: I9c54a5a245e7674b8370fc27ba7d0b9995f2ed08
2018-02-20 17:54:20 +00:00
Alan Viverette
6bd35eb6f5 Use "jar" as default type, "compile" as default scope
Only include dependencies for scope "compile."

Bug: 73263586
Test: ./update_current.py -s -t <build-id>
Change-Id: I2ac6055cb4c1ad1f8d7924869f54f50e5e662742
2018-02-13 15:21:18 -05:00
Colin Cross
5498f8548b Add a tool to inject data into an elf, macho, or PE symbol
Test: symbol_inject -i a.out -o a.out2 -s symbol -v value
Change-Id: I16cd8facbae754f679bef07ab0ba23638286e1d7
2018-02-06 19:12:21 +00:00
Alan Viverette
1593d3de6e Emit static dependencies into pom2mk-generated Makefile
Bug: 70516592
Test: ./update_current.py -s -t 4495436
Change-Id: Ic7dd433de78f9666c7388ed8ec1deebee7269368
2018-01-05 14:12:18 -05:00
Nan Zhang
5925b0fda4 Add -p, -e options in merge_zips for supporting par format.
Move __init__.py files population to merge_zips, and add options: -p, and
-e. Since par needs entry_point.txt to search the program entry point, so
add -e option to support it.

Test: ./merge_zips -p -e entry_point.txt out in1.zip in2.zip
Bug: 70568913
Change-Id: Ida10faf125e1b94dffc98b30240db3b90fd75b25
2017-12-20 12:57:10 -08:00
Colin Cross
e87040b2eb Allow Soong to generate its own build documentation
Add a rule in soong that re-executes soong_build in order to
generate build documentation.  This allows Soong to customize
the documentation.

Bug: 70516282
Test: m soong_docs
Change-Id: If143cfacd6ac20274cd7bb8d8fab0c07025a5553
2017-12-12 11:12:32 -08:00
Dan Willemsen
22de216799 Increase per-process file limits for multiproduct_kati
On large branches (250+ configurations), the open file count can go
over the default 1024 soft limit on Ubuntu. Many systems have increased
that default, but for the ones that haven't, at least opt into the
hard limit until this can be refactored to use fewer open files.

Bug: 70370883
Test: prlimit -n256:4096 build/soong/build_test.bash -only-config
Test: ulimit -Sn 256; build/soong/build_test.bash -only-config (darwin)
Change-Id: I7a952ffc89a0149ab65b04db1523d348daa6ab3e
2017-12-11 22:53:16 +00:00
Alan Viverette
75b95f8a61 Update pom2mk to allow duplicate module names if rewritten
Fixes: 70162730
Test: ./update_current.py -s -t 4482279
Change-Id: I874d7bfb50fd2d2bc488f5458cfe57b2e0d4d4e5
2017-12-05 12:53:44 -05:00
Jeff Gaston
088e29ed38 Revert "Revert "Soong support for namespaces""
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .

Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^ build/soong:work^' \
      'build/blueprint:work build/soong:work'
      # and see that the only changes were:
      # 1. adding some new files
      # 2. changing some line numbers
Test: m -j nothing # which runs unit tests

Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
2017-12-04 17:44:29 -08:00
Jeff Gaston
90cfb09214 sbox error message now lists the files that were created
which should make debugging faster.

Bug: 66921582

Test: ./out/soong/host/linux-x86/bin/sbox --output-root outs --sandbox-path out/.temp/sbox-work -c "cd __SBOX_OUT_DIR__ && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p" outs/a outs/b outs/z # and observe the below output:

mismatch between declared and actual outputs
in sbox command(cd out/.temp/sbox-work/sbox343858828 && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p)

  in sandbox out/.temp/sbox-work/sbox343858828,
  failed to create 2 files:
    b: not a file
    z: does not exist
  did create 16 files:
    a
    asubdir/child
    c
    d
    e
    f
    g
    h
    i
    j
    ...6 more

Change-Id: I75e37834c44d4279dec874701d67ce8bb01b872c
2017-12-01 13:21:35 -08:00
Jeff Gaston
178d5fefc0 Revert "Soong support for namespaces"
This reverts commit 63a250a336.

Reason for revert: Some failures:
  namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file 
  such as New Build Breakage: aosp-master/build_test @ 4475274

Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
2017-11-30 00:43:43 +00:00
Jeff Gaston
a8063afc16 Merge "Soong support for namespaces" 2017-11-29 23:01:39 +00:00
Treehugger Robot
b8160761f7 Merge "Revert "Revert "Test for dangling rules in make checkbuild""" 2017-11-29 21:09:48 +00:00
Jeff Gaston
63a250a336 Soong support for namespaces
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^ build/soong:work^' \
      'build/blueprint:work build/soong:work'
      # and see that the only changes were:
      # 1. adding some new files
      # 2. changing some line numbers
Test: m -j nothing # which runs unit tests

Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
2017-11-29 13:03:40 -08:00
Colin Cross
37193495cf Revert "Revert "Test for dangling rules in make checkbuild""
This reverts commit fb941913a3.

Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.

Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
2017-11-25 05:26:10 +00:00
Colin Cross
e909e1e079 Add support for --ignore_duplicates to merge_zips
Add a --ignore_duplicates option to cause merge_zips to
silently take the first entry in the case of duplicate entries
in the zips to be merged.  This will be used for soong Jacoco
support to combine the instrumented jar with the original jar
to pick up any classes that were excluded.

Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: I49ede3f07ce0ed7701b4db7058da2e4f11b8043e
2017-11-22 16:08:35 -08:00
Colin Cross
f3831d0e08 Support exclusions and Blueprint-style ** globs in zip2zip
Jacoco support will use zip2zip to create a jar that is a subset
of another jar, and will need exclusion filters and recursive
globs.  Switch zip2zip from filepath.Match to pathtools.Match,
and check each included file against the exclusion list.

Bug: 69629238
Test: zip2zip_test.go
Change-Id: Ibe961b0775987f52f1efa357e1201c3ebb81ca9c
2017-11-22 16:08:34 -08:00
Jeff Gaston
b61e3f79ec multiproduct_kati support for a custom list of products
Test: multiproduct_kati --products="aosp_arm,aosp_arm64"
Bug: 67478260
Change-Id: Icf7b8b0c30a2dc8e093ab109172675ff74f43adc
2017-11-15 15:31:38 -08:00
Colin Cross
c7d1254119 Merge "Revert "Test for dangling rules in make checkbuild"" 2017-11-13 23:41:24 +00:00
Colin Cross
fb941913a3 Revert "Test for dangling rules in make checkbuild"
This reverts commit 7709a05770.

Reason for revert: Broke mac checkbuilds

Change-Id: If71ec9198ca8943d90ce9d5e3a277faf239e38a5
2017-11-13 23:38:57 +00:00
Treehugger Robot
df93350ccd Merge "Test for dangling rules in make checkbuild" 2017-11-13 22:43:30 +00:00
Colin Cross
7709a05770 Test for dangling rules in make checkbuild
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory.  This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it.  The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly).  These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.

Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
2017-11-13 12:47:42 -08:00
Treehugger Robot
08bb51cdc5 Merge "Send soong output to std.log in multiproduct_kati" 2017-11-12 01:33:47 +00:00
Dan Willemsen
f196396a1c Send soong output to std.log in multiproduct_kati
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).

Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
2017-11-11 15:44:51 -08:00
Treehugger Robot
dbe8b78a5d Merge "pom2mk: Allow setting LOCAL_SDK_VERSION" 2017-11-10 23:53:44 +00:00
Dan Willemsen
15a8e793f3 pom2mk: Allow setting LOCAL_SDK_VERSION
For unbundled builds, LOCAL_SDK_VERSION needs to be set, otherwise we'll
try to use frameworks-res, which isn't available.

Bug: 64723465
Test: Use with prebuilts/sdk/update_current.py
Change-Id: If5e1233b4bcb09b3d0e2251e8dd1a8ecfc87eb19
2017-11-10 14:03:48 -08:00
Jeff Gaston
8a88db5a35 Have sbox remove its output directory before running
Bug: 38205169
Test: m -j
Change-Id: I73abb9921596c96fa056449bdf8250c2928bceda
2017-11-08 13:44:03 -08:00
Dan Willemsen
4eeb44b83f Merge "Change storage behavior of multiproduct_kati" 2017-11-07 22:25:24 +00:00
Dan Willemsen
4647be4afe Merge changes Ieaff757f,I5bdbba8d
* changes:
  pom2mk: Fix unlisted dependency types
  pom2mk: Add a flag to specify a specific version of artifacts
2017-11-07 20:59:02 +00:00
Dan Willemsen
e348076296 Change storage behavior of multiproduct_kati
Instead of deleting artifacts/logs from successful build (unless -keep
is set), and keeping unsuccessful artifacts, keep all logs and remove
all artifacts (unless -keep is set, then we'll compress the artifacts).

If -dist is passed in, we'll put an archive of the logs into the
DIST_DIR. Even compressed, the rest of the artifacts are still a bit too
large to dist (~5.6GB on AOSP).

Test: build/soong/build_test.bash
Test: build/soong/build_test.bash -keep
Test: build/soong/build_test.bash -dist
Change-Id: I87f55978c18c8ff2e517b8271554ba383003742f
2017-11-07 11:30:19 -08:00
Dan Willemsen
5f9d8a6698 pom2mk: Fix unlisted dependency types
Android Support Library 26+ does not specify dependency types, so parse
all pom files first, then propagate missing dependency types if we've
got the information.

Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 26.0.0-beta2 com/android/support
Change-Id: Ieaff757ff198c9a7b4b006623340b382728c1fd4
2017-11-06 20:55:02 -08:00
Jeff Gaston
93f0f37417 Use flag.Parse in sbox
Also removeTempDir -> !keepOutDir
Also shorten the help comment

Bug: 68336760
Test: rm out -rf && m -j
Test: sbox -c # and observe that the output does not have a stacktrace

Change-Id: I6fbb385132128ab1273c8398ff2425191049751e
2017-11-03 17:30:21 -07:00
Dan Willemsen
47e44a4c5c pom2mk: Add a flag to specify a specific version of artifacts
For repositories that contain multiple versions of the same artifact,
we'd only like to create makefiles for one of the versions of the
artifact.

Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 25.3.1 com/android/support
Change-Id: I5bdbba8d2143a47610d56d679381eb01b3cf176d
2017-11-03 13:40:08 -07:00
Jeff Gaston
02a684b8b2 Make $(depfile) work with sbox
Most notably, the sandbox depfile path should be passed into the tool.

Bug: 68336760
Test: m -j checkbuild

Change-Id: I22f944a3f57d613fda26de0ea777a915cafcd020
2017-10-30 17:49:29 -07:00
Colin Cross
5c6ecc179f Don't panic in turbine with no classes
ALLOW_MISSING_DEPENDENCIES=true can cause missing dependencies to be
hidden from GenerateAndroidBuildActions, which is triggering the
len(jars) == 0 check in compileJavaHeader.  For the
ALLOW_MISSING_DEPENDENCIES=true case the check is unnecessary, as the
build rules will be replaced with runtime errors.  On the off chance
the check would have been hit in a real build, allow merge_zips to
create an empty zip file if it is asked to.

Test: m ALLOW_MISSING_DEPENDENCIES=true
Change-Id: I3171f921d51229ddf38a1a647d32566658c673fa
2017-10-24 16:01:44 -07:00
Jeff Gaston
11b5c51d4e split soong_zip into a library and a binary
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)

Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
2017-10-19 18:18:04 +00:00