Commit graph

244 commits

Author SHA1 Message Date
Colin Cross
b1a5e9cadf zip2zip: add flag to uncompress files
Add -0 flag to convert files in a zip to stored instead of deflated.

Bug: 69500920
Test: zip2zip_test.go
Change-Id: I6c2b10f3b200a53a3339e3c97a78f65192b309ca
2018-10-08 15:20:56 -07:00
Nan Zhang
40b41b4038 Use SoongJavacWrapper for javadoc
Test: m -j docs; javac_wrapper_test
Bug: b/116874473
Change-Id: I51d440c815d3db1a0a5dc26ad4e376189dc0aefe
2018-10-02 16:45:37 -07:00
Dan Willemsen
2997123f73 Refactor Kati logic
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.

Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.

Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.

Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
2018-09-28 23:09:24 -07:00
Colin Cross
05518bc13b soong_zip: Add tests
Add test that cover basic command line usage of soong_zip.  -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.

Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
2018-09-28 13:56:06 -07:00
Dan Willemsen
bcc1dbf957 Refactor multiproduct_kati
Move the actual per-product builds out of the main function, combining
the product configuration and build sections. This makes it a lot more
readable.

This ends up saving a lot of FDs -- a -only-config build on aosp with 77
products used to require ~500 FDs, it succeeds with a limit of 150 FDs
now. I'm leaving the code to bump our FD limit though, since 2x the
number of internal products is too close to the 1024 limit for comfort.

Bug: 70370883
Test: prlimit -n150:150 build/soong/build_test.bash -only-config
Change-Id: Ia559beadc19deb8a5b9d50af6e0276e846fd8608
2018-09-05 13:57:10 -07:00
Dan Willemsen
41538385e5 Add an incremental mode to multiproduct_kati / build_test
This uses a very large amount of intermediates, but can speed up
subsequent multiproduct_kati runs by not re-reading all the makefiles
unless necessary.

Since we'll no longer have the full build output on every run,
explicitly save away a std_full.log of the log from the last time we ran
Kati.

Test: build/soong/build_test.bash -products aosp_arm -incremental
Change-Id: Iaae580c0e49a2024528617d39b2b477e29d99250
2018-09-05 13:53:59 -07:00
Colin Cross
632987ac21 pom2bp: clean up templates
Move some of the more complicated conditional logic out to helper
functions.  Use the {{- }} syntax to strip previous whitespace to
allow spacing out the remaning conditionals in the templates.

Test: (cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp)
Change-Id: I766bd0e1837aa04375f322fbe796d923cd99ecde
2018-08-30 11:18:50 -07:00
Colin Cross
8a9900a75f pom2bp: use java_import_host for host modules
java_library_host modules shouldn't depend on java_import modules.

Bug: 112707915
Test: (cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp)
Change-Id: Ia5ac353a696c9f51f1b50b334a327937ddcb47e7
2018-08-29 15:37:19 -07:00
Colin Cross
86bc9d4622 gofmt -w .
Test: m checkbuild
Change-Id: Ia4aec5ddadf1f1b00f7c567522ca89fd839504f0
2018-08-29 15:37:15 -07:00
Anton Hansson
179d89bc5b Merge "Pass extra args from multiproduct_kati to config." 2018-08-28 07:27:59 +00:00
Jeff Gaston
d492853a52 pom2bp support to specify which modules are host modules
Bug: 112804807
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ib816db8274cb05c9d0ed2cb557799ddcf6ee74fd
2018-08-24 14:32:50 -04:00
Anton Hansson
43da16a440 Pass extra args from multiproduct_kati to config.
Allows specifying make goals when using multiproduct_kati, for example:
$ multiproduct_kati -only-config dump-products

This makes dumping products with multiproduct_kati easier than before.

Test: multiproduct_kati -only-config -products=full,aosp_arm64 -out=/tmp/p dump-products
Change-Id: If526bd2ba1b7bbadcb230c020202ef9e421e1852
2018-08-20 16:16:02 +01:00
Dan Willemsen
59339a29e1 Fix go vet issues
Test: go vet ./...
Change-Id: Ifb936ccc5e2b5a2c3fcbbbcb54f680e2973ea1b3
2018-07-22 21:18:45 -07:00
Dan Willemsen
34218ece56 Make multiproduct_kati return error
Test: insert error, run build/soong/build_test.sh and check $?
Change-Id: I3462defd1ff04247a8b729e22988cbe002b0ce69
2018-07-19 22:57:18 -07:00
Dan Willemsen
c35b3813bb Fix missing errors during lunch
Bug: 111499523
Test: lunch foo
Test: m nothing 2>/dev/null
Change-Id: Ife888d4b731bc0a0dd253ac0c8c97d6c37593d73
2018-07-16 19:59:10 -07:00
Colin Cross
cdec7e7ee1 Merge changes I5b8522aa,I541aea75
* changes:
  Fix lint errors caught by go test
  Fix .kotlin_builtins glob
2018-07-17 00:14:33 +00:00
Treehugger Robot
6d6faa1a1f Merge changes from topic "merge_zips_strip"
* changes:
  Make merge_zips -stripFile use blueprint style globs
  Add tests for merge_zips
2018-07-17 00:09:58 +00:00
Colin Cross
fe5a3b7fac Fix lint errors caught by go test
Test: m checkbuild
Change-Id: I5b8522aa9da22fc595e4f26c23e7836de59bdfa2
2018-07-16 15:24:43 -07:00
Colin Cross
4c03f68763 Make merge_zips -stripFile use blueprint style globs
merge_zips -stripFile was only considering the name of the file and
ignoring the path.  Make it more useful by supporting blueprint style
globs.  The previous behavior can be recreated by prefixing with **/.

Bug: 111389216
Test: m checkbuild
Change-Id: I25760fe3f1f77704dd9da9d107d9a38a415d681f
2018-07-16 15:24:43 -07:00
Colin Cross
2486065c43 Add tests for merge_zips
Test: merge_zip_test.go
Change-Id: I8fc577946b40cad193864aa2ebda1bef3a0e050e
2018-07-16 15:24:22 -07:00
Dan Willemsen
b82471ad6d Add a unified status reporting UI
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.

For inputs:

Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.

Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.

multiproduct_kati loses its custom status routines, and uses the common
one instead.

For outputs:

The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.

The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.

A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.

Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.

Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
2018-07-12 14:15:31 -07:00
Colin Cross
461ba49a0a pom2bp: move misplaced {{end}}
manifest property should only be added to aar imports and not jar
imports.

Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ifaab81582c87f0af4419f90c3928d0d5d52cec33
2018-07-10 13:46:58 -07:00
Colin Cross
cf53e608a7 pom2bp: extract minSdkVersion from manifest into Android.bp
When manifest merging in Soong is turned on the modules created
by pom2bp will need to have min_sdk_version specified in order
to prevent manifest_fixer from setting it to the current SDK
level.  Read the minSdkVersion attribute out of the
AndroidManifest.xml file inside each .aar file.

Bug: 110848854
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.bp
Change-Id: Ieaf37ba6eccaf32dc5d3411566335830b1a327ff
2018-07-09 13:10:35 -07:00
Dan Willemsen
184901135c Revert "Revert "Revert "Revert "Add path interposer""""
This reverts commit 09f4540d66.

Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.

Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.

Also fixes some potential nil pointer dereferences.

Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
2018-05-31 14:59:33 -07:00
Dan Willemsen
09f4540d66 Revert "Revert "Revert "Add path interposer"""
This reverts commit c59a92cb1a.

Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.

Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
2018-05-25 23:26:00 +00:00
Dan Willemsen
c59a92cb1a Revert "Revert "Add path interposer""
This reverts commit 96c957ae20.

Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.

Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.

Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
2018-05-25 14:07:00 -07:00
Dan Willemsen
96c957ae20 Revert "Add path interposer"
This reverts commit a14704c12b.

Reason for revert: breaking builds

Change-Id: I920b0ff41823428a0baf59b3fd82cbcc766babca
2018-05-19 00:54:13 +00:00
Dan Willemsen
a14704c12b Add path interposer
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.

This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.

Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.

Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
2018-05-18 13:24:36 -07:00
Dan Willemsen
04d76ef47d Fix multiproduct_kati product list
When the config loading was changed to using AndroidProducts.mk lists
from soong_ui instead of finding them itself, multiproduct_kati stopped
being able to find all the products. Move the finder initialization
earlier so that we can dump the full list.

Test: build/soong/build_test.bash -only-config
Test: forrest runs of every downstream build_test
Change-Id: I085d40f9008a47b850783499365977a0ff70ac66
2018-05-15 00:54:25 -07:00
Dan Willemsen
4339853a20 Add a dependency fixer for proto deps
protoc dependency files, at least for C++ outputs, uses the form of:

  a/b.c \
  a/b.h: <dep1> <dep2>...

Ninja will fail the command when it parses a dep file and there's more
than one output file (even though it doesn't care what the output file
name is). So this tool will parse the original file, and output a
version with only a single output file.

Bug: 67329638
Test: NINJA_ARGS="-t deps ...pb.c" m
Test: NINJA_ARGS="-t deps ...srcjar" m
Test: NINJA_ARGS="-t deps ...srcszip" m
Test: Run dep_fixer across all of taimen's dep files, no failures.
Test: Run dep_fixer against the processed files, no changes.
Test: Run androidmk across all of our Android.mk files, inspect the diffs
Change-Id: I4263b7d5faea37285afa6b24dedf5964aa7d19dc
2018-05-07 16:21:59 -07:00
Colin Cross
66f78820e1 Regenerate R.java files from LOCAL_STATIC_ANDROID_LIBRARIES
If a static android library lib1 has static_libs: ["lib2"] then the
R.class files for packages in lib2 will be merged into the jar for
lib1.  If an app has lib1 in its static_libs it will get the R.class
files from lib2 through lib1, instead of regenerating the R.java
files with numbering that matches the resource table of the app.

Pass transtive static android library dependencies on the aapt2
command line so that aapt2 will always regenerate the R.java
constants for those packages.

Also extract the packages that have R.java files after each aapt2
invocation.  This is not necessary for Soong, but is passed to
make to let it force regenerating the packages using
--extra-packages.

Bug: 78300023
Test: m checkbuild
Change-Id: I0f3444af44d2a9f370d1f156c908972f8cc3a1ee
2018-05-02 13:11:01 -07:00
Colin Cross
70dd38f09e Add pom2bp
Convert pom2mk to pom2bp that writes out Android.bp files instead
of Android.mk files.  pom2mk stays for now until the last users
of it are cleaned up.

Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2bp -regen Android.mk
Change-Id: I584d63c4228bad32f9e1914b06bde807078d6a55
2018-04-24 13:37:38 -07:00
Colin Cross
6b382bcd04 Add pom2mk -exclude
Add an option to pom2mk to exclude modules by name.

Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk -exclude androidx.car_car
Change-Id: I083907ef364384aace524ced81820567f5075d76
2018-04-23 16:16:52 -07:00
Colin Cross
9f5633d60c Add pom2mk -regen
pom2mk -regen Android.mk will read the previous arguments out of
Android.mk, rerun pom2mk with those arguments, and write the result
to Android.mk.

Bug: 78300023
Test: cd prebuilts/sdk/current/support && pom2mk -regen Android.mk
Change-Id: Ie8e0f97ae5b87c8078cf8e753489326d80da1896
2018-04-23 16:16:41 -07:00
Jeff Gaston
358f25e97d pom2mk to include extra-deps as top-level deps too
Specifically, the use case is android-support-car requires some stubs that don't have a Maven pom.
So those stubs can't easily be listed as a dependency in the pom.
However, we do want the dependency to exist when importing to Make and detected by extract_deps.py

Bug: 76692459
Test: m -j blueprint_tools \
      && update_prebuilts.py -s <build_id> \
      && git diff HEAD^ -- fix_dependencies.mk \
      && m -j checkbuild

Change-Id: If546ecf7416580a01bde7b906976ada4b1145253
2018-04-23 15:59:05 -04:00
Dan Willemsen
3fb1faeeb9 Use Config/DeviceConfig functions to access ProductVariables
An upcoming change will stop exporting ProductVariables from Config, so
switch to using existing accessor functions, and add more when they're
missing.

Bug: 76168832
Test: out/soong/build.ninja is identical
Change-Id: Ie0135bdbd2df3258ef3ddb53e5f8fc00aa9b97f7
2018-04-11 01:45:14 +00:00
Jeff Gaston
af2191e201 Have pom2mk list all duplicates modules
For easier debugging if there are multiple duplicates

Bug: 76692459
Test: ./update_current.py -x <build-id>

Change-Id: I3079cb4ed1cfd365d792a7b41c9cdb4e7a1e148f
2018-03-29 17:44:36 -04:00
Jeff Gaston
8f08482957 Have pom2mk include runtime deps too
It still excludes test deps though

Bug: 76692459
Test: ./update_current.py -x <build-id>

Change-Id: I55a786e9feab31f99b95b7cebe84869757e46270
2018-03-29 17:44:33 -04:00
Colin Cross
f46e37f5f7 Fix format string issues
Fix issues caught by go vet.

Test: m checkbuild
Change-Id: Ib8d740457c15432dabe1575a6707726ddaf93084
2018-03-28 15:54:52 -07:00
Alan Viverette
2b53a0c4b8 Always include JAR dependencies (not just with --static-deps) in Makefile
Bug: 76692459
Test: ./update_current.py -x <build-id>
Change-Id: Ie0f77e52089adbe153d668fe39902545f469f2d2
2018-03-28 14:32:10 -04:00
Colin Cross
628d55d7ef Make zipsync list file output empty if there are no files
The make javac rule uses [ -s srcjar-list ] to decide whether or
not to run javac.  zipsync was putting a trailing "\n" in the
list file, so the file was never empty.  There was one case of a
package that contained no source files (so java-source-list was
empty) and had a res directory with an xml file that declared no
resources (so aapt.jar was created but empty).  The 1-byte
srcjar-list file caused javac to run and error with:
javac: no source files

Test: m checkbuild
Change-Id: I26b394b66bf81e4f5abbd27e4dc06bee1d9420a8
2018-03-22 13:02:39 -07:00
Colin Cross
436b76564d Replace extract-srcjars.sh with zipsync tool
extract_srcjars.sh uses zipinfo and unzip, which fail with an
error on an empty zip file.  Instead of trying to hack around
this (which is hard to make guarantees for since they are
non-hermetic host tools), replace them with a go tool to unzip
a set of zip files into a directory.

Bug: 73885582
Test: m checkbuild
Change-Id: I151fed347ed5196726e36866ffc27bc831799afb
2018-03-15 16:42:43 -07:00
Nan Zhang
1db85407aa Remove timestamp based filelist file for tracking Python dependencies
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.

Test: m -j checkbuild && real examples:
Bug: b/70568913

Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
2018-03-09 14:12:48 -08:00
Colin Cross
86803cfe6e add a library to report build numbers without causing rebuilds
Allow native modules to specify use_version_lib, which will make
an android::build::GetBuildNumber() function available.  For host
builds, the function will return the build number at the time that
the module was linked.  For device modules it will return the
value of the ro.build.version.incremental property.

Bug: 71719137
Test: build_version_test
Test: m build_version_test && touch build/make/core/Makefile build/soong/cc/libbuildversion/tests/build_version_test.cpp && m build_version_test shows different build numbers for binary and library tests.
Change-Id: I6f7d40b7574bb8206866c4e39bad9c710c796e32
2018-03-02 16:55:51 -08:00
Colin Cross
8673b5b959 Rewrite symbol_inject to be testable
Parse the machine-specific symbol tables into a custom symbol table
object and use a single function to find the offset and size of the
symbol in the custom table.  Make the functions to convert a
machine-specific symbol table into the custom table also take mocks,
and provide functions to dump mocks from real files.  Add tests
that take the mocks and verify they produce the right offset and
size.

Test: cmd/symbol_inject/*_test.go
Change-Id: I25654032b5017dd13a9a1fe29f8b8826ce5bc82b
2018-03-02 16:47:13 -08:00
Colin Cross
dfce764476 Fix finding next symbol when multiple symbols have the same address
Some exe files have a .data symbol at the same address as the
soong_build_number symbol.  If the .data symbol is after
soong_build_number in the symbol list, symbol_inject would think
the end address was the same as the start address, and use
uint32(-1) as the size.

Use sort.Search to find the first symbol whose section number is
the same as the target symbol, but whose address is higher than
the target symbol.

Test: manual
Change-Id: I51d6e53c6b906222ba68c5cf93be944843e23550
2018-02-28 15:04:59 -08:00
Colin Cross
c4a18e0291 Sort macho symbol table entries
macho symbol table entries are not always in order, which breaks
finding the next symbol to find the size of the target symbol.

Test: build_version_test
Change-Id: I41d1c3c3ff9929694e9ec2b034553d6b7ddef937
2018-02-23 22:45:55 -08:00
Colin Cross
64c6d4bf14 Fix symbol_inject on win32 exes
Win32 exes seem to prefix each symbol with an underscore like
macho.

Test: build_version_test
Change-Id: I9de0a0f5f25f93ad17d34b7c1b993511c77a57eb
2018-02-23 18:24:48 -08:00
Colin Cross
baccf5b984 Use __SBOX_OUT_DIR__ in sbox output file list
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.

Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.

Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
2018-02-21 14:55:34 -08:00
Colin Cross
52226ad920 Fix injecting data into a .o file
Use section.Offset instead of Addr.

Test: symbol_inject -i test.o -o test2.o -s symbol -v value
Change-Id: I9c54a5a245e7674b8370fc27ba7d0b9995f2ed08
2018-02-20 17:54:20 +00:00
Alan Viverette
6bd35eb6f5 Use "jar" as default type, "compile" as default scope
Only include dependencies for scope "compile."

Bug: 73263586
Test: ./update_current.py -s -t <build-id>
Change-Id: I2ac6055cb4c1ad1f8d7924869f54f50e5e662742
2018-02-13 15:21:18 -05:00
Colin Cross
5498f8548b Add a tool to inject data into an elf, macho, or PE symbol
Test: symbol_inject -i a.out -o a.out2 -s symbol -v value
Change-Id: I16cd8facbae754f679bef07ab0ba23638286e1d7
2018-02-06 19:12:21 +00:00
Alan Viverette
1593d3de6e Emit static dependencies into pom2mk-generated Makefile
Bug: 70516592
Test: ./update_current.py -s -t 4495436
Change-Id: Ic7dd433de78f9666c7388ed8ec1deebee7269368
2018-01-05 14:12:18 -05:00
Nan Zhang
5925b0fda4 Add -p, -e options in merge_zips for supporting par format.
Move __init__.py files population to merge_zips, and add options: -p, and
-e. Since par needs entry_point.txt to search the program entry point, so
add -e option to support it.

Test: ./merge_zips -p -e entry_point.txt out in1.zip in2.zip
Bug: 70568913
Change-Id: Ida10faf125e1b94dffc98b30240db3b90fd75b25
2017-12-20 12:57:10 -08:00
Colin Cross
e87040b2eb Allow Soong to generate its own build documentation
Add a rule in soong that re-executes soong_build in order to
generate build documentation.  This allows Soong to customize
the documentation.

Bug: 70516282
Test: m soong_docs
Change-Id: If143cfacd6ac20274cd7bb8d8fab0c07025a5553
2017-12-12 11:12:32 -08:00
Dan Willemsen
22de216799 Increase per-process file limits for multiproduct_kati
On large branches (250+ configurations), the open file count can go
over the default 1024 soft limit on Ubuntu. Many systems have increased
that default, but for the ones that haven't, at least opt into the
hard limit until this can be refactored to use fewer open files.

Bug: 70370883
Test: prlimit -n256:4096 build/soong/build_test.bash -only-config
Test: ulimit -Sn 256; build/soong/build_test.bash -only-config (darwin)
Change-Id: I7a952ffc89a0149ab65b04db1523d348daa6ab3e
2017-12-11 22:53:16 +00:00
Alan Viverette
75b95f8a61 Update pom2mk to allow duplicate module names if rewritten
Fixes: 70162730
Test: ./update_current.py -s -t 4482279
Change-Id: I874d7bfb50fd2d2bc488f5458cfe57b2e0d4d4e5
2017-12-05 12:53:44 -05:00
Jeff Gaston
088e29ed38 Revert "Revert "Soong support for namespaces""
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .

Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^ build/soong:work^' \
      'build/blueprint:work build/soong:work'
      # and see that the only changes were:
      # 1. adding some new files
      # 2. changing some line numbers
Test: m -j nothing # which runs unit tests

Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
2017-12-04 17:44:29 -08:00
Jeff Gaston
90cfb09214 sbox error message now lists the files that were created
which should make debugging faster.

Bug: 66921582

Test: ./out/soong/host/linux-x86/bin/sbox --output-root outs --sandbox-path out/.temp/sbox-work -c "cd __SBOX_OUT_DIR__ && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p" outs/a outs/b outs/z # and observe the below output:

mismatch between declared and actual outputs
in sbox command(cd out/.temp/sbox-work/sbox343858828 && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p)

  in sandbox out/.temp/sbox-work/sbox343858828,
  failed to create 2 files:
    b: not a file
    z: does not exist
  did create 16 files:
    a
    asubdir/child
    c
    d
    e
    f
    g
    h
    i
    j
    ...6 more

Change-Id: I75e37834c44d4279dec874701d67ce8bb01b872c
2017-12-01 13:21:35 -08:00
Jeff Gaston
178d5fefc0 Revert "Soong support for namespaces"
This reverts commit 63a250a336.

Reason for revert: Some failures:
  namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file 
  such as New Build Breakage: aosp-master/build_test @ 4475274

Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
2017-11-30 00:43:43 +00:00
Jeff Gaston
a8063afc16 Merge "Soong support for namespaces" 2017-11-29 23:01:39 +00:00
Treehugger Robot
b8160761f7 Merge "Revert "Revert "Test for dangling rules in make checkbuild""" 2017-11-29 21:09:48 +00:00
Jeff Gaston
63a250a336 Soong support for namespaces
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^ build/soong:work^' \
      'build/blueprint:work build/soong:work'
      # and see that the only changes were:
      # 1. adding some new files
      # 2. changing some line numbers
Test: m -j nothing # which runs unit tests

Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
2017-11-29 13:03:40 -08:00
Colin Cross
37193495cf Revert "Revert "Test for dangling rules in make checkbuild""
This reverts commit fb941913a3.

Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.

Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
2017-11-25 05:26:10 +00:00
Colin Cross
e909e1e079 Add support for --ignore_duplicates to merge_zips
Add a --ignore_duplicates option to cause merge_zips to
silently take the first entry in the case of duplicate entries
in the zips to be merged.  This will be used for soong Jacoco
support to combine the instrumented jar with the original jar
to pick up any classes that were excluded.

Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: I49ede3f07ce0ed7701b4db7058da2e4f11b8043e
2017-11-22 16:08:35 -08:00
Colin Cross
f3831d0e08 Support exclusions and Blueprint-style ** globs in zip2zip
Jacoco support will use zip2zip to create a jar that is a subset
of another jar, and will need exclusion filters and recursive
globs.  Switch zip2zip from filepath.Match to pathtools.Match,
and check each included file against the exclusion list.

Bug: 69629238
Test: zip2zip_test.go
Change-Id: Ibe961b0775987f52f1efa357e1201c3ebb81ca9c
2017-11-22 16:08:34 -08:00
Jeff Gaston
b61e3f79ec multiproduct_kati support for a custom list of products
Test: multiproduct_kati --products="aosp_arm,aosp_arm64"
Bug: 67478260
Change-Id: Icf7b8b0c30a2dc8e093ab109172675ff74f43adc
2017-11-15 15:31:38 -08:00
Colin Cross
c7d1254119 Merge "Revert "Test for dangling rules in make checkbuild"" 2017-11-13 23:41:24 +00:00
Colin Cross
fb941913a3 Revert "Test for dangling rules in make checkbuild"
This reverts commit 7709a05770.

Reason for revert: Broke mac checkbuilds

Change-Id: If71ec9198ca8943d90ce9d5e3a277faf239e38a5
2017-11-13 23:38:57 +00:00
Treehugger Robot
df93350ccd Merge "Test for dangling rules in make checkbuild" 2017-11-13 22:43:30 +00:00
Colin Cross
7709a05770 Test for dangling rules in make checkbuild
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory.  This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it.  The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly).  These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.

Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
2017-11-13 12:47:42 -08:00
Treehugger Robot
08bb51cdc5 Merge "Send soong output to std.log in multiproduct_kati" 2017-11-12 01:33:47 +00:00
Dan Willemsen
f196396a1c Send soong output to std.log in multiproduct_kati
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).

Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
2017-11-11 15:44:51 -08:00
Treehugger Robot
dbe8b78a5d Merge "pom2mk: Allow setting LOCAL_SDK_VERSION" 2017-11-10 23:53:44 +00:00
Dan Willemsen
15a8e793f3 pom2mk: Allow setting LOCAL_SDK_VERSION
For unbundled builds, LOCAL_SDK_VERSION needs to be set, otherwise we'll
try to use frameworks-res, which isn't available.

Bug: 64723465
Test: Use with prebuilts/sdk/update_current.py
Change-Id: If5e1233b4bcb09b3d0e2251e8dd1a8ecfc87eb19
2017-11-10 14:03:48 -08:00
Jeff Gaston
8a88db5a35 Have sbox remove its output directory before running
Bug: 38205169
Test: m -j
Change-Id: I73abb9921596c96fa056449bdf8250c2928bceda
2017-11-08 13:44:03 -08:00
Dan Willemsen
4eeb44b83f Merge "Change storage behavior of multiproduct_kati" 2017-11-07 22:25:24 +00:00
Dan Willemsen
4647be4afe Merge changes Ieaff757f,I5bdbba8d
* changes:
  pom2mk: Fix unlisted dependency types
  pom2mk: Add a flag to specify a specific version of artifacts
2017-11-07 20:59:02 +00:00
Dan Willemsen
e348076296 Change storage behavior of multiproduct_kati
Instead of deleting artifacts/logs from successful build (unless -keep
is set), and keeping unsuccessful artifacts, keep all logs and remove
all artifacts (unless -keep is set, then we'll compress the artifacts).

If -dist is passed in, we'll put an archive of the logs into the
DIST_DIR. Even compressed, the rest of the artifacts are still a bit too
large to dist (~5.6GB on AOSP).

Test: build/soong/build_test.bash
Test: build/soong/build_test.bash -keep
Test: build/soong/build_test.bash -dist
Change-Id: I87f55978c18c8ff2e517b8271554ba383003742f
2017-11-07 11:30:19 -08:00
Dan Willemsen
5f9d8a6698 pom2mk: Fix unlisted dependency types
Android Support Library 26+ does not specify dependency types, so parse
all pom files first, then propagate missing dependency types if we've
got the information.

Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 26.0.0-beta2 com/android/support
Change-Id: Ieaff757ff198c9a7b4b006623340b382728c1fd4
2017-11-06 20:55:02 -08:00
Jeff Gaston
93f0f37417 Use flag.Parse in sbox
Also removeTempDir -> !keepOutDir
Also shorten the help comment

Bug: 68336760
Test: rm out -rf && m -j
Test: sbox -c # and observe that the output does not have a stacktrace

Change-Id: I6fbb385132128ab1273c8398ff2425191049751e
2017-11-03 17:30:21 -07:00
Dan Willemsen
47e44a4c5c pom2mk: Add a flag to specify a specific version of artifacts
For repositories that contain multiple versions of the same artifact,
we'd only like to create makefiles for one of the versions of the
artifact.

Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 25.3.1 com/android/support
Change-Id: I5bdbba8d2143a47610d56d679381eb01b3cf176d
2017-11-03 13:40:08 -07:00
Jeff Gaston
02a684b8b2 Make $(depfile) work with sbox
Most notably, the sandbox depfile path should be passed into the tool.

Bug: 68336760
Test: m -j checkbuild

Change-Id: I22f944a3f57d613fda26de0ea777a915cafcd020
2017-10-30 17:49:29 -07:00
Colin Cross
5c6ecc179f Don't panic in turbine with no classes
ALLOW_MISSING_DEPENDENCIES=true can cause missing dependencies to be
hidden from GenerateAndroidBuildActions, which is triggering the
len(jars) == 0 check in compileJavaHeader.  For the
ALLOW_MISSING_DEPENDENCIES=true case the check is unnecessary, as the
build rules will be replaced with runtime errors.  On the off chance
the check would have been hit in a real build, allow merge_zips to
create an empty zip file if it is asked to.

Test: m ALLOW_MISSING_DEPENDENCIES=true
Change-Id: I3171f921d51229ddf38a1a647d32566658c673fa
2017-10-24 16:01:44 -07:00
Jeff Gaston
11b5c51d4e split soong_zip into a library and a binary
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)

Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
2017-10-19 18:18:04 +00:00
Colin Cross
a482065454 Use soong_javac_wrapper in soong javac compiles
Test: m checkbuild
Change-Id: I0f9ff0a719ac550a011c2878fc51595dc3427aed
2017-10-18 17:09:54 +00:00
Colin Cross
ae96e5376a Hide "bootstrap class path not set in conjunction with -source" messages
javac warns whenever -source is used with a version that does not
match javac, because it wants an rt.jar provided for the given
source level.  We are not using -source to prevent usage of APIs
that don't exist in the newer rt.jar, so hide the messages.

Test: javac_wrapper_test.go
Change-Id: Ic94b61767436848fc164e377dc3cc5a63726f1cf
2017-10-18 05:44:04 +00:00
Dan Willemsen
051133ba54 Add --dumpvar[s]-mode to replace config.mk uses
build/soong/soong_ui.bash --dumpvars-mode \
    --vars="..." \
    --abs-vars="..." \
    --var-prefix="..." \
    --abs-var-prefix="..."

is similar to the previous:

  CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
  make -f build/core/config.mk dump-many-vars \
    DUMP_MANY_VARS="..." \
    DUMP_MANY_ABS_VARS="..." \
    DUMP_VAR_PREFIX="..." \
    DUMP_ABS_VAR_PREFIX="..."

and

  build/soong/soong_ui.bash --dumpvar-mode [--abs] VAR

is similar to the previous:

  CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
  make -f build/core/config.mk dumpvar-[abs-]-VAR

But uses soong_ui and ckati, so that we use a consistent make parser and
sandboxing configurations.

One major output difference between the pure make implementation and
this one is that report_config in Go is implemented using embedded
newlines in single quotes, while the make implementation uses `` with
embedded echo commands. This seems to work fine for both bash and zsh,
and report_config isn't meant to be machine-parsed anyways.

Test: build/soong/soong_ui.bash --dumpvar-mode report_config
Test: build/soong/soong_ui.bash --dumpvar-mode TARGET_DEVICE
Test: build/soong/soong_ui.bash --dumpvar-mode --abs PRODUCT_OUT
Test: build/soong/soong_ui.bash --dumpvar-mode --abs ALL_PRODUCTS
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="report_config TARGET_DEVICE" --abs-vars="ALL_PRODUCTS"
Test: build/soong/soong_ui.bash --dumpvars-mode --vars=TARGET_DEVICE --abs-vars=PRODUCT_OUT --var-prefix=v_ --abs-var-prefix=a_
Change-Id: I0fbd0732bbf6fcfcd24084cf3c830a91a4b6bfc2
2017-10-17 12:57:41 -07:00
Dan Willemsen
37cba1471f Merge "Switch product configs from make to ckati" 2017-10-13 23:22:00 +00:00
Dan Willemsen
b2e6c2e571 Switch product configs from make to ckati
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).

envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.

Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config  (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
2017-10-12 18:11:08 -07:00
Jeff Gaston
c3bdc975a4 Make some params in soong_zip explicit and/or exported
in preparation to split soong_zip into a lib and a bin

Bug: 67478260
Test: m -j
Change-Id: I725da6f0c9b13069e3e5b036e2ac8cbdde9ba6bf
2017-10-12 12:51:52 -07:00
Treehugger Robot
29b41ed830 Merge "multiproduct_kati: Support skipping known-broken products" 2017-10-07 00:38:27 +00:00
Dan Willemsen
9957b9ccdc multiproduct_kati: Support skipping known-broken products
This can make it easier to add build/soong/build_test.sh to continuous
build branches to catch new breaks, but where some products are known to
be broken.

Test: build/soong/build_test.bash --skip-products aosp_arm,aosp_arm64
Change-Id: Ib240570b68957077047ca75b78658429cb8c6a49
2017-10-06 15:05:05 -07:00
Colin Cross
0cf45cdc1e Add -stripFile argument to merge_zips
Putting resources in dex jars is going to need to pull files from
classpath jars that do not match *.class.

Test: m -j checkbuild
Change-Id: Ia37b8b9387e5f5636769afc6937b9aa3c9ec1ced
2017-10-04 17:19:43 -07:00
Treehugger Robot
691ad1490c Merge changes from topic "host_bionic"
* changes:
  Extract the linker and embed it into host bionic binaries
  Add prefix_symbols support to cc_object
  Fix ToolchainClangCflags for host bionic
  Don't add a rpath to the linker
2017-09-21 00:46:41 +00:00
Dan Willemsen
c77a0b3f9c Extract the linker and embed it into host bionic binaries
The linux kernel requires that the ELF interpreter (runtime linker)
that's referenced by PT_INTERP be either an absolute path, or a relative
path from the current working directory. We'd prefer a relative path
from the binary, similarly to how we handle looking up shared libraries,
but that's not supported.

Instead, extract the load sections from the runtime linker ELF binary
and embed them into each host bionic binary, omitting the PT_INTERP
declaration. The kernel will treat it as a static binary, and we'll use
a special entry point (linker_wrapper) to fix up the arguments passed by
the kernel before jumping to the embedded linker. From the linker's
point of view, it looks like the kernel loaded the linker like normal.

Bug: 31559095
Test: Enable host bionic,
      out/soong/host/linux_bionic-x86/nativetest64/libdemangle_test/libdemangle_test
Change-Id: I8d0aea9790b5e86fcc3ea6e2d00cfa33907e2853
2017-09-20 15:34:33 -07:00
Colin Cross
635acc9446 Rearrange manifest file handling in merge_zips and soong_zip
Jar always puts default MANIFEST.MF files in if none was specified.
Copying that behavior in soong_zip causes problems with merge_zips,
because it ends up taking the default manifest from the classes.jar
instead of the user's manifest from res.jar.  We don't want the
user's manifest in the classes.jar, otherwise a change to the
manifest will cause all the class files to rebuild.  Instead,
move the manifest insertion to the final merge_zips stage.

Test: m -j checkbuild
Change-Id: Id6376961dbaf743c2fb92843f9bdf2e44b963be0
2017-09-20 13:20:45 -07:00
Nan Zhang
13f4cf5c9e Add -stripDir and -zipToNotStrip option in merge_zips.
In some cases, we need delete META-INF dir to void mis-using some
classes files in the Java runtime.

For now, when we delete META-INF, we need keep Manifest file in the jar
since merge_zips doesn't have ability to add a default one.

Bug: b/65455145
Test: m clean && m -j

Change-Id: Iea44b1a98d348cd8df1fa7374907733b1add6935
2017-09-19 18:44:43 -07:00
Nan Zhang
d5998cce7d Don't add data_descripters when merging uncompress zip entries for merge_zips.
Also filter out META-INF/TRANSITIVE dir, and report warnings when
merge_zips see duplicates entries with different CRC hash.

Bug: b/65455145
Test: m clean && m -j java (locally)

Change-Id: I47172ffa27df71f3280f35f6b540a7b5a0c14550
2017-09-13 23:01:49 -07:00
Colin Cross
34540315a0 Allow some duplicates in merged jars
Only take the first MANIFEST.MF or module-info.class file.

Test: m -j checkbuild
Change-Id: Ifbf9fe272437ef2c2bd51ab4849ac8d7ef37b6fc
2017-09-06 15:01:07 -07:00