When making this script deterministic we accidentally changed it so that
none of the entries in the output zip file were compressed. Re-add
the compression.
Adding the compression makes atest go from 67M to 34M, at the cost of
0.33s startup time instead of 0.28s.
Test: Presubmits
Change-Id: I4e58eb76235ddabf0195694cdd3644f26b7e84ec
Now a stacktrace involving precompile_python.py won't be shown,
and it will show all errors instead of just the first one.
Bug: 331488610
Test: Presubmits
Change-Id: I47376ca6764c0f84d1aaf5db329664e20143640a
This CL fixes timestamps for zip entries to a hardcoded value to ensure
deterministic zip archives across checkouts, not just for incremental builds.
The value chosen for the fixed date is the same as the one used by soong_zip.
Bug: b/322788229
Test: Ran a few builds and verified that mdate for zipped files are
fixed as well as the generated zip files have identical hashes.
Change-Id: I3cd6fcf559d4d83d8813c93319b46e267ae64a2b
Timestamps in pyc files and zip archives were causing cache misses in RBE.
Bug: b/322788229
Test: Ran a few local builds and verified improved cache hit rate for metalava actions.
Change-Id: I9375694d26a107f5c0c43bbd685bd8e86349d187
This signifigantly improves the startup time of soong-built
python binaries. For example, running
`m apexer && time out/host/linux-x86/bin/apexer` gives
0.734s before this cl, and 0.094s after.
Fixes: 259718110
Test: Presubmits
Change-Id: Ib19e83e2c60c39a849525be117279c318de3afa7
The new behavior has been enabled by default, and these
flags aren't necessary anymore.
Fixes: 245583294
Test: m py_dont_import_folder_of_entrypoint_test && /ssd/aosp-master/out/host/linux-x86/testcases/py_dont_import_folder_of_entrypoint_test/x86_64/py_dont_import_folder_of_entrypoint_test
Change-Id: I5b6f98da51791bc5d28662ef799a10c1bb6a35a0
The python interpreter will by default add the directory of
the entrypoint script to the beginning of sys.path. This
can be disabled in python 3.11+ (which is not released yet)
using the PYTHON_SAFE_PATH environment variable or the -P flag.
As a workaround to have this behavior in older python versions,
we can make an __soong_entrypoint_redirector__.py file at the
root of the zip file that is the entrypoint, and then that
file will redirect to the real entrypoint.
This brings non-embedded-launcher python modules closer to
the embedded launcher version. The embedded launcher binaries
already act like this because they start at an __main__.py file
at the root of the zip file.
Bug: 245583294
Test: m py_dont_import_folder_of_entrypoint_test && out/host/linux-x86/nativetest64/py_dont_import_folder_of_entrypoint_test/py_dont_import_folder_of_entrypoint_test
Change-Id: I39aaf04fb19c3ba7f5c9d98220872d6d08abf736
stub_template_host.txt added all the top-level modules to the
PYTHONPATH, which isn't correct, and caused absl.logging to
override the built-in logging module.
Removing this also makes it more consistent with python binaries
built with embedded_launcher: true. embedded_launcher: true
binaries don't add the top-level modules.
Fixes: 245583294
Test: m py_dont_add_top_level_dirs_test && out/host/linux-x86/testcases/py_dont_add_top_level_dirs_test/x86_64/py_dont_add_top_level_dirs_test
Change-Id: Id3069565d2b2c4b2bda0ff5301e757a7b4201751
Python programs run via a wrapper script. That wrapper script
can be started with open file descriptors, for example by using
the <(echo foo) syntax in bash. The shouldn't be closed when
starting the real script.
Bug: 203436762
Test: Presubmits
Change-Id: Ib20450f713b15c8a5527d1e433bfd9635cead85f
Make sure the shebang of the par file is consistent with the main python
script rather than use /usr/bin/python arbitrarily.
Bug: 185399398
Bug: 185094069
Test: m atest ; atest-dev --help
m atest-py2 ; atest-py2-dev --help
Change-Id: Ice4590b3d173e6d11e608c2ca29b297df0dc4089
Uses more python rather than C++, and skips less of Py_Main.
Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
Prevent problem for some users using PAR file built by android's
build system which cause by the different site package of python
installed by user themselves.
Bug: 120469915
Test: 1. m atest
atest --help
2. apply aosp/861470
full build
Change-Id: I72c83f9397f0e88eecfcd32806f1746520299447
Otherwise, python will complain the variable has not been initialized,
in case ExtractRunFiles throws an exception.
Test: m
Change-Id: I3d9e3cda5e558436ee4182c44e4345151c031e41
The problem came from Python libraries doesn't know the information that
we enabled embedded_launcher (only Python binary knows about that). And
we can't simply remove runfiles dir for Python libraries since host
Python mode need this.
Bug: b/80441699
Test: m perf_profo_flames
Change-Id: I73ffc4d7504f95a708ae7cca47bc6c15a673aa31
This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
The base module handles all the common functionalites, such as version
compatibilty check, version variations split, source file format check,
source/data file duplicate check.
The library/binary module focuses on how to generate binary build actions,
such as setting up stub script, zipping, filling in __init__.py in
runfiles dir tree.
Bug: b/31676493
Test: go test under python package
Change-Id: I06608369f350f7195873d459e1c8d1bdb811e77e
