Commit graph

120 commits

Author SHA1 Message Date
Elliott Hughes
5850f6f3be Use the canonical idiom for sockaddr_storage.
This works out a bit silly/ugly because the bits/ header file has the
wrong name, so I've also changed the map from kernel struct to boolean
to be a map from kernel struct to filename. That not only fixes this,
it's a bit more readable too. (Just yesterday, when I had no real reason
to change it, I was asking myself "why is this a boolean?"!)

Bug: http://b/236042740
Test: treehugger
Change-Id: I3eee25b493ea97d46cc5dc5fde07f7c5e77d2a46
2023-12-05 23:13:01 +00:00
Elliott Hughes
001fabaa90 Merge "Avoid multiple definitions of struct tcphdr." into main 2023-12-05 23:09:32 +00:00
Elliott Hughes
3906f3bc83 Merge "Avoid multiple definitions of sigaction." into main 2023-11-29 19:03:26 +00:00
Elliott Hughes
b4a9b17a55 Avoid multiple definitions of struct tcphdr.
No-one's hit this in practice, that I know of, but there are very few
instances of this old workaround for kernel/userspace mismatches still
present, and (as part of the much harder and less effective `struct
sigaction` cleanup), we should just remove them.

Bug: http://b/236042740
Test: treehugger
Change-Id: I6c71d4353044cf57cfa8a9796a4c3d6a4d51cd86
2023-11-28 15:41:54 -08:00
Elliott Hughes
df53b16fca Avoid multiple definitions of sigaction.
Before this change, we have the kernel's sigaction in the uapi headers,
and our own sigaction in <bits/signal_types.h> and we rely on callers
making sure to use `#define` to move the kernel type out of the way if
they include a uapi header directly. This is obviously error-prone and
undesireable, and not what we usually do now.

What we _usually_ do now is use the header scrubber's ability to replace
a struct definition with a `#include <bits/STRUCT.h>`, but that doesn't
work here because struct sigaction relies on a lot of other types,
some of which also come from uapi headers.

So instead use our second best trick, which is to "move the kernel struct
out of the way" at header scrubbing time instead. This means that someone
who does `#include <linux/signal.h>` or `#include <asm/signal.h>` won't
get `struct sigaction` (they'll only have `struct __kernel_sigaction`
instead), but it does mean that they can't get two incompatible
definitions if they include a uapi header both directly and indirectly.

So although this doesn't do what I'd set out to do, it's still an
improvement in some cases, and it's our preferred idiom in most cases
anyway. (I'll come back once this is in to tidy up the two other kernel
structs where we're still using the deprecated "rename out of the way
using #define" trick, but this change is already hairy enough, and
there's a possibility it will break code that didn't care that it was
getting the kernel `struct sigaction` rather than the userspace one.)

Bug: http://b/236042740
Test: treehugger
Change-Id: Icff50e330c09c587e8f77ba0fb7cffffd9c3b708
2023-11-28 14:20:43 -08:00
Elliott Hughes
844e8fc324 Stop the kernel header scrubber from duplicating includes.
We'll probably never hit this (because the case I hit it with, struct
sigaction, isn't amenable to the "replace with #include <bits/STRUCT.h>"
trick), but in case we do, and because a set expresses our intent better
than a list, keep the list^Wset of generated includes in a set.

Bug: http://b/236042740
Test: treehugger
Change-Id: I21f5c08515eab1b28e6a36fc00149b6bc7740b7e
2023-11-28 14:01:46 -08:00
Elliott Hughes
180edefbd2 Remove extraneous text from header comments.
Test: treehugger
Change-Id: I8e27e8ac15f5f1380046accbd7875da1c3b512f2
2023-11-02 00:08:05 +00:00
Edward Liaw
d903a736fb Add test for replacing macro arguments
Add test for aosp/2792161

Bug: 297317502
Test: cpp.py
Change-Id: I2eadd3b9371ec2f8b8c24107888cad5e3ae26f60
Signed-off-by: Edward Liaw <edliaw@google.com>
2023-10-17 20:09:42 +00:00
Edward Liaw
c291c39137 Replace macro arguments when applying replaceTokens
replaceTokens was only replacing tokens for cpp name clashes in the
macro body.  This change will also replace tokens in the arguments.

Bug: 297317502
Test: bionic/libc/kernel/tools/update_all.py
Change-Id: I102d000a8a4cea507b00c867df2a16106d8aed89
Signed-off-by: Edward Liaw <edliaw@google.com>
2023-10-17 17:42:40 +00:00
Elliott Hughes
b10c99d077 Keep ioprio_value; <linux/ioprio.h> relies on this.
We were keeping the macros that call this function, but not the function
itself. That's not helpful.

Bug: http://b/297317502
Test: treehugger
Change-Id: Icf8f734a129fe73ec740bb7cfbb11deb01a98cb3
2023-10-13 00:11:01 +00:00
Elliott Hughes
873c1ca3cb Add SYS_riscv_flush_icache.
Fix the uapi import script to look at the riscv/asm/ directory too, and
re-run it to add the missing SYS_riscv_* entry (there's only this one)
to glibc-syscalls.h.

Test: treehugger
Change-Id: Ie52c6ca1943c05bb615932174e49e7fb79725a7b
2023-04-25 12:13:56 -07:00
Elliott Hughes
0f0c18f695 uapi headers: simplify the <linux/compiler.h> stuff.
We don't really need <linux/compiler.h> and <linux/compiler_types.h>. We
already have a mechanism to remove unused macros, so let's do that. We
don't currently have a way to remove unused #includes, so we still need
<linux/compiler.h> and <linux/compiler_types.h> files (but I've clarified
the comments in them).

I've kept the empty definitions of `__user` and `__force` for source
compatibility. (We had one security test at least that was assuming
a kernel struct definition will "just work".)

Bug: http://b/262917450
Test: treehugger
Change-Id: Iacbbbc1aeef9a4fac52dabd7811ab875cc267d4f
2023-03-30 12:47:08 -07:00
Elliott Hughes
875d40559e Use <bits/timespec.h> from uapi headers too.
Bug: https://github.com/android/ndk/issues/1852
Test: treehugger
Change-Id: I554b31d2c4c443d37506e97a36099efbd3ad0e11
2023-03-20 15:07:16 +00:00
Colin Cross
4ac332279d Replace __packed with __attribute__((__packed__)) in uapi headers
Linux uapi headers use __packed as an abbreviation for
__attribute__((__packed__)).  In the upstream kernel headers this
is defined in linux/compiler_attributes.h, but it is not defined
in the uapi headers.  Bionic defines it in sys/cdefs.h, which will
almost always provide the needed definition, but there are uapi
headers that use __packed and do not include any system headers
that would include sys/cdefs.h.  Musl's sys/cdefs.h does not
define __packed, which leads to linker errors when __packed is
treated as a variable name.

Replace __packed with __attribute__((__packed__)) when importing
uapi headers to break the dependency on sys/cdefs.h.

Test: m USE_HOST_MUSL=true on oriole-userdebug in internal master
Change-Id: I45f80f4e5aec868b80ce8eb9d757a3e08ae3e1a9
2022-12-16 11:51:40 -08:00
Pat Tjin
00fde2aff6 Modify path for python clang support package
Due to commit 594844244237091347b62e8fbbc8a2851258db50 (update
stable to r468909b) the path for python3-clang support was moved from
   prebuilts/clang/host/linux-x86/clang-stable/lib64/python3/site-packages
to:
   prebuilts/clang/host/linux-x86/clang-stable/lib/python3/site-packages

Change the imports to point to the correct place.

Without the change:
  Traceback (most recent call last):
    File "bionic/libc/kernel/tools/./clean_header.py", line 72, in <module>
      import sys, cpp, kernel, glob, os, re, getopt
    File "bionic/libc/kernel/tools/cpp.py", line 19, in <module>
      import clang.cindex
  ModuleNotFoundError: No module named 'clang'

Bug: 254401594
Test: execute clean_headers.py
Signed-off-by: Pat Tjin <pattjin@google.com>
Change-Id: Ibf1423fbe78ee009366e26905c4a09e68ac8ff9b
2022-10-21 18:27:24 +00:00
Elliott Hughes
bba6ddff12 Kernel header scrubber: remove unused map entries.
I was going to add riscv64 and define __riscv_xlen to 64 here, but our
"preprocessor" in python can't simpify `MACRO_NAME == 123` expressions
anyway. Rather than add another empty map, remove the other unused maps
and explain why this functionality is still useful as long as the legacy
32-bit ABIs are still with us.

Test: treehugger
Change-Id: I553059942920e7d323a0ca8a6fd5f39bc2d35063
2022-10-06 20:40:44 +00:00
Elliott Hughes
de1df210bb Remove a mips-specific hack.
Rather than add risc-v to this map, let's just remove the map!

Test: treehugger
Change-Id: I118c74c61af95a96f8e31ae6e422619ed86c6a3b
2022-10-05 00:05:05 +00:00
Elliott Hughes
dc810a3c13 Remove unused variable.
Looks like we're not using this any more, so rather than add risc-v,
let's just remove this list.

Test: treehugger
Change-Id: I24bb22235c55d48c7dcbae94514ce8dcd03521c3
2022-10-04 20:21:20 +00:00
Elliott Hughes
05ef5d46c9 Add riscv64 to the list of uapi architectures.
Signed-off-by: Mao Han <han_mao@linux.alibaba.com>
Signed-off-by: Xia Lifang <lifang_xia@linux.alibaba.com>
Signed-off-by: Chen Guoyin <chenguoyin.cgy@linux.alibaba.com>
Signed-off-by: Wang Chen <wangchen20@iscas.ac.cn>
Signed-off-by: Lu Xufan <luxufan@iscas.ac.cn>
Test: bionic/libc/kernel/tools/generate_uapi_headers.sh
Change-Id: I926e1ceaab78b1bb9ae00cf39eb14005aae38e48
2022-09-30 21:37:57 +00:00
Christopher Ferris
3fc4e11607 Add support for removing structs that cross blocks.
The latest kernel has a struct that needs to be removed,
but the struct crosses blocks. Add support for removing a
struct that does cross blocks. This support is very primitive,
and will not parse any really complicated struct that uses
defines in complicated ways.

Combine the kernel_structs_to_remove and kernel_struct_replacements
structure into a single map. This allows marking a structure
to be removed as replaced with an #include <bits/STRUCT.h>.

The new support for the remove of structures is all in the
removeStructs function.

Raise an exception if the struct parsing does not work properly.

Add new unit tests for all of the new code.

In addition, fix the algorithm for deleting the uapi directory
before it gets updated. A new file BUILD was checking in that
directory, so delete everything in the directory except that
BUILD file.

Test: Unit tests pass.
Test: Running update_all.py results in the no unexpected changes.
Change-Id: I9a8cef0321beaf71d03b5b874327747a7edb6119
2022-08-09 17:49:31 -07:00
Christopher Ferris
e0cb4e1094 Fix old python2 test issue.
The map function returns a list in python2, but not in python3.
Force the test in cpp.py get_blocks() function to return a list.

Test: All cpp.py unit tests pass.
Change-Id: I1a63b92ac6cdb785830846fbd7e726eca7520e96
2022-08-05 12:59:05 -07:00
Colin Cross
d27506716a Define sockaddr_storage in bionic
Bionic currently renames the kernel's __kernel_sockaddr_storage
to sockaddr_storage.  We would like to reuse the bionic kernel
headers for musl, but musl defines its own sockaddr_storage,
causing redefinition errors when the musl and kernel headers
are both included.

Give bionic its own definition of sockaddr_storage so the kernel
one can be left as __kernel_sockaddr_storage, avoiding the
conflict in musl.

Some of the structs in netinet/in.h contain sockaddr_storage members.
There are definitions in linux/in.h, but they now use the
__kernel_sockaddr_storage naming.  To avoid having to copy the structs
from linux/in.h into netinet/in.h, add some #define hackery to
rename __kernel_sockaddr_storage to sockaddr_storage when including
linux/in.h from netinet/in.h.

Bug: 190084016
Test: m checkbuild
Test: bionic-unit-tests
Change-Id: I8f654511722b3a4a8e0bb146e4418ebcd370305c
2021-07-14 14:20:36 -07:00
Treehugger Robot
df901df77a Merge "Support replacing nested kernel structs" 2021-07-12 23:13:59 +00:00
Christopher Ferris
fcc3b4f45d Update to v5.13 kernel headers.
Kernel headers coming from:

Git: https://android.googlesource.com/kernel/common/
Branch: android-mainline
Tag: android-mainline-5.13

Test: Builds, bionic unit tests.
Change-Id: Ia4360ca4a2808206f64187dcb234d73c5752624b
2021-07-01 01:30:21 +00:00
Colin Cross
68ec53e141 Support replacing nested kernel structs
When replacing structs with bits/ includes, count the number of
opening and closing braces to find the end of the struct.

Test: tools/update_all.py
Change-Id: Ic9712a385da517710ceffccbdb223c89ae0b5f65
2021-06-29 14:22:30 -07:00
Christopher Ferris
e6b7969dd1 Merge "Fix last python3 issues." 2021-04-21 20:02:10 +00:00
Christopher Ferris
a2142d2835 Fix last python3 issues.
Includes pointing to the python3 version of the clang bindings.

Also, remove stale .gitignore line.

Test: Ran bionic/libc/kernel/tools/update_all.py and verified
Test: the files generated the same exact way.
Change-Id: I4eb9dd7382bca013f70d921b6ef48c7e7478615a
2021-04-20 17:36:07 -07:00
Dan Albert
8607c084c2 Fix overly-zealous strip.
The argument to this is the characters to strip, so `line.strip(line)`
just returns the empty string.

Test: None?
Bug: None
Change-Id: I4f62bffcd00936e4eef837a28b78023fcad54bb0
2021-04-20 16:05:02 -07:00
Christopher Ferris
ac7ec11473 Update for python3.
This fixes all of the problems with our kernel scripts, but not
the clang python script problems.

I also removed the updateGitFiles function since that code was
just silently failing any way. I replaced all calls with updateFiles.

Test: Ran script using python2 to verify it still works.
Test: Run script in python3 verifying that it starts to run.

Change-Id: I223a31a8324c59e6bc4067f48a6110361b3e26e8
2021-04-19 17:02:38 -07:00
Elliott Hughes
6b586e7709 Force everything to use python3 for consistency.
Rather than "whatever people have installed as 'python' on their machine".

I've removed check-symbols.py because that's been broken for years and
we never even noticed, and I'm not sure it's worth fixing.

Test: treehugger, manual
Change-Id: Ieb996bbdf790a18d4b1fb46a409cc240ba2a2a49
2021-04-15 13:39:08 -07:00
Christopher Ferris
ccf17f0ec7 Clean up header files from kernel directory.
Add a make distclean for each supported arch before generating headers
and at the end so these files are not left hanging around in the
kernel directory.

Bug: 174795055

Test: Verified all generated files are deleted after running.
Test: Verified this generates the exact same files as before this change.
Change-Id: I0f66f580af2ea50c190eb532032017e782d7cae6
2020-12-03 18:18:52 -08:00
Elliott Hughes
cf346532fc More cleanup for #inclusivefixit.
Found manually with grep, since the script seems to miss stuff.

Test: treehugger
Change-Id: I5933cbade9792801d4a0bec1ccb077efa6ad8fbc
2020-07-31 10:35:03 -07:00
Christopher Ferris
430527b4cd Update the generate script.
The arm kvm.h file has been deprecated, so nobody should be including it.
Therefore, remove the hack to copy the file out of the tools directory.

Also, update to remove the arm kvm.h file.

Test: Ran generate and update script and verified only the arm kvm.h
Test: was deleted.
Test: Builds and boots on a walleye.
Change-Id: I95dcc8877bdb2da2858b0b9d0aa95d1d2072cf9a
2020-06-04 12:32:09 -07:00
Christopher Ferris
af09c709d5 Update to v5.7 kernel headers.
Kernel headers coming from:

Git: https://android.googlesource.com/kernel/common/
Branch: android-mainline
Tag: android-mainline-5.7

Update the generate_uapi_headers.sh where the types.h file is now in
the right place, but kvm.h is not.

Test: Booted cuttlefish/bonito.
Test: Ran bionic-unit-tests on cuttlefish/bonito.
Change-Id: Ice9ce370a658e320b80f564b34a4431927fcf100
2020-06-02 10:58:28 -07:00
Christopher Ferris
bb9fcb4636 Update to v5.6 kernel headers.
Kernel headers coming from:

Git: https://android.googlesource.com/kernel/common/
Branch: android-mainline
Tag: android-mainline-5.6

Add a new method for removing structures. This is to deal with the kernel
headers changing some definitions of timeval to __kernel_old_timeval
and itimerval to __kernel_old_itimerval. Remove the __kernel_old_XX
strutures and change the other structures to the previous definitions.

This only works so long as these structures stay the same, if they
diverge, then a different strategy will need to be implemented.

Test: Booted cuttlefish/walleye.
Test: Ran bionic-unit-tests on cuttlefish/walleye.
Change-Id: I0a61f4fa6e4155c602e0414d9b38c2e1637829af
2020-04-10 10:53:48 -07:00
Christopher Ferris
d32ca14ea3 Update to v5.5 kernel headers.
Kernel headers coming from:

Git: https://android.googlesource.com/kernel/common/
Branch: android-mainline
Tag: android-mainline-5.5

Test: Boots on walleye.
Test: Ran bionic-unit-tests on walleye.
Test: Boots on cuttlefish.
Test: Ran bionic-unit-tests on cuttlefish.
Change-Id: I57387d3c31e0ba5ad125ffe291cecf365c7b374e
Merged-In: I57387d3c31e0ba5ad125ffe291cecf365c7b374e
2020-02-07 20:14:42 +00:00
Elliott Hughes
c2faf235c0 Stop generating unused headers.
These just cause confusion because they often have different
values/layouts, but they're never actually used.

Test: treehugger
Change-Id: I424034088e017c919f62fcefa7d6d3f903f31cfb
2020-02-03 17:56:06 -08:00
Elliott Hughes
09e77f35ab _FILE_OFFSET_BITS=64 support for fcntl.
On Android, fcntl is always implemented by fcntl64(2). This means that
an LP32 binary can `fcntl(F_SETLK, struct flock)` (because fcntl64(2)
passes through to the 32-bit fcntl(2) to handle F_SETLK), and it can
also `fcntl(F_SETLK64, struct flock64)`. What it can't do before this
patch is set _FILE_OFFSET_BITS=64 and then `fcntl(F_SETLK, struct
flock)` where that `struct flock` is actually implicitly `struct
flock64`.

Move the kernel uapi structs out of the way, define them ourselves based
on __LP64__ and _FILE_OFFSET_BITS, and fix up the relevant F_ constants.

(Also add a .clang-format to turn off clang-format in libc/include/.)

Bug: N/A
Test: treehugger (and strace!)
Change-Id: Iccd6c83d9133e1efcf93a7b49a6ae0f1bbd3d58b
2020-01-29 19:36:14 -08:00
Josh Gao
afe7e53003 Clone with --depth 1 in generate_uapi_headers.sh --download-kernel.
Test: manual
Change-Id: Id60b354fb1914470315976dd103d2665a6fefab8
2019-12-13 17:24:44 -08:00
Christopher Ferris
9584fa46e9 Update to android-mainline kernel headers v5.4.
Update generate_uapi_headers.sh to checkout the android mainline
kernel. Also, add a small modification to look for the kernel directory
in common not linux-stable.

Remove deprecated android headers from android/uapi/linux. Also,
remove f_accessory.h since it's in the android mainline kernel.

Test: Builds and runs on walleye.
Change-Id: Ia371305e19f56e6bcc2db6d5b4d299819f07ffc6
2019-12-12 14:59:25 -08:00
Christopher Ferris
fa59a10bcd Update to kernel headers v5.2.6.
Also, fix a bug in update_all.py when the syscalls file does not change.

Test: Builds and boots on a taimen.
Change-Id: If85b00daef2c176f804a0861894f5bbca9c6d5df
2019-08-05 12:59:58 -07:00
Daniel Mentz
d12d6f67bc clean_header: Run outside of $ANDROID_BUILD_TOP
Enable the use case where we run clean_header.py from outside of
$ANDROID_BUILD_TOP. Previously, this script required the current working
directory to be under $ANDROID_BUILD_TOP. Running it from a different
directory resulted in the following error message:

 clean_header.py: error: Not in android tree pointed at by ANDROID_BUILD_TOP (....)

Change-Id: I48210ea1a0033228a9aaa4124d28247b07cee6d4
2019-04-15 15:19:31 -07:00
Daniel Mentz
6d6b4cedd1 clean_header: Fix error handling for no-such-file case
The function cleanupFile should always return a single value (as opposed
to a tuple or list). In addition, if it encounters an error, it is
expected to return a value that evaluates to False. As it stands,
however, it returns (None, None) in certain error cases. Change this
function to return None, in those cases.

We previously saw the following error message, when we tried to run
clean_header.py on a non-existent file.

Traceback (most recent call last):
  File "clean_header.py", line 208, in <module>
    b.updateGitFiles()
  File "utils.py", line 164, in updateGitFiles
    self._writeFile(dst)
  File "utils.py", line 136, in _writeFile
    f.write(self.new_data[dst])
TypeError: expected a string or other character buffer object

Change-Id: I5f717dd1a4388f598f0fd4bfd5e6129017de9095
2019-04-15 15:03:39 -07:00
Elliott Hughes
c4c2e24d5f <bits/glibc-syscalls.h>: only regenerate when we have new uapi headers.
Test: update_all.py
Change-Id: Iaa92dce263197f5a0e7d2dce5e00a31372dcb3e9
2019-04-11 14:19:17 -07:00
Daniel Mentz
bb4cf7b8a7 clean_header: Don't print extraneous newline char
Don't print an extraneous newline character at the end of the include
file if clean_header.py is used in the non-update mode. This is
necessary to achieve consistent results in the update and non-update
modes.

Running clean_header in the non-update mode and redirecting the output
to a file should have the same result as using the update mode and have
the script write to the file directly.

Change-Id: I6b176c5365840d66e4499bacd205f1fa77302a2b
2019-03-29 15:17:48 -07:00
Daniel Mentz
316f4a499c clean_header: Write to correct dst_file
Write to correct dst_file when in update mode. This enables use cases
like the following:

../../../bionic/libc/kernel/tools/clean_header.py -k original-kernel-headers/ -d kernel-headers/ -u linux/ion.h

Previously, we had to cd into kernel-headers/ and then run

../../../../bionic/libc/kernel/tools/clean_header.py -k ../original-kernel-headers/ -d . -u linux/ion.h

because the tool didn't allow the relative path to be different from the
destination path.

Change-Id: I8c5e284ce7a6737d77a2f5ead3e7e5db01317425
2019-03-29 21:41:15 +00:00
Christopher Ferris
658b16fd90 Rewrite removeVarsAndFuncs.
The current version has these bugs:
- Adding a semicolon after a function results in the removal of structures
  following the function.
- Function like macros get removed on accident rather than on purpose.
- It removes extern "C" { completely, which might not be a bug, but doesn't
  seem right.

I couldn't easily fix any of these problems because the code depends heavily
on the header being correct.

New unit tests added for the function to cover all of these cases.

A follow-on CL will include the updated headers.

Bug: 112290385

Test: Passes all new unit tests.
Test: When run on the current kernel headers, the generated headers are
Test: nearly the same, missing data is being added.
Change-Id: Ib22a5f2e78873544e8a9d54e385af1156b2a72bb
2019-01-10 15:15:15 -08:00
Christopher Ferris
f2484aaed7 Fix handling of #elif.
When a construct like:

if defined(something)
  blocks1
elif 1
  blocks2
else
  blocks3
endif

The parser would put the first clause but then simply omit the elif and
put all of blocks2 without a terminating #endif. The code also did
something similar when the #else was an #endif.

Also convert all of the unit tests to real unit tests and only run
them if you run cpp.py by itself. Added new unit tests to cover the
new cases.

Test: Ran cpp.py unit tests.
Test: Reran update_all.py and verified nothing changed, and that running
Test: it on the new kernel headers that exposed this problem.
Change-Id: Ie168511303c4e15afdb60c37baef75a966ca29a8
2018-10-26 22:42:26 +00:00
Christopher Ferris
764049dce4 Modify the script to use the linux source tree.
The current code checks out the android kernel tree. Long ago, we moved
to using the linux kernel source code. Modify the script to actually
do this.

Bug: 6653610

Test: Ran the script using the --download_kernel option and without.
Change-Id: I769e769edea272f4f8c9c8d2e76bd336059c66e3
2018-05-24 15:10:33 -07:00
Christopher Ferris
5ed15ba733 Remove ion.h from bionic uapi headers.
Bug: 77976082

Test: Ran the generate script and verified ion.h is deleted.
Change-Id: I33b3aeef66f08f35dd496a073931b2fe2c80a1d6
2018-04-24 13:33:30 -07:00