While the 9a and 9 pro are very similar, for comunity based development this is substantial.
I am often very critial, but I must give props to the grapeneos team.
But one could argue that the team can move fast on 9a because they can piggyback existing (GrapheneOS) distros.
Treble's compatibility system isn't very relevant to us right now. There's a new Android release every month: a monthly, quarterly or yearly release. The devices we currently support (Pixels) receive each of these updates officially. Most Android devices do not get the monthly or quarterly updates, only the yearly ones. Other devices rely on partial backports of security patches (Android Security Bulletins) to an initial release, which are provided for ~3-4 years after the initial yearly release. If we supported typical Android devices with that situation, then we'd at least partially rely on Treble to provide the latest OS version. Pixels are currently the only devices meeting our hardware security requirements listed at https://grapheneos.org/faq#future-devices. Having proper updates for 7 years from launch is just part of that, most of the requirements are for hardware-based security features like the secure element features, hardware memory tagging, pointer authentication, USB controller support for blocking new connections and disabling USB data, etc.
GrapheneOS uses the 6.1 and 6.6 long term support branches of the Linux kernel with 6.12 as the next one that's likely going to be used to replace 6.6 for on-device virtual machines and the emulator with Android 16.
But they're drivers that are not upstreamed and which therefore make it hard to move to a newer kernel, right?
We automate a huge portion of the work via https://github.com/GrapheneOS/adevtool. We do a GrapheneOS build with it and it outputs state you can see in https://github.com/GrapheneOS/vendor_state which is then used to automatically figure out all of the missing overlays, SELinux policy, firmware files, etc. When we have our own devices in partnership with an OEM we won't need to use adevtool. We can borrow a lot from Pixels for those devices though.
Pixels are very similar to each other, which does make things simpler. The entire kernel source tree is identical for 6th, 7th, 8th and 9th generation Pixels. They all use the Linux 6.1 long term support branch on bare metal and the Linux 6.6 branch for on-device virtual machines. They'll likely advance to new Linux kernel branches together rather than ending up very split across different ones as they were in the past. That makes things easier for us.
Pixels also share most of the same drivers for the SoC and lots of other drivers. Those drivers support the different generations of hardware with the same codebase for the most part. There are still 6 different Wi-Fi/Bluetooth drivers across them but 5 of those are variations of a very similar Broadcom Wi-Fi/Bluetooth driver and only 1 is a separate Qualcomm Atheros driver (Pixel 7a).
We have various hardware-based hardening features such as our hardware-based disabling of the USB-C port with variations across different hardware generations (https://grapheneos.org/features#usb-c-port-and-pogo-pins-con...) and similar features. Our exploit protection features also uncover lots of memory corruption bugs across devices in their drivers. We do have a lot of device-specific work fixing uncovered bugs. Hardware memory tagging in particular finds nearly every heap memory corruption bug occurring during regular use including out-of-bound reads so that finds a lot of bugs we need to handle. Many of the bugs we find with hardware memory tagging and other memory corruption exploit protections are in drivers or the portable Bluetooth software stack which is thankfully one of the components Android is currently gradually rewriting in Rust along with the media stack.
If we supported a device with much different drivers, there wouldn't be much work to deal with that directly but enabling our features like our hardware memory tagging support would require fixing a bunch of memory corruption bugs occurring during regular use across it. Supporting other Android devices with the Android Open Source Project is easy. Supporting them with GrapheneOS is significantly harder due to various hardening features needing integration at a low level along with others uncovering a lot of latent bugs which were occurring but not being noticed most of the time. The ones which get noticed often due to breaking things get fixed, but many latent memory corruption bugs remain there unless the OEM heavily tests with HWASan or MTE themselves, which is unlikely. Pixels are tested with HWASan and MTE by Google but yet we still have to fix a lot ourselves largely because testing them in a device farm is different than users actually using them with assorted Bluetooth accessories, etc.
It's no harder than it would be dealing with them if they were upstream. Google ports all the Pixel drivers to newer LTS branches and a recent branch of the mainline kernel themselves.
With the recent Android 15 QPR2 release last month (March 2025), 6th/7th generation Pixels moved from the 5.10 branch to 6.1 and 8th generation Pixels moved from 5.15 to 6.1. 6th, 7th, 8th and 9th generation Pixels share the same kernel and kernel driver source tree now. They have them ported to 6.6 and mainline kernels too, it's just not ready to ship yet. 6.6 is used for virtual machines run on the devices and the emulator. 6.12 will be what's used for Android 16.
You can see at https://android.googlesource.com/kernel/google-modules/aoc/+... that they still port the 6th gen Pixel drivers to newer mainline kernels. They're ported to 6.13 and probably moving along to 6.14 soon. It doesn't mean they're going to ship them. Only LTS branches make sense to ship and they need long term stabilization first. The likely model is going to become ~12 months of stabilization and then ~12 months of usage since LTS kernel branches are moving back to 2 years of support. It was essentially increased to 6 years for 6th gen Pixels having 5 years of support, but they moved along to upgrading to newer LTS branches for 8th gen Pixels moving to 7 years of support. Greg KH works for and with Google on the LTS kernel maintenance / testing so it's actually quite Android centric rather than server centric. Long term support desktop/server distributions historically maintain their own LTS branches so they're not really the ones involved in that for the most part.
Drivers that are upstream don't actually get much free maintenance and testing. People making API changes blindly update the drivers without testing them. They get lots of updates and refactoring, but not much ongoing maintenance. Maintainers still need to deal with it. It's very common for upstream drivers to continuously regress. They can go a long time without actually working properly across releases due to how few people use them. Most people are using distributions with frozen package versions like Debian, not a rolling, and people using a rolling release like Arch Linux can use an LTS kernel branch to avoid a lot of this. The drivers for embedded hardware and things not used much by enthusiasts on desktops often break without it being noticed.
Android made a Generic Kernel Image system with ABI stability for drivers which does not benefit Pixels because they update the drivers to match the latest GKI kernel they ship. Similarly, Pixels don't really need the Treble HAL ABI forwards compatibility because they update all the vendor code to the latest monthly, quarterly and yearly OS releases anyway. It's helpful that drivers don't need to add all the new standard features to keep providing working support for new OS versions though. It's nice having it all nearly all neatly standardized and stable. We like Treble because of the sandboxing. The forwards compatibility benefits are largely unrealized because the vendors needing it aren't doing updates much anyway. Qualcomm is moving to 8 years of full update support for Snapdragon to partially match Pixels anyway.
First: I did momentarily forget that you're only targeting Pixel devices that are actively getting updates from Google. In light of that, so long as Google stays on top of maintaining those devices, yeah in your case that's probably fine. I'm somewhat accustomed to less responsible vendors and a lot of my views are shaped by that.
That said, I'm not wholly convinced that Google's downstream kernels are as good as running from upstream. AFAICT, for example, GrapheneOS is currently shipping a kernel for the Pixel 6 that's 3 minor versions behind. Trying to track things through the Android development process is... unintuitive... to me, so forgive me if I've missed something... If I go to https://github.com/GrapheneOS/device_google_raviole-kernels_... , grab a .ko file that shows as being committed yesterday, and run modinfo on it, I get a version of 6.1.131 which https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.1... says is from March 13 and has been superseded by 6.1.134 at this writing (from checking https://www.kernel.org/ ). Contrast https://archlinux.org/packages/core/x86_64/linux-lts/ which says that Arch's LTS kernel is at 6.12.23 which is the latest of that line. EDIT: Actually, the much better comparison is that Debian 12 is shipping 6.1.133 according to https://packages.debian.org/stable/kernel/linux-image-arm64 now. So the super stable slow moving distro is in fact still ahead of Android, even slightly lagging as it is.
As to breakage/testing... Yes, someone has to test new versions. Ideally that'd be a CI farm flashing and testing devices, but I appreciate that it's not exactly a trivial problem. Nonetheless, if that results in Graphene not shipping the latest bugfixes, I feel like that's an extremely awkward position to be in.
Adding new Pixels including whole new generations tends to be quite easy. When we still supported Snapdragon Pixels in the official GrapheneOS branch, it would have been fairly easy to add support for a theoretical Sony or Motorola device meeting our security requirements (https://grapheneos.org/faq#future-devices). Now that we don't have a Snapdragon device, we'd need to deal with fixing or working around all the bugs uncovered in Snapdragon drivers, integrating support for the USB-C controller into our USB-C port control feature (https://grapheneos.org/features#usb-c-port-and-pogo-pins-con...), adding back our code for coercing Qualcomm XTRA into being privacy respecting, etc. Snapdragon doesn't have memory tagging yet like Tensor (and recent flagship Exynos/MediaTek now), but pretending it did, we'd need to solve a lot of issues uncovered by it unless Qualcomm and the device OEM heavily tested with it.
See >>43669913 for more info including about kernels. 6th, 7th, 8th and 9th generation Pixels share the same Linux 6.1 source tree and kernel drivers since Android 15 QPR2 in March 2025.
Pixel 9a is still using a branch of Android 15 QPR1 due to how device launches work so most of the work involved taking our last Android 15 QPR1 release from early March and rebasing it onto the Pixel 9a device branch tag where they forked off from an earlier Android 15 QPR1 release and backported current security patches to it. We then had to backport our changes since early March. The device branch will go away in a couple months and it will be on the same branch as everything else until it's end-of-life as usual. We could spend more time to integrate it into our main Android 15 QPR2 based branch ourselves but we can also simply wait a couple months. As an earlier example, the Pixel 8a was released in May 2024 based on Android 14 QPR1 rather than the current Android 14 QPR2. It was incorporated into mainline Android 14 QPR3 in June only a few weeks later. We know Android 16 is already going to deal with this so spending our time on this instead of implementing new privacy, security, usability and compatibility improvements would be a waste.
Pixels only recently started moving to new LTS releases and will likely be moving to a new LTS release each year going forward. Moving the older generations to 6.1 to match current devices was done in March 2025. They'll likely move along together to a new branch each year going forward.
Linux kernel LTS revisions are nothing like LTS revisions of most other projects. They're largely untested patches blindly applied by the LTS maintainers based on patches to mainline being marked for stable backports. If the patches apply cleanly, they ship. If they don't apply cleanly, they largely don't ship. Whether it works is an open question.
> GrapheneOS is currently shipping a kernel for the Pixel 6 that's 3 minor versions behind
That's not quite right.
We're shipping the latest Linux 6.1 and Linux 6.6 GKI LTS branch releases from Greg KH. They're currently in between 2 upstream revisions upstream, not on a specific one. The devices all use both 6.1 and 6.6, not just 6.1. They use 6.1 for bare metal and 6.6 for virtual machines. Even the Pixel 6 has been ported to 6.13 by Google but that doesn't mean that's a stable enough kernel ready to ship.
The Android kernel branches also have a bunch of backports not included in the kernel.org LTS releases, including security patches and improvements they decided were important. Google does their own backporting and fixes in the GKI branch and Greg KH merges those into the GKI LTS branch. The kernel branch we use is the combination of the Google GKI backporting/fixes with the kernel.org backporting. The kernel.org LTS releases are far messier than you realize, and combining these things is messy too.
Linux LTS kernels are not very well tested and have tons of regressions. Quickly updating to the new LTS versions is problematic and we regularly encounter major regressions, especially in certain areas like f2fs and USB. We still update right away to the new GKI LTS versions. We're currently on the latest GKI LTS releases for each branch.
You'd have to ask Greg KH why there are still delays despite Google supporting it. It still seems to be him doing most of the kernel.org LTS and also GKI LTS work by himself, without nearly as much review or help by others as you would think. This is also tied into the severe regressions regularly happening with the LTS releases. Those can be security regressions too. Immediately updating to them is not necessarily a great idea with how much goes wrong at the moment.
They unfortunately sometimes lag behind the kernel.org releases. We used to merge the latest upstream kernel.org LTS releases ourselves but Greg KH convinced us we don't need to do that anymore and should just use the GKI LTS branch instead. We're not completely happy with it since it's not fully kept in sync but we're using our resources elsewhere at the moment.
> Actually, the much better comparison is that Debian 12 is shipping 6.1.133 according to https://packages.debian.org/stable/kernel/linux-image-arm64 now. So the super stable slow moving distro is in fact still ahead of Android, even slightly lagging as it is.
Debian is usually further behind than Greg KH's GKI LTS branch. Comparing at a snapshot in time doesn't mean much. GKI LTS branch should really be kept in sync but the GKI ABI stability system makes maintenance hard and is entirely worthless for Pixels. We would prefer if the whole GKI system did not exist. For Pixels, the kernel image and all the drivers are built with the same kernel source tree for Pixels so the whole system for driver ABI compatibility is just making things more complex and worse.
> As to breakage/testing... Yes, someone has to test new versions. Ideally that'd be a CI farm flashing and testing devices, but I appreciate that it's not exactly a trivial problem. Nonetheless, if that results in Graphene not shipping the latest bugfixes, I feel like that's an extremely awkward position to be in.
We do heavily test them. Our community helps with it. We OFTEN find regressions in the new LTS kernels. It often takes months before the issues we find and work around get fixed upstream. It's worth noting that due to mistreatment we've largely stopped helping them except for firmware, hardware or things we don't want to maintain downstream for some reason. It would be better if everyone collaborated on maintaining LTS kernels but instead it's largely 1 person and a couple others doing it with support from Google for testing, etc.
>So the super stable slow moving distro
Slow isn't referring to the speed Debian takes for hot fixes. It is referring to how long Debian stays hot fixing packages over updating to the latest version.
Let’s contrast that with a pure community led effort, motivated by freedom, privacy, and safety, that neither controls the OS nor the devices. It’s not what I tried to saltily explain above.
Respectfully, I hope that sunk in.
> Linux kernel LTS revisions are nothing like LTS revisions of most other projects. They're largely untested patches blindly applied by the LTS maintainers based on patches to mainline being marked for stable backports. If the patches apply cleanly, they ship. If they don't apply cleanly, they largely don't ship. Whether it works is an open question.
I'm also not super fond of frankenkernels. And... I'm confused how you feel about them? If backports suck, shouldn't you want to be chasing the very bleeding edge? I wasn't originally intending to argue that everything had to ride the very latest and greatest, but if backporting is inherently fragile and bug-prone, shouldn't you want to be on the very latest stable version (so as of this writing, 6.14.2)?
Quick, someone named strcat_s or strncat correct them!
Pixels moving 6th, 7th and 8th gen devices to 6.1 happened in March 2025. It's the first time they've moved to new LTS branches. It's likely going to move to using a newer LTS release where it would currently be using 6.6 and then moving to 6.12 after the next one comes out. We expect they move to having around a year to stabilize the new LTS and then use it for around a year before moving to the next. That fits nicely into the new 2 year lifespan for LTS kernels. This is a transition period. Once the longer than 2 year LTS kernels are gone, the quality of the LTS kernels will rise because there won't be as many to maintain. There are currently too many combined with too few people working on it. Greg KH having to handle both the kernel.org LTS and GKI LTS doing a huge portion of the work is clearly a problem. We'd also like to see the end of GKI ABI stability but that's highly unlikely. Yearly moves to new LTS kernels will at least make it a lot better.
> I would personally take [quickly checks kernel.org] 5.15.180 (latest 5.15) over 6.1.130 (not-latest 6.1)
Latest 5.15 has far fewer fixes backported for the same time periods than 6.1. The missing fixes in 5.15 are far more than a couple minor revisions. Similarly, 6.6 has more than 6.1 for the same time period and 6.12 has more than 6.6.
> And... I'm confused how you feel about them? If backports suck, shouldn't you want to be chasing the very bleeding edge? I wasn't originally intending to argue that everything had to ride the very latest and greatest, but if backporting is inherently fragile and bug-prone, shouldn't you want to be on the very latest stable version (so as of this writing, 6.14.2)?
Linux kernel code quality, review and testing is quite low. The bleeding edge kernels are nearly unusable in production for this use case (users running all kinds of software, using all kinds of different Bluetooth, USB, etc. accessories and so on while caring deeply about battery life) and have a ton of newly added security bugs which aren't found and fixed yet. We think that's a much bigger issue. We happily use Arch Linux for a lot of stuff but we use the LTS kernel package which is 6.12 at the moment.
If LTS quality was increased substantially, then we'd want to be on the latest LTS branch a while after the initial release, i.e. 6.12.15+ or so. However, at the moment, some serious regressions take them so long to find that it's still too new. We have high stability requirements where we can't have niche USB functionality, Bluetooth, video recording, etc. functionality regressing. The out-of-tree drivers are an area we don't have as much pain with since they're nearly all made for Exynos / Pixels and the drivers from the vendors so changes actually get tested well. The regressions are in the upstream code. More stuff coming from upstream would make LTS updates more, not less, painful, other than the GKI ABI stability nonsense we don't want.
They tried to get away with it on OpenBSD, macOS, etc and got their hand chewed off.
I wish it was feasible to run GrapheneOS on more devices. For some reason, Google seems to be incapable of selling Pixels world wide.
I wish they had an option that just used libc. Maybe, someday someone will add a target architecture/port that just uses posix. I’d prefer that in Linux too.
Android already published the full source code for Stable releases but barely anything for Beta releases. They didn't publish anything for upcoming releases with support for new devices. AOSP main branch received most changes through the Stable releases being merged into it. Most components were developed internally. Certain components were developed publicly in AOSP so they had to repeatedly merge back and forth between the internal and public main branches.
GrapheneOS would benefit from having early access to the monthly, quarterly and yearly releases as Android OEM partners do. The only benefit of having access to the main development branch would be the ability to backport more fixes than they do along with doing it earlier. We occasionally backported fixes from AOSP main for the few components developed publicly through it, which is what's going to be mostly going away. It was a big help to us as access to the upcoming quarterly and yearly releases would be. Monthly updates are too small for it to really matter but it'd still help.
Also worth noting the monthly security patch backports (Android Security Bulletins) are a separate thing from the new OS release each month. Those are backports of many of the High and Critical severity patches to older Android versions, often a month or two after they were released in the actual OS releases each month which are the monthly, quarterly and yearly releases (it's one of the 3 each month, with 3 quarterly releases and 1 yearly release per year although Android 16 is coming earlier this year instead of a 3rd quarterly release).
I don't want to use Android, I want to use Linux and Phosh or similar. But so far, the supported hardware is junk.
The Linux kernel is increasingly the elephant in the room when it comes to security and hasn't experienced anything like the massive progress made in Android's security in userspace. Piling on many more exploit mitigations to the Linux kernel won't really change this. We need to do a lot more work on it than we already do.
GrapheneOS has hardware virtualization support, which is going to be one of the ways to avoid depending so much on the Linux kernel's fragile security. The main usage for it in GrapheneOS will be running nested GrapheneOS instances for better sandboxing rather than running other operating systems. Android supports using the virtualization support to run other operating systems via the Terminal app and we have support for GUI applications, speaker, microphone and opt-in GPU acceleration backported to the Terminal app. The main use case for that app will be running desktop applications from other operating systems for the desktop mode. Windows 11 support would be a compelling addition to it and we may implement that in the next year or so.
Nice to know that all supported Pixel phones are not only on the same kernel version, but actually are build fro the same source tree now.
Do you also contribute your fixes back to the upstream projects like the upstream Linux kernel, AOSP or Google?
Many of the security features you are using are already included in AOSP, why does Google not activate them by default? Do they have a different balancing of performance, stability and compatibility on the one side and security on the other? I understand that Google has a different view on privacy for business reasons.
No interest in Android apps or Windows (hah). (Though maybe I'll try Waydroid one of these days.)
I don't know anything about your distro, not the graphics stack or what the package manager is or even if there is one? Meanwhile my starlite tablet is awesome because it works just like my desktop Fedora or Mint, though I installed Phosh on it.
Security is nice, but not before there is even a single feasible device on the market. Librem is just barely limping along, and I mean barely with a five year old handset that was obsolete when it debuted.
If your kernel is so advanced it really should be upstreamed, so these other distros could use it and support new Pixels. Y'all working together with other mobile projects would be so much better than the current surveillance dystopia we are currently living in. Maybe it's hard, but it is incredibly important. I can help though have limits.
https://github.com/lukasstraub2/intercept-anything
Go is hardly the only thing where LD_PRELOAD doesn't work.
We've made significant contributions to the Linux kernel, AOSP and Pixels in the past. We continue doing it to the extent that it helps GrapheneOS users. We no longer spend our time doing work for them if it doesn't have a clear benefit to our users.
Android's security team previously got us security partner access and was in the process of getting us OEM partner access. Android's partner management team blocked us from getting OEM partner access and revoked our security partner access. Due to this, we've reduced our reports of vulnerabilities upstream and have fixed numerous vulnerabilities in GrapheneOS without reporting them. We still report all firmware and hardware vulnerabilities but we make a decision about reporting software vulnerabilities solely based on what's best for GrapheneOS users.
> Many of the security features you are using are already included in AOSP
That's not really the case. The vast majority of our privacy and security features were developed for GrapheneOS. Features being built on top of standard functionality doesn't mean that they're present in AOSP.
For example, GrapheneOS has our own integration of hardware memory tagging into our hardened_malloc project and we turned the overall feature into something which can be used in production without making the OS unusable. We had to fix issues with the standard hardware memory tagging integration in the OS and Vanadium (Chromium) along with fixing numerous bugs uncovered by it. We had to integrate it into our user-facing crash reporting system and had to create a system for opting into using it for user installed apps where users can enable it for either specific user installed apps or for all user installed apps with a per-app opt-out. Android has the foundation for hardware memory tagging support but it's not used as a production feature for hardening and we're not simply enabling it. We have a much better userspace implementation in hardened_malloc and while we currently use the standard kernel allocator integration, we want to improve that to be closer to what we do with it in hardened_malloc. We currently use the standard Linux kernel MTE integration for the kernel allocators and the standard Chromium PartitionAlloc MTE integration but both need to be improved to provide better security properties as hardened_malloc does. They're also missing the other forms of hardening used by hardened_malloc which go nicely with memory tagging. The stock OS developer option for memory tagging is not the same thing and only makes it available for usage without actually using it. It has to then be enabled via ADB but there's no way to use it everywhere we do or in the same way we do. AOSP having that doesn't mean it provides what we do with it at all.
> Do they have a different balancing of performance, stability and compatibility on the one side and security on the other?
Yes, but you're wrong about where our privacy and security features come from.
Regarding LD_PRELOAD, I highly doubt that this is required by POSIX. macOS (which is a certified UNIX) uses DYLD_INSERT_LIBRARIES instead. OpenBSD (which is known for their pedantic documentation) uses LD_LIBRARY_PATH, doesn't mention any standards, and refers the reader to SunOS 4.0. If this is somehow standardised, I'd love to read the actual document.
We ship GKI LTS updates quickly but the GKI LTS branch can sometimes lag behind by up to a few weeks as it was doing when you looked. You happened to look at the maximum point it lags behind. It should really be kept closely in sync with the kernel.org releases but it's almost entirely Greg KH dealing with it and he deals with a lot of other stuff too.
Sounds like I should complain to them instead. Yet they are known for being unreachable.