Date: Thu, 11 May 2017 03:24:35 +0200 From: "PaX Team" <pageexec@...email.hu> To: Mathias Krause <minipli@...glemail.com>, Kees Cook <keescook@...omium.org> CC: Daniel Cegielka <daniel.cegielka@...il.com>, "kernel-hardening@...ts.openwall.com" <kernel-hardening@...ts.openwall.com> Subject: Re: It looks like there will be no more public versions of PaX and Grsec. i think i'm not the only one to notice that the past two weeks have been an interesting experience to say the least. ever since our announcement about the fate of our test patch series we've been silent but not because we had nothing to say (as you've no doubt guessed by now, this mail is it). rather, we've used the opportunity the announcement created to see just what comes out of the woodwork. you see, we've always wondered about this kernel security game, who wants what, who's doing what, who's p(l)aying whom, etc. in other words, we wondered if the participants of the game would show their true colors in the situation. and show they did, it was well worth the wait. as far as i could piece together the narrative propagated by Kees Cook of Google, Greg Kroah-Hartman of the Linux Foundation, LWN's Jonathan Corbet, the Linux Foundation's CII, and other less known players, the story is that said entities have found (some of them after years of ignoring the problem) that their security needs could be best solved by upstreaming our code but all their attempts to work out a deal with us for said purpose have failed and thus we are to blame for everything under the sun and then some. and if that weren't enough, they then went on the offensive with lots of self-congratulating bravado and outright attacks on us and our projects. one can't help but notice that behind all the garbage spewed at us, someone was getting very worried about the fate of the KSPP fork of our work. what i can tell them now is that they will have a whole lot more to worry about once they finished reading our response. before we'll get to particular claims, we'll first look at some history to set the stage for some of our answers. my side started in 2000 as a hobby project that i worked on entirely in my free time. i used to jokingly refer to it as a 'weekend project' as that's literally when i had time for it due to my day jobs. later years didn't change much except for the periods when i was without a job (amounting to over a decade) and thus had more free time on my hand. Brad started his project a bit later than me and has always worked on it in his free time. as a true community project (KSPP's is very far from it, see below) we have received lots of generous help over the years in the form of hardware and financial donations that however didn't quite cover our costs so at some point we were forced to run the project in a more formal manner to be able to accept corporate sponsorship (this is what OSS was originally created for). at some point we also decided to begin supporting selected kernel versions for a longer period of time in addition to the latest upstream kernel. this proved to be a nice bonus and even incentive for corporate sponsors. all was well until a few years ago when we learned of several blatant copyright violations (not of just our code but by extension that of the upstream kernel too) and also trademark violations. pursuing these wasn't too successful and in order to prevent further abuse we decided to stop the public distribution of our stable patches and provide it as an entirely commercial product from then on (all still under the GPL, despite much speculation to the contrary). this proved to be a better decision and all seemed well for a few months until the Washington Post article on kernel security appeared. this turned out to be a game changer event in that linux security got a serious look at from high level corporate executives and resulted in Google tasking Kees Cook to start to work on the KSPP with the express purpose of upstreaming grsecurity - all without us. at this point we'd like to make it clear that any statements from Kees/Greg/etc to the contrary are simply *lies*, neither i nor Brad have ever been contacted by any of the corporations paying for the KSPP. since proving a negative is hard, instead hereby we give any corporate attorney, executive (or any other officer who thinks that they gave us an offer that we refused) permission to post any private emails including financial details, contract terms, etc that prove us wrong. as a sidenote, there's some irony in that Kees and Greg managed to tell contradicting lies as according to one we refused to get paid to upstream our code whereas the other said that the Linux Foundation had funded 'members of the grsecurity team'. in reality, neither happened as such offers that we could refuse or accept have never been made. it's also telling that when upstreaming our code came up in private conversations where Kees was present, his best suggestion at the time (around 2015 summer) was to try the freshly expanded CII instead of, say, his own employer Google (last month after he got an early notification about our plans regarding the test patches, he in fact admitted that Google was never going to pay us for the upstreaming work). the CII is also an interesting animal in that they can't seem to figure out what exactly it is they want. during the summer of 2015 i was CC'd into an email thread on cii-discuss where some of our users suggested that the CII take up funding our projects. at the time the CII stated that they would not provide funds that'd amount to effective employment and also that they'd be only interested in upstreaming our work. when i asked if they were then willing to fund thousands of hours of such work, i got no response and left it at that. now imagine my surprise when i saw their recently published annual report for 2016 which has this following beauty in there: One challenge we’ve encountered this year is finding skilled people to take on the work. While the desire to work on open source exists, without compensation it’s simply not feasible for many developers to do so. Emese Renfy, for example, had to step back from Kernel Self Protection Work because funding from CII took so long to approve. We’ve worked to resolve this with our new online grant system. Another solution is to find those who are already working on open source as a hobby and allow them to continue their work at a fully-funded, professional level. For example, Chris Lamb and Ximin Luo, have been able to give up their day job to focus exclusively on the Reproducible Builds project. so it now looks like that funds amounting to effective employment are possible, too bad they never told me about it (this is of course in direct contradiction to what they claimed in their blog and i again give them permission to post any emails/etc that prove the contrary). on a somewhat amusing sidenote, i can't help but notice that someone at the CII must have taken lessons from the KSPP in their copy-pasting skills as they managed to get Emese's last name wrong not once but twice then repeated the same in their blog for good measure. so what do we have so far about this KSPP business? Google and other companies made a decision to get our work upstream using their own resources only - so far so good, it's their business in every sense of the word. not involving us however turned out to be an unwise decision when they realized that their employees are completely unprepared for this work (we'll discuss some examples later). this then led to all kinds of social engineering attempts at trying to get us to help them out - all without getting paid for our time, as if we were just supposed to drop all other work and spend our free time on fixing their botched up attempts. as if this still wasn't disgusting and outragous enough, they went on the combative and had the guts to accuse us of not wanting to cooperate without also mentioning that they expected all this work from us for free taken away from our free time. at this point it should be clear that the KSPP is anything but a community project. rather, it's a joint effort of commercial companies such as Google/Intel/etc to spend their corporate resources on ripping our code and abusing our hard-earned reputation *and* have the audacity to expect us to assist in all this using our free time. this state of affairs made us realize that if this is what this 'community' wants, then they shall get it. now they're on their own and by looking at all the reactive mud slinging ever since, they're terrified about the fate of their fork. based on the level of incompetence they showed so far, it's not hard to see why. this brings us to second part of our response where we'll take a look at some of the claims Kees made and show them for what they really are. > It does underscore the critical need to upstream stuff, though. > Forks of projects might disappear at any time. :( this is somewhat rich coming from a Google employee considering the fate of various linux forks used in their devices. one may also wonder what will happen to the KSPP fork, especially now that Fuchsia is also set to obsolete all that work and remove the incentive to fund it much longer. upstreaming stuff also isn't at all enough to prevent code from disappearing, one needs to look no further than the recent removal of avr32 support due to bitrot. considering the force that sustains the KSPP (corporate money and interest) one can guess how long said code will survive bitrot vs. our track record that is the result of a fundamentally different kind of motivation. as another counterexample to why that 'critical need' can be overrated, one can also look to the unprivileged ping code (somewhat ironically, it's a security related feature, not unlike the topic here), introduced on this very same list years ago by Vasiliy Kulikov via a Google Summer of Code project. it was later maintained by upstream developers who then introduced several exploitable vulnerabilities in it. > Additionally, while PaX Team, grsecurity, and ephox's work > were technically separate development efforts (for example, just look > at how PAX_USERCOPY differed between PaX and grsecurity), FYI, USERCOPY wasn't separate efforts at all, we worked together on USERCOPY and decided to carry certain features in grsec only. this is a typical workflow for us, there've always been things that started out in PaX and were made into a complete feature in grsec or where i took back some code into PaX that i considered needed infrastructure/features in there. > Looking at the results in grsecurity, though, it's clear that they > chose to integrate with upstream instead of maintaining a forked > implementation. i think you got that backwards, our code is the original, the KSPP (and consequently upstream's) copy is just that, a fork (look at where you get the gcc plugin updates from among others). that said, why would we keep two similar copies of our own code around? as for 'integration', what i actually did was to clean up the mess you created, just look at the bits i 'awarded' with CONFIG_BROKEN_SECURITY and some more examples below. > This is further supported by grsecurity being paid by CII to upstream > the gcc plugin infrastructure, correction, grsecurity, which according to your own definition is a kernel patch, can't have been and thus never was paid by the CII, Emese was (and she isn't a grsecurity developer any more than i am an upstream linux developer). > I think it's an entirely false claim that upstream is creating > more work that normal forward porting. and i think you're wrong on that (see below). it's in fact yet another example to show your lack of understanding of what some of our features do. > Finally, even if you can somehow disregard the thousands of upstream > changes benefiting grsecurity[...] what exactly benefits *grsecurity* (you know, what you defined as a kernel patch) and not kernel users in general (which then means that said changes have nothing to do with us)? i can't think of much if anything in recent memory. > they have benefited from the upstreaming review by upstream > finding bugs in various grsecurity features oh yes, this myth that just can't die. we'll see in about a second what exactly those 'bugs' are. > Arnd Bergmann alone found tons of issues with the initify plugin > and grsecurity fixed them) actually that was something like 4 bugs in the plugin and we didn't fix them, Emese did. > Hardened usercopy found bugs in grsecurity's slab implementation, now we're getting somewhere, let's dive in. the first 'bug' was that *4.6* introduced red_left_pad to SLUB objects which needs to be taken into account when computing object boundaries and my code didn't. what you forgot to tell the world at the time (and ever since) is that it was you who took my code from *4.5* which didn't have red_left_pad and thus was perfectly fine as is. the actual bug was that you blindly copy-pasted my code (a recurring theme) without doing the one job you had: maintain this code and follow upstream evolution. the second 'bug' was the use of ksize vs. ->object_size which, if memory serves, a slab maintainer insisted on using. what this 'bugfix' achieved in one fell swoop is that all SLUB objects with padding can now be used to leak that padding which pretty much destroys the purpose of (your copy of) USERCOPY. just think about it, by definition that padding is uninitialized memory and thus a prime target for memory leak attempts that thanks to your 'bugfix' can now occur unimpeded. as an aspiring maintainer your proper response should have been to explain this to the slab maintainers and also why ksize is an abomination itself (its users by definition exercise UB) and that the problem it sets out to solve should be an API redesign instead (remove ksize and create a new allocation function that takes the size as reference and updates it on return so that the caller can learn how much memory it ended up getting). so that's two bugs so far, except none of them is ours but yours instead but nice try selling them otherwise. unfortunately the buck doesn's stop just there yet, there're more problems with this expert reviewed and maintained code (i wonder how on earth we managed to survive this far without such expertise). the third bug is that the comment describing __check_heap_object is meaningless garbage as already allocated objects by definition cannot be 'incorrectly sized' (at least not at this level, SIZE_OVERFLOW would be something closer to that purpose) and of course the purpose of USERCOPY has nothing to do with verifying object sizes anyway. what USERCOPY does want to verify is the size of the memory *copy* attempt (that is, the copy must fully fall inside the kernel object). this brings us to the fourth bug which is the verification of the kernel object pointer for being 'impossible'. that of course was never the purpose of USERCOPY as its design assumes a threat model where an attacker controls only the size of the copy but not the object pointer. if the latter is also assumed, all bets are off, there's no way such checks can catch an actual attack (checking pointer provenance is a very hard problem). there're further problems caused by your 'improvements' on our code. one is the needless change of the log message that breaks existing log analyzers looking for a pattern. the other problem is that the replacement of do_group_exit with a BUG is entirely unnecessary as a USERCOPY event is recoverable, there's no need to panic the whole system because of it (a bit like how a REFCOUNT event is recoverable to an extent). i also wonder what locks would be broken by BUG when copy*user functions can sleep and thus are not supposed to be called under locks anyway. (and if you didn't mean spinlocks then how would BUG handling break them?) last but not least, a quick look at your claim to fame due to: > [USERCOPY] gained strncpy_from_user() coverage [...] this is an entirely pointless exercise as an audit of the few callers of said function can show their correctness instead. so your 'coverage' adds nothing but pointless performance impact to it. FWIW, similar considerations made us not check copying with compile-time constant sizes at the time. i believe this brief overview at your achievements regarding USERCOPY shows that instead of fixing any bugs in our code, you in fact added a bunch of them. > The current refcount_t work based on PAX_REFCOUNT uncovered a > crash bug in grsecurity's implementation for the record, it was a bug introduced in the PaX 4.9 port only and you failed to spot it when copying it to a newer kernel (you also failed to notice what the underlying problem i wanted to fix was and thus didn't fix it yourself either at the time). note also that the reporting problem i tried to fix there (and fixed it differently in later patches) is still not properly addressed in your refcount port but at least you documented it now. > Upstreaming grsecurity features brings a huge amount of testing > to bear on the code, which, like all the other things in upstream, > grsecurity directly benefits from. not all things upstreamed benefit us (or other users for that matter). just look at VMAP_STACK (upstream NIH'd version of Brad's KSTACKOVERFLOW feature) and the entirely avoidable damage (and security bugs) it caused to end users ever since its release. > And to top all of this off, while upstreaming the latent_entropy > plugin, I noticed a English typo that was present in all the > grsecurity gcc plugins, and when I sent them a trivial spelling > fix patch for all their plugins[...] if only it had been all plugins... clearly you never tried a recursive grep and thus missed the same typo in RAP and SIZE_OVERFLOW. > [...] (rather than just letting it stand and making work for > them to catch it during forward porting and fix it everywhere > else), they publicly mocked the patch (which they applied). given the above, it'd have been better to just send an email about the typo and let us fix it ourselves properly which we ended up doing anyway instead of using your incomplete patch. > Upstreaming can be extremely time consuming. Grsecurity made it clear > from long ago that they had no intention of upstreaming things because > they wanted to use their time differently. not true at all. what we've always said was that upstreaming our code requires so much time that noone can reasonably expect us to do it in our free time. that the KSPP exists entirely by corporate decree is proof of that. what would actually limit any potential upstreaming effort on our side is the need to balance our time spent on upstreaming vs. time spent on new R&D. come to think of it, if the companies running the KSPP had played it smart, they'd have gone out of their way to engage us as that's the only way they'd have a chance to close the ever widening feature gap. > That said, when I asked grsecurity if there I was any way they would > accept payment to upstream things, they did briefly agree and > upstreamed the gcc plugin infrastructure. So they were willing to > upstream if paid, yet ultimately decided to stop, and to continue to > forward port their patches. i don't know who you asked as 'grsecurity' but the only thing i recall was some private conversations back in 2015 where you suggested to try to get the CII to fund an upstreaming project (where you yourself had no saying) and that turned out to be a downer. you also knew full well that Brad wouldn't have anything to do with the CII due to one of their member companies (that entirely coincidentally is also behind the KSPP) having violated his trademark. one can't also help but wonder why you didn't offer Google's money? because you have never been authorized by Google to act on their behalf (and thus can't have possibly made any such offers)? so i'm not sure what you think you did to get us 'accept payment to upstream things' but you've clearly never been in a position to make such an offer (and no, poorly disguised social engineering attempts don't count). by the way, it wasn't Brad who upstreamed the gcc plugins stuff but Emese (she's neither a PaX nor a grsecurity developer). > [...] I again summarily reject the notion that upstreaming > grsecurity features creates "more work for grsecurity without > value in return". > [...] > To your specific examples, __ro_after_init is literally a one-line > change: they just make in __read_only. This was some of my first > attempts to make their forward porting work easy while upstream slowly > incorporated features. this is false and only shows that you don't at all understand what __read_only really does. in particular, __read_only is enforced much earlier in modules (so much for it being 'after init') and thus all writes to such objects must be instrumented. this means that all new __ro_after_init uses must be carefully audited and any writes to them instrumented. this is anything but 'literally a one-line change' (though i can at least get the compiler to do the work for me to an extent). > The work around PAX_MEMORY_SANITIZE made the slab debug paths > faster for everyone. i wonder, how many systems actually enable those slab debug features at runtime due to their performance impact. > So, neither of us can speak for grsecurity, but I reject your belief > that upstream has somehow created needless work for them. fortunately it's got nothing to do with belief but simple facts that happen to contradict your belief. > Which brings me to a question I haven't seen anyone ask yet: why does > grsecurity exist? thanks for your concern for our existence but i think you should ask why the KSPP exists instead and how much longer it will be around. based on your statements here and in later mails it seems that you're very worried that your ability to copy-paste yet more of our code will diminish over time. unfortunately for you, putting up a fake happy face and pretending nothing happened won't actually make it any better for the KSPP. back to your question, you don't have to guess and can instead just look at what world class domain experts think of our work in their testimonials on the grsecurity website. it seems to me that you and the KSPP are stuck in the world of linux whereas our work has affected much more than that already. so in a sense you're right to say that we don't care about all linux users since we actually care about everyone else too and use our projects (the code, conference presentations, blogs, etc) to showcase defense technologies that everyone can benefit from, not just linux users. when will the KSPP reach this level, if it's on the corporate agenda all, that is? and if all this counts for nothing for you then consider our existence as the apparent enabler of your career. > You're speaking entirely in theoreticals, but I understand what you're > trying to say. All forward porting runs this risk. Kernel internals > change in ways that threaten even unchanged grsecurity features. To > bring this down to earth, I would ask "how does grsecurity perform > testing?" I can point to the many ways how upstream performs testing, > including LKDTM for several of the security-sensitive features. As > seen in the PAX_REFCOUNT porting work, the 4.9 grsecurity patch was > clearly never actually tested since _any_ exercise of the PAX_REFCOUNT > protection would Oops the kernel. you're wrong, i actually tested the code just not a last-minute one-liner change that was 'obviously correct' (well, it wasn't, it happens). on the other hand it's pretty funny that you pointed out this one bug as it was you who blindly copy-pasted it *and* never tested it yourself otherwise you would have found it, not Jann Horn. another funny thing is that you clearly never understood the REFCOUNT reporting code as LKDTM didn't even exercise those paths at the time. last but not least, the actual effect of the bug wasn't necessarily an oops, it all depended on how the offsets were encoded in the conditional jumps (as i explained all this in my response to Jann and you). > [...] but I can point you directly to how upstream tests, how upstream > developers find bugs in grsecurity features, and how things can improve once > upstreamed (again, for example, the massive uaccess consolidation). yep, things improve indeed, like the new vulnerabilities caused by ping sockets, VMAP_STACK, HARDENED_USERCOPY, SLAB_FREELIST_RANDOM, HARDENED_REFCOUNT, etc... > I think I have thoroughly disproved this position. Something that I > think is hard to see for people not involved in day-to-day upstream > work is how very different the development workflows are between > upstream and grsecurity. Upstream is normally evolutionary, doing > things in easy to digest pieces, where as grsecurity could just land > massive changes between releases. believe it or not but we also develop our features step-by-step, not unlike how any sane developer, upstream or not, does it. > These are all complaints to be made to grsecurity. They are the ones > who provides those features originally, did not upstream them, and > then took them away. i'm wondering, when are you going to stop the hypocrisy and this rhetoric of blaming us for not upstreaming our own code in our free time while the KSPP companies have been happily paying you and others for doing the same? > > So, here's my list of questions for the KSPP: > > 1/ When will I be able to switch to a vanilla Linux kernel that is > > equivalently hardened as a grsecurity/PaX kernel used to be? > > How could anyone answer that question? I can't see the future. well, it seems that not that long ago you did see the future as: grsec is going to stay a decade ahead of anyone else (not just Linux) > And besides, it's not like grsecurity was providing comprehensive > protections. If you're using arm64, you might feel like you're already > in a better position with upstream (got PAN emu, got hardened > usercopy, still no RAP). actually, by definition you can't be in a better position with upstream unless you're claiming that we actually reduce security with our code (please, do bring up my disabling of KASLR, happy to discuss it again). in fact due to the bugfixes we have for your USERCOPY fork (that is, we kept our own working code), our code *is* actually in a better position already. as for RAP, it works on any arch by design, i just don't enable it on archs where i haven't fixed all the function pointer abuse yet and marked up asm code (both arm and arm64 work already here). > Upstream's goal is protecting as many people as possible. the KSPP's goal is to further the agenda of the companies behind it (which is extracting profits for shareholders). that has nothing to do with "protecting as many people as possible" but everything to do with business goals du jour. if what you claim was true, they would have done it since the beginning and in a way that is not restricted to only linux users. > > 2/ Who will maintain this code and how? > > 3/ Who ensures the coverage and quality won't suffer for each new > > kernel release? > > The upstream development community, just like everything else. I > answered both of these already above. the correct answer is 'whoever the KSPP companies put on the job' though their performance so far doesn't leave one with great confidence as to the sustainability of the project. cheers till the next round, PaX Team

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.