[Replying to both emails]
My hope is that most of this stems from my cursory work in replying and a general misunderstanding.
I've seen people advocating for replacing the memory allocator in Tor Browser since I started the effort five years ago here: https://github.com/iSECPartners/publications/blob/master/reports/Tor%20Brows... My replies were primarily focused on replying to the advocations I've seen in the past, and the misconceptions I've seen repeated there (like LD_PRELOAD).
On Wed, 21 Aug 2019 at 12:14, Daniel Micay danielmicay@gmail.com wrote:
On Sat, Aug 17, 2019 at 09:17:40PM +0000, Tom Ritter wrote:
On Sat, 17 Aug 2019 at 15:06, procmem at riseup.net <procmem at riseup.net> wrote:
Question for the Tor Browser experts. Do you know if it is possible to remotely fingerprint the browser based on the memory allocator it is using? (via JS or content rendering)
Fingerprint what aspect of the browser/machine?
Performance-based fingerprinting of the browser can easily differentiate between using a different malloc implementation. That can already obtain a lot of fingerprinting information about the hardware and OS so this may not actually matter much, but it's entirely possible.
Agreed.
We are thinking of switching Tor Browser to use the minimalist and security oriented hardened_malloc written by Daniel Micay. Thanks.
I wouldn't advise giving up partitioning for.... what exactly? What features does this allocator have that 68's jemalloc doesn't?
The hardened_malloc allocator heavily uses partitioning, and has a much stronger implementation than the very weak approach in mozjemalloc. It statically reserves memory regions for metadata and a dedicated region for each arena, with each size class receiving a dedicated sub-region within the arena. These sub-regions are placed within their own guard region and each have a high entropy random base. It never mixes address space between these regions or reuses the address space. This is much different than what you call 'partitioning' in mozjemalloc which does not really qualify. What you're talking about is mozjemalloc exposing an API for choosing the arena from the code, which can certainly be done with hardened_malloc too. However, in mozjemalloc, the address space for different arenas is mixed together and reused between them. It's really a stretch to call this partitioning, and it doesn't have the baseline separation of size classes either.
People can read about hardened_malloc in the README:
https://github.com/GrapheneOS/hardened_malloc/blob/master/README.md#hardened...
I don't know why you're making the misleading claim that people would need to give up partitioning.
My reply about giving up partitioning here (which I clarify in the second email) is that using LD_PRELOAD will negate partitioning.
It's also really a stretch to call what Mozilla is doing in mozjemalloc partitioning in the first place, so your claim is really quite backwards...
Let me start by asserting emphatically and agreeing that hardened_malloc is a much stronger allocator than mozjemalloc or jemalloc. PartitionAlloc is also for that matter.
Mozilla's partitioning has always been focused on preventing the Use-After-Free problems that have plagued Firefox for years. Allocate a DOM object, retain a pointer to it, free it, replace it with an Arraybuffer, align the vtable entry, redirect execution. We allocate ArrayBuffer contents and strings in separate arenas to avoid the immediate reuse of these bytes.
On Wed, 21 Aug 2019 at 13:40, Daniel Micay danielmicay@gmail.com wrote:
On Mon, Aug 19, 2019 at 04:09:36PM +0000, Tom Ritter wrote:
Okay I'm going to try and clear up a lot of misconceptions and stuff here. I don't own Firefox's memory allocator but I have worked in it, recently, and am one of the people who are working on hardening it.
This makes it clear why you're spreading misinformation. You're going out of your way to make false and misleading claims about mozjemalloc and hardened_malloc, particularly your bogus comparisons between them.
My comparisons were rushed and cursory. I apologize and I'll clarify my conclusions at the end of the email.
Bolting on a few weak implementations of hardening features to an allocator inherently very friendly to memory corruption exploitation does not make it anything close to being hardened allocator, sorry.
Fair.
Firefox's memory allocator is not jemalloc. It's probably better referred to as mozjemalloc. We forked jemalloc and have been improving it (at least from our perspective.) Any analysis of or comparison to jemalloc is - at this point - outdated and should be redone from scratch against mozjemalloc on mozilla-central.
It's not particularly different and the comparison isn't outdated.
As above, the comparisons I were referring to were the five year document I wrote, and past trac tickets and threads here.
LD_PRELOAD='/path/to/libhardened_malloc.so' /path/to/program will do nothing or approximately nothing. mozjemalloc uses mmap and low level allocation tools to create chunks of memory to be used by its internal memory allocator. To successfully replace Firefox memory allocator you should either use LD_PRELOAD _with_ a --disable-jemalloc build OR Firefox's replace_malloc functionality: https://searchfox.org/mozilla-central/source/memory/build/replace_malloc.h
LD_PRELOAD is not how hardened_malloc is supposed to be used outside of testing it anyway. It's meant to be integrated in libc, in which case --disable-jemalloc would be enough, although it can also be integrated into a specific program. That doesn't sidestep the importance of doing other hardening in libc and the rest of the system though.
We agree on this.
Fingerprinting: It is most likely possible to be creative enough to fingerprint what memory allocator is used. If we were to choose from different allocators at runtime, I don't think that fingerprinting is the worst thing open to us - it seems likely that any attacker who does such a attack could also fingerprinting your CPU speed, RAM, and your ASLR base addresses which depending on OS might not change until reboot.
They can obtain a lot more than just information about the hardware. A lot of the hardware and OS information that fingerprinting mitigations try to hide are leaked via performance measurements. It can also leak a lot of data from within the browser.
Also agreed. I wasn't mentioning things like the broad class of pixel-stealing attacks, or the user activity-class of attacks. Which is also the reason I advocate for getting Fuzzyfox into a usable state.
The only reason I can think of to choose between allocators at runtime is to introduce randomness into the allocation strategy. An attacker relying on a blind overwrite may not be able to position their overwrite reliably AND it has the cause the process to crash otherwise they can just try again.
The hardened_malloc design provides far more than randomization
Of course. But the randomization I'm referring to isn't the randomization inside the allocator, it's the random choice of *which* allocator to use.
Allocators can introduce randomness themselves, you don't need to choose between allocators to do that.
This is not something that can simply be bolted onto an existing allocator design with a good approach. It needs to be more heavily integrated into the design, and the same applies to an even greater extent to more important security features than weak fine-grained randomization.
I agree.
In virtually all browser exploits we have seen recently the attacker creates exploitation primitives that allow partial memory read/write and then full memory read/write. Randomness introduced is bypassed and ineffective. I've seen a general trend away from randomness for this purpose. The exception is when the attacker is heavily constrained - like exploiting over IPC or in a network protocol. Not when the attacker has a full Javascript execution environment available to them.
When exploiting a memory corruption vulnerability, you can target the application's memory (meaning, target a DOM object or an ArrayBuffer) or you can target the memory allocator's metadata. While allocator metadata corruption was popular in the past, I haven't seen it used recently.
The importance of out-of-line metadata is far beyond simply preventing exploitation through the allocator metadata. It's crucial for a hardened allocator to have a reliable source of information about allocations without trusting data read from freed allocations or next to the memory allocations.
My understanding of this statement is that the metadata must be protected against non-security-related corruption; if I'm misunderstanding I'm happy to be educated.
Okay all that out of the way, let's talk about allocators.
I skimmed https://github.com/GrapheneOS/hardened_malloc and it looks like it has:
I can see how my skimming, and simplistic bullet points, could be insulting to someone who has invested such a great deal of work into their project. I'm sorry; it's too easy to forget that on the other side of text is a person who deserves our consideration, and that on the other side of code is a significant investment of effort.
- support for arenas
This is a completely dishonest and ridiculous misrepresentation of the design and security properties laid out in that README. People should read it for themselves and they'll see that your attempt at spinning misinformation about it is a complete joke.
I'd intended this as a simplified response to the expected points I'd seen brought up in the past. I apologize if it came across as intentionally dishonest.
mozjemalloc:
- arenas (we call them partitions)
Unlike hardened_malloc, they're mixed together and address space is reused between them rather than having strong isolation. The approach in hardened_malloc also partitions each size class. Calling the mozjemalloc arenas partitions as if it's an implementation of a security feature is a joke.
As I mentioned before, our implementation of partitions is focused on preventing the immediate reuse of bytes to negate the common UAF pattern we have seen for years. hardened_malloc's is clearly stronger.
- randomization (support for, not enabled by default due to limited
utility, but improvements coming)
- double free protection
- zero-filling
In Progress:
- we're actively working on guard regions
As covered above, you're being misleading with each of these points, by portraying these things as something black and white that the allocator either has or doesn't have
That's a fair criticism.
In particular, talking about randomization and guard regions as if this is a matter of having them or not having them is ridiculous. There is more to invalid free detection than double free detection and how well it works has a lot of variation. It can be deterministic detection like the hardened_malloc implementation, or probabilistic detection that an attacker could much more easily bypass. The reuse of freed allocations also matters a lot, since once it's handed out again, a free based on a past generation allocation won't be considered invalid, despite it being wrong and dangerous. This is why the design of memory allocation reuse and quarantines matters so much. The documentation on thread caching in the hardened_malloc README elaborates on why that's not compatible with a hardened allocator due to interfering with doing anything like this properly.
Also a fair assessment.
Future Work:
- out of line metadata
There's a huge variation in what this means. The hardened_malloc metadata is fully out-of-line in a dedicated region, with that address space never mixed / reused with anything else. The same applies to all the size class regions within arenas.
That's very strong protection - I hope we can integrate the same level of security in the future.
- MPK
For what exactly?
Still TBD; but our initial thoughts are using it in the JIT Engine and to effect XOM for certain use cases.
But the benefit gained by slapping in an LD_PRELOAD and calling it a day is small to zero. Probably negative because you'll not utilize partitions by default. You'd need a particurally constrained vulnerability to actually prevent exploitation - it's more likely you'll just cost the attacker another 2-8 hours of work.
The claim that mozjemalloc has partitioning and hardened_malloc does not couldn't be further from the truth.
A misunderstanding: I didn't claim that; I claimed that using LD_PRELOAD would not make use of partitions.
I find it ridiculous how you attempt to attack the project with these lies to promote your own work, which is hardly comparable at all. It's not the same thing. Bolting on a few security features to an allocator design that's exploitation friendly from the ground up doesn't make it a hardened allocator. That's even more true when the implementations of those features are unnecessarily weak.
My intention was not to promote my own work; or negate yours. I'm sorry it came across that way. As I said at the top of this email, hardened_malloc is clearly a much stronger allocator that mozjemalloc.
I'll try to clear up my intention with my email here:
Replacing the allocator used by Tor Browser naively (LD_PRELOAD) would be a net loss in security. I think (?) you'd agree. Replacing the allocator used by Tor Browser correctly would grant a hardened allocator but would not significantly affect how the past several exploits written against Firefox would be written, and would not be a significant impediment to exploit authors who are using the type of vulnerabilities we have seen exploited of late.
It would certainly affect the exploitation of more limited vulnerabilities (fixed-byte over-reads/writes).
Out of line metadata is on-the-surface-attractive but... that tends to only help when you have a off-by-one/four write and you corrupt metadata state because it's the only thing you *can* do. With out of line metadata, you can just corrupt a real object and effect a different type of corruption. I'm pretty skeptical of the benefit at this point, although I could be convinced. We don't see metadata corruption attacks anymore - but I'm not sure if it's because we find better exploit primitives or better vulnerabilities.
Out-of-line metadata is not simply about preventing attacks on the metadata itself. It provides much more than that. I'm not sure why you think your ignorant opinions on these topics matter, when you so clearly don't know what you're talking about at all.
Unfair! :)
In particular, if you wanted to pursue hardened_malloc you would need to use replace_malloc and wire up the partitions correctly. Randomization will almost certainly not help (and will hurt performance)*.
The hardened_malloc design is focused on reliable, deterministic memory corruption mitigations. Randomization is used where possible, and in a way that has a low impact on performance. The high entropy base randomization for each size class region within arenas has no significant impact on performance. The randomization for large allocation (> 128k) guard regions and that quarantine has no substantial impact on performance. The impact from slot randomization and the slab allocation quarantine is measurable but not high, and the slab quarantine has no substantial impact.
My statement on performance was based on my experiments enabling our limited randomization in the Content Process (numbers lost, but they came from https://bugzilla.mozilla.org/show_bug.cgi?id=1376408#c18). It really hurt DOM node traversal. I'll concede it's possible that hardened_malloc may have less any performance cost here - I haven't tested it. But... I'm skeptical.
(You seem to agree that randomization is not a strong security feature?)
MPK sounds nice but you have to use it correctly (which requires application code changes), you have to ensure there are no MPK gadgets, and oh wait no one can use it because it's only available in Linux on server CPUs. =(
Using MPK is not one of the major features of hardened_malloc and the usage doesn't rely on not having MPK gadgets. This is explicitly documented in the README. It's the only optional security feature that's not enabled by default. It should be pointed out that most of security offered by hardened_malloc is not a feature that can be turned on or off because it's the design itself that's hardened, not the fact that it has some optional security features bolted onto it.
Fair! (Although I'm confused about your comment saying you can provide protection without eliminating MPK gadgets.)
In conclusion, while it's possible hardened_malloc could provide some small security increase over mozjemalloc, the gap is much smaller than it was when I advocated for allocator improvements 5 years ago, the effort is definitely non-trivial, and the gap is closing.
No, you're just making false attacks and misleading comparisons / spin to promote your own work, which is trash.
=(
You're being incredibly dishonest and unethical.
I'd hope that the principal of charity would lead to the conclusion that instead I was being simplistic and ignorant.
You didn't even bother to inform yourself about hardened_malloc by actually reading through the documentation. Instead, you just jump to conclusions and present yourself as an expert on topics you are clearly incredibly ignorant about. You really don't know what you're talking about, and your post on this mailing list is offensive. Your post as a whole is nonsense, and your conclusion is bogus.
Which conclusion? The one I didn't make (but certainly implied) that mozjemalloc is comparable security-wise to hardened_malloc? I agree, such a conclusion is bogus.
Or the one that replacing the allocator will not have a significant effect on the exploits written for the past couple years of Firefox expoits?
This seems to be the central misunderstanding, and why I agree that you are entitled to take offense.
[ending statements]
I don't have any knowledge of, or context about, your past interactions with Mozilla. We all wear many hats; but this discussion isn't about Mozilla, Firefox, or replacing Firefox's memory allocator - it's about Tor Browser. If you'd like to escalate your concerns, they can be forwarded to Tor's Community Council; that information is available at https://trac.torproject.org/projects/tor/wiki/org/CommunityCouncil
-tom