On Sat, Aug 17, 2019 at 09:17:40PM +0000, Tom Ritter wrote:

On Sat, 17 Aug 2019 at 15:06, procmem at riseup.net <procmem at riseup.net> wrote:

...

Question for the Tor Browser experts. Do you know if it is possible to remotely fingerprint the browser based on the memory allocator it is using? (via JS or content rendering)

Fingerprint what aspect of the browser/machine?

Performance-based fingerprinting of the browser can easily differentiate between using a different malloc implementation. That can already obtain a lot of fingerprinting information about the hardware and OS so this may not actually matter much, but it's entirely possible. The strength of fingerprinting mitigations when JavaScript is enabled are extremely overestimated / overvalued and in many cases don't actually hide the information that they attempt to hide due to secondary ways of obtaining it like careful performance measurements.

...

We are thinking of switching Tor Browser to use the minimalist and security oriented hardened_malloc written by Daniel Micay. Thanks.

I wouldn't advise giving up partitioning for.... what exactly? What features does this allocator have that 68's jemalloc doesn't?

The hardened_malloc allocator heavily uses partitioning, and has a much stronger implementation than the very weak approach in mozjemalloc. It statically reserves memory regions for metadata and a dedicated region for each arena, with each size class receiving a dedicated sub-region within the arena. These sub-regions are placed within their own guard region and each have a high entropy random base. It never mixes address space between these regions or reuses the address space. This is much different than what you call 'partitioning' in mozjemalloc which does not really qualify. What you're talking about is mozjemalloc exposing an API for choosing the arena from the code, which can certainly be done with hardened_malloc too. However, in mozjemalloc, the address space for different arenas is mixed together and reused between them. It's really a stretch to call this partitioning, and it doesn't have the baseline separation of size classes either.

People can read about hardened_malloc in the README:

https://github.com/GrapheneOS/hardened_malloc/blob/master/README.md#hardened...

I don't know why you're making the misleading claim that people would need to give up partitioning. It's also really a stretch to call what Mozilla is doing in mozjemalloc partitioning in the first place, so your claim is really quite backwards...

Hi,

On 18 Aug 2019, at 08:35, Shawn Webb shawn.webb@hardenedbsd.org wrote:

Having the heap implementation selectable at runtime would enable users to make the determination for themselves, while also making future integration efforts easier through modularization/abstraction APIs (I'm making a silly, naive, and likely wrong, assumption that such APIs don't already exist.)

I hope I'm not coming off as "hey, do this." I'm just thinking out loud in an admittedly naive fashion.

Anyone have any thoughts?

Allowing users to select the allocator at runtime would split the anonymity set, *if* the allocator was detectable remotely.

This is why we try to use safe defaults, and avoid user-selected options. When we do give users options, we create a few well-defined settings (like the security slider).

T

Hey Tom,

Thank you for your response. You've made some great points. My response is inline.

On Mon, Aug 19, 2019 at 04:09:36PM +0000, Tom Ritter wrote:

Okay I'm going to try and clear up a lot of misconceptions and stuff here. I don't own Firefox's memory allocator but I have worked in it, recently, and am one of the people who are working on hardening it.

Firefox's memory allocator is not jemalloc. It's probably better referred to as mozjemalloc. We forked jemalloc and have been improving it (at least from our perspective.) Any analysis of or comparison to jemalloc is - at this point - outdated and should be redone from scratch against mozjemalloc on mozilla-central.

LD_PRELOAD='/path/to/libhardened_malloc.so' /path/to/program will do nothing or approximately nothing. mozjemalloc uses mmap and low level allocation tools to create chunks of memory to be used by its internal memory allocator. To successfully replace Firefox memory allocator you should either use LD_PRELOAD _with_ a --disable-jemalloc build OR Firefox's replace_malloc functionality: https://searchfox.org/mozilla-central/source/memory/build/replace_malloc.h

Completely agreed. And, using LD_PRELOAD to hook into the allocator is improper, anyways, since it won't catch early uses of the allocator. And, as you mention, it wouldn't even work with Firefox given mozjemalloc. Firefox is not the only application to want to have control over the allocator.

The only way to guarantee catching early allocator use is to switch the system's allocator (ie, libc itself) to the new one. Otherwise, the application will end up with two allocator implementations being used: the application's custom one and the system's, included and used within libc (and other system libraries, of course.)

Fingerprinting: It is most likely possible to be creative enough to fingerprint what memory allocator is used. If we were to choose from different allocators at runtime, I don't think that fingerprinting is the worst thing open to us - it seems likely that any attacker who does such a attack could also fingerprinting your CPU speed, RAM, and your ASLR base addresses which depending on OS might not change until reboot.

My post was more along the lines of: what system-level components, if replaced, have a potentially visible effect on current (or future) fingerprinting techniques?

And: If, or how, does breaking monocultures affect fingerprinting? Breaking monocultures is typically done to help secure an environment through diversity, causing an attacker to have to spend more resources in quest for success.

The only reason I can think of to choose between allocators at runtime is to introduce randomness into the allocation strategy. An attacker relying on a blind overwrite may not be able to position their overwrite reliably AND it has the cause the process to crash otherwise they can just try again.

Allocators can introduce randomness themselves, you don't need to choose between allocators to do that.

I'm assuming you're talking about randomness of the address space? When it comes to browsers, ASLR is dead. Local execution of remotely-sourced arbitrary code, an attack vector ASLR was never meant to protect against.

Thus, discussion of whether choice of allocator improves effectiveness of ASLR when applied to the browser is moot.

In virtually all browser exploits we have seen recently the attacker creates exploitation primitives that allow partial memory read/write and then full memory read/write. Randomness introduced is bypassed and ineffective. I've seen a general trend away from randomness for this purpose. The exception is when the attacker is heavily constrained - like exploiting over IPC or in a network protocol. Not when the attacker has a full Javascript execution environment available to them.

When exploiting a memory corruption vulnerability, you can target the application's memory (meaning, target a DOM object or an ArrayBuffer) or you can target the memory allocator's metadata. While allocator metadata corruption was popular in the past, I haven't seen it used recently.

Okay all that out of the way, let's talk about allocators.

I skimmed https://github.com/GrapheneOS/hardened_malloc and it looks like it has:

• out of line metadata
• double free protection
• guard regions of some type
• zero-filling
• MPK support
• randomization
• support for arenas

mozjemalloc:

• arenas (we call them partitions)
• randomization (support for, not enabled by default due to limited

utility, but improvements coming)

• double free protection
• zero-filling

In Progress:

• we're actively working on guard regions

Future Work:

• out of line metadata
• MPK

harden_malloc definitely has more bells and whistles than mozjemalloc. But the benefit gained by slapping in an LD_PRELOAD and calling it a day is small to zero. Probably negative because you'll not utilize partitions by default. You'd need a particurally constrained vulnerability to actually prevent exploitation - it's more likely you'll just cost the attacker another 2-8 hours of work.

100% agreed with your thoughts on LD_PRELOAD here, with the additions of my notes above.

Out of line metadata is on-the-surface-attractive but... that tends to only help when you have a off-by-one/four write and you corrupt metadata state because it's the only thing you *can* do. With out of line metadata, you can just corrupt a real object and effect a different type of corruption. I'm pretty skeptical of the benefit at this point, although I could be convinced. We don't see metadata corruption attacks anymore - but I'm not sure if it's because we find better exploit primitives or better vulnerabilities.

In particular, if you wanted to pursue hardened_malloc you would need to use replace_malloc and wire up the partitions correctly. Randomization will almost certainly not help (and will hurt performance)*. MPK sounds nice but you have to use it correctly (which requires application code changes), you have to ensure there are no MPK gadgets, and oh wait no one can use it because it's only available in Linux on server CPUs. =(

• One place randomization will help is on the other side of an IPC

boundary. e.g. in the parent process. I'm trying to get that enabled for mozjemalloc in H2 2019.

In conclusion, while it's possible hardened_malloc could provide some small security increase over mozjemalloc, the gap is much smaller than it was when I advocated for allocator improvements 5 years ago, the effort is definitely non-trivial, and the gap is closing.

I'm curious about how breaking monocultures affect attacks. I think supporting hardened_malloc (or <insert arbitrary allocator here>) would provide at least the framework for academic exercises.

Thanks,

On Mon, Aug 19, 2019 at 04:09:36PM +0000, Tom Ritter wrote:

Okay I'm going to try and clear up a lot of misconceptions and stuff here. I don't own Firefox's memory allocator but I have worked in it, recently, and am one of the people who are working on hardening it.

This makes it clear why you're spreading misinformation. You're going out of your way to make false and misleading claims about mozjemalloc and hardened_malloc, particularly your bogus comparisons between them.

Bolting on a few weak implementations of hardening features to an allocator inherently very friendly to memory corruption exploitation does not make it anything close to being hardened allocator, sorry.

Firefox's memory allocator is not jemalloc. It's probably better referred to as mozjemalloc. We forked jemalloc and have been improving it (at least from our perspective.) Any analysis of or comparison to jemalloc is - at this point - outdated and should be redone from scratch against mozjemalloc on mozilla-central.

It's not particularly different and the comparison isn't outdated. I've made some substantial contributions to jemalloc upstream and wrote the integration into Rust. I'm deeply familiar with the design choices and implementation of jemalloc. Those design choices lead to having a memory allocator that's extremely friendly to exploitation. A lot of this is covered in the hardened_malloc documentation, without going out of the way to directly compare it to specific allocators. If the documentation is not currently clear enough, I could provide a comparison to jemalloc as an example of comparing a very hardened allocator implementation to one that's the direct opposite. It makes exploitation significantly easier overall than the traditional dlmalloc style design.

LD_PRELOAD='/path/to/libhardened_malloc.so' /path/to/program will do nothing or approximately nothing. mozjemalloc uses mmap and low level allocation tools to create chunks of memory to be used by its internal memory allocator. To successfully replace Firefox memory allocator you should either use LD_PRELOAD _with_ a --disable-jemalloc build OR Firefox's replace_malloc functionality: https://searchfox.org/mozilla-central/source/memory/build/replace_malloc.h

LD_PRELOAD is not how hardened_malloc is supposed to be used outside of testing it anyway. It's meant to be integrated in libc, in which case --disable-jemalloc would be enough, although it can also be integrated into a specific program. That doesn't sidestep the importance of doing other hardening in libc and the rest of the system though.

Fingerprinting: It is most likely possible to be creative enough to fingerprint what memory allocator is used. If we were to choose from different allocators at runtime, I don't think that fingerprinting is the worst thing open to us - it seems likely that any attacker who does such a attack could also fingerprinting your CPU speed, RAM, and your ASLR base addresses which depending on OS might not change until reboot.

They can obtain a lot more than just information about the hardware. A lot of the hardware and OS information that fingerprinting mitigations try to hide are leaked via performance measurements. It can also leak a lot of data from within the browser.

The only reason I can think of to choose between allocators at runtime is to introduce randomness into the allocation strategy. An attacker relying on a blind overwrite may not be able to position their overwrite reliably AND it has the cause the process to crash otherwise they can just try again.

The hardened_malloc design provides far more than randomization, and the fine-grained randomization within size class regions is one of the least important and lowest priority features.

https://github.com/GrapheneOS/hardened_malloc/blob/master/README.md

Allocators can introduce randomness themselves, you don't need to choose between allocators to do that.

This is not something that can simply be bolted onto an existing allocator design with a good approach. It needs to be more heavily integrated into the design, and the same applies to an even greater extent to more important security features than weak fine-grained randomization.

In virtually all browser exploits we have seen recently the attacker creates exploitation primitives that allow partial memory read/write and then full memory read/write. Randomness introduced is bypassed and ineffective. I've seen a general trend away from randomness for this purpose. The exception is when the attacker is heavily constrained - like exploiting over IPC or in a network protocol. Not when the attacker has a full Javascript execution environment available to them.

When exploiting a memory corruption vulnerability, you can target the application's memory (meaning, target a DOM object or an ArrayBuffer) or you can target the memory allocator's metadata. While allocator metadata corruption was popular in the past, I haven't seen it used recently.

The importance of out-of-line metadata is far beyond simply preventing exploitation through the allocator metadata. It's crucial for a hardened allocator to have a reliable source of information about allocations without trusting data read from freed allocations or next to the memory allocations.

Okay all that out of the way, let's talk about allocators.

I skimmed https://github.com/GrapheneOS/hardened_malloc and it looks like it has:

• out of line metadata

It has fully out-of-line metadata in a reserved region with the entirety of the mutable allocator state. This is not simply for the sake of protecting the metadata.

• double free protection

The hardened_malloc approach provides deterministic, immediate detection of every invalid free for both slab and large allocations. It detects much more than detect double free and doesn't rely on a probabilistic approach or inline / semi-inline metadata to provide it.

The quarantine implementations for slab allocations, slabs and large allocation also massively help with this. For slab allocations it has to go out of the way to preserve the deterministic, immediate double free detection since a naive quarantine implementation would interfere.

• guard regions of some type

It has much more than 'guard regions of some type'. It's designed around being able to implement features like this is a very meaningful and impactful way with low performance cost.

It reserves a slab allocation region, partitions that into arenas and partitions those into dedicated regions for each size class with large high entropy random guard regions around them.

Inside these size class regions, it skips slab locations to leave behind guard slabs, providing a sparse heap. These are currently inserted at a deterministic interval, but the plan is to randomize that to mitigate heap sprays within the size class regions.

Freed slabs beyond the small amount of cached slabs are also replaced with fresh guard regions, which is helpful. There's a quarantine for these too, to keep them memory protected once they're become just like the never allocated slab locations.

Large allocations (> 128k) have randomly sized guards around them using a random range based on a ratio to the allocation size. When a large allocation is freed, it's quarantined by replacing it with a guard region held onto as long as possible based on the large allocation quarantine size.

• zero-filling

It also provides write-after-free detection via checking the zero filling at allocation time, and the slab allocation quarantine is designed to work well with this.

• MPK support

This isn't one of the major features.

• randomization

This isn't simply a basic yes or no point about an allocator.

The slot randomization within slabs is the least impactful / important form of this in the implementation, aside from the planned randomization for choosing slabs within size class regions.

The quarantine randomization and high entropy bases for partitions are much more important. The quarantine randomization and the FIFO portion of it are also going to mix very well with the planned approach to using memory tagging for deterministic use-after-free detection, while still having deterministic sequential overflow detection and randomized tags as a probabilistic general purpose memory corruption mitigation.

• support for arenas

This is a completely dishonest and ridiculous misrepresentation of the design and security properties laid out in that README. People should read it for themselves and they'll see that your attempt at spinning misinformation about it is a complete joke.

mozjemalloc:

• arenas (we call them partitions)

Unlike hardened_malloc, they're mixed together and address space is reused between them rather than having strong isolation. The approach in hardened_malloc also partitions each size class. Calling the mozjemalloc arenas partitions as if it's an implementation of a security feature is a joke.

• randomization (support for, not enabled by default due to limited

utility, but improvements coming)

• double free protection
• zero-filling

In Progress:

• we're actively working on guard regions

As covered above, you're being misleading with each of these points, by portraying these things as something black and white that the allocator either has or doesn't have when that couldn't be further from reality.

In particular, talking about randomization and guard regions as if this is a matter of having them or not having them is ridiculous. There is more to invalid free detection than double free detection and how well it works has a lot of variation. It can be deterministic detection like the hardened_malloc implementation, or probabilistic detection that an attacker could much more easily bypass. The reuse of freed allocations also matters a lot, since once it's handed out again, a free based on a past generation allocation won't be considered invalid, despite it being wrong and dangerous. This is why the design of memory allocation reuse and quarantines matters so much. The documentation on thread caching in the hardened_malloc README elaborates on why that's not compatible with a hardened allocator due to interfering with doing anything like this properly.

Future Work:

• out of line metadata

There's a huge variation in what this means. The hardened_malloc metadata is fully out-of-line in a dedicated region, with that address space never mixed / reused with anything else. The same applies to all the size class regions within arenas.

• MPK

For what exactly?

harden_malloc definitely has more bells and whistles than mozjemalloc.

The hardened_malloc implementation is far simpler and more minimal. It has a core design focused heavily on having quantifiable, best in class security. The security is built into the design from the ground up and cannot simply be bolted onto to it or another allocator. I had to write it in the first place because I was unable to turn OpenBSD malloc into something comparable by simply continuing to bolt on additional security features to it which was the original approach. It should also be noted that OpenBSD malloc is a hardened allocator and shares many of the same security properties from the core design. The main distinction in the core approach is that hardened_malloc is heavily designed around taking advantage of the large address space on modern machines.

But the benefit gained by slapping in an LD_PRELOAD and calling it a day is small to zero. Probably negative because you'll not utilize partitions by default. You'd need a particurally constrained vulnerability to actually prevent exploitation - it's more likely you'll just cost the attacker another 2-8 hours of work.

The claim that mozjemalloc has partitioning and hardened_malloc does not couldn't be further from the truth. The opposite is true. What you claim to be partitioning is not a proper implementation, and hardened_malloc does have a proper implementation not only for arenas, but for dedicated size class regions within those. I find it ridiculous how you attempt to attack the project with these lies to promote your own work, which is hardly comparable at all. It's not the same thing. Bolting on a few security features to an allocator design that's exploitation friendly from the ground up doesn't make it a hardened allocator. That's even more true when the implementations of those features are unnecessarily weak.

Out of line metadata is on-the-surface-attractive but... that tends to only help when you have a off-by-one/four write and you corrupt metadata state because it's the only thing you *can* do. With out of line metadata, you can just corrupt a real object and effect a different type of corruption. I'm pretty skeptical of the benefit at this point, although I could be convinced. We don't see metadata corruption attacks anymore - but I'm not sure if it's because we find better exploit primitives or better vulnerabilities.

Out-of-line metadata is not simply about preventing attacks on the metadata itself. It provides much more than that. I'm not sure why you think your ignorant opinions on these topics matter, when you so clearly don't know what you're talking about at all.

In particular, if you wanted to pursue hardened_malloc you would need to use replace_malloc and wire up the partitions correctly. Randomization will almost certainly not help (and will hurt performance)*.

The hardened_malloc design is focused on reliable, deterministic memory corruption mitigations. Randomization is used where possible, and in a way that has a low impact on performance. The high entropy base randomization for each size class region within arenas has no significant impact on performance. The randomization for large allocation (> 128k) guard regions and that quarantine has no substantial impact on performance. The impact from slot randomization and the slab allocation quarantine is measurable but not high, and the slab quarantine has no substantial impact.

MPK sounds nice but you have to use it correctly (which requires application code changes), you have to ensure there are no MPK gadgets, and oh wait no one can use it because it's only available in Linux on server CPUs. =(

Using MPK is not one of the major features of hardened_malloc and the usage doesn't rely on not having MPK gadgets. This is explicitly documented in the README. It's the only optional security feature that's not enabled by default. It should be pointed out that most of security offered by hardened_malloc is not a feature that can be turned on or off because it's the design itself that's hardened, not the fact that it has some optional security features bolted onto it.

• One place randomization will help is on the other side of an IPC

boundary. e.g. in the parent process. I'm trying to get that enabled for mozjemalloc in H2 2019.

In conclusion, while it's possible hardened_malloc could provide some small security increase over mozjemalloc, the gap is much smaller than it was when I advocated for allocator improvements 5 years ago, the effort is definitely non-trivial, and the gap is closing.

No, you're just making false attacks and misleading comparisons / spin to promote your own work, which is trash. You're being incredibly dishonest and unethical. You didn't even bother to inform yourself about hardened_malloc by actually reading through the documentation. Instead, you just jump to conclusions and present yourself as an expert on topics you are clearly incredibly ignorant about. You really don't know what you're talking about, and your post on this mailing list is offensive. Your post as a whole is nonsense, and your conclusion is bogus. I recommend actually trying to read the documentation and learning a bit about memory allocators, memory corruption and exploit mitigations before lecturing people and attacking other projects.

Expect to hear more about this in the future, because I'll be contacting Mozilla about the fact that you're spreading dishonest attacks and misinformation about my work, which is a continuation of past harm inflicted on me by Mozilla employees. You're digging up a past conflict that was supposed to be put aside, so nice work with that. It's nice to have another example to point to about damages inflicted by Mozilla. I expect this to be made right just like the past attacks by Mozilla. I want a retraction and an apology, but either way this is being included in documentation on damages.

If people had the cycles to invest in something like this, I would actually advocate for helping us test and benchmark Fuzzyfox, and see if we can get the browser into a usable state with Fuzzyfox so we could enable it in Tor Browser.

Maybe if you didn't spend your time trying to inflict harm on other projects and individuals, people would be more interested in helping. I highly recommend that no one contributes any time to Mozilla, which has a history of taking advantage of contributors, lying to them and abusing them. Mozilla is built on abusing volunteer labour, and your post here is just a continuation of that. It's a thoroughly unethical company with disgusting, unethical people like yourself at the forefront of it. I have no idea why anyone would want to help scumbags like you.

On Wed, 21 Aug 2019 at 11:57, Nicolas Vigier boklm@mars-attacks.org wrote:

On Wed, 21 Aug 2019, Daniel Micay wrote:

...

No, you're just making false attacks and misleading comparisons / spin to promote your own work, which is trash. You're being incredibly dishonest and unethical. You didn't even bother to inform yourself about

It's fine to disagree with Tom about what he wrote in his previous email, however calling him dishonest and unethical seems very wrong to me. If anything he wrote was not correct or misleading

If someone is going to criticize other people's work and dismiss it as nearly useless without even somewhat informing themselves about it, they should expect to be called out on that. You might find my reply offensive, but I found the email that I was replying to extremely offensive and I had to subscribe to this list and figure out how to send a reply to a past email from the archive to defend the value of my work. It wasn't fair or accurate criticisms or comparisons.

It's not the first attack that I responded to today or the last. It's one of many. The depth and tone of my responses varies based on what I'm responding to. If I can assume good faith, I will do it, but I could not do that here with how it was presented. All of this is time is taken away from working on the projects and that hurts, but so does leaving it unchallenged.

I doubt it was intentional and more likely it was some honest mistakes.

I don't see how you can suggest it wasn't the intention. It's dismissing the project / work and the value of it without even putting in basic effort to learn what it is and what it does. It's presented as informed, expert commentary when it isn't. I had a serious problem with it and I responded in the way I felt was suitable. I intended to express how I felt about it which wouldn't have been accomplished by using forced diplomatic wording. I said that they were being dishonest and unethical with their actions. If people don't want to be called out for that, they shouldn't do it.

Mozilla has a history of harming me. I've documented this as one more case of attacks from Mozilla to go along with everything else. I see no reason to put up with it or tolerate it. Mozilla should expect that one day they're going to be held accountable. If people at Mozilla aren't aware of the unethical behavior it regularly engages in including an exploitative approach to contributors, they should inform themselves. My issue is primarily with Mozilla as an organization and a culture rather than any specific individuals participating in that. I think the problem is ultimately that self-righteous, dishonest organization presenting itself as a benevolent force of good when it really doesn't line up with the reality. It taints how the people involved approach things. Since these past issues were never addressed, and the company hasn't changed, any attacks from people at Mozilla are a spark igniting this existing conflict. It's not my responsibility to inform all their employees about what the organization has done and failed to resolve.

I'm not planning on participating on this list beyond defending myself here and in future cases.

Yeah same, this convo went from 'wow an interesting discussion about allocators' to 'fuck you Tom' real quick and without provocation.

From a human standpoint, maybe try and be your best self? Or (if you prefer) from a practical standpoint, maybe berating on one of the devs that would be reviewing your allocator patches isn't the best path forward to achieving your goals?

best, -Richard

On 8/21/19 1:28 PM, Ryan Duff wrote:

...

If someone is going to criticize other people's work and dismiss it as

nearly useless without even somewhat informing themselves about it, they should expect to be called out on that.

I don't have a dog in this fight but, as an outside observer, I never got the impression that this is what Tom was doing. I read it as "hardened_malloc is better but we are trying to do these /n /things to try to close that gap". I didn't read it as an attack at all.

On Wed, Aug 21, 2019 at 1:37 PM Daniel Micay <danielmicay@gmail.com mailto:danielmicay@gmail.com> wrote:

On Wed, 21 Aug 2019 at 11:57, Nicolas Vigier <boklm@mars-attacks.org
<mailto:boklm@mars-attacks.org>> wrote:
>
> On Wed, 21 Aug 2019, Daniel Micay wrote:
>
> >
> > No, you're just making false attacks and misleading comparisons
/ spin
> > to promote your own work, which is trash. You're being incredibly
> > dishonest and unethical. You didn't even bother to inform
yourself about
>
> It's fine to disagree with Tom about what he wrote in his previous
email,
> however calling him dishonest and unethical seems very wrong to me. If
> anything he wrote was not correct or misleading

If someone is going to criticize other people's work and dismiss it as
nearly useless without even somewhat informing themselves about it,
they should expect to be called out on that. You might find my reply
offensive, but I found the email that I was replying to extremely
offensive and I had to subscribe to this list and figure out how to
send a reply to a past email from the archive to defend the value of
my work. It wasn't fair or accurate criticisms or comparisons.

It's not the first attack that I responded to today or the last. It's
one of many. The depth and tone of my responses varies based on what
I'm responding to. If I can assume good faith, I will do it, but I
could not do that here with how it was presented. All of this is time
is taken away from working on the projects and that hurts, but so does
leaving it unchallenged.

> I doubt it was intentional and more likely it was some honest
mistakes.

I don't see how you can suggest it wasn't the intention. It's
dismissing the project / work and the value of it without even putting
in basic effort to learn what it is and what it does. It's presented
as informed, expert commentary when it isn't. I had a serious problem
with it and I responded in the way I felt was suitable. I intended to
express how I felt about it which wouldn't have been accomplished by
using forced diplomatic wording. I said that they were being dishonest
and unethical with their actions. If people don't want to be called
out for that, they shouldn't do it.

Mozilla has a history of harming me. I've documented this as one more
case of attacks from Mozilla to go along with everything else. I see
no reason to put up with it or tolerate it. Mozilla should expect that
one day they're going to be held accountable. If people at Mozilla
aren't aware of the unethical behavior it regularly engages in
including an exploitative approach to contributors, they should inform
themselves. My issue is primarily with Mozilla as an organization and
a culture rather than any specific individuals participating in that.
I think the problem is ultimately that self-righteous, dishonest
organization presenting itself as a benevolent force of good when it
really doesn't line up with the reality. It taints how the people
involved approach things. Since these past issues were never
addressed, and the company hasn't changed, any attacks from people at
Mozilla are a spark igniting this existing conflict. It's not my
responsibility to inform all their employees about what the
organization has done and failed to resolve.

I'm not planning on participating on this list beyond defending myself
here and in future cases.
_______________________________________________
tor-dev mailing list
tor-dev@lists.torproject.org <mailto:tor-dev@lists.torproject.org>
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

---

tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

On Wed, 21 Aug 2019 at 16:28, Ryan Duff ry@nduff.com wrote:

...

If someone is going to criticize other people's work and dismiss it as

nearly useless without even somewhat informing themselves about it, they should expect to be called out on that.

I don't have a dog in this fight but, as an outside observer, I never got the impression that this is what Tom was doing. I read it as "hardened_malloc is better but we are trying to do these n things to try to close that gap". I didn't read it as an attack at all.

Skipping actually reading the documentation about what it does and dismissing it as only being a couple features that could be added to jemalloc is ignorant misinformation and what I find offensive. Presenting it as a situation where they could simply bolt on some security features to jemalloc and provide something comparable is the problem. The list was also ridiculous, and simply wrong. What I found dishonest is that it's portrayed as if it's an expert's assessment of the differences rather than a bunch of poorly informed assumptions and assorted misinformation. It was not a fair attempt to actually evaluate hardened_malloc and compare it, but rather a lazy attempt to dismiss it and downplay the advantages on the security front. Now, consider that this is a personal project that I work on, using my own time, with only donations supporting my work. It's not a product from a company or a collaborative project. No one else has written a single line of code in it, or played a role in the design. It's an attack on my work, and myself, including my credibility since I present it as having substantial advantages and here's this person who seems like an expert on the topic (but hasn't even read the README or looked in any detail at it) from Mozilla dismissing it.

The gap between a hardened allocator design and a performance-oriented allocator design has if anything been widening due to divergence in design trade-offs / compromises. The glibc allocator recently got a lot worse on the security front (it was already awful) by adding thread caching, which is fantastic for performance. There have been many similar performance improvements in jemalloc reducing security in the past couple years.

Memory tagging is only going to increase the divergence with how many possibilities it opens up and how well it synergizes with hardened allocator designs.

[Replying to both emails]

My hope is that most of this stems from my cursory work in replying and a general misunderstanding.

I've seen people advocating for replacing the memory allocator in Tor Browser since I started the effort five years ago here: https://github.com/iSECPartners/publications/blob/master/reports/Tor%20Brows... My replies were primarily focused on replying to the advocations I've seen in the past, and the misconceptions I've seen repeated there (like LD_PRELOAD).

On Wed, 21 Aug 2019 at 12:14, Daniel Micay danielmicay@gmail.com wrote:

On Sat, Aug 17, 2019 at 09:17:40PM +0000, Tom Ritter wrote:

...

On Sat, 17 Aug 2019 at 15:06, procmem at riseup.net <procmem at riseup.net> wrote:

...

Question for the Tor Browser experts. Do you know if it is possible to remotely fingerprint the browser based on the memory allocator it is using? (via JS or content rendering)

Fingerprint what aspect of the browser/machine?

Performance-based fingerprinting of the browser can easily differentiate between using a different malloc implementation. That can already obtain a lot of fingerprinting information about the hardware and OS so this may not actually matter much, but it's entirely possible.

Agreed.

...

...

We are thinking of switching Tor Browser to use the minimalist and security oriented hardened_malloc written by Daniel Micay. Thanks.

I wouldn't advise giving up partitioning for.... what exactly? What features does this allocator have that 68's jemalloc doesn't?

The hardened_malloc allocator heavily uses partitioning, and has a much stronger implementation than the very weak approach in mozjemalloc. It statically reserves memory regions for metadata and a dedicated region for each arena, with each size class receiving a dedicated sub-region within the arena. These sub-regions are placed within their own guard region and each have a high entropy random base. It never mixes address space between these regions or reuses the address space. This is much different than what you call 'partitioning' in mozjemalloc which does not really qualify. What you're talking about is mozjemalloc exposing an API for choosing the arena from the code, which can certainly be done with hardened_malloc too. However, in mozjemalloc, the address space for different arenas is mixed together and reused between them. It's really a stretch to call this partitioning, and it doesn't have the baseline separation of size classes either.

People can read about hardened_malloc in the README:

https://github.com/GrapheneOS/hardened_malloc/blob/master/README.md#hardened...

I don't know why you're making the misleading claim that people would need to give up partitioning.

My reply about giving up partitioning here (which I clarify in the second email) is that using LD_PRELOAD will negate partitioning.

It's also really a stretch to call what Mozilla is doing in mozjemalloc partitioning in the first place, so your claim is really quite backwards...

Let me start by asserting emphatically and agreeing that hardened_malloc is a much stronger allocator than mozjemalloc or jemalloc. PartitionAlloc is also for that matter.

Mozilla's partitioning has always been focused on preventing the Use-After-Free problems that have plagued Firefox for years. Allocate a DOM object, retain a pointer to it, free it, replace it with an Arraybuffer, align the vtable entry, redirect execution. We allocate ArrayBuffer contents and strings in separate arenas to avoid the immediate reuse of these bytes.

On Wed, 21 Aug 2019 at 13:40, Daniel Micay danielmicay@gmail.com wrote:

On Mon, Aug 19, 2019 at 04:09:36PM +0000, Tom Ritter wrote:

...

Okay I'm going to try and clear up a lot of misconceptions and stuff here. I don't own Firefox's memory allocator but I have worked in it, recently, and am one of the people who are working on hardening it.

This makes it clear why you're spreading misinformation. You're going out of your way to make false and misleading claims about mozjemalloc and hardened_malloc, particularly your bogus comparisons between them.

My comparisons were rushed and cursory. I apologize and I'll clarify my conclusions at the end of the email.

Bolting on a few weak implementations of hardening features to an allocator inherently very friendly to memory corruption exploitation does not make it anything close to being hardened allocator, sorry.

Fair.

...

Firefox's memory allocator is not jemalloc. It's probably better referred to as mozjemalloc. We forked jemalloc and have been improving it (at least from our perspective.) Any analysis of or comparison to jemalloc is - at this point - outdated and should be redone from scratch against mozjemalloc on mozilla-central.

It's not particularly different and the comparison isn't outdated.

As above, the comparisons I were referring to were the five year document I wrote, and past trac tickets and threads here.

...

LD_PRELOAD='/path/to/libhardened_malloc.so' /path/to/program will do nothing or approximately nothing. mozjemalloc uses mmap and low level allocation tools to create chunks of memory to be used by its internal memory allocator. To successfully replace Firefox memory allocator you should either use LD_PRELOAD _with_ a --disable-jemalloc build OR Firefox's replace_malloc functionality: https://searchfox.org/mozilla-central/source/memory/build/replace_malloc.h

LD_PRELOAD is not how hardened_malloc is supposed to be used outside of testing it anyway. It's meant to be integrated in libc, in which case --disable-jemalloc would be enough, although it can also be integrated into a specific program. That doesn't sidestep the importance of doing other hardening in libc and the rest of the system though.

We agree on this.

...

Fingerprinting: It is most likely possible to be creative enough to fingerprint what memory allocator is used. If we were to choose from different allocators at runtime, I don't think that fingerprinting is the worst thing open to us - it seems likely that any attacker who does such a attack could also fingerprinting your CPU speed, RAM, and your ASLR base addresses which depending on OS might not change until reboot.

They can obtain a lot more than just information about the hardware. A lot of the hardware and OS information that fingerprinting mitigations try to hide are leaked via performance measurements. It can also leak a lot of data from within the browser.

Also agreed. I wasn't mentioning things like the broad class of pixel-stealing attacks, or the user activity-class of attacks. Which is also the reason I advocate for getting Fuzzyfox into a usable state.

...

The only reason I can think of to choose between allocators at runtime is to introduce randomness into the allocation strategy. An attacker relying on a blind overwrite may not be able to position their overwrite reliably AND it has the cause the process to crash otherwise they can just try again.

The hardened_malloc design provides far more than randomization

Of course. But the randomization I'm referring to isn't the randomization inside the allocator, it's the random choice of *which* allocator to use.

...

Allocators can introduce randomness themselves, you don't need to choose between allocators to do that.

This is not something that can simply be bolted onto an existing allocator design with a good approach. It needs to be more heavily integrated into the design, and the same applies to an even greater extent to more important security features than weak fine-grained randomization.

I agree.

...

In virtually all browser exploits we have seen recently the attacker creates exploitation primitives that allow partial memory read/write and then full memory read/write. Randomness introduced is bypassed and ineffective. I've seen a general trend away from randomness for this purpose. The exception is when the attacker is heavily constrained - like exploiting over IPC or in a network protocol. Not when the attacker has a full Javascript execution environment available to them.

When exploiting a memory corruption vulnerability, you can target the application's memory (meaning, target a DOM object or an ArrayBuffer) or you can target the memory allocator's metadata. While allocator metadata corruption was popular in the past, I haven't seen it used recently.

The importance of out-of-line metadata is far beyond simply preventing exploitation through the allocator metadata. It's crucial for a hardened allocator to have a reliable source of information about allocations without trusting data read from freed allocations or next to the memory allocations.

My understanding of this statement is that the metadata must be protected against non-security-related corruption; if I'm misunderstanding I'm happy to be educated.

...

Okay all that out of the way, let's talk about allocators.

I skimmed https://github.com/GrapheneOS/hardened_malloc and it looks like it has:

I can see how my skimming, and simplistic bullet points, could be insulting to someone who has invested such a great deal of work into their project. I'm sorry; it's too easy to forget that on the other side of text is a person who deserves our consideration, and that on the other side of code is a significant investment of effort.

...

• support for arenas

This is a completely dishonest and ridiculous misrepresentation of the design and security properties laid out in that README. People should read it for themselves and they'll see that your attempt at spinning misinformation about it is a complete joke.

I'd intended this as a simplified response to the expected points I'd seen brought up in the past. I apologize if it came across as intentionally dishonest.

...

mozjemalloc:

• arenas (we call them partitions)

Unlike hardened_malloc, they're mixed together and address space is reused between them rather than having strong isolation. The approach in hardened_malloc also partitions each size class. Calling the mozjemalloc arenas partitions as if it's an implementation of a security feature is a joke.

As I mentioned before, our implementation of partitions is focused on preventing the immediate reuse of bytes to negate the common UAF pattern we have seen for years. hardened_malloc's is clearly stronger.

...

• randomization (support for, not enabled by default due to limited

utility, but improvements coming)

• double free protection
• zero-filling

In Progress:

• we're actively working on guard regions

As covered above, you're being misleading with each of these points, by portraying these things as something black and white that the allocator either has or doesn't have

That's a fair criticism.

In particular, talking about randomization and guard regions as if this is a matter of having them or not having them is ridiculous. There is more to invalid free detection than double free detection and how well it works has a lot of variation. It can be deterministic detection like the hardened_malloc implementation, or probabilistic detection that an attacker could much more easily bypass. The reuse of freed allocations also matters a lot, since once it's handed out again, a free based on a past generation allocation won't be considered invalid, despite it being wrong and dangerous. This is why the design of memory allocation reuse and quarantines matters so much. The documentation on thread caching in the hardened_malloc README elaborates on why that's not compatible with a hardened allocator due to interfering with doing anything like this properly.

Also a fair assessment.

...

Future Work:

• out of line metadata

There's a huge variation in what this means. The hardened_malloc metadata is fully out-of-line in a dedicated region, with that address space never mixed / reused with anything else. The same applies to all the size class regions within arenas.

That's very strong protection - I hope we can integrate the same level of security in the future.

...

• MPK

For what exactly?

Still TBD; but our initial thoughts are using it in the JIT Engine and to effect XOM for certain use cases.

...

But the benefit gained by slapping in an LD_PRELOAD and calling it a day is small to zero. Probably negative because you'll not utilize partitions by default. You'd need a particurally constrained vulnerability to actually prevent exploitation - it's more likely you'll just cost the attacker another 2-8 hours of work.

The claim that mozjemalloc has partitioning and hardened_malloc does not couldn't be further from the truth.

A misunderstanding: I didn't claim that; I claimed that using LD_PRELOAD would not make use of partitions.

I find it ridiculous how you attempt to attack the project with these lies to promote your own work, which is hardly comparable at all. It's not the same thing. Bolting on a few security features to an allocator design that's exploitation friendly from the ground up doesn't make it a hardened allocator. That's even more true when the implementations of those features are unnecessarily weak.

My intention was not to promote my own work; or negate yours. I'm sorry it came across that way. As I said at the top of this email, hardened_malloc is clearly a much stronger allocator that mozjemalloc.

I'll try to clear up my intention with my email here:

Replacing the allocator used by Tor Browser naively (LD_PRELOAD) would be a net loss in security. I think (?) you'd agree. Replacing the allocator used by Tor Browser correctly would grant a hardened allocator but would not significantly affect how the past several exploits written against Firefox would be written, and would not be a significant impediment to exploit authors who are using the type of vulnerabilities we have seen exploited of late.

It would certainly affect the exploitation of more limited vulnerabilities (fixed-byte over-reads/writes).

...

Out of line metadata is on-the-surface-attractive but... that tends to only help when you have a off-by-one/four write and you corrupt metadata state because it's the only thing you *can* do. With out of line metadata, you can just corrupt a real object and effect a different type of corruption. I'm pretty skeptical of the benefit at this point, although I could be convinced. We don't see metadata corruption attacks anymore - but I'm not sure if it's because we find better exploit primitives or better vulnerabilities.

Out-of-line metadata is not simply about preventing attacks on the metadata itself. It provides much more than that. I'm not sure why you think your ignorant opinions on these topics matter, when you so clearly don't know what you're talking about at all.

Unfair! :)

...

In particular, if you wanted to pursue hardened_malloc you would need to use replace_malloc and wire up the partitions correctly. Randomization will almost certainly not help (and will hurt performance)*.

The hardened_malloc design is focused on reliable, deterministic memory corruption mitigations. Randomization is used where possible, and in a way that has a low impact on performance. The high entropy base randomization for each size class region within arenas has no significant impact on performance. The randomization for large allocation (> 128k) guard regions and that quarantine has no substantial impact on performance. The impact from slot randomization and the slab allocation quarantine is measurable but not high, and the slab quarantine has no substantial impact.

My statement on performance was based on my experiments enabling our limited randomization in the Content Process (numbers lost, but they came from https://bugzilla.mozilla.org/show_bug.cgi?id=1376408#c18). It really hurt DOM node traversal. I'll concede it's possible that hardened_malloc may have less any performance cost here - I haven't tested it. But... I'm skeptical.

(You seem to agree that randomization is not a strong security feature?)

...

MPK sounds nice but you have to use it correctly (which requires application code changes), you have to ensure there are no MPK gadgets, and oh wait no one can use it because it's only available in Linux on server CPUs. =(

Using MPK is not one of the major features of hardened_malloc and the usage doesn't rely on not having MPK gadgets. This is explicitly documented in the README. It's the only optional security feature that's not enabled by default. It should be pointed out that most of security offered by hardened_malloc is not a feature that can be turned on or off because it's the design itself that's hardened, not the fact that it has some optional security features bolted onto it.

Fair! (Although I'm confused about your comment saying you can provide protection without eliminating MPK gadgets.)

...

In conclusion, while it's possible hardened_malloc could provide some small security increase over mozjemalloc, the gap is much smaller than it was when I advocated for allocator improvements 5 years ago, the effort is definitely non-trivial, and the gap is closing.

No, you're just making false attacks and misleading comparisons / spin to promote your own work, which is trash.

=(

You're being incredibly dishonest and unethical.

I'd hope that the principal of charity would lead to the conclusion that instead I was being simplistic and ignorant.

You didn't even bother to inform yourself about hardened_malloc by actually reading through the documentation. Instead, you just jump to conclusions and present yourself as an expert on topics you are clearly incredibly ignorant about. You really don't know what you're talking about, and your post on this mailing list is offensive. Your post as a whole is nonsense, and your conclusion is bogus.

Which conclusion? The one I didn't make (but certainly implied) that mozjemalloc is comparable security-wise to hardened_malloc? I agree, such a conclusion is bogus.

Or the one that replacing the allocator will not have a significant effect on the exploits written for the past couple years of Firefox expoits?

This seems to be the central misunderstanding, and why I agree that you are entitled to take offense.

[ending statements]

I don't have any knowledge of, or context about, your past interactions with Mozilla. We all wear many hats; but this discussion isn't about Mozilla, Firefox, or replacing Firefox's memory allocator - it's about Tor Browser. If you'd like to escalate your concerns, they can be forwarded to Tor's Community Council; that information is available at https://trac.torproject.org/projects/tor/wiki/org/CommunityCouncil

-tom