Firefox has a feature where you can build a memory allocator library and shim it into Firefox - this is called building a memory replacement library. It requires an additional build-time configuration option.
Our first investigation was with PartitionAlloc being built as a memory replacement library. This mostly worked, except we could not replicate the implementation of memalign, which required editing the core internals of PartitionAlloc. It's also important to note that our use of PartitionAlloc did not actually partition the allocations based on any intelligent design. This is an ongoing problem.
Currently, when Firefox performs an allocation, it will snake its way through the layers of abstraction until it arrives at our replacement library. The only information that has been passed into our replacement library is the size of the allocation. We do not have any context about what the allocation is for, and therefore which partition we would like to place it in. (We theorized about walking the call stack in assembly and binning allocation based on that but this was an untested idea.)
The work with PartitionAlloc paused.
After some time, Firefox began investigating partitioning allocations using jemalloc3's partitioning capabilities. jemalloc3 is a new version of the currently used allocator, jemalloc. Mozilla's migration process is first to move to jemalloc3, and then to start partitioning allocations. Currently, their move to jemalloc3 is paused because <actually I'm not entirely sure why. I think "performance" and "mobile".>
However the code for jemalloc3 is present, and making use of it is only a configuration change away. jemalloc3 solves one of the problems of PartitionAlloc - the allocator actually, completely works. Mostly. There is at least one issue which I believe is <an outstanding bug in jemalloc causing mismatched VirtualAlloc/Free calls - but this may be fixed now>. We did some performance comparisons of jemalloc vs jemalloc3 and it was reasonable: http://imgur.com/a/A6TV9
To actually partition allocations we can implement a replace library that makes use of jemalloc3's partitioning capabilities. But we run into the same problem as before - we don't have any context to know what partition to put an allocation into.
One thought was that we could use memory tags, but after investigation, that did not seem feasible. We _can_ go through the Firefox codebase and change the allocation call to include an arena constant, but this patch will be difficult to maintain, we are likely to miss some allocation situations, and eventually Mozilla will do this themselves.
References: https://lists.torproject.org/pipermail/tbb-dev/2015-May/000269.html was my most-recent message on this topic. It references a github issue that is now closed. The patch referenced below should not be needed if this patch solves Mozilla's issue. Does it? Was jemalloc's fix merged? Questions I don't know the answer to.
https://lists.torproject.org/pipermail/tbb-dev/2015-May/000272.html Was my followup.
Future Directions and Open Questions: - Do we have a clear picture on why mozilla isn't shipping jemalloc3? - Is the jemalloc3 code in ESR38 considered bug-free, or did the known github issue not get patched in it? - Is there concern over switching TBB's allocator over to jemalloc3? - Is there concern over enabling the replace-malloc switch in TBB? - If both of those are done, any TBB distribution can, at start-time, choose to use a randomizing allocator in the form of a replacement library. The beginnings of one is attached. The library does not need to be used by default, but can be included in every install.
Hi,
thanks for the great write-up! I looked a bit around and asked people to fill at least some of the gaps/answer some of your questions.
Tom Ritter:
Firefox has a feature where you can build a memory allocator library and shim it into Firefox - this is called building a memory replacement library. It requires an additional build-time configuration option.
Our first investigation was with PartitionAlloc being built as a memory replacement library. This mostly worked, except we could not replicate the implementation of memalign, which required editing the core internals of PartitionAlloc. It's also important to note that our use of PartitionAlloc did not actually partition the allocations based on any intelligent design. This is an ongoing problem.
Currently, when Firefox performs an allocation, it will snake its way through the layers of abstraction until it arrives at our replacement library. The only information that has been passed into our replacement library is the size of the allocation. We do not have any context about what the allocation is for, and therefore which partition we would like to place it in. (We theorized about walking the call stack in assembly and binning allocation based on that but this was an untested idea.)
The work with PartitionAlloc paused.
After some time, Firefox began investigating partitioning allocations using jemalloc3's partitioning capabilities. jemalloc3 is a new version of the currently used allocator, jemalloc. Mozilla's migration process is first to move to jemalloc3, and then to start partitioning allocations. Currently, their move to jemalloc3 is paused because <actually I'm not entirely sure why. I think "performance" and "mobile".>
https://bugzilla.mozilla.org/show_bug.cgi?id=1201802 is the tracking bug and looking at the child tickets, yes, performance is one issue.
However the code for jemalloc3 is present, and making use of it is only a configuration change away. jemalloc3 solves one of the problems of PartitionAlloc - the allocator actually, completely works. Mostly. There is at least one issue which I believe is <an outstanding bug in jemalloc causing mismatched VirtualAlloc/Free calls - but this may be fixed now>. We did some performance comparisons of jemalloc vs
Yes, it is and it is on mozilla-central and on mozilla-aurora, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1201738
jemalloc3 and it was reasonable: http://imgur.com/a/A6TV9
To actually partition allocations we can implement a replace library that makes use of jemalloc3's partitioning capabilities. But we run into the same problem as before - we don't have any context to know what partition to put an allocation into.
One thought was that we could use memory tags, but after investigation, that did not seem feasible. We _can_ go through the Firefox codebase and change the allocation call to include an arena constant, but this patch will be difficult to maintain, we are likely to miss some allocation situations, and eventually Mozilla will do this themselves.
References: https://lists.torproject.org/pipermail/tbb-dev/2015-May/000269.html was my most-recent message on this topic. It references a github issue that is now closed. The patch referenced below should not be needed if this patch solves Mozilla's issue. Does it? Was jemalloc's fix merged? Questions I don't know the answer to.
I think so, yes, the fix is in Mozilla's jemalloc.
https://lists.torproject.org/pipermail/tbb-dev/2015-May/000272.html Was my followup.
Future Directions and Open Questions:
- Do we have a clear picture on why mozilla isn't shipping jemalloc3?
- Is the jemalloc3 code in ESR38 considered bug-free, or did the known
github issue not get patched in it?
The Github issue (#213) is not fixed in ESR 38.
- Is there concern over switching TBB's allocator over to jemalloc3?
- Is there concern over enabling the replace-malloc switch in TBB?
- If both of those are done, any TBB distribution can, at start-time,
choose to use a randomizing allocator in the form of a replacement library. The beginnings of one is attached. The library does not need to be used by default, but can be included in every install.
It depends on what you mean with "concern" I guess. Doing this is very likely "nightly territory" first but from my point of view I'd start rather sooner than later playing with it and assume it ould be a good thing to try out in our hardened builds first (after it surviced soe nightly testing).
Georg
-
Georg Koppen -
Tom Ritter