Reducing memory footprint using jemalloc

what is by now known as the "Facebook malloc" is also released as a opens source product: jemalloc. It features efficient memory management for multi-trhead application and aggressive memory redemption. This last feature is of interest for us as the memory footprint of HEP event processing application my greatly vary from one event to the other. I tested jemalloc in reconstruction of high pile up events in cms (using release *CMSSW_5_0_0_pre7"). I used the version linked in cmsJERun which is included in the CMSSW distribution. The results show that in running on a multicore machine JEMalloc allows to efficiently amortize the cost of peak allocation by each job making the total RSS used at any moment much smaller then the sum of the peak memory usage.

Single process

The following plot shows the value of VSS and RSS measured after each event for various runs of the very same application (reconstruction of 250 events) reading either from EOS or from a local file and using either standard malloc or jemalloc.

the plot is obtained parsing the log file with

foreach log (*_250*.out)
grep "MemoryCheck: e" $log | awk '{print e $5 ", " $8 }' > `basename $log .out`.csv
end

and merging the file using pr -m -t -s,\  ... before importing into Exel.

VSS and RSS for 250 events:

VSS and RSS for 250 events (alternative take):

One can observe, firstly, the steps in both virtual and resident memory while reading from EOS (with xrootd) using standard malloc. These steps are absent when using jmalloc or simply reading from a local file. The other cleat feature is how jmalloc effectively redeem memory after each large event. VSS is not reduces though and remain at the same level of the one used by standard malloc when xrootd is avoided.

Multi process

The effect of a lower memory footprint is even more evident in a multi-process environment where, obviously, different processes will, at any given time, use a very different RSS depending of the event in hand. We shall therefore expect that using jemalloc the total RSS (and in case of shared memory PSS) to be much lower than using standard malloc.

The followiing two plots shows the evolution of the total PSS and RSS, sampled each 30 seconds, for four and eight children processing 500 and 1000 events respectively for standard malloc and jemalloc. Memory has been sampled using smem

touch $1
while(1)
~/w1/smem-0.9/smem -t |& tail -1 >> $1
sleep 30
end

tcat sample_je8.log | awk '{ print $5 ", " $6}' > sample_je8.csv
cat sample_std8.log | awk '{ print $5 ", " $6}' > sample_std8.csv
pr -m -t -s,\  sample_std8.csv sample_je8.csv > multi8_pss_rss.csv

Total PSS and RSS for 4 children 500 events sampled each 30 seconds:

Total PSS and RSS for 8 children 1000 events sampled each 30 seconds::

Particularly in the case of eight children we note how using jemalloc the event-by-event variations in memory footprint are well amortized reducing even the peak effective memory use below 8GB while using standard malloc the value of 10GB is rapidly reached and maintained.

long run 10K events

The plot below shows the memory evolution (RSS and VSS) for four jobs on the full file with 10K events from bottom to top (rss) I remind that we run at cern with kernel 2.6.18-274.12.1.el5

• reco only using jemalloc reading from local file
• same reading from EOS
• reco+alca+dqm using jemalloc reading from local file
• same using standard malloc

all the plots shows a structure most probably due to the event content. As ususal RSS from jemalloc follows very closely the real needs of memory of the jobs. its VSS instead grows quite fast, more than standard malloc, in particular for reco+alca+dqm Reading from EOS shows a similar RSS behavior than reading from local file, while VSS is systematically larger by about 100 MB.

While is still very possible that the grows of VSS with jmalloc is just an artifact of the OS, it could also be an indication of some sort of memory leak

Reco (jmalloc reading either from EOS or from local file) and reco+alca+dqm either jmalloc or standard:

-- VincenzoInnocente - 13-Dec-2011