[RULE] Fwd: [forum] Some hard numbers on XFree86 memory usage (and no, not ps/top)

[Vadim Plessky]

Hi Guys,

I forward this mail from XFree86  Forum mailing list.
Can someone with KDrive running test script below and provide numbers to the 
list and Ivan Todoroski (he is in cc:)?

Greetings,
Vadim

----------  Forwarded Message  ----------

Subject: [forum] Some hard numbers on XFree86 memory usage (and no, not 
ps/top)
Date: Saturday 19 April 2003 10:31
From: Ivan Todoroski <grnch_lists at gmx.net>
To: forum at xfree86.org

I was browsing through the archives yesterday, and saw a recent discussion
 about whether XFree86 is bloatware or not. The debate contained much heat
 and emotion, but very little substance, and the distinction was blurred
 somewhat between distribution bloat (which can be easily trimmed down for
 special purposes) and runtime memory bloat (which cannot so easily).

I'll concentrate on the memory usage here, and I'll try to dispell some
misconceptions in the hope of a more reasonable discussion. I really hope
this doesn't turn into another flamewar.

It is well known that the VSZ numbers reported by ps/top for the X server
process are too inflated, which causes much needless confusion, so I decided
 to dig around a bit.

It turns out that atleast on Linux one can determine pretty accurately the
physical RAM usage of a process by analyzing the information in the /proc
filesystem. I created a script for this, and it is attached at the end of
 this message along with description of its operation. For now, let's
 concentrate on the numbers.

$ procmem 451
   PID    VSZ DEVMAP LIBMAP  ALLOC    RSS  CMDLINE
   451  63296  27216   2952  33128  34844  /usr/bin/X11/X -dpi 100 +bs
-nolisten tcp

This is the state of the XFree86 server after running a KDE session for
several hours. Currently loaded programs are KDE itself along with a few
kicker applets, the Mozilla Browser with several tabs open, and Mozilla Mail
into which I'm typing this message.

First thing to realize is that VSZ is the size of the mapped virtual address
space for the process, and that address space != physical RAM.

The above numbers mean the X server process has mapped 63 megs virtual
 address space, of which 27 megs is mapped directly to devices (in this case
 mostly the video card), 3 megs in shared libraries and other files which
 might be mapped into process address space via shared mmap(), and 33 megs
 real allocated physical memory via anonymous mmap().

This equation always holds: VSZ = DEVMAP + LIBMAP + ALLOC


DEVMAP is the address space mapped to PCI I/O registers, AGP memory and
what-not. Reading/writing from those addresses does not load/store anything
 in RAM, it loads/stores directly into some device(s), so it does not take up
 any physical RAM, and shouldn't be counted towards the process' physical
 memory usage.

LIBMAP is more problematic. It is the size of the shared libraries loaded by
the process, but you need to be careful to whom you attribute this memory
because a shared library is, well, shared across multiple process.

In the case of XFree86 the only shared library that should be attributed is
the executable image of /usr/bin/X11/XFree86 itself (about 1.5 meg). Looking
at /proc/451/maps, the other libraries loaded are libc, libdl, etc which are
all used by other processes already, so they would occupy memory even if the
 X server were not loaded.

Now, if it were Mozilla, then the situation would be different. Mozilla loads
many shared libs which are only used by itself, so those should rightfully be
attributed to the Mozilla process. How much of LIBMAP should be attributed
 has to be decided on a per-process basis by looking at the file
 /proc/$pid/maps to see what shared libraries are loaded.

Another nitpick is that this LIBMAP address space does not directly occupy
RAM, since it's backed by physical files on disc. But during use the
 operating system always caches those libraries into memory, and if they are
 actively used by processes you can pretty much assume that they occupy RAM
 the whole time. The only difference opposed to normal allocated memory is
 that when memory gets low the cache can simply be flushed and reclaimed, no
 need to be stored in swap.

ALLOC is the allocated memory, i.e. the address space you can safely
 attribute to the process as real occupied memory. It doesn't strictly mean
 that all of it is used right now, but it does mean that the process has
 asked this much memory from the OS recently.

So, by this logic, the amount of physical memory currently used by XFree86 is
ALLOC + size_of_XFree86_image (1.5 megs), i.e. slightly above 34 megs.

Incidentaly, this is about the size of RSS (resident set size) above, because
I have plenty of RAM and no part of XFree86 is swapped out. If it were, then
RSS would be smaller.

So, there you have it... 34 megs after prolonged use.

Whether this is bloat or not I leave up to others to decide. I do have an
opinion about it, but I'll not state it here in order to keep this message
objective. Please note that I'm running with backing-store enabled (+bs),
 which also consumes some memory.

One can probably further argue that not all of this memory is selfishly used
by the X server, that much of it is allocated on behalf of X clients for
pixmaps and what-not, but exactly how much is allocated on behalf, and
 whether the memory is managed in the most efficient manner (sharing of some
 stuff between clients etc), I cannot tell since I know next to nothing about
 XFree86 internals.

OK, so to get a clearer picture of the pure overhead, let's start only the X
server on another display with no clients.


$ procmem 25347
   PID    VSZ DEVMAP LIBMAP  ALLOC    RSS  CMDLINE
25347  24708  17088   2840   4780   6292  X :1


Here we see the reason why people suffer mild traumas when they run 'ps' and
notice that the X server they just started has already "occupied" 25 megs,
when in reality it only has allocated 4.7 megs itself, plus 1.5 megs for the
executable image which comes around to about 6.2 megs, which is again equal
 to RSS because none of it is swapped.

So, it looks like 6.2 megs for just the graphical mesh with an X cursor on
 it. Of course, it's much more complex than that, again I leave up to you to
 decide whether this is bloat or not.

What I can say for certain is that 6.2 megs initial overhead is quite a
 hurdle for embedded device that want to run X.


As you can see from all the above, it is possible to get accurate memory
 usage even with 'ps' alone, provided you a) only look at the RSS numbers, b)
 ensure that no part of the process memory is swapped out (by temporarily
 switching off swap, for instance).

But for a more accurate picture of address space distribution, use the
'procmem' script.

I would be really interested if someone already running the Kdrive server on
Linux would run this script and report back here. It would be interesting to
see Kdrive's memory usage with only the X server started, and also after
prolonged usage with several heavy-weight X clients (KDE, GNOME, Mozilla,
OpenOffice, etc). I really don't feel like downloading and compiling XFree86
just to try this out.

Reports from non-XFree86 servers running on Linux would also be enlightening.


Another interesting thing is that the drivers and various modules loaded by
XFree86 don't show up in the /proc/451/maps file... but there are a few
anonymous mmap() blocks with the executable flag set, which I'm guessing is
probably where the code from these modules is kept during runtime.

It looks like XFree86 bypasses somehow the OS shared library loader, instead
it loads and links those modules manually into it's own allocated memory.
 This would unfortunately mean that they are copied into RAM for each X
 server process - sharing them between multiple X processes would not be
 possible, but on the other hand, running multiple X servers on the same
 machine is not a common scenario, so it's not a big deal. I could be wrong
 about this, of course.



======================================================
Here is the little script. For each process it looks into /proc/$pid/maps and
accumulates the sizes of mapped address ranges as follows:

If the last column starts with "/dev/" it assumes the range is mapped to some
device; if the last column is empty it assumes the range is an anonymous
mmap() (i.e. allocated memory); otherwise assumes the range is mapped to a
shared library.


Note that this script is not POSIX shell compatible, it depends on a few
features in bash 2.x.

/usr/local/bin/procmem
--------------------
#!/bin/bash

if [ "$1" = "--help" -o "$1" = "-h" ]; then
     echo "Syntax: `basename $0` [pid]..."
     echo "If no pids are specified, all processes are listed"
     exit 1
fi

cd /proc 2> /dev/null || { echo "No /proc directory."; exit 1; }


get_cmdline() {
     local cmd
     cmd="`tr '\000' ' ' < $1/cmdline`"
     test -z "$cmd" && cmd="[`tr -d '()' < $1/stat | cut -d' ' -f2`]"
     echo "$cmd"
}

get_rss() {
     grep "^VmRSS:" $1/status | awk '{ print $2 }'
}


# Determine the width of the stdout tty (if it is a tty) for clipping
width=`stty -F $$/fd/1 size 2> /dev/null | awk '{ print $2 }'`

clip_line() {
     test -n "$width" && expr substr "$1" 1 "$width" || echo "$1"
}


test -z "$1" && set -- `echo [0-9]* | tr " " "\n" | sort -n`

header=$(printf "%5s %6s %6s %6s %6s %6s  %s\n" \
PID VSZ DEVMAP LIBMAP ALLOC RSS CMDLINE)

clip_line "$header"

for pid; do
     totalmap=0
     devmap=0
     libmap=0
     alloc=0

     test -f $pid/maps || continue

     { while read -u 99 range _ _ _ _ lib; do
         size=$(( -(0x${range/-/-0x}) ))
         let totalmap+=size
         case $lib in
             /dev/*)
                 let devmap+=size
             ;;
             "")
                 let alloc+=size
             ;;
             *)
                 let libmap+=size
         esac
     done; } 99< $pid/maps  # '99<' is a workaround for a stupid bash bug

     output=$(printf "%5d %6d %6d %6d %6d %6d  %s\n" \
     $pid $((totalmap/1024)) $((devmap/1024)) $((libmap/1024)) \
     $((alloc/1024)) "`get_rss $pid`" "`get_cmdline $pid`")

     clip_line "$output"
done
--------------------


_______________________________________________
Forum mailing list
Forum at XFree86.Org
http://XFree86.Org/mailman/listinfo/forum

-------------------------------------------------------

-- 
Best Regards,

Vadim Plessky
SVG Icons * BlueSphere Icons 0.3.0 released
http://svgicons.sourceforge.net




_______________________________________________
Original home page of the RULE project: www.rule-project.org
Original Rule Development Site http://savannah.gnu.org/projects/rule/
Original RULE mailing list: Rule-list at nongnu.org, hosted at http://mail.nongnu.org/mailman/listinfo/rule-list