Yesterday, zmedico wrote about building multiple packages in parallel with Portage-2.2_rc2. In Gentoo Prefix, we had a sneak peak to this feature, so I have had some time to play with it on my dual-quad core box. Some timing results that you may like:

emerge -e system (excluding sys-devel/gcc)

As a baseline:

With --jobs=1 and MAKEOPTS=16, load-average=9:

real 77m54.290s

user 41m46.086s

sys 29m14.598s

Because I was skeptical of what --jobs could really do, I decided to start with small number of parallel jobs:

With --jobs=3, MAKEOPTS=16, load-average=9:

real 61m30.181s

user 42m23.398s

sys 32m32.009s

While that was running, I noticed a very significant amount of time where my cores were idle, thanks to the handy little xfce-extra/xfce4-cpugraph widget. So, I turned --jobs up again:

With --jobs=5, MAKEOPTS=16, load-average=9:

real 58m5.388s

user 42m35.721s

sys 34m46.950s

Meh, not much improvement there. Surprising, but I suspect that I may be reaching the limits of the parallelization (dependencies, etc).

With --jobs=10, MAKEOPTS=16, load-average=9:

real 58m9.824s

user 42m43.525s

sys 37m57.234s

(And actually, a quick visual scan showed load averages staying below 4. Only a few times did I see the average above 8 )

Relying solely on load-average to keep my system usable:

With --jobs=40, MAKEOPTS=40 load-average=15:

real 58m45.106s

user 43m15.129s

sys 40m47.949s

The highest load average I visually saw was 23. But my load average was mostly always greater than 4, so this means that my procs are obviously getting used more but I must have hit another bottleneck.

NOTES:

-emerge -pe system was preformed before each time trial to assume the depgraph was in cache.

-84 packages total

-no ccache/distcc running

Conclusion: 20 minutes? ~30% speedup. Wow. Good! Quite significant even. Assuming you have cores/procs to spare, go ahead and crank up those --jobs . It is nice to be able to make the ./configure step not be the bottleneck anymore. 😉 I will keep testing and see if I can get the time down even farther (although, unlikely based on the last time trial).

Test requests? Please leave a comment.

Share this: Twitter

Facebook

Like this: Like Loading... Related