Author Message

rudregues









Joined: 29 Jan 2013

Posts: 227

Location: Brazil ApprenticeJoined: 29 Jan 2013Posts: 227Location: Brazil

Posted: Sat Mar 10, 2018 4:50 pm Post subject: [TIP] Calculate your world compile time



1. Get a list of all installed packages without the version at the end.

2. Redirect this to show building times in seconds.

3. Print only building times.

4. Sum all lines of building times.

5. Redirect the sum to a file.

6. Evalute the sum from the file.

7. Print the evaluated sum in HH:MM:SS format.

8. Delete the file to keep the system clean



I was struggling very hard to find an utility that calculates the total time I'd compiled all my packages, so I developed a command. Here are the explanation:1. Get a list of all installed packages without the version at the end.2. Redirect this to show building times in seconds.3. Print only building times.4. Sum all lines of building times.5. Redirect the sum to a file.6. Evalute the sum from the file.7. Print the evaluated sum in HH:MM:SS format.8. Delete the file to keep the system clean Code: qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds

' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime



Mine is 15h:51m:28s right now.





EDIT: steveL did a big enhancement over my command, eliminating steps 3, 5, 6 and 8:

Code: qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'





EDIT2: Mitigating/troubleshooting the outlier problem. People with absurd build times can use two commands to adapt the situation.



1) Calculating total sum excluding outliers defined by the user. I want everything bigger than 5400 seconds (1h30min) to be removed from calculation: Code: qlist -I | xargs qlop -t | awk '{ if ($2 < 5400) secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'

2) Discovering the problematic build times (maybe report a bug?). I want to point out which packages build times are bigger than 5400 seconds (1h30min): Code: qlist -I | xargs qlop -t | awk '{ if ($2 > 5400) printf("%s %dd:%dh:%dm:%ds

", $1, $2 / 86400, ($2 % 86400) / 3600, ($2 % 3600) / 60, $2 % 60); }'

_________________

Emerging en gentoo



Last edited by rudregues on Tue Apr 10, 2018 5:20 pm; edited 5 times in total

albright









Joined: 16 Nov 2003

Posts: 2588

Location: Near Toronto AdvocateJoined: 16 Nov 2003Posts: 2588Location: Near Toronto

Posted: Sun Mar 11, 2018 12:56 am Post subject: OK, that is fun



my desktop i7-6700: 16h:36m:10s





my thinkpad x1 carbon (4th gen / i7-6600U): 21h:42m:15s



very similar installations (but distcc probably skews result)

_________________

.... there is nothing - absolutely nothing - half so much worth

doing as simply messing about with Linux ...

(apologies to Kenneth Graeme)

fedeliallalinea









Joined: 08 Mar 2003

Posts: 24503

Location: here BodhisattvaJoined: 08 Mar 2003Posts: 24503Location: here

Posted: Sun Mar 11, 2018 8:22 am Post subject: You can skip to write in a file

Code: qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' | eval && printf '%dh:%dm:%ds

' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60))

_________________

Questions are guaranteed in life; Answers aren't.

steveL









Joined: 13 Sep 2006

Posts: 5153

Location: The Peanut Gallery WatchmanJoined: 13 Sep 2006Posts: 5153Location: The Peanut Gallery

Posted: Sun Mar 11, 2018 11:18 am Post subject: Re: [TIP] Calculate your world compile time Luke.. use the awk, Luke ;) Code: qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }' (untested, as not in gentoo atm.)

Juippisi









Joined: 30 Sep 2005

Posts: 480

Location: /home DeveloperJoined: 30 Sep 2005Posts: 480Location: /home

Posted: Sun Mar 11, 2018 5:30 pm Post subject: Is this one of those cases where "I copied a bash line from the internet and now my / is empty"?



Just kidding,

Code:

# qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds

' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime

11h:50m:56s





Sure felt a lot longer when I did -e world a while back. i7-2700k with '-march=native -O3 -fgraphite-identity -ftree-loop-distribution -floop-nest-optimize -flto=8 -fuse-linker-plugin -pipe -fstack-protector-all -mindirect-branch=thunk'

Fitzcarraldo









Joined: 30 Aug 2008

Posts: 1872

Location: United Kingdom VeteranJoined: 30 Aug 2008Posts: 1872Location: United Kingdom

Posted: Sun Mar 11, 2018 6:59 pm Post subject: It's a rather academic exercise, though, isn't it? It doesn't take into account one's time spent investigating between failed merges due to blockers and bugs in packages and/or ebuilds. In many cases I have had to spend hours trying to fix a problem and get a package to merge, and its build time may be a few minutes. Over time I have spent more time debugging than the actual time it took to build the packages.



Code: clevow230ss /home/fitzcarraldo # qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds

' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime

40h:18m:44s

_________________

Clevo W230SS: amd64 nvidia-drivers & xf86-video-intel.

Compal NBLB2: ~amd64 xf86-video-ati. Dual boot Win 7 Pro 64-bit.

OpenRC eudev elogind & KDE on both.

Fitzcarraldo's blog _________________

Hu









Joined: 06 Mar 2007

Posts: 15832

ModeratorJoined: 06 Mar 2007Posts: 15832

Posted: Sun Mar 11, 2018 7:10 pm Post subject: Juippisi wrote: Is this one of those cases where "I copied a bash line from the internet and now my / is empty"? Any time you see an eval in the program, that's a good question to ask. Although sometimes useful, incautious use of eval is extremely dangerous.

Ant P.









Joined: 18 Apr 2009

Posts: 6690

WatchmanJoined: 18 Apr 2009Posts: 6690

Posted: Sun Mar 11, 2018 7:34 pm Post subject: Code: ~ $ qlist -I | xargs qlop -t | cut -f2 -d" " | awk '{s+=$1} END {print "secs="s}' > /tmp/btime && eval $(cat /tmp/btime) && printf '%dh:%dm:%ds

' $(($secs/3600)) $(($secs%3600/60)) $(($secs%60)) && rm /tmp/btime

5055h:12m:42s





My output's a bit skewed by this one entry...

Code: openscenegraph: 18068512 seconds average for 1 merges



Probably related to Probably related to this bug in qlop

Zucca









Joined: 14 Jun 2007

Posts: 1803

Location: KUUSANKOSKI, Finland VeteranJoined: 14 Jun 2007Posts: 1803Location: KUUSANKOSKI, Finland

Posted: Sun Mar 11, 2018 8:26 pm Post subject: Re: [TIP] Calculate your world compile time steveL wrote: Luke.. use the awk, Luke ;) Code: qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }' (untested, as not in gentoo atm.) Works. Returns the exact same time.



My time is 76h:19m:7s ... FX-8350 shows its age.

_________________

..: Zucca :..



Code: ERROR: '--failure' is not an option. Aborting...

szatox









Joined: 27 Aug 2013

Posts: 1916

VeteranJoined: 27 Aug 2013Posts: 1916

Posted: Sun Mar 11, 2018 9:36 pm Post subject: Quote: 103h:16m:18s

My poor hamster powered pc

I bought this box nearly a decade (sic!) ago and it was dirt cheap already.... And I don't really feel like replacing it any time soon. That APU A8-3870 works well enough. My poor hamster powered pcI bought this box nearly a decade (sic!) ago and it was dirt cheap already.... And I don't really feel like replacing it any time soon. That APU A8-3870 works well enough.

Goverp









Joined: 07 Mar 2007

Posts: 893

l33tJoined: 07 Mar 2007Posts: 893

Posted: Mon Mar 12, 2018 10:08 am Post subject: What would be more useful is similar arithmetic in emerge --ask to estimate the time for the command in question. Then this script becomes essentially emerge -ea world .

_________________

Greybeard

szatox









Joined: 27 Aug 2013

Posts: 1916

VeteranJoined: 27 Aug 2013Posts: 1916

Posted: Mon Mar 12, 2018 8:31 pm Post subject: Goverp, this script completed in 20 seconds, which is significantly faster than 4 days emerge -aev @world would take.

Also, each of us has different @world, different set of USE flags and different hardware which means we simply _cant_ compare results anyway. There is no way to get any meaningful data from it besides "how badly screwed I am when my hard drive dies". And the answer is "you better verify your backups"

Goverp









Joined: 07 Mar 2007

Posts: 893

l33tJoined: 07 Mar 2007Posts: 893

Posted: Tue Mar 13, 2018 10:40 am Post subject: szatox, I'm not sure what you think I was suggesting, but nothing that would take 4 days. My point was that emerge could do the same arithmetic, and tell you how long the command would take if you say "Yes", thereby at least telling you whether to take a coffee, go out to the cinema, or take a small vacation while portage recompiled whatever.

_________________

Greybeard

Zucca









Joined: 14 Jun 2007

Posts: 1803

Location: KUUSANKOSKI, Finland VeteranJoined: 14 Jun 2007Posts: 1803Location: KUUSANKOSKI, Finland

Posted: Tue Mar 13, 2018 11:56 am Post subject: That would be useful... Also the estimations for each package in the listing, since by calcutaling total time the time for each packge is already known and wouldne add much overhead.



This can be done wia processing --pretend output, but wastes time.

_________________

..: Zucca :..



Code: ERROR: '--failure' is not an option. Aborting...

tuggbuss









Joined: 20 Mar 2017

Posts: 222

ApprenticeJoined: 20 Mar 2017Posts: 222

Posted: Fri Mar 30, 2018 9:28 am Post subject: Code: 3h:35m:50s



i7 - 6900K

64GB RAM

NVMe

Nvidia blob

Xfce4

rudregues









Joined: 29 Jan 2013

Posts: 227

Location: Brazil ApprenticeJoined: 29 Jan 2013Posts: 227Location: Brazil

Posted: Fri Apr 06, 2018 9:17 pm Post subject: tuggbuss wrote: Code: 3h:35m:50s



i7 - 6900K

64GB RAM

NVMe

Nvidia blob

Xfce4 This is humiliation...

_________________

Emerging en gentoo

krinn









Joined: 02 May 2003

Posts: 7447

WatchmanJoined: 02 May 2003Posts: 7447

Posted: Sat Apr 07, 2018 6:30 am Post subject: Ant P. wrote: My output's a bit skewed by this one entry...

Code: openscenegraph: 18068512 seconds average for 1 merges

Can't beat me

Can't beat me Code: qlop -vgH fuse

fuse-2.7.4: Sat Sep 19 00:01:48 2009: 1158050441 days, 5 hours, 1 minute, 10 seconds

fuse-2.7.4: Sat Sep 19 03:21:32 2009: 22 seconds

fuse-2.8.1: Wed Nov 11 00:44:36 2009: 18 seconds

fuse-2.8.1: Wed Jun 2 23:19:18 2010: 14 seconds

fuse-2.8.1: Wed Sep 15 21:20:45 2010: 34 seconds

fuse-2.8.5: Mon Oct 11 05:01:42 2010: 23 seconds

fuse-2.8.6: Wed Sep 28 02:18:14 2011: 18 seconds

fuse-2.8.6: Mon Dec 5 17:28:47 2011: 23 seconds

fuse-2.8.7: Mon Apr 16 02:45:26 2012: 38 seconds

fuse-2.9.2: Sun Dec 2 17:55:34 2012: 58 seconds

fuse-2.9.3: Sun Dec 22 10:38:25 2013: 15 seconds

fuse-2.9.5: Thu Apr 14 12:07:27 2016: 16 seconds

fuse-2.9.7: Sun Jan 29 21:19:25 2017: 25 seconds

fuse-2.9.7-r1: Sun Mar 25 03:44:15 2018: 9 seconds

fuse-2.9.7-r1: Sun Mar 25 08:28:48 2018: 14 seconds

fuse: 15 times

Irre









Joined: 09 Nov 2013

Posts: 354

Location: Stockholm GuruJoined: 09 Nov 2013Posts: 354Location: Stockholm

Posted: Sat Apr 07, 2018 8:17 pm Post subject: Code: qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'

502h:0m:44s

Linux localhost 4.12.10-gentoo #2 SMP Thu Aug 31 09:20:09 GMT 2017 x86_64 Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz GenuineIntel GNU/Linux

Something is wrong!



[Moderator edit: added [c ode] tags to preserve output layout. -Hu] 502h:0m:44sLinux localhost 4.12.10-gentoo #2 SMP Thu Aug 31 09:20:09 GMT 2017 x86_64 Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz GenuineIntel GNU/LinuxSomething is wrong!

josephg









Joined: 10 Jan 2016

Posts: 783

Location: usually offline l33tJoined: 10 Jan 2016Posts: 783Location: usually offline

Posted: Sat Apr 07, 2018 9:41 pm Post subject: and i run a tight ship too!

think i just topped everyoneand i run a tight ship too! Code: $ qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'

777h:50m:29s

$ uname -a

Linux localhost 4.9.76-gentoo-r1-josephg #7 SMP Sun Feb 18 20:38:56 GMT 2018 i686 Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz GenuineIntel GNU/Linux

and this is my fastest machine sigh i don't keep up with the rest of the world

atleast it's faster runtime than my neighbours' shiny new windoze laptops

_________________

"Growth for the sake of growth is the ideology of the cancer cell." Edward Abbey and this is my fastest machinesigh i don't keep up with the rest of the worldatleast it's faster runtime than my neighbours' shiny new windoze laptops_________________

krinn









Joined: 02 May 2003

Posts: 7447

WatchmanJoined: 02 May 2003Posts: 7447

Posted: Sat Apr 07, 2018 10:06 pm Post subject:

i didn't first output my result because it is just bork, but here it is to make everyone a few more happy with their "tiny" 500-700hours to build Code: qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'

341606371735829h:20m:0s

Linux beleg 4.14.11 #2 SMP PREEMPT Wed Jan 10 15:11:16 CET 2018 x86_64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz GenuineIntel GNU/Linux



khayyam









Joined: 07 Jun 2012

Posts: 6227

Location: Room 101 WatchmanJoined: 07 Jun 2012Posts: 6227Location: Room 101

Posted: Sat Apr 07, 2018 10:47 pm Post subject: krinn wrote: Code: qlop -vgH fuse

fuse-2.7.4: Sat Sep 19 00:01:48 2009: 1158050441 days, 5 hours, 1 minute, 10 seconds

krinn ... hehe, just over 3 million years, that makes you gentoo's oldest user, hows that beard of yours? :)



Code: % printf "%'.f

" $((1158050441 / 364))

3,181,457

... well, if you don't calculate leap years.



best ... khay

Tony0945









Joined: 25 Jul 2006

Posts: 4014

Location: Illinois, USA AdvocateJoined: 25 Jul 2006Posts: 4014Location: Illinois, USA

Posted: Sat Apr 07, 2018 11:03 pm Post subject: Athlon X4 (bulldozer) Not bad for a $50 cpu Code: qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'

2h:17m:38s



EDIT: Can't believe Kaveri is this fast. Code: # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'

0h:0m:0s



rudregues









Joined: 29 Jan 2013

Posts: 227

Location: Brazil ApprenticeJoined: 29 Jan 2013Posts: 227Location: Brazil

Posted: Mon Apr 09, 2018 9:59 pm Post subject: I edited my first post in order to help people with huge build times.



Don't know though what to do with Tony0945's Kaveri...

_________________

Emerging en gentoo

<3









Joined: 21 Oct 2004

Posts: 950

l33tJoined: 21 Oct 2004Posts: 950

Posted: Tue Apr 10, 2018 12:04 am Post subject: can we make an ebuild of this to put in portage or add this to gentoolkit?

ali3nx









Joined: 21 Sep 2003

Posts: 676

Location: Winnipeg, Canada l33tJoined: 21 Sep 2003Posts: 676Location: Winnipeg, Canada

Posted: Tue Apr 10, 2018 2:29 am Post subject: dual cpu 10 core hyperthreaded xeon



Total: 1153 packages (1153 reinstalls), Size of downloads: 0 KiB



Total: 1153 packages (1153 reinstalls), Size of downloads: 0 KiB Code: fenrir ~ # qlist -I | xargs qlop -t | awk '{secs += $2} END { printf("%dh:%dm:%ds

", secs / 3600, (secs % 3600) / 60, secs % 60); }'

9h:46m:11s

fenrir ~ # uname -a

Linux fenrir 4.16.0-gentoo #3 SMP Tue Apr 3 03:50:21 CDT 2018 x86_64 Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz GenuineIntel GNU/Linux



_________________

Compiling Gentoo since version 1.4

Thousands of Gentoo Installs Completed

Emerged on every continent but Antarctica

Compile long and Prosper!