English 日本語 Results of the public multiformat listening test (July 2014) These are the summary results of the multiformat listening test. The purpose was to test Opus, AAC and Ogg Vorbis at 96 kbps against a classic MP3 128 kbps, and find out which codec produces the best sound quality.

Summary

The sound quality of Opus, AAC, Ogg Vorbis and MP3 was evaluated by the public blind listening test. As a standard procedure of the ABC/HR methodology, the lossy audio files were paired with lossless original before the test. Thirty eight donators from public rated their perceived fidelity of those lossy files. Thirty five diverse music tracks and five speech tracks were used as test tracks.

The Opus encoder 1.1, the AAC encoder of iTunes, and the aoTuV at 96 kbps and the LAME at 128 kbps were tested. The most common response was on or between "Imperceptible" and "Perceptible, but not annoying".

The Opus codec was the clear winner, and the AAC encoded by Apple iTunes won the second place. The Ogg Vorbis was the third. The MP3, which was allowed to use 30kbps more bitrate than those contenders, was tied to Ogg Vorbis in joint third place.

FAAC (libfaac), which was known to be inferior before the test, was used as low quality anchors.

Contents

How to interpret the charts

Each chart is drawn with 6 codecs on the X axis and the rating given (1.0 to 5.0) on the Y axis. The mean rating given to each codec is indicated by the middle point of each vertical I-shaped line segment. Each I-shaped segment represents the 95% confidence interval of the mean rating (using bootstrap analysis) for each codec.

Ratings of all tracks (40 tracks)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible 95% Confidence Interval 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 Opus AAC Ogg MP3

All sets of tracks (5 sets, each 8 tracks)

Ratings of sets of tracks

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps C C C C C C T T T T T T M M M M M M R R R R R R A A A A A A Opus AAC Ogg MP3 FAAC 96k FAAC q30

Set name Opus

1.1 AAC

Apple Ogg

Vorbis MP3

LAME FAAC

96k FAAC

q30 Set name Opus

1.1 AAC

Apple Ogg

Vorbis MP3

LAME FAAC

96k FAAC

q30 Chris's set of tracks 4.66 4.26 4.22 4.22 2.62 1.15 Tracks randomly picked from album libraries 4.70 4.42 4.41 4.29 2.82 1.22 Mpeg Sequences 4.75 4.48 4.03 4.24 2.76 1.17 Tracks picked from previous 2011 AAC test 4.60 4.39 4.25 4.28 2.48 1.21 Tracks picked from new HA thread 103989 4.56 4.45 4.28 4.16 2.58 1.23

Ratings of each track

1 1 2 2 3 3 4 4 5 5 Chris's set of tracks SinceAlways Guitar Waiting Pops velvet Perc. trumpet Wind girl Techno Can't Wait Until Tonight (Dry Wurlitzer Mix) Pops 35_SQAM_glockenspiel_cut Inst. Robots_old Techno Tracks randomly picked from album libraries Asleep__4.11-4.31_ Strings Greatest_Love_of_All_2min57 Pops Hey Tonight Rock Severance__1.31-1.51_ Pops Shinsho_pool_3min45_4min4 Pops SlavesOfFear Metal The Chastising of Renegade Rock TrosYGareg Pops Mpeg Sequences 4-Sound-English-male.441 Voice 9-Have-big-expensive-car.441 Pops 12-German-male-speech.441 Voice 15-Good-evening.441 Voice 21-classic.441 Classic 24-Greensleeves-Korean-male-speech.441 Voice 25-This-is-the-end.441 Voice 27-last-song-drums-and-trampets.441 Perc. Tracks picked from previous 2011 AAC test bonhemian_rhapsody Rock clapton_44k Guitar Coral Pops ExitMusic Vocal liberate Metal NewYorkCity Pops sandman Rock take_your_finger_from_my_hair Rock Tracks picked from new HA thread 103989 Changes Rock Girl_In_The_Fire__Sample_ Rock Hotel California Rock Jupiter, the Bringer of Jollity Classic Last_Of_The_Mohicanz__Sample_ Electro Only Time Newage Through The Fire And Flames Metal With Love (Outro) Techno Opus AAC Ogg MP3 FAAC 96k FAAC q30

Chris's set of tracks

SinceAlways

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Waiting

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

velvet

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

trumpet

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

girl

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Can't Wait Until Tonight (Dry Wurlitzer Mix)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

35_SQAM_glockenspiel_cut

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Robots_old

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Tracks randomly picked from album libraries

Asleep__4.11-4.31_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Greatest_Love_of_All_2min57

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Hey Tonight

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Severance__1.31-1.51_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Shinsho_pool_3min45_4min4

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

SlavesOfFear

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

The Chastising of Renegade

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

TrosYGareg

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Mpeg Sequences

4-Sound-English-male.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

9-Have-big-expensive-car.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

12-German-male-speech.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

15-Good-evening.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

21-classic.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

24-Greensleeves-Korean-male-speech.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

25-This-is-the-end.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

27-last-song-drums-and-trampets.441

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Tracks picked from previous 2011 AAC test

bonhemian_rhapsody

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

clapton_44k

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Coral

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

ExitMusic

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

liberate

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

NewYorkCity

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

sandman

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

take_your_finger_from_my_hair

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Tracks picked from new HA thread 103989

Changes

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Girl_In_The_Fire__Sample_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Hotel California

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Jupiter, the Bringer of Jollity

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Last_Of_The_Mohicanz__Sample_

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Only Time

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Through The Fire And Flames

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

With Love (Outro)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Donators, tracks and contributions (363 results, 339 valid results)

Ratings of each donator

1 1 2 2 3 3 4 4 5 5 kiavan92 (38) Alexxander (35) anonymous2 (35) win32vb (32) Kamedo2 (25) Steve Forte Rio (24) IgorC (21) Caspian (16) Gainless (13) Joungmin Lee (11) 2Bdecided (10) singlethread (10) Hex144 (8) anonymous9 (7) timuer (7) anonymous3 (5) anonymous5 (4) HS (4) naylor83 (4) Panther025 (4) anonymous1 (3) anonymous10 (3) psychoticmeow (3) TL (3) yourlord (3) anonymous7 (2) johnb (2) NeoteriX (2) anonymous11 (1) anonymous12 (1) anonymous4 (1) anonymous8 (1) DH (1) Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings of the top 12 donators

Ratings submitted by kiavan92 (38 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Alexxander (35 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by anonymous2 (35 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by win32vb (32 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Kamedo2 (25 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Steve Forte Rio (24 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by IgorC (21 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Caspian (16 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Gainless (13 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by Joungmin Lee (11 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by 2Bdecided (10 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Ratings submitted by singlethread (10 results)

1 Very annoying 2 Annoying 3 Slightly annoying 4 Perceptible, but not annoying 5 Impercep- tible Opus AAC Ogg MP3 FAAC 96k FAAC q30

Bitrates of tracks

In each codec, settings were calibrated to provide ~96kbps on a large variety of music on average, except for the MP3 and anchors.

Track Bitrate [kbps] 0 20 40 60 80 100 120 140 160 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps Opus AAC Ogg MP3 FAAC 96k FAAC q30

These are the file size based bitrates.

Sample Length Opus AAC Ogg MP3 FAAC 96 FAAC q30 -------------------------------------------------------------------------- 1 18.191 82 103 97 144 98 58 SinceAlways 2 20.155 111 106 107 137 98 54 Waiting 3 11.879 112 114 124 169 98 55 velvet 4 10.031 133 101 101 156 98 60 trumpet 5 15.957 122 101 134 155 98 51 girl 6 29.933 116 99 111 144 97 54 Can't Wait Until Tonight (Dry Wurlitzer Mix) 7 23.45 131 85 79 77 83 41 35_SQAM_glockenspiel_cut 8 30 100 108 105 128 98 44 Robots_old 9 20 103 109 116 148 98 48 Asleep__4.11-4.31_ 10 20.178 103 100 89 120 98 54 Greatest_Love_of_All_2min57 11 20 92 102 96 131 98 55 Hey Tonight 12 20 120 100 94 131 98 54 Severance__1.31-1.51_ 13 18.924 104 109 88 108 98 48 Shinsho_pool_3min45_4min4 14 22 90 100 107 131 98 56 SlavesOfFear 15 20 104 109 122 146 98 49 The Chastising of Renegade 16 22.304 124 99 96 127 98 55 TrosYGareg 17 9.028 96 95 99 121 98 46 4-Sound-English-male.441 18 9.707 119 104 113 154 99 48 9-Have-big-expensive-car.441 19 7.84 100 95 104 116 98 47 12-German-male-speech.441 20 8.566 94 99 100 129 98 52 15-Good-evening.441 21 9.656 112 105 89 133 98 52 21-classic.441 22 8.72 133 101 121 176 99 45 24-Greensleeves-Korean-male-speech.441 23 9.184 98 99 95 125 98 53 25-This-is-the-end.441 24 9.966 104 105 112 128 99 48 27-last-song-drums-and-trampets.441 25 28.505 101 106 94 120 98 53 bonhemian_rhapsody 26 14.986 104 107 122 145 98 53 clapton_44k 27 12.701 104 107 102 135 98 48 Coral 28 20.106 111 101 85 141 98 47 ExitMusic 29 5.007 97 116 143 160 98 54 liberate 30 29.849 93 103 102 131 98 55 NewYorkCity 31 30.002 97 99 106 134 98 49 sandman 32 22.092 114 109 133 138 98 60 take_your_finger_frin_my_head 33 29.932 113 106 119 137 98 51 Changes 34 13.742 121 111 108 126 98 53 Girl_In_The_Fire__Sample_ 35 30.762 106 110 112 143 98 52 Hotel California 36 29.234 105 103 89 123 98 56 Jupiter, the Bringer of Jollity 37 25.481 104 110 114 147 98 48 Last_Of_The_Mohicanz__Sample_ 38 24.671 96 110 100 130 98 48 Only Time 39 29.789 97 99 117 135 98 59 Through The Fire And Flames 40 16.403 103 103 99 144 98 56 With Love (Outro) -------------------------------------------------------------------------- Mean 18.973 106.7 103.7 106.1 135.5 97.6 51.7 Unit second kbps kbps kbps kbps kbps kbps Sample Length Opus AAC Ogg MP3 FAAC 96 FAAC q30

Bitrate vs ratings 2D scatter plot

Opus 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps] Ratings AAC 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps] Ogg 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps] Ratings MP3 3.2 3.4 3.6 3.8 4.0 4.2 4.4 4.6 4.8 5.0 64 80 96 112 128 144 160 176 192 bitrate [kbps]

Listen and download tracks

The Links on the left side are the lossless original tracks, and the right side are the lossy tracks.

Post screening

Invalid results were discarded according to the following criteria, which had been made public prior to the test:

If a listener ranked the low anchor at 5.0 on a sample, the listener's results for that sample were discarded.

If a listener ranked the mid-low anchor at 5.0 on a sample, the listener's results for that sample were discarded.

If a listener ranked the low anchor higher than the mid-low anchor on a sample, the listener's results for that sample were discarded.

If a listener ranked the reference worse than 4.5 on a sample, the listener's results for that sample were discarded.

If a listener ranked the reference worse than 5.0 on 25% or more of submitted results, all of that listener's results were discarded.

If a listener submitted 25% or more invalid results, all of the listener's results were discarded.

The discarded results were less than 7% of all results.

Bootstrap analysis

bootstrap.py --compare-all -p 1000000 -s 1000000 aq23.txt bootstrap.py v1.0 2011-02-03 Copyright (C) 2011 Gian-Carlo Pascutto License Affero GPL version 3 or later <http://www.gnu.org/licenses/agpl.html> Reading from: C:\Users\PCC\Downloads\friedman124\aq23.txt Read 6 treatments, 40 samples => 15 comparisons Means: AppleAAC Opus Ogg MP3V5 FAAC96k FAACq30 4.400 4.653 4.235 4.237 2.652 1.194 Unadjusted p-values: Opus Ogg MP3V5 FAAC96k FAACq30 AppleAAC 0.000* 0.023* 0.016* 0.000* 0.000* Opus - 0.000* 0.000* 0.000* 0.000* Ogg - - 0.981 0.000* 0.000* MP3V5 - - - 0.000* 0.000* FAAC96k - - - - 0.000* Opus is better than AppleAAC (p=0.000) Ogg is worse than AppleAAC (p=0.023) Ogg is worse than Opus (p=0.000) MP3V5 is worse than AppleAAC (p=0.016) MP3V5 is worse than Opus (p=0.000) FAAC96k is worse than AppleAAC (p=0.000) FAAC96k is worse than Opus (p=0.000) FAAC96k is worse than Ogg (p=0.000) FAAC96k is worse than MP3V5 (p=0.000) FAACq30 is worse than AppleAAC (p=0.000) FAACq30 is worse than Opus (p=0.000) FAACq30 is worse than Ogg (p=0.000) FAACq30 is worse than MP3V5 (p=0.000) FAACq30 is worse than FAAC96k (p=0.000) p-values adjusted for multiple comparison: Opus Ogg MP3V5 FAAC96k FAACq30 AppleAAC 0.000* 0.044* 0.043* 0.000* 0.000* Opus - 0.000* 0.000* 0.000* 0.000* Ogg - - 0.981 0.000* 0.000* MP3V5 - - - 0.000* 0.000* FAAC96k - - - - 0.000* Opus is better than AppleAAC (p=0.000) Ogg is worse than AppleAAC (p=0.044) Ogg is worse than Opus (p=0.000) MP3V5 is worse than AppleAAC (p=0.043) MP3V5 is worse than Opus (p=0.000) FAAC96k is worse than AppleAAC (p=0.000) FAAC96k is worse than Opus (p=0.000) FAAC96k is worse than Ogg (p=0.000) FAAC96k is worse than MP3V5 (p=0.000) FAACq30 is worse than AppleAAC (p=0.000) FAACq30 is worse than Opus (p=0.000) FAACq30 is worse than Ogg (p=0.000) FAACq30 is worse than MP3V5 (p=0.000) FAACq30 is worse than FAAC96k (p=0.000)

ANOVA analysis

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/ Blocked ANOVA analysis Number of listeners: 40 Critical significance: 0.05 Significance of data: 0.00E+000 (highly significant) --------------------------------------------------------------- ANOVA Table for Randomized Block Designs Using Ratings Source of Degrees Sum of Mean variation of Freedom squares Square F p Total 239 385.20 Testers (blocks) 39 4.52 Codecs eval'd 5 369.44 73.89 1282.39 0.00E+000 Error 195 11.24 0.06 --------------------------------------------------------------- Fisher's protected LSD for ANOVA: 0.106 Means: Opus AppleAAC MP3V5 Ogg FAAC96k FAACq30 4.65 4.40 4.24 4.24 2.65 1.19 ---------------------------- p-value Matrix --------------------------- AppleAAC MP3V5 Ogg FAAC96k FAACq30 Opus 0.000* 0.000* 0.000* 0.000* 0.000* AppleAAC 0.003* 0.002* 0.000* 0.000* MP3V5 0.976 0.000* 0.000* Ogg 0.000* 0.000* FAAC96k 0.000* ----------------------------------------------------------------------- Opus is better than AppleAAC, MP3V5, Ogg, FAAC96k, FAACq30 AppleAAC is better than MP3V5, Ogg, FAAC96k, FAACq30 MP3V5 is better than FAAC96k, FAACq30 Ogg is better than FAAC96k, FAACq30 FAAC96k is better than FAACq30

Formatted ratings

The ratings parsed for ANOVA analysis, Bootstrap analysis and graphmaker 5.

Opus AAC Ogg MP3 FAAC96k FAACq30 %feature 10 0.1.9-win32 iTunes 11, qaac 2.41 aoTuV b6.03 LAME 3.99.5 faac-1.28-mod from rarewares faac-1.28-mod from rarewares %feature 11 --bitrate 96 --cvbr 96 -q2.2 -V5 -b 96 -q 30 %feature 12 107kbps 104kbps 106kbps 136kbps 98kbps 52kbps %genre Chris's set of tracks 4.960 4.860 4.660 4.420 2.630 1.330 4.580 4.490 4.030 4.290 2.610 1.070 4.345 4.382 4.373 4.373 2.464 1.127 4.840 4.270 4.240 3.910 2.440 1.040 4.492 3.238 3.954 4.077 2.577 1.238 4.433 3.767 3.378 4.578 2.567 1.056 4.700 4.558 4.425 4.492 3.067 1.242 4.911 4.533 4.667 3.611 2.633 1.056 %genre Tracks randomly picked from album libraries 4.885 4.908 4.785 4.885 2.946 1.062 4.667 4.367 4.322 4.267 2.967 1.100 4.400 4.567 4.133 4.056 2.811 1.200 4.650 3.950 4.000 4.283 2.717 1.367 4.811 4.544 4.289 3.878 2.922 1.400 4.913 4.313 4.850 4.738 2.850 1.225 4.760 4.430 4.710 4.270 2.880 1.180 4.488 4.250 4.150 3.963 2.488 1.238 %genre Mpeg Sequences 4.970 4.820 3.770 4.470 3.010 1.220 4.711 4.044 4.278 4.167 2.589 1.122 4.820 4.800 3.590 4.090 2.710 1.190 4.314 4.514 4.243 4.186 2.486 1.186 4.686 4.571 3.957 3.943 2.886 1.429 5.000 4.317 4.233 4.400 2.917 1.000 4.888 4.413 4.200 4.563 2.700 1.225 4.571 4.343 3.971 4.057 2.771 1.000 %genre Tracks picked from previous 2011 AAC test 4.629 4.529 4.043 3.714 2.486 1.257 4.286 4.214 4.457 4.300 2.414 1.043 4.940 4.500 4.530 4.620 2.570 1.290 4.933 4.433 4.133 3.800 2.333 1.000 4.243 4.429 4.614 4.500 2.486 1.486 4.800 4.067 4.133 4.483 2.383 1.000 4.925 4.325 4.400 4.725 2.663 1.363 4.043 4.657 3.700 4.129 2.529 1.214 %genre Tracks picked from new HA thread 103989 4.363 4.213 4.425 4.250 2.538 1.188 4.363 4.500 4.125 4.025 2.938 1.100 4.814 4.586 4.129 4.457 2.729 1.229 4.483 4.417 3.800 4.100 2.733 1.300 4.678 4.211 4.667 4.322 2.267 1.178 4.514 4.686 4.200 4.186 2.529 1.114 4.650 4.475 4.675 3.913 2.375 1.513 4.650 4.500 4.175 3.988 2.488 1.175 %samples SinceAlways Guitar %samples Waiting Pops %samples velvet Perc. %samples trumpet Wind %samples girl Techno %samples Can't Wait Until Tonight (Dry Wurlitzer Mix) Pops %samples 35_SQAM_glockenspiel_cut Inst. %samples Robots_old Techno %samples Asleep__4.11-4.31_ Strings %samples Greatest_Love_of_All_2min57 Pops %samples Hey Tonight Rock %samples Severance__1.31-1.51_ Pops %samples Shinsho_pool_3min45_4min4 Pops %samples SlavesOfFear Metal %samples The Chastising of Renegade Rock %samples TrosYGareg Pops %samples 4-Sound-English-male.441 Voice %samples 9-Have-big-expensive-car.441 Pops %samples 12-German-male-speech.441 Voice %samples 15-Good-evening.441 Voice %samples 21-classic.441 Classic %samples 24-Greensleeves-Korean-male-speech.441 Voice %samples 25-This-is-the-end.441 Voice %samples 27-last-song-drums-and-trampets.441 Perc. %samples bonhemian_rhapsody Rock %samples clapton_44k Guitar %samples Coral Pops %samples ExitMusic Vocal %samples liberate Metal %samples NewYorkCity Pops %samples sandman Rock %samples take_your_finger_from_my_hair Rock %samples Changes Rock %samples Girl_In_The_Fire__Sample_ Rock %samples Hotel California Rock %samples Jupiter, the Bringer of Jollity Classic %samples Last_Of_The_Mohicanz__Sample_ Electro %samples Only Time Newage %samples Through The Fire And Flames Metal %samples With Love (Outro) Techno %link_path tracks/ .wav .opus1.1.b96k.opus .qaac.cvbr96k.mp4 .aotuv.q2.2.ogg .lame3.99.5.v5.mp3 .faac.abr96k.mp4 .faac.vbr30q.mp4

Download logs

You can download a ZIP file containing all logs. Encryption keys and parsed text included.

Contact

Kamedo2: Twitter@kamedo2