





Introduction

There are three popular Red Hat Enterprise Linux clones: CentOS, Scientific Linux and Oracle Linux. All of these clone projects download source RPM packages from Red Hat and re-compile them to produce their own distributions.

These Linux distributions are often installed on servers which are connected to the internet. In that task it is essential to take care of security bugs quickly to avoid a system compromise. All of these three clones issue their own security advisories and updates.

I decided to compare the contents of these advisories as well as the delay publishing the advisories. This comparison is for version 6 of these distributions.

I was partially inspired to do this comparison after I saw a pathetic graph on Oracle’s web site which tries to get people to convert their CentOS installations to Oracle Linux. The graph that Oracle uses there includes only kernel security bugs even though it can be equally important or more important to quickly fix security bugs in other system software, such as internet facing service daemons. (The page has been archived by WebCite at http://www.webcitation.org/6A44JQ7rI and the graph at http://www.webcitation.org/6A705qdDS.)

Advisory contents: Red Hat Enterprise Linux

Archives at: https://www.redhat.com/archives/enterprise-watch-list/

CVE identifiers: yes

Unique advisory identifier: yes

Link to the upstream advisory: N/A

Description of the issues: yes

PGP signature: yes

This is “the upstream” that everyone else is copying from. RHEL advisories are very well written and contain all the essential information about the fixed security problems. They are issued through some automated system which makes sure that all information is where it should be. The advisory contents are very consistent: they can be easily parsed with a program to extract the essential contents for further processing.

My observation here is that you get what you pay for (even though the advisories themselves are freely available): Red Hat’s security advisories are of very high quality.

Advisory contents: CentOS

Archives at: http://lists.centos.org/pipermail/centos-announce/

CVE identifiers: no

Unique advisory identifier: yes

Link to the upstream advisory: yes

Description of the issues: no

PGP signature: no

CentOS advisories are very terse. They only contain the advisory identifier, the severity and the name of the updated software on the subject line. The contents of the advisories consist of a link to the upstream advisory and a list of the updated RPM packages with SHA256 sums. In the past there has been inconsistencies and it is difficult to machine parse old advisories, but this has improved a lot over the years.

Last year (2011) there was a period of several months when the CentOS project did not issue any security advisories or updates for CentOS 6. Many CentOS users got frustrated and worried about their system security for obvious reasons, as evidenced by looking at various CentOS related mailing lists, forums and blogs at that time. Hopefully the project has solved their problems and this will not happen again.

Advisory contents: Oracle Linux

Archives at: https://oss.oracle.com/pipermail/el-errata/

CVE identifiers: yes (but not in the header)

Unique advisory identifier: yes

Link to the upstream advisory: yes

Description of the issues: very brief

PGP signature: no

Oracle Linux security advisories are very similar to CentOS advisories, except there is RPM changelog excerpt at the end. In the past there has been several inconsistencies which make machine parsing old advisories difficult. The CVE identifiers are in the changelog excerpt at the end of the advisories and therefore difficult to find.

Oracle Linux differentiates itself from the other two clones by providing an alternative kernel called “Oracle Unbreakable Enterprise Kernel”. Updates to this alternative kernel are not included in this comparison because it does not make sense to compare apples to oranges and the author fails to see any added value from using this special kernel.

Advisory contents: Scientific Linux

Archives at: http://listserv.fnal.gov/archives/scientific-linux-errata.html

CVE identifiers: yes

Unique advisory identifier: no

Link to the upstream advisory: no

Description of the issues: yes

PGP signature: no

Scientific Linux security advisories are more verbose than advisories of the other clones. They contain a good description of the fixed security issues. The advisories do not have any unique identifiers which makes it difficult to refer to any specific advisory. The advisories are mostly consistent (this has improved over the years), but it is difficult to match them to the corresponding upstream advisories.

It appears that Scientific Linux missed two RHEL advisories during the first half of 2012. At least they are not in the mailing list archives. The corresponding updated RPM packages did appear in the repositories though.

Update: One of these two missing advisories (RHSA-2012:1009) was about new software (java-1.7.0-openjdk) which was added in 6.3 release. Scientific Linux had not yet released version 6.3 at that time, thus no advisory was needed.

Security advisory publishing delays

The following graph displays the time (in days) it took for the different clones to publish a security advisory after the upstream issued an advisory (this includes all RHEL 6 advisories issued in the first half of 2012):

There are a couple of gaps for Scientific Linux. These security updates were made available in the repositories, but there were no advisories published about these updates. The reason for this is unknown to the author.

Update: One of these two missing advisories (RHSA-2012:1009) was about new software (java-1.7.0-openjdk) which was added in 6.3 release. Scientific Linux had not yet released version 6.3 at that time, thus no advisory was needed.

The following graph shows the average publishing delays:

The following graph shows how frequently each of the distributions was the fastest one to publish an advisory:

The following graph shows how frequently each of the distributions was the slowest one to publish an advisory:

Important and Critical security advisory publishing delays

All security bugs are not created equal. Some are more important than the others. Updates with “low” or “moderate” security impact can often wait until a later time (depending on the particular environment). Therefore it makes sense to have a second look and focus only on security issues that are marked “important” or “critical” by the upstream vendor.

The following graph displays the time (in days) it took for the different clones to issue a security advisory after the upstream issued an advisory with important or critical severity level:

The following graph shows the average publishing delays for important and critical advisories:

The following graph shows how frequently each of the distributions was the fastest one to publish an important or critical advisory:

The following graph shows how frequently each of the distributions was the slowest one to publish an important or critical advisory:

Critical security advisory publishing delays

One more look, this time focusing only on critical advisories.

The following graph displays the time (in days) it took for the different clones to issue a security advisory after the upstream issued an advisory with critical severity level:

The following graph shows the average publishing delays for critical advisories:

The following graph shows how frequently each of the distributions was the fastest one to publish a critical advisory:

The following graph shows how frequently each of the distributions was the slowest one to publish a critical advisory:

Conclusions

All three clones have some delays in publishing security updates and advisories because it takes time to re-package the source RPMs published by Red Hat.

The CentOS project managed to publish one security advisory before the upstream advisory was sent on the Red Hat mailing list (RHSA-2012:0451 vs. CESA-2012:0451). This is quite an accomplishment.

If fast security fixes are important to you, you should purchase a subscription for the original upstream product (Red Hat Enterprise Linux).

There are no huge differences in the delays between the different clones. Currently all of these distributions produce updates for important and critical issues quickly enough for most purposes.

Scientific Linux was generally slightly slower than CentOS to publish advisories, but this could be simply a result of the fact that their advisories actually have a lot of content which needs to be edited. CentOS advisories are quick to produce as they do not contain any problem descriptions and only have a link to the corresponding Red Hat advisory.

Update: One reader pointed out: “Delays in issuing bug fixes are good for you – no delay means they did not test the stuff before pushing it out.” I completely agree with this. Publishing security updates is a balancing act between the need to get things out quickly and on the other hand making sure that the fixed version actually works. However the real testing of the backported patches has been already done by Red Hat at the time when they publish their advisory and source RPM. The clones only need to perform a limited amount of testing when packaging the patched version to ensure that there is no compatibility issues with their releases.

Oracle Linux: truth in advertising?

Oracle Linux gets a special mention here. It does not have any special advantage over CentOS or Scientific Linux even though their marketing department wants to make you believe otherwise, quoting from their web site (archived by WebCite at http://www.webcitation.org/6A44EPTRC):

Oracle invests significantly in testing Oracle Linux and releasing critical bug fixes faster, enabling enterprises to deploy with confidence.

As can be seen from the above graphs Oracle was never the fastest clone to publish a critical security advisory within the evaluated time period. In fact it was the slowest distribution in 82% of critical cases. Truth in advertising? Does not seem that way. However the marketing materials do not have an answer to the question “faster than who?”…

Comments welcome

Feedback and other comments are welcome. You can add your comments below. Please stay on topic and do not start a flame war as everyone seems to have their favorite RHEL clone and some people seem to be very emotional about it. Feel free to contact the author privately if your comment is not suitable for public display.