Guest post by Ed Thurstan

Abstract

It is apparent from the data that CRUTEM4 temperatures adjustments have, in part, been made with reference only to the earlier CRUTEM3 data, rather than raw temperature data. Further, the adjustments depend on the month for the data, and these adjustments are made for 20 or 30 consecutive years.

In the case of Adelaide (946720), for 30 years from 1857, CRUTEM4

Lowers all January temps by 1.4 o C

C Lowers all Feb temps by 0.9 o C

C Lowers all March temps by 1.7 o C

C With April to December all lowered by 0.5 to 1.1oC.

Thereafter, there are no adjustments until 2000, when a smattering of adjustments appear, mostly raising the temperature.

There are many examples of this practice. The total effect of all the differences between CRUTEM4 and CRUTEM3 where there is corresponding data is to accentuate warming trend by lowering pre 1995 temps by 0.1 to 0.2oC, and raising post 1995 temps by a similar amount.

This does not mean that the overall effect of all CRUTEM4 updates will induce the same magnitude change in HADCRUT4 anomaly, as no account has been taken of deleted and added stations, or relative numbers of stations in the database, vs the number which display differences as analysed here.

I believe that CRUTEM4 is seriously flawed, due to the apparent selective mechanical adjustments to blocks of station temperatures where the criteria for adjustment are a function of the month name. Until this is adequately explained, CRUTEM4 should be withdrawn.

Background

In March 2012 the Hadley Climate Research Unit released the land temperature dataset CRUTEM4, along with the station data from which it was constructed. A cursory inspection of the new dataset revealed some irregularities in Australian data. In particular, there were puzzling differences between CRUTEM3 and CRUTEM4 data.

A program was written to compare the two complete sets CRUTEM data, and highlight the differences. It compares the two sets to report:

· Stations in CRUTEM3 which do not appear in CRUTEM4. That is, they have been dropped in the construction of CRUTEM4.

· Stations in CRUTEM4 which do not appear in CRUTEM3. That is, new stations.

· Stations which appear in both sets and which have an arithmetical difference in any month of any year. The whole 12 months are reported for these. All missing data (reported as -99 in CRUTEM data) was converted to zero. This means that some data is lost if a valid temperature appears in one set with a matching -99 in the other dataset.

To exclude minor differences due to issues (like roundoff, precision) which could arise in comparing two datasets derived in different ways, a threshold of 0.22 was applied. That is, years are selected only if they contain differences whose absolute value exceeds 0.22oC.

Some simple statistics about the two databases include:

1. Station/Years of data: In CRUTEM3 set – 399,303; In CRUTEM4 set – 466,246.

2. Number of -99 months: In CRUTEM3 set – 674,993; In CRUTEM4 set – 568,606.

3. Number of CRUTEM3 stations dropped from CRUTEM4 286

4. Number of stations added to CRUTEM4 738

5. Matching station years where there is at least 1 month difference 104,296

6. Total stations in CRUTEM3 5,113

7. Total stations in CRUTEM4 5,565

Results

Old temperature data has been adjusted

In the following examples, only a snapshot of small parts of the data is presented to illustrate the point. Positive differences imply that CRUTEM4 is higher than CRUTEM3.

30050 is LERWICK UK; 42160 is ILULISSAT-JAKOBSHAVN Greenland;66450 is BASEL-BINNINGEN Switzerland; 67000 is GENEVE-COINTRIN Switzerland.

Adjustments of this magnitude can be seen through most of the CRUTEM4 database, but especially in Europe and adjacent areas. With data from such early years being so sparse, it is difficult to see both why such adjustments have been made, and the basis on which they were made. They were clearly made with reference solely to CRUTEM3 data, and not from original data. There seems little value in adjusting such early data, unless the purpose is to lower early temperatures.

There are strange repeating adjustments

Strange adjustments have been systematically applied to CRUTEM3 data to create CRUTEM4 data. For example, for a period of 22 years, from 1951 to 1972 Station 915540 (Vanuatu) has the following set of adjustments applied:

With the original CRUTEM3 data showing

1951 26.5 26.3 26.5 26.0 25.2 24.4 24.2 24.1 24.2 25.1 25.6 26.1

1952 26.8 26.6 26.3 25.6 24.8 23.9 23.0 22.6 22.8 23.8 24.5 24.8

1953 25.3 25.4 25.1 24.3 23.7 22.6 22.5 23.1 23.1 24.2 25.0 24.4

1954 25.2 25.1 24.8 24.2 23.6 23.3 23.3 22.2 23.5 23.7 23.9 24.8

1955 24.6 25.2 24.1 24.1 23.9 22.6 23.1 22.3 23.7 24.0 24.5 24.7

1956 24.9 25.6 24.8 24.3 23.9 23.7 22.4 22.5 24.1 24.4 24.4 25.7

1957 25.7 25.7 25.1 25.0 23.1 21.6 21.7 23.2 23.1 23.5 24.3 25.0

1958 25.6 26.3 25.7 24.9 23.9 23.6 21.5 22.0 23.9 23.8 24.1 25.4

1959 25.3 25.7 25.4 24.6 23.7 23.1 23.4 22.8 23.3 24.2 25.3 25.5

1960 24.9 24.8 24.9 24.4 24.1 23.6 22.6 23.3 24.0 23.8 24.7 24.4

1961 26.0 26.6 26.0 25.0 24.9 24.3 23.9 23.7 24.3 24.5 25.1 25.7

1962 26.1 25.8 25.2 24.9 24.7 23.8 23.3 24.1 23.9 24.4 24.7 25.3

1963 25.6 25.9 25.6 25.0 23.7 24.4 23.0 23.8 23.7 23.6 24.4 25.3

1964 25.8 26.5 26.2 25.3 24.5 24.4 23.2 24.0 23.9 24.5 25.5 25.4

1965 25.5 25.8 25.5 24.6 23.5 23.1 22.5 21.9 22.8 23.1 24.1 25.4

1966 25.8 25.9 25.8 25.1 23.5 23.4 22.4 23.0 23.3 23.8 24.3 24.5

1967 25.4 26.0 25.6 24.6 24.7 24.0 22.9 23.5 23.8 24.1 24.2 25.5

1968 25.5 26.0 25.4 24.5 24.0 23.5 22.9 22.6 23.2 24.1 24.5 25.3

1969 25.6 25.9 26.1 25.6 25.0 24.4 23.3 23.6 23.2 24.1 24.7 25.3

1970 25.9 26.0 26.5 25.1 24.4 23.8 23.6 24.2 24.0 24.5 24.5 25.5

1971 25.3 25.7 25.2 24.8 24.1 24.1 23.2 23.9 23.9 24.1 24.5 24.8

1972 25.3 25.6 25.2 24.8 25.1 23.7 22.4 21.7 23.1 24.0 25.2 25.5

And the original CRUTEM4 data showing

1951 26.9 26.7 26.9 27.0 26.0 25.0 25.0 25.3 25.2 25.9 26.4 26.7

1952 27.2 27.0 26.7 26.6 25.6 24.5 23.8 23.8 23.8 24.6 25.3 25.4

1953 25.7 25.8 25.5 25.3 24.5 23.2 23.3 24.3 24.1 25.0 25.8 25.0

1954 25.6 25.5 25.2 25.2 24.4 23.9 24.1 23.4 24.5 24.5 24.7 25.4

1955 25.0 25.6 24.5 25.1 24.7 23.2 23.9 23.5 24.7 24.8 25.3 25.3

1956 25.3 26.0 25.2 25.3 24.7 24.3 23.2 23.7 25.1 25.2 25.2 26.3

1957 26.1 26.1 25.5 26.0 23.9 22.2 22.5 24.4 24.1 24.3 25.1 25.6

1958 26.0 26.7 26.1 25.9 24.7 24.2 22.3 23.2 24.9 24.6 24.9 26.0

1959 25.7 26.1 25.8 25.6 24.5 23.7 24.2 24.0 24.3 25.0 26.1 26.1

1960 25.3 25.2 25.3 25.4 24.9 24.2 23.4 24.5 25.0 24.6 25.5 25.0

1961 26.4 27.0 26.4 26.0 25.7 24.9 24.7 24.9 25.3 25.3 25.9 26.3

1962 26.5 26.2 25.6 25.9 25.5 24.4 24.1 25.3 24.9 25.2 25.5 25.9

1963 26.0 26.3 26.0 26.0 24.5 25.0 23.8 25.0 24.7 24.4 25.2 25.9

1964 26.2 26.9 26.6 26.3 25.3 25.0 24.0 25.2 24.9 25.3 26.3 26.0

1965 25.9 26.2 25.9 25.6 24.3 23.7 23.3 23.1 23.8 23.9 24.9 26.0

1966 26.2 26.3 26.2 26.1 24.3 24.0 23.2 24.2 24.3 24.6 25.1 25.1

1967 25.8 26.4 26.0 25.6 25.5 24.6 23.7 24.7 24.8 24.9 25.0 26.1

1968 25.9 26.4 25.8 25.5 24.8 24.1 23.7 23.8 24.2 24.9 25.3 25.9

1969 26.0 26.3 26.5 26.6 25.8 25.0 24.1 24.8 24.2 24.9 25.5 25.9

1970 26.3 26.4 26.9 26.1 25.2 24.4 24.4 25.4 25.0 25.3 25.3 26.1

1971 25.7 26.1 25.6 25.8 24.9 24.7 24.0 25.1 24.9 24.9 25.3 25.4

1972 25.7 26.0 25.6 25.8 25.9 24.3 23.2 22.9 24.1 24.8 26.0 26.1

Similar adjustments can be seen in many other stations. The above Vanuatu data comes from a tropical area where temperature varies only 2 to 4oC over the year. In this situation, an adjustment of 0.4 to 1.2oC degree seems extreme. Furthermore, there are no adjustments after 1972. But the same type of adjustment appears in more temperate 946720 (Adelaide Australia). For the 30 year period 1857 to 1886 the following adjustment has been applied to CRUTEM3 to create CRUTEM4.

when the two Datasets show

CRUTEM3

1857 22.6 28.3 19.7 17.5 13.0 11.6 11.8 12.2 14.2 15.4 18.4 23.4

1858 26.4 24.0 22.3 18.3 13.5 12.0 10.3 11.7 12.6 15.8 21.1 22.5

1859 23.8 22.4 20.3 17.1 13.2 10.9 10.7 12.4 13.3 17.7 19.7 22.9

1860 26.1 23.8 22.4 17.0 14.4 12.5 11.7 13.4 15.5 17.0 19.9 22.5

1861 23.2 22.3 23.2 19.1 14.2 13.7 10.6 11.4 14.3 17.7 20.0 19.8

1862 25.4 22.4 23.0 17.0 14.9 11.9 13.2 13.1 15.3 18.6 21.4 23.6

1863 23.5 24.0 22.3 20.7 15.8 13.3 11.7 12.2 13.5 16.3 19.0 21.9

1864 23.0 21.9 21.1 18.1 15.3 11.7 11.5 11.7 15.4 16.1 20.6 21.0

1865 21.6 21.7 21.3 19.5 13.6 11.8 10.6 12.7 14.6 17.1 21.7 20.8

1866 23.9 25.2 21.2 19.0 15.8 12.6 11.6 12.6 13.8 16.6 17.8 22.1

1867 24.1 24.3 20.7 18.2 15.6 14.0 11.5 12.6 13.2 16.2 18.9 20.1

1868 20.6 22.9 23.4 17.8 15.8 12.1 10.4 12.1 14.6 17.9 20.2 22.0

1869 22.1 22.6 21.3 17.4 13.4 12.4 11.3 12.8 11.8 16.4 20.4 21.7

1870 23.4 25.3 21.7 18.7 13.9 12.7 10.8 11.5 12.9 17.2 18.0 21.9

1871 23.0 23.8 20.3 18.6 15.4 13.4 11.6 13.4 14.8 16.6 18.6 24.0

1872 25.9 23.3 22.4 17.1 13.5 12.6 11.1 9.9 14.1 16.2 21.0 20.0

1873 23.9 22.6 19.9 16.7 14.9 11.8 10.7 12.6 14.2 18.3 17.3 23.6

1874 24.0 21.7 19.8 19.4 14.0 11.7 9.8 11.5 12.1 17.6 17.9 21.9

1875 23.6 22.9 20.9 18.1 13.2 11.9 10.6 12.2 13.8 16.6 18.2 19.3

1876 22.8 22.1 23.8 16.5 13.3 10.8 10.1 11.5 13.5 16.1 18.6 23.4

1877 23.1 24.7 20.3 17.7 14.2 11.7 11.2 13.6 12.4 16.1 17.2 20.6

1878 25.6 23.0 21.6 18.3 14.1 9.9 11.7 12.9 14.3 17.5 20.1 21.4

1879 24.4 23.9 20.9 18.6 12.4 11.6 10.1 12.1 13.2 16.2 18.6 21.1

1880 25.6 26.4 21.5 17.5 13.7 12.1 10.4 12.9 13.7 15.4 17.8 21.9

1881 23.2 21.7 21.4 17.4 14.9 10.6 10.6 11.8 13.9 15.3 18.6 21.3

1882 22.9 24.0 22.6 17.4 15.1 10.6 9.6 11.2 14.0 16.9 20.8 21.6

1883 23.6 21.9 20.9 18.4 13.2 13.1 10.8 11.3 12.4 15.3 19.6 20.9

1884 21.2 23.6 22.1 17.1 13.9 12.2 9.8 13.3 14.1 15.5 18.9 19.5

1885 21.6 21.6 18.9 17.2 15.9 10.8 10.7 12.8 14.0 18.0 19.2 23.2

1886 24.4 20.6 20.4 17.3 14.0 11.6 11.6 12.4 16.1 14.7 19.9 21.9

CRUTEM4

1857 21.2 27.4 18.0 16.4 12.1 11.0 11.4 11.4 13.7 14.7 17.5 22.6

1858 25.0 23.1 20.6 17.2 12.6 11.4 9.9 10.9 12.1 15.1 20.2 21.7

1859 22.4 21.5 18.6 16.0 12.3 10.3 10.3 11.6 12.8 17.0 18.8 22.1

1860 24.7 22.9 20.7 15.9 13.5 11.9 11.3 12.6 15.0 16.3 19.0 21.7

1861 21.8 21.4 21.5 18.0 13.3 13.1 10.2 10.6 13.8 17.0 19.1 19.0

1862 24.0 21.5 21.3 15.9 14.0 11.3 12.8 12.3 14.8 17.9 20.5 22.8

1863 22.1 23.1 20.6 19.6 14.9 12.7 11.3 11.4 13.0 15.6 18.1 21.1

1864 21.6 21.0 19.4 17.0 14.4 11.1 11.1 10.9 14.9 15.4 19.7 20.2

1865 20.2 20.8 19.6 18.4 12.7 11.2 10.2 11.9 14.1 16.4 20.8 20.0

1866 22.5 24.3 19.5 17.9 14.9 12.0 11.2 11.8 13.3 15.9 16.9 21.3

1867 22.7 23.4 19.0 17.1 14.7 13.4 11.1 11.8 12.7 15.5 18.0 19.3

1868 19.2 22.0 21.7 16.7 14.9 11.5 10.0 11.3 14.1 17.2 19.3 21.2

1869 20.7 21.7 19.6 16.3 12.5 11.8 10.9 12.0 11.3 15.7 19.5 20.9

1870 22.0 24.4 20.0 17.6 13.0 12.1 10.4 10.7 12.4 16.5 17.1 21.1

1871 21.6 22.9 18.6 17.5 14.5 12.8 11.2 12.6 14.3 15.9 17.7 23.2

1872 24.5 22.4 20.7 16.0 12.6 12.0 10.7 9.1 13.6 15.5 20.1 19.2

1873 22.5 21.7 18.2 15.6 14.0 11.2 10.3 11.8 13.7 17.6 16.4 22.8

1874 22.6 20.8 18.1 18.3 13.1 11.1 9.4 10.7 11.6 16.9 17.0 21.1

1875 22.2 22.0 19.2 17.0 12.3 11.3 10.2 11.4 13.3 15.9 17.3 18.5

1876 21.4 21.2 22.1 15.4 12.4 10.2 9.7 10.7 13.0 15.4 17.7 22.6

1877 21.7 23.8 18.6 16.6 13.3 11.1 10.8 12.8 11.9 15.4 16.3 19.8

1878 24.2 22.1 19.9 17.2 13.2 9.3 11.3 12.1 13.8 16.8 19.2 20.6

1879 23.0 23.0 19.2 17.5 11.5 11.0 9.7 11.3 12.7 15.5 17.7 20.3

1880 24.2 25.5 19.8 16.4 12.8 11.5 10.0 12.1 13.2 14.7 16.9 21.1

1881 21.8 20.8 19.7 16.3 14.0 10.0 10.2 11.0 13.4 14.6 17.7 20.5

1882 21.5 23.1 20.9 16.3 14.2 10.0 9.2 10.4 13.5 16.2 19.9 20.8

1883 22.2 21.0 19.2 17.3 12.3 12.5 10.4 10.5 11.9 14.6 18.7 20.1

1884 19.8 22.7 20.4 16.0 13.0 11.6 9.4 12.5 13.6 14.8 18.0 18.7

1885 20.2 20.7 17.2 16.1 15.0 10.2 10.3 12.0 13.5 17.3 18.3 22.4

1886 23.0 19.7 18.7 16.2 13.1 11.0 11.2 11.6 15.6 14.0 19.0 21.1

The updates accentuate the global warming argument

There is pronounced tendency to flex the graph of global temperature vs time in a way which accentuates warming in recent years. However, the dominant effect is to lower temperatures prior to 1995.

The differences between CRUTEM3 and CRUTEM4, with zero tolerance, were consolidated to annual temperature differences, and plotted against time along with the count of the stations which contributed to the graph. This gives

The adjustments indicate that in CRUTEM4:

· A comparatively small number of stations have been cooled between about 1820 and 1900 anywhere up to 0.4oC.

· A substantial number of stations have been cooled about 0.1oC between about 1910 and 1995.

· Between about 100 and 400 stations have been warmed by up to about 0.2oC since about 1995.

This does not translate into the same change in the anomaly vs time graph, because it is not the whole dataset, but simply the changed data between CRUTEM3 and CRUTEM4. Nor does it take into account the effect of dropped and added stations. But it does indicate the direction of the change.

Nitpicking Differences

The nature of the CRUTEM database makes it obviously difficult to manage. Data comes from many sources. It is unreasonable to expect Hadley or UEA personnel to understand the geography of the data they receive. So errors will appear in the database. Some positional errors can be dismissed on the grounds that CRUTEM is directed at anomalies. This means that if a station’s position data is wrongly represented, then provided the error is contained within the same 5o v 5o gridcell, the anomaly is unaffected.

However, it is difficult to see how Station number 237070, name listed as “Unknown, Russia”, with a Lat/Long of -99.9/-999.9 (ie unknown), could escape being found by Quality Control, while still feeding temperatures to the gridding/anomaly calculation. Station 288020 is similarly identified, and also supplies data.

“Normals” are, by convention, calculated over 30 years from 1961 to 1990. While this appears to be observed, it is quite common to calculate Standard Deviations over a different period, commonly 1941 to 1990.

Extreme differences occur when the same station Number is used for two different locations. For example, In CRUTEM3, Station 840270 is the high altitude Tulcan El Rosal in Ecuador. But in CRUTEM4 station 840270 is Esmeraldas Tachina, also in Ecuador, but several hundred km distant from its CRUTEM3 namesake, and at almost sea level. This appears to be a CRUTEM3 error, probably difficult to find once the error is made.

Sydney Airport (947670) is still badly identified. The data runs from 1859 to the present, but only data after 1990 comes from Sydney Airport. The earlier data comes from Sydney Observatory, 10km away.

Effect of added and deleted stations

No attempt is made in this report to assess the effect of the 286 station deletions and the 738 additional stations, except to observe that most of the additions are high latitude NH stations.

Discussion

When updating a database such as CRUTEM, I would expect the steps to be roughly

· Tidy up the precursor database, which should be mostly stable data that has had years of scrutiny, and which would require limited correction or additions.

· Delete data which is considered poor quality.

· Add new data, edited and homogenised.

· Run a check as has been done in this report, looking at differences which might suggest irregularities.

But Hadley/CRU do not have appears to have done this. They have added and deleted stations, but it seems strange that very early data – 18th and 19th Century data, should be added, especially when much of it is sparse, of perhaps questionable quality and not germane to the current temperature/time/CO 2 discussion.

But the biggest problem with the new HADCRUT4 database is the frequent practice of systematic, repetitive temperature modifications on blocks of station data. What reason could there be for making the following set of adjustments on Adelaide data for every year from 1857 to 1886 ?

The perpetrators of this change have clearly not gone back to original raw data and re-appraised their original homogenisation processes. They have simply taken every year of CRUTEM3 data (with it’s possible homogenisation adjustments), from 1857 to 1886, and

· Deducted 1.4oC if the month name was January,

· Deducted 0.9oC if February,

· Deducted 1.7oC if March

· And so on.

It is difficult to conceive w­­­­­­hat the justification could be for these changes. There are many instances of such changes in HADCRUT4, covering periods of 5 to 30 years, in both ancient and recent data. The adjustment vector differs in each, but the magnitudes of the individual elements are equally large. Their signs may vary.

The temperature differences, where there are corresponding data in each of the datasets works to promote the impression of a temperature surge in the 1990s.

It has done this by lowering temperatures before about 1995, and raising them thereafter.

The effect of this on the final gridded anomaly data has yet to be calculated. But it will be less than depicted above. The above graph shows the difference in the data induced by changes in data which appear in both CRUTEM3 and CRUTEM4. It does not include the effect of

· Dropped stations

· Added stations

· Stations which record a valid temperature in one set, but a Null (-99) in the other.

There are other relatively minor issues regarding data quality. I am surprised that neither CRU quality control nor users of the data have picked up these errors.

The CRUTEM4 database appears even more like one which has been assembled by amateurs with little concept of accuracy or integrity. If you were of a suspicious nature, you might get the impression that the changes are targeted at accentuating the Warmist message.

Until the matters raised in this report have been adequately explained, CRUTEM4, and therefore HADCRUT4 are seriously flawed.

Acknowledgements

Thank you Warwick Hughes (http://www.warwickhughes.com/blog/) for encouraging me to write this report, and for helpful suggestions on the construction of the report.

Data available

The original data is available at: http://www.cru.uea.ac.uk/cru/data/temperature/.

The data and processing programs used in preparation of this report are available from the author at thurstan@bigpond.net.au., or from Warwick Hughes blog http://www.warwickhughes.com/blog/ They are in two zipped packages.

Data

This comprises

· The CRUTEM3 and CRUTEM4 station data for all examples mentioned in this report.

· The output of a full CRUTEM run, with some analysis of the output. It will not run under XL97, which is limited to 64K rows, when this workbook holds over 400,000 rows.

This is about 26mb compressed.

Programs

· An Excel 2010 Workbook. The one that produces the output used in this report. It is a Visual Basic driven Excel workbook. It extracts stations missing from each dataset, and the differences. An untested XL97 version is included. It will not run the full CRUTEM database, as XL97 is restricted to 64K rows, and CRUTEM generates over 100,000. But it is possible to split the two sets of input into smaller parcels. Australia/New Zealand generated only about 650 rows of output.

· Instructions for use are on Worksheet “MAIN”.

Share this: Print

Email

Twitter

Facebook

Pinterest

LinkedIn

Reddit



Like this: Like Loading...