The results of CRUTEM have been replicated, reimplemented and/or corroborated at least six times by now, using a variety of analyses, written in different languages, by different people.
Not counting the satellite data and other lines of physical evidence, we have:
I’ll have to admit I was a little astounded at the agreement between Jones’ and my analyses, especially since I chose a rather ad-hoc method of data screening that was not optimized in any way. Note that the linear temperature trends are essentially identical; the correlation between the monthly anomalies is 0.91.
So we have by now temperature analyses all pointing to the same result and written in a variety of languages: R, Fortran, Python, STATA, Perl, and so on, some based on reimplementing code, and some based only on descriptions of the analysis steps. This rather gives the lie to the notion put about by ’sceptics’ that this type of replication is impossible without ‘the code’. Indeed further than that, it shows that replication cannot be highly sensitive to the low-level implementation detail, categorisation of ‘good’ and ‘bad’ data, and even some high level details of the data processing.
When ’sceptics’ talk about replication of temperature analysis the analogy is often made to a chemical or physical experiment, where you supposedly need the lab books etc in order to exactly replicate what was done to the decimal point. However the reality is that those who are replicating results typically do so in their own labs using their own scientific instruments, and simply follow the essential features of the protocol that they are trying to replicate. They may even try to falsify the result using a completely different experiment design with different controls. Getting exactly the same answer to the decimal point isn’t expected, because the circumstances can never be identical.
Think about your high school physics, when you rolled a weight down a ramp you didn’t roll the same weight down the same ramp as everyone else in the world did. Nor I presume did you book a trip to the leaning tower of Pisa to chuck weights off the top. You probably didn’t go up in a space shuttle to test F=ma either. In ’sceptic’ land, this makes the theory of gravity a hoax and you a scientific fraud.
By the same analogy the ’sceptics’ demand for ‘the code’ is like insisting that it’s insufficient for replication if experimenters simply describe the off-the-shelf equipment they used and what they did with it. It is like insisting that replication is impossible unless the original scientists provide them with the same scientific instruments they used in their experiment and access to their lab. Worse than that, such is the degree of nitpicking that it is like demanding that scientists record details that aren’t even obviously relevant, such as what they had for breakfast when they did the experiment, how much they weighed that day, or what was the number 1 record that week.
Over on Bishop Hill, dcardno argues:
The problem is that the “instrument” used by various climate investigators was largely software – it didn’t really exist other than in the code, and had never been seen before. The analogy is not to an often-performed drug test, but an entirely new and unproven method of determining eligibility.
Consider results of a chemical experiment: If I were to reference results from a mass spectrometer, I can be reasonably confident that any other experimenter can reproduce those results; “mass spectrometer” is a reasonably well-defined product. If they cannot reproduce my results, we might look at their lab techniques, process temperature control, timing and so on before we suspected that a difference in spectrometer is to blame. Eventually, though, if that was the only uncontrolled variable, we would investigate there, as well – and if different spectrometers gave significantly different results, we would learn something, either about my claims or the state of our instruments.
On the other hand, if I produce a result, but base it on a newly-developed ‘mass spectophotometron’ other investigators have no way to verify or disprove my results, particularly if my ‘high level’ description of what the device does is either inadervertently or deliberately incomplete. In that case, in order to be taken seriously, I have to demonstrate that my ‘mass spectophotometron’ actually measures some relevant property, and that it does so in a consistent and meaningful way.
Exactly so. However all experiments use off the shelf equipment in a novel protocol or they wouldn’t tell us anything new, so this objection if valid would apply to absolutely any experiment. And the point is that the statistical processing done on the temperature record is also made up of well known and easily described off the shelf techniques. This is not brand new stuff – it is well known techniques assembled to address a new problem.
So again it comes down to describing what you did, and that means describing the details that matter. In the case of temperature record, the fact that a variety of other analyses exist with very similar results obviously implies that people did have enough information to replicate. Not only that, but that the answer is right.
dcardno continues (my emphasis):
This is analogous to the CRU / Mann / etc. results – based on the descriptions in their papers, capable investigators could not reproduce their findings, and could not verify that the black box code (their ‘mass spectophotometron’) functioned as decribed. Rather than demonstrate that their analysis was correct, they hid their methodolgies and prevented independent access to their raw data. As shown by the various e-mails, they knew that their results were irreproduceable – so they blustered, appealed to authority, and attacked anyone who actually wanted to see their ‘mass spectophotometron’ and see evidence that it actually worked.
How could they have known something that we now know for sure is not true? The CRU results have been reproduced.
The Pedant-General argues:
If I do my own stuff and come up with a different answer, the team, the media and the IPCC ignore it entirely.
[...]
The ONLY way to strike at the team and the consensus is to demonstrate that the consensus scientists have arrived at their results incorrectly. You have to demonstrate flaws in THEIR work.
To which my response is, so what if you are ignored? You have a right to speak but not a right to be listened to. I don’t buy it in the first place – if a result based on reasonable analysis of the same or even similar data existed, it would be addressed. This has already happened in the case of the UAH analysis, for example. If the ’sceptics’ really have an analysis that gives a different answer to CRUTEM or Mann or whoever then let’s see it. Let’s see your work. Of course, that means actually doing analysis and science, and not ‘auditing’.
In essence what the ’sceptics’ demand amounts to a proposal that the scientific method which has worked well for centuries should now be re-engineered to meet a different requirement. Instead of being optimised to find out the truth about the world, in spite of error, fraud and human fallibility, it should now be re-engineered to serve the needs of nitpickers and those in search of fraud. But Science doesn’t care whether somebody added up a column of numbers correctly, and it’s already perfectly capable of routing around errors like that, malicious or otherwise. Science only cares about the damn answer. You get that by doing Science.
Ultimately the fact is that there isn’t evidence of a serious problem in CRUTEM and there never was. Certainly ‘open source science’ and exactly what details should be archived for exact reproduction is an interesting question to consider in its own right. Based on my own experience of archiving my own code for my own future reference, I would argue that you probably can’t do this easily without stashing away a virtual machine containing the complete implementation, and/or a source code control system containing not only your code but almost all dependencies. However it is rather ironic for ’sceptics’ to insist based on no evidence at all that there is a transparency problem with implications so serious that it requires dropping everything and turning the world upside down to address it, while they themselves ignore the far more compelling evidence for the need to do something about greenhouse gas emissions. Perhaps in 30 or 60 years, when the ’sceptics’ have proved beyond any doubt that releasing code and data to the level of detail they demand is necessary, worth the costs, and not harmful, we can do it then.
Meanwhile, maybe releasing ‘the code’ is helpful or maybe it just propagates error. Maybe archiving absolutely every detail of every analysis such that it can be reproduced 50 years hence is worth the onerous costs, maybe it isn’t. And meanwhile, you now have the code for 6 or so implementations which you haven’t investigated yet, but you’re stuck on the one from CRU. What about the others? Why haven’t you got cracking on them? Don’t like the ones in Fortran – try the one in Perl. Don’t like that the Fortran ones don’t have unit tests etc (what were you expecting, poetry?) try the python implementation.
And when it comes to reproducibility of results years hence, you can also think about this: how would you propose to reproduce the results of Gilbert N. Plass, which used early computer models in the 1950s? Good luck getting those to run on your Mac.
Where to start?
This merely goes over exactly the same ground and merely reheats all the points you have already made in Bishop Hill’s original post and answers none of the substantive objections raised against them.
” Of course, that means actually doing analysis and science, and not ‘auditing’.”
This is particularly especially nauseating. The audit work that Steve McIntyre has been doing is much much much more onerous because he has had to reverse engineer what has been done.
So is this:
“In essence what the ’sceptics’ demand amounts to a proposal that the scientific method which has worked well for centuries should now be re-engineered to meet a different requirement. Instead of being optimised to find out the truth about the world, in spite of error, fraud and human fallibility, it should now be re-engineered to serve the needs of nitpickers and those in search of fraud.”
The scientific method has indeed worked well for centuries. It’s just that it hasn’t been employed by the climate ’scientists’ – hey we can both use scare quotes. The sceptics wish to return climate ’science’ to proper actual science, yet they are being lambasted and obstructed at every turn.
Do you really not see any problem at all with Jones’ – almost proud – admission that no-one had ever asked to see his workings? All you are getting from the climate ’science’ is confirmation bias.
Pedant-General:
Not surprisingly, I (cited by our host) tend to agree with our host.
But I’m curious. What is it you really mean by ‘actual science’? ln context, you seem to mean something about reproducibility.
Yet if I want to reproduce some of the astrophysical work that I have been interested, I’m sunk. Doing so would require that I have time on the Hubble telescope, or the like, in order to collect my own observations of the given area. That also requires that the observations aren’t of singular things like supernovae. Even if I could get time on the Hubble, that particular supernova is not going to recur for me, so that portion of astrophysics can’t be reproduced — for a particular notion of reproducitibility.
So, do you conclude that astrophysics is not ‘actual science’ either?
It’s that sort of puzzlement I have with what you really mean by ‘actual science’, or rather ‘proper actual science’. Not scare quotes, btw, just that it’s a term you’re using that doesn’t seem to have been defined yet.
Well Frank, on this issue, I think we will have to agree to disagree – as the Pundit-General notes, we are just going over the same ground. I agree with P-G’s assessment that most of the independent verification of various climate alarmist’s results is either confirmation bias, or less “independent” that it would appear – through shared authorship or the links of collegiality: for some reason I am not much impressed if another member of the hockey team declares Mann’s results to be “robust.”
…the statistical processing done on the temperature record is also made up of well known and easily described off the shelf techniques. This is not brand new stuff – it is well known techniques assembled to address a new problem.
Not quite – the de-centred PC analysis is simply wrong – to use my earlier example, it is a mass spectophotometron’ that simply doesn’t work. “Hiding the decline” is again, simply wrong – until there is substantive evidence that the ‘divergence’ can be traced to a unique (late) 20th Century factor (and so far, all we have is hand-waving in that regard) then it is an unacceptable practice to brush the issue under the rug.
…it requires dropping everything and turning the world upside down to address it, while they themselves ignore the far more compelling evidence for the need to do something about greenhouse gas emissions.
Frank, please stop frothing at the mouth. Releasing a couple of computer files is hardly “turning the world upside down” – particularly compared to the policy response ardently promoted by Hansen et al of taxing every transaction that involves the production of consumption of energy (in other words, virtually every economically-significant human interaction), and unknown cost in terms of standard of living and economic growth. Surely if the situation is so dire to require such a response, it at least merits a truly independent second look!
However all experiments use off the shelf equipment in a novel protocol or they wouldn’t tell us anything new, so this objection if valid would apply to absolutely any experiment.
Yes – but note you are considering off the shelf equipment. The black box climate models are not off the shelf – they are bespoke equipment intended for a single purpose – it behoves the user to prove that it operates as intended. It would be a huge step in that direction to allow someone else to examine it and verify the logic and the coding.
Maybe archiving absolutely every detail of every analysis such that it can be reproduced 50 years hence…
That’s a pretty obvious strawman, Frank. Rather than insisting that results can be reproduced 50 years from publication, I think most sceptics would be happy to see them reproduced 50 days after publication – unfortunately, even that can’t be managed.
“Scientific replication is not rote repetition”
Not what’s being argued here.
Of course rote replication is not sufficient. In some circumstances, it may not even be necessary but that’s not the case here at all. However, if you believe that there is a flaw in the work, starting with rote repitition is a very very good basis. confirm you know hwo the original work was done, then show where and why that work is weak.
Crucially, if the method fails or defies rote repitition it utterly fails the replication test and it’s not science at all.
Before we deal with anything else, can you at least confirm that you understand this really rather simple point?
Robert,
“That also requires that the observations aren’t of singular things like supernovae. “
Huh? Are we even talking about the same topic? What part of needing to see the raw data used and the code used to process it is a singular event that will not happen again?
dcardno,
“Rather than insisting that results can be reproduced 50 years from publication, I think most sceptics would be happy to see them reproduced 50 days after publication – unfortunately, even that can’t be managed.”
You are being too soft. All the major journals have clear policies on this – many require data to be made available as a condition of publication – not 50 days after, but actually prior to publication.
I simply cannot understand why anyone can defend the refusal to abide by those rules.
pedant:
There are many things that you could mean by ‘proper actual science’, and you still haven’t said what all you mean by that. Nor even address my example.
Continuing with the supernova, fine, you only want all raw data and all code used to process it. At least that’s all you’re saying at the moment. More consideration would show that you need more than that, as I discussed back in the article Frank links to. But stay there.
‘all’ data is in terabytes for that supernova. ‘all’ code is quite a lot — from the code that is used to aim the telescopes to preprocessing steps and through to the code that produced the particular pretty picture that you think is suspicious and want to ‘reproduce’. The code involved is in many institutions across the world, as is the raw data. What is archived is cleaned up data several steps down the processing chain. You won’t be able to get all the code.
As to getting all the data, or merely all the data that is archived … how much are you willing to pay? How many institutions are you willing to contact? If the answers are zero, and 1, respectively — as they appear to be for climate — then, again, you have to reject astrophysics as well.
Get to the data for some of the larger projects and you’re talking petabytes, not terabytes. And exabytes are coming in. Particle physics, other parts of astrophysics, biology, are all at that level of data volume. And you’re not going to be able to copy all that to your home computer.
Robert – the data capture that you are dealing with on your supernova example is not the first optical (or radio) observation of that phenomenon. There is lots of metadata about astonomical observations (or gene sequencing or particle accelerators, two other high data-volume areas you cited – we could add medical diagnostics and imaging): we understand optics – even adaptive optics and post-processing; most of that can be taken as read. We have similar understandings of radio reception and interference, and gene sequencing technology or density reconstruction – although it is not that long ago that the algorithms for -say- axial tomography had to be tested and experimentally verified before we could use them in diagnostics, even when they were producing images that were clearly in accord with expectation. We have no such metadata about the various calculations performed by climate investigators, and so no reason to suppose a priori that they work.
Who is ‘we’ that you are talking about? That same ‘we’ is who you’re saying are to decide whether something is or is not science.
Does ‘we’ include my jr. high neighbor? He certainly doesn’t understand how astrophysical observations are taken.
Does ‘we’ include every adult? Again, few could explain, much less carry out, adaptive optics or seismological inversion calculations.
Or is ‘we’ just you and pedant and the list of those you two personally approve of? I certainly can’t see any general philosophical justification for you being the sole deciders of what is or is not science.
Richard Feynman would be on the short list of people who could be said to understand quantum mechanics. Yet his comment was that nobody understood quantum mechanics. By your definition, then, quantum mechanics is not science. Still my computer works, so I suspect the problem is with your definition.
I certainly can’t see any general philosophical justification for you being the sole deciders of what is or is not science.
But you seem to have no trouble with Phil Jones or Mike Mann telling you.
Again, few could explain, much less carry out, adaptive optics or seismological inversion calculations.
Agreed – so would this justify a refusal to release data, methods, etc, to anyone who, in your opinion, was incapable of understanding it or them? Just who is “defining science” then, Robert? This is, of course, exactly the tack taken by the leading climate investigators – unfortunately, they seemed to define “capable of understanding” as a synonym for “agrees with (or won’t challenge) my results.” I take many things as fact that I cannot verify for myself; I do so on the basis that there is a whole web of people (most of whom I do not know) who can verify them, and after attempting to falsify them have been unable to do so. That system breaks down when no one who can look at the detailed results has any interest in falsifying them – and that is manifestly the case in the closed world of climate science alarmism.
Still my computer works, so I suspect the problem is with your definition.
Or your understanding.
dcardano:
You’re the one talking about Mann and Jones, not me. For my taste, rather more than 2 people should be involved in deciding what is or is not science. So far, though, you haven’t defined ‘we’, so it could indeed just be the two of you — dcardano and pedant-general.
Your little rant shows just that you still haven’t defined who the ‘we’ are that get to decide what is or is not science. And engaging in mind-reading, which you do badly. It is amusing to see Roy Spencer added to the ‘closed world of climate science alarmism’. He’s done his own test recently and supported that indeed the surface record shows the sort of thing CRU claims it does. Quite surprised himself.
You definition of science being things that ‘we’ ‘understand’ is just a poor one. You fail to define who ‘we’ are. And, as my Feynman quote illustrated, being able to understand something is not a requirement for the something to be scientific.
Defining science is a difficult thing, and I’ve seen professionals (philosophers of science) take swings at it. If you and pedant think you’ve solved that problem, or some part of it, that would be interesting and I’d get to learn something.
You’re the one talking about Mann and Jones…
But they are the one deciding who gets access to the data and methodologies that underlie their claims that observed warming is anthropogenic. In effect, they are defining climate science by determining who gets to participate, and who doesn’t.
I have not tried to define anything – the Popperian definition of science is (broadly speaking) investigation and testing of hypotheses subject to falsification. If the hypotheses can’t be examined because they are embodied in computer code that (third party) investigators can’t look at, then they can’t be falsified. If the data can’t be examined because it is proprietary (excuse number one) or has been lost (excuse number two), then the hypotheses that flow from the data can’t be falsified. If the hypothesis can’t be falsified… well, it fails that Popperian definition of science.
“We” is, as I said, anyone outside the CRU / GISS orbit, Robert – and to the best of my knowledge Spencer’s work has always been done without access to the CRU data / methodologies (and by the way, you might have a look at subsequent work by Spencer on that topic, as well). I did not say that we have to understand something for it to be science, as per your Feynman example as an extreme case – only that what we are doing is subject (at least in principal) to falsification. To my knowledge, no one has proposed an experiment that would falsify string theory, but that is not because the theorists are not willing to subject their work to examination; it is because we can’t think of an experiment that can be performed with existing technologies that would test the theory and expose it to falsification. The fact that in principal (if not in practice) it is subject to falsification (someday we will have the capacity to test at the required energies, or some very clever person will suggest an effect observable with then-current technology) makes all the difference in deciding whether string theory meets Popper’s definition of science.
dcardno,
why the emphasis on Popper for a definition of science? It is almost a commonplace of undergrad philosophy of science that there are problems with using falsification, or falsifiability of hypotheses, as the criterion for demarcation between science and non-science.
I think you share, with the Pedant-General above, an unsatisfactory appreciation of what science is. P-G claims that “The sceptics wish to return climate ’science’ to proper actual science”. But what is “proper actual science”?
Why don’t the skeptics, instead of demanding to see computer code, actually get out and measure something themselves? Then they can submit the results to journals and the rest of us can scrutinize their data and computer code.
Ian McEwan put it well on the Australian Broadcasting Corporations’s Lateline TV programme last night:
“So, I’d really like to see the debate move the other way around: let the sceptics start moving across the face of the Earth and gather empirical data, put it up for peer review and see how it goes, and if they can reassure us that they have solid scientific evidence that, strangely enough, the Earth’s temperature is not rising, then we’ll all be very, very happy. We’ll pop open the champagne.”
http://www.abc.net.au/lateline/content/2010/s2847828.htm
Petered – sorry for the late response; I thought this thread had gone cold. As for the reference to Popper – because it was convenient (I can remember it) and reasonably comprehensive. Would you care to expand on the ‘commonplace’ observation about the problems of falsifiability – I have never heard of them. (but that’s no proof one way or ‘tother). I would hate to borrow Robert’s refrain that you are seeking to “define science” – but I just can’t see a heck of a lot of use for ‘non-falsifiable’ assertions; to my engineer-turned-finance guy mind, that sounds like ‘just-so stories’
Why don’t the skeptics, instead of demanding to see computer code, actually get out and measure something themselves?
Because the burden of proof is on the person making the assertion. Nevertheless, with an openness that easily matches Mann’s or Jones’, I can assert that I have examined every temperature record on Earth and multiple proxies, and there is no warming. Of course, you can’t see my data (it’s under NDAs, it’s really badly organized, I lost it) nor can you see the details of my analysis (it’s proprietary, it’s none of your business, it’s adequately described in the article).
Now are you happy?
If not, why not?
Finally, no one doubts that there has been “warming” – that’s been going on since the end of the Little Ice Age. The doubt is about the (exclusive) attribution to CO2, and the assumptions that prevention is possible (let alone desirable), and that preventions will be cheaper than adaptation.
dcardno,
yes, perhaps the thread had “gone cold”, and perhaps I should have left it that way, but- speaking personally- I just can’t bear to see the “contrarians” (my preferred term to “sceptics” or denialists” have the last word. Call it a weakness if you will. That’s me.
I will separate my comments into separate posts.
(1) As to Popper, I have noticed a tendency for climate contrarians to quote what they take to be a simple, naive falsificationist view of science (I call it “Popper Lite”), as if this was the first and last word about the nature of science. It is not. In fact, there are serious problems with both the naive and more serious (Popperian) versions of falsification. Read, for example, Sir A.J. Ayer’s account in his ‘Philosophy in the 20th Century’. “This is not to deny that Popper gives a luminous account of at least one form of scientific procedure, but the basis of his sytem is insecure”. Popper and his followers claim to have solved the problem of induction, and to have banished induction as a form of valid scientific method. However, and as Ayer points out, we reason inductively, and validly, all the time.
A.F. Chalmers, in ‘What is this thing called Science?’ (Univ. Queensland Press) points to the problem of ascertaining what is an observation that will refute a theory or hypothesis (or conjecture, in Popper’s term). What is the clear unambiguous piece (or pieces) of evidence that you think would refute global warming “theory”?
Oops,
the first sentence after (1) shoudl read:
As to Popper, I have noticed a tendency for climate contrarians to quote a simple, naive falsificationist view of science (I call it “Popper Lite”), as if this was the first and last word about the nature of science.
The lack of quality control is entirely my fault. :-(
(2) You claim: “Because the burden of proof is on the person making the assertion.”
I disagree. Your assertion appears to me rather similar to that of a person who, confronted by somebody who has just come in from the rain, dripping wet and claiming “It’s raining hard!”, responds with: “Raining? I don’t believe you. The burden of proof that it’s raining is on you, old son”. You see, dcradno, I think that if the evidence for a particular argument, theory, viewpoint, call it what you will, is overwhelming, then the burden of proof falls on those who would deny the theory or argument at issue. I think myself that the accumulation of evidence in favour of the view that carbon dioxide contributes in some measurable and substantial way to recent (and, in all likelihood, future) warming of the planet’s atmosphere and oceans is just too substantial to be denied. But that’s just my perspective, and is what separates you and other contrarians from me.
However, I would be more sympathetic to the climate contrarians if they made an honest effort to explain, in terms of mechanism, why the climate behaves as it does. It is just not good enough to claim (as I have heard, from an engineering colleague of mine- I’m a physical scientist myself) that we’re warming because the planet is coming out of a Little Ice Age. I mean, really! As if that explains anything! GHGs offer a causal mechanism and explanation for climate change. Asserting that “we’re warming because we’re coming out of ice age” offers no such causal explanation.
You are also mistaken when you suggest that AGW proponents claim that warming is due exclusively to CO2. This has NEVER been claimed by adherents of AGW. (Challenge: show me where they do claim that.) The claim has always been that natural fluctuations, due to solar and volcanic activity variations, together with the effects from GHGs, plus other forcings due to aerosols, etc. can account for the temperature variations of the last century or so.
We’ve gotten away a little from the topic of Frank’s post. The issues of replication & reproduction of results seem relevant to climate science and important. None of the contrarians has, to my best knowledge, responded adequately to the challenge that we now appear to have before us a number of analyses of the temperature data, using a variety of codes, that (seem to) arrive at basically the same answer. To be sure, “replication” (exact copying) has not been undertaken (it is of little relevance to actual science anyway, as Frank points out), but the results have surely been reproduced.
peterd,
“None of the contrarians has, to my best knowledge, responded adequately to the challenge that we now appear to have before us a number of analyses of the temperature data, using a variety of codes, that (seem to) arrive at basically the same answer.”
Exactly. And this has been going on for a while – every now and then somebody pops up claiming to have spotted a problem in the code for GISS, or CRU, or whatever the fashionable record for attacks is that week. They never explain how the problem affects the other records that use different code. (This is another reason why sharing code might be counterproductive – if everyone used the same code base – such errors would pollute multiple records).
It’s as if we had several clocks of different designs, all saying it is about 11pm. The ’sceptics’ fixate on one of them, and say this one is fast. Well, why does it agree with the others then?
What is the clear unambiguous piece (or pieces) of evidence that you think would refute global warming “theory”?
Peterd – this is a really good question… unfortunately I can’t offer a really good response, or even any response, although I have been thinking about it for a few days now. It has forced me to think about falsifiability in observational sciences, as distinct from experimental ones, and I can see where you are dissatisfied with falsifiablity, or “Popper lite” (tastes great / less filling!). I am still not convinced that alarmists have made their case – but I have to think more carefully about why.
None of the contrarians has, to my best knowledge, responded adequately to the challenge that we now appear to have before us a number of analyses of the temperature data… that (seem to) arrive at basically the same answer.
Not quite – we have a number of assertions about the results of an analysis that we are not allowed to see. It isn’t that much of a surprise that they agree, at least directionally. As an example, either on this thread or at Robert Grumbine’s site he was commenting on an observation by Roy Spencer that he had found that a set of temperature records more-or-less randomly selected, had been in close agreement with a particular dataset from CRU. Robert jumped on that observation, and was chortling that surely Roy Spencer wasn’t one of the ‘inner circle’ of climate science, and obviously his findings validated whatever dataset he was comparing to. The odd thing that Spencer noted a couple of days later was that the CRU results were (or claimed to have been) adjusted for UHI, while his was not – it was raw temperature.
It is possible that this was just an inadvertent error; an unadjusted file was mistakenly labelled as adjusted – but unless someone had made that comparison, it would never be known, and we would be left with the impression of warming in excess of what a (hypothetical) non-UHI affected temperature record would show us (in the alternative – there was exceptionally close agreement between a random raw file that included urban sites, and a UHI-adjusted file: I find that to be a credulity stretcher). When investigators have pre-concieved notions about what they will find, they are too quick to accept results that match those expectations: this dataset was simply taken as ‘correctly adjusted” or the mechanism that produced it was thought to be correct – because the results matched expectations. If it had gone the other way, and the adjusted results showed a decline in temperature, I suspect that the algorithims would have been exhaustively tested and recalibrated. In a non-experimental situation, the quality control offered by replication and attempted falsification is absent; allowing inspection of data and code is a step in regaining that quality control.