Hacked E-mails from Climate Scientists released

[KrauserKrauser]

As a complete layman to the programming behind this reading through that article makes me think that the main concern I have with this is not the fact that the climate may or may not be changing, but that depending on a computer model that has artificial corrections, purposeful alteration data trends and buggy as shit code to make policy decisions without any oversight is retarded as hell.

If the data has been historically requested for analysis, been denied and now shown to be purposefully denied by this organization to deny oversight by anyone else, this organization is going to reap the whirlwind for their hubris.

[Edi]

Just how long have all these thousands of emails, code files, reams of data etc been out there in public? When did the hack occur?

Because if this is just days old, all those who are crowing about the "global warming hoax" can go fuck themselves. They could not possibly have analyzed anything in that shit in this short a time and gotten the kind of conclusive results they are saying they have. And fuck James Inhofe for good measure too. With a pineapple up the ass.

This will be a good long while settling before there is anything useful to be gotten out of it. And the rabid anti-global warming crowd is going to try and muddy things up and politicize this at every turn as much as they can, you can be sure of that.

[gizmojumpjet]

Just how long have all these thousands of emails, code files, reams of data etc been out there in public? When did the hack occur?

~11/19/2009

Because if this is just days old, all those who are crowing about the "global warming hoax" can go fuck themselves. They could not possibly have analyzed anything in that shit in this short a time and gotten the kind of conclusive results they are saying they have. And fuck James Inhofe for good measure too. With a pineapple up the ass.

How can you have seen someone claiming that they've gotten conclusive results if you don't even know when the data first came out? Please, link me to someone claiming they've found something conclusive in the data.

This will be a good long while settling before there is anything useful to be gotten out of it.

Please define "a good long while." What in your mind is an appropriate length of time one must spend with this data before starting to make judgments based on the content?

And the rabid anti-global warming crowd is going to try and muddy things up and politicize this at every turn as much as they can, you can be sure of that.

Climate change is already politicized. People who are skeptical of the "consensus" view aren't the only ones politicizing the global warming issue. When scientific findings are used to shape public policy, it's inevitable.

[Alyrium Denryle]

How can other researchers replicate the results if the raw data has gone down the rabbit hole?

By collecting their own you fucking retard. They can also ask for the standardized dataset and the methodology used to do the standardization. They do this all the time. As I have laid out for you, the raw data has only two purposes. Plagiarism, and Criticism Not In Good Faith. When dealing with complex models, the raw datapoints are useless for replication. You are looking to replicate the trend using your own dataset in order to test the robustness of the model. If given starting conditions, you should be able to get the other persons model to predict the temperatures found in your dataset. If there is something you missed, you will see major discrepancies.

That is how these kinds of tests are replicated.

The raw data existed at one time, and now (apparently) does not. That is the definition of "deleted." It has been made to no longer exist.

Deleted 30 years ago because you were running out of space in your filing cabinet for dozens and dozens of dozens of 5.25'' floppy disks is different from actually deleting the data out of spite.

Why shouldn't data you collected using the people's money be available to the people?

Science is not a democratic process. Nor is it something akin to the french revolution. What exactly are you going to do with that data? Nothing but hurl criticism based on ignorance. Having to defend ourselves from that shit increases our already monumental stress levels.

You lack the training or intellectual capacity to see anything but a string of numbers you cannot understand and your eyes glaze over. I am sick and fucking tired of this bullshit intellectual populism you espouse.

We do what we do because we are (for the most part) one of the few groups of people on this planet which has the ability and willingness to

The product of the work I do while on the clock becomes my employer's property; what makes you any different?

I am not your employee, bitch. Scientists do not serve you directly, and the data we obtain is beyond your capacity to analyze. We as individuals have editorial control over our research and between us and the university we work under, own the rights in a legal sense to the intellectual property. This is what the law says.

The work you do for your employer is theirs by contract.

That is the difference.

It doesn't matter; also, can you support your assertion that no actual scientists submitted FOIA requests for climate data from CRU?

That is why I said almost guarantee. There is a chance someone was a scientist. However if that were the case they would have just asked for it instead of using legal compulsion. I support the assertion with professional experience. If I want someone's data all I need to do is ask and typically agree to put the owners name on a paper I publish with it.

The 80's weren't the Dark Age of Computers.

Would you like me to post some computer specs?

If anyone should be on the cutting edge of data retrieval and storage it should be those folks in the business of generating large amounts of data.

When you have data from that many weather stations in a massive file, back in the early 80s? You dont save your raw data going back that long. You compile it to save space.

The fact that they did not or were not able to sounds to me like negligence or incompetence at the very least.

Well you are not exactly making an informed decision on the matter.

Good faith? The content of the leaked emails does not instill a great sense in me that the CRU is operating in good faith, and has rather attempted to subvert the peer review process.

To the extent your claim is true Tu Quoque fallacy. The only way to subvert peer review is if the individual in question is a reviewer of several journals or sits on the editorial board. If that is the case, what typically happens upon editorial rejection is submission to another journal. However, and this is the fun part, the person submitting a manuscript files a conflict of interest form. Individuals who they have worked with in the past or with whom they have personal grudges can be excluded from having editorial control over that manuscript. It would be very very hard to do, and if someone manages to do it and gets caught, their careers are over.

On the same token, if someone actually did falsify data, their careers are over. Done. They will be expelled from their labs and academically black listed. So once the dust settles from this if there is any truth to allegations of data falsification, heads will roll and these people will be lucky to get jobs cleaning glassware in a community college.

Talking about deleting data rather than turn it over to a FOIA request makes it seem to me like we're talking about data that's actually subject to such a request.

Oh please. You have never heard anyone say "I would rather give all my money away to homeless people than give that SOB a dime!" before? It is the same statement with the nouns changed.

Science isn't founded on trust, its founded on data and results that can be replicated and reviewed. But we don't have the original data anymore, we only have the "value-added" data; how can another we determine if the "value-added" dataset is reliable when we cannot replicate the process by which it was arrived at?

You cant. by you I mean lay people. You dont need the original data to replicate the results. It wont do you any good. You need their model and a separate data set.

I'm not making the argument that any Tom, Dick, or Harry can take the raw data and make valid conclusions based on it. There are people who could, however, if only they had access to it.

And the people filing FOIA requests are not the ones who are capable of doing it. The people that are need only ask.

As for your remarks, yes, I think that global-warming deniers tend to misinterpret data to accommodate their pet theories, so I feel that it is in the public interest to not grant them special access to raw scientific data. I feel the same way about creationists as well.

Pretty much.

Kindly refrain from putting words in my mouth. I never said all scientific data should be subject to FOIA requests.

That you have not thought your position through to its obvious conclusion is not his fault.

If someone won't disclose publicly owned data so that it can be reviewed properly, FOIA provides a method by which they can be compelled to do so. That is unless they delete it, which is apparently what they did.

If the data is publicly owned. Now, I do not know what the rules are in the UK, however in the US, data is not publicly owned. I imagine the same is true in the UK. This is why universities can patent intellectual property. As I recall, and I may have missed something there are several thousand emails, at least some of the requests were Denied.

[Bakustra]

I am talking about the general idea of whether scientific data should be protected from FOIA requests or not. You seem to be fixating on the idea that the data should be freely available, whereas I am arguing that the protection of scientific papers from plagiarism is more important.

I wish you'd stop talking in generalities when we are discussing specific datasets. Why do you keep doing that? Stop trying to turn this into a discussion about what limits should be placed on FOIA requests. We're discussing specific data that apparently is/was subject to such a request. Whether or not you or I think it should have been subject to such a request is immaterial to the discussion at hand.

In other words, I support the idea of making data freely available after a certain point in general. Of course, you are avoiding the implications of your own statements: if FOIA applies to all scientific data, then what stops unscrupulous individuals from simply using FOIA requests to steal the material. I am not familiar with the means of climatology and the analysis of temperatures, but what you seem to be saying is that scientific papers are invalid if all the raw data used is not published with them.

Kindly refrain from putting words in my mouth. I never said all scientific data should be subject to FOIA requests.

Would you be willing to provide more context on what these "MMs" mentioned in the e-mail are? You seem to be claiming that these are scientists that aren't receiving any of the data, especially given that the person in question mentions sending station data to another individual in the same e-mail.

The MM's referenced are Stephen McIntyre and Ross McKitrick which anyone with half a brain could have determined for themselves.

Justify this, please. I also ask you to point me to the papers that have been published

http://www.cru.uea.ac.uk/cru/pubs/full/

and ask you why you think that FOIA is an appropriate tool for ensuring scientific openness if you do not support the idea that unqualified persons are equally valid in their conclusions.

If someone won't disclose publicly owned data so that it can be reviewed properly, FOIA provides a method by which they can be compelled to do so. That is unless they delete it, which is apparently what they did.

As for your remarks, yes, I think that global-warming deniers tend to misinterpret data to accommodate their pet theories, so I feel that it is in the public interest to not grant them special access to raw scientific data. I feel the same way about creationists as well.

It's not possible to misinterpret data that has been deleted or to which you otherwise have no access. It's also not possible to duplicate results if you don't have access to the data.

I will say that I have perhaps hijacked the thread slightly by referring to my personal opinions on the usefulness of FOIA when dealing with scientific data. I will note that you seem to have modified your position slightly, so if you would be so good as to simply lay out your position with regards to who should have access to raw scientific data.

Of course, I have noted, here and in your other posts in this thread, a generalized accusation of dishonesty towards the scientists involved. I would like you to back up this argument, that all the people involved in the CRU and the people they have been e-mailing back and forth with are such a monolithic block that you can condemn them all on the director's statements involving the FOIA. Furthermore, I see you implying that the data involved was reviewed improperly by the climatology journals that it was published in. I would like you to justify this as well, if you can.

Of course, I fully expect you to take advantage of the fact that you never stated this outright and retreat from the implications.

Further, when somebody asks you to provide a paper, that doesn't mean simply linking to a list of all the papers published by the Climate Research Unit. You have read through at least some of the stolen e-mails, provide an actual, concise list of papers that the data in question would have been used in, or a statement that would indicate it was used in all the papers, please.

[gizmojumpjet]

I will say that I have perhaps hijacked the thread slightly by referring to my personal opinions on the usefulness of FOIA when dealing with scientific data. I will note that you seem to have modified your position slightly, so if you would be so good as to simply lay out your position with regards to who should have access to raw scientific data.

I haven't modified my position at all: Deleting data that's subject to FOIA requests is illegal; threatening to do it is unethical and possibly criminal as well. I'm not a lawyer. I think that scientific data should be made available to the public that funded the research within certain reasonable restrictions. I don't, for example, have a problem with the delays in Hubble data that you've mentioned, or that data that has national security implications should freely available.

I would like you to back up this argument, that all the people involved in the CRU and the people they have been e-mailing back and forth with are such a monolithic block that you can condemn them all on the director's statements involving the FOIA. Furthermore, I see you implying that the data involved was reviewed improperly by the climatology journals that it was published in. I would like you to justify this as well, if you can.

Of course, I fully expect you to take advantage of the fact that you never stated this outright and retreat from the implications.

I would be happy to back up that argument if that was the argument I have been making, which it is not. Your habit of putting words in my mouth is getting tiresome.

Further, when somebody asks you to provide a paper, that doesn't mean simply linking to a list of all the papers published by the Climate Research Unit. You have read through at least some of the stolen e-mails, provide an actual, concise list of papers that the data in question would have been used in, or a statement that would indicate it was used in all the papers, please.

The raw data being referenced is that which was used to compile the HadCRUT3 dataset, available for download here, and introduced in this paper.

[Bakustra]

I will say that I have perhaps hijacked the thread slightly by referring to my personal opinions on the usefulness of FOIA when dealing with scientific data. I will note that you seem to have modified your position slightly, so if you would be so good as to simply lay out your position with regards to who should have access to raw scientific data.

I haven't modified my position at all: Deleting data that's subject to FOIA requests is illegal; threatening to do it is unethical and possibly criminal as well. I'm not a lawyer. I think that scientific data should be made available to the public that funded the research within certain reasonable restrictions. I don't, for example, have a problem with the delays in Hubble data that you've mentioned, or that data that has national security implications should freely available.

My apologies then. You seemed to have gone from saying that scientific data should be available to the public, then "not to every Tom, Dick, and Harry" then back again to publicly available. My mistake.

I would like you to back up this argument, that all the people involved in the CRU and the people they have been e-mailing back and forth with are such a monolithic block that you can condemn them all on the director's statements involving the FOIA. Furthermore, I see you implying that the data involved was reviewed improperly by the climatology journals that it was published in. I would like you to justify this as well, if you can.

Of course, I fully expect you to take advantage of the fact that you never stated this outright and retreat from the implications.

I would be happy to back up that argument if that was the argument I have been making, which it is not. Your habit of putting words in my mouth is getting tiresome.

If I have misrepresented your argument, my apologies. Why, if I my ask, did you include asides such as "so that it can be reviewed properly?" Why did you use this as a reason why we shouldn't trust the CRU, and tar the entire group and the people e-mailed by the group, with the same brush? If you are not attacking the conclusions of the CRU or subscribing to the conspiracy theories about global warming, then what are you doing, if I may ask?

Further, when somebody asks you to provide a paper, that doesn't mean simply linking to a list of all the papers published by the Climate Research Unit. You have read through at least some of the stolen e-mails, provide an actual, concise list of papers that the data in question would have been used in, or a statement that would indicate it was used in all the papers, please.

The raw data being referenced is that which was used to compile the HadCRUT3 dataset, available for download here, and introduced in this paper.

Thank you.

[MKSheppard]

Having read some more on this in various blogs; I believe that the emails are just the fluffy delicious icing on the cake. The real yumminess is in the programming source code for various things that was included along with the emails.

For example, teh Harry_Read_me file is a real gold mine that's just only being exploited. As someone posted on Climate Audit:

Yes, I think Harry has been told to try and replicate their current datasets and especially the gridded output databsets.

This starts with station data from GHCN and other places, which gets processed (to make .cts files). These then get merged into a ‘database’. The database consisting of .dtb/.dts and other files. This isn’t database, in the sense that a programmer in the 21st century might think of a database. we aren’t talking realional or SQL here. Hell there isn’t even any sign of indexing. This is how universities did computing back in the 1970s. data came in on cards or tapes, and you’d process them to produce a new pile of cards or output on tape.

To produce the gridded datasets, the .dtb file data gets converted to some text files. These can then be read in by some IDL scripts. IDL is a higher level llanguage than Fortran, has lots of special graphics stuff. And most usefully has a routine for triangulating data. This IDL routine is what does the interpolation between station data points to generatee the gridded data points.

All of this, is explained by Tim in the readme’s. Harry seems to be so offended by the fact that a readme starts with an underscore, that he doesn’t actually read them.

It’s my guess that there are no professional software developers at CRU. Nor are there any professional data librarians or archivists. The code and data just exists, a communal pile of junk. If a postgraduate needs something they just have to go and code it for themselves. In practice, in this environment, some poor sod gets a reputation as the person who know how to make the computers work. This was someone called Mark, then Tim got lumbered, and now young Harry has had the baton passed to him.

what yo uhave to consider is that CRU isn’t a governemnt agency, say like the office of national Statistics or the met Office. They are basically a university department. Status for a university comes in terms of PhDs produced, papers authored and so on. Money isn’t going to be taken away from that, in order to fund someone managing the data properly.

And anyway, PhD students are a hell of a lot cheaper than professioanl software developers

and

Raven, on my own computer I have lots of experimental results that sit in several distinct data files (because they are measures of different things, resulting from different phases of some protocol). Programs then load up some subset of them, as required for what I’m doing at the time.

This is just a guess, but I imagine they had the same kind of situation. CRU didn’t simply maintain a worldwide panel on temparatures, they also had research teams putting that together with other datasets for the specific work of various papers.

So various data of various kinds lay about on the hard drives. Whoever did the actual computer work knew what to load up for particular analyses. They didn’t document that knowledge well, and when they left it was effectively gone.

I have sometimes received old data (and here I mean just five years old) from people who cannot precisely remember what columns are what; or in several different files corresponding to several different treatments; and the original authors confess they are unsure which file is which treatment. In such cases I have to go to their papers and experiment–”audit” their results–until I am confident I know what’s what.

I’m afraid this kind of thing is pretty common with low-stakes data sets from experiments that were relatively cheap. In this case, with a data set so obviously central to many hundreds of academic projects, it is pretty amazing. (If indeed we are seeing a sort of breakdown in the passing on of knowledge about that temperature data set.)

and

Judging from the comments in HARRY_READ_ME.txt putting together the database from all the input was an awful chore spanning three years. And this exposes what I had suspected was a very real problem. You have several different people doing climate research and they all apparently make their own databases from the raw input. The raw input seems to change as stations are added or drop out or numbering systems change and so forth.

What the world needs someplace is a common standard temperature record database that they can all draw from where the data is standardized. It would be a very difficult job but once done, could provide some continuity and make future studies much easier and less expensive. You could have better assurance that people are comparing apples to apples and not constructing databases that differ because of how they handled data input inconsistencies.

Following HARRY_READ_ME.txt is like a techie version of a Stephen King novel and I imagine that just about anyone compiling any kind of climate records writes their own version of it.

[MKSheppard]

I've had a chance to think this over, and I think this leak will be beneficial for the Climate Change scientific community.

In the long run, it'll force a professionalization and better attitude towards data keeping.

But in the short term, a lot of careers which had been built up on teh GIGO are going to get buttfucked without lube.

I predict the buttfucking from their fellow scientists is going to come from non-US/European scientists -- India recently opened up a Climate Change institute, and the Russians have been trying to make inroads into this field for a bit.

The extant groups in the US/Europe will slowly be forced to audit themselves -- and they'll do it kicking and screaming; because they've taken a LOT of money in the form of grants.

For example, IIRC the CRU at East Anglia (them of whom the hack happened to), was slated to be downsized/reduced by their university early in the 1990s, due to a lack of funding. But then they started generating papers based on data; and taking grants from a whole bunch of people; since 1990 they've gotten funding from the following sources:

DEPT OF ENVIRONMENT £179,484.00
DEPT OF ENVIRONMENT £101,914.00
NATO - NORTH ATLANTIC TREATY ORGANISATION £33,678.00
EUROPEAN UNION - EU / CEC £59,446.00
EUROPEAN UNION - EU / CEC £39,833.00
NATIONAL RIVERS AUTHORITY £4,800.00
EUROPEAN UNION - EU / CEC £62,605.00
NERC £104,230.00
US DEPT OF ENERGY £128,000.00
EUROPEAN UNION - EU / CEC £42,464.00
MASSACHUSETTS UNIV. £20,953.00
NATIONAL RIVERS AUTHORITY £19,780.00
NERC £18,639.00
EUROPEAN UNION - EU / CEC £209,843.00
US DEPT OF ENERGY £132,000.00
UNIV CORP ATMOS RES £12,903.00
NERC £24,859.00
US DEPT OF ENERGY £125,000.00
EUROPEAN UNION - EU / CEC £183,333.00
EUROPEAN UNION - EU / CEC £71,333.00
EUROPEAN UNION - EU / CEC £33,333.00
US DEPT OF ENERGY £99,555.00
BRITISH SUGAR PLC/IACR - BROOM'S BARN £8,148.00
NERC £92,383.00
US DEPT OF ENERGY £102,752.00
UK WATER INDUSTRY RESEARCH LTD £5,000.00
US DEPT OF ENERGY £106,151.00
EUROPEAN UNION - EU / CEC £26,809.00
EUROPEAN UNION - EU / CEC £123,600.00
EUROPEAN UNION - EU / CEC £99,492.00
NERC £39,255.00
US DEPT OF ENERGY £212,500.00
NERC £34,660.00
HEFCE / JIF £6,608,541.00
MET OFFICE £45,052.00
EUROPEAN UNION - EU / CEC £178,812.00
SCOTTISH OFFICE £14,164.00
CEC £134,978.00
EPSRC £105,178.00
EPSRC £69,125.00
NERC £226,981.00
MET OFFICE £13,870.00
ENVIRONMENT AGENCY £9,000.00
EUROPEAN UNION - EU / CEC £105,499.00
US DEPT OF ENERGY £262,629.00
MET OFFICE £14,492.00
EUROPEAN UNION - EU / CEC £426,895.00
SCOTTISH ENVIRONMENT PROTECTION AGENCY £730.00
ENVIRONMENT AGENCY £26,516.00
ENVIRONMENT AGENCY £52,177.00
CEC £84,431.00
COUNCIL FOR THE CENTRAL LAB. OF THE RES. COUNCILS £50,000.00

Want to bet some of those people are feeling a bit cheated and may be preparing the lawyer swarm to get some of their money back?

[Edi]

Just how long have all these thousands of emails, code files, reams of data etc been out there in public? When did the hack occur?

~11/19/2009

Right. Less than a week ago, as I suspected.

Because if this is just days old, all those who are crowing about the "global warming hoax" can go fuck themselves. They could not possibly have analyzed anything in that shit in this short a time and gotten the kind of conclusive results they are saying they have. And fuck James Inhofe for good measure too. With a pineapple up the ass.

How can you have seen someone claiming that they've gotten conclusive results if you don't even know when the data first came out? Please, link me to someone claiming they've found something conclusive in the data.

Since you like to go out of your way to be a shitlicking pigfucker, I'll treat you like one and use really small words so that they have a chance of penetrating through your thick skull:

I suspected that the leak of the emails and other stuff was just a few days old, which you kindly confirmed. We're already seeing all kinds of bullshit from global warming skeptics about this, when they could not possibly have analyzed all that data or the source code to get any kind of relevant results. Yet the bullshit is still coming on.

Kindly refer to the link Norseman posted on the first page about whackadoodles.

This will be a good long while settling before there is anything useful to be gotten out of it.

Please define "a good long while." What in your mind is an appropriate length of time one must spend with this data before starting to make judgments based on the content?

Enough time and enough expertise to be able to analyze it with more than just "I skimmed bits and pieces and cherry-picked the stuff that sounds worst when taken out of context", which is what we've mostly seen so far. A friend of mine does cancer research and a lot of that is analysis of results coming from the lab, working with databases and other such stuff and unless you really know what the fuck you're doing, the raw data means zilch. You need to know what to look for, where to look for it and how it relates to other things before you can get anything conclusive and it sure as fuck doesn't happen by skimming.

And the rabid anti-global warming crowd is going to try and muddy things up and politicize this at every turn as much as they can, you can be sure of that.

Climate change is already politicized. People who are skeptical of the "consensus" view aren't the only ones politicizing the global warming issue. When scientific findings are used to shape public policy, it's inevitable.

So they should be given a free pass for stirring up shit and making trouble, when many of them are engaged in outright lies? Fuck that.

I agree with Shep that this will eventually settle down, but if there's asskicking to be done, it'll be other scientists who do it, not some ignorant asshats with delusions of competence.

[Alyrium Denryle]

Climate change is already politicized. People who are skeptical of the "consensus" view aren't the only ones politicizing the global warming issue. When scientific findings are used to shape public policy, it's inevitable.

And you think that is OK? That people should be permitted to sling shit at scientists because they dont like what the Universe tells said scientists, to the point that said scientists cannot do their jobs without having their work Edited by lay-plebes who are informed not by data by ideology? Seriously?

[phongn]

Having read some more on this in various blogs; I believe that the emails are just the fluffy delicious icing on the cake. The real yumminess is in the programming source code for various things that was included along with the emails.

For example, teh Harry_Read_me file is a real gold mine that's just only being exploited. As someone posted on Climate Audit

Unfortunately, the things you posted are distressingly common in science these days; essentially we're forcing scientists to become programmers - and not particularly well-trained ones at that. Asking them to do actual software engineering is far beyond them or their training and education. I see this all the time where new grad students and postdocs are given a week-long programming course and then thrown into the deep end.

[KrauserKrauser]

What's even better is we are requiring them to be computer programmers and then making plans and legislation based on the resulting computer models. Brilliant!

[Alyrium Denryle]

What's even better is we are requiring them to be computer programmers and then making plans and legislation based on the resulting computer models. Brilliant!

Well the models are fine. The models produced manage to accurately predict past and current temperatures. They are just a bitch for new people to operate.

The major problem is one of redundancy and fine tuning. As it is each lab has to create their own complete model rather than being able to tweak or modify an existing one.

The code is a mess. However it is not the sort of mess that would invalidate the science.

I think.

[phongn]

Well the models are fine. The models produced manage to accurately predict past and current temperatures. They are just a bitch for new people to operate.

The major problem is one of redundancy and fine tuning. As it is each lab has to create their own complete model rather than being able to tweak or modify an existing one.

That's terrible practice. Why is everyone recreating the wheel? Every line of code written is another chance (a high chance) of another bug; shouldn't there be a pool of tested, validated and high-quality modeling code?

The code is a mess. However it is not the sort of mess that would invalidate the science.

I think.

Putting on my software hat, it doesn't look good.

[KrauserKrauser]

Because if I learned anything in my AP Coputer Science class ten years ago it was that busted codes ALWAYS results in working computer models. ALWAYS!

So often in fact that you should base the world's energy production plans on the outcome of this fucked code containing model. Because it's not like the model is entirely dependent on the code or anything...

As to them accurately predicting the temperature changes? WTF? We've been cooling in the last few years regardless of the continued spiking of the CO2 levels, obviously this does not disprove MMGW but the computer models of the previous years all predicted a marked rise in temperature in the last few years IIRC.

[Alyrium Denryle]

That's terrible practice. Why is everyone recreating the wheel? Every line of code written is another chance (a high chance) of another bug; shouldn't there be a pool of tested, validated and high-quality modeling code?

That is what I was getting at. There should be a standard template, a baseline set of code upon which models are constructed.

Instead the wheel gets reinvented over and over and over again by people doing jury rigged programing that, while it does what is is supposed to do, is terribly inefficient, buggy, and difficult to examine. Same problem happens in biology but it is not as bad because a lot of the time the programs eventually get worked over by a decent software engineer and published. This is how we have MEGA, Genepop etc.

As to them accurately predicting the temperature changes? WTF? We've been cooling in the last few years regardless of the continued spiking of the CO2 levels, obviously this does not disprove MMGW but the computer models of the previous years all predicted a marked rise in temperature in the last few years IIRC.

There are two options. The code does not function OR there is something else going on in the complex system which is also likely.

So often in fact that you should base the world's energy production plans on the outcome of this fucked code containing model. Because it's not like the model is entirely dependent on the code or anything...

Just because the code is fucked up, inefficient and buggy does not mean it cannot function. Plenty of programs were not fully worked over before release. All you need to do is go down to G&C where they talk about games and such to see that.

The question is whether or not the fucked up part of the code effects the predictions of the model.

This has not been as far as I know, evaluated.

[Serafine666]

So they should be given a free pass for stirring up shit and making trouble, when many of them are engaged in outright lies? Fuck that.

And you think that is OK? That people should be permitted to sling shit at scientists because they dont like what the Universe tells said scientists, to the point that said scientists cannot do their jobs without having their work Edited by lay-plebes who are informed not by data by ideology? Seriously?

It's worth noting that the statement you're both quoting doesn't attempt to justify any actions of the skeptics. It didn't seem to be defending them, justifying their shit, or saying anything that would lead to demanding of gizmojumpjet whether he's OK with the way skeptics have conducted themselves. When data is used to justify political policy, the data has become politicized whether the scientists intended it or not and when both sides start slinging over-the-top rhetoric (likening skeptics to Holocaust deniers, opining that we should drag an airline executive out of his office and shoot him for every person who dies by climate-related drowning), it becomes slightly dishonest to pretend that one side is pure as the driven snow, doing no wrong and having nothing to do with the politics while the other side is rabid politically-obsessed haters and evil besides. Nothing justifies the extremes of EITHER side... much less someone correctly pointing out that the process has already become political.

[Edi]

The data itself is not political. It just is. A description of reality. The response in what to do with it, how to react to it, that's a political issue.

But when some asshat just flat out lies about it because he does not like the consequences of the facts, it's time to pull off the gloves and tell them to fuck off. If they're willing to lie in order to suppress actual discussion of the facts or to derail it, they should be tossed out on their ear and excluded from making decisions on the issue.

That's the point. A denier of facts is not a legitimate participant in a discussion of those same facts when they have been established. There are enough real issues and problems that shit-stirrers like that are simply not needed.

[Alyrium Denryle]

Nothing justifies the extremes of EITHER side... much less someone correctly pointing out that the process has already become political.

Except that is not what he is doing. He is crying conspiracy with regard to missing raw data and insisting that The People (read: politically motivate laypeople) should be able to "review" said data. In other words he is insisting that people be given free license to sling mud for political gain and to derail a meaningful scientific discussion of what is going on in the universe. That kind of bullshit should not be permitted.

[Serafine666]

The data itself is not political. It just is. A description of reality. The response in what to do with it, how to react to it, that's a political issue.

Precisely. The data isn't itself political but when it is used in a political way, it has become politicized.

But when some asshat just flat out lies about it because he does not like the consequences of the facts, it's time to pull off the gloves and tell them to fuck off. If they're willing to lie in order to suppress actual discussion of the facts or to derail it, they should be tossed out on their ear and excluded from making decisions on the issue.

That's the point. A denier of facts is not a legitimate participant in a discussion of those same facts when they have been established. There are enough real issues and problems that shit-stirrers like that are simply not needed.

Indeed they are not. The misfortune of this debate is that when it finally reaches the point where the significant scientific objections of the few responsible skeptics have been answered (or confirmed), the political policies that assume catastrophic climate change will have been in place for years and the economic prosperity they've traded to avert the terrors that have been predicted cannot be retrieved. I think, in a certain sense, the possibility that harm will be done for no gain is what motivates some skeptics to lose their minds and start doing stupid shit... like hacking email accounts and treating their contents as the final nail in the coffin.

[Serafine666]

Except that is not what he is doing. He is crying conspiracy with regard to missing raw data and insisting that The People (read: politically motivate laypeople) should be able to "review" said data. In other words he is insisting that people be given free license to sling mud for political gain and to derail a meaningful scientific discussion of what is going on in the universe. That kind of bullshit should not be permitted.

I would hardly think he needs to justify a process that is already ongoing. The fact that political policies are already in the works without all relevant data on hand is why we have mud-slinging for political gain: one side thinks it suspicious that the other is working as fast as possible to get a policy in place, especially since many political policies (at least in the United States) become increasingly difficult to implement the longer the populace has to become well-informed.

[MKSheppard]

I agree with Shep that this will eventually settle down, but if there's asskicking to be done, it'll be other scientists who do it, not some ignorant asshats with delusions of competence.

By that logic; well, lots of things aren't credible.

We know how the media uses "battleships" to refer to anything larger than a rowboat. So any analysis the media does in regards to defense issues is not credible and they are nothing but ignorant asshats.

But I digress. Back onto the topic at hand.

The source code included in the leak is the real smoking howitzer in the whole thing. The emails are like I said, garnishing on the cake.

The truly delicious stuff is in the comments in the source code; which are chock full of statements like:

Code: Select all

function mkp2correlation,indts,depts,remts,t,filter=filter,refperiod=refperiod,$
datathresh=datathresh
;
; THIS WORKS WITH REMTS BEING A 2D ARRAY (nseries,ntime) OF MULTIPLE TIMESERIES
; WHOSE INFLUENCE IS TO BE REMOVED. UNFORTUNATELY THE IDL5.4 p_correlate
; FAILS WITH >1 SERIES TO HOLD CONSTANT, SO I HAVE TO REMOVE THEIR INFLUENCE
; FROM BOTH INDTS AND DEPTS USING MULTIPLE LINEAR REGRESSION AND THEN USE THE
; USUAL correlate FUNCTION ON THE RESIDUALS.
;

pro maps12,yrstart,doinfill=doinfill
;
; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions
; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
;

Code: Select all

; Plots (1 at a time) yearly maps of calibrated (PCR-infilled or not) MXD
; reconstructions
; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.

or this

Link

From the CRU code file osborn-tree6/briffa_sep98_d.pro , used to prepare a graph purported to be of Northern Hemisphere temperatures and reconstructions.

Code: Select all

;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,- 0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
;
yearlyadj=interpol(valadj,yrloc,timey)

This, people, is blatant data-cooking, with no pretense otherwise. It flattens a period of warm temperatures in the 1940s — see those negative coefficients? Then, later on, it applies a positive multiplier so you get a nice dramatic hockey stick at the end of the century.

Alright, enough posting the fun stuff.

Now, I've always found the rationale that the Climate guys have used to avoid releasing most of their data to the public quite....dodgy, to use a Britishism.

I can understand sitting on the data a bit, or only releasing a select bit or two before the stuff is published -- I'm doing that right now with my research at Archives II in support of a "American Secret Projects Book" by Tony Buttler.

But once the papers have been published, credit has been assigned, etc what possible reason is there to hide everything?

Well, now we know from a random look at the code by outside parties -- it's poorly written, makes CS people and IT people want to drive hot nails into their eyeballs; and all that fun stuff.

Now, this isn't a problem confined to the climate change scientific community; as someone commented at Link.

That kind of quality is not unusual for code written by scientists with no training in computer science or software engineering. Pretty much the only thing that matters is getting the calculation right (hmm… or not, as the case may be). You don’t get style points for usable/flexible/elegant code when you publish the paper. A lot of the stuff is terrible, even the packages with expensive licenses. I’ve seen a C program that used ‘YES’ and ‘NO’ strings for booleans and strcmp for comparing the values, in time-critical code (the code was written in an academic group in the molecular modeling / drug discovery field).

The real big problem comes when you compare this with the HARD SCIENCE disciplines. I've been exposed to a lot of the HARD SCIENCES as a result of my research at Archives II.

The typical aircraft design proposal to meet a set of government requirements (for example, OS-130 was the requirement for a VF Day Fighter that led to the F8U Crusader); consists of several binders usually.

You have the summary, which has the basic stuff about the plane, it's dimensions, a three view drawing, expected performance. The basic stuff you'd need to put it into a wargame.

Then come the binders full of ENJUNEERING stuff, with hundreds of pages of graphs drawn by engineers (in the days before computer printers) showing lift, drag, center of gravity, etc etc etc etc. Then come the boxes of blueprints for the proposed aircraft; showing in great detail various stuff, like how the airframe structure is put together.

And the engineering binders detail in great detail the various equations that the engineers used to come up with their results; what their inputs were, etc.

These huge proposal files get sent to either Wright Patterson (for USAF) or to whatever equivalent Command the Navy has; where the services' inhouse engineers pore over the data given to them by the contractor, because contractor's early estimates are quite.....overoptimistic.

In one famous instance, Lockheed once withdrew from a competition after a very embarassing error was discovered in it's proposal.

When Boyd returned to the Pentagon, a full colonel was waiting. He chewed Boyd out for insulting a senior officer. He said the general in charge of Research and Development for the Air Force was so angry that he was about to transfer Boyd to Alaska. The two men marched down the hall to the general's office. Boyd was still carrying the briefcase he had taken to Wright-Pat.

"Major Boyd, I have just one question," the general said. "Did you tell that colonel at Wright-Pat he was a lying fucker?"

"Yes, Sir, I did."

"You are out of here. You are being transferred." The general launched his own chewing-out session about respecting senior officers and insubordination and how lucky Boyd was that he was only being transferred. When he paused, Boyd said, "Sir, do you want to know why I said that?"

"No."

"I think you do. Give me one minute." He opened his briefcase.

Reluctantly, the general looked at the drag polars. "Know how to read these, General?"

"Yes."

The general moved his finger over the polars. "They're saying ..."

"Yes, Sir, they are saying the smaller the wing, the greater the lift."

"That means ..."

"Yes, Sir, that means the greatest lift would come if there were no wing at all."

The general picked up his telephone and called Wright-Pat. And Boyd swears that as the general picked up the phone, he muttered, They are lying fuckers."

Originally four entrants were in the IFA Competition: LTV's V-1000, McDonnell Douglas' F-4E(F), Northrop's F-5E, and Lockheed's CL-1200. Lockheed dropped out of the competition because it was unwilling to admit to an early miscalclation in an aeronautical equation that, if believed, would have required its airplane design to be chained to the runway to keep it from flying while the engine was off.

I've also been exposed to the design process used by NAVSHIPS to design cruisers -- there are several boxes of stuff at Archives II detailing the design of the Typhon guided missile ships -- and entire boxes, as in several linear feet of archive boxes are full of hundreds of pages of computer printouts listing nothing but the numbers going in and out of the programs they used to calculate the hull stresses for the Typhon cruisers/destroyers/frigates.

Anyway back to the argument at hand. When the government looks into buying military hardware like a new aircraft or warship -- it demands a very high standard of transparency from the contractor(s) or internal design departments (until the late 60s, virtually all of the USN's warships were designed inhouse by the USN).

The same thing undoubtly occurs in the nuclear power field -- I haven't been exposed to the work generated from that; but the nuclear field is very intensive about double, triple and quadruple checking work, making sure everything is all good, the code used to calculate shielding is sound, etc; So that your new nuclear reactor design doesn't melt down the moment power is drawn from it.

But when you look at the Climate Change field; it's opaque and closed off from external scrutiny.

"My models say that x will happen!"

"Um, can we see the code that says this result?"

"NEVAH!"

We demand transparency in the design process for aircraft and ships the government buys, and for the design of nuclear reactors; but when it comes to climate change models, transparency is a dirty word.

This really wouldn't be a problem if it was just a bunch of guys prognosticating eternal doom or selling quack science -- my new design produces FREE ENERGY FOREVER -- those guys can be ignored.

But these guys' work is the foundation for a lot of proposed draconian laws and regulations, like Cap and Trade, Carbon Credits, etc etc etc that would severely restrict economic and industrial output all across the world.

It's for precisely this lack of openness and transparency that the Indians have recently started up their own climate change unit; to provide their government and politico-economic leaders with information that isn't selectively biased.

[MKSheppard]

To elaborate further on the transparency issue; if you're going to propose such draconian acts like cutting emissions by 75% by 2040; etc, I'd damn well like to see all your data before I go on such a draconian course of action.

You're not showing me any of it?

*files under quack file, next to guy who claims he can get 200 MPG cars by simply changing carburettor design*

Normally, this process would have been done in the beginning, but it has been short circuited via politicization; to the point that it's only happening now, after a lot of money has been paid off to scientific centers, and PhDs have been generated based on these assumptions.

It really is going to be pretty damn brutal.

[Alyrium Denryle]

The truly delicious stuff is in the comments in the source code; which are chock full of statements like:

Good thing that The Shep has no idea what they are referring to. They are referring to the methods they used to statistically adjust certain parts of their data to make it useable.

There were IIRC problems with raw data, they were aware of it, and statistically corrected the data to match what was actually true.

In other sections you posted they were referring to doing allometry, which depending on their dataset is entirely appropriate depending on what that particular code is doing.