Proving AGW just got NP hard.

[GreyGeek]

http://prl.aps.org/accepted/L/0707aY...6bf3f1fa9e8bc6

Extracting dynamical equations from experimental data is NP hard

Toby S. Cubitt, Jens Eisert, and Michael M. Wolf
Accepted Thursday Feb 16, 2012
The behavior of any physical system is governed by its underlying dynamical equations. Much of physics is concerned with discovering these dynamical equations and understanding their consequences. In this work, we show that, remarkably, identifying the underlying dynamical equation from any amount of experimental data, however precise, is a provably computationally hard problem (it is NP-hard), both for classical and quantum mechanical systems. As a by-product of this work, we give complexity-theoretic answers to both the quantum and classical embedding problems, two long-standing open problems in mathematics (the classical problem, in particular, dating back over 70 years).

http://news.sciencemag.org/scienceno...s-is-hard.html

Mathematicians recognize a set of truly hard problems that can't be simplified, Cubitt explains. They also know that these problems are all variations of one another. By showing that the challenge of turning physics data into equations is actually one of those problems in disguise, the team showed this task is also truly hard. As a result, any general algorithm that turns a data set into a formula that describes the system over time can't be simplified so that it can run on a computer, the team reports in an upcoming issue of Physical Review Letters.

That makes it doubly difficult for AGW theorists because they use the AR(1) model (first order autoregression), a time-series modeling tool, with the assumption the next year's temperature is dependent on this year's temperature, but not on any of the previous years. Previous years data, even paleodata, is collected so that the assumption, among many others, can be applied. The goal of regression is to select the parameters of the model so as to minimize the sum of the squared residuals. This is referred to as ordinary least squares (OLS) estimation and results in best linear unbiased estimates (BLUE) of the parameters if and only if the Gauss-Markov assumptions are satisfied, i.e., all errors have the same variance, and any two different values of the error term are drawn from "uncorrelated" distributions in which coefficients are not allowed to depend on the underlying coefficients which are not unobservable, hence the term "synthetic".

It raises the question many have asked before: just what are those computer models that project dynamic atmospheric behavior 50 years into the future actually computing? After I read the HARRY_README.TXT file it had me wondering as well. But, when I read the 1,072 emails that the whistle blower released I understood what the game was.

[NickStone]

Instead of trying to argue the causes of global warming / climate change I personally think that we should be cleaning up our planet. To stop pumping out pollutants in to the atmosphere or rivers/seas and cutting trees on a massive scale as the planet needs trees to produce oxygen.

[Ole Juul]

Instead of trying to argue the causes of global warming / climate change I personally think that we should be cleaning up our planet. To stop pumping out pollutants in to the atmosphere or rivers/seas and cutting trees on a massive scale as the planet needs trees to produce oxygen.

I think that's the core of the argument. Those that are "pumping out pollutants" or profiting from it, are not so eager to stop doing that. Around here we should be thinking about spring, but it was -19° C (minus 2° F) the other night and there is lots of snow, including on the road. It's wonky all right! But I'm still not convinced that we can do anything about it other than put another log on the fire and leave the long underwear on at night. My take is that AGW is more about how long we can discuss it before we need to get off our butts and do some of the things we should have been doing anyway - regardless of the reason for doing them.

[NickStone]

... Around here we should be thinking about spring, but it was -19° C (minus 2° F) the other night and there is lots of snow, including on the road...

Spring! in my calendar it is still winter. So when does Spring start where you live?

[Ole Juul]

Spring! in my calendar it is still winter. So when does Spring start where you live?

We've not seen the ground (or the road) since November. I consider it spring when the snow is gone, but things should be starting to grow around April. Who knows what it will be this year. Last "summer" we had 5 months of almost constant 30mph wind, and that was unusual too. Historical data seems completely irrelevant these days. I think GG is right, there's no "formula" that's going to work here.

[GreyGeek]

I finally got around to checking out the FOIA files released late last year. There are over 5,000 emails (out of 220,000 - most of which are still locked up).
It is introduced by the README.txt. In it the numbers in carrots refer to the number of the text file containing the quote taken from that email posted by the person whose name is given.

I've often wondered who released the files. The "intro" to the README.txt file suggests that it is a person who believes that AGW is a threat, but that those who are representing the theory are less then honest or ethical.

/// FOIA 2011 -- Background and Context ///


"Over 2.5 billion people live on less than $2 a day."

"Every day nearly 16.000 children die from hunger and related causes."

"One dollar can save a life" -- the opposite must also be true.

"Poverty is a death sentence."

"Nations must invest $37 trillion in energy technologies by 2030 to stabilize
greenhouse gas emissions at sustainable levels."

Today's decisions should be based on all the information we can get, not on
hiding the decline.

This archive contains some 5.000 emails picked from keyword searches. A few
remarks and redactions are marked with triple brackets.

The rest, some 220.000, are encrypted for various reasons. We are not planning
to publicly release the passphrase.

We could not read every one, but tried to cover the most relevant topics such
as...





/// The IPCC Process ///

<1939> Thorne/MetO:

Observations do not show rising temperatures throughout the tropical
troposphere unless you accept one single study and approach and discount a
wealth of others. This is just downright dangerous. We need to communicate the
uncertainty and be honest. Phil, hopefully we can find time to discuss these
further if necessary [...]


<3066> Thorne:

I also think the science is being manipulated to put a political spin on it
which for all our sakes might not be too clever in the long run.


<1611> Carter:

It seems that a few people have a very strong say, and no matter how much
talking goes on beforehand, the big decisions are made at the eleventh hour by
a select core group.


<2884> Wigley:

Mike, The Figure you sent is very deceptive [...] there have been a number of
dishonest presentations of model results by individual authors and by IPCC [...]


<4755> Overpeck:

The trick may be to decide on the main message and use that to guid[e] what's
included and what is left out.


<3456> Overpeck:

I agree w/ Susan [Solomon] that we should try to put more in the bullet about
"Subsequent evidence" [...] Need to convince readers that there really has been
an increase in knowledge - more evidence. What is it?


<1104> Wanner/NCCR:

In my [IPCC-TAR] review [...] I crit[i]cized [...] the Mann hockey[s]tick [...]
My review was classified "unsignificant" even I inquired several times. Now the
internationally well known newspaper SPIEGEL got the information about these
early statements because I expressed my opinion in several talks, mainly in
Germany, in 2002 and 2003. I just refused to give an exclusive interview to
SPIEGEL because I will not cause damage for climate science.


<0414> Coe:

Hence the AR4 Section 2.7.1.1.2 dismissal of the ACRIM composite to be
instrumental rather than solar in origin is a bit controversial. Similarly IPCC
in their discussion on solar RF since the Maunder Minimum are very dependent on
the paper by Wang et al (which I have been unable to access) in the decision to
reduce the solar RF significantly despite the many papers to the contrary in
the ISSI workshop. All this leaves the IPCC almost entirely dependent on CO2
for the explanation of current global temperatures as in Fig 2.23. since
methane CFCs and aerosols are not increasing.



<2009> Briffa:

I find myself in the strange position of being very skeptical of the quality of
all present reconstructions, yet sounding like a pro greenhouse zealot here!


<2775> Jones:

I too don't see why the schemes should be symmetrical. The temperature ones
certainly will not as we're choosing the periods to show warming.


<1219> Trenberth:

[...] opposing some things said by people like Chris Landsea who has said all the
stuff going on is natural variability. In addition to the 4 hurricanes hitting
Florida, there has been a record number hit Japan 10?? and I saw a report
saying Japanese scientists had linked this to global warming. [...] I am leaning
toward the idea of getting a box on changes in hurricanes, perhaps written by a
Japanese.


<0890> Jones:

We can put a note in that something will be there in the next draft, or Kevin
or I will write something - it depends on whether and what we get from Japan.


<0170> Jones:

Kevin, Seems that this potential Nature paper may be worth citing, if it does
say that GW is having an effect on TC activity.


<0714> Jones:

Getting people we know and trust [into IPCC] is vital - hence my comment about
the tornadoes group.


<3205> Jones:

Useful ones [for IPCC] might be Baldwin, Benestad (written on the solar/cloud
issue - on the right side, i.e anti-Svensmark), Bohm, Brown, Christy (will be
have to involve him ?)


<4923> Stott/MetO:

My most immediate concern is to whether to leave this statement ["probably the
warmest of the last millennium"] in or whether I should remove it in the
anticipation that by the time of the 4th Assessment Report we'll have withdrawn
this statement - Chris Folland at least seems to think this is possible.





/// Communicating Climate Change ///

<2495> Humphrey/DEFRA:

I can't overstate the HUGE amount of political interest in the project as a
message that the Government can give on climate change to help them tell their
story. They want the story to be a very strong one and don't want to be made
to look foolish.


<0813> Fox/Environment Agency:

if we loose the chance to make climate change a reality to people in the
regions we will have missed a major trick in REGIS.


<4716> Adams:

Somehow we have to leave the[m] thinking OK, climate change is extremely
complicated, BUT I accept the dominant view that people are affecting it, and
that impacts produces risk that needs careful and urgent attention.


<1790> Lorenzoni:

I agree with the importance of extreme events as foci for public and
governmental opinion [...] 'climate change' needs to be present in people's
daily lives. They should be reminded that it is a continuously occurring and
evolving phenomenon


<3062> Jones:

We don't really want the bull**** and optimistic stuff that Michael has written
[...] We'll have to cut out some of his stuff.


<1485> Mann:

the important thing is to make sure they're loosing the PR battle. That's what
the site [Real Climate] is about.


<2428> Ashton/co2.org:

Having established scale and urgency, the political challenge is then to turn
this from an argument about the cost of cutting emissions - bad politics - to
one about the value of a stable climate - much better politics. [...] the most
valuable thing to do is to tell the story about abrupt change as vividly as
possible


<3332> Kelly:

the current commitments, even with some strengthening, are little different
from what would have happened without a climate treaty.
[...] the way to pitch the analysis is to argue that precautionary action must be
taken now to protect reserves etc against the inevitable


<3655> Singer/WWF:

we as an NGO working on climate policy need such a document pretty soon for the
public and for informed decision makers in order to get a) a debate started and
b) in order to get into the media the context between climate
extremes/desasters/costs and finally the link between weather extremes and
energy


<0445> Torok/CSIRO:

[...] idea of looking at the implications of climate change for what he termed
"global icons" [...] One of these suggested icons was the Great Barrier Reef [...]
It also became apparent that there was always a local "reason" for the
destruction - cyclones, starfish, fertilizers [...] A perception of an
"unchanging" environment leads people to generate local explanations for coral
loss based on transient phenomena, while not acknowledging the possibility of
systematic damage from long-term climatic/environmental change [...] Such a
project could do a lot to raise awareness of threats to the reef from climate
change


<4141> Minns/Tyndall Centre:

In my experience, global warming freezing is already a bit of a public
relations problem with the media

Kjellen:

I agree with Nick that climate change might be a better labelling than global
warming

Pierrehumbert:

What kind of circulation change could lock Europe into deadly summer heat waves
like that of last summer? That's the sort of thing we need to think about.





/// The Medieval Warm Period ///

<5111> Pollack:

But it will be very difficult to make the MWP go away in Greenland.


<5039> Rahmstorf:

You chose to depict the one based on C14 solar data, which kind of stands out
in Medieval times. It would be much nicer to show the version driven by Be10
solar forcing


<5096> Cook:

A growing body of evidence clearly shows [2008] that hydroclimatic variability
during the putative MWP (more appropriately and inclusively called the
"Medieval Climate Anomaly" or MCA period) was more regionally extreme (mainly
in terms of the frequency and duration of megadroughts) than anything we have
seen in the 20th century, except perhaps for the Sahel. So in certain ways the
MCA period may have been more climatically extreme than in modern times.





/// The Settled Science ///

<0310> Warren:

The results for 400 ppm stabilization look odd in many cases [...] As it stands
we'll have to delete the results from the paper if it is to be published.


<1682> Wils:

[2007] What if climate change appears to be just mainly a multidecadal natural
fluctuation? They'll kill us probably [...]


<2267> Wilson:

Although I agree that GHGs are important in the 19th/20th century (especially
since the 1970s), if the weighting of solar forcing was stronger in the models,
surely this would diminish the significance of GHGs.
[...] it seems to me that by weighting the solar irradiance more strongly in the
models, then much of the 19th to mid 20th century warming can be explained from
the sun alone.


<5289> Hoskins:

If the tropical near surface specific humidity over tropical land has not gone
up (Fig 5) presumably that could explain why the expected amplification of the
warming in the tropics with height has not really been detected.


<5315> Jenkins/MetO:

would you agree that there is no convincing evidence for kilimanjaro glacier
melt being due to recent warming (let alone man-made warming)?


<2292> Jones:

[tropical glaciers] There is a small problem though with their retreat. They
have retreated a lot in the last 20 years yet the MSU2LT data would suggest
that temperatures haven't increased at these levels.


<1788> Jones:

There shouldn't be someone else at UEA with different views [from "recent
extreme weather is due to global warming"] - at least not a climatologist.


<4693> Crowley:

I am not convinced that the "truth" is always worth reaching if it is at the
cost of damaged personal relationships


<2967> Briffa:

Also there is much published evidence for Europe (and France in particular) of
increasing net primary productivity in natural and managed woodlands that may
be associated either with nitrogen or increasing CO2 or both. Contrast this
with the still controversial question of large-scale acid-rain-related forest
decline? To what extent is this issue now generally considered urgent, or even
real?


<2733> Crowley:

Phil, thanks for your thoughts - guarantee there will be no dirty laundry in
the open.


<2095> Steig:

He's skeptical that the warming is as great as we show in East Antarctica -- he
thinks the "right" answer is more like our detrended results in the
supplementary text. I cannot argue he is wrong.


<0953> Jones:

This will reduce the 1940-1970 cooling in NH temps. Explaining the cooling with
sulphates won't be quite as necessary.


<4944> Haimberger:

It is interesting to see the lower tropospheric warming minimum in the tropics
in all three plots, which I cannot explain. I believe it is spurious but it is
remarkably robust against my adjustment efforts.


<4262> Klein/LLNL:

Does anybody have an explanation why there is a relative minimum (and some
negative trends) between 500 and 700 hPa? No models with significant surface
warming do this


<2461> Osborn:

This is an excellent idea, Mike, IN PRINCIPLE at least. In practise, however,
it raises some interesting results [...] the analysis will not likely lie near to
the middle of the cloud of published series and explaining the reasons behind
this etc. will obscure the message of a short EOS piece.


<4470> Norwegian Meteorological Institute:

In Norway and Spitsbergen, it is possible to explain most of the warming after
the 1960s by changes in the atmospheric circulation. The warming prior to 1940
cannot be explained in this way.





/// The Urban Heat Effect ///

<4938> Jenkins/MetO:

By coincidence I also got recently a paper from Rob which says "London's UHI
has indeed become more intense since the 1960s esp during spring and summer".


<0896> Jones:

I think the urban-related warming should be smaller than this, but I can't
think of a good way to argue this. I am hopeful of finding something in the
data that makes by their Figure 3.


<0044> Rean:

[...] we found the [urban warming] effect is pretty big in the areas we analyzed.
This is a little different from the result you obtained in 1990.
[...] We have published a few of papers on this topic in Chinese. Unfortunately,
when we sent our comments to the IPCC AR4, they were mostly rejected.


<4789> Wigley:

there are some nitpicky jerks who have criticized the Jones et al. data sets --
we don't want one of those [EPRI/California Energy Commission meeting].

Jones:

The jerk you mention was called Good(e)rich who found urban warming at
all Californian sites.


<1601> Jones:

I think China is one of the few places that are affected [urban heat]. The
paper shows that London and Vienna (and also New York) are not affected in the
20th century.


<2939> Jones:

[...] every effort has been made to use data that are either rural and/or where
the urbanization effect has been removed as well as possible by statistical
means. There are 3 groups that have done this independently (CRU, NOAA and
GISS), and they end up with essentially the same results.
[...] Furthermore, the oceans have warmed at a rate consistent with the land.
There is no urban effect there.





/// Temperature Reconstructions ///

<1583> Wilson:

any method that incorporates all forms of uncertainty and error will
undoubtedly result in reconstructions with wider error bars than we currently
have. These many be more honest, but may not be too helpful for model
comparison attribution studies. We need to be careful with the wording I think.


<4165> Jones:

what he [Zwiers] has done comes to a different conclusion than Caspar and Gene!
I reckon this can be saved by careful wording.


<3994> Mitchell/MetO

Is the PCA approach robust? Are the results statistically significant? It seems
to me that in the case of MBH the answer in each is no


<4241> Wilson:

I thought I'd play around with some randomly generated time-series and see if I
could 'reconstruct' northern hemisphere temperatures.
[...] The reconstructions clearly show a 'hockey-stick' trend. I guess this is
precisely the phenomenon that Macintyre has been going on about.


<3373> Bradley:

I'm sure you agree--the Mann/Jones GRL paper was truly pathetic and should
never have been published. I don't want to be associated with that 2000 year
"reconstruction".


<4758> Osborn:

Because how can we be critical of Crowley for throwing out 40-years in the
middle of his calibration, when we're throwing out all post-1960 data 'cos the
MXD has a non-temperature signal in it, and also all pre-1881 or pre-1871 data
'cos the temperature data may have a non-temperature signal in it!


<0886> Esper:

Now, you Keith complain about the way we introduced our result, while saying it
is an important one. [...] the IPCC curve needs to be improved according to
missing long-term declining trends/signals, which were removed (by
dendrochronologists!) before Mann merged the local records together. So, why
don't you want to let the result into science?


<4369> Cook:

I am afraid that Mike is defending something that increasingly can not be
defended. He is investing too much personal stuff in this and not letting the
science move ahead.


<5055> Cook:

One problem is that he [Mann] will be using the RegEM method, which provides no
better diagnostics (e.g. betas) than his original method. So we will still not
know where his estimates are coming from.





/// Science and Religion ///

<2132> Wigley:

I heard that Zichichi has links with the Vatican. A number of other greenhouse
skeptics have extreme religious views.


<4394> Houghton [MetO, IPCC co-chair]

[...] we dont take seriously enough our God-given responsibility to care for the
Earth [...] 500 million people are expected to watch The Day After Tomorrow. We
must pray that they pick up that message.


<0999> Hulme:

My work is as Director of the national centre for climate change research, a
job which requires me to translate my Christian belief about stewardship of
God's planet into research and action.


<3653> Hulme:

He [another Met scientist] is a Christian and would talk authoritatively about
the state of climate science from the sort of standpoint you are wanting.





/// Climate Models ///

<3111> Watson/UEA:

I'd agree probably 10 years away to go from weather forecasting to ~ annual
scale. But the "big climate picture" includes ocean feedbacks on all time
scales, carbon and other elemental cycles, etc. and it has to be several
decades before that is sorted out I would think. So I would guess that it will
not be models or theory, but observation that will provide the answer to the
question of how the climate will change in many decades time.


<5131> Shukla/IGES:

["Future of the IPCC", 2008] It is inconceivable that policymakers will be
willing to make billion-and trillion-dollar decisions for adaptation to the
projected regional climate change based on models that do not even describe and
simulate the processes that are the building blocks of climate variability.


<2423> Lanzante/NOAA:

While perhaps one could designate some subset of models as being poorer in a
lot of areas, there probably never will be a single universally superior model
or set of models. We should keep in mind that the climate system is complex, so
that it is difficult, if not impossible to define a metric that captures the
breath of physical processes relevant to even a narrow area of focus.


<1982> Santer:

there is no individual model that does well in all of the SST and water vapor
tests we've applied.


<0850> Barnett:

[IPCC AR5 models] clearly, some tuning or very good luck involved. I doubt the
modeling world will be able to get away with this much longer


<5066> Hegerl:

[IPCC AR5 models]
So using the 20th c for tuning is just doing what some people have long
suspected us of doing [...] and what the nonpublished diagram from NCAR showing
correlation between aerosol forcing and sensitivity also suggested.


<4443> Jones:

Basic problem is that all models are wrong - not got enough middle and low
level clouds.


<4085> Jones:

GKSS is just one model and it is a model, so there is no need for it to be
correct.





/// The Cause ///

<3115> Mann:

By the way, when is Tom C going to formally publish his roughly 1500 year
reconstruction It would help the cause to be able to refer to that
reconstruction as confirming Mann and Jones, etc.


<3940> Mann:

They will (see below) allow us to provide some discussion of the synthetic
example, referring to the J. Cimate paper (which should be finally accepted
upon submission of the revised final draft), so that should help the cause a
bit.


<0810> Mann:

I gave up on Judith Curry a while ago. I don't know what she think's she's
doing, but its not helping the cause


<3594> Berger:

Phil,
Many thanks for your paper and congratulations for reviving the global warming.


<0121> Jones:

[on temperature data adjustments] Upshot is that their trend will increase


<4184> Jones:

[to Hansen] Keep up the good work! [...] Even though it's been a mild winter in
the UK, much of the rest of the world seems coolish - expected though given the
La Nina. Roll on the next El Nino!


<5294> Schneider:

Even though I am virtually certain we shall lose on McCain-Lieberman, they are
forcing Senators to go on record for for against sensible climate policy





/// Freedom of Information ///

<2440> Jones:

I've been told that IPCC is above national FOI Acts. One way to cover yourself
and all those working in AR5 would be to delete all emails at the end of the
process


<2094> Briffa:

UEA does not hold the very vast majority of mine [potentially FOIable emails]
anyway which I copied onto private storage after the completion of the IPCC
task.


<2459> Osborn:

Keith and I have just searched through our emails for anything containing
"David Holland". Everything we found was cc'd to you and/or Dave Palmer, which
you'll already have.


<1473> McGarvie/UEA Director of Faculty Administration:

As we are testing EIR with the other climate audit org request relating to
communications with other academic colleagues, I think that we would weaken
that case if we supplied the information in this case. So I would suggest that
we decline this one (at the very end of the time period)


<1577> Jones:

[FOI, temperature data]
Any work we have done in the past is done on the back of the research grants we
get - and has to be well hidden. I've discussed this with the main funder (US
Dept of Energy) in the past and they are happy about not releasing the original
station data.