Data Availability Bias in Oil Market: Kemp

Why is there such good data about oil in the United States but such poor data about everywhere else?

Accurate information is essential for good decision-making, so it is remarkable how little reliable and timely data exists about the production and consumption of crude oil and refined fuels outside the United States.

The situation in the other advanced economies, not to mention emerging markets, is mostly guesswork.

The result is that oil analysts cannot even agree on production and consumption yesterday and today, let alone predict what will happen tomorrow.

And because the best and most readily accessible data is for the United States, the market puts excessive emphasis on what happens there and neglects developments elsewhere.

The obsession with weekly rig counts, production estimates and crude inventories in the United States as a sign of wider supply-demand trends in the oil market has been a case in point.

But as long as U.S. data is more accurate, detailed and timely than the numbers for other countries, this example of "availability bias" is set to continue.
            
ENERGY CRISIS
Some U.S. data comes from private companies such as Baker Hughes, which inherited the decades-old rig count from the  Hughes Tool Company, but most is produced by the federal government.

The U.S. Energy Information Administration, the independent statistical and analysis arm of the Department of Energy, provides by far the best data on oil and other energy markets anywhere in the world.

The EIA's magnificent data collection and publication effort was born amid bitter recriminations about the federal government's failure to foresee the energy shortages of the 1970s.

In early 1974, just months after Arab countries embargoed oil exports to the United States, more U.S. voters blamed oil companies and their own government for shortages and price rises than the oil exporters.

"The real trouble in the energy crisis is that the government does not seem to know how serious the shortage will be", according to 53 percent of those questioned in one survey conducted in February 1974.

Widespread suspicions arose that oil and gas companies were exacerbating the crisis by deliberately withholding production to drive prices higher.

Even as late as 1979, more people were blaming high prices on companies deliberately withholding oil and gas production (65 percent) and refusing to drill wells unless prices were raised (61 percent) than OPEC (59 percent).

"Neither industry nor government had given the public advanced warning of energy shortages. Even after they occurred, there seemed to be no clear explanations or comprehensible data," historian Richard Vietor wrote later about the confusion and conspiracy theories ("Energy Policy in America", 1984).

Reputable publications including Newsweek, Time and the Washington Post were filled with articles suggesting domestic oil and gas producers were colluding with the Organization of the Petroleum Exporting Countries to engineer shortages and raise prices.

"The news media only added to the confusion. Mistrust abounded as to the reality of the crisis and its causes," Vietor concluded.

Conspiracy theories flourished because it turned out the government relied almost exclusively on the oil and gas industries for information on the state of production and reserves.

"In terms of getting the facts in the energy area, the federal government has completely delegated their responsibility to the oil industry," complained the chairman of a congressional committee. "This deplorable situation has to end."

Senior officials from the administration testified they were satisfied with data they received from the National Petroleum Council, the American Petroleum Institute, the American Gas Association and other industry associations, but Congress was adamant the government needed its own independent information.

"The American people want to know if there is an oil shortage. The American people want to know if there are oil tankers anchored offshore waiting for a price increase or available storage before they unload. The American people want to know whether major oil companies are sitting on shut-in wells and hoarding production in hidden tanks and at abandoned service stations," Senator Henry "Scoop" Jackson demanded to know.

Jackson chaired the powerful Senate Permanent Subcommittee on Investigations and turned the heat up on the industry.

"The American people want to know why oil companies are making soaring profits. The American people want to know if this so-called energy crisis is only a pretext, a cover ... to raise prices, to repeal environmental laws and to force the adoption of new tax subsidies," Jackson wondered in January 1974.
            
DATA DEMAND
Between 1973 and 1975, at least 25 of the 39 permanent congressional committees then in existence held hearings into some aspect of the energy crisis.

The recurrent theme through the 1970s was the need for better data. Congress published reports on everything from "Conflicting information on fuel shortages" (1974) and "Energy data requirements of the federal government" (1974) to "Energy information shortcomings and the gasoline shortage" (1979).

From this was born the most ambitious and comprehensive effort ever undertaken by any country in times of peace to collect detailed information on the production, processing, distribution and consumption of energy.

The Federal Energy Administration Act, enacted in May 1974 when memories of the crisis were still sharp, established a post of federal energy administrator and endowed it with sweeping powers to take such actions as necessary "to assure that adequate provision is made to meet the energy needs of the nation" (Public Law 93-275).

The administrator was empowered to "collect, evaluate, assemble, and analyze energy information on reserves, production, demand and related economic data" (section 5(b)(9)).

"All persons owning or operating facilities or business premises who are engaged in any phase of energy supply or major energy consumption shall make available to the administrator such information and periodic reports, records, documents and other data ... as the administrator may prescribe by regulation or order", the law stipulated (section 13(b)).

In case that was insufficient, the law gave the administrator power to issue compulsory surveys and questionnaires, conduct physical examinations of plants and subpoena evidence (sections 13(c)-(e)).
             
AVAILABILITY BIAS
The Federal Energy Administration was subsequently reorganised and its information-gathering powers assigned to a new Energy Information Administration within the Department of Energy. But the sweeping powers conferred at the height of the energy crisis remain the basis for the agency's unrivalled collection and analysis efforts to the present day.

In 1995, the agency became the first component of the Energy Department to launch its own website. It now has more than 1.5 million visitors to its website every month and over 200,000 web pages ("EIA celebrates 20 years on the Internet", June 30, 2015).

The problem is that no other country provides anything like the same quality and depth of information about its energy industries.

Other advanced economies such as Britain, Germany and Australia provide data that is less comprehensive and far less timely, often on websites that are byzantine in their complexity.

In the case of emerging markets like China and Saudi Arabia, the data is either missing or considered a state secret.

With emerging markets accounting for more than half of global oil demand for the first time in 2013 and 2014, most of the oil market is now opaque.

Inevitably, analysts, traders, investors and journalists tend to focus on the most readily available information and then extrapolate to the rest of the market, a variant on the data availability bias identified by behavioural economists.

Congress's preoccupation with data in the 1970s explains why we have such rich information about what is happening in the United States.

But it has created an enormous distortion. Better data on the rest of the world and especially emerging markets must be the top priority if policymakers want energy markets to operate more smoothly.

By John Kemp