Corporate ESG Data is Still Mostly Useless

14 min readAug 4, 2020

This is the first blog post that has been commissioned by someone other than me! I recently met an institutional equity portfolio manager that asked me if I could do some homework on the state of ESG investing and data today. He told me that he has heard that demand for ESG investing has gone “through the roof,” but he was skeptical about whether the products offered today truly give investors what they’re looking for. We agreed on 4 questions to answer about ESG data-dependent investing: who has the best data, how can we objectively use the data, to where is the money going, and which newer companies are promising in this space?

1. Who has the best data?

There are a lot of ESG data providers that investors use to influence their investment decisions, by one count there are around 200 independent operations.[1] The most recent and relevant comparison between the major players comes from the appendix of a February 2020 MDPI article, “Measurement of Corporate Social Responsibility: A Review of Corporate Sustainability Indexes, Rankings and Ratings.”[2] The chart was so comprehensive with its comparison of 15 major players that it had to be broken into 5 sub-charts over 13 pages. I did some cut and paste work in PowerPoint to break it into 5 images (see appendix, bring Advil). For a simpler but less updated comparison of the breadth of the offerings, to the right is a chart published in the 2017 Journal of Environmental Investing by my NYU professor, Tensie Whelan.[3]

Providers distinguish themselves in this horse race by using different (and often undisclosed) recipes of public information, proprietary questionnaires, and quantitative analysis to print a host of ratings, rankings, and indices. Just peeking at the differences in methodology, output, breadth and depth of each provider, we can start to get an understanding of how complex and nuanced this field is.

Maybe if we find the best data provider, we can ignore everything else? Ok, what is “good” ESG data, anyway? One of the most prominent ESG reporting standards publishers, the Global Reporting Initiative (GRI), has a great framework for ESG data quality: accuracy, balance, clarity, comparability, reliability, and timeliness.[4] To be both high quality and useful, the ESG data also needs to be transparent of stakeholders, material, and complete. So which providers best fit these characteristics? Luckily, an environmental consulting firm, SustainAbility, recently conducted a survey of 319 ESG experts that can help guide us toward the provider with the highest quality and most useful data.

From the survey there isn’t one provider that can definitively be crowned best, but CDP, RobecoSam, and Sustainalytics are the top 3 in quality and usefulness, as perceived by “experts.” The SustainAbility annual report, “Rate the Raters,” seems to be a widely cited report in the industry, so perhaps we can get away with believing that these experts know what they’re talking about. Frankly, I’m not too happy with the applicability of these infographics. Why the groups? Wouldn’t it be useful to know if RobecoSAM received all 5s and CDP received all 4s?

But ok, we now have a base understanding of the best providers, or do we? CDP only conducts research on environmental factors, leaving out the “S” and “G”, and RobecoSAM only covers 2,500 companies. Sustainalytics covers 11,000 companies, but each rating is only based on 70 indicators, while CDP incorporates 170 indicators just on the “E” part! To make things more complicated, different providers, using roughly the same data, often come up with vastly different ratings for each company,[5] so how do we know which provider has the best analysis?[6] The average correlation between ESG ratings of the same company is 0.61. Investors truly rely on Moody’s and S&P to provide credit ratings on borrowers, and the correlation between their ratings of the same issuer is 0.99%.

Overwhelmed? Me too. If I had to pick one to rely on, I would likely go with Sustainalytics. They’ve added more indicators since 2017, and so have the other providers, but I love a good word mash.

2. How can we objectively use the data?

If you frequent this blog, you know that I’m not the most ardent advocate of stock picking, so I am very tempted to tackle the connection between ESG ratings and stock performance. However, to stay singularly focused and answer our 4 questions, it is safe to assume that some investors believe ESG data should be incorporated into investing. In fact, we have an idea of how investors typically use this data. Of the $31 trillion that was invested with some ESG consideration in 2018, most of the time the manager wanted to extrapolate financial implications from ESG data or simply avoid the worst companies (see below, and yes, the managers could pick multiple options).

For an investor to gain useful financial insights or effectively isolate the least sustainable companies, the data must exhibit those quality characteristics listed above: accuracy, balance, clarity, comparability, reliability, and timeliness. In 2011, the percentage of S&P 500 constituents that published sustainability reports was 20%, and in 2018 it improved to 78%, but is the increase in data truly useful?[7] Let’s tackle each quality characteristic individually:

Accuracy: whether the data has been correctly measured to a reasonable margin of error

There are no regulatory bodies that audit sustainability reports, the primary source of ESG data, and self-reported estimates of emissions, for example, are tough to verify, but some studies have shed some light on the subject. Marchi and Hamilton (2006) found evidence suggesting chemical plants underreport emissions of toxic chemicals like lead and nitric acid.[8] Zahran et al. (2014) found evidence that self-reported lead emissions were significantly less accurate before the EPA implemented more strict reporting requirements.[9] Downie and Stubbs (2012) found evidence that Australian companies reported vastly different carbon emissions for similar activities,[10] and Del Guidice and Rigamonti (2020) found evidence that self-reported ESG data makes for less accurate sustainability ratings than externally audited ESG data.[11] It’s not an overwhelming body of evidence, but these studies tell me that self-reported ESG data likely is not the most accurate.

Balance: whether the data considers equally the positive and negative aspects of an organization’s performance considering only relevant data

If companies tend to selectively disclose data in their sustainability reports, I would consider that data unbalanced. Unfortunately, In and Park (2019) and Marquis et al (2016) discuss how companies do just that: selectively disclose within sustainability reports in response to changing corporate environments and communications strategies.[12] [13] In the same vein, Stanny (2013) shows that Fortune 500 companies appear to deliberately conceal information about their climate performance and disclose the minimum amount of data to avoid criticism.[14] In a study of major Canadian emitters, Talbot and Boiral (2013) found that managers admitted having disclosed “misleading or incomplete information in the past” in 35% of cases studied.[15] ESG ratings generated by data providers are choc’ full of generalizations as well, often highly influenced by unrelated factors like company size, location, and industry, as reported by the American Council for Capital Formation (2018).[16] It appears that the primary source of ESG data is highly skewed towards marketing communications, and the secondary ratings analysis of that data is variable, proprietary, and critically flawed.

Clarity: whether the data is easily accessible and understandable

For those who pay the expensive fees of an ESG data provider, sure, the data that has been reported or estimated by the provider is easily accessible, but methodologies for benchmarks, ratings or rankings are often far from clear. In Rate the Raters (2020), it was reported that most ESG investors use a provider’s corporate data, but “throw out the scores” because methodology is often a proprietary black box of statistics. ESG investors also complained about “holes in corporate data and the need for companies to improve disclosure, reporting and transparency.” They said that the number one preferred change in the coming years is “improved quality and disclosure of methodology.” The state of today’s ESG data is not profoundly clear.

Comparability: whether the data is reported consistently in a manner that enables stakeholders to analyze performance changes over time and against peers.

This is the real issue with today’s ESG data. We’ve come a decent way in the last decade or so, but we’re still nowhere close to fully comparable ESG data. Comparability is an interesting characteristic for the GRI to include because it’s not mutually exclusive within this framework. Can you analyze performance over time if data isn’t accurate, balanced, or clearly understood?

Wegener et al (2019) provides evidence of “a lack of comparability, and therefore commensurability, in reported greenhouse gas emissions” among oil and gas corporations that could generate “misleading sustainability reports,” [17] but my favorite study on this subject is Cardoni et al (2018). These researchers started with a universe of 41 public oil and gas companies (sector most studied because it’s the most sensitive to ESG considerations) that provided 2018 sustainability reports.

After reviewing and discarding companies that reported incomparable frameworks, they were left with only 16 companies with GRI or similar frameworks (39%). GRI standards for oil and gas companies provide indicators within 51 material aspects (categories). Of these 51 GRI aspects, only indicators within 10 aspects appear in more than 50% of sustainability reports from those 16 comparable companies (see appendix). There was no single aspect that appeared on 100% of the sustainability reports, but 14 of the 16 companies reported at least some information on emissions, and within the emissions aspect you can see below how often a company reported a material indicator.

From this graphic we can read that only 4 of 41 large public oil and gas companies disclosed GRI indicator G4-EN17: Other Indirect GHG emissions (Scope 3). This is just one example, but scope 3 emissions are very important measurements, they encapsulate all GHG emissions that the company is responsible for within its unowned value chain, like its suppliers or customers. Scope 3 emissions often represent the largest portion of a company’s carbon footprint, and if we care at all about measuring corporate emissions, we can’t even begin to have that conversation until we address scope 3.[18] For example, scope 1 counts the emissions of Exxon’s oil tankers as they sail around the world, scope 2 counts the pro-rata emissions of the powerplant they use to turn on the lights of their corporate office, but it takes scope 3 to count the emissions from all the cars that buy Exxon’s gas. Scope 3 emissions account for 90% — 95% of total emissions for oil and gas companies.[19]

The study went on to show that only 2 of 8 indicators that were reported by at least 11 companies displayed sufficient data quality to confidently compare the set. The study’s author concluded, “Our findings reveal that, despite the availability of a large amount of ESG data and the efforts of organizations to increase the quality of these data, a relevant problem of comparability still exists. This mainly depends on the absence of mandatory regulatory constraints in terms of social reporting.” [20]

This is just one study, but its implications seem to be widespread. For example, only 10 of the 395 sustainability reporting companies in the S&P 500 followed one reporting framework, like GRI. The other 97.5% of reporting companies take snippets from multiple frameworks or never reference one.9

Reliability: whether the data has been externally verified or cited, and assumptions are supported in evidence

External verification of ESG data is not widespread. 36% of S&P 500 constituents with sustainability reports claim some form of nonfinancial audit, but it hardly used for more than one or two GHG data points. Only 3% of that group acknowledged that their holistic environmental and social data were audited.[21] Ability and willingness to report nonfinancial data decreases with company size, so we can safely bet that companies too small for the S&P 500 utilize external auditing less often.

Timeliness: whether the data is recent enough for investors to make informed decisions

According to “Rate the Raters,” investors interviewed this year “expressed strong critiques of ratings [for] use of old or backwards-looking data.” I can’t find much data on the timeliness of these reports, but I have no problem assuming company typically reports annually if at all.

In aggregate, if the primary and secondary data we use to make ESG related investment decisions aren’t accurate, balanced, clear, comparable, reliable, or timely, then what quality of decisions can we truly be making from them? There is hope in the form of increasingly utilized reporting frameworks and standards, but today I cannot find evidence that suggests ESG data is robust enough to accurately extrapolate financial implications or isolate the least sustainable companies, the most prevalent uses of this data.

3. To where is the money going?

In 2011, about $13.6 trillion of professionally managed global assets had some form of ESG mandate.[22] In 2018, that number rose to $30.7 trillion. Domestically, it grows at a much greater rate, from $3.7 trillion to $12 trillion, roughly 26% of all professionally managed investment assets in the country.[23] At any scale, that growth rate is formidable, but when we’re talking about markets worth trillions of dollars growing 20% each year for almost a decade, there’s certainly a major tectonic shift occurring right under our feet. During a major industrial evolution like this, someone is bound to be making out like a bandit. So, where is all the money going?

At first, I thought the investment managers were making all the money. I couldn’t find an estimate for ESG mandate investment fees, so I calculated my own estimates for 2011 and 2018. GSI published assets and allocations, so I just needed average annual fees for each asset class.[24],[25] According to this quick math, investment managers took in over $100 billion in revenue for managing these growing funds in 2018, head and shoulders above the $600 million taken by the data providers. Then I noticed that ESG investment fees were roughly the same in 2018 as 2011, due mostly to the decrease in fund expense ratios. I knew investment fund fees had been dropping, but I hadn’t expected the magnitude of the drop would cancel out a 3x increase in assets over the last 7 years! I only used ETF fees for this calculation (except for alts) because the data is limited, and I wanted to keep it consistent to show the relative change over time.

Morningstar’s annual fee survey only supported a 30% drop in all fund fees over our time frame, which is still substantial, but my data show a 60% decline.[26]

I can hypothesize that ESG investments have experienced a greater drop in fees than their peers precisely because of the tremendous increase in ESG data provision during this timeframe. Perhaps ESG funds were more expensive to manage in 2011 because of the lack of data, but have since become as cheap as passive index funds with the rapid increase of ESG disclosure, aggregation, and technology. If this is the case, is it accurate to say that asset managers are the folks that are getting rich off this trend? Maybe.

Perhaps the data providers are making all the money? After all, why would there be so many new entrants to this market if there wasn’t enough perceived economic profit? Actually, the ESG data provider market is growing at a nice 20% clip, but it’s still objectively small, accounting for only $600 million globally in 2019.[27] That’s pretty small for a global market, especially compared to the $100 billion the asset managers are raking in, but then again, the ESG data market didn’t really exist 10 years ago, so it’s a new $600 million. Perhaps the consultants are the ones making all the money? Not exactly. Even though IBISWorld reports that domestic environmental consulting was a $16 billion industry in 2019, it was an even bigger market 7 years prior.[28]

Is anyone making money off the rise of ESG investing? Maybe sustainable companies are profiting off the reduction in cost of capital when investor interest increases?[29] Maybe companies can reduce the volatility of their stock and push up its price by improving ESG metrics?[30] Perhaps investors are the ones making the real money?[31]

I don’t think anyone is making a ton of money in this market to the extent that I was expecting. Corporate reporting of ESG data is expensive and likely won’t pay for itself. The global data provider market is small, crowded, and commoditized at its best quality. The managers aren’t getting more rich because fees have collapsed, and the consultants have been flat for a decade. That being said, asset managers are still pulling in $100 billion each year, and during the an earnings call in January, CEO Laurence Fink said that BlackRock intends to “double its ESG ETF offerings to 150 over the next few years, to include sustainable versions” of its flagship index funds.[32]

4. Which newer companies might be promising in this space?

Unfortunately, I lost my NYU-borne access to Pitchbook before this section, but I found a couple of interesting companies that are making strides in the ESG data industry. There is clearly a growing movement for companies to provide this type of data, but collection and self-reporting data is time consuming, inaccurate, and expensive. Whether or not regulators enter this arena, companies that can take cost-effective measurements of material ESG indicators will have a seat at the table moving forward. Blue Sky Analytics and Hypervine are technology startups that have partnered with Al Gore’s Climate TRACE project using complex satellite imagery to provide accurate real-time and location-specific emissions data. Great improvements in technology have made collecting and aggregating data more efficient, and companies like Truvalue Labs are using AI in interesting ways to gather and interpret data.

To wrap up, the state of ESG investing today is generally unreliable, incomparable, inaccurate, and incomplete. I can understand using data from a highly credible company to analyze changes over time, but that may be the only reliable way to utilize ESG data today. There is hope that this may change in the future if reporting standards are more widely adopted, disclosure is improved, and external verification is mandated.

Appendix: