Introduction

Agricultural carbon markets are created to provide incentives for growers to adopt “climate-smart” farming practices. These practices reduce greenhouse gas (GHG) emissions on the farm and can help soil sequester carbon. Computer models of soil carbon flux play an important part in this process: models estimate carbon change year by year, and enable growers to earn carbon credits more quickly. Allowing carbon modeling to augment or even replace some direct soil sampling lowers the cost of administering carbon and climate smart programs while making programs available to more growers.

For an in-depth discussion of agricultural carbon markets, see here.

Figure 1: The geographical area of CIBO’s Verra project, showing climate zones represented.

In 2021, CIBO initiated an agricultural carbon program with Verra under the newly-issued VM0042 protocol. This program supports corn and soybean farmers in the US Midwest (see Figure 1) who adopt regenerative practices for cover crop, tillage, and nitrogen usage, using the SALUS (Systems Approach to Land Use Sustainability) ecosystem model to estimate soil carbon flux. An extensive validation of the SALUS model was required to demonstrate the accuracy, efficacy, and precision of modeled results. In this paper, we share our experience with the model validation process.

Voluntary Carbon Markets

According to the Congressional Research Service, a carbon market “…generally refers to an economic framework that supports buying and selling of environmental commodities that signify GHG emission reductions or sequestration.” With the growing awareness of greenhouse gasses’ role in climate change, many companies have committed to reducing greenhouse gas emissions. Organizations that aspire to be carbon neutral often achieve this through applying several strategies at once: reducing direct carbon emissions, choosing supply-chain partners who are also reducing carbon emissions (i.e., Scope 3 emissions reductions), and purchasing carbon credits.

The process of purchasing carbon credits is sometimes called “offsetting.” A company offsets its carbon emission “debt” by paying someone else to reduce carbon emissions. Both parties benefit from this: the credit purchaser pays to reduce their overall carbon footprint, and the credit seller gets material compensation for reducing emissions. Carbon markets were developed to facilitate this exchange.

CIBO’s Mission: Enabling Agricultural Carbon Markets 

Many processes produce greenhouse gases, and carbon credits can be generated in many ways. One of the best ways to remove carbon dioxide from the atmosphere is to sequester it in the soil. Farmers can generate carbon credits by reducing their greenhouse gas emissions and adopting practices that increase carbon sequestration on their land. Because farming and grazing lands are plentiful, these agricultural carbon credits are of growing interest in the carbon marketplace.

Several obstacles stand in the way of the farmer who wants to generate carbon credits. In addition to the cost of adopting new land management practices, such as cover cropping, reduced tillage, and reduced nitrogen fertilizer application, benefits to yield and operational profitability from improved soil health can take several years to realize. While the data is clear that such an investment is worthwhile in the long run, the necessity of making a large up-front investment can make these practice changes difficult for many farmers who are already operating on thin margins. Ideally, selling carbon credits would provide up-front funds for such improvements, but the credits are often not available until the farmer can prove that sequestration has taken place.

In addition, carbon sequestered in the soil is difficult to quantify. Direct measurement via soil sampling is inherently uncertain and so expensive that it can quickly consume the profits a farmer might make from selling the carbon credits.

At CIBO Technologies, our mission is scaling regenerative agriculture with reliable, trustworthy technology that delivers high-quality accounting for agricultural carbon. CIBO does this with CIBO Impact, our advanced software platform that helps enterprises scale their agricultural carbon programs to meet their carbon and climate commitments.

CIBO powers regenerative agriculture initiatives of grower-focused food and agriculture companies ready to meet their carbon and sustainability commitments. CIBO achieves our partners’ goals by leveraging our scaled software platform, CIBO Impact, to develop, deploy and manage sustainability programs that combine advanced science-based ecosystem modeling, AI-enhanced computer vision, MVR capabilities, and the most complete programs engine to connect growers, enterprises and ecosystems.

CIBO creates visibility into the carbon footprint of entire supply chains and supply sheds. We use ecosystem modeling to estimate carbon credits and minimize the need for soil sampling. To accomplish this, we need to demonstrate that our ecosystem modeling delivers consistent,  accurate estimates of soil carbon flux.

The Model Validation Report (MVR) Process

Several registries are available for certifying agricultural carbon credits. CIBO chose to register a project first with Verra. CIBO’s was the first VM0042 project under validation in North America. Most of the major registries have very similar requirements for model validation, so we believe our experience is broadly applicable to other registries as well.

The registry protocol, VM0042 “METHODOLOGY FOR IMPROVED AGRICULTURAL LAND MANAGEMENT,” details how agricultural carbon credits are accounted for and assigned.  This protocol specifies how models, such as CIBO’s SALUS model, can be used to estimate carbon flux.  The accompanying model validation protocol, VMD0053 “MODEL CALIBRATION, VALIDATION, AND UNCERTAINTY GUIDANCE FOR THE METHODOLOGY FOR IMPROVED AGRICULTURAL LAND MANAGEMENT,” details how such an ecosystem model must be tested to demonstrate its accuracy and applicability.

A large part of the process involves comparing modeled results to results found in published literature. That is, a researcher studies the effects of farming practice changes and publishes a set of observations related to soil carbon change. At CIBO, we create an in-silico experiment matching the real-world conditions of the study, and compare our modeled results with the published outcome. Although the process seems simple, in practice it proved to be a monumental effort, with many pitfalls.

Finding the Right Datasets

This first step in validating a model is to find appropriate “ground truth” data for comparison to model simulations. The model validation protocol VMD0053 specifies, “Measured datasets must be drawn from peer-reviewed and published experimental datasets with measurements of the emissions source(s) of interest (soil organic carbon (SOC) stock change and/or N2O and CH4 change, as applicable), ideally using control plots to test the practice category.” At present, every organization that wants to use modeling as part of the agricultural carbon process must independently search for and identify the appropriate published datasets.

To ensure that no potentially useful datasets were ignored, CIBO started by casting a wide net: we developed a set of search terms to capture potentially relevant papers, then reviewed each paper for actual relevance. Datasets that are not agricultural in nature, experiments that are located in other parts of the world, or datasets that rely on crops and/or practices that CIBO is not modeling for the Verra project were excluded.

In all, CIBO had a team of 12 scientists running searches and filtering publications. The initial searches identified hundreds of possible papers, which were eventually narrowed down to approximately 60 relevant publications. Some of these were later discovered to have technical problems (such as omissions of key variables) that rendered them unusable. Others were found to be duplicates of datasets already ingested. In the end, CIBO curated 39 datasets from peer-reviewed, published sources, with a total of 298 treatment comparisons. This is one of the most comprehensive collections, to date, of agricultural emissions and carbon flux data for the crops and practices covered in the CIBO Verra program.

Extracting and Curating Data

Generally speaking, each dataset consists of two parts relevant to the model validation process: the measured soil organic carbon (SOC) values and the accompanying information about what locations and agricultural management practices were used in the experiment. The measured SOC data are needed to calculate the “ground truth” carbon credits resulting from practice changes that occurred in the field experiments. The location and management information about a field experiment are needed to create a simulated experiment that mimics the real world conditions of the plant. Such a simulated experiment represents instructions for the SALUS model about the real-world experiment. Executing the computer experiment produces simulation results from which simulated carbon credits can be calculated and compared to the measured carbon credits.

Extracting the two types of data from the literature is a complicated and manual process that often requires expert knowledge, because the data come in various formats that have to be harmonized prior to utilization. Agricultural management practices, in particular, are generally presented in narrative form, and may not include enough information to create the simulated experiment necessary to run the crop model. In these cases, CIBO uses proprietary crop management models to infer management practices like cultivar characteristics, planting date, etc. that vary across production systems in the United States.

In addition to management practices, soil and weather data about the locations of the field experiments are needed to create the computer experiment for running the SALUS model. These data are obtained through CIBO’s soil and weather services which require the geographic coordinates of the field experiment locations to be known. These geographic coordinates were extracted from the curated data and datasets with no geographic coordinates were not used.

Calculating Carbon Credits

For the purposes of this project, a carbon credit is simply the annual soil carbon gain/loss between a baseline practice or control treatment (e.g., continuous corn with no cover crop) and a regenerative practice (e.g., adding a cover crop to a corn-corn rotation), expressed as metric tonnes of CO2-equivalents gained or lost per acre per year. The measured carbon credits were calculated directly from the raw SOC data extracted through the data curation process.

The modeled carbon credits were calculated from the model outputs obtained by executing the computer experiments for both the baseline and regenerative practices for the same number of years the field experiments were conducted. The results presented in the next sections assess the ability of the SALUS model in simulating the measured carbon credits based on information about the field experiments supplied to the model as inputs, and given uncertainties in the measurements quantified by the Pooled Measurement Uncertainty (PMU, see next section for more detail).

CIBO’s Results

Across all curated data, CIBO found 298 relevant measurements for comparing simulated and observed results. These results are grouped by practice category (soil disturbance or cropping) and crop functional group (C3 annual grass, C4 annual grass, or annual legume). For each combination of practice category and crop functional group, we calculate the pooled measurement error, the model bias, and the model error standard deviation. Then we compare our own model bias to the pooled measurement error found in the datasets.

Pooled Measurement Uncertainty

Pooled measurement uncertainty (PMU) is meant to represent the uncertainty associated with experimental measurements of SOC. PMU is defined as the weighted average of standard errors of all the measured values for a given practice change. PMU should, ideally, be derived from observed variation between replicated measurements within an experiment. However, most studies do not provide data from replicates that could be used to estimate measurement uncertainty. Where replicates for a study are provided, they typically consist of several measurements across experimental blocks and/or within individual treatment plots, which capture the combined uncertainty stemming from both measurement error and sampling uncertainty (e.g., due to spatial variability). Although there is likely to be spatial variability within an individual treatment plot, the assessment of PMU for this Model Validation Report considered measurements collected within the same plot at the same depth and time as replicates and averaged those measurements for this analysis.

Estimating unreported measurement uncertainties were especially challenging. Although VMD0053 does not provide explicit guidelines for these cases, it allows for an alternative computation method for PMU as a whole: rather than computing PMU for each practice change, PMU may be computed using all studies in the validation dataset using the same measurement technique (e.g., dry combustion, loss on ignition, or older chemical approaches for soil organic carbon determination) regardless of the crop functional group studied. This is not an ideal way to calculate a SOC stock uncertainty, however, as it does not account for the uncertainties associated with bulk density measurements or account for the size of the soil depth interval measured. CIBO took a hybrid approach for missing observed uncertainty, by calculating a PMU for every reported SOC concentration and bulk density measurement method, then propagating these uncertainties through the calculation of SOC stock for each unreported measurement uncertainty according to its measurement method. These were then combined with the observed measurement uncertainties to calculate PMU by cumulative depth for each practice change and crop functional group combination.

CIBO’s approach to calculating PMU is explicitly designed to calculate PMU for the entire soil profile sampled. There is wide variation in the depth increments and total depths to which the studies comprising the validation dataset sampled. Converting to a cumulative depth for each given observation depth allows CIBO to combine studies together to calculate a more accurate PMU. For example, a small number of observations were taken from 23-30 cm. Using the cumulative depth approach, the observations from 23-30 cm could build on the uncertainty derived from any soil samples shallower than 23 cm, and be combined with uncertainty derived from observations of 0-30 cm and 15-30 cm depths. By using this cumulative depth approach, CIBO is able to take a more holistic view of sampling uncertainty that is less sensitive to individual  studies with particularly high or particularly low sample variability.

Finally, each calculation of net impact to SOC stocks due to a practice change requires at least four estimates of SOC content and four estimates of bulk density: initial and final measurements for both treatments at every depth sampled in the study. Many studies neglected to measure SOC at the beginning of the study. These studies assume, implicitly or explicitly, that the initial conditions of the differing treatments are the same. CIBO treats a missing measurement as if it had the same uncertainty as available measurements for a given study. This is a conservative choice, as it underestimates measurement uncertainty, decreasing PMU associated with the studies, and requiring a similarly smaller model bias to pass validation.