Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century

Research output: Contribution to journalComment/debateResearchpeer review

Authors

  • Raphael A. Viscarra Rossel
  • Thorsten Behrens
  • Eyal Ben-Dor
  • Sabine Chabrillat
  • José Alexandre Melo Demattê
  • Yufeng Ge
  • Cecile Gomez
  • César Guerrero
  • Yi Peng
  • Leonardo Ramirez-Lopez
  • Zhou Shi
  • Bo Stenberg
  • Richard Webster
  • Leigh Winowiecki
  • Zefang Shen

Research Organisations

External Research Organisations

  • Curtin University
  • Kompetenzzentrum Boden (KOBO)
  • Tel Aviv University
  • Helmholtz Centre Potsdam - German Research Centre for Geosciences (GFZ)
  • Universidade de Sao Paulo
  • University of Nebraska
  • Université Montpellier
  • Indian Institute of Science Bangalore
  • Universidad Miguel Hernandez
  • Food and Agriculture Organization of the United Nations
  • BÜCHI Labortechnik AG
  • Zhejiang University
  • Swedish University of Agricultural Sciences
  • Rothamsted Research
  • Center for International Forestry Research and World Agroforestry (CIFOR-ICRAF)
View graph of relations

Details

Original languageEnglish
Article numbere13271
Number of pages9
JournalEuropean journal of soil science
Volume73
Issue number4
Early online date26 Jun 2022
Publication statusPublished - 14 Jul 2022

Abstract

Spectroscopic measurements of soil samples are reliable because they are highly repeatable and reproducible. They characterise the samples' mineral–organic composition. Estimates of concentrations of soil constituents are inevitably less precise than estimates obtained conventionally by chemical analysis. But the cost of each spectroscopic estimate is at most one-tenth of the cost of a chemical determination. Spectroscopy is cost-effective when we need many data, despite the costs and errors of calibration. Soil spectroscopists understand the risks of over-fitting models to highly dimensional multivariate spectra and have command of the mathematical and statistical methods to avoid them. Machine learning has fast become an algorithmic alternative to statistical analysis for estimating concentrations of soil constituents from reflectance spectra. As with any modelling, we need judicious implementation of machine learning as it also carries the risk of over-fitting predictions to irrelevant elements of the spectra. To use the methods confidently, we need to validate the outcomes with appropriately sampled, independent data sets. Not all machine learning should be considered ‘black boxes’. Their interpretability depends on the algorithm, and some are highly interpretable and explainable. Some are difficult to interpret because of complex transformations or their huge and complicated network of parameters. But there is rapidly advancing research on explainable machine learning, and these methods are finding applications in soil science and spectroscopy. In many parts of the world, soil and environmental scientists recognise the merits of soil spectroscopy. They are building spectral libraries on which they can draw to localise the modelling and derive soil information for new projects within their domains. We hope our article gives readers a more balanced and optimistic perspective of soil spectroscopy and its future. Highlights: Spectroscopy is reliable because it is a highly repeatable and reproducible analytical technique. Spectra are calibrated to estimate concentrations of soil properties with known error. Spectroscopy is cost-effective for estimating soil properties. Machine learning is becoming ever more powerful for extracting accurate information from spectra, and methods for interpreting the models exist. Large libraries of soil spectra provide information that can be used locally to aid estimates from new samples.

Keywords

    calibration, machine learning, model localization, reflectance spectroscopy, regression, soil constituents, spectral libraries, validation

ASJC Scopus subject areas

Cite this

Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century. / Viscarra Rossel, Raphael A.; Behrens, Thorsten; Ben-Dor, Eyal et al.
In: European journal of soil science, Vol. 73, No. 4, e13271, 14.07.2022.

Research output: Contribution to journalComment/debateResearchpeer review

Viscarra Rossel, RA, Behrens, T, Ben-Dor, E, Chabrillat, S, Demattê, JAM, Ge, Y, Gomez, C, Guerrero, C, Peng, Y, Ramirez-Lopez, L, Shi, Z, Stenberg, B, Webster, R, Winowiecki, L & Shen, Z 2022, 'Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century', European journal of soil science, vol. 73, no. 4, e13271. https://doi.org/10.1111/ejss.13271
Viscarra Rossel, R. A., Behrens, T., Ben-Dor, E., Chabrillat, S., Demattê, J. A. M., Ge, Y., Gomez, C., Guerrero, C., Peng, Y., Ramirez-Lopez, L., Shi, Z., Stenberg, B., Webster, R., Winowiecki, L., & Shen, Z. (2022). Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century. European journal of soil science, 73(4), Article e13271. https://doi.org/10.1111/ejss.13271
Viscarra Rossel RA, Behrens T, Ben-Dor E, Chabrillat S, Demattê JAM, Ge Y et al. Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century. European journal of soil science. 2022 Jul 14;73(4):e13271. Epub 2022 Jun 26. doi: 10.1111/ejss.13271
Viscarra Rossel, Raphael A. ; Behrens, Thorsten ; Ben-Dor, Eyal et al. / Diffuse reflectance spectroscopy for estimating soil properties : A technology for the 21st century. In: European journal of soil science. 2022 ; Vol. 73, No. 4.
Download
@article{af1b2694c5cf465a99c9d8965351ada7,
title = "Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century",
abstract = "Spectroscopic measurements of soil samples are reliable because they are highly repeatable and reproducible. They characterise the samples' mineral–organic composition. Estimates of concentrations of soil constituents are inevitably less precise than estimates obtained conventionally by chemical analysis. But the cost of each spectroscopic estimate is at most one-tenth of the cost of a chemical determination. Spectroscopy is cost-effective when we need many data, despite the costs and errors of calibration. Soil spectroscopists understand the risks of over-fitting models to highly dimensional multivariate spectra and have command of the mathematical and statistical methods to avoid them. Machine learning has fast become an algorithmic alternative to statistical analysis for estimating concentrations of soil constituents from reflectance spectra. As with any modelling, we need judicious implementation of machine learning as it also carries the risk of over-fitting predictions to irrelevant elements of the spectra. To use the methods confidently, we need to validate the outcomes with appropriately sampled, independent data sets. Not all machine learning should be considered {\textquoteleft}black boxes{\textquoteright}. Their interpretability depends on the algorithm, and some are highly interpretable and explainable. Some are difficult to interpret because of complex transformations or their huge and complicated network of parameters. But there is rapidly advancing research on explainable machine learning, and these methods are finding applications in soil science and spectroscopy. In many parts of the world, soil and environmental scientists recognise the merits of soil spectroscopy. They are building spectral libraries on which they can draw to localise the modelling and derive soil information for new projects within their domains. We hope our article gives readers a more balanced and optimistic perspective of soil spectroscopy and its future. Highlights: Spectroscopy is reliable because it is a highly repeatable and reproducible analytical technique. Spectra are calibrated to estimate concentrations of soil properties with known error. Spectroscopy is cost-effective for estimating soil properties. Machine learning is becoming ever more powerful for extracting accurate information from spectra, and methods for interpreting the models exist. Large libraries of soil spectra provide information that can be used locally to aid estimates from new samples.",
keywords = "calibration, machine learning, model localization, reflectance spectroscopy, regression, soil constituents, spectral libraries, validation",
author = "{Viscarra Rossel}, {Raphael A.} and Thorsten Behrens and Eyal Ben-Dor and Sabine Chabrillat and Dematt{\^e}, {Jos{\'e} Alexandre Melo} and Yufeng Ge and Cecile Gomez and C{\'e}sar Guerrero and Yi Peng and Leonardo Ramirez-Lopez and Zhou Shi and Bo Stenberg and Richard Webster and Leigh Winowiecki and Zefang Shen",
note = "Funding Information: Raphael A. Viscarra Rossel received funding from the Australian Government via grant ACSRIV000077. ",
year = "2022",
month = jul,
day = "14",
doi = "10.1111/ejss.13271",
language = "English",
volume = "73",
journal = "European journal of soil science",
issn = "1351-0754",
publisher = "Wiley-Blackwell Publishing Ltd",
number = "4",

}

Download

TY - JOUR

T1 - Diffuse reflectance spectroscopy for estimating soil properties

T2 - A technology for the 21st century

AU - Viscarra Rossel, Raphael A.

AU - Behrens, Thorsten

AU - Ben-Dor, Eyal

AU - Chabrillat, Sabine

AU - Demattê, José Alexandre Melo

AU - Ge, Yufeng

AU - Gomez, Cecile

AU - Guerrero, César

AU - Peng, Yi

AU - Ramirez-Lopez, Leonardo

AU - Shi, Zhou

AU - Stenberg, Bo

AU - Webster, Richard

AU - Winowiecki, Leigh

AU - Shen, Zefang

N1 - Funding Information: Raphael A. Viscarra Rossel received funding from the Australian Government via grant ACSRIV000077.

PY - 2022/7/14

Y1 - 2022/7/14

N2 - Spectroscopic measurements of soil samples are reliable because they are highly repeatable and reproducible. They characterise the samples' mineral–organic composition. Estimates of concentrations of soil constituents are inevitably less precise than estimates obtained conventionally by chemical analysis. But the cost of each spectroscopic estimate is at most one-tenth of the cost of a chemical determination. Spectroscopy is cost-effective when we need many data, despite the costs and errors of calibration. Soil spectroscopists understand the risks of over-fitting models to highly dimensional multivariate spectra and have command of the mathematical and statistical methods to avoid them. Machine learning has fast become an algorithmic alternative to statistical analysis for estimating concentrations of soil constituents from reflectance spectra. As with any modelling, we need judicious implementation of machine learning as it also carries the risk of over-fitting predictions to irrelevant elements of the spectra. To use the methods confidently, we need to validate the outcomes with appropriately sampled, independent data sets. Not all machine learning should be considered ‘black boxes’. Their interpretability depends on the algorithm, and some are highly interpretable and explainable. Some are difficult to interpret because of complex transformations or their huge and complicated network of parameters. But there is rapidly advancing research on explainable machine learning, and these methods are finding applications in soil science and spectroscopy. In many parts of the world, soil and environmental scientists recognise the merits of soil spectroscopy. They are building spectral libraries on which they can draw to localise the modelling and derive soil information for new projects within their domains. We hope our article gives readers a more balanced and optimistic perspective of soil spectroscopy and its future. Highlights: Spectroscopy is reliable because it is a highly repeatable and reproducible analytical technique. Spectra are calibrated to estimate concentrations of soil properties with known error. Spectroscopy is cost-effective for estimating soil properties. Machine learning is becoming ever more powerful for extracting accurate information from spectra, and methods for interpreting the models exist. Large libraries of soil spectra provide information that can be used locally to aid estimates from new samples.

AB - Spectroscopic measurements of soil samples are reliable because they are highly repeatable and reproducible. They characterise the samples' mineral–organic composition. Estimates of concentrations of soil constituents are inevitably less precise than estimates obtained conventionally by chemical analysis. But the cost of each spectroscopic estimate is at most one-tenth of the cost of a chemical determination. Spectroscopy is cost-effective when we need many data, despite the costs and errors of calibration. Soil spectroscopists understand the risks of over-fitting models to highly dimensional multivariate spectra and have command of the mathematical and statistical methods to avoid them. Machine learning has fast become an algorithmic alternative to statistical analysis for estimating concentrations of soil constituents from reflectance spectra. As with any modelling, we need judicious implementation of machine learning as it also carries the risk of over-fitting predictions to irrelevant elements of the spectra. To use the methods confidently, we need to validate the outcomes with appropriately sampled, independent data sets. Not all machine learning should be considered ‘black boxes’. Their interpretability depends on the algorithm, and some are highly interpretable and explainable. Some are difficult to interpret because of complex transformations or their huge and complicated network of parameters. But there is rapidly advancing research on explainable machine learning, and these methods are finding applications in soil science and spectroscopy. In many parts of the world, soil and environmental scientists recognise the merits of soil spectroscopy. They are building spectral libraries on which they can draw to localise the modelling and derive soil information for new projects within their domains. We hope our article gives readers a more balanced and optimistic perspective of soil spectroscopy and its future. Highlights: Spectroscopy is reliable because it is a highly repeatable and reproducible analytical technique. Spectra are calibrated to estimate concentrations of soil properties with known error. Spectroscopy is cost-effective for estimating soil properties. Machine learning is becoming ever more powerful for extracting accurate information from spectra, and methods for interpreting the models exist. Large libraries of soil spectra provide information that can be used locally to aid estimates from new samples.

KW - calibration

KW - machine learning

KW - model localization

KW - reflectance spectroscopy

KW - regression

KW - soil constituents

KW - spectral libraries

KW - validation

UR - http://www.scopus.com/inward/record.url?scp=85135921473&partnerID=8YFLogxK

U2 - 10.1111/ejss.13271

DO - 10.1111/ejss.13271

M3 - Comment/debate

AN - SCOPUS:85135921473

VL - 73

JO - European journal of soil science

JF - European journal of soil science

SN - 1351-0754

IS - 4

M1 - e13271

ER -