Too few samples and/or parameter difficult to predict and/or noise in the reference method are possible causes of large difference between SEC and SECV

From NIR Forum discussion Posted on Tuesday, February 08, 2011 - 4:42 am:    Dear all,  First, I want to say thank you in advance for every answer. I'm first time on this forum.  I have read a lot of things here and I  think that it is really useful. I have a question regarding the prediction of wood properties with NIR spectra.  I have a set of spectra from wood sample (calibration and test set) and I would like develop the best model for wood properties (eg wood density). However, I get higher error for cross validation (SECV) then for calibration set (SEC) and test set(SEP).Maybe I make a mistake in the application of cross validation. I use Unscrambler software. Can anyone tell me how to use the option of cross validation in Unscrambler software?  Thank you and best regards,  Nebojsa

Posted on Tuesday, February 08, 2011 - 5:44 am:    HI,  How large is the difference between SEC and SECV?  Before calibrating, you have to know the SEL (error of the reference method) and the SD of the calibration sample.  How many samples?  the gap between SEC and SECV is due to  - Too few samples and/or  - Parameter difficult to predict and/or  - Noise in the reference method.  SECV is always higher than SEC. My rule is to have SECV<=1.05*SEC with 2 groups of CV. Then I am pretty sure the model is robust.  Pierre

Posted on Tuesday, February 08, 2011 - 7:22 am:

NIR model should produce SEP lesser than 2.0SEL or 1.5SEL at least.

"There is also a statement of criteria that the NIR model should produce SEP lesser than 2.0SEL or 1.5SEL at least. If the SEP is larger then your model needs more calibration development and/or more samples."

(https://www.researchgate.net/post/Does_anybody_know_how_to_compare_NIR_error_and_lab_analysis_error1)



"Standard errors of performance (SEP) are frequently twice the magnitude of the standard error of the laboratory (SEL) in successful NIR calibrations. In spite of this the repeatability of NIR measurement is almost always better than the repeatability of the reference procedure."

(V. MÍKA, J. POZDÍŠEK, P. TILLMANN, P. NERUŠIL, K. BUCHGRABER4, L. GRUBER (2003) Development of NIR calibration valid for two different grass sample collections, Czech J. Anim. Sci., 48, 2003 (10): 419–424.)



"Westerhaus (1985, cited by Stimson, et al 1991) recommended that the SEP should be no greater than twice the SEL."

(G. McL. Dryden (2003) Near Infrared Reflectance Spectroscopy: Applications in Deer Nutrition. A report for the Rural Industries Research and Development Corporation, RIRDC Publication No W03/007 RIRDC Project No UQ-109A)

It is easier to avoid overfitting the data by using PCR over PLS, particularly if there is considerable error in the lab method (SECV <= 1.05*SEC)

From NIR Forum discussion Posted on Tuesday, February 08, 2011 - 4:42 am:    Dear all,  First, I want to say thank you in advance for every answer. I'm first time on this forum.  I have read a lot of things here and I  think that it is really useful. I have a question regarding the prediction of wood properties with NIR spectra.  I have a set of spectra from wood sample (calibration and test set) and I would like develop the best model for wood properties (eg wood density). However, I get higher error for cross validation (SECV) then for calibration set (SEC) and test set(SEP).Maybe I make a mistake in the application of cross validation. I use Unscrambler software. Can anyone tell me how to use the option of cross validation in Unscrambler software?  Thank you and best regards,  Nebojsa

Posted on Tuesday, February 08, 2011 - 5:44 am:    HI,  How large is the difference between SEC and SECV?  Before calibrating, you have to know the SEL (error of the reference method) and the SD of the calibration sample.  How many samples?  the gap between SEC and SECV is due to  - Too few samples and/or  - Parameter difficult to predict and/or  - Noise in the reference method.  SECV is always higher than SEC. My rule is to have SECV<=1.05*SEC with 2 groups of CV. Then I am pretty sure the model is robust.  Pierre

Posted on Tuesday, February 08, 2011 - 7:22 am:    Dear Pierre,  thank you for your answer.  How large is the difference between SEC and SECV?  - The difference is very large SECV - 0.043 SEC - 0.020  How many samples?  - 74 for calibration and 20 for test set.  - SD for calibration is 0.049 (mean 0.698) and for test set is 0.047 (mean  0.713)  - for cross validation I use setup random, number of segments 10 and samples  per segment 7.  Nebojsa

Posted on Wednesday, February 09, 2011 - 4:35 am:    Bonjour,  it means R2CV ~~ 0.0. You have a serious problem.  What is your SEL ? Likely too large and/or SDy too low.  Several papers mention wood density calibrations. Do refer to the literature to compare with your samples (mean, SD and SEL) and the way the samples are scanned.  Pierre

Posted on Wednesday, February 09, 2011 - 6:17 am:    Dear Pierre,  thank you. Of course, I will chek my samples and probably is a problem in their choice. Is there any reference for rule SECV<=1.05*SEC or difference between SEC (SECV) and SEP? It is more important then R2?!  Nebojsa

Posted on Wednesday, February 09, 2011 - 6:52 am:    Nebojsa,  as I said it's my rule. you can calculate a Ftest on the ratio SECV/SEP, but the result depends on the number of samples in cal and val. SEP is more important than R2, but with SEP = SDy there is no need for analyzes. (except in process control when predictions of the "standard" product will give the same predicted values over time)  Pierre

Posted on Friday, February 18, 2011 - 11:00 am:    Nebojsa,  It appears that the SEP values and the SECV value you find are fairly comparable, and that the SEC is quite low. That suggests to me that you are using too many factors in your model. How many factors have you selected?  You have not mentioned what calibration method you are using. Since you are using Unscrambler, I assume you are using PLS? Have you tried PCR? In my experience, it is easier to avoid overfitting the data by using PCR, particularly if there is considerable error in the lab method. Have you evaluated the reproducibility of the lab method (the SEL that Pierre mentioned)? The results of your validation are limited by the SEL, because the RMSEP must always be >= SEL.  Another way to see if you have used too many factors (or over-fit your data) is to observe the factors. They should look like smooth spectra-like curves, with a minimum of high frequency noise.  Best wishes,  Dave

Standard error of prediction (SEP) should not be greater than 1.3 times the standard error of calibration (SEC)

"As recommended by the instrument/software vendor, generally, standard error of prediction (SEP) should not be greater than 1.3 times the standard error of calibration (SEC) and the bias should not be greater than 0.6 times the SEC (50). High values of SEP or bias indicate that the errors are significantly larger for the new
cross-validation samples and that the calibration data may not include all the necessary variability or be over fit."

(P. 265 in: Stuart L. Cantor, Stephen W. Hoag, Christopher D. Ellison, Mansoor A. Khan, and Robbe C. Lyon (2011). NIR Spectroscopy Applications in the Development of a Compacted Multiparticulate System for Modified Release. AAPS PharmSciTech, Vol. 12, No. 1, March 2011)

"A large difference indicates that too many latent variables are used in the model and noise is modeled. "

(P.318

In Li et al., (2007) Nondestructive measurement and fingerprint analysis of soluble solid content of tea soft drink based on Vis/NIR spectroscopy, J. of Food Eng, 82, 316-323.)

Standard error of cross-validation (SECV or SEP)

P.318

In Li et al., (2007) Nondestructive measurement and fingerprint analysis of soluble solid content of tea soft drink based on Vis/NIR spectroscopy, J. of Food Eng, 82, 316-323.

A large difference indicates that too many latent variables are used in the model and noise is modeled.

P.318

In Li et al., (2007) Nondestructive measurement and fingerprint analysis of soluble solid content of tea soft drink based on Vis/NIR spectroscopy, J. of Food Eng, 82, 316-323.

PLS2 regression give better results than PLS1 regression only if Y variables are strongly correlated

"When several dependent data are available for calibration, two approaches can be used in PLS regression: either properties are calibrated for one at a time (PLS1), or properties are calibrated at once (PLS2). In PLS1 model, the Y response consists of a single variable. When there is more than one Y response a separated model must be constructed for each Y response. In PLS2 model, responses are multivariate. PLS1 and PLS2 models provide different prediction set and PLS2 regression give better results than PLS1 regression only if Y variables are strongly correlated."

(Page 134 in: O. Galtier, O. Abbas, Y. Le Dréau, C. Rebufa, J. Kister, J. Artaud, N. Dupuy 2011. Comparison of PLS1-DA, PLS2-DA and SIMCA for classification by origin of crude petroleum oils by MIR and virgin olive oils by NIR for different spectral regions. Vibrational Spectroscopy 55 (2011) 132–140)

Indirect prediction method may need to check for robustness

"While good calibration models were obtained for dry matter, it seems more difficult to predict acidity based on the NIR spectrum. The concentration of acids in most fruit and vegetables is typically considerably smaller than that of sugars, and probably too small to affect the NIR spectrum significantly. The water absorption bands dominate the spectrum of fruit and vegetables, and it is not likely that minor constituents can be measured well. Obviously, when the concentration of such a minor constituent is correlated to, e.g., sugar content, the calibration results may seem reasonable but then the method is indirect and robustness issues are to be expected when applied to a different batch."

(Nicolaï, B.M., Beullens, K., Bobelyn, E., Peirs, A., Saeys, W., Theron, K.I., Lammertyn, J.

Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: A review (2007) Postharvest Biology and Technology, 46 (2), pp. 99-118. )

Non-linearity can be accounted for by extra latent variables of PLS

"So far there does not seem to be convincing evidence that nonlinear techniques, such as ANNs or kernel- ased methods can really offer advantages with respect to the classical linear algorithms. This is due to the fact that NIR spectroscopy is essentially a very linear technique. Further, Li et al. (1999) stated that both PCR and PLS can provide linear approximations to subtle deviations from ideal linear behaviour by using extra latent variables to account for the nonlinearity."

(Nicolaï, B.M., Beullens, K., Bobelyn, E., Peirs, A., Saeys, W., Theron, K.I., Lammertyn, J.

Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: A review (2007) Postharvest Biology and Technology, 46 (2), pp. 99-118. )

Definition of Robustness of Calibration Model

"Calibration models are called robust when the prediction accuracy is relatively insensitive towards unknown changes of external factors. The main factors which may affect model performance are (Wang et al., 1991): (i) the calibration model developed on one instrument is transported to another instrument that produces instrumental responses that differ from the responses obtained on the first instrument; (ii) the instrumental responses measured on a single instrument drift because of temperature fluctuations, electronic drift, and changes in wavelength or detector stability over time; and (iii) the samples belong to different batches."

(Nicolaï, B.M., Beullens, K., Bobelyn, E., Peirs, A., Saeys, W., Theron, K.I., Lammertyn, J.

Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: A review (2007) Postharvest Biology and Technology, 46 (2), pp. 99-118. )

A sucrose absorption band in the 900–930 nm range

"A number of researchers have shown that the SSC and:or DM of a number of fruits can be
predicted by NIR spectroscopy (e.g. Birth et al., 1985; Dull et al., 1989; Kawano et al., 1992;
Slaughter, 1995). Generally, NIR light from 800 to 1000 nm has been used and success attributed to a sucrose absorption band in the 900–930 nm range."

(V.Andrew McGlone, Sumio Kawano, Firmness, dry-matter and soluble-solids assessment of postharvest kiwifruit by NIR spectroscopy, Postharvest Biology and Technology, Volume 13, Issue 2, April 1998, Pages 131-141, ISSN 0925-5214)

Second-derivative absorbances at 874 and 902 nm had a high correlation with the sugar content of melons

"Abstract
A method for visualizing the sugar content in the flesh of melons was developed. This method was based on the sugar absorption band in the near-infrared (NIR) region to avoid bias caused by the color information of a sample. NIR spectroscopic analysis revealed that each of the two second-derivative absorbances at 874 and 902 nm had a high correlation with the sugar content of melons. A high-resolution cooled charged couple device camera with band-pass filters, which included the above two wavelengths, was used to capture the spectral absorption image of a half-cut melon. A color distribution map of the sugar content on the surface of the melon was constructed by applying the NIR spectroscopy theory to each pixel of the acquired images."

(J Agric Food Chem. 2002 Jan 2;50(1):48-52.

Near-infrared imaging spectroscopy based on sugar absorption band for melons.

Tsuta M, Sugiyama J, Sagara Y.)

Discussion on scatter plot having discontinuous data

"Due to the randomized data split process for calibration and prediction sets, some
graphs show discontinuity (e.g. the TA between 0.14 and 0.18 g/kg). However, the lake of
data observed in the prediction results (Fig. 5) may affect in the calibration results (Table 8)."

(Liu, Y., & Ying, Y. (2007). Noninvasive method for internal quality evaluation of pear fruit using fiber-optic FT-NIR spectrometry. International Journal of Food Properties, 10(4), 877-886.)

SDR = SD/RMSEP: SDR = 2 would mean two group sorting but with much larger error overlap

"A further statistic, SDR, was also used and defined as:
SDR = SD/RMSEP
where SD was the data set standard deviation. In theory (for large normally distributed data set) it is equal to, and indicates more directly than either R2 and RMSEP separately can, the relative predictive performance of a model: the higher the value, the greater the power. We consider a value of three the minimum for sorting/grading purposes, enabling the data set to be split into three groups (of width equivalent to RMSEP) with the extreme two groups not overlapping appreciably. A value of two would mean two groups but with much larger error overlap"

(McGlone, V.A. and Kawano, S., (1998) Firmness, dry-matter and soluble-solids assessment of postharvest kiwifruit by NIR spectroscopy, Postharvest Biology and Technology, 13, 131-141.)

Wavelength of 914 nm was sensitive to the SSC of satsuma mandarins

"However, it is likely that the wavelengths 650-680 nm for SSC mainly attributed to the color of bayberry juice. It is same to the sensitive wavelengths to acidity of 685-695 nm, because there is nonexistent of organic acids in this region of the spectrum. So in our research, to SSC, wavelengths 970-990 nm might be of particular importance, and to acidity, 910-925 nm were better. This found was similar to the earlier literature, such as He (1998) found a wavelength of 914 nm was sensitive to the SSC of satsuma mandarins. And near 900 nm were sensitive wavelengths corresponding to organic acid of oranges."

(Shao, Y. and He. Yong (2007) Nondestructive measurement of the internal quality of bayberry juice using Vis/NIR spectroscopy, Journal of Food Engineering, 79, 1015-1019.)

Important wavelengths can be determined from PLS regression coefficients and difference in spectra

"The wavelengths important in classifying the species of interest were determined based on PLS regression coefficients and differences in spectra."

"The regression coefficients indicating important wavelengths in the calibration model are shown in Fig. 1."

(Fengyou, J., Maghirang, E., Dowell, F. Abel, C. and Ramaswamy, S., 2007. Differentating tobacco budworm and corn earworm using near-infrared spectroscopy, Journal of Economic Entomology, Vol. 100, No. 3, 759-763.)