Large differences between SEC and SEP indicate too many latent variables are introduced in the model and that data noises is also being modeled.

"Acceptable models should
have lower SEC and SEP, high correlation coefficients
and small differences between SEC and SEP. Large differences
indicate that too many latent variables are introduced
in the model and that data noises is also being
modeled."
page 674 in

Visible/near infrared spectrometric technique for
nondestructive assessment of tomato ‘Heatwave’ (Lycopersicum
esculentum) quality characteristics
by Yongni Shao, Yong He , Antihus H. Go´mez , Annia G. Pereir , Zhengjun Qiu ,
Yun Zhang

Journal of Food Engineering 81 (2007) 672–678

Sugar concentrations relied strongly on the 910 nm sugar CH related bands.

Appl Spectrosc. 2003 Feb;57(2):139-45.

Short-wavelength near-infrared spectra of sucrose, glucose, and fructose with respect to sugar concentration and temperature.

Golic M1, Walsh K, Lawson P.

Author information

Abstract

Short-wavelength near-infrared (SW-NIR) (700-1100 nm) spectra of aqueous solutions of sucrose, D-glucose, and D-fructose were monitored with respect to change in temperature and sugar concentration. Sugar OH and CH related vibrations were identified by analysis of the spectra of sugar solutions in deuterium oxide (D2O), and of sucrose-d8 solutions in D2O. Absorption spectra were explained in terms of the second and third overtones of OH stretching vibrations and the third overtone of CH2 and CH stretchings. In deuterated solutions, CH and CH2 higher overtone vibration bands became apparent. The major spectral effect of decreased temperature or increased sugar concentration was a decrease in absorbance at 960 nm and an increase in absorbance at 984 nm, interpreted as an increase in the degree of H bonding. Partial least-squares (PLS) calibrations on sugar concentrations (with spectra collected at several sample temperatures) relied strongly on the 910 nm sugar CH related bands, whereas calibrations on temperature depended equally on all OH associated vibrations (750, 840, 960, and 985 nm).

PMID:

 

14610949

 

DOI:

 

10.1366/000370203321535033

[PubMed - indexed for MEDLINE]

More scattering means more light is absorbed in transmission mode and less light absorbed in reflectance mode.

"A fungal–infected kernel would
also scatter more light than a sound, vitreous kernel because
the invasion of the fungus causes the kernel endosperm to
become powdery (Hesseltine and Shotwell, 1973; Lillehoj et
al., 1976). This scattering would cause more NIR (>750 nm)
radiation to be absorbed in transmission mode, and less NIR
radiation to be absorbed in reflectance mode. Powdery
substances with refractive indices different than air, such as
in the air–endosperm interface of infected kernels, cause
more light to be reflected (Birth and Hecht, 1987), as opposed
to the more crystal–like property of normal kernels."

(Page 1250 in
Pearson, T.C., Wicklow, D.T., Maghirang, E.B., Xie, F., Dowell, F.E. (2001) DETECTING FLATOXIN IN SINGLE CORN KERNELS BY TRANSMITTANCE AND REFLECTANCE SPECTROSCOPY, Transactions of the ASAE, Vol 44(5), 1247 - 1254.)

No deep cooling necessary for InGaAs detector in a range of 900 to 1700 nm

1. "Thermal noise in InGaAs photodetectors is a big issue, so we pay thousands of ... I need 30 – 35 seconds for each sample, will FT-NIR suit? ... with upper limit of 1900 nm, you see that they do not need cooling system and as ... The 2200nm cut off detectors are 10x more noisy than the 1700nmtype, and the ..."

NIR Discussion Forum: Detectors - IM Publications

www.impublications.com › ... › Bruce Campbell's List › Calibration transfer

Feb 17, 2011 - 42 posts

2. "While other sensor technologies are equally capable of covering all or parts of the NIR spectrum covered by standard InGaAs FPAs (900 to 1700 nm), most other technologies have some drawback that make then less attractive. InSb can be used to cover the same range with a high performance sensor but requires cryogenic cooling, resulting in a higher cost, and lower reliability camera due to the sterling cooler. Short wavelength HgCdTe, like InGaAs does not require cryogenic cooling, but is relatively more expensive due to limited commercial availability of he detector material."

From Extended short wavelength spectral response

from InGaAs focal plane arrays

Theodore R. Hoelter, Jeffrey B. Barton

Indigo Systems Corporation, 50 Castilian Drive, Goleta, CA USA 93117

Infrared Technology and Applications XXIX, Bjørn F. Andresen, Gabor F. Fulop, Editors,

Proceedings of SPIE Vol. 5074 (2003) © 2003 SPIE · 0277-786X/03/$15.00

3. "The IK1112 camera is a highly sensitive infrared camera (SWIR, NIR) with 320x256 pixel and a framrate of 110fps. The sensitivity interval reaches from 900nm to 1700nm. The sensor doesn't need active cooling and thus powered by USB bus (no external power supply needed)."

http://www.ehd.de/products/InGaAscameras/InGaAs_SWIR_Infrared_Cameras.html

4. "Unlike other IR detectors, InGaAs detectors do not require deep cooling. However, moderate thermoelectric cooling reduces the detector noise and improves the image quality"

http://83.169.23.21/files/downloads/xenics/eu/NIR_SWIR_FPA_Cameras_eu.pdf

Aluminum presents a constant absorbance throughout the NIR range.

"Aluminum was the chosen material because it presents a constant absorbance throughout the NIR range, i.e. it is optically neutral. This property allows the spectra of the samples to be measured without any interference from the platform." From P. Mishra et al. J. Near Intrared Spectrosc. 23, 15-22 (2015)

1100 to 1800 nm winds up being the critical range for many analysis.

Extracted from
http://www.impublications.com/discus/messages/5/213.html?1060210133

"Then there is the detector selection. If we take the visible/NIR cutoff as where the optimum detector choice changes, it is about at 1100 nm. In the wavelength range below 1100, there is limited vibrational information available, so many scanning instruments have that as the lower end of the wavelength range. If you scan only below 1100, many of us would say you are not really doing NIR Spectroscopy, but rather visible. 
For data collected in the reflection mode, the data above 1800 nm or so is of limited utility because absorptions tend to be so strong as to make quantitation in the presence of surface reflection less reliable. 
So, in my experience, 1100 to 1800 nm winds up being the critical range for many analysis, and I recommend to my clients that they make sure that the instrument they buy operates reliably in this range. However, the adjoining wavelengths can add versatility that at times can be very important for specific applications. 

By DJ Dahm "

"Generally, 700 -1800 nm covers almost all applications. If you are considering 700 – 1100 nm then more caution would be advised There will be some calibrations which cannot be replaced with Herschel wavelengths.

By Tony Davies (Td)"

Correlation is not the most important statistic in regression analysis.

Extracted from
http://www.impublications.com/discus/messages/5/6974.html?1280931124


"Correlation is not the most important statistic in regression analysis. The standard error of prediction (SEP) which is the error in predicting the independent validation samples is the first test of any NIR method. You can compare this to the SER for your reference analysis. 

Hope this helps, 

Best wishes, 

Tony"

"The correlation coefficient is influenced by the range. The most important information is given to us by the SEP which answers the question "How variable are the answers from this model"? If the SEP is less than that required for the analysis then you have a method. R and r^2 are important but not the most important or useful statistics. 

Best wishes, 

Tony"

(Tony Davies (td) )

Effect of stray light on NIR spectrum (e.g. flattened peak)

From NIR Discussion Forum » ICNIRS
(http://www.impublications.com/discus/messages/43/9407.html?1298057637)

Alisha (agnosus) Member Username: agnosus Post Number: 14 Registered: 1-2009

Posted on Tuesday, February 08, 2011 - 3:48 am:    Thanks Howard & Karl, It sure makes a lot more sense now. I am using an FT system for tablet analysis in transmission mode using a narrow InGaAs detector. What I have noticed is that I never get absorbance values as high as 7 AU (usually 3 AU is the max) and also I usually don't get much information above 1400 nm. I am trying to understand if this is the detector sensitivity (dark noise) or stray light that is putting a cap on my absorbance values? Also note the spectral shape I am getting that is very different to that of Foss machine (this is a different formulation so comparison might not be right). I have attached a spectrum below:

Howard Mark (hlmark) Senior Member Username: hlmark Post Number: 399 Registered: 9-2001

Posted on Tuesday, February 08, 2011 - 9:00 am:    Alisha - the "flattopping" above 1500 nm is certainly characteristic of stray light. On the other hand, FTIR is known to be resistant (not immune, but definitly resistant) to stray light, so there's somewhat of a contradiction here. One way you can easily tell how much of the signal is noise is to run the spectrum two times in a row, and then compare the two spectra. The parts that are different represents noise, the parts that are the same represent actual signal. It's possible that the limit you're running into is digital, in the number of bits in the instrument's A/D converter. Usually that shows up differently, but when you've got a "mystery" you have to consider all possibilities, as Sherlock Holmes famously said in different words.

Karl Norris (knnirs) Senior Member Username: knnirs Post Number: 45 Registered: 8-2009

Posted on Tuesday, February 08, 2011 - 10:25 am:    Alisha, Your spectrum indicates you are not using an air reference. What is your reference,and can you provide the spectrum of your reference? I would guess that your reference has a Log(1/T) of about 3, so that your sample has a Log(1/T) of from 2.5 to 6. Stray light in measuring a sample such as a tablet is often from radiation going around the tablet and reaching the detector without going through the tablet. I will be glad to give you the benifit of my many years of experience in NIR, but I think we should switch to e-mail. My address is: knnirs@gmail.com. Karl

From http://www.impublications.com/discus/messages/5/348.html?1106275130

A quick determination that may indicate the level of stray light would be to measure a highly absorbing sample, say 5mm of water or more in transmission, or a very absorbing sample in reflection, and note where the peaks are flattened and appear "saturated". That is the limit Tony refers to, and the percent stray light is given approximately by the conversion from the log function, as the light measured is essentially all stray light. If the peak "saturates" at 2 AU, the stray light is 1% of the NIR at that region. The stray light you thus determine at 1940 or 2130 n(From m (for example) is also present at other measurement wavelengths. 

Best wishes, 
Dave


(From Brown, C.W. (2004) Ultraviolet, Visible, Near-Infrared Spectrophotometers. Analytical Instrumentation Handbook, Third Edition  edited by Jack Cazes, p.127 - 140)

"The second source of instrumental error is stray light. This problem can be inherent in the spectrometer, but it can also be caused by the operator. Stray light is any light reaching the detector without passing through the sample. Thus, if a sample were completely opaque at a certain wavelength, any photons that were detected would be due to stray light. These photons could be passing around the sample, through holes in the sample as might be caused by air bubbles in a liquid, or they "might by the result of poor shielding, permitting room light to reach the detector from some external light source, for example, room lights. The effect of stray light is to add a constant power (intensity) of light, Ps, to both the numerator and denominator in the absorbance expression:

A = log ((P0 + Ps)/(P + Ps))

As the concentration increases, P approaches zero, and A asymptotically approaches a maximum level given by log[(P0 + Ps)/Ps]. For Ps equal to 10% of P0, the maximum value of A is 1.04. As this is a log relation, A approaches 1.04 asymptotically with the concentration. Thus, stray light should be suspected any time when nonlinear data is encountered.
     Stray light in a spectrophotometer can be measured by inserting an opaque blocking filter into the optical path. A signal observed by the detector under these condition is due solely to stray radiation. A 10 g/L solution of potassium iodide does not transmit appreciably below 259 nm, but is essentially completely transparent above 290 nm when observed in a 10 nm cuvet (Poulson, 1964). If a spectrophotometer is set at a lower wavelength, say 240 nm, any signal that is observed must originate from stray radiation. In determining the stray light below 259 nm, the cuvet holding the solution should be placed in the spectrophotometer and scanned from the longer wavelength, transparent region to the shorter, opaque region. In this way the detector signal will be gradually decreased to its lowest level. False readings can sometimes be obtained by inserting a cuvet containing the sample in the spectrophotometer at an opaque wavelength, as the abrupt decrease in signal may not register correctly."

Noise in regression coefficients: an indicator of data overfitting

"One well-known sign of over-fitting is the appearance of noise in regression coefficients; this often takes the form of a reduction in apparent structure and the presence of sharp peaks with a high degree of directional oscillation, features which are usually estimated subjectively. "

(Preventing over-fitting in PLS calibration models of near-infrared (NIR) spectroscopy data using regression coefficients

A.A. Gowen, G. Downey, C. Esquerre, C. P. O'Donnell

Journal of Chemometrics

Special Issue: WSC-7: 7th Winter Symposium on Chemometrics

Volume 25, Issue 7, pages 375–381, July 2011)

Too few samples and/or parameter difficult to predict and/or noise in the reference method are possible causes of large difference between SEC and SECV

From NIR Forum discussion Posted on Tuesday, February 08, 2011 - 4:42 am:    Dear all,  First, I want to say thank you in advance for every answer. I'm first time on this forum.  I have read a lot of things here and I  think that it is really useful. I have a question regarding the prediction of wood properties with NIR spectra.  I have a set of spectra from wood sample (calibration and test set) and I would like develop the best model for wood properties (eg wood density). However, I get higher error for cross validation (SECV) then for calibration set (SEC) and test set(SEP).Maybe I make a mistake in the application of cross validation. I use Unscrambler software. Can anyone tell me how to use the option of cross validation in Unscrambler software?  Thank you and best regards,  Nebojsa

Posted on Tuesday, February 08, 2011 - 5:44 am:    HI,  How large is the difference between SEC and SECV?  Before calibrating, you have to know the SEL (error of the reference method) and the SD of the calibration sample.  How many samples?  the gap between SEC and SECV is due to  - Too few samples and/or  - Parameter difficult to predict and/or  - Noise in the reference method.  SECV is always higher than SEC. My rule is to have SECV<=1.05*SEC with 2 groups of CV. Then I am pretty sure the model is robust.  Pierre

Posted on Tuesday, February 08, 2011 - 7:22 am:

NIR model should produce SEP lesser than 2.0SEL or 1.5SEL at least.

"There is also a statement of criteria that the NIR model should produce SEP lesser than 2.0SEL or 1.5SEL at least. If the SEP is larger then your model needs more calibration development and/or more samples."

(https://www.researchgate.net/post/Does_anybody_know_how_to_compare_NIR_error_and_lab_analysis_error1)



"Standard errors of performance (SEP) are frequently twice the magnitude of the standard error of the laboratory (SEL) in successful NIR calibrations. In spite of this the repeatability of NIR measurement is almost always better than the repeatability of the reference procedure."

(V. MÍKA, J. POZDÍŠEK, P. TILLMANN, P. NERUŠIL, K. BUCHGRABER4, L. GRUBER (2003) Development of NIR calibration valid for two different grass sample collections, Czech J. Anim. Sci., 48, 2003 (10): 419–424.)



"Westerhaus (1985, cited by Stimson, et al 1991) recommended that the SEP should be no greater than twice the SEL."

(G. McL. Dryden (2003) Near Infrared Reflectance Spectroscopy: Applications in Deer Nutrition. A report for the Rural Industries Research and Development Corporation, RIRDC Publication No W03/007 RIRDC Project No UQ-109A)

It is easier to avoid overfitting the data by using PCR over PLS, particularly if there is considerable error in the lab method (SECV <= 1.05*SEC)

From NIR Forum discussion Posted on Tuesday, February 08, 2011 - 4:42 am:    Dear all,  First, I want to say thank you in advance for every answer. I'm first time on this forum.  I have read a lot of things here and I  think that it is really useful. I have a question regarding the prediction of wood properties with NIR spectra.  I have a set of spectra from wood sample (calibration and test set) and I would like develop the best model for wood properties (eg wood density). However, I get higher error for cross validation (SECV) then for calibration set (SEC) and test set(SEP).Maybe I make a mistake in the application of cross validation. I use Unscrambler software. Can anyone tell me how to use the option of cross validation in Unscrambler software?  Thank you and best regards,  Nebojsa

Posted on Tuesday, February 08, 2011 - 5:44 am:    HI,  How large is the difference between SEC and SECV?  Before calibrating, you have to know the SEL (error of the reference method) and the SD of the calibration sample.  How many samples?  the gap between SEC and SECV is due to  - Too few samples and/or  - Parameter difficult to predict and/or  - Noise in the reference method.  SECV is always higher than SEC. My rule is to have SECV<=1.05*SEC with 2 groups of CV. Then I am pretty sure the model is robust.  Pierre

Posted on Tuesday, February 08, 2011 - 7:22 am:    Dear Pierre,  thank you for your answer.  How large is the difference between SEC and SECV?  - The difference is very large SECV - 0.043 SEC - 0.020  How many samples?  - 74 for calibration and 20 for test set.  - SD for calibration is 0.049 (mean 0.698) and for test set is 0.047 (mean  0.713)  - for cross validation I use setup random, number of segments 10 and samples  per segment 7.  Nebojsa

Posted on Wednesday, February 09, 2011 - 4:35 am:    Bonjour,  it means R2CV ~~ 0.0. You have a serious problem.  What is your SEL ? Likely too large and/or SDy too low.  Several papers mention wood density calibrations. Do refer to the literature to compare with your samples (mean, SD and SEL) and the way the samples are scanned.  Pierre

Posted on Wednesday, February 09, 2011 - 6:17 am:    Dear Pierre,  thank you. Of course, I will chek my samples and probably is a problem in their choice. Is there any reference for rule SECV<=1.05*SEC or difference between SEC (SECV) and SEP? It is more important then R2?!  Nebojsa

Posted on Wednesday, February 09, 2011 - 6:52 am:    Nebojsa,  as I said it's my rule. you can calculate a Ftest on the ratio SECV/SEP, but the result depends on the number of samples in cal and val. SEP is more important than R2, but with SEP = SDy there is no need for analyzes. (except in process control when predictions of the "standard" product will give the same predicted values over time)  Pierre

Posted on Friday, February 18, 2011 - 11:00 am:    Nebojsa,  It appears that the SEP values and the SECV value you find are fairly comparable, and that the SEC is quite low. That suggests to me that you are using too many factors in your model. How many factors have you selected?  You have not mentioned what calibration method you are using. Since you are using Unscrambler, I assume you are using PLS? Have you tried PCR? In my experience, it is easier to avoid overfitting the data by using PCR, particularly if there is considerable error in the lab method. Have you evaluated the reproducibility of the lab method (the SEL that Pierre mentioned)? The results of your validation are limited by the SEL, because the RMSEP must always be >= SEL.  Another way to see if you have used too many factors (or over-fit your data) is to observe the factors. They should look like smooth spectra-like curves, with a minimum of high frequency noise.  Best wishes,  Dave

Standard error of prediction (SEP) should not be greater than 1.3 times the standard error of calibration (SEC)

"As recommended by the instrument/software vendor, generally, standard error of prediction (SEP) should not be greater than 1.3 times the standard error of calibration (SEC) and the bias should not be greater than 0.6 times the SEC (50). High values of SEP or bias indicate that the errors are significantly larger for the new
cross-validation samples and that the calibration data may not include all the necessary variability or be over fit."

(P. 265 in: Stuart L. Cantor, Stephen W. Hoag, Christopher D. Ellison, Mansoor A. Khan, and Robbe C. Lyon (2011). NIR Spectroscopy Applications in the Development of a Compacted Multiparticulate System for Modified Release. AAPS PharmSciTech, Vol. 12, No. 1, March 2011)

"A large difference indicates that too many latent variables are used in the model and noise is modeled. "

(P.318

In Li et al., (2007) Nondestructive measurement and fingerprint analysis of soluble solid content of tea soft drink based on Vis/NIR spectroscopy, J. of Food Eng, 82, 316-323.)

Standard error of cross-validation (SECV or SEP)

P.318

In Li et al., (2007) Nondestructive measurement and fingerprint analysis of soluble solid content of tea soft drink based on Vis/NIR spectroscopy, J. of Food Eng, 82, 316-323.

A large difference indicates that too many latent variables are used in the model and noise is modeled.

P.318

In Li et al., (2007) Nondestructive measurement and fingerprint analysis of soluble solid content of tea soft drink based on Vis/NIR spectroscopy, J. of Food Eng, 82, 316-323.

PLS2 regression give better results than PLS1 regression only if Y variables are strongly correlated

"When several dependent data are available for calibration, two approaches can be used in PLS regression: either properties are calibrated for one at a time (PLS1), or properties are calibrated at once (PLS2). In PLS1 model, the Y response consists of a single variable. When there is more than one Y response a separated model must be constructed for each Y response. In PLS2 model, responses are multivariate. PLS1 and PLS2 models provide different prediction set and PLS2 regression give better results than PLS1 regression only if Y variables are strongly correlated."

(Page 134 in: O. Galtier, O. Abbas, Y. Le Dréau, C. Rebufa, J. Kister, J. Artaud, N. Dupuy 2011. Comparison of PLS1-DA, PLS2-DA and SIMCA for classification by origin of crude petroleum oils by MIR and virgin olive oils by NIR for different spectral regions. Vibrational Spectroscopy 55 (2011) 132–140)

Indirect prediction method may need to check for robustness

"While good calibration models were obtained for dry matter, it seems more difficult to predict acidity based on the NIR spectrum. The concentration of acids in most fruit and vegetables is typically considerably smaller than that of sugars, and probably too small to affect the NIR spectrum significantly. The water absorption bands dominate the spectrum of fruit and vegetables, and it is not likely that minor constituents can be measured well. Obviously, when the concentration of such a minor constituent is correlated to, e.g., sugar content, the calibration results may seem reasonable but then the method is indirect and robustness issues are to be expected when applied to a different batch."

(Nicolaï, B.M., Beullens, K., Bobelyn, E., Peirs, A., Saeys, W., Theron, K.I., Lammertyn, J.

Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: A review (2007) Postharvest Biology and Technology, 46 (2), pp. 99-118. )

Non-linearity can be accounted for by extra latent variables of PLS

"So far there does not seem to be convincing evidence that nonlinear techniques, such as ANNs or kernel- ased methods can really offer advantages with respect to the classical linear algorithms. This is due to the fact that NIR spectroscopy is essentially a very linear technique. Further, Li et al. (1999) stated that both PCR and PLS can provide linear approximations to subtle deviations from ideal linear behaviour by using extra latent variables to account for the nonlinearity."

(Nicolaï, B.M., Beullens, K., Bobelyn, E., Peirs, A., Saeys, W., Theron, K.I., Lammertyn, J.

Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: A review (2007) Postharvest Biology and Technology, 46 (2), pp. 99-118. )

Definition of Robustness of Calibration Model

"Calibration models are called robust when the prediction accuracy is relatively insensitive towards unknown changes of external factors. The main factors which may affect model performance are (Wang et al., 1991): (i) the calibration model developed on one instrument is transported to another instrument that produces instrumental responses that differ from the responses obtained on the first instrument; (ii) the instrumental responses measured on a single instrument drift because of temperature fluctuations, electronic drift, and changes in wavelength or detector stability over time; and (iii) the samples belong to different batches."

(Nicolaï, B.M., Beullens, K., Bobelyn, E., Peirs, A., Saeys, W., Theron, K.I., Lammertyn, J.

Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: A review (2007) Postharvest Biology and Technology, 46 (2), pp. 99-118. )

A sucrose absorption band in the 900–930 nm range

"A number of researchers have shown that the SSC and:or DM of a number of fruits can be
predicted by NIR spectroscopy (e.g. Birth et al., 1985; Dull et al., 1989; Kawano et al., 1992;
Slaughter, 1995). Generally, NIR light from 800 to 1000 nm has been used and success attributed to a sucrose absorption band in the 900–930 nm range."

(V.Andrew McGlone, Sumio Kawano, Firmness, dry-matter and soluble-solids assessment of postharvest kiwifruit by NIR spectroscopy, Postharvest Biology and Technology, Volume 13, Issue 2, April 1998, Pages 131-141, ISSN 0925-5214)