Revista Mexicana Ciencias Agrícolas   volume 13   number 4   May 16 - June 29, 2022

DOI: https://doi.org/10.29312/remexca.v13i4.2514

Article

Filling missing daily data in records of climatological time series

Jesica Natalia Valencia-González¹§

Ramón Arteaga-Ramírez¹

Mario Alberto Vásquez-Peña¹

Abel Quevedo-Nolasco²

¹Postgraduate in Agricultural Engineering and Integral Use of Water-Chapingo Autonomous University. Mexico-Texcoco Highway km 38.5, Texcoco, State of Mexico, Mexico. CP. 56230. (nvg.912@gmail.com, mvazquezd@correo.chapingo.mx).

²Postgraduate College. Mexico-Texcoco Highway km 36.5, Montecillo, Texcoco, State of Mexico, Mexico. CP. 56230. (anolasco@colpos.mx).

§Corresponding author: rarteagar@taurus.chapingo.mx.

Abstract

The lack of daily data in weather stations is frequent and this does not allow the series to be used in agroclimatic studies. With the above, the temporal and spatial variation of the variables that make up the agroclimate of a region is not known. The objective of this work was to estimate and verify by means of the methods: normal ratio, Fourier series and square of the inverse of the distance, the method with the least error for the filling of missing daily data of the variables: precipitation, solar brightness, evaporation, maximum temperature, minimum temperature and relative humidity of the weather stations surrounding the rice production area in the department of Valle del Cauca, Colombia. Nine stations were analyzed, which do not present distances greater than 50 km, nor altitudinal differences of more than 750 m. These were used with different study periods according to the variable in progress and the methods were evaluated with the statistical indices: root mean square error and coefficient of determination, the first allowed knowing the maximum admissible value of error and the second, the level of adjustment between the observed and the estimated values. Therefore, these allowed inferring that the variable solar brightness and evaporation obtained the best results with the normal ratio; the minimum temperature and relative humidity with the Fourier series and the square of the inverse of the distance for precipitation and maximum temperature.

Keywords: Fourier series, missing data, normal ratio, square of the inverse of the distance.

Reception date: March 2022

Acceptance date: May 2022

Introduction

Climate analysis requires time series with continuous, homogeneous meteorological data that contain the maximum possible time interval (Nauman and Vargas, 2012). The absence of data in meteorological records is common due to several situations, such as: the replacement of the operator, failures of the recording devices or operational negligence, which limits the carrying out of agroclimatic studies (Gallegos et al., 2016). Therefore, Medina et al. (2008) propose that an attempt should be made to complete the databases, with data generated by some statistical methodology that guarantees their randomness, and that they adjust to the structure of the underlying stochastic process.

From the point of view of Nauman and Vargas (2012), a reference station or series is considered to be one that has a wide range of records (if possible, it must cover the instrumental period), in addition, they must represent different or particularized climatic regions. It is necessary to carry out the diagnosis on reference weather stations since the stability of the results is conditioned to the greater availability of information. As mentioned by Mintegui and López (1990), the estimation of missing data has generated methods that allow completing the climatological databases, where the variables in progress, regardless of their theoretical basis, respond to the structure of a statistical study.

The normal ratio has been a useful method to complement missing data of a series from the data of the series of the n neighboring and contemporary stations, which have a high degree of correlation with the series to be completed (Paulhus and Kohler, 1952). Similarly, harmonic analysis using Fourier series has been used to estimate daily series of maximum and minimum temperature, pressure, evaporation and relative humidity, since these meteorological elements behave as a periodic function, being represented as the sum of sines and cosines, assigning each one a weighting coefficient, whose treatment has tolerated adapting processes of transmission, compression and reconstruction of the required information (Cortés et al., 2007).

The square of the inverse of the distance (IDW) is one of the most used methods in hydrology, climatology and meteorology, since the area that is interpolated must be that of a location-dependent variable and thus estimate values at unknown points, with the distance between the stations (Eischeid et al., 2000). The objective of this work was to estimate and verify by means of the methods: normal ratio (NR), Fourier series (FS) and square of the inverse of the distance (IDW), the method with the least error for the filling of the missing daily data of the variables precipitation, solar brightness, evaporation, maximum temperature, minimum temperature and relative humidity of the weather stations surrounding the rice production area in the south of the department of Valle del Cauca, Colombia.

Materials and methods

A bibliometric review of the scientific papers on the methods of filling missing data was carried out in the Scopus database: ‘normal ratio, Fourier series and the square of the inverse of distance’. This library was chosen for having a wide variety of quality scientific journals, as well as a metasearch engine that facilitates the evaluation of scientific production by various filters (Biblioteca Scopus, 2019).

Location of the study area

Rice production in the Department of Valle del Cauca (Colombia) includes the municipalities of Jamundí, Quinamayo, Timba, Potrerito and Villa Paz, which border the department of Cauca (Figure 1).

Figure 1. Location of the study area.

Weather stations used

Nine stations around the rice-producing area (not exceeding 50 km away) were chosen, which provided information on daily records of the variables: Precipitation (mm), maximum and minimum temperature (°C), relative humidity (%), solar brightness (sunshine hours) and evaporation (mm), with more than 20 years of information. The databases were obtained from the Institute of Environmental Studies of Colombia (IDEAM, for its acronym in Spanish) and from the hydro-climatological bulletins of the Autonomous Corporation of Valle del Cauca (CVC) (Table 1).

Table 1. Description of weather stations

Characterization of information from weather stations

The selected stations do not have the same period of information of the climatological variables of the current study, those stations whose records complied with the greatest recorded information of daily data, which are observed at 8:00 am, were chosen, thus establishing the study periods for each variable and the total percentage of missing data (Table 2).

Table 2. Missing data for each variable according to its study period.

NS= number of stations; SP= study period; NY= number of years; TD= total data for each variable; TMD= total missing data; %MD= percentage of missing data.

Estimation of missing daily data

The filling of the daily historical series of the study area was carried out using the following methods.

Normal ratio (NR)

With the climatol package programmed in R, similar to the methodology proposed by Guijarro (2004), whose process consists of: the calculation of each series, as if it did not exist, by interpolation of standardized data of neighboring series, the analysis of the differences between the calculated and observed series to detect erroneous data, jumps in the mean and trends and finally the automatic filling of the missing data and the correction of isolated errors. The equation of the method is as follows:

.

Where: PX= missing precipitation data at station x, Nx, Na, Nb, …, Nn= value of the average daily precipitation of the station of the missing data (x) and the auxiliary stations a, b, n (means of all historical series) and Pa, Pb, Pn= value of the precipitation recorded in the auxiliary stations of the day where the data in station x is missing.

Harmonic analysis of Fourier series (FS)

This is a function used for periodic variables such as: maximum and minimum temperature, relative humidity and evaporation. It is known as the periodic function s(t), where the coefficients ai and bi are calculated and, as a result, the signal s(t) can be broken down into the sum of sines and cosines, whose frequencies are harmonic multiples of the fundamental frequency (Nauman and Vargas, 2012). It is estimated as follows:

     ,  .

Where: P= is the period P= 2 π/ω; ω= angular frequency function; N= is the number of data to be used; cos= the cosine function; sen= the sine function. In this study, this method was processed in the Excel program version 2018. To determine the homogeneity of the data of the meteorological variables, a quantification of missing data was carried out and subsequently, the information considered discrepant or inconsistent was eliminated, in order to obtain more representative data groupings; and as previously indicated, the lost data allowed estimating the statistical indices proposed to evaluate the behavior of each method with respect to each variable.

Square of the inverse of distance (IDW)

The procedure of the square of the inverse of the distance is explained by Campos (1998), where the influence of the value of the variable in a station, for the calculation of this at any point, is inversely proportional to the distance between the two points.

  .

 Where: Px= the estimated data of the incomplete station; Pi= rainfall values of the rain gauges used for estimation; di= distance from each place to point that is being estimated; N= number of neighboring stations.

According to Campos (1998), it is recommended to use four auxiliary stations (the closest), so that each one is located in one of the quadrants that define the coordinate axes that pass through the incomplete station, usually north-south and east-west. The procedure was performed in the Excel 2018 program. To use these methods, a loss of observed data was made, which were calculated and then compared in order to define the goodness of each of these.

That information belonging to the actual data volume per variable (TD) and that was later assumed as non-existent during the application of the filling methods was called ‘lost data’. The number of lost data was set according to the percentage of missing data of the TD (Table 2 and 3). Thus, a random percentage of data was considered to perform the test methods, and to use this information to calculate the statistical indices.

Table 3. Lost data for each variable according to its study period.

TD= total data of each variable; TLD= total lost data; %LD= percentage of lost data.

Estimation of the error

In order to estimate the accuracy of the above methods to reconstruct the missing data and select the best one, the root mean square error (RMSE) (Teegavarapu and Chandramouli, 2005) and the coefficient of determination (R²) (Cervantes et al., 2013) were used. The ideal model is when R²≈ 1 or with a good fit ≥ 0.5 and the RMSE≈ 0 (Pereira, 2004).

 .

Where: Vobs= observed data; Vest= estimated data; n= number of observations or estimates.

 .

Where: ai= data estimated by the method; ti= observed data; = average of the data estimated by the method;  = average of the observed data.

Results and discussion

In a search in Scopus, 17 documents related to the use of Fourier series or analysis, from 1986 to the present, were found. Next, the inverse distance weighting (IDW) method, 46 files at the daily study level, between 2001 and 2019, were recognized. Scopus records 33 documents with the use of the normal ratio (NR) method, which correspond to the study areas of: engineering, agriculture, earth and environmental sciences, in the estimation of missing data at the daily level, used from 1963 to the present date. This search shows recent evidence on the present topic, which suggests that the current publications may be striking and point to agricultural research achieving, in the long term, leading agriculture to reach other promising research niches in the future.

Precipitation

For this variable, nine stations were used to complete its records and error (RMSE) intervals from 7.6 mm to 13 mm and from 6.8 mm to 13.9 mm were obtained with the NR and IDW methods, respectively, these errors are accepted since, for the study area, the annual precipitation corresponds to 1 706 mm year-1 and the lowest precipitation occurs in July with about 62 to 80 mm month-1. The values obtained are higher than those reported by Kanda et al. (2017), who completed this variable for 12 stations of 31 years of daily records and estimated errors (RMSE) for the entire historical series, for seven, fifteen and thirty missing days accumulated and the average error of the historical series; through the IDW method, they obtained 2.531, 2.497, 2.524 and 2.432 mm respectively and in the NR method, they report 2.375, 2.382, 2.387 and 2.372 mm correspondingly.

Likewise, Xia et al. (1999) report an average error of 1.5 and 1 mm day-1 for IDW and NR in a study conducted for 44 stations with a period of 5 years of daily records. However, a maximum R² between the stations of 0.45 and 0.48 was obtained in the NR and IDW methods, these are higher according to Gallegos et al. (2016), they indicate an R² value of 0.376 and 0.363 in the NR and IDW methods for seven auxiliary stations used in San Luis Potosí, Mexico. By implication, distance is a logical criterion that must be considered in the estimation of filling of climatic series, obviously, neighboring stations should be close to the point to be analyzed, without intermediate obstacles such as mountains and barriers that modify their behavior (Vincent and Gullet, 1999).

Therefore, an aspect related to the distance is height, since for the study area the stations do not exceed 50 km horizontally or 750 m vertically, this allows having similar climates, as in the Ing. Miranda and Ing. Cauca stations, which are 13 km apart and are 28 m high from each other, and yet they show similar fits of R² in two different filling methods (Figure 2).

Figure 2. Relationship of the observed and estimated precipitation of the Ing. Miranda station through the NR method.

Having said the above, in order to obtain more adequate fillings in large amounts of daily information (greater than 10 years of daily records), some authors suggest using stations no more than 50 km apart (De Luis et al., 2007; Toro et al., 2015). Until now, it has been shown that a reference series is calculated by a weighted average between neighboring stations, by using the correlation coefficient or distance as a weighting factor; however, the IDW method is the one that is reported as the one that best fits the filling of this climatological series and this agrees with what was obtained in this work (Teegavarapu and Chandramouli, 2005; Toro et al., 2015).

Solar brightness

Regarding the variable solar brightness (sunshine hours), a range of errors (RMSE) from 1.6 h to 2.4 h and from 1.4 to 2.5 h was obtained, with coefficients of determination of 0.71 and 0.78 by the NR and IDW methods for the five stations used (Figure 3). These results are acceptable and lower than those of Grossi et al. (2010), who achieved similar daily averages of 2.1, 2.9, 2.5 and 2.2 sunshine hours for January, October, November and December, respectively, by completing monthly time series of 20 years of sunshine hours in 11 stations in Uruguay using the Kriging method.

In Latin America, very few countries recognize the importance of solar radiation and lack historical data from field trials on this climatic element (Degiovanni et al., 2010), so that, given the similarities they show in the RMSE index and the R² of the estimates, either of the two methods could be used; since, the coefficient determination allows distinguishing that there is dependence between the weighting parameters or weights, and the distance of the stations (Toro et al., 2015).

Figure 3. Relationship of the observed and estimated solar brightness of the Ing. Cauca station through the NR method.

Minimum temperature

In this case, maximum RMSE values of up to 1.5 °C, 1.03 °C, 1.64 °C, and R² of 0.21, 0.25, 0.1 occur in the NR, FS and IDW methods used for the five stations used, it was observed that they have a trend with each other, although the highest RMSE are obtained with the IDW method. When relating the estimated values with those observed through each method, low coefficients of determination were found; therefore, they do not show a significant relationship (Figure 4). A characteristic of this argument is that, for this variable, there is little spatial and temporal variability, in addition to the altitudinal characteristics of the place and its equatorial location.

Figure 4. Relationship of the observed and estimated minimum temperature of the Ciat Quilichao station.

Previous studies highlight that the NR method is viable given that the RMSE results obtained are below 3.5 °C, reported by Kanda et al. (2017), who completed the number of missing data in 12 stations of Karakorum Himalaya. The reliability of the NR and IDW methods is corroborated by Gallegos et al. (2016), who reveal RMSE values of 4.1 and 3.7 °C respectively, for a study period of 16 years of daily records with 25.4% of missing information, it is interesting to note that, for this study, there is 24% of missing information in series of 28 years of daily records and the statistical indices of RMSE were much lower; which indicates that both methods will be more suitable for completing missing information in periods of study of more than 25 years.

Also, a work used by Cervantes et al. (2019) is shown, which shows RMSE results not greater than 0.16 °C, for estimates made to four stations for a study period of five years of daily records. The minimum temperature occurs more regularly over time; since, the temporal variations of the meteorological elements are directly or indirectly related to the movements of the earth (Riehl, 1965). Thus, the best method of the minimum temperature is chosen, the Fourier series after the use of its second harmonic. This method allows calculating the most likely missing value of the temperature of any day of the year of a given place.

Maximum temperature

Regarding the filling of the historical series of the maximum temperature, the results of the NR, FS and IDW methods obtained by the RMSE were 1.9 °C, 2 °C, 1.8 °C, respectively. It was observed that the RMSE values are similar to each other, comparing the values reported by Kanda et al. (2017), in the NR and IDW methods, they found 2.6 and 2.5 °C of maximum error, the values being higher than those obtained in this study.

One of the representative estimates made by the NR and IDW methods for the maximum temperature was made for the Ing. Bengala station, which presented the value of R²= 0.5 (Figure 5); this indicates that these methods have a moderate goodness of fit with respect to this variable. A different case occurs with the Ing. Cauca station, which did not obtain an acceptable coefficient of determination. Gallegos et al. (2016) estimated missing records for this variable and obtained maximum R² fits of 0.641 and 0.493 for NR and IDW for 16 years of daily records and minimum fits of up to 0.25 of R² in both methods, which indicates that those obtained in this work are admissible; Cervantes et al. (2019) estimated 229 missing data using the FS method in a five year historical series and reported RMSE values of up to 0.149 °C in four stations, which indicates that this method shows better results with fewer missing data.

Figure 5. Relationship of the observed and estimated maximum temperature of the Ing. Bengala station.

Relative humidity

Once the variable relative humidity was diagnosed, throughout the data set, a wide range in its daily records was identified, which range from 45.7% to 99.4% in the four available stations; for which, in the filling of this series, it is allowed to admit up to 10.4%, 9.02%, 8% of RMSE for the NR, FS and IDW methods, considering that, for this variable, the way to complete the missing data in the historical records has not been explored in great detail at a daily level, these methods have been chosen for this study.

On the other hand, Coutinho et al. (2018) completed this variable at the monthly level for a 13-year study, in which they obtained values of error lower than 2.9% with four different methods. These results are admissible, since this periodic variable tends to be represented with values greater than 70% in places near the equator (such as the location of this study) and additionally, the altitudinal difference between the stations used for this estimate does not exceed 150 masl (Araya, 2011).

Regarding the results acquired by the R², they were: 0.07, 0.32, 0.18 of the NR, FS and IDW methods, whose values are not an adequate statistic that by itself can predict the behavior of said variable (Teegavarapu and Chandramouli, 2005), since the data differ from each other due to the filling process, it can be noted, among these representations, the advantage of showing not only the data that were not randomly replaced by missing data, it also gives an idea of how much the data deviate from the actual data (Araya, 2014).

Of the stations used, it is observed that Ciat Quilichao is the one that tends to show a better fit in the estimation using the FS method. This is possible thanks to the high degree of humidity shown by the study area, since, for the observed and estimated values, a large number of concentrated data is evident (Figure 6). On the other hand, the relationship of the observed and the estimated humidity for the Ing. Miranda station with the NR method, an observed value obtained a set of estimated data for the current evaluation period.

Figure 6. Relationship of the observed and estimated relative humidity of the Ciat Quilichao station through the FS method.

Evaporation

The evaporation showed RMSE values of 1.8 mm, 2.1 mm, 1.61 mm, in the NR, FS and IDW methods, whose average error obtained is 1.84 mm in the four stations used, being considered minimal. An interpretation of the results presented is that the filling of the historical series of this variable is more convenient using the second-order approximation of the Fourier series, because, for the selected study area, there is a wide variability in the relative humidity, and this directly influences the manifestation of evaporation and implicitly influences the reconstruction of its time series.

The NR and IDW methods for the Ciat Quilichao station show to be convenient, since, in their regressions, they exhibit slight trends or accumulation in the slope between the values of 3 and 5.5 mm of the observed evaporation. Compared to the Topacio station, this lacks a clear trend, and a wide distribution was observed in the lower right part of the regression, which allows inferring that this variable for this station presents some extreme values in certain sections of the present historical series (Figure 7).

Figure 7. Relationship of the observed and estimated evaporation of the Topacio station through the FS method.

The scientific literature developed so far externalizes that the filling of missing data of climatological series comprises a detailed and cautious analysis by variable, of which, it could be mentioned that the lack of an explicit method for each case study, if not of a method that adjusts to the behavior of the variable, is conclusive.

Conclusions

It is determined that the method of the square of the inverse of the distance was the one that behaved best for the filling of the different variables used, followed by that of the normal ratio, so both are recommended for the filling of series of daily climatological data. A method that should not be applied to all variables is that of Fourier series, since it must be used in periodic series in time. With the observed data of the variables and those estimated with the different methods, admissible errors were obtained for all the variables, not so for the coefficient of determination, the latter is due to the geographical, latitudinal and altitudinal characteristics of the study area.

Cited literature

Araya, J. L. 2011. Resultados de un control de calidad de datos de temperatura superficial. Costa Rica. Tecnología en Marcha. 1(24):33-49.

Araya, J. L. 2014. Experiencias en la aplicación operativa de un método multivariado de imputación de datos meteorológicos. Costa Rica. Tecnología en Marcha. 3(27):70-79.

Biblioteca Scopus. 2019. Universidad Autónoma Chapingo. https://www.scopus.com.

Campos, A. D. F. 1998. Procesos del ciclo hidrológico. Editorial Universitaria Potosina. 3a (Ed.). Facultad de Ingeniería. Universidad Autónoma San Luis Potosí, México. 500 p.

Cervantes, R.; Arteaga, R.; Vázquez, M. A.; Ojeda, B. W. y Quevedo, N. A. 2013. Modelos Hargreaves Priestley-Taylor y redes neuronales artificiales en la estimación de la evapotranspiración de referencia. Ingeniería Investigación y Tecnología. 16(2):163-176.

Cervantes, R.; Arteaga, R.; Vázquez, M. A.; Ojeda, B. W. y Quevedo, N. A. 2019. Red neuronal artificial y series de Fourier para pronóstico de temperaturas en el Distrito de Riego 075. Sinaloa, México. Tecnología y Ciencias del Agua. 1(10):253-268.

Cortés, J. A.; Medina, A. F. A. y Chaves, O. J. A. 2007. Del análisis de Fourier a las wavelets análisis de Fourier. Colombia. Scientia Technica. 34(13):151-156.

Coutinho, E. R.; Silva, R. M.; Madeira, J. G. F.; Coutinho, P. R. Boloy, R. A. M. y Delgado, A. R. S. 2018. Aplicación de redes neuronales artificiales (ANN) en el relleno de brechas de series meteorológicas de tiempo. Rev. Bras. Meteorol. 33(2):317-328. Doi: https://doi.org/10.1590/0102-7786332013.

Degiovanni, B. V.; Martínez, R. C. P. y Motta, O. F. 2010. Producción ecoeficiente del arroz en América Latina, Tomo I. Conferencia internacional de arroz para América Latina y el Caribe. (Ed.). Cali-Colombia. 513 p.

De-Luis, M.; Longares, L. A.; Stepanek, P. y González, J.C. 2007. Tendencias estacionales de la precipitación en la cuenca del Ebro 1951-2000. España. Geographicalia. 52:53-78.

Eischeid, J. K.; Pasteris, P. A.; Díaz, H. F.; Plantico, M. S. and Lott, N. J. 2000. Creating a serially complete, national daily time series of temperature and precipitation for the western united states. United states. J. Appl. Meteorol. 39(9):1580-1591. Doi:10.1175/15200450(2000) 039<1580:CASCND>2.0.CO;2.

Gallegos, J.; Arteaga, R.; Vázquez, M. A. y Juárez, J. 2016. Estimación de la falta de precipitación diaria y registros de temperatura máxima y mínima en San Luis Potosí. México. Ingeniería Agrícola y Biosistemas. 8(1):3-16. Doi: 10.5154/r.inagbi.2015.11.008.

Grossi, G. H.; Raichijk, C. y Righini, R. 2010. Algunos aspectos de la climatología solar del Uruguay. Brasil. Rev. Bras. Meteorol. 4(25):479-486.

Guijarro, J. A. 2004. Climatol: software libre para la depuración y homogeneización de datos climatológicos. In: el clima, entre el mar y la montaña. (Ed.). A-4. Asociación Española de Climatología. España. 493-502 pp.

Kanda, N.; Negi, H. S.; Rishi, M. S. and Shekhar, M. S. 2017. Performance of various techniques in estimating missing climatological data over snowbound mountainous areas of Karakorum Himalaya. United States. Meteorological Applications. 25(3):337-349. Doi:10.1002/met.1699.

Medina, R. D., Montoya, E. C. y Jaramillo, R. A. 2008. Estimación de valores faltantes en series históricas de lluvia. Colombia. Cenicafé. 59(3):260-273.

Mintegui, J. y López, F. 1990. La ordenación agrohidrológica en la planificación. Servicio central de publicaciones del gobierno Vasco. 1 (Ed.). Vasco. 306 p.

Nauman, G. y Vargas, W. M. 2012. Estabilidad de la estimación de la onda anual en escala diaria de la temperatura. Brasil. Rev. Bras. Meteorol. 4(27):401-412.

Paulhus, J. L. H. and Kohler, M. A. 1952. Interpolation of missing precipitation records. United States. Monthly Weather Review. 80(8):129-133.

Pereira, A. R. 2004. The Priestley Taylor parameter and the decoupling factor for reference evapotranspiration. Canada. Agric. Forest Meteorol. 125(3):305-313.

Riehl, H. 1965. Tropical meteorology. McGraw Hill. New York, United states of America. 392 p.

Teegavarapu, R. S. and Chandramouli, V. 2005. Improved weighting methods, deterministic and stochastic data-driven models for estimation of missing precipitation records. Países Bajos. J. Hydrol. 312(1-4):191-20

Toro, A. M.; Arteaga, R.; Vázquez, M. A. e Ibáñez, L. A. 2015. Relleno de series diarias de precipitación, temperatura mínima, máxima de la región norte del urabá antioqueño. Rev. Mex. Cienc. Agríc. 6(3):577-588.

Xia, Y. Fabian, P.; Stohl, A. and Winterhalter, M. 1999. Forest climatology: estimation of missing values for Bavaria, Germany. Canada. Agric. Forest Meteorol. 96(1-3):131-14. Doi: 10.1016/S0168-1923(99)00056-8.