Revista Mexicana Ciencias Agrícolas   volume 12   number 4   May 16 - June 29, 2021

DOI: https://doi.org/10.29312/remexca.v12i4.2930

Investigation note

Meta-evaluation of support programs for the field in Mexico 2001-2017

Teodoro Domínguez Torres§

Jaime Arturo Matus Gardea

Miguel Ángel Martínez Damián

José Miguel Omaña Silvestre

Oliverio Hernández Romero

Laura Elena Garza Bueno

Postgraduate College-Campus Montecillo. Highway México-Texcoco km 36.5, Montecillo, Texcoco, State of Mexico, Mexico. CP. 56230. (matusgar@colpos.mx, angel01@colpos.mx; miguelom@colpos.mx; ohr@colpos.mex).

§Corresponding author: teodorodominguez2017@gmail.com.

Abstract

This work of a socioeconomic nature proposes to know the level of use and usefulness of the results of the evaluations of the field support programs in Mexico, in the period 2001-2017, it is a meta-evaluation or evaluation of the evaluation, which considers particularly those two key aspects of the evaluation process. To carry out the study, information was obtained from the possible users of the evaluations through a survey applied online or directly. The foregoing is relevant since in Mexico the formal evaluation of SADER’s field support programs before SAGARPA dates back to the end of 1997, with the Alliance for the Field, to the current programs and also in an important period of time. From 2001 to date, the federal government has maintained a technical cooperation agreement with FAO to conduct this process. Thus, this work is also relevant for decision-making since it corroborates, among other things, the limited use of evaluations carried out in decision-making for the improvement of programs. Finally, it is expected to contribute to motivate a technical-methodological and normative redefinition of the external evaluation process of the field support programs in Mexico and thus achieve the objective of improving the performance of said programs to which they are assigned year after year. important public resources to improve the standard of living of the population living in rural areas.

Keywords: evaluation, FAO-SADER, use, utility.

Reception date: April 2021

Acceptance date: May 2021

Contemporary public administration theory emphasizes the advantages of having a results-oriented public management, in contrast to a classical orientation that emphasizes management procedures and processes. Orienting management to results generates a dynamic within the organization that causes the improvement of organizational performance and the results to be achieved (Mejía, 2005). Precisely, the process that accounts for the extent to which the results are obtained is that of the evaluation, hence the relevance of the subject in question in the sense that the meta-evaluation has the final purpose of contributing to the improvement of the evaluation processes for the best decision making.

Likewise, the implicit theory is that this dynamic by itself generates feedback and learning processes on the part of the actors involved, which in turn generate corrective actions that produce management improvement. This reasoning explains the appearance of numerous management evaluation efforts in several Latin American countries, some creating a set of evaluation instruments, and others trying to create national management evaluation systems (Rosas and Sánchez, 2017).

However, little is known about how these evaluation systems and tools actually contribute to the improvement of public management in Latin America in general and in Mexico in particular. Nor is it known what are the factors that facilitate or limit the use of evaluation as a management tool and more specifically, how this instrument is used to promote results-oriented public management.

Generating knowledge on these issues can provide useful lessons, both for those who administer the existing systems, as well as for those who consider this tool as part of the modernization agenda of public administration and finally, for students of public management as a discipline (Mejía, 2005).

This work seeks to explore, in general terms, the levels of use and usefulness of the results of the evaluations of the field support programs in Mexico from the period 2001 to 2017, from the point of view of the main actors involved in the processes of evaluation and above all, of the probable users of the evaluation reports in making decisions for the improvement of the programs, that is: legislators, executive directors, operational directors, operations and those responsible for evaluation. It should be noted that there was no information from the legislative field, so in this analysis it will only be possible to refer to it through information from the existing literature.

The research topic treated here is relevant because it touches crucial aspects

i) the national need for public development policies based on a results-based management (RBM) approach in accordance with the provisions of the law incorporated in 2008, which implies a continuous process of evaluation and improvement with efficient, effective use, timely and economical of the resources used; likewise, the use of knowledge and experiences captured in the performance evaluation system (SED) by public policy actors and by society (Sánchez and Rosas, 2015); ii) rural problems associated with rising unemployment rates, low production and income levels, and poverty in broad segments of the Mexican population. That is, the proposals made to help solve the problems identified in the work have been raised with the intention that Mexico take advantage of the great opportunity that it currently has to improve the quality of evaluation work and integrate its results into management. public, as a way of influencing more effectively in alleviating the most urgent national problem: the poverty in which half of its citizens live (Cardozo, 2006a); and iii) experiences in similar works in Europe concluded that the results of the evaluations are very little used. In the same sense, Díaz and Rosales (2003) express themselves; Cardozo (2006b) based on the evaluations they review in the Costa Rican and Mexican context, respectively.

Next, the research questions that guide this research work are enunciated, which limit the level of analysis of the same and contribute to generate new research questions for the analysis of the evaluation in Mexico, of the evaluation processes of the programs of the current SADER: what is the level of use of the results of the evaluations of the field support programs, in the period 2001-2017 and their usefulness in making decisions for their improvement? What are the main factors that have limited the use of the evaluation results of field support programs in Mexico?

In this sense, the objectives of the research are: a) to know the level of use and usefulness of the results of the evaluations of the field support programs in Mexico, in the period 2001-2017, in making decisions for their improvement; and b) determine the main factors that have limited the use of the evaluation results of the field support programs in Mexico. Likewise, the hypotheses proposed are the following: 1) the results of the evaluations of the field support programs in Mexico are not used by the actors involved and are not useful in making decisions for their improvement; and 2) the main factors that have limited the use of the results of the evaluations of the field support programs in Mexico are essentially of a technical-methodological nature (relevance, credibility, quality and the findings themselves, as well as communication practices of the evaluator and the timeliness of the reports).

Due to the foregoing, the study is amply justified by virtue of the fact that over the last 22 years a significant amount of public resources has been allocated to carrying out external evaluations of support programs for the field in Mexico, such as the Alliance for the Countryside, Training and Extension, Mechanization, Agricultural Development, Livestock Development, Rural Development, etc., as well as the Support Program for the Acquisition of Productive Assets, the Support Program for Investment in Equipment and Infrastructure, the Development Program. of Capacities, Technological Innovation and Rural Extensionism, that of Health and Food Safety, that of Concurrence with the Federal Entities, etc.

In this sense, it is relevant to note that in Mexico, from 2001 to 2017, approximately 2 200 state evaluations have been carried out, which have been carried out by around 550 consultants. Likewise, the FAO has carried out nearly 100 works of national magnitude on the FAO evaluation website related to the evaluation of SAGARPA now SADER programs.

Notwithstanding the foregoing, it is relevant to know how much said information is used for decision-making in the improvement of SADER programs in the country, which would help to provide guidelines, to the corresponding authorities, on the possible pending issues to be considered. in the evaluation processes that are currently being carried out, since some scholars on the subject refer that the use of the results of evaluations of public programs in Mexico has been very limited, but that their magnitude is unknown.

In addition, the meta-assessment can analyze, among other things, the background and context of the assessment; the definition of the terms used; the value criteria, the determination of costs and resources, the evaluation of the research process and its results and the existence of recommendations on the use or destination of the evaluation (Schwandt and Halpern, 1988; Scriven, 1991 and 2007).

It should be noted that the experience of external evaluation of the field support programs in Mexico, methodologically guided by FAO, in accordance with the participation agreement signed by this international organization and SAGARPA since 2001 (SAGARPA, 20015; SAGARPA, 2016), was marked from the beginning by processes of qualification of the evaluation reports by FAO itself that, although they have not been considered meta-evaluations at any time, an analysis of compliance with the methodology proposed in the TOR has been required by said international body, in terms of the areas of evaluation-processes, results and impacts-indicators to be considered, times required in the evaluation stages, definition of samples, questionnaires applied to the different actors, obtaining, capturing and debugging databases , presentation of results and edition, forms and times of disclosure, etc.

Finally, it is worth mentioning that the attempt to qualify the evaluation results reports, during the first years of the process being conducted by FAO, had implications on the operating rules of the secretariat’s programs to the extent that, depending on the quality of the evaluation reports delivered by the states to SAGARPA-FAO was the total amount assigned to the programs to operate in the states.

That is, the qualification of state evaluation reports was included as one more criterion in the formula for calculating the contributions of resources to the states. This situation, apparently, did not last long, probably due to the claim of the states, as it is considered a factor with a strong load of subjectivity motivated by technical-administrative processes.

The present work of a socio-economic nature has as a frame of reference the theory of the evaluation of public policies, framed in the general context of the virtuous circle of public policies, where the evaluation process plays an important role in decision-making for the improvement of these (Figure 1).

Figure 1. Virtuous circle of public policies in Mexico.

Likewise, the analysis units of this research are made up of the different instances that have to do with the evaluation process; in general, and with decision making regarding the use of the evaluation results, in particular. In this sense, these instances are: 1) the legislative level -deputies and senators-; 2) the national executive level (secretary, undersecretaries, general directors, general coordinators, delegates); 3) the state leadership level (state deputy delegates, directors and deputy directors); 4) the state operational level (DDR heads, program heads, CADER heads and department heads); and 5) the level of those responsible for evaluation -state evaluators-.

To obtain the information that served as the basis for this research, a survey was applied online or directly to the personnel of the levels mentioned above, in order to obtain the information necessary to support the objectives and research hypotheses. It should be noted here that the deputies and senators of the Republic to whom the survey was applied, via email, none of them answered it, so it was not possible to have the assessments of the legislative field (level 1). Thus, the distribution of the interviewees is as shown in Table 1.

Table 1. Number and type of interviewee.

As indicated at the beginning, this meta-evaluation study emphasizes only the level of use and the usefulness of the results of evaluations of field support programs in Mexico in making decisions for their improvement, considering that these simple factors of analysis could have normative and methodological implications that have probably been neglected by the respective official instances throughout the time in which this type of evaluations have been carried out in Mexico.

Thus, in relation to the use and usefulness of the evaluation results, it was hypothesized that the actors who direct or operate the field support programs in Mexico do not use the results of the evaluations and they are not useful in making decision-making for their improvement. In this sense and also taking into account the perception of the evaluation managers themselves, according to the information obtained, it was found that, in general, there is a very limited use of these results due to the different levels of action, since they consider in their Most of them say that the results of the evaluations are little or not used according to their appreciation and experience (Figure 2), which coincides with what was found by other authors who have studied the subject.

Figure 2. Level of use of the evaluation of support programs for the field of in Mexico in decision-making by level of action and those responsible for evaluation.

On the other hand, in relation to the usefulness of the results of the evaluations, it was found that, although the various actors consider that the use of said results is little or nothing, they perceive, for the most part, that the information is useful and very useful which is derived from the evaluations of the programs for decision-making in their improvement (Figure 3).

Figure 3. Level of utility of the results of the evaluations according to the global appreciation of the various participation actors.

In this sense, in relation to the usefulness of the evaluation results of the field support programs in Mexico, according to the managerial and operational managers at the national and state level, as well as the evaluation managers themselves, it could be taking into account the items shown in Table 2.

Table 2. Areas of interest in which the information derived from the evaluation of the field support programs in Mexico has been used according to the various actors.

Due to the afore mentioned and in accordance with the hypothesis that expresses that the results of the evaluations of the field support programs in Mexico are not used by the actors and are not useful in making decisions for their improvement, it is partially accepted in the sense that the information obtained shows that although, in general, the results of the evaluations are not used, they are considered useful at all.

On the other hand, regarding the main factors that have limited the use of the results of the evaluations of the field support programs in Mexico, according to the different actors, the following are found in order of importance: i) the results evaluation are presented in an inopportune manner for decision making; ii) the absence of legal mechanisms that motivate or regulate its use; iii) the information presented in the evaluations is neither objective nor reliable; iv) the methodology used to evaluate is not relevant; v) the results do not come from methodologies and topics of interest in the state; and vi) they are not recognized as users of the evaluation results.

Finally, in accordance with the afore mentioned and with the hypothesis raised in the sense that the main factors that have limited the use of the results of the evaluations of the field support programs in Mexico are essentially of a technical-methodological nature (relevance, credibility, quality and the findings themselves, as well as the evaluator’s communication practices and the timeliness of the reports) this hypothesis is corroborated.

Conclusions

This research was based on the identification of the levels of use and usefulness of the results of the evaluations of the field support programs in Mexico, in making decisions for the improvement of the field. In general, it was found that, although the results of the evaluations can be considered useful, there is a very limited use of them by the different levels of action, which calls for a reconsideration by all those involved in the evaluation process, especially those involved in the evaluation process. the corresponding official bodies (SADER, SFP, SHCP, CONEVAL, etc.), on the organizational, methodological and regulatory aspects considered over time, which possibly have not clearly and explicitly addressed the aspects of the use and utility of the Evaluation results according to the type of audience or possible user of the same.

In relation to the main factors that have limited the use of the results of the evaluations of the field support programs in Mexico, it was found, according to the evaluations of the directive and operative managers (national executive, state directive and state operative) As well as the evaluation officials in the country themselves, which in some cases include state evaluation coordinators, that the factors that have most limited the use of evaluation results are essentially of a technical-methodological nature, probably motivated by the lack of clarity regarding the determination of the usefulness of the information generated in the evaluation processes and the specific forms of use possible by type of audience or user of the evaluation.

Cited literature

Cardozo, M. 2006a. La evaluación de políticas y programas públicos. El caso de los programas de desarrollo social en México. H. Cámara de Diputados, LIX Legislatura-Miguel Ángel Porrúa, librero (Ed.). México, DF. 374 p.

Cardozo, M. 2006b. Políticas de lucha contra la pobreza en México. Principales resultados y limitaciones. Fermentum. Revista Venezolana de Sociología y Antropología. 16(45):15-56.

Díaz, L. F. y Rosales, R. 2003. Metaevaluación: Evaluación de políticas, programas y proyectos sociales, San José de Costa Rica. (Ed.). Universidad Estatal a Distancia. 73 p.

Mejía, J. 2005. La evaluación como herramienta para una gestión pública orientada a resultados. La práctica de la evaluación en el ámbito público mexicano. CLAD. 56 p. http://www.clad.org.ve.

Rosas, A. J. y Sánchez, B. A. 2017. La evaluación de las políticas públicas en Jalisco: una aproximación desde la metaevaluación. Política y Cultura. 47:201-223.

SAGARPA. 2015. Secretaría de Agricultura, Ganadería, Desarrollo Rural, Pesca y Alimentación. Lineamientos para las actividades de planeación, monitoreo y evaluación en los estados 2015. Subsecretaría de alimentación y competitividad. Dirección General De Planeación y Evaluación. Dirección General Adjunta de Planeación y Evaluación. México, DF.

SAGARPA. 2016. Secretaría de Agricultura, Ganadería, Desarrollo Rural, Pesca y Alimentación. Lineamientos para las actividades de planeación, monitoreo y evaluación en los estados 2016-2017. Dirección General De Planeación y Evaluación. Ciudad de México.

Sánchez, A. y Rosas, J. 2015. La evaluación de la política social en los gobiernos municipales de México. En Pablo Pineda (Coord.). Cuestiones sociales y políticas, Instituto de Investigación en Gobierno y Políticas Públicas-Universidad de Guadalajara/Prometeo Editores. 71-83 pp.

Schwandt, T. and Edward, H. 1988. Linking auditing and metaevaluation, Newbury Park: Sage Publications.188 p.

Scriven, M. 1991. Evaluation Thesaurus. 4th edition. Newbury Park: Sage Publications. 391 p.

Scriven, M. 2007. Key evaluation Checklist, Kalamazoo, Western Michigan University. The Evaluation Center. http://www.wmich.edu/evalctr/checklists/kec-feb07.pdf.