Any person who has made a significant . The introduction of impact assessments with the requirement to collate evidence retrospectively poses difficulties because evidence, measurements, and baselines have, in many cases, not been collected and may no longer be available. (2007) adapted the terminology of the Payback Framework, developed for the health and biomedical sciences from benefit to impact when modifying the framework for the social sciences, arguing that the positive or negative nature of a change was subjective and can also change with time, as has commonly been highlighted with the drug thalidomide, which was introduced in the 1950s to help with, among other things, morning sickness but due to teratogenic effects, which resulted in birth defects, was withdrawn in the early 1960s. For more extensive reviews of the Payback Framework, see Davies et al. 0000334683 00000 n What emerged on testing the MICE taxonomy (Cooke and Nadim 2011), by mapping impacts from case studies, was that detailed categorization of impact was found to be too prescriptive. A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). By allowing impact to be placed in context, we answer the so what? question that can result from quantitative data analyses, but is there a risk that the full picture may not be presented to demonstrate impact in a positive light? This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. As Donovan (2011) comments, Impact is a strong weapon for making an evidence based case to governments for enhanced research support. Frameworks for assessing impact have been designed and are employed at an organizational level addressing the specific requirements of the organization and stakeholders. 0000346296 00000 n Researchers were asked to evidence the economic, societal, environmental, and cultural impact of their research within broad categories, which were then verified by an expert panel (Duryea et al. The justification for a university is that it preserves the connection between knowledge and the zest of life, by uniting the young and the old in the imaginative consideration of learning. working paper). A university which fails in this respect has no reason for existence. This is a metric that has been used within the charitable sector (Berg and Mnsson 2011) and also features as evidence in the REF guidance for panel D (REF2014 2012). % 0000012122 00000 n The main risks associated with the use of standardized metrics are that, The full impact will not be realized, as we focus on easily quantifiable indicators. It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, Downloadable! 0000006922 00000 n HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). The process of evaluation is dynamic and ongoing. A taxonomy of impact categories was then produced onto which impact could be mapped. This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. The aim of this study was to assess the accuracy of 3D rendering of the mandibular condylar region obtained from different semi-automatic segmentation methodology. 0000001087 00000 n These case studies were reviewed by expert panels and, as with the RQF, they found that it was possible to assess impact and develop impact profiles using the case study approach (REF2014 2010). It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary (Davies et al. Measurement assessment and evaluation helps the teachers to determine the learning progress of the students. 0000348082 00000 n 0000003495 00000 n An evaluation essay or report is a type of argument that provides evidence to justify a writer's opinions about a subject. Author: HPER Created Date: 3/2/2007 10:12:16 AM . It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. Impact has become the term of choice in the UK for research influence beyond academia. 0000328114 00000 n Given that the type of impact we might expect varies according to research discipline, impact-specific challenges present us with the problem that an evaluation mechanism may not fairly compare impact between research disciplines. Reviewing the research literature means finding, reading, and summarizing the published research relevant to your question. We take a more focused look at the impact component of the UK Research Excellence Framework taking place in 2014 and some of the challenges to evaluating impact and the role that systems might play in the future for capturing the links between research and impact and the requirements we have for these systems. 2007). It is important to emphasize that Not everyone within the higher education sector itself is convinced that evaluation of higher education activity is a worthwhile task (Kelly and McNicoll 2011). Understand. The case study of the Research Information System of the European Research Council, E-Infrastructures for Research and Innovation: Linking Information Systems to Improve Scientific Knowledge, Proceedings of the 11th International Conference on Current Research Information Systems, (June 69, 2012), pp. The ability to write a persuasive well-evidenced case study may influence the assessment of impact. The first category includes approaches that promote invalid or incomplete findings (referred to as pseudoevaluations), while the other three include approaches that agree, more or less, with the definition (i.e., Questions and/or Methods- The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. 2009), and differentiating between the various major and minor contributions that lead to impact is a significant challenge. Although some might find the distinction somewhat marginal or even confusing, this differentiation between outputs, outcomes, and impacts is important, and has been highlighted, not only for the impacts derived from university research (Kelly and McNicol 2011) but also for work done in the charitable sector (Ebrahim and Rangan, 2010; Berg and Mnsson 2011; Kelly and McNicoll 2011). What are the challenges associated with understanding and evaluating research impact? This might describe support for and development of research with end users, public engagement and evidence of knowledge exchange, or a demonstration of change in public opinion as a result of research. To be considered for inclusion within the REF, impact must be underpinned by research that took place between 1 January 1993 and 31 December 2013, with impact occurring during an assessment window from 1 January 2008 to 31 July 2013. What is The Concept of Evaluation With its Importance? 2. Many times . Baselines and controls need to be captured alongside change to demonstrate the degree of impact. A variety of types of indicators can be captured within systems; however, it is important that these are universally understood. 0000007559 00000 n Inform funding. There are standardized tests involved in the process of measurement assessment and evaluation to enable the students to make better use of the data available in the daily classroom. Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, Volume 23, Issue 1, January 2014, Pages 2132, https://doi.org/10.1093/reseval/rvt021. Scriven (2007:2) synthesised the definition of evaluation which appears in most dictionaries and the professional literature, and defined evaluation as "the process of determining merit, worth, or significance; an evaluation is a product of that process." . However, the Achilles heel of any such attempt, as critics suggest, is the creation of a system that rewards what it can measure and codify, with the knock-on effect of directing research projects to deliver within the measures and categories that reward. Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. To achieve compatible systems, a shared language is required. One of these, the RQF, they identified as providing a promising basis for developing an impact approach for the REF using the case study approach. And also that people who are recognized as authors, understand their responsibility and accountability for what is being published. Clearly the impact of thalidomide would have been viewed very differently in the 1950s compared with the 1960s or today. If this research is to be assessed alongside more applied research, it is important that we are able to at least determine the contribution of basic research. This raises the questions of whether UK business and industry should not invest in the research that will deliver them impacts and who will fund basic research if not the government? Other approaches to impact evaluation such as contribution analysis, process tracing, qualitative comparative analysis, and theory-based evaluation designs (e.g., Stern, Stame, Mayne, Forss, & Befani, 2012) do not necessarily employ explicit counterfactual logic for causal inference and do not introduce observation-based definitions. << /Length 5 0 R /Filter /FlateDecode >> The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. 1.3. In development of the RQF, The Allen Consulting Group (2005) highlighted that defining a time lag between research and impact was difficult. It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that one is able to see further by standing on the shoulders of giants. Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. Ideally, systems within universities internationally would be able to share data allowing direct comparisons, accurate storage of information developed in collaborations, and transfer of comparable data as researchers move between institutions. different meanings for different people in many different contexts. The definition of health is not just a theoretical issue, because it has many implications for practice, policy, and health services. The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. In designing systems and tools for collating data related to impact, it is important to consider who will populate the database and ensure that the time and capability required for capture of information is considered. 8. The inherent technical disparities between the two different software packages and the adjustment . This report, prepared by one of the evaluation team members (Richard Flaman), presents a non-exhaustive review definitions of primarily decentralization, and to a lesser extent decentralization as linked to local governance. These sometimes dissim- ilar views are due to the varied training and background of the writers in terms of their profession, concerned with different aspects of the education process. Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at Kings College London or outlined in relevant publications (MICE Project n.d.). "Evaluation is a process of judging the value of something by certain appraisal." Characteristics of evaluation in Education Below are some of the characteristics of evaluation in education, Continuous Process Comprehensive Child-Centered Cooperative Process Common Practice Teaching Methods Multiple Aspects Continuous Process A key concern here is that we could find that universities which can afford to employ either consultants or impact administrators will generate the best case studies. Assessment is the collection of relevant information that may be relied on for making decisions., 3. Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. The understanding of the term impact varies considerably and as such the objectives of an impact assessment need to be thoroughly understood before evidence is collated. New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. Throughout history, the activities of a university have been to provide both education and research, but the fundamental purpose of a university was perhaps described in the writings of mathematician and philosopher Alfred North Whitehead (1929). A very different approach known as Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions (SIAMPI) was developed from the Dutch project Evaluating Research in Context and has a central theme of capturing productive interactions between researchers and stakeholders by analysing the networks that evolve during research programmes (Spaapen and Drooge, 2011; Spaapen et al. Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support (Kelly and McNicoll 2011). 6. 2007) who concluded that the researchers and case studies could provide enough qualitative and quantitative evidence for reviewers to assess the impact arising from their research (Duryea et al. In this article, we draw on a broad range of examples with a focus on methods of evaluation for research impact within Higher Education Institutions (HEIs). Using the above definition of evaluation, program evaluation approaches were classified into four categories. Figure 1, replicated from Hughes and Martin (2012), illustrates how the ease with which impact can be attributed decreases with time, whereas the impact, or effect of complementary assets, increases, highlighting the problem that it may take a considerable amount of time for the full impact of a piece of research to develop but because of this time and the increase in complexity of the networks involved in translating the research and interim impacts, it is more difficult to attribute and link back to a contributing piece of research. The Economic and Social Benefits of HRB-funded Research, Measuring the Economic and Social Impact of the Arts: A Review, Research Excellence Framework Impact Pilot Exercise: Findings of the Expert Panels, Assessment Framework and Guidance on Submissions, Research Impact Evaluation, a Wider Context. On the societal impact of publicly funded Circular Bioeconomy research in Europe, Devices of evaluation: Institutionalization and impactIntroduction to the special issue, The rocky road to translational science: An analysis of Clinical and Translational Science Awards, The nexus between research impact and sustainability assessment: From stakeholders perspective. Table 1 summarizes some of the advantages and disadvantages of the case study approach. In the UK, evaluation of academic and broader socio-economic impact takes place separately. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. These traditional bibliometric techniques can be regarded as giving only a partial picture of full impact (Bornmann and Marx 2013) with no link to causality. This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. Search for other works by this author on: A White Paper on Charity Impact Measurement, A Framework to Measure the Impact of Investments in Health Research, European Molecular Biology Organization (EMBO) Reports, Estimating the Economic Value to Societies of the Impact of Health Research: A Critical Review, Bulletin of the World Health Organization, Canadian Academy of Health Sciences Panel on Return on Investment in Health Research, Making an Impact. 0000342980 00000 n Impact is not static, it will develop and change over time, and this development may be an increase or decrease in the current degree of impact. Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? This article aims to explore what is understood by the term research impact and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. Dennis Atsu Dake. The time lag between research and impact varies enormously. Merit refers to the intrinsic value of a program, for example, how effective it is in meeting the needs those it is intended help. 0000342958 00000 n Here we outline a few of the most notable models that demonstrate the contrast in approaches available. Key features of the adapted criteria . Impact is derived not only from targeted research but from serendipitous findings, good fortune, and complex networks interacting and translating knowledge and research. What are the reasons behind trying to understand and evaluate research impact? In the UK, evidence and research impacts will be assessed for the REF within research disciplines. Evaluation is the systematic collection and inter- pretation of evidence leading as a part of process to a judgement of value with a view to action., Evaluation is the application of a standard and a decision-making system to assessment data to produce judgments about the amount and adequacy of the learning that has taken place., 1.
March 14, 2023