Method and Framework2 – The Climate Futures Project

The ‘HOW’ of assessing, comparing,
and interpreting models

In the Unpacking Models Initiative (TUMI), we apply a common framework to assess, compare, and interpret the assumptions and results of various modelling studies. However, we believe that insights are only as good as the methods used to arrive at them. Therefore, in this page, we explain the method, the processes, and rationales employed to arrive at insights (assessments, comparisons, interpretations) based on disparate modelling studies.

Model Factsheets

Cross-Model Analysis

Framework Documentation

ASSESSING CLIMATE POLICY MODELS

Are model scenarios transparently described and justified for their stated purpose? How do results from one model compare with those of other, similar studies?

Assessments of the modelling studies are aimed at evaluating the transparency and credibility of the studies with respect to addressing their stated objectives. The framework for assessment employed in this project was developed based on studies of good practices for computational models which inform decision-making.

A review of these papers indicated a few common themes:

Clarity of Purpose, Importance of Model Specification and The Process Involved, Assessing Data Quality, Dealing with Uncertainty, and Validation of The Model and its Results.

We arrived at the final indicators through an iterative process of applying the indicators to contemporary modelling studies, gauging their applicability and usefulness, and engaging in a peer review process for the framework. More details on the criteria are presented below.

Each model is assessed based on six main criteria:

Transparency and Credibility of Inputs to the Model

Appropriateness of Model Choice to Research Objective

Assessment of Scenario Construction Process

Approach to Uncertainty

Transparency

Validation of Outputs

These are further broken into sub-criteria. Each of them is given a score (adequate, partially adequate, and inadequate) based on the extent to which the criteria have been met. The framework of the assessment criteria and decision rules to determine final assessment score are summarised in the table below.

scroll

Assessment criteria	Assessment sub-criteria	Sub-criteria scoring	Decision rule to determine final assessment score (adequate/ partially adequate/ inadequate)

Transparency and credibility of inputs to the model	Data source transparent and credible? Projections justified? Data up-to-date?	• Yes • Yes	Adequate If all three parameters are met for both techno economic data and socio economic drivers
		• Yes • No	Partially Adequate If parameter 1 and 2 are met, but 3 is unmet or poorly met for both techno-economic data and socio-economic drivers
		• No • No	Inadequate

Appropriateness of model choice to research objective	Model structure transparent? Fit for purpose justified? Conclusions drawn based on model strengths and qualified for limitations?	• Yes • Yes	Adequate If parameters 1, 2 and 3 are marked ‘adequate’,
		• Yes • No	Partially Adequate Any two parameters 1 and 2 are marked yes.
		• No • No	Inadequate

Assessment of scenario construction process	Rationale for scenarios explained? Process of developing scenarios explained? Alternative socio-economic pathways explained?	• Yes • Yes	Adequate If parameters 1, 2, and 3 are marked yes
		• Yes • No	Partially Adequate If any two parameters are marked yes
		• No • No	Inadequate

Approach to uncertainty	Uncertainties in inputs transparent? Uncertainties in a model's causal mechanisms transparent? Change in uncertainty with time?	• Yes • Yes	Adequate If parameters 1, 2, and 3 are marked adequate
		• Yes • No	Partially Adequate If any two of three parameters is adequate
		• No • No	Inadequate

Transparency and validation of outputs	Explanation of how outputs are shaped by inputs, model mechanisms Implications of uncertainties considered in results and policy implications Validation efforts presented clearly?	• Yes • Yes	Adequate If parameters 1, 2, 3 are marked adequate
		• Yes • No	Partially Adequate If any two parameters are adequate
		• No • No	Inadequate

Si. No.	Assessment Criteria	Assessment Sub-criteria	Sub-criteria Scoringe	Decision rule to determine final assessment score (Adequate/Partially adequate/ Inadequate)

1.	Transparency and credibility of inputs to the model	Data source transparent and credible? Projections justified?	• Yes • Yes	Adequate If all three parameters are met for both techno economic data and socio economic drivers
			• Yes • No	Partially Adequate If parameter 1 and 2 are met, but 3 is unmet or poorly met for both techno-economic data and socio-economic drivers
			• No • No	Inadequate

2.	Appropriateness of model choice to research objective	Model structure transparent? Fit for purpose justified?	• Yes • Yes	Adequate If parameters 1, 2 and 3 are marked ‘adequate’,
			• Yes • No	Partially Adequate Any two parameters 1 and 2 are marked yes.
			• No • No	Inadequate

3.	Assessment of scenario construction process	Rationale for scenarios explained? Process of developing scenarios explained? Alternative socio-economic pathways explained?	• Yes • Yes	Adequate If parameters 1, 2, and 3 are marked yes
			• Yes • No	Partially Adequate If any two parameters are marked yes
			• No • No	Inadequate

4.	Approach to uncertainty	Uncertainties in inputs transparent? Uncertainties in a model's causal mechanisms transparent? Change in uncertainty with time?	• Yes • Yes	Adequate If parameters 1, 2, and 3 are marked adequate
			• Yes • No	Partially Adequate If any two of three parameters is adequate
			• No • No	Inadequate

5.	Transparenc and validation of outputs	Explanation of how outputs are shaped by inputs, model mechanisms Implications of uncertainties considered in results and policy implications Validation efforts presented clearly?	• Yes • Yes	Adequate If parameters 1, 2, 3 are marked adequate
			• Yes • No	Partially Adequate If any two parameters are adequate
			• No • No	Inadequate

Si. No.	Assessment Criteria	Assessment Sub-criteria	Sub-criteria Scoringe	Decision rule to determine final assessment score (Adequate/Partially adequate/ Inadequate)

1.	Transparency and credibility of inputs to the model	Data source transparent and credible? Projections justified?	• Yes • Yes	Adequate If all three parameters are met for both techno economic data and socio economic drivers
			• Yes • No	Partially Adequate If parameter 1 and 2 are met, but 3 is unmet or poorly met for both techno-economic data and socio-economic drivers
			• No • No	Inadequate

2.	Appropriateness of model choice to research objective	Model structure transparent? Fit for purpose justified?	• Yes • Yes	Adequate If parameters 1, 2 and 3 are marked ‘adequate’,
			• Yes • No	Partially Adequate Any two parameters 1 and 2 are marked yes.
			• No • No	Inadequate

3.	Assessment of scenario construction process	Rationale for scenarios explained? Process of developing scenarios explained? Alternative socio-economic pathways explained?	• Yes • Yes	Adequate If parameters 1, 2, and 3 are marked yes
			• Yes • No	Partially Adequate If any two parameters are marked yes
			• No • No	Inadequate

4.	Approach to uncertainty	Uncertainties in inputs transparent? Uncertainties in a model's causal mechanisms transparent? Change in uncertainty with time?	• Yes • Yes	Adequate If parameters 1, 2, and 3 are marked adequate
			• Yes • No	Partially Adequate If any two of three parameters is adequate
			• No • No	Inadequate

5.	Transparenc and validation of outputs	Explanation of how outputs are shaped by inputs, model mechanisms Implications of uncertainties considered in results and policy implications Validation efforts presented clearly?	• Yes • Yes	Adequate If parameters 1, 2, 3 are marked adequate
			• Yes • No	Partially Adequate If any two parameters are adequate
			• No • No	Inadequate

Implications of model constructs for six key policy priorities

Projections coming from climate policy models provide valuable inputs for policy decision-making. Climate policy models typically project estimates for emissions and energy across different scenarios, which help to indicate the possible pathways India can follow to achieve its climate goals. However, the outputs of these exercises should be interpreted carefully, with a view to assessing implications for India’s policy priorities in a balanced manner. This requires a deeper look into the studies’ underlying assumptions and outputs, particularly in relation to current trends. In this context, we consider the modelling setups and outputs in relation to six policy priorities:

Development Pathway

Emissions

Energy Transition Pathways

Investment

Equity and Resources Impact

Energy Security

The rationale for choosing these parameters and how we consider them are noted below.

Policy Parameter	Rationale	Focus
Development Pathway	As a lower middle-income country, India aspires to grow strongly, develop, and eradicate poverty. This section asks how a model envisions the pathway that India follows to achieve such national objectives, and whether this diverges from current trends. The implicit pathway has important implications for level of emissions, macro variables (e.g., GDP, job), technology choice, and behavioural changes etc. for low-carbon transition from development perspectives.	Macro-structural assumptions in determining a pathway (e.g., urbanization, growth, jobs, total and sectoral energy demand, and electrification) Patterns of the development and its alignment with the current socio-economic structure
Energy Transition Pathway	Transforming the energy system is fundamental to mitigate emissions, and also presents an opportunity for an alternative development pathway. It typically requires shifting away from fossil fuel-based energy sources and improving energy efficiency at national and sectoral levels. This section addresses how comprehensively a modelling study addresses the energy transition pathway.	Sectoral energy needs Expected energy mix Enabling conditions Technological implications
Emissions	The level of projected emissions is critical to understand the feasibility of decarbonisation efforts and meeting India’s climate goals. Comprehensive (economy-wide) and robust estimates of emissions are therefore critical also to develop a low-carbon development strategy. This section covers how a model estimates emissions, in terms of the coverage, methodology, consideration of uncertainty, and robustnes in emission estimates.	Emission projection vs. backcasting Scope of coverage in emissions estimates Implication of technological and demand trends on robustness of estimates
Investments	Investment in infrastructure, disruptive low carbon echnologies, institutional development, and capacity building drive economic growth and climate actions. This section assesses how modelling studies approach investment needs estimates, what is included or excluded, and how these are estimated.	Suitability of modelling framework to estimate investments Investment estimates based onassumptions of technological choices, lock-in, cost assumptions, sectoral choice
Equity and Resource Impacts	With low incomes and low levels of access to resources and services, it is important that models consider the distributional and resource implications of climate policies. This section discusses whether and how a modelling study explores such impacts.	Variation in economic outcomes across socioeconomic classes, sectors and regions Roles of informal economy and employment Implications for natural resource needs
Energy Security	Considering future increases in domestic energy demand, and increasing levels of geo-political uncertainties, energy independence is an important consideration for energy security and macroeconomic stability. This section looks into whether and how a modelling study consider India’s future reliance on fuel and material imports, as well as their implications.	Consideration of fuel and material import dependence into energy capacity and investment estimates

The ‘HOW’ of assessing, comparing, and interpreting models