Dynamic Personality

Last month, the University of Bath published a research paper that has been controversial amongst the building energy modelling community.

The research project took a typical real house, modelled it very carefully, using actual measured energy and other parameters to refine the parameters of the model to get it to match, as closely as possible. The model was then ‘retrofitted’ with various energy saving measures, to establish the impact of 21 different measures such as adding insulation. The modelling tool focussed on in the paper was a Dynamic Simulation tool, called IES – other software is available. It is however likely the team created a variety of models with different tools which we might hear about in future papers.

This in itself is a very interesting exercise, and it would be very useful to learn about what steps the team took to refine the model to get it to align with reality. People that aren’t involved in modelling will probably be thinking to the contrary, this is bonkers and placing the cart firmly before the horse, but for those of us who’s work it is to make energy consumption predictions, it’s very useful to look backwards and see how well we did.

Perhaps surprisingly, this paper was picked up by the Telegraph, who seized upon it as proof that energy modellers don’t know what they are doing. Here’s the article:

The Telegraph is generally thought to be a relatively sober and grown up, if decidedly right-wing publication. Unfortunately, I must say this article tends towards outrage and sensationalism. The first paragraph of the piece reads

“Homeowners and companies are being hit with unexpectedly high energy bills because planners continually make false promises about the ‘green’ credentials of new buildings, a major study has found.”

Firstly, the research looks at a house, using what is generally a commercial simulation tool. Maybe we can forgive the Telegraph for being a little confused about who is being victimised here. The Energy Performance Certificates that such calculations underpin are not intended to predict energy bills (although they perhaps unwisely do include an indication of cost). There are no promises here, false or otherwise, but perhaps we energy modelers have failed to communicate this properly. The sneering use of the word green, complete with inverted commas is a little bizarre alongside the outrage about energy bills, trying to both have and eat the cake.

I could go on dismantling the piece paragraph by paragraph and as fun as that might be, it would make for an exceedingly dull blog.

The article goes on to compare the issue to the VW emissions scandal, a point Mark Siddall has made previously. It’s an interesting point that effectively highlights the problem. However, VW actively undermined the testing procedure; I’d argue that in construction, our testing procedures themselves are at fault – no-one would bother fitting a defeat device to a gas meter just to get a better EPC. Even the new MEES is based on modelled rather than measured data.

A technical but important point that the Telegraph conveniently overlook is the role of standardisation in the compliance testing. We are required to adopt the National Calculation Methodology, which specifies things like how many people there are per square meter of office (or classroom, or storage cupboard). I’m slightly ashamed to admit in our office we’re sometimes crammed in at a rate of about 6m² per person, much more dense than the NCM would assume.

Similarly, many tools especially DSM do not adequately account for thermal bridging, a particular interest of mine.

One key point revealed by the research was that, when 108 professionals were asked to estimate the impact of 21 common energy saving measures, “A quarter of those interviewed were judged to be no better than if a member of the public had responded at random.”

This is a serious indictment. But is it really fair? Our friends Susie and Claire at Inkling, certainly think not. On the whole I tend to agree, although I think several of the points they make are debateable.

They say “It is unusual for modellers to be asked questions in this way.” In my experience, it’s not unusual – if we recommend say External Wall Insulation, the building owners want to know how much that will save them. In my opinion, it’s easy to overstate “how multiple building variables interact dynamically”. It’s true to say it happens, but in my work I’ve observed such effects are often quite small.

It’s quite true to say DSM “…is rarely performed … on new homes”, and I think this is a significant factor in the apparently inaccurate answers. Schools, offices and most commercial buildings behave somewhat differently to homes due to the usage patterns and typical design features. Many professionals using DSM energy modellers simply won’t have much experience of modelling homes – the industry is structured such that it’s unusual to be skilled in both as we are at Greengauge. If the researchers had performed the same exercise on a group of SAP assessors, I would put good money on the SAP assessors doing better, simply because it’s what they are used to thinking about. Similarly, if the research had taken an office example, I bet the DSM users would have done better. I’d go a step further and say that experienced Passivhaus designers would do even better than the SAP assessors in the domestic exercise because the nature of Passivhaus design forces us to become intimately familiar with exactly these questions.

Susie and Claire briefly touch on the complexities of buildings that must be inputted to a model, which they are right to identify as an issue.

I, like Susie and Claire “take issue with the blanket criticism meted out to the modelling community”, although I may not have used such strong terms. It seems entirely reasonable to expect someone who professionally predicts the energy consumption of buildings to have a good level of understanding about what factors have the strongest bearing on energy use. The question of whether we should work in siloes of domestic and non-domestic is another matter.

I consider myself fortunate, and somewhat unusual, to be familiar with a variety of modelling methods – I’m comfortable with SAP (the domestic Part L assessment tool), TAS (a DSM tool) and the Passivhaus Planning Package (PHPP). I have presented a paper to the UK Passivhaus conference on the Energy prediction gap between TAS and PHPP, and used all three tools on several projects with interesting, but often confusing and contradictory results.

It seems to me that there are several shortcomings throughout the industry that contribute to this performance gap.

Poorly designed policy, legislation and technical guidance
Badly designed models that respond to the regs/guidance
Inadequate and/or inappropriate training and experience requirements for energy modellers
Inadequate QA processes

Firstly the policy, legislation and guidance design. There are several interlinked issues that Part L and related policies must address. CO₂ emissions, energy bills (protecting consumers), energy poverty, and thermal comfort. The models that are used to demonstrate compliance with Part L express results in terms of CO₂. This is fine in principle, but can set up inappropriate incentives. Under pressure from clients, building energy modellers become skilled at gaming the model for the cheapest route to compliance, rather than the most accurate prediction of energy use.

On twitter, I suggested that DSM tools can be quite ‘black box’. What I meant by that, was that as modellers we put in lots of inputs, and it’s hard to know exactly what’s going on ‘under the bonnet’ to generate the outputs we receive and analyse. There’s a lot of maths going on that is fairly difficult to understand. I have a degree in engineering and while I think I get most of the basic principles, to be honest if asked to write it out I’d struggle. Instead we build up mental models of what’s going on inside the computer model, which itself is of course a representation of reality, so we get a Chinese whisper of understanding.

In contrast, simpler methods such as PHPP are much more transparent. Given some time, I feel like I could have a good stab at re-writing PHPP from scratch, or at least most parts of it. It is what I’d call a much more human scale model, in the sense that the maths fits in the brain of one person, along with the email I need to write to an angry contractor, the thermal conductivity and vapour permeability of dozens of types of insulation, and what I might have for tea. Most importantly, I know and understand its short comings. A static U-value is a representation of the heat flowing through a real wall – it’s not perfect but I understand its imperfections and can think about them as I interrogate my results.

Yes, it’s not as sophisticated – but my question is “does that matter?” Sometimes it does, yes, which is why we maintain our range of tools at Greengauge. But I would argue that most of the time, a simple model is more useful, often quicker to get results from, (and adjust and re-model), and easier to interpret. This Is particularly useful at the early stages of a project where cost and speed of work and feedback are important.

In discussion with academic colleagues, I have heard concerns raised about students plugging strange designs into models and not being alarmed by the strange results. This is a very important skill, and to be fair to students, one that comes mainly through experience rather than academic learning. It would be great to see students challenged to build models that emulate the physical results that Bath have measured as a learning exercise – maybe this is done already.

Finally, there’s the QA process. This is a particular weakness of Part L in my opinion, and something the Passivhaus methodology is better at. Passivhaus isn’t just PHPP, it’s a design process, followed by QA process, which is informed by PHPP. I sometimes tell people we first make sure the model and the design are aligned, then we make sure the building and the design are aligned. In an industry where a cladding panel gets swapped out to make a £2/m² saving it’s easy to see that the latter is often not done properly. Crucially, the QA is completed by a third party, and while this costs it works. The lack of performance gap in Certified Passivhauses is testament to that.

In conclusion

The University of Bath paper uses an interesting methodology that many are questioning. But it reveals some uncomfortable truths some of which we have to read between the lines to get. As a profession, we can and must do better at energy modelling, but to get there we need to take a holistic look at what we do and why we do it. As we go on that journey, I propose we go armed with Occam’s Razor, and one of the first items under my knife would be the blanket application of DSM.