Improve performance of model evaluation when no units are present #6161

#4855 is not merged yet, but this is a reminder to work on this issue after merging

As @nden pointed out in #4855, adding unit support has introduced a small performance penalty for models that don't use units. We should try and reduce this penalty so that performance is unaffected by the unit support as much as possible.