I think you have the right tactic, focusing on the physical efficiency of your home and where you are losing heat/cooling
Whatever solution you decide after getting that done will work much better and last longer
for sure get a smart thermostat, so you can make a schedule for heating and cooling...that will save lots too
the new heat pumps are vastly better than the ones made a decade ago
Also, if you have access to wood, look at the wood burning stoves. especially ones with a catalytic combustor
modern wood burning stoves are crazy efficient compared to older ones too
but if you have to buy wood, eats into overall cost savings