Insulate First
It is generally considered best to insulate a house if possible before looking at a new heating system. Certainly any lofts with less than 250mm of insulation and any unfilled cavities should be treated, these are cost effective and relatively simple measures
The next priority should be any uninsulated sloping ceilings as these can be a huge cause of heat loss, with only half an inch of plaster and lath between the occupants and the outside world. This can be done either between or below the rafters (or preferably both).
Whilst double glazing is expensive and not appropriate for some older properties secondary glazing can be very effective, see www.extraglaze.co.uk. Draught-stripping should also be considered, chimneys can be blocked using chimney sheep, www.chimneysheep.co.uk, gaps in floorboards and under skirtings can be filled with stop gap, www.stopgap.com. Ventilation may need to be maintained for combustion appliances, if possible this should be provided directly to the stove or boiler.
The Green Homes Voucher scheme has been extended till end of March 2022, see https://www.gov.uk/guidance/apply-for-the-green-homes-grant-scheme. Most homeowners can claim up to two thirds of the cost of improvements up to a maximum of £5,000 but those on benefits could claim up to 100% up to £10,000. Trustmark approved installers must be used. There are primary and secondary measures under the scheme, new glazing is a secondary measure which can only be claimed for if a primary measure, eg a heat pump, is being claimed. Houses in Wales with an energy rating of E,F or G can claim grants through the NEST scheme - see www.nestwales.org.ukHouseholds in Powys can get zero interest loans through the Robert Owen Community Bank for household improvements, including insulation, boilers and renewable energy. See https://rocbf.co.uk/home_loans