It may be much more sinister than that. I am not sure how track and trace works but even in the early days the receiver of a mobile call could get the geoographic location of the caller. Rescue services such as the RAC and AA were using this over 15 years ago. It is possible that once you have answered the call they are mapping your position. We know where you live.
Speaking from some experience within the telecoms industry, the geo-location of a mobile is something that can be easily determined at the network level. In the earlier days of mobile networks, specialised components were deployed to give the cell information and estimates of the physical location based on signal strength, etc. These days, such elements are highly integrated at the core network and are as secure as any of the rest of the network – I expect GCHQ/NSA/FSB/etc have access but not to be downloaded on an app…
Your reference to the RAC and AA having location information was almost certainly thanks to a commercial agreement between them and the mobile operator(s) enacted by system to system level query-responses. This was not and to the ebst of my knowledge is not available to the public – including scammers who have no system level access to networks.
Accessing your mobile via virus and IP network attack is more likely – but falls into the same category as attacks on your PC or Mac IMO.
As Nigel says, "'they' may well know where you live anyway" since there is so much data out there to identify each of us to the nth degree.
Hope this helps!
Simon