The government already has access to every phone number in existence. They can already track every phone to figure out who attended a protest or whatever. Filtering down to "all phone numbers who've ever connected to Signal" doesn't exactly narrow anything down. They don't have any metadata about who you were chatting with.