The Signal client is open source (some say it isn't BC the Play version has an prompt for updates or Smith, but whatever, there are a half dozen clients) so an audit would reveal this pretty quickly.
@You Will (Not) Escape ☸️ Signal does not have reproducible builds for iOS. This is hard to do but not impossible. Telegram provides it but you need a jailbroken device to verify you can build exactly what is published on the App Store
You can also just look at Signal's reputation and see that they fight against what most would consider common sense exploitation prevention in the name of privacy. We actually can just trust that companies which act in good faith will continue to do so