Sounds like a case of somebody rolling their own cryptography without knowing what they were doing. Why are they not using a mainstream cryptographic hash that has been analyzed for years without finding vulnerabilities?