The issue is that the engineer is using the wrong hashing algorithm to generate the hash for the custom detection policy. Cisco AMP for Endpoints requires the use of SHA-256 hashes for simple custom detections, as stated in the Configure a Simple Custom Detection List on the AMP for Endpoints Portal document.
SHA-256 hashes are 64 hexadecimal characters long, while MD5 hashes are 32 hexadecimal characters long.
Therefore, if the engineer tries to upload a hash created using MD5, the dashboard will indicate that the hash is not 64 characters and is non-zero, as shown in the image below:
To resolve the issue, the engineer should use a tool or a website that can generate SHA-256 hashes from files, such as this one, and upload the correct hash to the custom detection list. References : Configure a Simple Custom Detection List on the AMP for Endpoints Portal, Create an Advanced Custom Detection List in Cisco Secure Endpoint, Working with Advanced Malware Protection (AMP) for Endpoints

Cisco Cognitive Threat Analytics helps you quickly detect and respond to sophisticated, clandestine attacks that are already under way or are attempting to establish a presence within your environment. The solution automatically identifies and investigates suspicious or malicious web-based traffic. It identifies both potential and confirmed threats, allowing you to quickly remediate the infection and reduce the scope and damage of an attack, whether it's a known threat campaign that has spread across multiple organizations or a unique threat you've never seen before.
Detection and analytics features provided in Cognitive Threat Analytics are shown below:
+ Data exfiltration: Cognitive Threat Analytics uses statistical modeling of an organization's network to identify anomalous web traffic and pinpoint the exfiltration of sensitive data. It recognizes data exfiltration even in HTTPS-encoded traffic, without any need for you to decrypt transferred content
+ Command-and-control (C2) communication: Cognitive Threat Analytics combines a wide range of data, ranging from statistics collected on an Internet-wide level to host-specific local anomaly scores. Combining these indicators inside the statistical detection algorithms allows us to distinguish C2 communication from benign traffic and from other malicious activities. Cognitive Threat Analytics recognizes C2 even in HTTPSencoded or anonymous traffic, including Tor, without any need to decrypt transferred content, detecting a broad range of threats
...