the cert-file must only contain the PEM encoded certificate. Which looks like that:
Open the file with a text editor. If the file starts with human readable data like that:
Version: 3 (0x2)
Serial Number...and so on
Than that has to be deleted so that only the encoded certificate remains.
Backup before editing.
Since (AFAIK) the default behaviour of OpenSSL is to include the human readable certificate information into the cert-file, Synology might consider to work on the import check mechanism.