In OSX Server 5 Apple has changed paths to most everything, placing copies of all binaries and config files within the Server.app package itself. This can cause a lot of confusion as you might be inadvertently using the non-longer-applicable unix binaries and trying to make changes to conf files which no longer apply, as the old binaries and files are not removed.
Used to be that postfix binaries were located at: /usr/sbin/postfix and postfix configuration files were located at: /etc/postfix . Not so anymore!
In Server 5, Postfix and other binaries are now located at: /Applications/Server.app/Contents/ServerRoot/usr/sbin
In Server 5, Postfix configuration files are now located at: /Library/Server/Mail/Config/postfix
Correcting the shell PATH
Your shell may still be finding the old binaries which are no longer used. This can result in a lot of confusion when you are controlling a instance of postfix binaries which are not being used anymore!
At the command-line, try enter: which postfixIf the result is /usr/sbin/postfix , you are trying to using the older binary which is NOT USED by OSX Server 5.
See the article below to modify the .profile, so that your shell will use the correct binaries located within the Server.app
ref: Why do postconf -n and postfix reload produce unexpected output on OS X Server 5?
Alerts and Notifications
If Mail is not hosted on the server, you must still enable Mail services so that server notifications and alerts are delivered via SendMail.
Adding a catchall to OS X Server Mail
In Server 5, the method to create a catchall mailbox for a domain is different from previous versions.
See: https://rossbennetts.com/2016/03/13/adding-a-catchall-to-os-x-server-mail/
Spam Filtering
Spam filtering remains underwhelming in OSX Server. Rather than trying to add the zillion anti-spam applications and burdening the mail server instance, consider using Mailcleaner as Spam Filtering Solution for OSX Server or a third-party service such as DuoCircle.
Postfix conf Settings
To check the value of a setting, use postconf:
/Applications/Server.app/Contents/ServerRoot/usr/sbin/postconf -d smtpd_client_restrictions
References
Krypted: Configure The Mail Service in OS X Server 5