Is there a way I can test my server's connection to the WHMCS licensing server?
Yes - WHMCS offers a guide and a tool to help you troubleshoot your connection to their licensing server. The IP's of the WHMCS licensing server is below, so it may be beneficial to confirm if these IP's are whitelisted.
If you have confirmed your connection to the above IP's is not restricted, then you can use WHMCS's built-in tool to troubleshoot further. To test the connection between your server and the WHMCS license server, a built-in test tool is available on the admin area login page:
More information on how to use this test and how to better understand this output can be found here:
help.whmcs.com | WHMCS - Troubleshooting Licence Connection Error