Server response time

Hi, for some reason my site is connecting to both: http and https versions of website when visiting. This is killing my website speed. See attached images about GTmetrix Waterfall chart about this issue. First image is the double connection issue, second image about details from http version and third image about https details. I am not sure how to fix this. And this happens when connecting with http:// to my site not when connecting with https://. I red something about that xmlrpc.php file being unnecessary nowadays but I don't know if removing that is the solution.