Accessing website with ip/~user requires more CPU resources?

Hello, I'm load testing my website with apache benchmark. I think that numbers are really bad. I've not associated the domain name to my VPS, so I access the website with IP/~user. Can this require, somehow, more resources? I've a VPS with 4GB Ram, 4vCPU. Wordpress with 0 plugin other than the theme, and xenforo with 0 plugin other than the theme. With this: ab -k -n 10000 -c 3 -l URL
I get: Server Load 3.393066 (4 CPUs) Memory Used 35% (1,358,560 of 3,881,408) Swap Used 23.7% (248,488 of 1,048,572)
With ab -k -n 1000 -c 7 -l URL
I get: Server Load 8.6 (4 CPUs)
3 concurrent users with 4GB 4vCPU seems really bad.