File Manager incorrectly detects valid UTF-8 files as GB18030
Hello,
I’m seeing an encoding detection issue in cPanel File Manager on:
cPanel & WHM 11.136.0.31
A file that is valid UTF-8 is being detected as gb18030 when opening it through File Manager.
For example:
file -bi /home/CLIENTE/laravel/.env
returns:
text/plain; charset=utf-8
And:
file --mime-encoding /home/CLIENTE/laravel/.env
returns:
utf-8
UTF-8 validation also succeeds:
iconv -f UTF-8 -t UTF-8 /home/CLIENTE/laravel/.env >/dev/null 2>&1
echo $?
returns:
0
The newer Fileman API also identifies the file correctly:
uapi --user=CLIENTE Fileman get_file_content \
dir=/home/CLIENTE/laravel \
file=.env
returns:
from_charset: utf-8
to_charset: utf-8
However, the encoding detection used by File Manager returns:
cpapi2 --user=CLIENTE Encoding guess_file \
file=laravel%2F.env
result:
encoding: gb18030
I traced this to:
/usr/local/cpanel/bin/guess_file_encoding
Running it directly reproduces the problem:
/usr/local/cpanel/bin/guess_file_encoding /home/CLIENTE/laravel/.env
returns:
gb18030
The script uses:
Encode::Detect::Detector::detect($buffer)
So it appears that Encode::Detect::Detector is producing a false positive for GB18030, even though the file contains valid UTF-8.
The File Manager template also appears to call Encoding::guess_file_opts when the charset is set to _DETECT_, so this incorrect result becomes the default encoding shown in the editor.
I considered adding a UTF-8 validity check before calling Encode::Detect::Detector, so that:
valid UTF-8 -> utf-8
otherwise -> Encode::Detect::Detector
Before modifying a cPanel-managed file, I would like to know:
- Is this a known issue?
- Is there a supported setting to prefer UTF-8 when it is valid?
- Is there an official workaround or patch?
- Would checking for valid UTF-8 before calling
Encode::Detect::Detectorbe considered a safe workaround?
Character encoding verification in File Manager is enabled. This is therefore different from the older _DETECT_ conversion issue caused by disabling encoding verification.
Thank you.
-
Hey there! This is actually a known issue as certain special characters, which are likely present in that .env file, cause the guess_file_encoding tool to detect the wrong encoding value.
We do have an older case on this at CPANEL-39433 and I've reached out to the development team to see if we can get some movement on this. Thanks for bringing this up!
0
Please sign in to leave a comment.
Comments
1 comment