cpanel mysql update breaks mysql, in yet another way
cpanel wouldnt update at all, cos mysql was too old, so i allowed it to update once... big mistake
old my.cnf:
binlog-do-db=db1
binlog-do-db=db2
binlog-do-db=db3
binlog-do-db=db4
new my.cnf:
binlog-do-db=db4
ouch...
guess now i'll do a full mysql backup, copy it to all the slaves, and start the reload process.
-
Hello :) Could you elaborate on the specific method you used to upgrade MySQL? Was it through Web Host Manager? What version did you upgrade to and from? Thank you. 0 -
Hi cPanelMichael, I don't know where to find that information. The details of the update may be listed in your ticket #4810867 0 -
That ticket addresses the cPanel update itself. Could you open a separate ticket regarding the changes made to your /etc/my.cnf file so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome. Thank you. 0 -
Hi cPanelMichael, Really, I'd prefer if cpanel staff didn't access the server. It's clear to me that the cpanel upgrade tried adding to the my.cnf file, by taking all the settings and removing the duplicates. cPanel seems completely unaware that some settings are supposed to be duplicated. cPanel is great if you don't know how to configure mysql for yourself, however if you run a server with advanced settings then you have to be very careful that it does not break your mysql. The damage is already done, and now I am moving our servers away from cpanel to prevent any future damage. 0 -
[quote="cornernote, post: 1635781">Hi cPanelMichael, Really, I'd prefer if cpanel staff didn't access the server. It's clear to me that the cpanel upgrade tried adding to the my.cnf file, by taking all the settings and removing the duplicates. cPanel seems completely unaware that some settings are supposed to be duplicated. cPanel is great if you don't know how to configure mysql for yourself, however if you run a server with advanced settings then you have to be very careful that it does not break your mysql. The damage is already done, and now I am moving our servers away from cpanel to prevent any future damage.
I've really never seen this happen - and I run hundreds of servers with custom my.cnf settings. You should let cPanel troubleshoot the problem if you're going to complain about it on their forums.0 -
@vanessa, Do you have binlog-do-db multiple times (or any other similar setting such as replicate-do-db)? If you want some (but not all) databases in your binlog, you have to define binlog-do-db on multiple lines. Do you use binary logging or replication at all? If you do, its amazing you have had no issues. It has broken replication during updates before, see this blog post for the exact details: [url=http://openquery.com.au/blog/mysql_install_db-mysqld-bootstrap-binary-log-cpanel]mysql_install_db, mysqld "bootstrap, binary log, cPanel | Open Query blog 0 -
I actually have four servers that I manage where replication is used in this fashion, but not necessarily by choice :) [url=http://code.openark.org/blog/mysql/quick-reminder-avoid-using-binlog-do-db]Quick reminder: avoid using binlog-do-db | code.openark.org [url=http://www.mysqlperformanceblog.com/2009/05/14/why-mysqls-binlog-do-db-option-is-dangerous/]Why MySQL's binlog-do-db option is dangerous - MySQL Performance Blog I haven't seen a problem so far where these lines were removed following a MySQL update. I suppose if it's that big of an issue, chattr'ing the my.cnf isn't completely out of the question, but typically any admin would advise against doing that unless absolutely necessary. You can alternatively prevent cPanel from updating MySQL at all, and instead run the updates on your own terms when you are able to monitor what is going on. I'm not saying that this isn't a bug or that cPanel isn't doing this - I'm simply saying that *I* have not seen this problem while using a similar setup. Regarding my latter statement, you may have misunderstood me. If you're going to make a stink about a problem and even open a ticket for them to look into it, you should be willing to give them the ability to actually help. Sure, cPanel reps have the ability to create test environments, and they do. The few times I do contact cPanel support, I've experienced nothing less than them going above and beyond to resolve my issues, even when I contact them through "unofficial" channels. However, often times when troubleshooting they will need to be able to evaluate the environment in which the problem is occurring. Not every server is set up identically, and you've already indicated that you have a fairly advanced setup that their test environment is unlikely going to mimic out of the box. So if you're not going to let them troubleshoot the problem, you can't really accuse them of being incompetent or not understanding. 0 -
@vanessa, I am well aware of the issues with using binlog-do-db, and my application has been developed with these things in mind. Replication has never been broken by the issues you pointed out, nor do they have anything to do with this bug report. Replication has however been broken by the cpanel mysql updater on multiple occasions. As specified in my OP I do disable the mysql auto-updater because of previous problems which I linked to, and you seem to have no comment on. The author of the article described cpanel's work arounds as dangerous hacks that seemed to lack understanding of the cause to the actual problem. Replicating the issue should be very simple. I have explained the specific parts of the my.cnf that get removed. If they run a test and it doesn't happen for them, they are welcome to report that in this post. This has not been done, so I assume this has not been tested. I don't see how access to my box will assist them with that. After all they can't re-upgrade mysql. Even if they could, I don't want them to. If they need more information about the box I am be happy to provide it, however this has not been requested. I don't want to get further this debate with you about how I should configure my server, who should have access, or if cpanel did or didn't cause the issue. The facts are simple, cpanel's updater did not consider how the my.cnf can be configured, and broke mysql replication. I have every right to post this bug report here in the forum, and every right to choose who has access to my box. I wish you would stop using terms like "complain" and "make a stink". You seem to be trying to belittle my bug report. It's upto cpanel if they wish to address the issue or not. So please go about your day instead of trying to hijack this thread. 0 -
yo - don't post to a public forum if you don't want sensible replies. If you say something inaccurate or ignorant, expect someone to correct you. You also didn't post any of what you stated in your OP, in fact, the OP was barely readable. Chill. 0 -
I have no issue with sensible replies, and feel free to correct me if I am wrong about something (however not sure what needs correcting). So far you have only told me that I should give access to my server (with no good reason), I shouldn't use binlog-do-db (again, no good reason), stated that you don't have this problem (without clarification if you use binlog-do-db), and used terms to make it seem like I am making a fuss, when in fact I am simply reporting this as an issue. OP says "so i allowed it to update once". OP explains the issue that binlog-do-db was wiped from my.cnf. Are you trying to help with this thread, or just trying to increase your post count? Perhaps you can explain what is is you are trying to help with. Your comments are not assisting to get this issue resolved. In fact they are doing the opposite because you are advocating that cpanel staff should not investigate unless I give them server access. I can't see why they need it, nor has anyone given me any reason why they need it. 0 -
Defects should be reported via this link:
This solves nothing. What troubles me is this: [quote="cPanelMichael, post: 1634401">Hello :) Could you elaborate on the specific method you used to upgrade MySQL? Was it through Web Host Manager? What version did you upgrade to and from? Thank you.
[quote="cornernote, post: 1634951">Hi cPanelMichael, I don't know where to find that information. The details of the update may be listed in your ticket #4810867
With this setup that you require, you updated without knowing what you were updating to? How can that be? [QUOTE]Are you trying to help with this thread, or just trying to increase your post count?
I am trying to help, and my post count is fine. vanessa and cPanelMichael are trying to help you too. cPanel Technical Support is standing by, waiting to help you as well.0 -
Hi @Infopro, Thanks for pointing me to the defects page. I didn't see it. I'll report the issue there. I didnt try updating again, because I don't want to risk breaking replication again. I am running a live site that requires real-time replication to be functional. I prefer cpanel staff to not come onto my server because it increases the risk that something will be changed from zero to something above zero. If you need to verify something then you can try on your own non-live servers, or if you need further information you can ask me and I will gladly provide it. I prefer for cpanel staff not to resolve anything, because I have already rectified the problem that was caused. I am very concerned for the state of my servers and don't need additional changes at this point. If it is simply to better the cpanel product then testing should be performed away from my live environment. Suggesting testing on my live server increases the chance that something will go wrong, with no advantage other than to test and improve a product that I have already stated I won't be using any more. I am taking the time to report this issue, but I am not putting my server on the line, and quite frankly I am getting fed up with the persistent requests for access and lack of information requested, that I offered to provide if only a specific request was made. Can you please clarify what it is you need to check on my server and I will gladly check. The only thing that has been asked so far is the previous and new version numbers. I stated that I don't know where to find these, and no instructions have been given to obtain this information. If the techs login, where will they get this information? Personally I think moving away from cpanel will prevent future issues. Can you clarify why you think it solves nothing? I didn't think it was required to record the previous and upgraded version of cpanel. Apparently the tech that assisted with the upgrade felt the same way. In addition it seems cpanel itself doesn't store that information, so it seems that its not considered important until after the fact. If its recorded in a log file then you can point me to it and I'll update this thread with the information. I have expressed my concern with venessa's replies. I have no concern about cPanelMichael's post and didn't claim he was not trying to help. Not sure why you added his name or your name to the list after quoting my comment, seems you are concerned I will find those posts non-helpful, but that is not the case. 0 -
[quote="cornernote, post: 1636151">Hi @Infopro, Thanks for pointing me to the defects page. I didn't see it. I'll report the issue there. I didnt try updating again, because I don't want to risk breaking replication again. I am running a live site that requires real-time replication to be functional.
When you open a Defect report, you will be asked by cPanel Technical Support for access. You could of course provide specific details as well so that they can do some testing of their own, but the access needed is so they can view your existing setup. [quote="cornernote, post: 1636151"> I prefer cpanel staff to not come onto my server because it increases the risk that something will be changed from zero to something above zero. If you need to verify something then you can try on your own non-live servers, or if you need further information you can ask me and I will gladly provide it. I prefer for cpanel staff not to resolve anything, because I have already rectified the problem that was caused. I am very concerned for the state of my servers and don't need additional changes at this point. If it is simply to better the cpanel product then testing should be performed away from my live environment. Suggesting testing on my live server increases the chance that something will go wrong, with no advantage other than to test and improve a product that I have already stated I won't be using any more. I am taking the time to report this issue, but I am not putting my server on the line, and quite frankly I am getting fed up with the persistent requests for access and lack of information requested, that I offered to provide if only a specific request was made. Can you please clarify what it is you need to check on my server and I will gladly check. The only thing that has been asked so far is the previous and new version numbers. I stated that I don't know where to find these, and no instructions have been given to obtain this information. If the techs login, where will they get this information?
The specific question asked was: [QUOTE]Could you elaborate on the specific method you used to upgrade MySQL? Was it through Web Host Manager? What version did you upgrade to and from?
[quote="cornernote, post: 1636151"> Personally I think moving away from cpanel will prevent future issues. Can you clarify why you think it solves nothing?
Moving away from cPanel prevents you and your users from using cPanel. Of course if you have no need for cPanel that's something else all together I guess. Finding out what went wrong and solving the issues you had going forward, I think, a better plan. [quote="cornernote, post: 1636151"> I didn't think it was required to record the previous and upgraded version of cpanel. Apparently the tech that assisted with the upgrade felt the same way. In addition it seems cpanel itself doesn't store that information, so it seems that its not considered important until after the fact. If its recorded in a log file then you can point me to it and I'll update this thread with the information.
The version is at the top of the log file you pasted to your prior ticket. You were on an outdated, EOL version of cPanel, 11.38. [quote="cornernote, post: 1636151"> I have expressed my concern with venessa's replies. I have no concern about cPanelMichael's post and didn't claim he was not trying to help. Not sure why you added his name or your name to the list after quoting my comment, seems you are concerned I will find those posts non-helpful, but that is not the case.
He asked you a question trying to help, and you choose not to answer that, instead you stated you did not want to provide access, and then ended your reply with: [QUOTE] The damage is already done, and now I am moving our servers away from cpanel to prevent any future damage.
You manually killed the update. cPanel sorted this and got you going again. cPanel Technical Support is here to help. I'm very happy to hear that you got updated and away from 11.38 and that you sorted out your issues seemingly brought on by the upgrade. Still, I can only assume that you're not the only one with the setup you have, and if somethings not working properly with the cPanel product, cPanel Technical Support should know about it. As I said, to make the product better. You moving away from cPanel doesn't help. Whether you provide access or not is your call of course, but please do open a new ticket on this, as Michael suggested, and provide as many details as you can so that it could be duplicated, tested and any issues that are found, can get fixed. I have read no comments by cPanel Technical Support here or in your prior ticket that mention if your existing setup is even supported as you have it. If it's not, I would think taking precautions ahead of updating might have been prudent. GL!0 -
Hello, So I spun up a test server running 11.38.2 (build 25), and MySQL 5.0.96 -bash-4.1# /usr/local/cpanel/cpanel -V 11.38.2 (build 25) -bash-4.1# mysql -V mysql Ver 14.12 Distrib 5.0.96, for unknown-linux-gnu (x86_64) using readline 5.1 -bash-4.1#
I then modified the /etc/my.cnf file and added 4 lines:# cat my.cnf [mysqld] innodb_file_per_table=1 binlog-do-db=db1 binlog-do-db=db2 binlog-do-db=db3 binlog-do-db=db4 -bash-4.1#
I'm assuming here that I don't actually have to have any replication running, since you simply stated that the my.cnf file is changed during a MySQL update. But if that's wrong, then please do let me know. Then I ran a MySQL Update. Which updated MySQL to 5.1.73-bash-4.1# mysql -V mysql Ver 14.14 Distrib 5.1.73, for unknown-linux-gnu (x86_64) using readline 5.1 -bash-4.1#
I then checked the /etc/my.cnf file and it still looked identical to before the update...-bash-4.1# cat /etc/my.cnf [mysqld] innodb_file_per_table=1 binlog-do-db=db1 binlog-do-db=db2 binlog-do-db=db3 binlog-do-db=db4 -bash-4.1#
So I was not able to duplicate the problem you originally reported. I will try again now updating to MySQL 5.5, and will post the results shortly.0 -
Hello, I was able to reproduce the issue you experienced when updating from 5.1 to 5.5. The /etc/my.cnf file now does have those additional options missing... -bash-4.1# cat /etc/my.cnf [mysqld] innodb_file_per_table=1 binlog-do-db=db4 default-storage-engine=MyISAM -bash-4.1#
Now that I know I can reproduce the issue, I will file an internal case on this with our developers.0 -
[quote="cPanelPeter, post: 1636612">Hello, I was able to reproduce the issue you experienced when updating from 5.1 to 5.5. Now that I know I can reproduce the issue, I will file an internal case on this with our developers.
Perhaps having this info from the OP would have helped narrow it down - it would explain why it's unfamiliar, because not only is this a non-standard setup, but he's going from a very outdated version. I would assume then that this isn't an issue anymore for the OP, since upgrading from 5.1 to 5.5 is typically a one-time thing.0 -
Hello, While that's true (MySQL updates should be pretty rare) and many of these back and forth conversations could have been avoided with all the information at hand, the problem still needs to be addressed. I just checked again and this happens with an update to MySQL 5.6 as well (running 11.42.1.5). So I have opened an internal case on this issue. Case # 100717 - MySQL Update from 5.1 to 5.5/5.6 removes lines in /etc/my.cnf that can then break MySQL When our developers address this issue, it will appear in the change logs. I do want to thank everyone for their input and helping me get this bug tested and confirmed. Although it may not seem like it, all the responses were indeed helpful. 0 -
Thanks @cPanelPeter :) 0 -
What she said. :p 0 -
The issue (of MySQL configuration breaking or being modified in an undesired way during MySQL upgrades via WHM) is corrected in upcoming cPanel&WHM version 11.44, (development version 11.43). This was recently identified during testing and resolved in 11.43/11.44 via internal case 99141. As the issue also affects the specific scenario in older versions where MySQL 5.5 or 5.6 is an upgrade target, I have expressed the need to back-port the resolution in case 99141 to both cPanel&WHM versions 11.42 and 11.40. I don't have an ETA for version 11.43/11.44, but it will be made available first via the edge tier; the following resource can be used to determine when 11.43/11.44 has reached each tier: [url=http://go.cpanel.net/builds]http://go.cpanel.net/builds 0 -
[quote="vanessa, post: 1636651">Perhaps having this info from the OP would have helped narrow it down - it would explain why it's unfamiliar, because not only is this a non-standard setup, but he's going from a very outdated version. I would assume then that this isn't an issue anymore for the OP, since upgrading from 5.1 to 5.5 is typically a one-time thing.
As I said, I was more than happy to provide more information, however even if asked I couldn't have helped because I upgraded 5.0 to 5.1, then 5.1 to 5.5. I didn't know which version update caused the issue. There is a saying about assumptions being the mother of all ....ups. Very interesting that you run this setup on 4 servers and never experienced this issue. Perhaps you should check your my.cnf? [quote="cPanelPeter, post: 1636682">I just checked again and this happens with an update to MySQL 5.6 as well (running 11.42.1.5).
Does the fix also apply to other settings, for example replicate-do-db? I am unsure if there are other settings which are intended to be listed multiple times. These are the main 2 that I know of. Thank you for checking the issue in such detail and updating this thread with your findings. Thanks to all the cpanel staff who put the effort into testing this on their own servers without the need to risk my live server. I understand the information I provided was minimal, due to the lack of information recorded by myself, cpanel staff, and cpanel itself. I am still planning to move away from cpanel as it has caused more issues in the past than the benefit it has provided. I don't have "users" as such, I simply used cpanel to make my own life easier to manage sites. Now that I am running a less standard system, and that I am more familiar with system management I have found it easier to do these things manually rather than press a button and let cpanel do it's magic. [COLOR="silver">- - - Updated - - - One last thing... When upgrading cpanel/mysql, I had to enable mysql upgrades. I did this via a setting in WHM, in either "tweak settings" or "update preferences". I don't recall if I changed the setting back, I think because I left it overnight for cpanel staff to check the upgrade because it didn't go smoothly. Now I am trying to confirm that mysql upgrades is disabled, however I cannot find the setting. Is there a different method to disable mysql upgrades now, or am I looking in the wrong place?0 -
It's to be noted that, again, I did not experience the same issue. However, my findings are irrelevant because you neglected to mention that you were upgrading from 5.1 when this happened. I however, was upgrading within the same version tree. And yes, my my.cnf is fine on all four servers. There's also really no point on you continuing to take jabs at me. Your issue was acknowledged by cPanel, and I couldn't care less about your situation at this point. You asked for help but didn't want to give anyone the means to help you, then you talk about how unhelpful and incompetent everyone is until yet another person went out of their way for you and was able to reproduce your problem. Get over it and move on. [QUOTE] When upgrading cpanel/mysql, I had to enable mysql upgrades. I did this via a setting in WHM, in either "tweak settings" or "update preferences". I don't recall if I changed the setting back, I think because I left it overnight for cpanel staff to check the upgrade because it didn't go smoothly. Now I am trying to confirm that mysql upgrades is disabled, however I cannot find the setting. Is there a different method to disable mysql upgrades now, or am I looking in the wrong place?
/scripts/update_local_rpm_versions --edit target_settings.MySQL55 unmanaged /scripts/update_local_rpm_versions --edit target_settings.MySQL56 unmanaged
Docs: [url=http://docs.cpanel.net/twiki/bin/view/AllDocumentation/InstallationGuide/RPMCookbook]RPM Cookbook0 -
[quote="cornernote, post: 1637712">Does the fix also apply to other settings, for example replicate-do-db? I am unsure if there are other settings which are intended to be listed multiple times. These are the main 2 that I know of.
We re-retested and verified the resolution of case 99141 using additional directives from your configuration scenario and could no longer reproduce the original symptoms. As of cPanel&WHM 11.44, build 11.43.0.8, MySQL configuration customizations in /etc/my.cnf should be preserved, including directive ordering, spacing and formatting (new-lines and tabbed indentation), commented/disabled directives, and in-line comments after directives. Reference: cPanel&WHM 11.44 Change Log0
Please sign in to leave a comment.
Comments
23 comments