It’s 6 in the morning, but here I am working on OpenStack for some reason. I really must hate myself or something, or maybe I just do it for the reddit karma, who knows.

A few days ago I wiped my OpenStack cluster in favor of a new topology and drive policy to make better use of the hardware I have available. I really didn’t experience a whole lot of issues with this third installation with the usual exception of a few typos here and there so I had a running deployment pretty quickly with much more storage, but a serious problem hiding under the rug.

After verifying that my storage size had increased (and it had! yay!) I proceeded to attempt to spin up a large amount of small VMs (9 m1.tiny cirros instances) and quickly found myself staring down the barrel of a loaded gun. Very much unfortunately for me, mariadb was the one doing the pointing, and decided to pull the trigger. Everything came to a halt. Nothing was loading out of the database, my inventory was loading, hypervisor stats weren’t loading, and I couldn’t log into the DB with MySQL workbench. Fuck.

A frantic search through the logs revealed an issue that I hadn’t had before. “Failed to connect to database: Connection refused, too many connections.” Ugh! This seemed incredibly strange because the guide I’ve been using explicitly sets the maximum number of connections for mariadb to 4096. How in the hell was I using that many connections for 9 instances??!!!!! A little google-fu granted me the ability to view the variables set in mariadb and sure enough the variable max_connections was set to 214???? But I set it to 4096? Last I checked 214 is not equivalent to 4096. I’ve been known to make simple math mistakes before, but as an engineering student I’m pretty good with math.

Ok so what’s going on? Is mariadb not reading my config files? Google was not super helpful for this issue surprisingly. Lots of things like “you’re using the wrong config file” even though I’m not using MySQL, or “some other config file is overriding your config file.” None of those were remotely close to my issue. A quick test of the config file by injecting some garbage data showed that mariadb was in fact reading my config file, but for some reason max_connections was being limited to 214.

So here I was around 5:30 or so and I decide to search around on stackoverflow. A lot of off topic things as usual, but then success! I stumbled across this post which ended up being my exact issue! Apparently there is a limit to the number of files a user can have open. I didn’t realize this, but now it makes sense that the mysql user can only have so many files open and therefore can only facilitate so many connections. Editing limits.conf to allow mysql to open more files did the trick! After a reboot checking the max_connections variable in mariadb showed 4096! Awesome!

So now when I launch 9 instances my database connections don’t crash and everything is working like it should. I’ve even launched two more larger instances still with no problems. So it does seem that now the third time is the charm and I’m happy.

OpenStack has been an excellent project for learning. I’ve learned so much about CentOS as well as general systems management and mass deployment. I’ve had quite a few struggles, but all of them yielded results and a better understanding of the subject. My hope is that now I may be able to roll a custom deployment playbook for ansible to better automate the installation process of a cluster like this. I know that it already exists, but where’s the fun in that? It’ll be a good excuse to learn ansible plays better so it’ll be a good experience, and hopefully give me some powerful tools for when I end up deploying a production stack.

It seems like there are quite a few people that enjoy reading about my experiences with OpenStack and I hope that maybe my struggles will be their benefit. Either way I’ll continue posting about my experiences with OpenStack. It’s fun for me too and it’s a great way for me to keep track of my own experiences and document my fixes for issues I’ve faced.

Hope you’ve enjoyed so far! Catch you next time!