This post is about how I managed to, one part coax and one part coerce Travis-ci into doing what I wanted it to do.
It’s a bit lengthy and a bit technical so bear with me.
I could write a TL;DR for it, but I’m not very inclined to do so because I feel this post is meant to highlight the process rather than the end result more than anything else.
There comes a point in time in every developer’s life where he has to get his hands dirty with Databases. There also comes a point in every Open Source developer’s life where he decides to try out Travis-ci, because tests are good tight? Sure, they are.
Continuous Integration testing is even better!
Write tests, run them on every commit, get email telling you whenever you broke something, it’s great right? Sure it is! It catches human errors and gives you a safety net to fall back upon.
I’ve been working on a project for a while and I had a couple of unit-tests I’d written for it (Shoutout to Hal for nagging me to get those tests in). I also managed to write a hacky integration test that seems to do the full run through.
Now a couple of days ago, I decided to add in MySQL support for the application on the database side, so far I’d been happily using SQLite with no issues. I was using SQLAlchemy and figured I could just change the Database URI and MySQL would be drop-in replacement.
Naive move.
After a couple of hours of Googling, some help from the people in #mysql and more help from my team I finally figured that I needed to tweak the mysql configuration file (my.cnf hence forth)*.
After making the required changes in my.cnf and in the Schema, I head over to my application, fire it up and do a run through for the Nth time to make sure things work as expected… SUCCESS!
Now comes … the hard(er?) part.
Remember the integration test I told you about?
Well it used to run on Travis-ci just fine… until I decided to add MySQL support.
Those of you that have ever had the pleasure of using Travis, know that Github has tried to make things very simple. You tell Travis what you want it to do via the .travis.yml. You tell it what your tests are and how to run them and it’ll go ahead and do just that.
Great!
Or so it is until you want to configure MySQL in ways that are not exposed by the .travis.yml file.
There are always trade-offs between simplicity and power, so this should’ve been been expected, this should have been common sense. But like a child who doesn’t learn fire is hot until he pokes his finger in it, there are some lessons that make far more sense in retrospect.
I felt a tad disappointed and bewildered at first, not knowing where to begin.
Part of me abandoning all hope of having integration tests for my MySQL-support branch. But, and yes there is ‘but’, I decided to stick it out and see what I could figure out.
So I began my journey to hack** around the Travis build system.
There was a lot to do to even begin figuring out what to do.
Things become easier if you break down the process into the following major steps:
- Is it possible?
- What do I need to do it?
- How do I fulfill that need?
To answer #1 I needed to figure out if I had control or some way to interact with the MySQL server and it’s configuration other than the .travis.yml.
To do this, I started changing the Travis config file to see if I could cat the common locations of the MySQL config file, most notably /etc/my.cnf. I also tried starting, stopping and getting the status of the mysql service.
Then I tried executing a couple of MySQL commands to set the variables I needed.
None of these yielded positive results.
I decided figuring out what kind of environment I was running in before trying to move further might be a good idea. I ran the basics to figure out what was going on :- pwd, who, groups, mysql --help.
Then something happened, a ray light shone through the darkness.
While looking at the Travis log for the push I saw something that piqued my interest.
Travis was using sudo to install some of the other services I need in the Travis configuration file.
Aha!
An incandescent lightbulb came on somewhere.
Could it be?
Could I really have root access for this box? I quickly wrote up a sudo ls to test my theory and whaddya know? It worked! Now that I had root access I was 99% sure what I wanted to do was in fact possible regardless of how hard it might be.
I set out with more confidence knowing that I now had the powers of the root bestowed upon me.
Next up was figuring out #2.
There are plenty of ways to set the options I wanted for MySQL and I had to figure out which one would work. Following the principle of least resistance I tried setting the paramters I wanted using MySQL statements. I tried various combinations, preceding and succeeding the mysql statements with restarts.
No success.
Well this wasn’t working and as someone succinctly put it, “Insanity is doing the same thing, over and over again, but expecting different results.”
Now I like to think I am not insane and these combinations weren’t really getting me anywhere, except maybe on the Travis server abuse/mis-use list.
So I decided to try and tweak the configuration files.
The question was figuring out which one.
I began trying to ls -al the expected location(s), hmm, no luck. But mysql --help tells me there are more locations I can find the configuration file. So I just decide to cat all those at the same time^. This told me that the file I wanted was /etc/mysql/my.cnf.
Got it. I had #2.
I could edit this file and have what I wanted.
So on to #3.
Now ideally I should be able to overwrite this file with the bare minimum I needed and get it working.
Naive. Naive again.
Everything stopped working the moment I did this.
To be fair, the default my.cnf had a lot of configuration and it was probably there for a reason.
So I tried a different route.
I can just go in add the things I want in the existing file.
Sounds simple enough right?
Except for the part where I don’t have access to the file. Or an editor to edit the file.
Or do I?
Enter sed. Sometimes known by it’s longer and more expressive name stream editor. So I wrote up a couple of sed commands that add in the lines to the file in the required part of the file. After confirming that the substitution actually works, I pushed with sweaty palms.
Success!
Sure it’s not a foolproof solution and it might break whenever they modify the file, but it works and that’s more than I could say before I started.
And isn’t that what counts?
Scroll down to see what my glorious git log looked like after I was done.
^ Why at the same time you ask? Because Travis will error out on the first failure and I did not want to try all different combinations to figure out which one, it’s a waste of time and resources.
* If you’re interested in reading what the actual technical problem I was dealing with was, then read on.
I had a 257 long character key as one of the fields in my database (in a table in my database, you pedantic people). Now all this works fine as long as you’re running SQLite because SQLite doesn’t really care much about what you’re storing a whole lot, it typically accepts whatever you give it without question and dumps it in a file, it’s the same when you’re retrieving stuff from the database. In MySQL though, there are optimizations for speed and redundancy and lots of other stuff.
This means MySQL cares more about the data that you store in it. Enter DataTypes. With these DataTypes also come constraints on their size and length. As you might’ve guessed, MySQL doesn’t support storing a 257 character long key (VARCHAR or TEXT) with default settings.
It’s an InnoDB thing.
But it does give you a way to configure your options to force MySQL to allow larger keys and this is infact what I ended up doing.
The stuff you need in your my.cnf is:
#InnoDB config # http://dev.mysql.com/doc/refman/5.5/en/innodb-restrictions.html innodb_file_format = Barracuda innodb_large_prefix = 1 innodb_file_per_table=true
You’ll also have to change your row storage format to DYNAMIC.
You can do this in SQLAlchemy like so __table_args__={'mysql_row_format':'DYNAMIC', ...}
** I use ‘hack’ in the sense it was originally meant to be used; to imply exploratory interaction with a computer. See this and this for more details.
My Glorious git log
ffledgling's blog 
