What the hell is funsize? (Introduction)

And that’s exactly how our eyes look after a night of ceaseless monitor peering too.

This is a post I have been putting off writing for a while now.
But now that I’ve finally gotten down to it, let’s begin!

The first question anyone might ask is What the hell is a funsize?
Go look at Funsize Etymology.

Now that we have that out of the way, let’s actually begin.

Funsize or Senbonakura* is technically a “Partial MAR generation service” that generates “Partial MARs on demand”.
Too much jargon? No problemo, Senõr.
We’ll break it down bit by bit.

If you’ve ever used Firefox, and I hope you have, you’ll know that Firefox ships automatic updates to users that install in the background and get applied when you reboot.

Nifty, right?

What you may or may not know is the stuff that goes on behind the scenes.
Getting updates to your browser and getting them so that they don’t break your browser is harder than you might think. We and by we I mean your friendly, probably-not-in-your-neighbourhood Release Engineering Team do a lot of stuff to make sure that doesn’t happen.

There’s a huge Ginormous Pipeline.

My project focuses on a small part of it.

Applications are growing in size these days and so are their installers. Weighing in at 35MB for Linux64 install, 28MB for the Windows setup and a hefty 58MB for the MacOSX install of the latest Firefox Release Builds.They are big, so whenever you’re served an update, you aren’t given the entire new installer to install off of unless it is deemed absolutely necessary.

We at Mozilla value your bandwidth, because not everyone lives in a country with unlimited bandwidth. We at Mozilla also value our bandwidth because serving complete installers where it’s not necessary isn’t great for resources, server load and CDN costs.

Enter, Partial Updates.

Partial Updates, as the name suggests, are not Complete Updates.

The rationale behind Partial Updates is that we don’t really need to give you an update in the form of an entire new installer because even though things change from version to version; a lot of thing also remain the same.

So what we do instead is just give you the bits that change.

It’s called a ‘diff’, or more precisely in our case, a ‘binary diff’ in developer parlance. We figure out what changed between the version you have and the version you’re being updated to and then send you the diff. Diffs are typically much smaller than the entire installer.
Sometimes, up to 1/7th the size of a Complete update, depending on what platform and which channel you’re on.

This reduced size seems like a win-win for everybody involved, and it is … mostly.

Unfortunately generating these ‘Diff’s is not a computationally cheap process.

Infact it can take a while.

My super fast, work machine can take anywhere between 1-5 minutes to generate this, and the production machines that actually generate the official “Partials” aren’t nearly as powerful. So expect times to be longer than those on my developer machine. Now 3 minutes (on average, say) to generate an Update doesn’t sound like a lot.

And you’re right, it’s not a whole lot… for one Partial.

Here’s where I’d like to emphasize the diversity with which Mozilla and thus the scale at which RelEng works kicks in.

We have upto 97 different locales we ship.
For 4 different Operating Systems.
And we do such shipments everyday.
Nightly and Aurora in addition to the less frequent Bi-Weekly Beta and 6-Weekly Release Builds.
Oh and did I mention we go back upto 3 versions of Firefox everytime?

The numbers add up.

I’ll would let you do the math.
But just to drive the point home, we spend 9 days of compute time.

Every. Single. Week.
Just for generating updates.
And this is not even counting the other branches like UX or E10S.

Like I said, it all adds up.

Now this is where my application comes in.
Caveat lector: Senbonzakura/Funsize is far done, and there’s still stuff to be done before it becomes “profitable” so to speak, but that doesn’t mean that it’s goals aren’t ambitious.

So what does my application do?

It does one thing and one thing only; it generates Partials.
And it’s sole purpose, is to do so efficiently.
In essence, Senbonzakura/Funsize is a web service that generates a partial between any two versions of Firefox you need it to.

There are a lot of advantages to having a dedicated service for something like this.

• The service can optimize for speed for generation of partials in ways that the current “Generate-Partials-as-part-of-Release” process can’t.
  There are possibilities of diff and file level caching across versions and locales. Imagine generating a diff for one locale of Firefox and being  able to use it across locales. If we go by ballpark numbers that should mean reduction in compute time by a factor of 89. In practice it may not touch that number, but it will still be a sizeable factor nonetheless.

• We can generate partials that were not possible before, or had to be generated by hand.
  Sometimes a lot of users are using an older version of Firefox. A version that had a massive uptake at the time of release, but maybe not a whole lot of users updated it since.
  Now they’re all clamouring for the next big release.
  If the version in question is more than 3 versions before the current release, they’ll end up being given a complete installer instead of a partial, because we do want them to update, for their sake and ours. Once Senbonzakura/Funsize comes into play, the entire process of identifying a large user pool and migrating it to a new version via partials can be automated, because the service can generate a partial on demand.

Having a separate service do the heavy lifting also moves the update generation process out of the critical release pipeline. This results in faster builds and less chance of bustage.
There are also other smaller, albeit equally important use cases, where we need to migrate people from an no-longer-supported locale to the nearest supported one.

What does it run on, you ask?

It’s basically a flask app, with a celery backend, while tying into a self-written file cache and a SQLite or MySQL database, depending on your choice. We’re also doing a little bit of docker with a dash of AWS.

It sounds fairly simple, but I’ve had my share of architectural and debugging “nightmares”.
But maybe that’s just an Intern-I-have-no-idea-what-I’m-doing.jpg thing.

I’m an Intern, did you honestly expect better?

The application is still very much a work in progress and there’s a lot to be improved, but it’s getting better and hopefully I’ll have the chance and time to keep working on it to get it in even better shape.

Now, if the ELT could please switch the spotlight to point towards our audience.

<begin>
If you dear reader found this project interesting and you are able and willing to contribute, read on!

Mozilla’s Release Engineering Team is looking for talented young contributors like yourself to help out! Don’t waste a day, join #releng on IRC TODAY and ask how you can contribute to awesome!

If you’re looking for information on how to get started with Senbonzakura/Funsize in particular, I have this amazing, albeit incomplete HOWTO Start contributing guide that I wrote on a bus back from LA with my eyes closed.
</end id=’shameless plug’>

---

* It will always and forever be Senbonzakura to me.

---

Update

I did a talk about Senbonzakura/Funsize at the Mozilla SFO office (as part of my Internship) go watch it here!

What the hell is Funsize? (Etymology)

The origins of the name, the story behind it all, in a familiar annoying Magazine style Q&A.

Is it a new size of clothing?
No.

Is it the name of a compact family hatchback? (Car companies do silly names like this don’t they?)
It very possibly could’ve been, but it’s not.

Were you on prohibited substances when you came up with that name?
Um … No?

Now, now, my shocked audience, these are all fair questions.
Please take a seat and get comfortable, get popcorn and your choice of refreshing drink because this will be a long tale.

Or maybe not.

I began working on a project with the Release Engineering Team in May ’14. The project was originally titled ‘Senbonzakura’ after the ‘Zanpaktuo’ from ‘Bleach’. Upon introducing the name to my team, everyone had a hard time pronouncing and remembering how to spell it right.

Except Nick.
Nick is a boss and seems to have a knack for remembering long and hard-to-pronounce names. Maybe it’s from his days remembering all those unintelligible high atomic number elements’ names as an Atomic Scientist.

But I digress.
It was decided after much debate and controversy (not really) that the project had to be re-named.

My project deals with something called “Partial MARs”, so I figured calling it “Mars Mini” after the namesake chocolatey treats would be clever. Our non-American English speaking team-mates suggested calling it “Mars Funsize” instead because that’s what it’s known by in other parts of the world.

And so it came to be.

We christened our new-born Partial MAR generation service “Funsize”!
Which kind of fits, because I had a lot of fun working on it.

<begin>
And you can too!
If you think Python and scalable Web Applications are your kind of thing, you might want to take a look at the Contributing to Funsize and How to contribute to Release Engineering pages and see if Funsize is your idea of fun!
</end id=”shameless plug”>