As of December 6, 2019, Vidyard’s largest and oldest application (internally known as “Dashboard”) is fully running on Rails 5. Major version upgrades usually take a considerable amount of resources and this was no different — it took over a year of preparation and several months of dedicated effort before we were ready. This post reflects on our process and some of the bumps we ran into along the way.
Over a year ago we started to scoped out what it would take to upgrade Dashboard to Rails 5. We knew from previous upgrades (e.g. Rails 3.2 to 4.2 in December 2015) that we wanted to keep our other dependencies up-to-date and to backport as many changes as possible. Vidyard has also been slowly transitioning over to a microservices architecture which meant that as time went on there would be fewer pieces to worry about when we finally committed to the upgrade.
Managing interdependencies within a Gemfile can be one of the most painful parts of upgrading Rails. For example, bumping the version of ActiveSupport can cascade down to various third-party gems, potentially leading to painful modifications of large swaths of code. This is particularly evident in areas of a codebase that haven’t been maintained in quite some time.
How each dependency conflict should be handled depends on various factors. We opted to fork and patch a few gems (e.g. validates_url_format_of and google-api-client) in cases where they were unmaintained or where upgrading to the next compatible version would require major work. Several others, like our heavily modified version of audited, were simply phased out with the creation of new services that handled all related functionality.
Although Vidyard has been shifting towards microservices, Dashboard remains the most heavily trafficked repository by far, with dozens of pull requests made against it daily. Upgrading Rails in a separate feature branch wouldn’t have been feasible, especially when it started out as sort of a side project for all developers involved. Instead, we opted to dual-boot Dashboard under both Rails 4 and 5, similar to how GitHub and Shopify handled their upgrades. This proved beneficial in several ways:
To set up dual-booting, we made use of bootboot, a Bundler plugin created by our friends at Shopify. Bootboot allows us to conditionally require certain dependencies based off of an environment variable and automatically updates the associated lockfile for each environment (Gemfile.lock for Rails 4 and Gemfile_next.lock for Rails 5). Our Gemfile ended up looking like the following:
# Gemfile
...
plugin 'bootboot', '~> 0.1.1', :git => 'https://github.com/Vidyard/bootboot.git', :branch => 'auto-sync'
Bundler.settings.set_local('bootboot_env_prefix', 'DASHBOARD')
Plugin.send(:load_plugin, 'bootboot') if Plugin.installed?('bootboot')
if ENV.fetch('DASHBOARD_NEXT', 0).to_i == 1
enable_dual_booting if Plugin.installed?('bootboot')
gem 'rails', '~> 5.0.7.2'
group :test do
gem 'rails-controller-testing'
end
else
gem 'rails', '4.2.11.1'
group :test do
gem 'test_after_commit'
end
end
...We tried to keep Gemfile.lock and Gemfile_next.lock from diverging as much as possible by only updating Rails-specific gems and any conflicts that arose.
Although bootboot allowed us to conditionally load dependencies, we also needed a way for Rails 4- and 5-specific code to live together in harmony. We added the following helper to application.rb:
# config/application.rb
def rails_4?
current_version = Gem::Version.new(Rails::VERSION::STRING)
return (
current_version >= Gem::Version.new('4') &&
current_version < Gem::Version.new('5')
)
endThis allowed us to conditionally run logic like setting headers in our config:
# config/environments/production.rb
...
if rails_4?
config.static_cache_control = 'public, max-age=31536000'
else
config.public_file_server.headers = {
'Cache-Control' => 'public, max-age=31536000'
}
end
...Finally, we had to configure our CI pipeline to support this workflow. This is heavily organization-specific but the general approach we took was to create two separate Docker images using build arguments. We then added a step to run our test suite under both Rails 4 and 5. For the latter, we configured it such that it would never fail until we were closer to completion. It simply provided much needed feedback on what still needed fixing.
The release of Rails 6 on August 16, 2019 gave us the final push to start the upgrade process as older versions of Rails would no longer be receiving security patches under the core team’s maintenance policy. We formed a small team to focus on getting things over the finish line.
Our first goal was fixing up Dashboard’s Rails 5 test suite. We ran into a number of interesting problems which occasionally led to deep dives in various codebases. Some examples include:
update_attribute method skipped callbacks if the record’s attributes weren’t changed. A PR was made against Rails to fix this which was then backported to the 5.0-stable branch. However, it never made it to an actual release in the 5.0.x series so we simply opted to point our Gemfile at the branch instead. We wanted to avoid bumping Rails to the next minor version before fixing all deprecations under 5.0.belongs_to :created_by, :class_name => 'User', :foreign_key => 'created_by'), it will blow up recursively in certain situations. We renamed the column to better suit Rails conventions (e.g. creator_id). See rails/rails#26778 for additional context.ActiveRecord::Relation delegates to Array when serializing to CSV (via method_missing). This changes in Rails 5 as it now includes Enumerable and only delegates a select list of methods to Array, which means relations should be explicitly converted to arrays before serialization.ActionController::Parameters now returns an object instead of a hash. This is something to be aware of especially when serializing to another format. For example, Sidekiq converts job arguments to JSON; this is fine for hashes but it breaks for complex Ruby objects.rubocop -a --only Rails/HttpPositionalArguments). AppFolio’s blog goes into more detail on this approach.Once the test suite started passing we made it a mandatory step in our CI pipeline. Rails 5 was then set as the default in development and half of our staging instances were swapped over so developers could test changes under both environments when necessary. Over the next few weeks we monitored the state of things, making adjustments as necessary.
We knew from the beginning that we wanted to do a staged rollout in order to minimize impact to our users. We spun up a mirror of our services on AWS using Terraform, pointing the ECS task to the Docker image configured for Rails 5 instead. Fortuitously, Amazon also announced the ability to weight traffic at the load balancer level mid-November; this would form the basis of our rollout strategy over the following days.
The first thing we swapped over were the Sidekiq workers. Given the nature of the jobs and to simplify the upgrade, this was done separately from the applications that served our web and API traffic. One minor thing we ran into as a result of this decision was a (de-)serialization error (ArgumentError: undefined class/module ActiveRecord::ConnectionAdapters::AbstractMysqlAdapter::MysqlDateTime) as jobs were still being enqueued under Rails 4. We added a quick monkey patch to resolve this:
# config/initializers/abstract_mysql_adapter.rb
module ActiveRecord
module ConnectionAdapters
class AbstractMysqlAdapter
if !rails_4?
class MysqlDateTime < ActiveRecord::Type::DateTime
end
end
end
end
endNext, we began to shift traffic over the main instances in the following order: internal API requests (i.e. those made by other microservices within the company), public API requests, and web requests (i.e. rendering Rails views). All in all, it was a fairly uneventful process; error rates and request times were within acceptable bounds. Sweet success!
Major framework updates can be intimidating at times. With the right approach, however, things are much more manageable. Dual-booting has definitely proven its value for us by making it easy to incrementally upgrade small portions of an application. We’ll be following the same approach for Rails 5.2 over the next month or so (and potentially setting our sights on 6.0/master afterwards!).
Special thanks to the Rails community for making the upgrade as easy as possible, whether it is through postmortems of their own upgrade processes, raising GitHub issues/PRs, or creating gems to aid the migration effort.