Elasticsearch Upgrade and Migrations

Upgrading between major versions of Elasticsearch

It is possible to migrate your Elasticsearch deployment to use the newest version of Elasticsearch. It is currently possible to:

• migrate from Elasticsearch 5.x to 6.x
• migrate from Elasticsearch 2.x to 5.x
• migrate from Elasticsearch 1.x to 2.4.6

🚧

Support Ending for Elasticsearch versions 1.x through 2.4.6

Compose will no longer be supporting Elasticsearch 1.x, 2.1.1, 2.4.0, 2.4.4, and 2.4.6 as these versions have reached end-of-life. This means that users with deployments on these versions should migrate those deployments up to 5.6.9.

It is not possible to:

• migrate from Elasticsearch 2.x directly to Elasticsearch 6.x
• migrate from Elasticsearch 1.x to any version other than 2.4.6

📘

In-place Upgrades

If you are running Elasticsearch 5.6.16, then you can upgrade your deployment to 6.8.4 in place. Upgrade to through the Compose UI Settings panel, and click the Change Version button to start the upgrade process. Please make sure that your deployment is compatible with Elasticsearch 6.x before upgrading.

The process to migrate between all other major versions uses a recent or on-demand backup to restore the data into a new deployment.

Why restore to upgrade

As a matter of policy, when a major upgrade is needed, we only enable upgrade through restoring into a new deployment. It has a number of advantages for users:

• The original database stays running and production work can be uninterrupted.
• Users will be able to test the new database out of production and act on any application incompatibilities.
• The entire process can be re-run at any point.
• A fresh restoration reduces the likelihood of unneeded artifacts of the older version of the database are carried over to the new database.

Before migrating

One element of the deployment will not be carried over in this process; the usernames and passwords used to access the system. You can document your existing usernames for reuse at this point or you can create all new users and credentials when the migration is complete.

Select or create a backup

Go to the deployment's Backup tab. If you are happy to restore from one of the existing daily backups move on to the next step. If you want to create a new up-to-date backup, click the Backup Now button.

You will be sent to the Jobs tab to see the progress of the backup that is being created. When it is complete, return to the Backup tab where the new backup will be available in the table under On-Demand Backup section.

Begin a restore

For the backup you have selected to use, click on the circular arrow to the right of the backup's entry in the table.

This will take you to the New Elasticsearch Deployment view where you can configure your new Elasticsearch deployment. You can use the generated deployment name or enter a new name. You should select the location for your new deployment from the available datacenters or private clusters displayed in the drop-down menu.

Finally, select the version to upgrade to. You will be shown all versions the backup can be restored to and that does include the current version — that's allowed to enable cloning of databases. Click Create Deployment.

This will take you to the New Deployment screen and show you the progress of creating your newly upgraded deployment.

After migrating

Once the new deployment is running, remember to add users to it to enable access.

Migration Notes for New Elasticsearch 6.x Users

As in previous version upgrades, there are many changes. There are two in particular that might affect how you use Elasticsearch.

Pre-5.x Indices

Elasticsearch will usually support reading indices that are created in the previous major version. This means that any indices created in Elasticsearch 5.x will be readable by Elasticsearch 6.x.

If you have indices that were created in Elasticsearch 2.x, they will not be readable in Elasticsearch 6.x. Those indices will need to be re-indexed in Elasticsearch 5.x before migrating to 6.x. If your data contains indices made in Elasticsearch 2.x and you attempt to migrate to Elasticsearch 6.x, the nodes will fail to start and the migration will fail.

Re-indexing can be done in-place in Elasticsearch 5.x before migrating to 6.x through the Reindex API.

Index mappings

The ability to create multiple mapping types per index has been removed in Elasticsearch 6.x. Any multiple index mappings created in Elasticsearch 5.x will continue to work, but Elasticsearch 6.x only supports creation of a single mapping type per index.

Mapping types will be removed in Elasticsearch 7.x and above.

Other changes

Full documentation of breaking changes can be found in the Elasticsearch docs.

Migration Notes for New Elasticsearch 5.x Users

The change from 2.4.x to 5.x in Elasticsearch sees many, many changes. Some have backward compatibility baked in, such as the change from string types to text and keyword types, others are changes in the ways various aspects of existing operations function.

Replica counts

In Elasticsearch 2.x it was possible to set the default number of replicas and shards on indices. Older 2.x indices had a default of 2 replicas and 3 shards to ensure high availability.

With Elasticsearch 5.x, Compose deployments modify the template index instead, and we later backported this change to 2.x. The template index sets the defaults for new indices to 1 replica and 3 shards. The number of replicas and shards can also be set when an index is created.

Indices which have been migrated from previous versions of Elasticsearch will retain their replica count, so if they were modified, they will remain modified. We suggest that in that situation you check your replica counts after you complete your migration.

From Elasticsearch 1.x

Elasticsearch deployments running 1.x can be migrated to 2.4.6, but it's possible that you will have to make changes to your data before you migrate. To help, 1.x deployments have access to the Elasticsearch migration plugin by accessing /_plugin/migration on one of your deployment's connection strings.
For example if your HTTP Connection string under Connection Info is:
https://[username]:[password]@portal1557-29.legendary-elasticsearch-75.compose-34.composedb.com:27203/
then the plugin is found at:
https://[username]:[password]@portal1557-29.legendary-elasticsearch-75.compose-34.composedb.com:27203/_plugin/migration

The migration plugin will highlight everything that needs to be fixed prior to being able to successfully restore to 2.4.6

Still Need Help?

If this article didn't solve things, summon a human and get some help!