On-Premises Events Migration - Multi-Server Distributed

[NOTE: This guide is intended for environments that are running a “multi-server distributed” installation of On-Premises Sysdig Monitor. Environments running a single-server “all-in-one” installation should refer to the all-in-one guide instead.]

To improve search capabilities and offer improved performance, beginning with version 494 of On-Premises Sysdig Monitor, Elasticsearch is used to store Custom Events. In order to complete an update to version 494 or newer, it is mandatory to migrate previously-stored events into Elasticsearch.

Because the Elasticsearch support is not present in the Admin console in versions <= 439, in the steps below to complete the migration, you’ll first update to the new version, then add and tag new Elasticsearch node(s) before completing the migration of your events. Follow the steps below to complete the migration.

Prerequisites

Must be up & running Sysdig Monitor On-Premises version 439 or older

Step 1 - Verify existing events

Log in to the Sysdig Monitor application and familiarize yourself with recent events by navigating to the page Events > Custom Events. You may want to take a screenshot so you can refer to it later when confirming the success of the migration.

Step 2 - Initiate the update of the Sysdig Monitor version

Using the Admin console, click from the  tab to begin the update of the Sysdig Monitor application to the latest available version.

Step 3 - Add/tag Elasticsearch nodes

For a multi-server distributed installation, while it is technically possible to run with as little as a single Elasticsearch node, we suggest to have at least 3 Elasticsearch nodes in order to achieve high-availability and reduce the risk of Elasticsearch split brain.

Because Elasticsearch is not yet running the environment, Sysdig Monitor will not finish starting after the update that you just started. Once the Dashboard shows the update procedure has reached the point of “Waiting for all components to start” (see screenshot below), click the Cancel button.

Next, navigate to the   tab in the Admin console and click  to add/tag each of the new server(s) on which Elasticsearch will be run. Check the box to apply the “elasticsearch” tag for each node you add. If you need to revisit the details of adding and tagging a node, refer to the  On-Premises Installation Guide.

Step 4 - Start Sysdig Monitor

Once all new node(s) are visible in the Cluster tab, each with the “elasticsearch” tag, click to the  tab and click the button to start the Sysdig Monitor application.

Wait until the Dashboard indicates the Sysdig Monitor application is running. At this point your Elasticsearch node(s) will be running in a cluster.

Step 5 - Migrate the events

Once the update is completed and the Sysdig Monitor application is running, use SSH to login into one of the newly-added node(s) that is now running Elasticsearch. Store the MySQL server info in the following environment variables:

MYSQL='<INSERT_HERE_THE_IP/HOSTNAME_OF_YOUR_MYSQL_NODE>'
MYSQL_USER='<INSERT_HERE_THE_MYSQL_USER>'
MYSQL_PWD='<INSERT_HERE_THE_MYSQL_PASSWORD>'

If you have not changed the defaults from when Sysdig Monitor was first installed, the latter two settings will be:

MYSQL_USER='admin'
MYSQL_PWD='password'

While still logged into this host, execute the event migration by running the following container:

docker run --rm -it --net=host sysdig/onprem_migration:events --mysql_endpoint ${MYSQL} --mysql_user ${MYSQL_USER} --mysql_password ${MYSQL_PWD} --es_url http://127.0.0.1

A sample output of correct events migration:

2017-02-10 16:50:20.259  INFO 7 --- [           main] com.draios.services.MigrationService     : starting events migration.
2017-02-10 16:50:20.284  INFO 7 --- [           main] com.draios.services.MigrationService     : Starting from id 0
2017-02-10 16:50:20.830  INFO 7 --- [           main] com.draios.services.MigrationService     : Indexed 100 events; latest id is 100
2017-02-10 16:50:20.835  INFO 7 --- [           main] com.draios.services.MigrationService     : Avergage duration per batch (run 1 of size 100): 546ms
2017-02-10 16:50:20.843  INFO 7 --- [           main] com.draios.services.MigrationService     : db events: 100
2017-02-10 16:50:20.844  INFO 7 --- [           main] com.draios.services.MigrationService     : no more events to migrate, exiting.

Step 6 - Verify your events

Log in to the Sysdig Monitor application and check that all your events from before the migration are available by navigating to the page Events > Custom Events.

Step 7 - Clean up

Once the migration is completed, perform the following clean-up command to remove the remnants of the migration tool from the host where you just executed the migration:

docker rmi sysdig/onprem_migration:events
Have more questions? Submit a request