For a last couple of years, our development stack stays for SpringBoot+Vaadin running under Apache Tomcat and using MongoDB and ElasticSearch. We have a several installations on Windows and Linux platforms including different Windows Server versions. Our stack is carefully selected to support both platforms with good quality.

Time to time, customer decides to move from one OS platform to another and we have to migrate everything seamlessly. And that happened this time also. Customer is running everything on premise using Windows 2012 server and now a decision was made to migrate to CentOS 8. Decision was also made to do it step by step and we will start with MongoDB first. We should move our MongoDB replica set to new server and use it from our application running on existing server.

Prerequisites:

We run existing MongoDB version 4.2.x configured as replica set with one primary and one secondary both on the same machine. We don’t use any automatic failover scenario, the secondary is used here as backup and data are stored on separated disk.

Our migration plan:

Our goal is to add another secondary replica set member running on new CentOS 8 server, and then switch this secondary to become primary. In this way our SpringBoot application(when configured properly to use our replica set) will use this new primary MongoDB instance automatically for storing data and 2 secondary MongoDB instances become failover and data backup members.

Steps to accomplish our plan:

• as existing replica set is running locally on different ports, configuration is based on localhost host names. We now suppose to add new member from another server so we have to change replica set configuration host names to by fully qualified and resolvable from both servers.
• add new replica set member
• reconfigure replica-set to make this new secondary member to become primary.

a) Prepare existing replica-set for adding a new member — as mentioned earlier we run our replica-set locally and configuration host names refer to localhost now. As new replica-set member is suppose to run on another machine we need all hostnames to become no localhost based. To accomplish this we connect to primary using mongo client and do:

cfg = rs.conf()
cfg.members[0].host = "rs1.ourdomain.com:27017"
cfg.members[1].host = "rs2.ourdomain.com:27018"
rs.reconfig(cfg)

The last command should reconfigure our replica-set without any interruption. For details see change host name from MongoDB documentation.

b) Add new replica-set member — here we need to add member to replica-set configuration first and then install and run it on CentOS 8. To add a new member do following using Mongo client:

rs.add( { host: "rs3.ourdomain.com:27017", priority: 0, votes: 0 } )

Now we install MongoDB instance on CentOS 8 server and configure properly to be part of this existing replica-set. When running, this new instance should become secondary and initial synch should occur so after some time we get all data loaded/synced on our new secondary MongoDB. At this point we should have replica-set running with one primary(rs1.ourdomain.com) and 2 secondary members. (rs2.ourdomain.com and rs3.ourdomain.com) For adding a new member to replica-set see this part of MongoDB official documentation.

c) Reconfigure replica-set so CentOS 8 secondary become primary — to do so we connect MongoDB client to primary server and do following configuration change:

cfg = rs.conf()
cfg.members[0].priority = 0.5
cfg.members[0].votes = 1
cfg.members[1].priority = 0.5
cfg.members[1].votes = 1
cfg.members[2].priority = 1
cfg.members[2].votes = 1
rs.reconfig(cfg)

After reconfiguration is called, replica-set reconfigure based on priority and votes information . We set votes = 1 for all members, former primary and secondary was assigned with priority = 0.5 and new secondary is set priority = 1. That means our CentOS 8 secondary has highest priority and become primary. After reconfiguration it takes 10–12 seconds for replica-set to reconfigure. To see the current status just run rs.status(), something like this should be listed for 3rd member:

{
    "_id" : 2,
    "name" : "rs3.ourdomain.com:27017",
    "health" : 1,
    "state" : 1,
    "stateStr" : "PRIMARY",
    "uptime" : 4524,
    "optime" : {
        "ts" : Timestamp(1605348479, 2),
        "t" : NumberLong(14)
    },
    "optimeDurable" : {
        "ts" : Timestamp(1605348479, 2),
        "t" : NumberLong(14)
    },
    "optimeDate" : ISODate("2020–11–14T10:07:59Z"),
    "optimeDurableDate" : ISODate("2020–11–14T10:07:59Z"),
    "lastHeartbeat" : ISODate("2020–11–14T10:08:01.033Z"),
    "lastHeartbeatRecv" : ISODate("2020–11–14T10:07:59.446Z"),
    "pingMs" : NumberLong(0),
    "lastHeartbeatMessage" : "",
    "syncingTo" : "",
    "syncSourceHost" : "",
    "syncSourceId" : -1,
    "infoMessage" : "",
    "electionTime" : Timestamp(1605348479, 1),
    "electionDate" : ISODate("2020–11–14T10:07:59Z"),
    "configVersion" : 353722
}

To set primary back to former MongoDB instance simply set proper cfg.members[?].priority so that assumed primary has 1 and others 0.5 and reconfigure. Wait 10–12 seconds and it should be done.

You can find some details here in MongoDB official documentation for how to force member to become primary.

Details on how to configure Spring Data for MongoDB to use multiple replica-set instances can be found here or for example here.

Next time we will look at how to migrate ElasticSearch to new server using its clustering option.