MongoDB Upgrade From 2.4.x to 2.6.x

Recently, in our environment, we undertook an exercise  to test the new version of MongoDB and look n feel of the upgrade process. 

Our env has following components - 

2 MongoS processes.
3 config servers.
2 shard servers. 

All of the above was using version 2.4.8, which we intend to upgrade to latest version of 2.6.5. 

To begin the upgrade procedure, connect a 2.6 mongo shell to your MongoDB 2.4 mongos or mongod and run the db.upgradeCheckAllDBs() to check your data set for compatibility. This is a preliminary automated check. Assess and resolve all issues identified by db.upgradeCheckAllDBs().

mongos> use admin
switched to db admin
mongos> db.upgradeCheckAllDBs()

Checking database sharding

Checking collection sharding.col1
37055 documents processed
76072 documents processed
......
.....
.....
Checking database admin

Checking collection admin.system.indexes

Checking collection admin.system.users
Everything is ready for the upgrade!
true



-- Now one has to  Stop/Disable the Balancer, so that there are no cluster related operations.

mongos> sh.isBalancerRunning()
true
mongos> sh.stopBalancer()
Waiting for active hosts...
Waiting for the balancer lock...
Waiting again for active hosts after balancer is off...
mongos> sh.isBalancerRunning()
false


There is specific order of the upgrade when you wish to upgrade the sharded cluster.
1. Mongos upgrade
2. config server upgrade
3. shard server upgrade. 

1. Upgrade mongos.

-- unzip the binaries of 2.6 mongos and use it to start new mongos of 2.6 version

-- stop mongos 
root      2988  1941  0 16:24 pts/1    00:00:29 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos1.log --port 30002

root@INPUNPCLX:~# kill 2988

root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# /usr/bin/mongos --version
MongoS version 2.4.8 starting: pid=4471 port=27017 64-bit host=INPUNPCLX (--help for usage)

root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# mv /usr/bin/mongos /usr/bin/mongos.bak

root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# scp ./mongos /usr/bin/mongos
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# which mongos
/usr/bin/mongos
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# /usr/bin/mongos --version
MongoS version 2.6.5 starting: pid=4491 port=27017 64-bit host=INPUNPCLX (--help for usage)


Start a single 2.6 mongos instance with the configDB pointing to the cluster’s config servers and with the --upgrade option.

To run a mongos with the --upgrade option, you can upgrade an existing mongos instance to 2.6, or if you need to avoid reconfiguring a production mongos instance, you can use a new 2.6 mongos that can reach all the config servers.

To upgrade the meta data, run:

mongos --configdb <configDB string>  --logpath  xxxx --upgrade --port nnnnn

-- to start a new mongos 
 /usr/bin/mongos --configdb "localhost:19001,localhost:19002,localhost:19003" --upgrade  --logpath /db/mongos2.log --port 30002 &

7ac7218077eb4215", server: "INPUNPCLX", clientAddr: "N/A", time: new Date(1417182378230), what: "finished upgrade of config database", ns: "config.version", details: { from: 4, to: 5 } }
2014-11-28T19:16:18.248+0530 [mongosMain] upgrade of config server to v5 successful
2014-11-28T19:16:18.385+0530 [mongosMain] distributed lock 'configUpgrade/INPUNPCLX:30002:1417182377:1804289383' unlocked.
2014-11-28T19:16:18.385+0530 [mongosMain] Config database is at version v5


The mongos will exit upon completion of the --upgrade process.

Upgrade the remaining mongos instances to v2.6.
Upgrade and restart without the --upgrade option the other mongos instances in the sharded cluster. After upgrading all the mongos, see Complete Sharded Cluster Upgrade for information on upgrading the other cluster components

root@INPUNPCLX:~# ps -ef | grep mongos
root      2925     1  1 16:23 pts/1    00:02:38 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos1.log --port 30001
root      4596  4102  0 19:17 pts/5    00:00:00 grep --color=auto mongos
root@INPUNPCLX:~# kill 2925
root@INPUNPCLX:~# /usr/bin/mongos --configdb "localhost:19001,localhost:19002,localhost:19003" --logpath /db/mongos1.log --port 30001 &
[1] 4598
root@INPUNPCLX:~# 2014-11-28T19:17:56.370+0530 log file "/db/mongos1.log" 

-- start the another mongos 

root@INPUNPCLX:~# /usr/bin/mongos --configdb "localhost:19001,localhost:19002,localhost:19003" --logpath /db/mongos2.log --port 30002 &
[2] 4645
root@INPUNPCLX:~# 2014-11-28T19:18:19.123+0530 log file "/db/mongos2.log" exists; moved to "/db/mongos2.log.2014-11-28T13-48-19".

root@INPUNPCLX:~# ps -ef | grep mongos
root      4598  4102  0 19:17 pts/5    00:00:00 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos1.log --port 30001
root      4645  4102  1 19:18 pts/5    00:00:00 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos2.log --port 30002

-- Do a test of connection, with new mongos 
root@INPUNPCLX:~# mongo --port 30001 admin -u hardik -p xxxx
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:30001/admin

2. Upgrade config db 

Upgrade all 3 mongod config server instances, leaving the first system in the mongos --configdb argument to upgrade last.


--For example, on Linux, run 2.4 mongod with the --shutdown option as follows:
mongod --dbpath /var/mongod/data --shutdown

-- Start the new mongod instance. Ensure you start the 2.6 mongod with the same dbPath:
mongod --dbpath /var/mongod/data

root      2889     1  0 16:23 ?        00:01:39 /usr/bin/mongod --configsvr --config /etc/config3.conf
root      2893     1  0 16:23 ?        00:01:34 /usr/bin/mongod --configsvr --config /etc/config2.conf
root      4722  4102  0 19:20 pts/5    00:00:00 grep --color=auto mongod
root@INPUNPCLX:~# kill 2889

root@INPUNPCLX:~# /usr/bin/mongod --version
db version v2.4.8

root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# ./mongod --version
db version v2.6.5
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# scp ./mongod /usr/bin/mongod

root@INPUNPCLX:~/# mongod --version
db version v2.6.5
root@INPUNPCLX:/db/mongo01/data# /usr/bin/mongod --configsvr --config /etc/config3.conf &

root@INPUNPCLX:/db/mongo01/data# ps -ef | grep mongod

root      5377     1  0 19:36 ?        00:00:00 /usr/bin/mongod --configsvr --config /etc/config3.conf


root@INPUNPCLX:/db/mongo01/data# mongo --port 19003
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19003/test
configsvr> show dbs
admin   0.063GB
config  0.063GB
local   0.031GB
configsvr> use config
switched to db config

configsvr> db.version()
2.6.5
-- above displays the config server version

WHEN YOU CONNECT THE OTHER CONFIG SERVER YOU WILL SEE THE OLD VERSION

root@INPUNPCLX:/db/mongo01/data# mongo --port 19002
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19002/test
configsvr> use config
switched to db config
configsvr> db.version()
2.4.8

-- shutdown config2 server
root@INPUNPCLX:/db/mongo01/data# mongod --dbpath /db/config2/data --shutdown
killing process with pid: 2893

root@INPUNPCLX:/db/mongo01/data# /usr/bin/mongod --configsvr --config /etc/config2.conf &
[3] 5522
root@INPUNPCLX:/db/mongo01/data# about to fork child process, waiting until server is ready for connections.
forked process: 5524
child process started successfully, parent exiting

[3]+  Done                    /usr/bin/mongod --configsvr --config /etc/config2.conf
root@INPUNPCLX:/db/mongo01/data# ps -ef | grep mongod

root      5524     1  1 19:40 ?        00:00:00 /usr/bin/mongod --configsvr --config /etc/config2.conf

root@INPUNPCLX:/db/mongo01/data# mongo --port 19002
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19002/test
configsvr> use config
switched to db config
configsvr> db.version()
2.6.5

-- Shut down and start up the last config server 
root@INPUNPCLX:/db/mongo01/data# mongod --dbpath /db/config1/data --shutdown
killing process with pid: 2884
root@INPUNPCLX:~# /usr/bin/mongod --configsvr --config /etc/config1.conf &
[3] 6172
root@INPUNPCLX:~# about to fork child process, waiting until server is ready for connections.
forked process: 6174
child process started successfully, parent exiting

[3]+  Done                    mongod --configsvr --config /etc/config1.conf


root@INPUNPCLX:~# ps -ef | grep mongod
root      2852     1  3 16:23 ?        00:07:28 /usr/bin/mongod --shardsvr --config /etc/mongo02.conf
root      2856     1  1 16:23 ?        00:03:17 /usr/bin/mongod --shardsvr --config /etc/mongo01.conf
root      5377     1  0 19:36 ?        00:00:01 /usr/bin/mongod --configsvr --config /etc/config3.conf
root      5524     1  0 19:40 ?        00:00:01 /usr/bin/mongod --configsvr --config /etc/config2.conf
root      6174     1  0 19:49 ?        00:00:00 /usr/bin/mongod --configsvr --config /etc/config1.conf

root@INPUNPCLX:~# mongo --port 19001
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19001/test
configsvr> use config
switched to db config
configsvr> db.version()
2.6.5

3. Upgrade shard servers

Same way upgrade the shard servers same way you do configdb..
WARNING
Do not upgrade mongod instances until after you have upgraded all mongos instances.

-- check the shard server version
root@INPUNPCLX:~# mongo --port 20001
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:20001/test
> use config
switched to db config
> db.version()
2.4.8

-- kill the shard server 
root      2856     1  1 16:23 ?        00:03:18 /usr/bin/mongod --shardsvr --config /etc/mongo01.conf

root@INPUNPCLX:~# kill 2856

root@INPUNPCLX:~# /usr/bin/mongod --shardsvr --config /etc/mongo01.conf  &
child process started successfully, parent exiting

root      6269     1  0 19:55 ?        00:00:00 /usr/bin/mongod --shardsvr --config /etc/mongo01.conf

root@INPUNPCLX:~# mongo --port 20001
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:20001/test
Server has startup warnings:
2014-11-28T19:55:03.847+0530 ** WARNING: --rest is specified without --httpinterface,
2014-11-28T19:55:03.847+0530 **          enabling http interface
> use config
switched to db config
> db.version()
2.6.5

root@INPUNPCLX:~# ps -ef | grep mongod
root      2852     1  3 16:23 ?        00:07:29 /usr/bin/mongod --shardsvr --config /etc/mongo02.conf

root@INPUNPCLX:~# kill 2852

root@INPUNPCLX:~#  /usr/bin/mongod --shardsvr --config /etc/mongo02.conf &
[3] 6331
root@INPUNPCLX:~# 2014-11-28T19:57:58.410+0530 ** WARNING: --rest is specified without --httpinterface,
2014-11-28T19:57:58.410+0530 **          enabling http interface
about to fork child process, waiting until server is ready for connections.
forked process: 6333
child process started successfully, parent exiting


root@INPUNPCLX:~# mongo --port 20002
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:20002/test
Server has startup warnings:
2014-11-28T19:57:58.410+0530 ** WARNING: --rest is specified without --httpinterface,
2014-11-28T19:57:58.410+0530 **          enabling http interface
> use config
switched to db config
> db.version()
2.6.5

AS you can see now that even the shard servers are also upgraded. So one can use this method to upgrade the sharded clusters. Here I did not installed the Mongo RPM but used the tar'red version and copied the executables to /usr/bin myself. This process was adopted to keep control with DBA rather than to system to make trouble shooting easy. 

The only remaining bit that was not covered in post is the upgrade of Auth Schema. Which I will cover in separate post. 

Happy Learning!!