MongoDB Upgrade From 2.4.x to 2.6.x
Recently, in our environment, we undertook an exercise to test the new version of MongoDB and look n feel of the upgrade process.
Our env has following components -
2 MongoS processes.
3 config servers.
2 shard servers.
All of the above was using version 2.4.8, which we intend to upgrade to latest version of 2.6.5.
To begin the upgrade procedure, connect a 2.6 mongo shell to your MongoDB 2.4 mongos or mongod and run the db.upgradeCheckAllDBs() to check your data set for compatibility. This is a preliminary automated check. Assess and resolve all issues identified by db.upgradeCheckAllDBs().
mongos> use admin
switched to db admin
mongos> db.upgradeCheckAllDBs()
Checking database sharding
Checking collection sharding.col1
37055 documents processed
76072 documents processed
......
.....
.....
Checking database admin
Checking collection admin.system.indexes
Checking collection admin.system.users
Everything is ready for the upgrade!
true
-- Now one has to Stop/Disable the Balancer, so that there are no cluster related operations.
mongos> sh.isBalancerRunning()
true
mongos> sh.stopBalancer()
Waiting for active hosts...
Waiting for the balancer lock...
Waiting again for active hosts after balancer is off...
mongos> sh.isBalancerRunning()
false
There is specific order of the upgrade when you wish to upgrade the sharded cluster.
1. Mongos upgrade
2. config server upgrade
3. shard server upgrade.
1. Upgrade mongos.
-- unzip the binaries of 2.6 mongos and use it to start new mongos of 2.6 version
-- stop mongos
root 2988 1941 0 16:24 pts/1 00:00:29 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos1.log --port 30002
root@INPUNPCLX:~# kill 2988
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# /usr/bin/mongos --version
MongoS version 2.4.8 starting: pid=4471 port=27017 64-bit host=INPUNPCLX (--help for usage)
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# mv /usr/bin/mongos /usr/bin/mongos.bak
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# scp ./mongos /usr/bin/mongos
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# which mongos
/usr/bin/mongos
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# /usr/bin/mongos --version
MongoS version 2.6.5 starting: pid=4491 port=27017 64-bit host=INPUNPCLX (--help for usage)
Start a single 2.6 mongos instance with the configDB pointing to the cluster’s config servers and with the --upgrade option.
To run a mongos with the --upgrade option, you can upgrade an existing mongos instance to 2.6, or if you need to avoid reconfiguring a production mongos instance, you can use a new 2.6 mongos that can reach all the config servers.
To upgrade the meta data, run:
mongos --configdb <configDB string> --logpath xxxx --upgrade --port nnnnn
-- to start a new mongos
/usr/bin/mongos --configdb "localhost:19001,localhost:19002,localhost:19003" --upgrade --logpath /db/mongos2.log --port 30002 &
7ac7218077eb4215", server: "INPUNPCLX", clientAddr: "N/A", time: new Date(1417182378230), what: "finished upgrade of config database", ns: "config.version", details: { from: 4, to: 5 } }
2014-11-28T19:16:18.248+0530 [mongosMain] upgrade of config server to v5 successful
2014-11-28T19:16:18.385+0530 [mongosMain] distributed lock 'configUpgrade/INPUNPCLX:30002:1417182377:1804289383' unlocked.
2014-11-28T19:16:18.385+0530 [mongosMain] Config database is at version v5
The mongos will exit upon completion of the --upgrade process.
Upgrade the remaining mongos instances to v2.6.
Upgrade and restart without the --upgrade option the other mongos instances in the sharded cluster. After upgrading all the mongos, see Complete Sharded Cluster Upgrade for information on upgrading the other cluster components
root@INPUNPCLX:~# ps -ef | grep mongos
root 2925 1 1 16:23 pts/1 00:02:38 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos1.log --port 30001
root 4596 4102 0 19:17 pts/5 00:00:00 grep --color=auto mongos
root@INPUNPCLX:~# kill 2925
root@INPUNPCLX:~# /usr/bin/mongos --configdb "localhost:19001,localhost:19002,localhost:19003" --logpath /db/mongos1.log --port 30001 &
[1] 4598
root@INPUNPCLX:~# 2014-11-28T19:17:56.370+0530 log file "/db/mongos1.log"
-- start the another mongos
root@INPUNPCLX:~# /usr/bin/mongos --configdb "localhost:19001,localhost:19002,localhost:19003" --logpath /db/mongos2.log --port 30002 &
[2] 4645
root@INPUNPCLX:~# 2014-11-28T19:18:19.123+0530 log file "/db/mongos2.log" exists; moved to "/db/mongos2.log.2014-11-28T13-48-19".
root@INPUNPCLX:~# ps -ef | grep mongos
root 4598 4102 0 19:17 pts/5 00:00:00 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos1.log --port 30001
root 4645 4102 1 19:18 pts/5 00:00:00 /usr/bin/mongos --configdb localhost:19001,localhost:19002,localhost:19003 --logpath /db/mongos2.log --port 30002
-- Do a test of connection, with new mongos
root@INPUNPCLX:~# mongo --port 30001 admin -u hardik -p xxxx
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:30001/admin
2. Upgrade config db
Upgrade all 3 mongod config server instances, leaving the first system in the mongos --configdb argument to upgrade last.
--For example, on Linux, run 2.4 mongod with the --shutdown option as follows:
mongod --dbpath /var/mongod/data --shutdown
-- Start the new mongod instance. Ensure you start the 2.6 mongod with the same dbPath:
mongod --dbpath /var/mongod/data
root 2889 1 0 16:23 ? 00:01:39 /usr/bin/mongod --configsvr --config /etc/config3.conf
root 2893 1 0 16:23 ? 00:01:34 /usr/bin/mongod --configsvr --config /etc/config2.conf
root 4722 4102 0 19:20 pts/5 00:00:00 grep --color=auto mongod
root@INPUNPCLX:~# kill 2889
root@INPUNPCLX:~# /usr/bin/mongod --version
db version v2.4.8
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# ./mongod --version
db version v2.6.5
root@INPUNPCLX:~/mongodb-linux-x86_64-2.6.5/bin# scp ./mongod /usr/bin/mongod
root@INPUNPCLX:~/# mongod --version
db version v2.6.5
root@INPUNPCLX:/db/mongo01/data# /usr/bin/mongod --configsvr --config /etc/config3.conf &
root@INPUNPCLX:/db/mongo01/data# ps -ef | grep mongod
root 5377 1 0 19:36 ? 00:00:00 /usr/bin/mongod --configsvr --config /etc/config3.conf
root@INPUNPCLX:/db/mongo01/data# mongo --port 19003
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19003/test
configsvr> show dbs
admin 0.063GB
config 0.063GB
local 0.031GB
configsvr> use config
switched to db config
configsvr> db.version()
2.6.5
-- above displays the config server version
WHEN YOU CONNECT THE OTHER CONFIG SERVER YOU WILL SEE THE OLD VERSION
root@INPUNPCLX:/db/mongo01/data# mongo --port 19002
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19002/test
configsvr> use config
switched to db config
configsvr> db.version()
2.4.8
-- shutdown config2 server
root@INPUNPCLX:/db/mongo01/data# mongod --dbpath /db/config2/data --shutdown
killing process with pid: 2893
root@INPUNPCLX:/db/mongo01/data# /usr/bin/mongod --configsvr --config /etc/config2.conf &
[3] 5522
root@INPUNPCLX:/db/mongo01/data# about to fork child process, waiting until server is ready for connections.
forked process: 5524
child process started successfully, parent exiting
[3]+ Done /usr/bin/mongod --configsvr --config /etc/config2.conf
root@INPUNPCLX:/db/mongo01/data# ps -ef | grep mongod
root 5524 1 1 19:40 ? 00:00:00 /usr/bin/mongod --configsvr --config /etc/config2.conf
root@INPUNPCLX:/db/mongo01/data# mongo --port 19002
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19002/test
configsvr> use config
switched to db config
configsvr> db.version()
2.6.5
-- Shut down and start up the last config server
root@INPUNPCLX:/db/mongo01/data# mongod --dbpath /db/config1/data --shutdown
killing process with pid: 2884
root@INPUNPCLX:~# /usr/bin/mongod --configsvr --config /etc/config1.conf &
[3] 6172
root@INPUNPCLX:~# about to fork child process, waiting until server is ready for connections.
forked process: 6174
child process started successfully, parent exiting
[3]+ Done mongod --configsvr --config /etc/config1.conf
root@INPUNPCLX:~# ps -ef | grep mongod
root 2852 1 3 16:23 ? 00:07:28 /usr/bin/mongod --shardsvr --config /etc/mongo02.conf
root 2856 1 1 16:23 ? 00:03:17 /usr/bin/mongod --shardsvr --config /etc/mongo01.conf
root 5377 1 0 19:36 ? 00:00:01 /usr/bin/mongod --configsvr --config /etc/config3.conf
root 5524 1 0 19:40 ? 00:00:01 /usr/bin/mongod --configsvr --config /etc/config2.conf
root 6174 1 0 19:49 ? 00:00:00 /usr/bin/mongod --configsvr --config /etc/config1.conf
root@INPUNPCLX:~# mongo --port 19001
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:19001/test
configsvr> use config
switched to db config
configsvr> db.version()
2.6.5
3. Upgrade shard servers
Same way upgrade the shard servers same way you do configdb..
WARNING
Do not upgrade mongod instances until after you have upgraded all mongos instances.
-- check the shard server version
root@INPUNPCLX:~# mongo --port 20001
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:20001/test
> use config
switched to db config
> db.version()
2.4.8
-- kill the shard server
root 2856 1 1 16:23 ? 00:03:18 /usr/bin/mongod --shardsvr --config /etc/mongo01.conf
root@INPUNPCLX:~# kill 2856
root@INPUNPCLX:~# /usr/bin/mongod --shardsvr --config /etc/mongo01.conf &
child process started successfully, parent exiting
root 6269 1 0 19:55 ? 00:00:00 /usr/bin/mongod --shardsvr --config /etc/mongo01.conf
root@INPUNPCLX:~# mongo --port 20001
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:20001/test
Server has startup warnings:
2014-11-28T19:55:03.847+0530 ** WARNING: --rest is specified without --httpinterface,
2014-11-28T19:55:03.847+0530 ** enabling http interface
> use config
switched to db config
> db.version()
2.6.5
root@INPUNPCLX:~# ps -ef | grep mongod
root 2852 1 3 16:23 ? 00:07:29 /usr/bin/mongod --shardsvr --config /etc/mongo02.conf
root@INPUNPCLX:~# kill 2852
root@INPUNPCLX:~# /usr/bin/mongod --shardsvr --config /etc/mongo02.conf &
[3] 6331
root@INPUNPCLX:~# 2014-11-28T19:57:58.410+0530 ** WARNING: --rest is specified without --httpinterface,
2014-11-28T19:57:58.410+0530 ** enabling http interface
about to fork child process, waiting until server is ready for connections.
forked process: 6333
child process started successfully, parent exiting
root@INPUNPCLX:~# mongo --port 20002
MongoDB shell version: 2.6.5
connecting to: 127.0.0.1:20002/test
Server has startup warnings:
2014-11-28T19:57:58.410+0530 ** WARNING: --rest is specified without --httpinterface,
2014-11-28T19:57:58.410+0530 ** enabling http interface
> use config
switched to db config
> db.version()
2.6.5
AS you can see now that even the shard servers are also upgraded. So one can use this method to upgrade the sharded clusters. Here I did not installed the Mongo RPM but used the tar'red version and copied the executables to /usr/bin myself. This process was adopted to keep control with DBA rather than to system to make trouble shooting easy.
The only remaining bit that was not covered in post is the upgrade of Auth Schema. Which I will cover in separate post.
Happy Learning!!
