MongoDB 3.2 Sharded Cluster – rs.reconfig() does not work

MongoDB 3.2 Sharded Cluster – rs.reconfig() does not work

Following this document to reconfigure a sharded replica set does not work. The replica set seems to be confused. rs.status() show the correct output but db.serverStatus() show otherwise.

rs:PRIMARY> rs.status()
{
"set" : "rs",
"date" : ISODate("2017-01-18T03:05:06.202Z"),
"myState" : 1,
"term" : NumberLong(20),
"configsvr" : true,
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 1,
"name" : "node1:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 439,
"optime" : {
"ts" : Timestamp(1484708697, 1),
"t" : NumberLong(20)
},
"optimeDate" : ISODate("2017-01-18T03:04:57Z"),
"infoMessage" : "could not find member to sync from",
"electionTime" : Timestamp(1484708696, 1),
"electionDate" : ISODate("2017-01-18T03:04:56Z"),
"configVersion" : 83051,
"self" : true
}
],
"ok" : 1
}
rs:PRIMARY> db.serverStatus()
{
"host" : "node1:27017",
"process" : "mongod",
"repl" : {
"hosts" : [
"node1:27017"
],
"setName" : "rs",
"setVersion" : 83051,
"ismaster" : false,
"secondary" : true,
"primary" : "node1:27017",
"me" : "node1:27017",
"electionId" : ObjectId("7fffffff0000000000000014"),
"rbid" : 398723451

Error log on mongos

2017-01-17T21:38:40.143Z W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set rs0
2017-01-17T21:38:40.144Z W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set rs1
2017-01-17T21:38:50.440Z W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set rs0
2017-01-17T21:39:00.762Z W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set rs1
2016-12-20T21:18:27.060Z W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set rs0(config)
2016-12-20T21:18:29.187Z W NETWORK [mongosMain] Failed to connect to , reason: errno:65 No route to host
2016-12-20T21:18:29.188Z W NETWORK [mongosMain] Failed to connect to , reason: errno:65 No route to host
2016-12-20T21:19:44.217Z W NETWORK [mongosMain] Failed to connect to after 5000 milliseconds, giving up.
2016-12-20T21:19:44.218Z W NETWORK [mongosMain] No primary detected for set rs0(config)
2016-12-20T21:19:44.218Z I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set rs0(config)

Error log on the single node replica set mongod process

2017-01-17T23:50:36.601Z I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 
2017-01-17T23:50:36.602Z I NETWORK [initandlisten] waiting for connections on port 27017 ssl 
2017-01-17T23:50:36.602Z I REPL [ReplicationExecutor] New replica set config in use: {...} 
2017-01-17T23:50:36.603Z I REPL [ReplicationExecutor] This node is in the config 
2017-01-17T23:50:36.603Z I REPL [ReplicationExecutor] transition to STARTUP2 
2017-01-17T23:50:36.603Z I REPL [ReplicationExecutor] Starting replication applier threads 
2017-01-17T23:50:36.604Z I REPL [ReplicationExecutor] transition to RECOVERING 
2017-01-17T23:50:36.608Z I REPL [ReplicationExecutor] transition to SECONDARY 
2017-01-17T23:50:36.608Z I REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected 
2017-01-17T23:50:36.608Z I REPL [ReplicationExecutor] dry election run succeeded, running for election 
2017-01-17T23:50:36.608Z I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 38 
2017-01-17T23:50:36.608Z I REPL [ReplicationExecutor] transition to PRIMARY 
2017-01-17T23:50:37.668Z I SHARDING [rsSync] Sharding state recovery process found document { _id: "minOpTimeRecovery", configsvrConnectionString: "rs0/", shardName: "rs0", minOpTime: { ts: Timestamp 1481206005000|6, t: 15 }, minOpTimeUpdaters: 0 } 
2017-01-17T23:50:37.669Z I SHARDING [ShardingState initialization] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers 2017-01-17T23:50:37.690Z I SHARDING [ShardingState initialization] Updating config server connection string to: rs0/ 
2017-01-17T23:50:37.690Z I NETWORK [ShardingState initialization] Starting new replica set monitor for rs0/ 
2017-01-17T23:50:37.690Z I NETWORK [ReplicaSetMonitorWatcher] starting 
2017-01-17T23:50:37.703Z I SHARDING [thread1] creating distributed lock ping thread for process rs0/ (sleeping for 30000ms) 
2017-01-17T23:51:32.666Z I ACCESS [conn2] Successfully authenticated as principal user on admin 
2017-01-17T23:51:52.712Z W NETWORK [replSetDistLockPinger] Failed to connect to after 5000 milliseconds, giving up. 
2017-01-17T23:51:52.712Z W NETWORK [ShardingState initialization] Failed to connect to after 5000 milliseconds, giving up. 
2017-01-17T23:51:52.712Z W NETWORK [ShardingState initialization] No primary detected for set rs0(config) 2017-01-17T23:51:52.712Z W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set rs0(config) 
2017-01-17T23:51:52.712Z I SHARDING [ShardingState initialization] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for rs0(config)

Root Cause:

Because the config database information isn’t updated in the shard members. We need to remove the cache data from  db.system.version in step #6 of this document.

Hope this helps.

Regards,
Wei Shan

Advertisements
  1. Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: