MongoDB 3.2 – setShardVersion failed, stale config on lazy receive

MongoDB 3.2 – setShardVersion failed, stale config on lazy receive

While we were hacking the config servers today, we hit multiple errors.

  • MongoDB version 3.2.11
  • 2 shards
  • 3 replica set per shards

Any find() or aggregation query against mongoS

2016-11-24T15:03:31.699+0000 E QUERY [thread1] Error: command failed: {
 "code" : 10429,
 "ok" : 0,
 "errmsg" : "setShardVersion failed shard: s1:s1./hostnames7 { oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId('000000000000000000000000'), ns: \"collections\", version: Timestamp 0|0, versionEpoch: ObjectId('000000000000000000000000'), globalVersion: Timestamp 1000|0, globalVersionEpoch: ObjectId('5836fa3761234567600eebbf'), reloadConfig: true, ok: 0.0, errmsg:
\"could not refresh metadata for collections with requested shard version 0|0||000000000000000000000000, stored shard version is 1|0||1234fa37631...\" }"
} : aggregate failed :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
doassert@src/mongo/shell/assert.js:16:14
assert.commandWorked@src/mongo/shell/assert.js:290:5
DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1312:5
@(shell):1:1

db.printCollectionStats() against mongoS

error: {
 "$err" : "stale config on lazy receive :: caused by :: $err: \"[myzips.zips] shard version not ok in Client::Context: this shard contains versioned chunks for myzips.zips, but no version set in request ( ns : myzips.zip...\" ( ns : myzips.zips, received : 0|0||000000000000000000000000, wanted : 1|1||50fdd55b14faa2aa46422a7a, recv )",
 "code" : 9996
} at src/mongo/shell/query.js:128

Root cause:

  • config servers have new updated metadata about the sharded cluster that is not updated in both mongoSs and mongoDs (Yes, despite MongoDB official documentation, mongoDs does have a cache copy of your config servers metadata

Solution as below:

  1. Restart mongoSs
  2. Restart all your shards mongoDs

Regards,
Wei Shan

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s