If a shard it's becoming too big you cannot simply add a new shard to a collection, but the correct procedure is to split the shard in two parts. The original shard remains the same and Solr creates two new shards each of which contains a part of the documents of the original shard.
The two new shards will start to receive the requests directed to the original shard, and the latter must be deactivated manually. If needed you can move the new shards to other cluster nodes.
The command to split a shard into two parts is:
Here "shard1" is the id of the shard to be split and "examplecoll" is the name of the collection to which the shard belongs.
Note the async parameter: it's better to execute this command in an asynchronous way because it can last for a long time depending on the number of documents contained in the shard. To check the status of the task there's the command:
where "request-id" is the id of the asynchronous request returned by the split command.
When the shard has been split you should unload the old shard on every replica where it's hosted (here I suppose that there are only two replicas):
The UNLOAD command doesn't remove the data from disk: it should be removed by hand.
When you split a shard you often need also to move part of the documents (and workload) on some other cluster nodes. Up to now there's no automated way to accomplish the task, so you have to execute the commands by hand.
To move a shard replica on another node there are two steps:
1) create a new replica of the shard on the destination node:
Here the shard to be moved has id "shard1_0" because it's one of the two pieces into which the "shard1" has been split. The name of the core that will be the new replica of "shard1_0" is convoluted because it follows Solr naming pattern, but you can choose whatever you like.
At this point you must check with a query that the shard has been fully copied to the new replica. Then you can go ahead with the next step:
2) switch off the old replica on the source node:
Note that the node on which we created the replica listens on the port 9900 and it's different from the node on which the old replica has been deactivated (which listens to the port 7574).
These two steps should be executed for each replica of the shard you are moving.
The two new shards will start to receive the requests directed to the original shard, and the latter must be deactivated manually. If needed you can move the new shards to other cluster nodes.
Split the shard in two parts
The command to split a shard into two parts is:
curl 'http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=examplecoll&shard=shard1&async=true'
Here "shard1" is the id of the shard to be split and "examplecoll" is the name of the collection to which the shard belongs.
Note the async parameter: it's better to execute this command in an asynchronous way because it can last for a long time depending on the number of documents contained in the shard. To check the status of the task there's the command:
curl 'http://localhost:8983/solr/admin/collections?action=REQUESTSTATUS
&requestid=request-id'
where "request-id" is the id of the asynchronous request returned by the split command.
When the shard has been split you should unload the old shard on every replica where it's hosted (here I suppose that there are only two replicas):
curl 'http://localhost:7574/solr/admin/cores?action=UNLOAD
&core=examplecoll_shard1_replica1'
curl 'http://localhost:7500/solr/admin/cores?action=UNLOAD
&core=examplecoll_shard1_replica2'
The UNLOAD command doesn't remove the data from disk: it should be removed by hand.
Move a shard to another node
When you split a shard you often need also to move part of the documents (and workload) on some other cluster nodes. Up to now there's no automated way to accomplish the task, so you have to execute the commands by hand.
To move a shard replica on another node there are two steps:
1) create a new replica of the shard on the destination node:
curl 'http://localhost:9900/solr/admin/cores?action=CREATE
&name=examplecoll_shard1_0_replica1&collection=examplecoll&shard=shard1_0'
Here the shard to be moved has id "shard1_0" because it's one of the two pieces into which the "shard1" has been split. The name of the core that will be the new replica of "shard1_0" is convoluted because it follows Solr naming pattern, but you can choose whatever you like.
At this point you must check with a query that the shard has been fully copied to the new replica. Then you can go ahead with the next step:
2) switch off the old replica on the source node:
curl 'http://localhost:7574/solr/admin/cores?action=UNLOAD
&core=examplecoll_shard1_0_replica1'
Note that the node on which we created the replica listens on the port 9900 and it's different from the node on which the old replica has been deactivated (which listens to the port 7574).
These two steps should be executed for each replica of the shard you are moving.
Comments
Post a Comment