While swapping the store that is build from Hadoop, I can see it
copying the data onto the /tmp drive, and it says it was successful,
but the index doesn't seem to get onto the voldemort read-only store,
which still remains with index and data files of size 0.
I know I must have a bad config somewhere, but can't locate it. Any ideas?
The following shows the swap store, the empty data index directory and
also the temp directory with the index
Many thanks for any hints
Tim
[root@ip-10-244-181-204 voldemort]# $VOLDEMORT_HOME/bin/swap-store.sh
--file hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD
--name=density-tiles --cluster=config/cluster.xml
09/06/20 15:24:26 INFO readonly.StoreSwapper: Invoking fetch for node
0 for hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-0
09/06/20 15:24:26 INFO readonly.StoreSwapper: Invoking fetch for node
2 for hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-2
09/06/20 15:24:26 INFO readonly.StoreSwapper: Invoking fetch for node
3 for hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-3
09/06/20 15:24:26 INFO readonly.StoreSwapper: Invoking fetch for node
1 for hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-1
09/06/20 15:24:26 INFO readonly.StoreSwapper: Invoking fetch for node
4 for hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-4
09/06/20 15:24:27 INFO gui.ReadOnlyStoreManagementServlet: Executing
fetch of hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-0
09/06/20 15:24:27 INFO fetcher.HdfsFetcher: Starting copy of
hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-0/0.data
to /tmp/hdfs-fetcher/hdfs-fetcher/node-0/0.data
09/06/20 15:24:27 INFO readonly.StoreSwapper: Fetch succeeded on node 3
09/06/20 15:24:27 INFO readonly.StoreSwapper: Fetch succeeded on node 2
09/06/20 15:24:27 INFO fetcher.HdfsFetcher: Completed copy of
hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-0/0.data
to /tmp/hdfs-fetcher/hdfs-fetcher/node-0/0.data
09/06/20 15:24:27 INFO readonly.StoreSwapper: Fetch succeeded on node 1
09/06/20 15:24:27 INFO fetcher.HdfsFetcher: Starting copy of
hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-0/0.index
to /tmp/hdfs-fetcher/hdfs-fetcher/node-0/0.index
09/06/20 15:24:27 INFO readonly.StoreSwapper: Fetch succeeded on node 4
09/06/20 15:24:27 INFO fetcher.HdfsFetcher: Completed copy of
hdfs://ip-10-244-181-204.ec2.internal:9000/user/root/voldemort/PD/node-0/0.index
to /tmp/hdfs-fetcher/hdfs-fetcher/node-0/0.index
09/06/20 15:24:27 INFO gui.ReadOnlyStoreManagementServlet: Fetch complete.
09/06/20 15:24:27 INFO readonly.StoreSwapper: Fetch succeeded on node 0
09/06/20 15:24:27 INFO readonly.StoreSwapper: Attempting swap for node
0 dir = /tmp/hdfs-fetcher/hdfs-fetcher/node-0
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Swapping files
for store 'density-tiles' from /tmp/hdfs-fetcher/hdfs-fetcher/node-0
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Acquiring write
lock on 'density-tiles':
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Renaming data
and index files for 'density-tiles':
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Setting primary
files for store 'density-tiles' to
/tmp/hdfs-fetcher/hdfs-fetcher/node-0
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Rolling back
store 'density-tiles' to version 1.
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Rollback
operation completed on 'density-tiles', releasing lock.
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Swap operation
completed on 'density-tiles', releasing lock.
09/06/20 15:24:27 INFO readonly.StoreSwapper: Swap succeeded for node 0
09/06/20 15:24:27 INFO readonly.StoreSwapper: Attempting swap for node
1 dir = /tmp/hdfs-fetcher/hdfs-fetcher/node-1
09/06/20 15:24:27 INFO readonly.StoreSwapper: Swap succeeded for node 1
09/06/20 15:24:27 INFO readonly.StoreSwapper: Attempting swap for node
2 dir = /tmp/hdfs-fetcher/hdfs-fetcher/node-2
09/06/20 15:24:27 INFO readonly.StoreSwapper: Swap succeeded for node 2
09/06/20 15:24:27 INFO readonly.StoreSwapper: Attempting swap for node
3 dir = /tmp/hdfs-fetcher/hdfs-fetcher/node-3
09/06/20 15:24:27 INFO readonly.StoreSwapper: Swap succeeded for node 3
09/06/20 15:24:27 INFO readonly.StoreSwapper: Attempting swap for node
4 dir = /tmp/hdfs-fetcher/hdfs-fetcher/node-4
09/06/20 15:24:27 INFO readonly.StoreSwapper: Swap succeeded for node 4
09/06/20 15:24:27 INFO readonly.StoreSwapper: Swap succeeded on all
nodes in 0 seconds.
[root@ip-10-244-181-204 voldemort]#
[root@ip-10-244-181-204 voldemort]#
[root@ip-10-244-181-204 voldemort]#
[root@ip-10-244-181-204 voldemort]#
[root@ip-10-244-181-204 voldemort]# cd data/read-only/density-tiles/version-0/
[root@ip-10-244-181-204 version-0]# ls -lh
total 0
-rw-r--r-- 1 root root 0 Jun 20 15:21 0.data
-rw-r--r-- 1 root root 0 Jun 20 15:21 0.index
[root@ip-10-244-181-204 version-0]#
[root@ip-10-244-181-204 version-0]#
[root@ip-10-244-181-204 version-0]#
[root@ip-10-244-181-204 version-0]# ls -lh
/tmp/hdfs-fetcher/hdfs-fetcher/node-0
total 41M
-rw-r--r-- 1 root root 30M Jun 20 15:24 0.data
-rw-r--r-- 1 root root 11M Jun 20 15:24 0.index
[root@ip-10-244-181-204 version-0]#
ReadOnlyStorageEngine: Renaming data
and index files for 'density-tiles':
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Setting primary
files for store 'density-tiles' to
/tmp/hdfs-fetcher/hdfs-fetcher/node-0
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Rolling back
store 'density-tiles' to version 1.
09/06/20 15:24:27 INFO readonly.ReadOnlyStorageEngine: Rollback
operation completed on 'density-tiles', releasing lock.