replication policy ran out of suggestions

[William Francis]

I'm attempting to set up a two datacenter mogile cluster and migrate my
files from my old one to the new one. The new one has 5 machines in
each data center with a mysql master/master setup where each set of
machines points to their local mysql master.

I'm running into a problem during my test imports, however. I've
installed (I believe) correctly the Network and zonelocal plugins and
have added the config stuff as shown below. However, when I !watch on
one of my trackers, I see lots of replication errors in regards to
policy. My file_to_replicate table will list a bunch of fids(in the
tens out of a 11K files test import) that will slowly increase their
failcount. If I UPDATE nexttry to 0 it seems they will flush out
eventually, but it seems strange that when I force them to retry they
would work, but have errors at other times.

:: [replicate(24322)] policy_no_suggestions: replication policy ran out
of suggestions for us replicating fid 37162
:: Watchdog killing worker 32457 (replicate)
:: Child 32457 (replicate) died: 9 (expected)

This is with 2.30 built into debian packages from the CPAN sources.


my server_settings table has this:

| network_zones | x,y |
| zone_x | 10.0.128.0/22 |
| zone_y | 10.2.128.0/22 |

my class table:

| 1 | 1 | three | 3 | MultipleNetworks() |
| 1 | 2 | four | 4 | MultipleNetworks() |

mogilefsd looks like

db_dsn = DBI:mysql:mogilefs:w.x.y.z
listen = hostip:7001
db_user = xxx
db_pass = yyy
conf_port = 7001
listener_jobs = 10
delete_jobs = 1
replicate_jobs = 5
mog_root = /var/lib/mogdata
reaper_jobs = 1
plugin ZoneLocal

xxx:/etc/mogilefs# dpkg-query -L mogilefs-plugin-network
...
/usr/share/perl5/MogileFS/ReplicationPolicy
/usr/share/perl5/MogileFS/ReplicationPolicy/MultipleNetworks.pm
/usr/share/perl5/MogileFS/ReplicationPolicy/HostsPerNetwork.pm
/usr/share/perl5/MogileFS/Network.pm

xxx:/etc/mogilefs# dpkg-query -L mogilefs-plugin-zonelocal
...
/usr/share/perl5/MogileFS/Plugin
/usr/share/perl5/MogileFS/Plugin/ZoneLocal.pm

[dormando]

Hey,

At a glance, and given the description, it seems that you're pointing the
trackers in each datacenter, to different DB's?

Trackers must *always* point to he same master database. They can use
slave DBs for lookups and such, but if they don't point to the same DB,
its locking and some transactional stuff will collide and fail.

So the tracker in the standby/secondary DC should be talking over the
backhaul to the master DB..

The trackers in the differnet DC's should be returning localized paths,
and be replicating off of localized paths, so they at least attempt to not
go over the backhaul.

Let me know if you need any further help with this.. Maybe we could modify
the slave DB setu in some trivial way so the tracker can prefer a "local"
database for those reads, minimizing what's not local.

-Dormando