filtered pull replication for P2P/CBLListener

[Sören Havemester]

Is there any way to setup a filtered pull replication for P2P use? Is there something to be set up with CBLListener like a design/filter doc as in CouchDB?

Thanks,

Søren

[jam...@couchbase.com]

Yes you can use a filter function on a pull or push replication.

[Sören Havemester]

I've got two clients doing p2p e.g. syncing to the each others CBLListener. Setting up a filtered push or an unfiltered pull replications is easy and works fine.

But how and where do i setup the fiter for a pull? The filter should run on the "server" e.g. the CBLListener as far as I understand it.

[Jens Alfke]

On May 15, 2017, at 3:21 AM, Sören Havemester <res...@gmail.com> wrote:

But how and where do i setup the fiter for a pull? The filter should run on the "server" e.g. the CBLListener as far as I understand it.

Correct. On the “server” side, register a database filter function under a specific name. Then on the “client” side, set that same name as the filter property on the pull replication.

—Jens

[Sören Havemester]

I still can't make the filtered pull replication work - but everything seams to work with an HTTP client like Paw.

The response from CBLListener when running this code is:  {"status":24576,"error":"server error"} followed by {"ok":true} (Monitored via Charles). But there is no response containing the changes feed.

When i run the same request via Paw I get the correct answer form the CBLListener - {"results":[{"seq":371,"id":"-8K3Kn-aKp9amRCUIzijVB7","changes":[{"rev":"5-af054342256c93740dbd6e4f644cd6d5"}]}, ...

Running my app without the filter works great. Either via HTTPS and pinned certs with or without authentication - all's fine. Turning on the filter kills everything. 

Any idea? 

Thanks,

Sören

I construct a simple filter

         database?.setFilterNamed("sharedByProject", as: {

            (revision, params) -> Bool in

                let isShared = revision["isShared"] as? Bool ?? false

                return isShared

         })

the kick off a CBLListener:

            listener = CBLListener(manager: manager, port: 55554)

            listener.readOnly = true

            listener.requiresAuth = false

            listener.setBonjourName(Host.current().localizedName, type: "_pull-sync._tcp.")

            listener.txtRecordDictionary = txtDictionary

            

            do {

                try listener.start()

            } catch {

               ...

and after discovering another client in the network start a replication:

                   ...

        let components = NSURLComponents()

        components.scheme = "http"

        components.host = host

        components.port = port

        components.path = "/pulldatabase"

        

        let pull = database!.createPullReplication(components.url!)

        pull.continuous = false

        pull.filter = "sharedByProject"

        

        pull.start()

[Jens Alfke]

On May 24, 2017, at 6:51 AM, Sören Havemester <res...@gmail.com> wrote:

The response from CBLListener when running this code is:  {"status":24576,"error":"server error"} followed by {"ok":true} (Monitored via Charles). But there is no response containing the changes feed.

That’s strange, especially because 24576 isn’t a valid status code (though it is a round number in hex, 0x6000). Is there anything logged on the ‘server’ peer at the time?

When i run the same request via Paw I get the correct answer form the CBLListener - {"results":[{"seq":371,"id":"-8K3Kn-aKp9amRCUIzijVB7","changes":[{"rev":"5-af054342256c93740dbd6e4f644cd6d5"}]}, …

Could you show the HTTP request? I just had a look through the code and it should work; it sounds like it’s running into some bug while doing the filtering.

—Jens

[Sören Havemester]

Here is the request and the logs from the 'server' and 'client'

The Request is

http://Dackel.local.:55554/pulldatabase/_changes?feed=normal&heartbeat=300000&style=all_docs&filter=sharedByProject

complete by Charles:

POST /pulldatabase/_changes?feed=normal&heartbeat=300000&style=all_docs&filter=sharedByProject HTTP/1.1

Host: Dackel.local.:55554

Content-Type: application/json

User-Agent: CouchbaseLite/1.3 (Mac OS X 1.3.1/b9a1827f4bc22418501fd7f0d5b977a5ecfa7e37)

Content-Length: 107

Accept-Encoding: gzip

Connection: close

{"feed":"normal","heartbeat":300000,"style":"all_docs","filter":"sharedByProject","accept_encoding":"gzip"}

The server loges this (with "ListenerVerbose") 

08:20:09.302‖ Listener: POST /pulldatabase/_changes?feed=normal&heartbeat=300000&style=all_docs&filter=sharedByProject {+107}

08:20:09.302‖ Listener: Response[POST /pulldatabase/_changes]: Starting...

08:20:09.302‖ Listener: Response[POST /pulldatabase/_changes]: Returning from -init

08:20:09.303‖ Listener: Response[POST /pulldatabase/_changes] answers delayResponseHeaders=1

08:20:09.304‖ Listener:     Response[POST /pulldatabase/_changes] --> 1

08:20:09.304‖ Listener: Response[POST /pulldatabase/_changes] adding 11 bytes

08:20:09.304‖ Listener: Response[POST /pulldatabase/_changes] Finished!

08:20:09.304‖ Listener: Response[POST /pulldatabase/_changes] answers delayResponseHeaders=0

08:20:09.304‖ Listener: Response[POST /pulldatabase/_changes] answers isChunked=0

08:20:09.304‖ Listener: Response[POST /pulldatabase/_changes] answers status=1

08:20:09.305‖ Listener: Response[POST /pulldatabase/_changes] answers httpHeaders={2 headers}

08:20:09.305‖ Listener: Response[POST /pulldatabase/_changes] sending 11 bytes (of 524288 requested)

08:20:09.305‖ Listener: Response[POST /pulldatabase/_changes] answers isDone=1

08:20:09.305‖ Listener: Response[POST /pulldatabase/_changes] sending nil bytes

The client logs this (with "SyncVerbose")

08:30:20.559| Sync: CBLReplication[from http://Dackel.local.:55554/pulldatabase]: offline, progress = 0 / 0, err: (null)

08:30:20.559| Sync: CBLReplication[from http://Dackel.local.:55554/pulldatabase]: offline, progress = 0 / 0, err: (null)

08:30:20.559‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase] STARTING ...

08:30:20.569‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: Going online

08:30:20.569‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: No local checkpoint; not getting remote one

08:30:20.569‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase] Progress: set active = 1

08:30:20.569‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: postProgressChanged (0/0, active=1 (batch=0, net=1), lastSeq=(null), online=1, error=(null))

08:30:20.569‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase] starting ChangeTracker: mode=0, since=(null)

08:30:20.570‖ Sync: CBLSocketChangeTracker[0x6080001a23e0 pulldatabase]: POST //Dackel.local.:55554/pulldatabase/_changes?feed=normal&heartbeat=300000&style=all_docs&filter=sharedByProject

08:30:20.570‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: postProgressChanged (0/0, active=1 (batch=0, net=2), lastSeq=(null), online=1, error=(null))

08:30:20.570| Sync: CBLReplication[from http://Dackel.local.:55554/pulldatabase]: active, progress = 0 / 0, err: (null)

08:30:20.571| Sync: CBLReplication[from http://Dackel.local.:55554/pulldatabase]: active, progress = 0 / 0, err: (null)

08:30:20.584‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: Server is CouchbaseLite 1.3.1 (build 6)

08:30:20.584‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: Caught up with changes!

08:30:20.585‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: ChangeTracker stopped; error=(null)

08:30:20.585‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase] Progress: set active = 0

08:30:20.585‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase]: postProgressChanged (0/0, active=0 (batch=0, net=0), lastSeq=(null), online=1, error=(null))

08:30:20.585‖ Sync: CBLRestPuller[http://Dackel.local.:55554/pulldatabase] STOPPED

08:30:20.585‖ Replication: CBLRestPuller[http://Dackel.local.:55554/pulldatabase] took 0.016 sec; error=(null)

08:30:20.585| Sync: CBLReplication[from http://Dackel.local.:55554/pulldatabase]: stopped, progress = 0 / 0, err: (null)

08:30:20.585| Sync: CBLReplication[from http://Dackel.local.:55554/pulldatabase]: stopped, progress = 0 / 0, err: (null)

Thanks, Sören