Storing logs in Couchbase Server for analyzing

[parvez....@decurtis.com]

For an iOS application that is using Couchbase lite, we do have "enableLogging" method on CBLManager class that enables us to specify type of logging for which log messages will start appearing in console. 

1.Is there a way to store these log messages (from ios application) as documents inside the couchbase server so that analysis of these log messages can be done via a search plugin for example "Elastic Search" on the server ?

2.Is it also possible that we can direct log messages to a different couchbase server (can be called a Logging server) that will be storing log messages from various couchbase lite based ios apps that is having different couchbase servers or databases?

The point #2 can be stated in other words as: Can we setup a couchbase server (acting as a Logging Server) that will store log messages from different iOS apps that are connected to different databases.

Thanks

[Traun Leyden]

@Jens: I remember you worked on something to transport logs from the Couchbase Lite iOS up to the server.  Do you have any details on that?

[Jens Alfke]

On Apr 20, 2017, at 3:21 AM, parvez....@decurtis.com wrote:

1.Is there a way to store these log messages (from ios application) as documents inside the couchbase server so that analysis of these log messages can be done via a search plugin for example "Elastic Search" on the server ?

There is, in the Objective-C (iOS/Mac/tvOS) version, but it hasn’t gone through formal testing so it’s not part of the public API. Of course we can’t really hide anything because it’s right there in the source code…

<experimental>

CBLTimeSeries is a class that can collect small timestamped data values called “events” (as JSON objects) rapidly into a database. It packs them together into an array and stores that in a document. It will stuff up to 1000 events, or 100KB of JSON data, into a single document. This gives it much lower overhead than creating a document for every event. The downside is that it’s write-only: you can’t modify or delete individual events.

The class includes a convenience method to set up a push replication to a server; it can even purge the documents locally after they’ve been successfully pushed, so the local database won’t get bloated.

There’s a subclass CBLRemoteLogging that does this for log messages. You just get the sharedInstance at startup, and tell it what logging keywords/channels you want it to capture. Then you can use the inherited replication setup method to push it to your server.

(You’ll need to copy the headers into your project and #import them, to be able to call these APIs.)

On the server side, the documents have a special ‘_type’ property to identify them, and another property that’s an array of all the events. (Details are in CBLTimeSeries.m.) It’s quite straightforward to write a map/reduce view on the server that emits every individual event, and then you can query that.

</experimental>

I’ll try to answer questions, but as I said, this isn’t an official API so it won’t get the same level of support. (I.e. if you’re a Couchbase customer, don't ask our support engineers for help on this.)

—Jens

PS: We’d like to make this an official feature (on all platforms) in 2.x, but we can’t guarantee that x=0…

[parvez....@decurtis.com]

I am using the 1.3.1 via cocoapods for a Swift Project. Are the classes CBLTimeSeries, CBLRemoteLogging available in this version? I am not able to see them. Can you please provide  a sample code where we can see how to use them in ios swift projects?

Thanks 

[parvez....@decurtis.com]

I have tried making use of CBLManager class's redirectLogging method to get all logs that are enabled using enableLogging method. Then I create instances of CBLModel subclass

public class LogMessage: CBLModel{

    @NSManaged var messageType: String!

    @NSManaged var message: String!

    @NSManaged var created_at: String!

}

to create a document for each log message received and then saving it to the database as shown below:

public func saveLog(message: String, logMessageType: String){

        let logMessage = LogMessage(forNewDocumentIn: database!)

        logMessage.type = "log"

        logMessage.message = message

        logMessage.messageType = logMessageType

        logMessage.created_at = CBLJSON.jsonObject(with: Date())

        do{

            try logMessage.save()

        }catch{

            print("Encountered error: \(error) while saving log message")

        }

    }

How can we make above LogMessage model to be push only document and it should not be pulled back on same or other devices running the application? Since in my case logging is not meant to store logs locally inside the database. It can be possible that the same application is having a push / pull replication already there for replicating other documents so how to handle this situation?

Thanks

[Jens Alfke]

On Apr 21, 2017, at 2:32 AM, parvez....@decurtis.com wrote:

I have tried making use of CBLManager class's redirectLogging method to get all logs that are enabled using enableLogging method. Then I create instances of CBLModel subclass

I would not make each log message a separate document; it produces too many documents.

How can we make above LogMessage model to be push only document and it should not be pulled back on same or other devices running the application?

On the server side, detect documents containing logs and don’t add them to any channels that user accounts are pulling from.

Or just put the logs into an entirely different database that clients only push to, never pull from.

—Jens

[Pankaj Sharma]

Yes @Jens And Parvez

We will be having a complete different bucket for it and the document will have a channel which wont be subscribed by any user. This way we will stop pulling logs to other devices. Apart from this we would use TTL so that the document is purged after a configured period of time. The essence of this exercise is to know what errors are coming on clients having our apps. So we can keep on solving them in releases we will plan.

Regards

Pankaj