Documentation or starter examples for writing Vitess plugins?

[Steve Perkins]

I am interested in writing a custom logger plugin for internal use by my company.  I'm at least intermediate-level with Go, but I'm struggling to figure out how to get started.

There are numerous sub-packages beneath "cmd" in the Vitess git repo (i.e. https://github.com/youtube/vitess/tree/master/go/cmd), and some of these have code comments indicating that they are plugins.  However, I can't really tell where the "built-in command" line ends and the "plugin" line begins, and I'm sure that any of these are really examples to follow for building your own custom plugins.

Is there any documentation out there, or at least source code for a good starter example, of writing your own custom Vitess plugin?  How the code works, how to build/deploy/configure the plugin in your Vitess environment, etc?  I assume that surely you don't have to fork the entire Vitess git repo, and produce your own custom build of EVERYTHING with your own code statically linked.

Thanks!

[Sugu Sougoumarane]

Hi Steve,

I don't fully understand what you're looking for. Do you want to reuse vitess code to write a custom logger, or do you want to plug into an existing vitess instance to perform custom logging?

If it's the latter, are you talking about plugging in a custom version of what the querylog does?

--
You received this message because you are subscribed to the Google Groups "vitess" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vitess+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Perkins]

Perhaps.  Apparently there was some contact between my organization and your team a few months back.  Now that I'm getting pulled in to actually do the work, I'm just trying to connect the dots from the notes I was given.

The content rendered by the "/debug/querylog" monitoring endpoint would work.  However, I think the issue there is that we REALLY do need to capture every query for auditing purposes.  Unless I'm missing something huge, this monitoring endpoint is HTTP based.  Meaning that if (or rather "when") we have an interrupted HTTP connection, we'll miss some records while re-establishing it.

The person we spoke to indicated that, “[Vitess] supports registering plugins that get a stream of all requests, using tabletserver.StatsLogger.Subscribe(""). You can use it to get all queries made to the database."

So that's basically the path that I'm investigating now.  How to attach a (non-HTTP) listener process directly to this StatsLogger, and send the output to syslog, ElasticSearch, Kafka, or whatever it might be in the future.  However, from what I know of Go, and crawling through this codebase... I'm not sure what "registering a plugin" would look like in this context.  I'm afraid that would mean creating my own private build of Vitess tabletserver, with my logger functionality statically linked.

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+un...@googlegroups.com.

[Alain Jobart]

Hi Steve,

Actually, if you want to build your own plug-ins, you do have to compile your code within the Vitess tree. It is actually exactly what we do within YouTube, we take all the files from the open source tree, and add a few of our own, both as separate libraries in different paths, and extra xxx_plugin.go files that import these libraries. There is no dynamic linking of plugin code, it's all statically linked.

But your use case seems to be very generic though, if you were to create a plug-in that does what you describe (sending all queries to syslog), we would accept it in the main tree, and help you with reviews / approvals. So you wouldn't need your own build.

The code would look like this, in a new library:

func init() {

servenv.OnRun(func() { 

google.RegisterInit(func() {

if *logQueries {

go run()

}

})

}

On Thu, Nov 3, 2016 at 8:16 AM, 'Steve Perkins' via vitess <vit...@googlegroups.com> wrote:

Perhaps.  Apparently there was some contact between my organization and your team a few months back.  Now that I'm getting pulled in to actually do the work, I'm just trying to connect the dots from the notes I was given.

I think that the content rendered by the "/debug/querylog" monitoring endpoint would work.  However, I think the issue there is that we REALLY do need to capture every query for auditing purposes.  Unless I'm missing something huge, this monitoring endpoint is HTTP based.  Meaning that if (or rather "when") we have an interrupted HTTP connection, we'll miss some records.

The person we spoke to indicated that, “[Vitess] supports registering plugins that get a stream of all requests, using tabletserver.StatsLogger.Subscribe(""). You can use it to get all queries made to the database."

So that's basically the path that I'm investigating now.  How to attach a (non-HTTP) listener process directly to this StatsLogger, and send the output to syslog, ELK

On Wednesday, November 2, 2016 at 5:18:26 PM UTC-4, Sugu Sougoumarane wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alain Jobart]

Ignore code sample, here is the right version:

var (

logQueries   = flag.Bool("log_queries", false, "Enable query logging to syslog.")

)

func run() {

ch := tabletserver.StatsLogger.Subscribe("gwslog")

defer tabletserver.StatsLogger.Unsubscribe(ch)

        for out := range ch {

              /* write to syslog *.

        }

}

func init() {

    servenv.OnRun(func() { 

[Alain Jobart]

OK I need to stop sending emails too early ;)

Short version is: we can help you with the code, we'll take that plug-in into our tree, as it seems valuable, for both auditing and debugging.

As you may be able to tell by now, we have an internal version of the plugin you want that writes the queries to our internal auditing system. That internal auditing system is proprietary, that's why we don't have it in the open source code. But adding an optional plugin that writes to syslog would be very easy to do...

[Sugu Sougoumarane]

Hi Steve,

I still have a couple of questions:

1. If delivery of an event is not possible, should vitess drop the message in favoring execution, or should it never give up?

2. Does your audit log need to report SELECT statements?

For #1: If you need delivery guarantee, the stats logger may not be appropriate as it stands now. It was built as a best effort system. It tries to send, but if delivery fails, it just drops the message. In particular, if the syslog write can't keep up with the QPS, messages will be dropped.

If you need delivery guarantee, but don't need to report SELECT statements, you may want to consider the update stream, which reports from the binlogs. The update stream gives you delivery guarantee, but it's at the MySQL level and only serves raw DMLs.

If none of the above meets your needs, we'll need to build a different system. If all you need is something as simple as writing to syslog, then it would be fairly trivial since the rest of the pieces are already present.

[Steve Perkins]

Hi Alain and Sugu -

Regarding Sugu's most recent question:  Thank you very much for the detailed insight!  I've talked it over, and I think simply pulling from StatsLogger on a best effort basis is acceptable.  If this is to be merged upstream for potential use by anyone, then I'm mindful of keeping it as simple and intuitive as possible.

Regarding Alain's code snippets and high-level guidance:  I've been able to successfully stand up a local Vitess dev environment, and starting on Monday I'll finally focus on this full-time for a couple of weeks (assuming it doesn't take less time).  

However, I'm a bit confused as to what you mean by putting the code "in a new library".  I've grepped through the Vitess code, and there doesn't seem to be an existing example of the string "func run()" anywhere in the Go codebase.  I can probably handle the "subscribe to StatsLogger and write to syslog" part, as there's nothing too crazy about that... but I'm not exactly sure where to put the code.

One other minor question.  I notice that Golang has syslog support in the standard library's "log/syslog" package.  However, the package comments indicate that it is "frozen", and seem to steer users toward third-party packages instead.  Are you guys comfortable with me simply using the standard library package?

Thanks!

You received this message because you are subscribed to a topic in the Google Groups "vitess" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vitess/GHrmvmMJMmk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vitess+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Steve Perkins | Director of Engineering | m. 404-997-3018

3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 

"We could do that better. We’ve got the right technology." - Meet Brandon Williams, our newest Modern Workplace Innovator

[Alain Jobart]

Hi Steve,

There aren't any open source users of this feature yet. We use it internally at YouTube though, so we do have a plug-in in our code.

For the package location, I suggest creating it in:

go/vt/tabletserver/sysloglogger

as it is a feature related to tablet server.

Then in there you can define a flag to enable the logging or not.

Then let's add a plugin_sysloglogger.go file in go/cmd/vttablet that just imports the module. That way it will be in vttablet, but it's easy to remove entirely from the binary if the need arises.

Yes, simply using the standard library package would work. We already do that in the event logger code.

Let us know if you have more questions (don't hesitate to ask early so you're not stuck or going the wrong way), and good luck!

Alain.