Validation Benchmark

[Atamert Ölçgen]

Hi All,

I have been working a benchmark for (runtime) validation libraries. I think it is mature enough to share the results with the group.

The repo is here: https://github.com/muhuk/validation-benchmark

Raw benchmark output as EDN is here: https://gist.github.com/muhuk/93d7d9e46bf5191310aaa4557379d10e

More human readable results are in my blog: http://blog.muhuk.com/2016/04/18/performance_comparison_of_annotate_herbert_schema.html

If you are familiar with Annotate, Herbert or Schema, please take a quick look at their implementations: https://github.com/muhuk/validation-benchmark/tree/master/src/validation_benchmark/lib

Also contributions of other validation libraries are most welcome!

-----

As an aside: what is the group policy for attachments? Would it be ok if I attached images of benchmark results?

--

Kind Regards,
Atamert Ölçgen

◻◼◻

◻◻◼

◼◼◼

www.muhuk.com

[Jason Wolfe]

Thanks for putting this together!  

For schema, I think using `s/validate` is not a fair comparison with the other libraries -- as you mention, it throws an exception on error, and it also has to parse the schema each time.  

I think `s/checker` is the correct construct to use here -- it returns an error or nil for success (no exceptions), and only parses the schema once.  (This is what's used by `s/defn`, for example).  

Here's a pull request that implements this change:

https://github.com/muhuk/validation-benchmark/pull/6

I posted the updated results in the PR comments as well, which are qualitatively quite different from the ones currently reported in your blog post (with this change, schema is fastest on all but one benchmark, often by a factor of 3 or more).  

Please let me know if you have any questions. 

Thanks,

Jason

[Atamert Ölçgen]

Hi Jason,

Thank you for the PR. I've just merged it. 

For now I have added this to the blog post:

Results below and the comments about Schema being slow is no longer true. See this pull request for the results Jason got with quick mode benchmark.

I am thinking about generating a stand alone page (GH pages) so that I don't have to update or create a new post everytime the results are updated.

The README for Schema doesn't mention checker function, FYI.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jason Wolfe]

On Wed, Apr 20, 2016 at 12:31 AM, Atamert Ölçgen <mu...@muhuk.com> wrote:
> Hi Jason,
>
> Thank you for the PR. I've just merged it.
>
> For now I have added this to the blog post:
>
> Results below and the comments about Schema being slow is no longer true.
> See this pull request for the results Jason got with quick mode benchmark.
>
> I am thinking about generating a stand alone page (GH pages) so that I don't
> have to update or create a new post everytime the results are updated.
>
> The README for Schema doesn't mention checker function, FYI.

Thanks! I've just updated the README to explain check and checker as well.

> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/P3X9995Lrp8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to