[erlang-questions] MySQL driver
155 views
Skip to first unread message
Dave Smith
Apr 10, 2013, 4:43:01 PM4/10/13
to Erlang Questions
Hello list,
I'm curious what libraries people are currently using for MySQL access in Erlang. I have an old library on my github that I've kept basically working, but am wondering if there are any preferred/production libraries out there that people like better.
I ask because I'm considering retrofitting the pooling system in the driver I maintain to use poolboy, but figure it was worth checking if there are better alternatives out there. :)
After all, the best code to maintain is the code you don't maintain.
D.
Jeremy Ong
Apr 10, 2013, 4:46:07 PM4/10/13
to Dave Smith, Erlang Questions
I've been using emysql https://github.com/Eonblast/Emysql which has connection pools and supports storing and executing procedures in addition to arbitrary sql queries.
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Dmitry Kolesnikov
Apr 10, 2013, 4:49:54 PM4/10/13
to Dave Smith, Erlang Questions
Hello,
Thanks for asking…
I am using your library at production with small enhancement, which is part of pull request.
There are couple of minor concerns and issues with that library, especially in area of connection supervision but in-general is good.
Best Regards,
Dmitry
Thanks for asking…
I am using your library at production with small enhancement, which is part of pull request.
There are couple of minor concerns and issues with that library, especially in area of connection supervision but in-general is good.
Best Regards,
Dmitry
Henning Diedrich
Apr 10, 2013, 5:00:50 PM4/10/13
to Dave Smith, Erlang Questions
Hi Dave,
if I remember right, you were using the driver that came from Yxa via Yarif's erlang-emysql-driver https://github.com/dizzyd/erlang-mysql-driver ? (Here is the history of that. https://github.com/eonblast/Emysql#History)
It was found less than perfect when Nick and Jacob gave it something to chew building Rupture. I never spotted the problems myself I should add. That's purely from Nick Gerakine's and Jacob Vorreuter's words.
But they report that with much effort they could not get the races out of it. That's why they started emysql, which I later started maintaining, and which Jeremy just mentioned, https://github.com/eonblast/Emysql
Considerable work went into getting the pooling system of Emysql right. (What I don't know is when exactly it got buggy, it may have been after Jacob's Emysql fork that now lies dormant.) But I worked a while, with help from others to get all races out of the queuing.
However, the best fork of Emysql, I *think* is currently that of Chris Rempel at https://github.com/csrl/emysql as I got behind while making Erlvolt, the VoltDB driver, modeled in parts on Emysql. Chris did some impressive clean up. Now that Erlvolt is out, I'll turn around and catch up asap. But that's not a New York asap.
An alternative may be to go back to a more original Yxa version, like the one possibly still to be found in ejabberd. That may be more stable than Yariv's branch that you started out with. The "history" I linked to above has the reasons, why that branch may be not quite clean and hard to debug.
Best,
Henning
if I remember right, you were using the driver that came from Yxa via Yarif's erlang-emysql-driver https://github.com/dizzyd/erlang-mysql-driver ? (Here is the history of that. https://github.com/eonblast/Emysql#History)
It was found less than perfect when Nick and Jacob gave it something to chew building Rupture. I never spotted the problems myself I should add. That's purely from Nick Gerakine's and Jacob Vorreuter's words.
But they report that with much effort they could not get the races out of it. That's why they started emysql, which I later started maintaining, and which Jeremy just mentioned, https://github.com/eonblast/Emysql
Considerable work went into getting the pooling system of Emysql right. (What I don't know is when exactly it got buggy, it may have been after Jacob's Emysql fork that now lies dormant.) But I worked a while, with help from others to get all races out of the queuing.
However, the best fork of Emysql, I *think* is currently that of Chris Rempel at https://github.com/csrl/emysql as I got behind while making Erlvolt, the VoltDB driver, modeled in parts on Emysql. Chris did some impressive clean up. Now that Erlvolt is out, I'll turn around and catch up asap. But that's not a New York asap.
An alternative may be to go back to a more original Yxa version, like the one possibly still to be found in ejabberd. That may be more stable than Yariv's branch that you started out with. The "history" I linked to above has the reasons, why that branch may be not quite clean and hard to debug.
Best,
Henning
Dave Smith
Apr 10, 2013, 5:59:20 PM4/10/13
to Henning Diedrich, Erlang Questions
Hi Henning,
Thanks for the pointers. I must admit that I find the level of snark in the emysql history delightful, considering that it's at least as forked as erlang-mysql-driver now. :)
There are a number of good fixes for the emysql driver, but they are on 20 different forks across Github. Looking over csrl's, fixes, I'm pretty concerned those aren't in mainline since there are some non-trivial pooling problems he's addressed (would you like some irony with that irony? :)).
Do you know if eonblast is planning to incorporate all the work from those different forks? I think we need to build some momentum behind a driver and I'm certainly happy to direct people towards a maintained one.
IMO, it would also be handy if the driver wasn't pooled, but used poolboy since getting the pooling right is rather difficult. (I know that poolboy has been tested with some of Quviq's concurrent checking stuff and a number of race conditions addressed).
Thanks!
D.
Henning Diedrich
Apr 10, 2013, 6:21:40 PM4/10/13
to Dave Smith, Erlang Questions
Hi Dave,
that would be great if you'd join the effort there with the driver!
As I said, I can't wait to bring the mainline up to date, and I'll do it, it was delayed by the focus on Erlvolt recently. (Please check it out: https://github.com/Eonblast/Erlvolt -- http://blog.voltdb.com/877000-tps-with-erlang-and-voltdb/)
I'll take a look at poolboy. However, I've gone through the pool again and again, and so did some people smarter than me, I think it's solid.
What I came to believe about erlang-mysql-driver was that the concerns of its three different layers had been violated on the way and that will have made it hard to spot errors.
I also have a good orientation about the Emysql forks as they are and which ones are more valuable, as preparation for catching up. It's quite some work there to re-unite them, but additions and corrections are minor, with the exception of two competing approaches to transactions, where I'd love to get your opinion. Yes it grew to be the same "convincing case for the github Network graphs" :-D
Henning
Garrett Smith
Apr 10, 2013, 6:25:45 PM4/10/13
to Dave Smith, Erlang Questions
Someday someone will write a book on the history of the Erlang MySQL
libraries. Not a great read, but it'd have to be scholarly.
I've modified the Eonblast library with some connection timeout tweaks
(as in, supporting them):
https://github.com/gar1t/Emysql
libraries. Not a great read, but it'd have to be scholarly.
I've modified the Eonblast library with some connection timeout tweaks
(as in, supporting them):
https://github.com/gar1t/Emysql
Dave Smith
Apr 10, 2013, 6:41:10 PM4/10/13
to Garrett Smith, Erlang Questions
Re: pooling -- there's always bugs...ALWAYS BUGS. We were using poolboy at Basho and thought that it was so simple/basic that bugs should be easy to sort out...until Dr. Hughes walked into the room and pulled a Concurrent Debuggerator 3000 from his black bag (aka QuickCheck) and found 5 significant concurrency bugs in as many minutes...and this was on a library that only does pooling and nothing else. So...unless you tell me that Dr. Hughes has been over this pooling code with the aforementioned tool, I will eye the pooling code with a raised eyebrow. :)
csrl - pooling fixes, edge cases, etc. (underscores my point about pooling, but I digress)
jlarky - looks like a opscode branch that has some good bug fixes
taybin - adds better UTF support (it's a big patch though, so probably needs some work)
rflynn - bug fix for float encoding
gar1t - bug fixes + polish on API
What are the competing approaches on transactions?
D.
Henning Diedrich
Apr 10, 2013, 6:41:26 PM4/10/13
to Garrett Smith, Erlang Questions
Hi Garrett,
great, I'll check out your connection time outs, thanks a lot!
It was quite fun to find Jacob, Nick and Fredrik to write this [1] up. It certainly felt more like historical forensics from some point on although I was throughout only trying to find out which driver version made most sense. Which in the end came to light in a way I could not read out of the source.
If someone would like to share the story of his fork I'll be happy to include it, for the fun of it. Somehow, this all fits MySQL.
Henning
[1] https://github.com/eonblast/Emysql#History
great, I'll check out your connection time outs, thanks a lot!
It was quite fun to find Jacob, Nick and Fredrik to write this [1] up. It certainly felt more like historical forensics from some point on although I was throughout only trying to find out which driver version made most sense. Which in the end came to light in a way I could not read out of the source.
If someone would like to share the story of his fork I'll be happy to include it, for the fun of it. Somehow, this all fits MySQL.
Henning
[1] https://github.com/eonblast/Emysql#History
Jesper Louis Andersen
Apr 16, 2013, 6:06:21 AM4/16/13
to Henning Diedrich, Erlang Questions
I may be interested in helping out on this thing as well. I have some work to do changing driver infrastructure to Emysql and when that comes around (in a couple of days/weeks), i'll probably also try to look into some merging of patches if possible.
Please keep me in the loop :)
Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen
Please keep me in the loop :)
Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen
Pablo Polvorin
Apr 16, 2013, 4:07:43 PM4/16/13
to Erlang Questions
Is there a driver that supports the binary mysql protocol (not the textual, sql one) for prepared statements?.
I come in a situation recently on a query where we have to bind lots of parameters into a prepared statement, and the time to escape all of them by the driver and send the corresponding SET @N = .. was substantial, actually larger than the time it takes mysql to run the query. Using the binary protocol in this situation would help.
I found http://erlmysql.sourceforge.net/ that do support the binary protocol, and although it was much faster on this particular query (order of magnitudes faster) it sometimes crashed and I found the code more complex than other mysql drivers I've worked with, both reason to make me nervous about using it. Is anyone using it on production?, do you know alternatives?--
Pablo Polvorin
ProcessOne
I come in a situation recently on a query where we have to bind lots of parameters into a prepared statement, and the time to escape all of them by the driver and send the corresponding SET @N = .. was substantial, actually larger than the time it takes mysql to run the query. Using the binary protocol in this situation would help.
I found http://erlmysql.sourceforge.net/ that do support the binary protocol, and although it was much faster on this particular query (order of magnitudes faster) it sometimes crashed and I found the code more complex than other mysql drivers I've worked with, both reason to make me nervous about using it. Is anyone using it on production?, do you know alternatives?
Pablo Polvorin
ProcessOne
Loïc Hoguin
Apr 16, 2013, 4:43:29 PM4/16/13
to Pablo Polvorin, Erlang Questions
bank_mysql has binary prepared statements:
https://github.com/extend/bank_mysql
You can use it with the bank pool (https://github.com/extend/bank) or
without, whichever you prefer.
No user guide at this time but it should be pretty easy to read.
On 04/16/2013 10:07 PM, Pablo Polvorin wrote:
> Is there a driver that supports the binary mysql protocol (not the
> textual, sql one) for prepared statements?.
>
> I come in a situation recently on a query where we have to bind lots of
> parameters into a prepared statement, and the time to escape all of
> them by the driver and send the corresponding SET @N = .. was
> substantial, actually larger than the time it takes mysql to run the
> query. Using the binary protocol in this situation would help.
>
> I found http://erlmysql.sourceforge.net/ that do support the binary
> protocol, and although it was much faster on this particular query
> (order of magnitudes faster) it sometimes crashed and I found the code
> more complex than other mysql drivers I've worked with, both reason to
> make me nervous about using it. Is anyone using it on production?, do
> you know alternatives?
>
>
> *|
> |*
> <hd2...@eonblast.com <mailto:hd2...@eonblast.com>>
Erlang Cowboy
Nine Nines
http://ninenines.eu
https://github.com/extend/bank_mysql
You can use it with the bank pool (https://github.com/extend/bank) or
without, whichever you prefer.
No user guide at this time but it should be pretty easy to read.
On 04/16/2013 10:07 PM, Pablo Polvorin wrote:
> Is there a driver that supports the binary mysql protocol (not the
> textual, sql one) for prepared statements?.
>
> I come in a situation recently on a query where we have to bind lots of
> parameters into a prepared statement, and the time to escape all of
> them by the driver and send the corresponding SET @N = .. was
> substantial, actually larger than the time it takes mysql to run the
> query. Using the binary protocol in this situation would help.
>
> I found http://erlmysql.sourceforge.net/ that do support the binary
> protocol, and although it was much faster on this particular query
> (order of magnitudes faster) it sometimes crashed and I found the code
> more complex than other mysql drivers I've worked with, both reason to
> make me nervous about using it. Is anyone using it on production?, do
> you know alternatives?
>
>
> |*
> <mailto:jesper.lou...@erlang-solutions.com>> wrote:
>
> I may be interested in helping out on this thing as well. I have
> some work to do changing driver infrastructure to Emysql and when
> that comes around (in a couple of days/weeks), i'll probably also
> try to look into some merging of patches if possible.
>
> Please keep me in the loop :)
>
> Jesper Louis Andersen
> Erlang Solutions Ltd., Copenhagen
>
>
>
> On Apr 11, 2013, at 12:41 AM, Henning Diedrich <hd2...@eonblast.com
>
> I may be interested in helping out on this thing as well. I have
> some work to do changing driver infrastructure to Emysql and when
> that comes around (in a couple of days/weeks), i'll probably also
> try to look into some merging of patches if possible.
>
> Please keep me in the loop :)
>
> Jesper Louis Andersen
> Erlang Solutions Ltd., Copenhagen
>
>
>
> On Apr 11, 2013, at 12:41 AM, Henning Diedrich <hd2...@eonblast.com
> <mailto:hd2...@eonblast.com>> wrote:
>
> > Hi Garrett,
> >
> > great, I'll check out your connection time outs, thanks a lot!
> >
> > It was quite fun to find Jacob, Nick and Fredrik to write this
> [1] up. It certainly felt more like historical forensics from some
> point on although I was throughout only trying to find out which
> driver version made most sense. Which in the end came to light in a
> way I could not read out of the source.
> >
> > If someone would like to share the story of his fork I'll be
> happy to include it, for the fun of it. Somehow, this all fits MySQL.
> >
> > Henning
> >
> > [1] https://github.com/eonblast/Emysql#History
> >
> >
> > On Apr 11, 2013, at 12:25 AM, Garrett Smith <g...@rre.tt
>
> > Hi Garrett,
> >
> > great, I'll check out your connection time outs, thanks a lot!
> >
> > It was quite fun to find Jacob, Nick and Fredrik to write this
> [1] up. It certainly felt more like historical forensics from some
> point on although I was throughout only trying to find out which
> driver version made most sense. Which in the end came to light in a
> way I could not read out of the source.
> >
> > If someone would like to share the story of his fork I'll be
> happy to include it, for the fun of it. Somehow, this all fits MySQL.
> >
> > Henning
> >
> > [1] https://github.com/eonblast/Emysql#History
> >
> >
> > On Apr 11, 2013, at 12:25 AM, Garrett Smith <g...@rre.tt
> <mailto:diz...@dizzyd.com>> wrote:
> >>>>
> >>>>> Hello list,
> >>>>>
> >>>>> I'm curious what libraries people are currently using for
> MySQL access
> >>>>> in Erlang. I have an old library on my github that I've kept
> basically
> >>>>> working, but am wondering if there are any
> preferred/production libraries
> >>>>> out there that people like better.
> >>>>>
> >>>>> I ask because I'm considering retrofitting the pooling system
> in the
> >>>>> driver I maintain to use poolboy, but figure it was worth
> checking if there
> >>>>> are better alternatives out there. :)
> >>>>>
> >>>>> After all, the best code to maintain is the code you don't
> maintain.
> >>>>>
> >>>>> D.
> >>>>> _______________________________________________
> >>>>> erlang-questions mailing list
> >>>>> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>
> >>>>
> >>>>> Hello list,
> >>>>>
> >>>>> I'm curious what libraries people are currently using for
> MySQL access
> >>>>> in Erlang. I have an old library on my github that I've kept
> basically
> >>>>> working, but am wondering if there are any
> preferred/production libraries
> >>>>> out there that people like better.
> >>>>>
> >>>>> I ask because I'm considering retrofitting the pooling system
> in the
> >>>>> driver I maintain to use poolboy, but figure it was worth
> checking if there
> >>>>> are better alternatives out there. :)
> >>>>>
> >>>>> After all, the best code to maintain is the code you don't
> maintain.
> >>>>>
> >>>>> D.
> >>>>> _______________________________________________
> >>>>> erlang-questions mailing list
> >>>>> http://erlang.org/mailman/listinfo/erlang-questions
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> erlang-questions mailing list
> >>> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> erlang-questions mailing list
> >>> http://erlang.org/mailman/listinfo/erlang-questions
> >>>
> >
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>
> >>>
> >
> > _______________________________________________
> > erlang-questions mailing list
> > http://erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>
>
> _______________________________________________
> erlang-questions mailing list
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
>
> --
> Pablo Polvorin
> ProcessOne
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>
--
Loïc Hoguin
>
>
>
>
> --
> Pablo Polvorin
> ProcessOne
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>
--
Erlang Cowboy
Nine Nines
http://ninenines.eu
Pablo Polvorin
Apr 16, 2013, 10:31:23 PM4/16/13
to Loïc Hoguin, Erlang Questions
looks good, I'll try it, thanks!
--
Pablo Polvorin
ProcessOne
On 16 April 2013 17:43, Loïc Hoguin <es...@ninenines.eu> wrote:
bank_mysql has binary prepared statements: https://github.com/extend/bank_mysql
You can use it with the bank pool (https://github.com/extend/bank) or without, whichever you prefer.
No user guide at this time but it should be pretty easy to read.
On 04/16/2013 10:07 PM, Pablo Polvorin wrote:
Is there a driver that supports the binary mysql protocol (not the*|
textual, sql one) for prepared statements?.
I come in a situation recently on a query where we have to bind lots of
parameters into a prepared statement, and the time to escape all of
them by the driver and send the corresponding SET @N = .. was
substantial, actually larger than the time it takes mysql to run the
query. Using the binary protocol in this situation would help.
I found http://erlmysql.sourceforge.net/ that do support the binary
protocol, and although it was much faster on this particular query
(order of magnitudes faster) it sometimes crashed and I found the code
more complex than other mysql drivers I've worked with, both reason to
make me nervous about using it. Is anyone using it on production?, do
you know alternatives?
|*
On 16 April 2013 07:06, Jesper Louis Andersen
>>>>> erlang-q...@erlang.org <mailto:erlang-questions@erlang.org>
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-q...@erlang.org <mailto:erlang-questions@erlang.org>
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>
> _______________________________________________
> erlang-questions mailing list
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
http://erlang.org/mailman/listinfo/erlang-questions
--
Pablo Polvorin
ProcessOne
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
--
Loďc Hoguin
Erlang Cowboy
Nine Nines
http://ninenines.eu
--
Pablo Polvorin
ProcessOne
Reply all
Reply to author
Forward
0 new messages