I have been making a few cleanups and fixing a few bugs on a
(temporary?) fork of the codehaus.org copy of the Ruby Stomp gem. I
would encourage anyone who has issues that need fixing to submit
patches for code and or documentation as a pull request on GitHub.
The intention is to provide all changes back to the original project
and they can decide if they would like to accept them and officially
release the changes we are submitting as a new version of the official
gem on rubyforge.
GitHub automatically generates new gems from the master branch of the
repo and I'll put instructions for installing the gem from github in
the README which can be found at the link above.
Please accompany all code changes with matching Rspec specs, which
should be mocked/stubbed so they don't make calls directly to external
servers, or to collaborating classes.
TODO : One thing I would personally like to see is an easy way to make
a synchronous request to the STOMP server which is non-blocking. The
poll method seems broken to me, and receive appears to connect and
block until messages arrive. Someone want to take that one on or show
me the error of my ways? :-)
I just got your two messages about this git repo for a fork of the stomp gem (I'm also on the stomp lists). Glad to see more interest in stomp for ruby/rails.
I'm a bit confused why it was necessary to fork the project, and I hope it does not becomes confusing for users. When I had a number of bug fixes and changes for the gem, I submitted patches and quickly found myself with commit rights.
Is there a reason you didn't want to submit patches to the existing gem repo? I am watching the store and actively work on and use the gem, and would have been happy to help get your code committed, and I'm pretty such Brian would be happy to have more contributers as well.
There are a number of changes to the stomp gem I have been working on lately that fix some errors in the existing retry logic. I am close to committing these to the gem trunk; I wonder how we can coordinate active development on the gem with your fork of the code?
That said, if there is good stuff that comes out of your efforts, I would be happy to help you get your changes into the official stomp gem repository.
I have a few other specific responses in context below:
On Sat, May 10, 2008 at 1:12 PM, Glenn Rempe <glenn.re...@gmail.com> wrote:
> Please accompany all code changes with matching Rspec specs, which > should be mocked/stubbed so they don't make calls directly to external > servers, or to collaborating classes.
The current gem doesn't use or depend on rspec; I respect that this is how you like to work, but I would think it would be more appropriate to offer test unit classes, as that is what the gem uses now, and otherwise you end up with a mix of units and specs, and that's awkward.
> TODO : One thing I would personally like to see is an easy way to make > a synchronous request to the STOMP server which is non-blocking.
The way the gem works now is that when you make a request, it blocks until that request has been sent to the broker. Is what you want to be able to do the send, but then immediately move on, regardless of if the request has to be retried, or can even be sent right away? If so, it sounds like all you want is to use a different thread to do the send, such as putting the send in the block for a Thread.new.
Or, is what you mean a way to get a _response_ to a request where listening for the response is not blocking? This to me sounds like a contradiction - if it's synchronous, it means you are waiting for the response, which means it is blocking, or you are building into your logic on the main execution thread to check back at various points in the program flow to see if there is a response, which is a hard/impossible thing to contain in a method call. All the API could provide for that is the equivalent of the private _receive method which looks for a new message once then returns.
> The > poll method seems broken to me, and receive appears to connect and > block until messages arrive.
Poll works as I use it in a13g, perhaps it depends what you are trying to do with it, or how you expect it to work? What are you expecting?
Stomp as a protocol is designed so that you subscribe to destinations, and the broker will then push messages to you over your open connection, but that means some thread has to be listening/polling on that open connection with some frequency to receive the messages. So there you go, that's what poll and receive do.
Receive loops checking to see if there is new incoming IO on the tcp port, and once it gets a stomp frame, returns. The _receive message tries once, and then returns, so all receive does is call this in a loop till something comes back. Is what you want what _receive does, where it looks for a message once and then quits, so it is not blocking?
Poll is just a synchronizing wrapper to make sure you don't have more than one thread simultaneously trying to read the next message from off the connection (imagine how ugly that could get, if we get 2 threads competing to get IO off the connection, and end up with a single message read in pieces by different threads).
I guess I just don't understand how you expect this all to work where some thread isn't taken up busy trying to listen for incoming IO.
> Someone want to take that one on or show > me the error of my ways? :-)
The Client class in the stomp gem starts up a listen thread to receive messages on initialize, so it is not blocking the main execute thread. In your main code you then register listener blocks when you subscribe that will get called back asynchronously when your a message is received.
All you need to do is instantiate the Client, and use that to This is the same approach in active messaging, where threads are started to listen for messages. Did you have another approach in mind?
If you don't want it to block, but you do want to listen for and then handle messages whenever they arrive, that means you either need another thread, or have breaks in the processing of the main thread to comeback and look for incoming messages and handle them.
Anyway, let me know if I have misunderstood, and I look forward to your contributions to stomp and messaging,
Thanks for taking the time to provide a detailed response. Much
appreciated.
I'll provide some comments below:
On May 10, 12:34 pm, "Andrew Kuklewicz" <kooks...@gmail.com> wrote:
> I'm a bit confused why it was necessary to fork the project, and I hope it
> does not becomes confusing for users.
> When I had a number of bug fixes and changes for the gem, I submitted
> patches and quickly found myself with commit rights.
> Is there a reason you didn't want to submit patches to the existing gem
> repo?
> I am watching the store and actively work on and use the gem, and would have
> been happy to help get your code committed, and I'm pretty such Brian would
> be happy to have more contributers as well.
I am totally happy to have the stomp gem use any of my patches and
pull them into the code. My 'fork' is not so much to take the project
in a different direction, it is just the terminology used freely in
the Git/GiHub context where forks are encouraged, but used to feed
changes back to the master. I have found github to be an excellent
tool for getting additional work done by others and feeding that back
to the master since the barrier to entry for submitting changes is so
very low (no commit privs needed, just fork and go, and then send pull
requests for changes.
> There are a number of changes to the stomp gem I have been working on lately
> that fix some errors in the existing retry logic.
> I am close to committing these to the gem trunk; I wonder how we can
> coordinate active development on the gem with your fork of the code?
My git repo is actually able to be actively connected to both the SVN
master which I would pull your changes from, and then merge them into
my branch which is what I would push to GitHub. So if you or others
commit changes I can get those onto Git and keep everything in synch.
Or even better :-) You can fork my git repo, and push the work onto
github and we can merge our git repos easily. Or I'll give you commit
access to my git repo directly. Once all the changes are cool in this
working area we can bundle it all up in a patchset and commit it all
back to the SVN trunk for official release.
All this being said, if there is opposition to this approach, I am
fine with providing someone with patches agains the svn repo for any
changes that I have done and closing down the GitHub version if there
is confusion. I think I made clear the intentions in the readme on
the site though.
> That said, if there is good stuff that comes out of your efforts, I would be
> happy to help you get your changes into the official stomp gem repository.
Cool.
> > Please accompany all code changes with matching Rspec specs, which
> > should be mocked/stubbed so they don't make calls directly to external
> > servers, or to collaborating classes.
> The current gem doesn't use or depend on rspec; I respect that this is how
> you like to work, but I would think it would be more appropriate to offer
> test unit classes, as that is what the gem uses now, and otherwise you end
> up with a mix of units and specs, and that's awkward.
The reason I was setting up the structure for having tests in Rspec is
because:
a) It is a much better testing framework IMHO. :-)
b) It has a built in mocking and stubbing framework so it will be
possible to build tests that:
- Exercise only the class being tested (e.g. the Client and mock/
stub any calls to the Connection class).
- Use mocks/stubs to mock the connection and responses from the
external stomp server. This removes the dependency on having a
message queue running in order to run the tests (and allows you to
mock the expected response that the stomp spec says should be
returned, and not what different implementations of the spec actually
return.
- Rspec provides mechanisms to keep the testing code more DRY.
Notice I was able to remove two of the test unit files completely
since they three files contained all of the same exact tests with only
the setup of the connection being different. This makes maintenance
really hard. Check out my start on Rspec where you'll see I was able
to duplicate the functionality and can run different setups against
the same suite of test cases. You can see that in
http://github.com/grempe/stomp/tree/master/spec/client_spec.rb where I
specify it should 'it_should_behave_like "standard Client"'.
> > TODO : One thing I would personally like to see is an easy way to make
> > a synchronous request to the STOMP server which is non-blocking.
> The way the gem works now is that when you make a request, it blocks until
> that request has been sent to the broker.
> Is what you want to be able to do the send, but then immediately move on,
> regardless of if the request has to be retried, or can even be sent right
> away? If so, it sounds like all you want is to use a different thread to do
> the send, such as putting the send in the block for a Thread.new.
To clarify what I would like to be able to do. I want to be able to
make a call to @connection.somemethod and have it return the next
message on the specified queue. If there is a message, give it to me
and close the connection. If there is no message return nil. As I
look at the comments in the code this is what poll seems to offer
(comments say : # Return a pending message if one is available,
otherwise return nil.). However, if there are no messages on the
queue, poll blocks since its using the same receive method. No? My
expected behavior would wrap the process of opening a connection,
getting a message if avail, and then disconnecting. While this may
not be as efficient as a thread which is listening and not always
opening and tearing down connections it is useful in some cases.
> Anyway, let me know if I have misunderstood, and I look forward to your
> contributions to stomp and messaging,
> - Andrew Kuklewicz
> Happy coding!
Thanks again for the detailed response. Actually much of what you
said should be included in the rdoc comments in the code to make it
clearer for newcomers how some of the code is expected to work.
Perhaps I'll drop some of your comments in as a starting point if you
don't mind. :-)
I was really happy to see Brian's support behind this - I was feeling a bit protective on his behalf, as this was his baby, but with is support, I am totally for what you are doing/have done. To be honest, most of my interest in the gem to date has been to make it work for active messaging and fix bugs, not to go to the lengths you are now; I applaud your efforts.
On Sun, May 11, 2008 at 6:37 PM, Glenn Rempe <glenn.re...@gmail.com> wrote: > My git repo is actually able to be actively connected to both the SVN > master which I would pull your changes from, and then merge them into > my branch which is what I would push to GitHub. So if you or others > commit changes I can get those onto Git and keep everything in synch.
> Or even better :-) You can fork my git repo, and push the work onto > github and we can merge our git repos easily. Or I'll give you commit > access to my git repo directly. Once all the changes are cool in this > working area we can bundle it all up in a patchset and commit it all > back to the SVN trunk for official release.
> All this being said, if there is opposition to this approach, I am > fine with providing someone with patches agains the svn repo for any > changes that I have done and closing down the GitHub version if there > is confusion. I think I made clear the intentions in the readme on > the site though.
I have no personal opposition to what you have done. I reacted a bit conservatively only as Brian had not weighed in yet, and I wanted to make sure he was cool with this.
I'll try to get my changes in using your git rather than svn - I am not using git for any of my own projects yet, so it will be a good excuse to get more familiar.
> The reason I was setting up the structure for having tests in Rspec is > because: > ...
I like consistency - if all the tests are moving to specs, I am cool with it. I agree that mocking is essential in testing code like this.
> To clarify what I would like to be able to do. I want to be able to > make a call to @connection.somemethod and have it return the next > message on the specified queue. If there is a message, give it to me > and close the connection. If there is no message return nil.
The "_receive" private method is passed in a connection, and tries once to get a response frame, then returns nil if there is none. The "receive" public method tries to establish a connection by calling "socket", then uses "_receive" to try once to get a message. Where "receive" can block is when the connection needs to be (re)established, and the connection attribute reliable == true.
If the connection is good, it will therefore try to call "_receive" once only, and so do what you are looking for. If the connection is nil, it will try once to connect if reliable=false, otherwise it will retry until it gets a connection - and therefore blocks. So reliable=false essentially makes "receive" always behave in the non-blocking way that you describe.
Well, that is a slight gloss - "_receive" does block, but only in order to make sure that the entirety of a frame is read of the connection at once - again, you want it to synchronize around this, as you don't want competing threads to each get a piece of a single frame. So that blocking I assume you are not worried about - more it is the blocking retry logic that seems to be your (understandable) concern.
One of the changes I have been working on is cleaning up the way recieve and _receive are used (mostly because I found a bug in the retry logic, but also to make it better and clearer), so this is an area that I believe needs work anyway.
I am generally not a huge fan of the existing retry logic, which we are touching on a bit here. You either have no retries if you have reliable=false, or it retries infinitely (or until success) if reliable=true - I would rather have retry counts with a configurable max retry count - what do you think?
As I
> look at the comments in the code this is what poll seems to offer > (comments say : # Return a pending message if one is available, > otherwise return nil.). However, if there are no messages on the
queue, poll blocks since its using the same receive method. No?
First off - I don't use poll, and neither does the Client that comes with stomp gem; it is an unneeded wrapper on receive as far as I am concerned. It really ought to be deprecated, removed, or repurposed to do something useful. The only thing it does is return nil if there is not a connection already, whereas receive will open a connection if there is none. Never had a reason to use this.
So yeah, that comment is not exactly true, as described above: If reliable=false, then that is how both poll and receive will behave - returning nil if there is no message or it can't make a connection If reliable=true, it keeps retrying until it connects, and then keeps retrying until it returns a message.
> My > expected behavior would wrap the process of opening a connection, > getting a message if avail, and then disconnecting.
Set reliable=false, and that is what calling receive will do, except it does not automatically disconnect. We can add another method perhaps that also does he disconnect, but there should be some method such as receive that does not do this, as disconnecting will get rid of all your existing subscriptions, and you really don't always want to do that - activemessaging relies on the fact that once you have connected and subscribed, it can simply use receive to get messages off without having the overhead of reconnecting and resubscribing.
> While this may > not be as efficient as a thread which is listening and not always > opening and tearing down connections it is useful in some cases.
In some cases, I agree, and in those cases, we can have a different method, or finer grained options for retry logic. ActiveMessaging is built using the assumption of a single connection with subscriptions to receive messages as they come in, and not closing and resubscribing all the time.
Thanks again for the detailed response. Actually much of what you
> said should be included in the rdoc comments in the code to make it > clearer for newcomers how some of the code is expected to work. > Perhaps I'll drop some of your comments in as a starting point if you > don't mind. :-)
Of course not - I've spent alot of time in this code, for better or worse. I hope my long-windedness can be slightly helpful to the new guard.
Just as a brief follow up, I think the cleanup so far is good, and the better tests and docs are excellent. Glenn has really put the gem on good, solid footing for continued improvement and enhancement.
-Andrew
On Mon, May 12, 2008 at 1:06 PM, Andrew Kuklewicz <kooks...@gmail.com> wrote:
> I was really happy to see Brian's support behind this - I was feeling a bit > protective on his behalf, as this was his baby, but with is support, I am > totally for what you are doing/have done. To be honest, most of my interest > in the gem to date has been to make it work for active messaging and fix > bugs, not to go to the lengths you are now; I applaud your efforts.
> A few comments below:
> On Sun, May 11, 2008 at 6:37 PM, Glenn Rempe <glenn.re...@gmail.com> > wrote:
>> My git repo is actually able to be actively connected to both the SVN >> master which I would pull your changes from, and then merge them into >> my branch which is what I would push to GitHub. So if you or others >> commit changes I can get those onto Git and keep everything in synch.
>> Or even better :-) You can fork my git repo, and push the work onto >> github and we can merge our git repos easily. Or I'll give you commit >> access to my git repo directly. Once all the changes are cool in this >> working area we can bundle it all up in a patchset and commit it all >> back to the SVN trunk for official release.
>> All this being said, if there is opposition to this approach, I am >> fine with providing someone with patches agains the svn repo for any >> changes that I have done and closing down the GitHub version if there >> is confusion. I think I made clear the intentions in the readme on >> the site though.
> I have no personal opposition to what you have done. I reacted a bit > conservatively only as Brian had not weighed in yet, and I wanted to make > sure he was cool with this.
> I'll try to get my changes in using your git rather than svn - I am not > using git for any of my own projects yet, so it will be a good excuse to get > more familiar.
>> The reason I was setting up the structure for having tests in Rspec is >> because: >> ...
> I like consistency - if all the tests are moving to specs, I am cool with > it. > I agree that mocking is essential in testing code like this.
>> To clarify what I would like to be able to do. I want to be able to >> make a call to @connection.somemethod and have it return the next >> message on the specified queue. If there is a message, give it to me >> and close the connection. If there is no message return nil.
> The "_receive" private method is passed in a connection, and tries once to > get a response frame, then returns nil if there is none. > The "receive" public method tries to establish a connection by calling > "socket", then uses "_receive" to try once to get a message. > Where "receive" can block is when the connection needs to be > (re)established, and the connection attribute reliable == true.
> If the connection is good, it will therefore try to call "_receive" once > only, and so do what you are looking for. > If the connection is nil, it will try once to connect if reliable=false, > otherwise it will retry until it gets a connection - and therefore blocks. > So reliable=false essentially makes "receive" always behave in the > non-blocking way that you describe.
> Well, that is a slight gloss - "_receive" does block, but only in order to > make sure that the entirety of a frame is read of the connection at once - > again, you want it to synchronize around this, as you don't want competing > threads to each get a piece of a single frame. So that blocking I assume > you are not worried about - more it is the blocking retry logic that seems > to be your (understandable) concern.
> One of the changes I have been working on is cleaning up the way recieve > and _receive are used (mostly because I found a bug in the retry logic, but > also to make it better and clearer), so this is an area that I believe needs > work anyway.
> I am generally not a huge fan of the existing retry logic, which we are > touching on a bit here. > You either have no retries if you have reliable=false, or it retries > infinitely (or until success) if reliable=true - I would rather have retry > counts with a configurable max retry count - what do you think?
> As I >> look at the comments in the code this is what poll seems to offer >> (comments say : # Return a pending message if one is available, >> otherwise return nil.). However, if there are no messages on the
> queue, poll blocks since its using the same receive method. No?
> First off - I don't use poll, and neither does the Client that comes with > stomp gem; it is an unneeded wrapper on receive as far as I am concerned. > It really ought to be deprecated, removed, or repurposed to do something > useful. The only thing it does is return nil if there is not a connection > already, whereas receive will open a connection if there is none. Never had > a reason to use this.
> So yeah, that comment is not exactly true, as described above: > If reliable=false, then that is how both poll and receive will behave - > returning nil if there is no message or it can't make a connection > If reliable=true, it keeps retrying until it connects, and then keeps > retrying until it returns a message.
>> My >> expected behavior would wrap the process of opening a connection, >> getting a message if avail, and then disconnecting.
> Set reliable=false, and that is what calling receive will do, except it > does not automatically disconnect. > We can add another method perhaps that also does he disconnect, but there > should be some method such as receive that does not do this, as > disconnecting will get rid of all your existing subscriptions, and you > really don't always want to do that - activemessaging relies on the fact > that once you have connected and subscribed, it can simply use receive to > get messages off without having the overhead of reconnecting and > resubscribing.
>> While this may >> not be as efficient as a thread which is listening and not always >> opening and tearing down connections it is useful in some cases.
> In some cases, I agree, and in those cases, we can have a different method, > or finer grained options for retry logic. ActiveMessaging is built using > the assumption of a single connection with subscriptions to receive messages > as they come in, and not closing and resubscribing all the time.
> Thanks again for the detailed response. Actually much of what you >> said should be included in the rdoc comments in the code to make it >> clearer for newcomers how some of the code is expected to work. >> Perhaps I'll drop some of your comments in as a starting point if you >> don't mind. :-)
> Of course not - I've spent alot of time in this code, for better or worse. > I hope my long-windedness can be slightly helpful to the new guard.
One thing I would really like to see is low-level connection failover
within the Stomp gem itself. In my current hacked-up copy of
stomp.rb, I'm accomplishing this by storing an array of connection
objects instead of host/port variables. I then increment an index
into that array every time I try a reconnect within socket().
Basically, my socket() has an @hosts array and @host_index, and does
something like this:
If there's a better list for this discussion please let me know, I'm
very interested in seeing a robust Stomp failover just like JMS but I
haven't seen a good discussion group for the stomp gem itself.