I don't think I'm doing anything wrong in the code itself. For your
information I have 498 podcasts and currently 17961 episodes. I'm
pretty sure the script runs for a few days before somehow dying.
Anybody else experiencing this?
Cheers.
-- You received this message because you are subscribed to the Google Groups "feedzirra" group.
To post to this group, send email to feedzirra@googlegroups.com.
To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
Check that its not running out of memory. god is useful for to keeping
things running.
You might also want to checkout spawn and fork the process before doing a
loop and wait for it to complete. This will ensure your memory is release
after each loop, and automatically handle the ActiveRecord database
connection (which you may also be loosing after a period of time.)
Cheers,
sam
On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
> I don't think I'm doing anything wrong in the code itself. For your
> information I have 498 podcasts and currently 17961 episodes. I'm
> pretty sure the script runs for a few days before somehow dying.
> Anybody else experiencing this?
> Cheers.
> --
> You received this message because you are subscribed to the Google Groups
> "feedzirra" group.
> To post to this group, send email to feedzirra@googlegroups.com.
> To unsubscribe from this group, send email to
> feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com>
> .
> For more options, visit this group at
> http://groups.google.com/group/feedzirra?hl=en.
-- You received this message because you are subscribed to the Google Groups "feedzirra" group.
To post to this group, send email to feedzirra@googlegroups.com.
To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
thanks for the tips. I had thought about spawn & god but what's the
point of a daemon if the connection to Active Record goes away?! Also
how come the memory isn't freed. Can I call "something" in my loop to
"clean up"? Do I have to?
On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote:
> Hi,
> Check that its not running out of memory. god is useful for to keeping
> things running.
> You might also want to checkout spawn and fork the process before doing a
> loop and wait for it to complete. This will ensure your memory is release
> after each loop, and automatically handle the ActiveRecord database
> connection (which you may also be loosing after a period of time.)
> Cheers,
> sam
> On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>> I don't think I'm doing anything wrong in the code itself. For your
>> information I have 498 podcasts and currently 17961 episodes. I'm
>> pretty sure the script runs for a few days before somehow dying.
>> Anybody else experiencing this?
>> Cheers.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "feedzirra" group.
>> To post to this group, send email to feedzirra@googlegroups.com.
>> To unsubscribe from this group, send email to
>> feedzirra+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/feedzirra?hl=en.
> --
> You received this message because you are subscribed to the Google Groups
> "feedzirra" group.
> To post to this group, send email to feedzirra@googlegroups.com.
> To unsubscribe from this group, send email to
> feedzirra+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/feedzirra?hl=en.
-- You received this message because you are subscribed to the Google Groups "feedzirra" group.
To post to this group, send email to feedzirra@googlegroups.com.
To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
> thanks for the tips. I had thought about spawn & god but what's the
> point of a daemon if the connection to Active Record goes away?!
The DB can timeout a connection if its not being used or cut-out on network
failure. This probably isn't a problem in your situation though. It would be
if you were forking, as closing a fork will also close the DB connection.
> Also
> how come the memory isn't freed.
You can try some profiling techniques if you want to discover where your
memory is leaking. You're reading lots of external data and passing it
through several libraries, there could be tiny leaks anywhere.
I suspect most people give-up on this. Ruby makes programming easier at the
cost of less-refined memory management, this is why god scripts exist to
restart mongrel or unicorn processes to free-up trapped memory.
Can I call "something" in my loop to
> "clean up"? Do I have to?
You can try forcing garbage collection, but I doubt that will solve much:
> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote:
> > Hi,
> > Check that its not running out of memory. god is useful for to keeping
> > things running.
> > You might also want to checkout spawn and fork the process before doing a
> > loop and wait for it to complete. This will ensure your memory is release
> > after each loop, and automatically handle the ActiveRecord database
> > connection (which you may also be loosing after a period of time.)
> > Cheers,
> > sam
> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
> >> I don't think I'm doing anything wrong in the code itself. For your
> >> information I have 498 podcasts and currently 17961 episodes. I'm
> >> pretty sure the script runs for a few days before somehow dying.
> >> Anybody else experiencing this?
> >> Cheers.
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "feedzirra" group.
> >> To post to this group, send email to feedzirra@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com>
> .
> >> For more options, visit this group at
> >> http://groups.google.com/group/feedzirra?hl=en.
> > --
> > You received this message because you are subscribed to the Google Groups
> > "feedzirra" group.
> > To post to this group, send email to feedzirra@googlegroups.com.
> > To unsubscribe from this group, send email to
> > feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/feedzirra?hl=en.
> --
> You received this message because you are subscribed to the Google Groups
> "feedzirra" group.
> To post to this group, send email to feedzirra@googlegroups.com.
> To unsubscribe from this group, send email to
> feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com>
> .
> For more options, visit this group at
> http://groups.google.com/group/feedzirra?hl=en.
-- You received this message because you are subscribed to the Google Groups "feedzirra" group.
To post to this group, send email to feedzirra@googlegroups.com.
To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>> thanks for the tips. I had thought about spawn & god but what's the >> point of a daemon if the connection to Active Record goes away?!
> The DB can timeout a connection if its not being used or cut-out on network > failure. This probably isn't a problem in your situation though. It would be > if you were forking, as closing a fork will also close the DB connection.
>> Also >> how come the memory isn't freed.
> You can try some profiling techniques if you want to discover where your > memory is leaking. You're reading lots of external data and passing it > through several libraries, there could be tiny leaks anywhere.
> I suspect most people give-up on this. Ruby makes programming easier at the > cost of less-refined memory management, this is why god scripts exist to > restart mongrel or unicorn processes to free-up trapped memory.
>> Can I call "something" in my loop to >> "clean up"? Do I have to?
> You can try forcing garbage collection, but I doubt that will solve much:
> Seriously, if you're handling lots of data in a daemon during a long period > of time, fork it, process, clean.
> Cheers, > sam
>> Cheers.
>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: >> > Hi, >> > Check that its not running out of memory. god is useful for to keeping >> > things running. >> > You might also want to checkout spawn and fork the process before doing >> > a >> > loop and wait for it to complete. This will ensure your memory is >> > release >> > after each loop, and automatically handle the ActiveRecord database >> > connection (which you may also be loosing after a period of time.) >> > Cheers, >> > sam >> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>> >> I don't think I'm doing anything wrong in the code itself. For your >> >> information I have 498 podcasts and currently 17961 episodes. I'm >> >> pretty sure the script runs for a few days before somehow dying.
>> >> Anybody else experiencing this?
>> >> Cheers.
>> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "feedzirra" group. >> >> To post to this group, send email to feedzirra@googlegroups.com. >> >> To unsubscribe from this group, send email to >> >> feedzirra+unsubscribe@googlegroups.com. >> >> For more options, visit this group at >> >> http://groups.google.com/group/feedzirra?hl=en.
>> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "feedzirra" group. >> > To post to this group, send email to feedzirra@googlegroups.com. >> > To unsubscribe from this group, send email to >> > feedzirra+unsubscribe@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/feedzirra?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "feedzirra" group. >> To post to this group, send email to feedzirra@googlegroups.com. >> To unsubscribe from this group, send email to >> feedzirra+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/feedzirra?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "feedzirra" group. > To post to this group, send email to feedzirra@googlegroups.com. > To unsubscribe from this group, send email to > feedzirra+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/feedzirra?hl=en.
Is the daemon failing with some sort of error? I assume that it's just not failing quietly. The graph doesn't really do anything to help troubleshoot. Is there a reason you're doing a require inside the "add_entries" method? I don't think you want to be rerunning that everytime you run it.
> This graph shows how nuts the process is going (before to be killed I > guess) after the daemons has been running for about a day or 2.
> I haven't take the time yet to look at how God can help me - the last > time I looked at God it wasn't trivial :(
> But from the Feedzirra perspective (and thus, this is a question to > the main dev), where could be the problem?
> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>> thanks for the tips. I had thought about spawn & god but what's the >>> point of a daemon if the connection to Active Record goes away?!
>> The DB can timeout a connection if its not being used or cut-out on network >> failure. This probably isn't a problem in your situation though. It would be >> if you were forking, as closing a fork will also close the DB connection.
>>> Also >>> how come the memory isn't freed.
>> You can try some profiling techniques if you want to discover where your >> memory is leaking. You're reading lots of external data and passing it >> through several libraries, there could be tiny leaks anywhere.
>> I suspect most people give-up on this. Ruby makes programming easier at the >> cost of less-refined memory management, this is why god scripts exist to >> restart mongrel or unicorn processes to free-up trapped memory.
>>> Can I call "something" in my loop to >>> "clean up"? Do I have to?
>> You can try forcing garbage collection, but I doubt that will solve much:
>> Seriously, if you're handling lots of data in a daemon during a long period >> of time, fork it, process, clean.
>> Cheers, >> sam
>>> Cheers.
>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: >>> > Hi, >>> > Check that its not running out of memory. god is useful for to keeping >>> > things running. >>> > You might also want to checkout spawn and fork the process before doing >>> > a >>> > loop and wait for it to complete. This will ensure your memory is >>> > release >>> > after each loop, and automatically handle the ActiveRecord database >>> > connection (which you may also be loosing after a period of time.) >>> > Cheers, >>> > sam >>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>> >> I don't think I'm doing anything wrong in the code itself. For your >>> >> information I have 498 podcasts and currently 17961 episodes. I'm >>> >> pretty sure the script runs for a few days before somehow dying.
>>> >> Anybody else experiencing this?
>>> >> Cheers.
>>> >> -- >>> >> You received this message because you are subscribed to the Google >>> >> Groups >>> >> "feedzirra" group. >>> >> To post to this group, send email to feedzirra@googlegroups.com. >>> >> To unsubscribe from this group, send email to >>> >> feedzirra+unsubscribe@googlegroups.com. >>> >> For more options, visit this group at >>> >> http://groups.google.com/group/feedzirra?hl=en.
>>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "feedzirra" group. >>> > To post to this group, send email to feedzirra@googlegroups.com. >>> > To unsubscribe from this group, send email to >>> > feedzirra+unsubscribe@googlegroups.com. >>> > For more options, visit this group at >>> > http://groups.google.com/group/feedzirra?hl=en.
>>> -- >>> You received this message because you are subscribed to the Google Groups >>> "feedzirra" group. >>> To post to this group, send email to feedzirra@googlegroups.com. >>> To unsubscribe from this group, send email to >>> feedzirra+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/feedzirra?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "feedzirra" group. >> To post to this group, send email to feedzirra@googlegroups.com. >> To unsubscribe from this group, send email to >> feedzirra+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/feedzirra?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "feedzirra" group. > To post to this group, send email to feedzirra@googlegroups.com. > To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
True, it's pretty dumb to have this require there. I remember having tried to have it in my environment.rb but was unsuccessful. Anyways, I moved it to the top of the source file (so outside the method) and we'll see if it changes anything.
Concerning the logs, they are supposed to be in /log, right? There nothing relevant there :(
On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: > Is the daemon failing with some sort of error? I assume that it's just > not failing quietly. The graph doesn't really do anything to help > troubleshoot. Is there a reason you're doing a require inside the > "add_entries" method? I don't think you want to be rerunning that > everytime you run it.
>> This graph shows how nuts the process is going (before to be killed I >> guess) after the daemons has been running for about a day or 2.
>> I haven't take the time yet to look at how God can help me - the last >> time I looked at God it wasn't trivial :(
>> But from the Feedzirra perspective (and thus, this is a question to >> the main dev), where could be the problem?
>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>>> thanks for the tips. I had thought about spawn & god but what's the >>>> point of a daemon if the connection to Active Record goes away?!
>>> The DB can timeout a connection if its not being used or cut-out on network >>> failure. This probably isn't a problem in your situation though. It would be >>> if you were forking, as closing a fork will also close the DB connection.
>>>> Also >>>> how come the memory isn't freed.
>>> You can try some profiling techniques if you want to discover where your >>> memory is leaking. You're reading lots of external data and passing it >>> through several libraries, there could be tiny leaks anywhere.
>>> I suspect most people give-up on this. Ruby makes programming easier at the >>> cost of less-refined memory management, this is why god scripts exist to >>> restart mongrel or unicorn processes to free-up trapped memory.
>>>> Can I call "something" in my loop to >>>> "clean up"? Do I have to?
>>> You can try forcing garbage collection, but I doubt that will solve much:
>>> Seriously, if you're handling lots of data in a daemon during a long period >>> of time, fork it, process, clean.
>>> Cheers, >>> sam
>>>> Cheers.
>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: >>>> > Hi, >>>> > Check that its not running out of memory. god is useful for to keeping >>>> > things running. >>>> > You might also want to checkout spawn and fork the process before doing >>>> > a >>>> > loop and wait for it to complete. This will ensure your memory is >>>> > release >>>> > after each loop, and automatically handle the ActiveRecord database >>>> > connection (which you may also be loosing after a period of time.) >>>> > Cheers, >>>> > sam >>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>>> >> I don't think I'm doing anything wrong in the code itself. For your >>>> >> information I have 498 podcasts and currently 17961 episodes. I'm >>>> >> pretty sure the script runs for a few days before somehow dying.
>>>> >> Anybody else experiencing this?
>>>> >> Cheers.
>>>> >> -- >>>> >> You received this message because you are subscribed to the Google >>>> >> Groups >>>> >> "feedzirra" group. >>>> >> To post to this group, send email to feedzirra@googlegroups.com. >>>> >> To unsubscribe from this group, send email to >>>> >> feedzirra+unsubscribe@googlegroups.com. >>>> >> For more options, visit this group at >>>> >> http://groups.google.com/group/feedzirra?hl=en.
>>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> > Groups >>>> > "feedzirra" group. >>>> > To post to this group, send email to feedzirra@googlegroups.com. >>>> > To unsubscribe from this group, send email to >>>> > feedzirra+unsubscribe@googlegroups.com. >>>> > For more options, visit this group at >>>> > http://groups.google.com/group/feedzirra?hl=en.
>>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "feedzirra" group. >>>> To post to this group, send email to feedzirra@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> feedzirra+unsubscribe@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/feedzirra?hl=en.
>>> -- >>> You received this message because you are subscribed to the Google Groups >>> "feedzirra" group. >>> To post to this group, send email to feedzirra@googlegroups.com. >>> To unsubscribe from this group, send email to >>> feedzirra+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/feedzirra?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups "feedzirra" group. >> To post to this group, send email to feedzirra@googlegroups.com. >> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "feedzirra" group. > To post to this group, send email to feedzirra@googlegroups.com. > To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
I just use the rubygem "daemons" and it's supposed to output in log/name_of_the_daemon.rb.log but the last errors I had there were from March.
Attached a chart that shows when I restarted the daemon. We'll see over time if the require was the problem. Aside the daemon, would you have a good tip to profile my "Episode.fetch_all"?
On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: > It depends on how/what you're using to daemonize the process that is > running these updates. Whatever it is should have the ability to write > STDOUT and STDERR to some log file. If you aren't specifying that then > the process is probably just throwing an exception that isn't being > captured by anything.
> Best, > Paul
> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: >> Thanks for answering so quickly Paul.
>> True, it's pretty dumb to have this require there. I remember having >> tried to have it in my environment.rb but was unsuccessful. Anyways, I >> moved it to the top of the source file (so outside the method) and >> we'll see if it changes anything.
>> Concerning the logs, they are supposed to be in /log, right? There >> nothing relevant there :(
>> Cheers.
>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: >>> Is the daemon failing with some sort of error? I assume that it's just >>> not failing quietly. The graph doesn't really do anything to help >>> troubleshoot. Is there a reason you're doing a require inside the >>> "add_entries" method? I don't think you want to be rerunning that >>> everytime you run it.
>>>> This graph shows how nuts the process is going (before to be killed I >>>> guess) after the daemons has been running for about a day or 2.
>>>> I haven't take the time yet to look at how God can help me - the last >>>> time I looked at God it wasn't trivial :(
>>>> But from the Feedzirra perspective (and thus, this is a question to >>>> the main dev), where could be the problem?
>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>>>>> thanks for the tips. I had thought about spawn & god but what's the >>>>>> point of a daemon if the connection to Active Record goes away?!
>>>>> The DB can timeout a connection if its not being used or cut-out on network >>>>> failure. This probably isn't a problem in your situation though. It would be >>>>> if you were forking, as closing a fork will also close the DB connection.
>>>>>> Also >>>>>> how come the memory isn't freed.
>>>>> You can try some profiling techniques if you want to discover where your >>>>> memory is leaking. You're reading lots of external data and passing it >>>>> through several libraries, there could be tiny leaks anywhere.
>>>>> I suspect most people give-up on this. Ruby makes programming easier at the >>>>> cost of less-refined memory management, this is why god scripts exist to >>>>> restart mongrel or unicorn processes to free-up trapped memory.
>>>>>> Can I call "something" in my loop to >>>>>> "clean up"? Do I have to?
>>>>> You can try forcing garbage collection, but I doubt that will solve much:
>>>>> Seriously, if you're handling lots of data in a daemon during a long period >>>>> of time, fork it, process, clean.
>>>>> Cheers, >>>>> sam
>>>>>> Cheers.
>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: >>>>>> > Hi, >>>>>> > Check that its not running out of memory. god is useful for to keeping >>>>>> > things running. >>>>>> > You might also want to checkout spawn and fork the process before doing >>>>>> > a >>>>>> > loop and wait for it to complete. This will ensure your memory is >>>>>> > release >>>>>> > after each loop, and automatically handle the ActiveRecord database >>>>>> > connection (which you may also be loosing after a period of time.) >>>>>> > Cheers, >>>>>> > sam >>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>>>>> >> I don't think I'm doing anything wrong in the code itself. For your >>>>>> >> information I have 498 podcasts and currently 17961 episodes. I'm >>>>>> >> pretty sure the script runs for a few days before somehow dying.
>>>>>> >> Anybody else experiencing this?
>>>>>> >> Cheers.
>>>>>> >> -- >>>>>> >> You received this message because you are subscribed to the Google >>>>>> >> Groups >>>>>> >> "feedzirra" group. >>>>>> >> To post to this group, send email to feedzirra@googlegroups.com. >>>>>> >> To unsubscribe from this group, send email to >>>>>> >> feedzirra+unsubscribe@googlegroups.com. >>>>>> >> For more options, visit this group at >>>>>> >> http://groups.google.com/group/feedzirra?hl=en.
>>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> > Groups >>>>>> > "feedzirra" group. >>>>>> > To post to this group, send email to feedzirra@googlegroups.com. >>>>>> > To unsubscribe from this group, send email to >>>>>> > feedzirra+unsubscribe@googlegroups.com. >>>>>> > For more options, visit this group at >>>>>> > http://groups.google.com/group/feedzirra?hl=en.
>>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups >>>>>> "feedzirra" group. >>>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>>> To unsubscribe from this group, send email to >>>>>> feedzirra+unsubscribe@googlegroups.com. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/feedzirra?hl=en.
>>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "feedzirra" group. >>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>> To unsubscribe from this group, send email to >>>>> feedzirra+unsubscribe@googlegroups.com. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/feedzirra?hl=en.
>>>> -- >>>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>>> To post to this group, send email to feedzirra@googlegroups.com. >>>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>>> -- >>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>> To post to this group, send email to feedzirra@googlegroups.com. >>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups "feedzirra" group. >> To post to this group, send email to feedzirra@googlegroups.com. >> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "feedzirra" group. > To post to this group, send email to feedzirra@googlegroups.com. > To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
On Wed, May 26, 2010 at 5:04 PM, StuFF mc <m...@stuffmc.com> wrote: > I just use the rubygem "daemons" and it's supposed to output in > log/name_of_the_daemon.rb.log but the last errors I had there were > from March.
> Attached a chart that shows when I restarted the daemon. We'll see > over time if the require was the problem. Aside the daemon, would you > have a good tip to profile my "Episode.fetch_all"?
> script/performance isn't much helpful i.m.h.o.
> On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: >> It depends on how/what you're using to daemonize the process that is >> running these updates. Whatever it is should have the ability to write >> STDOUT and STDERR to some log file. If you aren't specifying that then >> the process is probably just throwing an exception that isn't being >> captured by anything.
>> Best, >> Paul
>> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: >>> Thanks for answering so quickly Paul.
>>> True, it's pretty dumb to have this require there. I remember having >>> tried to have it in my environment.rb but was unsuccessful. Anyways, I >>> moved it to the top of the source file (so outside the method) and >>> we'll see if it changes anything.
>>> Concerning the logs, they are supposed to be in /log, right? There >>> nothing relevant there :(
>>> Cheers.
>>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: >>>> Is the daemon failing with some sort of error? I assume that it's just >>>> not failing quietly. The graph doesn't really do anything to help >>>> troubleshoot. Is there a reason you're doing a require inside the >>>> "add_entries" method? I don't think you want to be rerunning that >>>> everytime you run it.
>>>> Log the output from that daemon.
>>>> Best, >>>> Paul
>>>> On Wed, May 26, 2010 at 4:00 PM, StuFF mc <m...@stuffmc.com> wrote: >>>>> Since I'm not sure you can add images to this group, it's also here: >>>>> http://emberapp.com/stuffmc/images/cpu-graph-rp
>>>>> This graph shows how nuts the process is going (before to be killed I >>>>> guess) after the daemons has been running for about a day or 2.
>>>>> I haven't take the time yet to look at how God can help me - the last >>>>> time I looked at God it wasn't trivial :(
>>>>> But from the Feedzirra perspective (and thus, this is a question to >>>>> the main dev), where could be the problem?
>>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
>>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>>>>>> thanks for the tips. I had thought about spawn & god but what's the >>>>>>> point of a daemon if the connection to Active Record goes away?!
>>>>>> The DB can timeout a connection if its not being used or cut-out on network >>>>>> failure. This probably isn't a problem in your situation though. It would be >>>>>> if you were forking, as closing a fork will also close the DB connection.
>>>>>>> Also >>>>>>> how come the memory isn't freed.
>>>>>> You can try some profiling techniques if you want to discover where your >>>>>> memory is leaking. You're reading lots of external data and passing it >>>>>> through several libraries, there could be tiny leaks anywhere.
>>>>>> I suspect most people give-up on this. Ruby makes programming easier at the >>>>>> cost of less-refined memory management, this is why god scripts exist to >>>>>> restart mongrel or unicorn processes to free-up trapped memory.
>>>>>>> Can I call "something" in my loop to >>>>>>> "clean up"? Do I have to?
>>>>>> You can try forcing garbage collection, but I doubt that will solve much:
>>>>>> Seriously, if you're handling lots of data in a daemon during a long period >>>>>> of time, fork it, process, clean.
>>>>>> Cheers, >>>>>> sam
>>>>>>> Cheers.
>>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: >>>>>>> > Hi, >>>>>>> > Check that its not running out of memory. god is useful for to keeping >>>>>>> > things running. >>>>>>> > You might also want to checkout spawn and fork the process before doing >>>>>>> > a >>>>>>> > loop and wait for it to complete. This will ensure your memory is >>>>>>> > release >>>>>>> > after each loop, and automatically handle the ActiveRecord database >>>>>>> > connection (which you may also be loosing after a period of time.) >>>>>>> > Cheers, >>>>>>> > sam >>>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>>>>>> >> I don't think I'm doing anything wrong in the code itself. For your >>>>>>> >> information I have 498 podcasts and currently 17961 episodes. I'm >>>>>>> >> pretty sure the script runs for a few days before somehow dying.
>>>>>>> >> Anybody else experiencing this?
>>>>>>> >> Cheers.
>>>>>>> >> -- >>>>>>> >> You received this message because you are subscribed to the Google >>>>>>> >> Groups >>>>>>> >> "feedzirra" group. >>>>>>> >> To post to this group, send email to feedzirra@googlegroups.com. >>>>>>> >> To unsubscribe from this group, send email to >>>>>>> >> feedzirra+unsubscribe@googlegroups.com. >>>>>>> >> For more options, visit this group at >>>>>>> >> http://groups.google.com/group/feedzirra?hl=en.
>>>>>>> > -- >>>>>>> > You received this message because you are subscribed to the Google >>>>>>> > Groups >>>>>>> > "feedzirra" group. >>>>>>> > To post to this group, send email to feedzirra@googlegroups.com. >>>>>>> > To unsubscribe from this group, send email to >>>>>>> > feedzirra+unsubscribe@googlegroups.com. >>>>>>> > For more options, visit this group at >>>>>>> > http://groups.google.com/group/feedzirra?hl=en.
>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google Groups >>>>>>> "feedzirra" group. >>>>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>>>> To unsubscribe from this group, send email to >>>>>>> feedzirra+unsubscribe@googlegroups.com. >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/feedzirra?hl=en.
>>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups >>>>>> "feedzirra" group. >>>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>>> To unsubscribe from this group, send email to >>>>>> feedzirra+unsubscribe@googlegroups.com. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/feedzirra?hl=en.
>>>>> -- >>>>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>>>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>>>> -- >>>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>>> To post to this group, send email to feedzirra@googlegroups.com. >>>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>>> -- >>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>> To post to this group, send email to feedzirra@googlegroups.com. >>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups "feedzirra" group. >> To post to this group, send email to feedzirra@googlegroups.com. >> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "feedzirra" group. > To post to this group, send email to feedzirra@googlegroups.com. > To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
It depends on how/what you're using to daemonize the process that is running these updates. Whatever it is should have the ability to write STDOUT and STDERR to some log file. If you aren't specifying that then the process is probably just throwing an exception that isn't being captured by anything.
On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: > Thanks for answering so quickly Paul.
> True, it's pretty dumb to have this require there. I remember having > tried to have it in my environment.rb but was unsuccessful. Anyways, I > moved it to the top of the source file (so outside the method) and > we'll see if it changes anything.
> Concerning the logs, they are supposed to be in /log, right? There > nothing relevant there :(
> Cheers.
> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: >> Is the daemon failing with some sort of error? I assume that it's just >> not failing quietly. The graph doesn't really do anything to help >> troubleshoot. Is there a reason you're doing a require inside the >> "add_entries" method? I don't think you want to be rerunning that >> everytime you run it.
>>> This graph shows how nuts the process is going (before to be killed I >>> guess) after the daemons has been running for about a day or 2.
>>> I haven't take the time yet to look at how God can help me - the last >>> time I looked at God it wasn't trivial :(
>>> But from the Feedzirra perspective (and thus, this is a question to >>> the main dev), where could be the problem?
>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>>>> thanks for the tips. I had thought about spawn & god but what's the >>>>> point of a daemon if the connection to Active Record goes away?!
>>>> The DB can timeout a connection if its not being used or cut-out on network >>>> failure. This probably isn't a problem in your situation though. It would be >>>> if you were forking, as closing a fork will also close the DB connection.
>>>>> Also >>>>> how come the memory isn't freed.
>>>> You can try some profiling techniques if you want to discover where your >>>> memory is leaking. You're reading lots of external data and passing it >>>> through several libraries, there could be tiny leaks anywhere.
>>>> I suspect most people give-up on this. Ruby makes programming easier at the >>>> cost of less-refined memory management, this is why god scripts exist to >>>> restart mongrel or unicorn processes to free-up trapped memory.
>>>>> Can I call "something" in my loop to >>>>> "clean up"? Do I have to?
>>>> You can try forcing garbage collection, but I doubt that will solve much:
>>>> Seriously, if you're handling lots of data in a daemon during a long period >>>> of time, fork it, process, clean.
>>>> Cheers, >>>> sam
>>>>> Cheers.
>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: >>>>> > Hi, >>>>> > Check that its not running out of memory. god is useful for to keeping >>>>> > things running. >>>>> > You might also want to checkout spawn and fork the process before doing >>>>> > a >>>>> > loop and wait for it to complete. This will ensure your memory is >>>>> > release >>>>> > after each loop, and automatically handle the ActiveRecord database >>>>> > connection (which you may also be loosing after a period of time.) >>>>> > Cheers, >>>>> > sam >>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>>>> >> I don't think I'm doing anything wrong in the code itself. For your >>>>> >> information I have 498 podcasts and currently 17961 episodes. I'm >>>>> >> pretty sure the script runs for a few days before somehow dying.
>>>>> >> Anybody else experiencing this?
>>>>> >> Cheers.
>>>>> >> -- >>>>> >> You received this message because you are subscribed to the Google >>>>> >> Groups >>>>> >> "feedzirra" group. >>>>> >> To post to this group, send email to feedzirra@googlegroups.com. >>>>> >> To unsubscribe from this group, send email to >>>>> >> feedzirra+unsubscribe@googlegroups.com. >>>>> >> For more options, visit this group at >>>>> >> http://groups.google.com/group/feedzirra?hl=en.
>>>>> > -- >>>>> > You received this message because you are subscribed to the Google >>>>> > Groups >>>>> > "feedzirra" group. >>>>> > To post to this group, send email to feedzirra@googlegroups.com. >>>>> > To unsubscribe from this group, send email to >>>>> > feedzirra+unsubscribe@googlegroups.com. >>>>> > For more options, visit this group at >>>>> > http://groups.google.com/group/feedzirra?hl=en.
>>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "feedzirra" group. >>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>> To unsubscribe from this group, send email to >>>>> feedzirra+unsubscribe@googlegroups.com. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/feedzirra?hl=en.
>>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "feedzirra" group. >>>> To post to this group, send email to feedzirra@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> feedzirra+unsubscribe@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/feedzirra?hl=en.
>>> -- >>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>> To post to this group, send email to feedzirra@googlegroups.com. >>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups "feedzirra" group. >> To post to this group, send email to feedzirra@googlegroups.com. >> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "feedzirra" group. > To post to this group, send email to feedzirra@googlegroups.com. > To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
Guys, discussing with a few folks while being at http://railswaycon.de (right now) they told me using a Daemon wasn't the best way of doing it. They recommend putting the process in a cron so it's started from fresh everytime. The problem is that I'd like to "fetch" to fetch non stop, this way being as synced as possible (right now worst case I have the podcast 20 minutes after they are published).
I would need the "new/next" cron to only start when the previous is finished. I was thinking to maybe program the next cron 1 minute after the current is finished. Any Ideas/recommandations?
> These types of errors are a real time suck. Hardest to track down. :(
> On Wed, May 26, 2010 at 5:04 PM, StuFF mc <m...@stuffmc.com> wrote: >> I just use the rubygem "daemons" and it's supposed to output in >> log/name_of_the_daemon.rb.log but the last errors I had there were >> from March.
>> Attached a chart that shows when I restarted the daemon. We'll see >> over time if the require was the problem. Aside the daemon, would you >> have a good tip to profile my "Episode.fetch_all"?
>> script/performance isn't much helpful i.m.h.o.
>> On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: >>> It depends on how/what you're using to daemonize the process that is >>> running these updates. Whatever it is should have the ability to write >>> STDOUT and STDERR to some log file. If you aren't specifying that then >>> the process is probably just throwing an exception that isn't being >>> captured by anything.
>>> Best, >>> Paul
>>> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: >>>> Thanks for answering so quickly Paul.
>>>> True, it's pretty dumb to have this require there. I remember having >>>> tried to have it in my environment.rb but was unsuccessful. Anyways, I >>>> moved it to the top of the source file (so outside the method) and >>>> we'll see if it changes anything.
>>>> Concerning the logs, they are supposed to be in /log, right? There >>>> nothing relevant there :(
>>>> Cheers.
>>>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: >>>>> Is the daemon failing with some sort of error? I assume that it's just >>>>> not failing quietly. The graph doesn't really do anything to help >>>>> troubleshoot. Is there a reason you're doing a require inside the >>>>> "add_entries" method? I don't think you want to be rerunning that >>>>> everytime you run it.
>>>>> Log the output from that daemon.
>>>>> Best, >>>>> Paul
>>>>> On Wed, May 26, 2010 at 4:00 PM, StuFF mc <m...@stuffmc.com> wrote: >>>>>> Since I'm not sure you can add images to this group, it's also here: >>>>>> http://emberapp.com/stuffmc/images/cpu-graph-rp
>>>>>> This graph shows how nuts the process is going (before to be killed I >>>>>> guess) after the daemons has been running for about a day or 2.
>>>>>> I haven't take the time yet to look at how God can help me - the last >>>>>> time I looked at God it wasn't trivial :(
>>>>>> But from the Feedzirra perspective (and thus, this is a question to >>>>>> the main dev), where could be the problem?
>>>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
>>>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>>>>>>> thanks for the tips. I had thought about spawn & god but what's the >>>>>>>> point of a daemon if the connection to Active Record goes away?!
>>>>>>> The DB can timeout a connection if its not being used or cut-out on network >>>>>>> failure. This probably isn't a problem in your situation though. It would be >>>>>>> if you were forking, as closing a fork will also close the DB connection.
>>>>>>>> Also >>>>>>>> how come the memory isn't freed.
>>>>>>> You can try some profiling techniques if you want to discover where your >>>>>>> memory is leaking. You're reading lots of external data and passing it >>>>>>> through several libraries, there could be tiny leaks anywhere.
>>>>>>> I suspect most people give-up on this. Ruby makes programming easier at the >>>>>>> cost of less-refined memory management, this is why god scripts exist to >>>>>>> restart mongrel or unicorn processes to free-up trapped memory.
>>>>>>>> Can I call "something" in my loop to >>>>>>>> "clean up"? Do I have to?
>>>>>>> You can try forcing garbage collection, but I doubt that will solve much:
>>>>>>> Seriously, if you're handling lots of data in a daemon during a long period >>>>>>> of time, fork it, process, clean.
>>>>>>> Cheers, >>>>>>> sam
>>>>>>>> Cheers.
>>>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: >>>>>>>> > Hi, >>>>>>>> > Check that its not running out of memory. god is useful for to keeping >>>>>>>> > things running. >>>>>>>> > You might also want to checkout spawn and fork the process before doing >>>>>>>> > a >>>>>>>> > loop and wait for it to complete. This will ensure your memory is >>>>>>>> > release >>>>>>>> > after each loop, and automatically handle the ActiveRecord database >>>>>>>> > connection (which you may also be loosing after a period of time.) >>>>>>>> > Cheers, >>>>>>>> > sam >>>>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>>>>>>> >> I don't think I'm doing anything wrong in the code itself. For your >>>>>>>> >> information I have 498 podcasts and currently 17961 episodes. I'm >>>>>>>> >> pretty sure the script runs for a few days before somehow dying.
>>>>>>>> >> Anybody else experiencing this?
>>>>>>>> >> Cheers.
>>>>>>>> >> -- >>>>>>>> >> You received this message because you are subscribed to the Google >>>>>>>> >> Groups >>>>>>>> >> "feedzirra" group. >>>>>>>> >> To post to this group, send email to feedzirra@googlegroups.com. >>>>>>>> >> To unsubscribe from this group, send email to >>>>>>>> >> feedzirra+unsubscribe@googlegroups.com. >>>>>>>> >> For more options, visit this group at >>>>>>>> >> http://groups.google.com/group/feedzirra?hl=en.
>>>>>>>> > -- >>>>>>>> > You received this message because you are subscribed to the Google >>>>>>>> > Groups >>>>>>>> > "feedzirra" group. >>>>>>>> > To post to this group, send email to feedzirra@googlegroups.com. >>>>>>>> > To unsubscribe from this group, send email to >>>>>>>> > feedzirra+unsubscribe@googlegroups.com. >>>>>>>> > For more options, visit this group at >>>>>>>> > http://groups.google.com/group/feedzirra?hl=en.
>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google Groups >>>>>>>> "feedzirra" group. >>>>>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> feedzirra+unsubscribe@googlegroups.com. >>>>>>>> For more options, visit this group at >>>>>>>> http://groups.google.com/group/feedzirra?hl=en.
>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google Groups >>>>>>> "feedzirra" group. >>>>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>>>> To unsubscribe from this group, send email to >>>>>>> feedzirra+unsubscribe@googlegroups.com. >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/feedzirra?hl=en.
>>>>>> -- >>>>>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>>>>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>>>>> -- >>>>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>>>> To post to this group, send email to feedzirra@googlegroups.com. >>>>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>>>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>>>> -- >>>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>>> To post to this group, send email to feedzirra@googlegroups.com. >>>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>>> -- >>> You received this message because you are subscribed to the Google Groups "feedzirra" group. >>> To post to this group, send email to feedzirra@googlegroups.com. >>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >>> For more options, visit this group at http://groups.google.com/group/feedzirra?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups "feedzirra" group. >> To post to this group, send email to feedzirra@googlegroups.com. >> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. >> For more
Seriously, use spawn! Cron is not better than a daemon for things that need to be running 24/7. The only thing cron does differently is that it closes the app when it finishes releasing all its memory, then loads it up again. Think of all the CPU you're loosing as your code is re-loaded after each loop. Forking (with spawn) avoids this as it just copies itself in memory and starts running in a new process, when it finished that memory is then released allowing you to repeat efficiently.
I have a lot of experience with this. I run Planetaki.com which is currently fetching around 60,000 feeds constantly. Spawn is used to fork off multiple "feed reapers" at the same time, and Timeout is used to ensure bad feeds are stopped if they take to long to complete. (Incidentally, we'll be moving to FeedZirra for parsing this week!)
Sam
On 31 May 2010 10:13, StuFF mc <m...@stuffmc.com> wrote:
> Guys, discussing with a few folks while being at http://railswaycon.de > (right now) they told me using a Daemon wasn't the best way of doing > it. They recommend putting the process in a cron so it's started from > fresh everytime. The problem is that I'd like to "fetch" to fetch non > stop, this way being as synced as possible (right now worst case I > have the podcast 20 minutes after they are published).
> I would need the "new/next" cron to only start when the previous is > finished. I was thinking to maybe program the next cron 1 minute after > the current is finished. Any Ideas/recommandations?
> > These types of errors are a real time suck. Hardest to track down. :(
> > On Wed, May 26, 2010 at 5:04 PM, StuFF mc <m...@stuffmc.com> wrote: > >> I just use the rubygem "daemons" and it's supposed to output in > >> log/name_of_the_daemon.rb.log but the last errors I had there were > >> from March.
> >> Attached a chart that shows when I restarted the daemon. We'll see > >> over time if the require was the problem. Aside the daemon, would you > >> have a good tip to profile my "Episode.fetch_all"?
> >> script/performance isn't much helpful i.m.h.o.
> >> On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: > >>> It depends on how/what you're using to daemonize the process that is > >>> running these updates. Whatever it is should have the ability to write > >>> STDOUT and STDERR to some log file. If you aren't specifying that then > >>> the process is probably just throwing an exception that isn't being > >>> captured by anything.
> >>> Best, > >>> Paul
> >>> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: > >>>> Thanks for answering so quickly Paul.
> >>>> True, it's pretty dumb to have this require there. I remember having > >>>> tried to have it in my environment.rb but was unsuccessful. Anyways, I > >>>> moved it to the top of the source file (so outside the method) and > >>>> we'll see if it changes anything.
> >>>> Concerning the logs, they are supposed to be in /log, right? There > >>>> nothing relevant there :(
> >>>> Cheers.
> >>>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: > >>>>> Is the daemon failing with some sort of error? I assume that it's > just > >>>>> not failing quietly. The graph doesn't really do anything to help > >>>>> troubleshoot. Is there a reason you're doing a require inside the > >>>>> "add_entries" method? I don't think you want to be rerunning that > >>>>> everytime you run it.
> >>>>> Log the output from that daemon.
> >>>>> Best, > >>>>> Paul
> >>>>> On Wed, May 26, 2010 at 4:00 PM, StuFF mc <m...@stuffmc.com> wrote: > >>>>>> Since I'm not sure you can add images to this group, it's also here: > >>>>>> http://emberapp.com/stuffmc/images/cpu-graph-rp
> >>>>>> This graph shows how nuts the process is going (before to be killed > I > >>>>>> guess) after the daemons has been running for about a day or 2.
> >>>>>> I haven't take the time yet to look at how God can help me - the > last > >>>>>> time I looked at God it wasn't trivial :(
> >>>>>> But from the Feedzirra perspective (and thus, this is a question to > >>>>>> the main dev), where could be the problem?
> >>>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> > wrote:
> >>>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
> >>>>>>>> thanks for the tips. I had thought about spawn & god but what's > the > >>>>>>>> point of a daemon if the connection to Active Record goes away?!
> >>>>>>> The DB can timeout a connection if its not being used or cut-out on > network > >>>>>>> failure. This probably isn't a problem in your situation though. It > would be > >>>>>>> if you were forking, as closing a fork will also close the DB > connection.
> >>>>>>>> Also > >>>>>>>> how come the memory isn't freed.
> >>>>>>> You can try some profiling techniques if you want to discover where > your > >>>>>>> memory is leaking. You're reading lots of external data and passing > it > >>>>>>> through several libraries, there could be tiny leaks anywhere.
> >>>>>>> I suspect most people give-up on this. Ruby makes programming > easier at the > >>>>>>> cost of less-refined memory management, this is why god scripts > exist to > >>>>>>> restart mongrel or unicorn processes to free-up trapped memory.
> >>>>>>>> Can I call "something" in my loop to > >>>>>>>> "clean up"? Do I have to?
> >>>>>>> You can try forcing garbage collection, but I doubt that will solve > much:
> >>>>>>> Seriously, if you're handling lots of data in a daemon during a > long period > >>>>>>> of time, fork it, process, clean.
> >>>>>>> Cheers, > >>>>>>> sam
> >>>>>>>> Cheers.
> >>>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> > wrote: > >>>>>>>> > Hi, > >>>>>>>> > Check that its not running out of memory. god is useful for to > keeping > >>>>>>>> > things running. > >>>>>>>> > You might also want to checkout spawn and fork the process > before doing > >>>>>>>> > a > >>>>>>>> > loop and wait for it to complete. This will ensure your memory > is > >>>>>>>> > release > >>>>>>>> > after each loop, and automatically handle the ActiveRecord > database > >>>>>>>> > connection (which you may also be loosing after a period of > time.) > >>>>>>>> > Cheers, > >>>>>>>> > sam > >>>>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
> >>>>>>>> >> I don't think I'm doing anything wrong in the code itself. For > your > >>>>>>>> >> information I have 498 podcasts and currently 17961 episodes. > I'm > >>>>>>>> >> pretty sure the script runs for a few days before somehow > dying.
> >>>>>>>> >> Anybody else experiencing this?
> >>>>>>>> >> Cheers.
> >>>>>>>> >> -- > >>>>>>>> >> You received this message because you are subscribed to the > Google > >>>>>>>> >> Groups > >>>>>>>> >> "feedzirra" group. > >>>>>>>> >> To post to this group, send email to > feedzirra@googlegroups.com. > >>>>>>>> >> To unsubscribe from this group, send email to > >>>>>>>> >> feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com> > . > >>>>>>>> >> For more options, visit this group at > >>>>>>>> >> http://groups.google.com/group/feedzirra?hl=en.
> >>>>>>>> > -- > >>>>>>>> > You received this message because you are subscribed to the > Google > >>>>>>>> > Groups > >>>>>>>> > "feedzirra" group. > >>>>>>>> > To post to this group, send email to feedzirra@googlegroups.com > . > >>>>>>>> > To unsubscribe from this group, send email to > >>>>>>>> > feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com> > . > >>>>>>>> > For more options, visit this group at > >>>>>>>> > http://groups.google.com/group/feedzirra?hl=en.
> >>>>>>>> -- > >>>>>>>> You received this message because you are subscribed to the Google > Groups > >>>>>>>> "feedzirra" group. > >>>>>>>> To post to this group, send email to feedzirra@googlegroups.com. > >>>>>>>> To unsubscribe from this group, send email to > >>>>>>>> feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com> > . > >>>>>>>> For more options, visit this group at > >>>>>>>> http://groups.google.com/group/feedzirra?hl=en.
> >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > Groups > >>>>>>> "feedzirra" group. > >>>>>>> To post to this group, send email to feedzirra@googlegroups.com. > >>>>>>> To unsubscribe from this group, send email to > >>>>>>> feedzirra+unsubscribe@googlegroups.com<feedzirra%2Bunsubscribe@googlegroups .com> > . > >>>>>>> For more options, visit this group at > >>>>>>> http://groups.google.com/group/feedzirra?hl=en.
> >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > Groups "feedzirra" group. > >>>>>> To post to this group, send email to feedzirra@googlegroups.com. > >>>>>> To unsubscribe from this
> Seriously, use spawn! Cron is not better than a daemon for things that need to be running 24/7. The only thing cron does differently is that it closes the app when it finishes releasing all its memory, then loads it up again. Think of all the CPU you're loosing as your code is re-loaded after each loop. Forking (with spawn) avoids this as it just copies itself in memory and starts running in a new process, when it finished that memory is then released allowing you to repeat efficiently.
> I have a lot of experience with this. I run Planetaki.com which is currently fetching around 60,000 feeds constantly. Spawn is used to fork off multiple "feed reapers" at the same time, and Timeout is used to ensure bad feeds are stopped if they take to long to complete. (Incidentally, we'll be moving to FeedZirra for parsing this week!)
> Sam
> On 31 May 2010 10:13, StuFF mc <m...@stuffmc.com> wrote: > Guys, discussing with a few folks while being at http://railswaycon.de > (right now) they told me using a Daemon wasn't the best way of doing > it. They recommend putting the process in a cron so it's started from > fresh everytime. The problem is that I'd like to "fetch" to fetch non > stop, this way being as synced as possible (right now worst case I > have the podcast 20 minutes after they are published).
> I would need the "new/next" cron to only start when the previous is > finished. I was thinking to maybe program the next cron 1 minute after > the current is finished. Any Ideas/recommandations?
> > These types of errors are a real time suck. Hardest to track down. :(
> > On Wed, May 26, 2010 at 5:04 PM, StuFF mc <m...@stuffmc.com> wrote: > >> I just use the rubygem "daemons" and it's supposed to output in > >> log/name_of_the_daemon.rb.log but the last errors I had there were > >> from March.
> >> Attached a chart that shows when I restarted the daemon. We'll see > >> over time if the require was the problem. Aside the daemon, would you > >> have a good tip to profile my "Episode.fetch_all"?
> >> script/performance isn't much helpful i.m.h.o.
> >> On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: > >>> It depends on how/what you're using to daemonize the process that is > >>> running these updates. Whatever it is should have the ability to write > >>> STDOUT and STDERR to some log file. If you aren't specifying that then > >>> the process is probably just throwing an exception that isn't being > >>> captured by anything.
> >>> Best, > >>> Paul
> >>> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: > >>>> Thanks for answering so quickly Paul.
> >>>> True, it's pretty dumb to have this require there. I remember having > >>>> tried to have it in my environment.rb but was unsuccessful. Anyways, I > >>>> moved it to the top of the source file (so outside the method) and > >>>> we'll see if it changes anything.
> >>>> Concerning the logs, they are supposed to be in /log, right? There > >>>> nothing relevant there :(
> >>>> Cheers.
> >>>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: > >>>>> Is the daemon failing with some sort of error? I assume that it's just > >>>>> not failing quietly. The graph doesn't really do anything to help > >>>>> troubleshoot. Is there a reason you're doing a require inside the > >>>>> "add_entries" method? I don't think you want to be rerunning that > >>>>> everytime you run it.
> >>>>> Log the output from that daemon.
> >>>>> Best, > >>>>> Paul
> >>>>> On Wed, May 26, 2010 at 4:00 PM, StuFF mc <m...@stuffmc.com> wrote: > >>>>>> Since I'm not sure you can add images to this group, it's also here: > >>>>>> http://emberapp.com/stuffmc/images/cpu-graph-rp
> >>>>>> This graph shows how nuts the process is going (before to be killed I > >>>>>> guess) after the daemons has been running for about a day or 2.
> >>>>>> I haven't take the time yet to look at how God can help me - the last > >>>>>> time I looked at God it wasn't trivial :(
> >>>>>> But from the Feedzirra perspective (and thus, this is a question to > >>>>>> the main dev), where could be the problem?
> >>>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> wrote:
> >>>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
> >>>>>>>> thanks for the tips. I had thought about spawn & god but what's the > >>>>>>>> point of a daemon if the connection to Active Record goes away?!
> >>>>>>> The DB can timeout a connection if its not being used or cut-out on network > >>>>>>> failure. This probably isn't a problem in your situation though. It would be > >>>>>>> if you were forking, as closing a fork will also close the DB connection.
> >>>>>>>> Also > >>>>>>>> how come the memory isn't freed.
> >>>>>>> You can try some profiling techniques if you want to discover where your > >>>>>>> memory is leaking. You're reading lots of external data and passing it > >>>>>>> through several libraries, there could be tiny leaks anywhere.
> >>>>>>> I suspect most people give-up on this. Ruby makes programming easier at the > >>>>>>> cost of less-refined memory management, this is why god scripts exist to > >>>>>>> restart mongrel or unicorn processes to free-up trapped memory.
> >>>>>>>> Can I call "something" in my loop to > >>>>>>>> "clean up"? Do I have to?
> >>>>>>> You can try forcing garbage collection, but I doubt that will solve much:
> >>>>>>> Seriously, if you're handling lots of data in a daemon during a long period > >>>>>>> of time, fork it, process, clean.
> >>>>>>> Cheers, > >>>>>>> sam
> >>>>>>>> Cheers.
> >>>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> wrote: > >>>>>>>> > Hi, > >>>>>>>> > Check that its not running out of memory. god is useful for to keeping > >>>>>>>> > things running. > >>>>>>>> > You might also want to checkout spawn and fork the process before doing > >>>>>>>> > a > >>>>>>>> > loop and wait for it to complete. This will ensure your memory is > >>>>>>>> > release > >>>>>>>> > after each loop, and automatically handle the ActiveRecord database > >>>>>>>> > connection (which you may also be loosing after a period of time.) > >>>>>>>> > Cheers, > >>>>>>>> > sam > >>>>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
> >>>>>>>> >> I don't think I'm doing anything wrong in the code itself. For your > >>>>>>>> >> information I have 498 podcasts and currently 17961 episodes. I'm > >>>>>>>> >> pretty sure the script runs for a few days before somehow dying.
> >>>>>>>> >> Anybody else experiencing this?
> >>>>>>>> >> Cheers.
> >>>>>>>> >> -- > >>>>>>>> >> You received this message because you are subscribed to the Google > >>>>>>>> >> Groups > >>>>>>>> >> "feedzirra" group. > >>>>>>>> >> To post to this group, send email to feedzirra@googlegroups.com. > >>>>>>>> >> To unsubscribe from this group, send email to > >>>>>>>> >> feedzirra+unsubscribe@googlegroups.com. > >>>>>>>> >> For more options, visit this group at > >>>>>>>> >> http://groups.google.com/group/feedzirra?hl=en.
> >>>>>>>> > -- > >>>>>>>> > You received this message because you are subscribed to the Google > >>>>>>>> > Groups > >>>>>>>> > "feedzirra" group. > >>>>>>>> > To post to this group, send email to feedzirra@googlegroups.com. > >>>>>>>> > To unsubscribe from this group, send email to > >>>>>>>> > feedzirra+unsubscribe@googlegroups.com. > >>>>>>>> > For more options, visit this group at > >>>>>>>> > http://groups.google.com/group/feedzirra?hl=en.
> >>>>>>>> -- > >>>>>>>> You received this message because you are subscribed to the Google Groups > >>>>>>>> "feedzirra" group. > >>>>>>>> To post to this group, send email to feedzirra@googlegroups.com. > >>>>>>>> To unsubscribe from this group, send email to > >>>>>>>> feedzirra+unsubscribe@googlegroups.com. > >>>>>>>> For more options, visit this group at > >>>>>>>> http://groups.google.com/group/feedzirra?hl=en.
> >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google Groups > >>>>>>> "feedzirra" group. > >>>>>>> To post to this group, send email to feedzirra@googlegroups.com. > >>>>>>> To unsubscribe from this group, send email to > >>>>>>> feedzirra+unsubscribe@googlegroups.com. > >>>>>>> For more options, visit this group at > >>>>>>> http://groups.google.com/group/feedzirra?hl=en.
> >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google Groups "feedzirra" group. > >>>>>> To post to this group, send email to feedzirra@googlegroups.com. > >>>>>> To unsubscribe from this group, send email to feedzirra+unsubscribe@googlegroups.com. > >>>>>> For more options,
agreed, cron is for things that get scheduled, not for things that need to be running all the time. I haven't used spawn, but I can say there are libraries out there to do this. Servolux (http://github.com/TwP/servolux) is pretty good and includes stuff for managing a pre-forking worker pool.
On Mon, May 31, 2010 at 4:43 AM, Samuel Lown <sam.l...@gmail.com> wrote: > Hi, > Seriously, use spawn! Cron is not better than a daemon for things that need > to be running 24/7. The only thing cron does differently is that it closes > the app when it finishes releasing all its memory, then loads it up again. > Think of all the CPU you're loosing as your code is re-loaded after each > loop. Forking (with spawn) avoids this as it just copies itself in memory > and starts running in a new process, when it finished that memory is then > released allowing you to repeat efficiently. > I have a lot of experience with this. I run Planetaki.com which is currently > fetching around 60,000 feeds constantly. Spawn is used to fork off multiple > "feed reapers" at the same time, and Timeout is used to ensure bad feeds are > stopped if they take to long to complete. (Incidentally, we'll be moving to > FeedZirra for parsing this week!) > Sam
> On 31 May 2010 10:13, StuFF mc <m...@stuffmc.com> wrote:
>> Guys, discussing with a few folks while being at http://railswaycon.de >> (right now) they told me using a Daemon wasn't the best way of doing >> it. They recommend putting the process in a cron so it's started from >> fresh everytime. The problem is that I'd like to "fetch" to fetch non >> stop, this way being as synced as possible (right now worst case I >> have the podcast 20 minutes after they are published).
>> I would need the "new/next" cron to only start when the previous is >> finished. I was thinking to maybe program the next cron 1 minute after >> the current is finished. Any Ideas/recommandations?
>> > These types of errors are a real time suck. Hardest to track down. :(
>> > On Wed, May 26, 2010 at 5:04 PM, StuFF mc <m...@stuffmc.com> wrote: >> >> I just use the rubygem "daemons" and it's supposed to output in >> >> log/name_of_the_daemon.rb.log but the last errors I had there were >> >> from March.
>> >> Attached a chart that shows when I restarted the daemon. We'll see >> >> over time if the require was the problem. Aside the daemon, would you >> >> have a good tip to profile my "Episode.fetch_all"?
>> >> script/performance isn't much helpful i.m.h.o.
>> >> On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: >> >>> It depends on how/what you're using to daemonize the process that is >> >>> running these updates. Whatever it is should have the ability to write >> >>> STDOUT and STDERR to some log file. If you aren't specifying that then >> >>> the process is probably just throwing an exception that isn't being >> >>> captured by anything.
>> >>> Best, >> >>> Paul
>> >>> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: >> >>>> Thanks for answering so quickly Paul.
>> >>>> True, it's pretty dumb to have this require there. I remember having >> >>>> tried to have it in my environment.rb but was unsuccessful. Anyways, >> >>>> I >> >>>> moved it to the top of the source file (so outside the method) and >> >>>> we'll see if it changes anything.
>> >>>> Concerning the logs, they are supposed to be in /log, right? There >> >>>> nothing relevant there :(
>> >>>> Cheers.
>> >>>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: >> >>>>> Is the daemon failing with some sort of error? I assume that it's >> >>>>> just >> >>>>> not failing quietly. The graph doesn't really do anything to help >> >>>>> troubleshoot. Is there a reason you're doing a require inside the >> >>>>> "add_entries" method? I don't think you want to be rerunning that >> >>>>> everytime you run it.
>> >>>>> Log the output from that daemon.
>> >>>>> Best, >> >>>>> Paul
>> >>>>> On Wed, May 26, 2010 at 4:00 PM, StuFF mc <m...@stuffmc.com> wrote: >> >>>>>> Since I'm not sure you can add images to this group, it's also >> >>>>>> here: >> >>>>>> http://emberapp.com/stuffmc/images/cpu-graph-rp
>> >>>>>> This graph shows how nuts the process is going (before to be killed >> >>>>>> I >> >>>>>> guess) after the daemons has been running for about a day or 2.
>> >>>>>> I haven't take the time yet to look at how God can help me - the >> >>>>>> last >> >>>>>> time I looked at God it wasn't trivial :(
>> >>>>>> But from the Feedzirra perspective (and thus, this is a question to >> >>>>>> the main dev), where could be the problem?
>> >>>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> >> >>>>>> wrote:
>> >>>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>> >>>>>>>> thanks for the tips. I had thought about spawn & god but what's >> >>>>>>>> the >> >>>>>>>> point of a daemon if the connection to Active Record goes away?!
>> >>>>>>> The DB can timeout a connection if its not being used or cut-out >> >>>>>>> on network >> >>>>>>> failure. This probably isn't a problem in your situation though. >> >>>>>>> It would be >> >>>>>>> if you were forking, as closing a fork will also close the DB >> >>>>>>> connection.
>> >>>>>>>> Also >> >>>>>>>> how come the memory isn't freed.
>> >>>>>>> You can try some profiling techniques if you want to discover >> >>>>>>> where your >> >>>>>>> memory is leaking. You're reading lots of external data and >> >>>>>>> passing it >> >>>>>>> through several libraries, there could be tiny leaks anywhere.
>> >>>>>>> I suspect most people give-up on this. Ruby makes programming >> >>>>>>> easier at the >> >>>>>>> cost of less-refined memory management, this is why god scripts >> >>>>>>> exist to >> >>>>>>> restart mongrel or unicorn processes to free-up trapped memory.
>> >>>>>>>> Can I call "something" in my loop to >> >>>>>>>> "clean up"? Do I have to?
>> >>>>>>> You can try forcing garbage collection, but I doubt that will >> >>>>>>> solve much:
>> >>>>>>> Seriously, if you're handling lots of data in a daemon during a >> >>>>>>> long period >> >>>>>>> of time, fork it, process, clean.
>> >>>>>>> Cheers, >> >>>>>>> sam
>> >>>>>>>> Cheers.
>> >>>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> >> >>>>>>>> wrote: >> >>>>>>>> > Hi, >> >>>>>>>> > Check that its not running out of memory. god is useful for to >> >>>>>>>> > keeping >> >>>>>>>> > things running. >> >>>>>>>> > You might also want to checkout spawn and fork the process >> >>>>>>>> > before doing >> >>>>>>>> > a >> >>>>>>>> > loop and wait for it to complete. This will ensure your memory >> >>>>>>>> > is >> >>>>>>>> > release >> >>>>>>>> > after each loop, and automatically handle the ActiveRecord >> >>>>>>>> > database >> >>>>>>>> > connection (which you may also be loosing after a period of >> >>>>>>>> > time.) >> >>>>>>>> > Cheers, >> >>>>>>>> > sam >> >>>>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>> >>>>>>>> >> I don't think I'm doing anything wrong in the code itself. For >> >>>>>>>> >> your >> >>>>>>>> >> information I have 498 podcasts and currently 17961 episodes. >> >>>>>>>> >> I'm >> >>>>>>>> >> pretty sure the script runs for a few days before somehow >> >>>>>>>> >> dying.
>> >>>>>>>> >> Anybody else experiencing this?
>> >>>>>>>> >> Cheers.
>> >>>>>>>> >> -- >> >>>>>>>> >> You received this message because you are subscribed to the >> >>>>>>>> >> Google >> >>>>>>>> >> Groups >> >>>>>>>> >> "feedzirra" group. >> >>>>>>>> >> To post to this group, send email to >> >>>>>>>> >> feedzirra@googlegroups.com. >> >>>>>>>> >> To unsubscribe from this group, send email to >> >>>>>>>> >> feedzirra+unsubscribe@googlegroups.com. >> >>>>>>>> >> For more options, visit this group at >> >>>>>>>> >> http://groups.google.com/group/feedzirra?hl=en.
>> >>>>>>>> > -- >> >>>>>>>> > You received this message because you are subscribed to the >> >>>>>>>> > Google >> >>>>>>>> > Groups >> >>>>>>>> > "feedzirra" group. >> >>>>>>>> > To post to this group, send email to >> >>>>>>>> > feedzirra@googlegroups.com. >> >>>>>>>> > To unsubscribe from this group, send email to >> >>>>>>>> > feedzirra+unsubscribe@googlegroups.com. >> >>>>>>>> > For more options, visit this group at >> >>>>>>>> > http://groups.google.com/group/feedzirra?hl=en.
>> >>>>>>>> -- >> >>>>>>>> You received this message because you are subscribed to the >> >>>>>>>> Google Groups >> >>>>>>>> "feedzirra" group. >> >>>>>>>> To post to this group, send email to feedzirra@googlegroups.com. >> >>>>>>>> To unsubscribe from this group, send email to >> >>>>>>>> feedzirra+unsubscribe@googlegroups.com. >> >>>>>>>> For more options, visit this group at >> >>>>>>>> http://groups.google.com/group/feedzirra?hl=en.
Sadly like a lot of open source things, Servolux is pretty undocumented. Watching at this: http://rdoc.info/projects/TwP/servolux , will anyone have a code sample showing how I could call my "fetch_episodes" from a worker?
I'm a bit confused.
ps: Obviously it still dies after a few days with the daemon.
On Mon, May 31, 2010 at 7:07 PM, Paul Dix <p...@pauldix.net> wrote: > agreed, cron is for things that get scheduled, not for things that > need to be running all the time. I haven't used spawn, but I can say > there are libraries out there to do this. Servolux > (http://github.com/TwP/servolux) is pretty good and includes stuff for > managing a pre-forking worker pool.
> Best, > Paul
> On Mon, May 31, 2010 at 4:43 AM, Samuel Lown <sam.l...@gmail.com> wrote: >> Hi, >> Seriously, use spawn! Cron is not better than a daemon for things that need >> to be running 24/7. The only thing cron does differently is that it closes >> the app when it finishes releasing all its memory, then loads it up again. >> Think of all the CPU you're loosing as your code is re-loaded after each >> loop. Forking (with spawn) avoids this as it just copies itself in memory >> and starts running in a new process, when it finished that memory is then >> released allowing you to repeat efficiently. >> I have a lot of experience with this. I run Planetaki.com which is currently >> fetching around 60,000 feeds constantly. Spawn is used to fork off multiple >> "feed reapers" at the same time, and Timeout is used to ensure bad feeds are >> stopped if they take to long to complete. (Incidentally, we'll be moving to >> FeedZirra for parsing this week!) >> Sam
>> On 31 May 2010 10:13, StuFF mc <m...@stuffmc.com> wrote:
>>> Guys, discussing with a few folks while being at http://railswaycon.de >>> (right now) they told me using a Daemon wasn't the best way of doing >>> it. They recommend putting the process in a cron so it's started from >>> fresh everytime. The problem is that I'd like to "fetch" to fetch non >>> stop, this way being as synced as possible (right now worst case I >>> have the podcast 20 minutes after they are published).
>>> I would need the "new/next" cron to only start when the previous is >>> finished. I was thinking to maybe program the next cron 1 minute after >>> the current is finished. Any Ideas/recommandations?
>>> > These types of errors are a real time suck. Hardest to track down. :(
>>> > On Wed, May 26, 2010 at 5:04 PM, StuFF mc <m...@stuffmc.com> wrote: >>> >> I just use the rubygem "daemons" and it's supposed to output in >>> >> log/name_of_the_daemon.rb.log but the last errors I had there were >>> >> from March.
>>> >> Attached a chart that shows when I restarted the daemon. We'll see >>> >> over time if the require was the problem. Aside the daemon, would you >>> >> have a good tip to profile my "Episode.fetch_all"?
>>> >> script/performance isn't much helpful i.m.h.o.
>>> >> On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: >>> >>> It depends on how/what you're using to daemonize the process that is >>> >>> running these updates. Whatever it is should have the ability to write >>> >>> STDOUT and STDERR to some log file. If you aren't specifying that then >>> >>> the process is probably just throwing an exception that isn't being >>> >>> captured by anything.
>>> >>> Best, >>> >>> Paul
>>> >>> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: >>> >>>> Thanks for answering so quickly Paul.
>>> >>>> True, it's pretty dumb to have this require there. I remember having >>> >>>> tried to have it in my environment.rb but was unsuccessful. Anyways, >>> >>>> I >>> >>>> moved it to the top of the source file (so outside the method) and >>> >>>> we'll see if it changes anything.
>>> >>>> Concerning the logs, they are supposed to be in /log, right? There >>> >>>> nothing relevant there :(
>>> >>>> Cheers.
>>> >>>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: >>> >>>>> Is the daemon failing with some sort of error? I assume that it's >>> >>>>> just >>> >>>>> not failing quietly. The graph doesn't really do anything to help >>> >>>>> troubleshoot. Is there a reason you're doing a require inside the >>> >>>>> "add_entries" method? I don't think you want to be rerunning that >>> >>>>> everytime you run it.
>>> >>>>> Log the output from that daemon.
>>> >>>>> Best, >>> >>>>> Paul
>>> >>>>> On Wed, May 26, 2010 at 4:00 PM, StuFF mc <m...@stuffmc.com> wrote: >>> >>>>>> Since I'm not sure you can add images to this group, it's also >>> >>>>>> here: >>> >>>>>> http://emberapp.com/stuffmc/images/cpu-graph-rp
>>> >>>>>> This graph shows how nuts the process is going (before to be killed >>> >>>>>> I >>> >>>>>> guess) after the daemons has been running for about a day or 2.
>>> >>>>>> I haven't take the time yet to look at how God can help me - the >>> >>>>>> last >>> >>>>>> time I looked at God it wasn't trivial :(
>>> >>>>>> But from the Feedzirra perspective (and thus, this is a question to >>> >>>>>> the main dev), where could be the problem?
>>> >>>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> >>> >>>>>> wrote:
>>> >>>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>> >>>>>>>> thanks for the tips. I had thought about spawn & god but what's >>> >>>>>>>> the >>> >>>>>>>> point of a daemon if the connection to Active Record goes away?!
>>> >>>>>>> The DB can timeout a connection if its not being used or cut-out >>> >>>>>>> on network >>> >>>>>>> failure. This probably isn't a problem in your situation though. >>> >>>>>>> It would be >>> >>>>>>> if you were forking, as closing a fork will also close the DB >>> >>>>>>> connection.
>>> >>>>>>>> Also >>> >>>>>>>> how come the memory isn't freed.
>>> >>>>>>> You can try some profiling techniques if you want to discover >>> >>>>>>> where your >>> >>>>>>> memory is leaking. You're reading lots of external data and >>> >>>>>>> passing it >>> >>>>>>> through several libraries, there could be tiny leaks anywhere.
>>> >>>>>>> I suspect most people give-up on this. Ruby makes programming >>> >>>>>>> easier at the >>> >>>>>>> cost of less-refined memory management, this is why god scripts >>> >>>>>>> exist to >>> >>>>>>> restart mongrel or unicorn processes to free-up trapped memory.
>>> >>>>>>>> Can I call "something" in my loop to >>> >>>>>>>> "clean up"? Do I have to?
>>> >>>>>>> You can try forcing garbage collection, but I doubt that will >>> >>>>>>> solve much:
>>> >>>>>>> Seriously, if you're handling lots of data in a daemon during a >>> >>>>>>> long period >>> >>>>>>> of time, fork it, process, clean.
>>> >>>>>>> Cheers, >>> >>>>>>> sam
>>> >>>>>>>> Cheers.
>>> >>>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> >>> >>>>>>>> wrote: >>> >>>>>>>> > Hi, >>> >>>>>>>> > Check that its not running out of memory. god is useful for to >>> >>>>>>>> > keeping >>> >>>>>>>> > things running. >>> >>>>>>>> > You might also want to checkout spawn and fork the process >>> >>>>>>>> > before doing >>> >>>>>>>> > a >>> >>>>>>>> > loop and wait for it to complete. This will ensure your memory >>> >>>>>>>> > is >>> >>>>>>>> > release >>> >>>>>>>> > after each loop, and automatically handle the ActiveRecord >>> >>>>>>>> > database >>> >>>>>>>> > connection (which you may also be loosing after a period of >>> >>>>>>>> > time.) >>> >>>>>>>> > Cheers, >>> >>>>>>>> > sam >>> >>>>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>> >>>>>>>> >> I don't think I'm doing anything wrong in the code itself. For >>> >>>>>>>> >> your >>> >>>>>>>> >> information I have 498 podcasts and currently 17961 episodes. >>> >>>>>>>> >> I'm >>> >>>>>>>> >> pretty sure the script runs for a few days before somehow >>> >>>>>>>> >> dying.
>>> >>>>>>>> >> Anybody else experiencing this?
>>> >>>>>>>> >> Cheers.
>>> >>>>>>>> >> -- >>> >>>>>>>> >> You received this message because you are subscribed to the >>> >>>>>>>> >> Google >>> >>>>>>>> >> Groups >>> >>>>>>>> >> "feedzirra" group. >>> >>>>>>>> >> To post to this group, send email to >>> >>>>>>>> >> feedzirra@googlegroups.com. >>> >>>>>>>> >> To unsubscribe from this group, send email to >>> >>>>>>>> >> feedzirra+unsubscribe@googlegroups.com. >>> >>>>>>>> >> For more options, visit this group at >>> >>>>>>>> >> http://groups.google.com/group/feedzirra?hl=en.
>>> >>>>>>>> > -- >>> >>>>>>>> > You received this message because you are subscribed to the >>> >>>>>>>> > Google >>> >>>>>>>> > Groups >>> >>>>>>>> > "feedzirra" group. >>> >>>>>>>> > To post to this group, send email to >>> >>>>>>>> > feedzirra@googlegroups.com. >>> >>>>>>>> > To unsubscribe from this group, send email to >>> >>>>>>>> > feedzirra+unsubscribe@googlegroups.com.
I wanted to let you know everything works fine since I started using the "delay" method from http://tobi.github.com/delayed_job/ instead of using a daemon. It then spawns himself without any work from me. Really, the only thing I do other than installing the plugin/gem is:
def self.fetch_all Podcast.all.each do |podcast| podcast.fetch_episodes end self.delay.fetch_all end
then I call the first time Episode.delay.fetch_all from one way (script/console) or another (capistrano) and boom!, it's running for weeks without any problem.
On Wed, Jun 9, 2010 at 5:15 PM, StuFF mc <m...@stuffmc.com> wrote: > Back on this thread.
> Sadly like a lot of open source things, Servolux is pretty > undocumented. Watching at this: http://rdoc.info/projects/TwP/servolux > , will anyone have a code sample showing how I could call my > "fetch_episodes" from a worker?
> I'm a bit confused.
> ps: Obviously it still dies after a few days with the daemon.
> On Mon, May 31, 2010 at 7:07 PM, Paul Dix <p...@pauldix.net> wrote: >> agreed, cron is for things that get scheduled, not for things that >> need to be running all the time. I haven't used spawn, but I can say >> there are libraries out there to do this. Servolux >> (http://github.com/TwP/servolux) is pretty good and includes stuff for >> managing a pre-forking worker pool.
>> Best, >> Paul
>> On Mon, May 31, 2010 at 4:43 AM, Samuel Lown <sam.l...@gmail.com> wrote: >>> Hi, >>> Seriously, use spawn! Cron is not better than a daemon for things that need >>> to be running 24/7. The only thing cron does differently is that it closes >>> the app when it finishes releasing all its memory, then loads it up again. >>> Think of all the CPU you're loosing as your code is re-loaded after each >>> loop. Forking (with spawn) avoids this as it just copies itself in memory >>> and starts running in a new process, when it finished that memory is then >>> released allowing you to repeat efficiently. >>> I have a lot of experience with this. I run Planetaki.com which is currently >>> fetching around 60,000 feeds constantly. Spawn is used to fork off multiple >>> "feed reapers" at the same time, and Timeout is used to ensure bad feeds are >>> stopped if they take to long to complete. (Incidentally, we'll be moving to >>> FeedZirra for parsing this week!) >>> Sam
>>> On 31 May 2010 10:13, StuFF mc <m...@stuffmc.com> wrote:
>>>> Guys, discussing with a few folks while being at http://railswaycon.de >>>> (right now) they told me using a Daemon wasn't the best way of doing >>>> it. They recommend putting the process in a cron so it's started from >>>> fresh everytime. The problem is that I'd like to "fetch" to fetch non >>>> stop, this way being as synced as possible (right now worst case I >>>> have the podcast 20 minutes after they are published).
>>>> I would need the "new/next" cron to only start when the previous is >>>> finished. I was thinking to maybe program the next cron 1 minute after >>>> the current is finished. Any Ideas/recommandations?
>>>> > These types of errors are a real time suck. Hardest to track down. :(
>>>> > On Wed, May 26, 2010 at 5:04 PM, StuFF mc <m...@stuffmc.com> wrote: >>>> >> I just use the rubygem "daemons" and it's supposed to output in >>>> >> log/name_of_the_daemon.rb.log but the last errors I had there were >>>> >> from March.
>>>> >> Attached a chart that shows when I restarted the daemon. We'll see >>>> >> over time if the require was the problem. Aside the daemon, would you >>>> >> have a good tip to profile my "Episode.fetch_all"?
>>>> >> script/performance isn't much helpful i.m.h.o.
>>>> >> On Wed, May 26, 2010 at 10:59 PM, Paul Dix <p...@pauldix.net> wrote: >>>> >>> It depends on how/what you're using to daemonize the process that is >>>> >>> running these updates. Whatever it is should have the ability to write >>>> >>> STDOUT and STDERR to some log file. If you aren't specifying that then >>>> >>> the process is probably just throwing an exception that isn't being >>>> >>> captured by anything.
>>>> >>> Best, >>>> >>> Paul
>>>> >>> On Wed, May 26, 2010 at 4:20 PM, StuFF mc <m...@stuffmc.com> wrote: >>>> >>>> Thanks for answering so quickly Paul.
>>>> >>>> True, it's pretty dumb to have this require there. I remember having >>>> >>>> tried to have it in my environment.rb but was unsuccessful. Anyways, >>>> >>>> I >>>> >>>> moved it to the top of the source file (so outside the method) and >>>> >>>> we'll see if it changes anything.
>>>> >>>> Concerning the logs, they are supposed to be in /log, right? There >>>> >>>> nothing relevant there :(
>>>> >>>> Cheers.
>>>> >>>> On Wed, May 26, 2010 at 10:11 PM, Paul Dix <p...@pauldix.net> wrote: >>>> >>>>> Is the daemon failing with some sort of error? I assume that it's >>>> >>>>> just >>>> >>>>> not failing quietly. The graph doesn't really do anything to help >>>> >>>>> troubleshoot. Is there a reason you're doing a require inside the >>>> >>>>> "add_entries" method? I don't think you want to be rerunning that >>>> >>>>> everytime you run it.
>>>> >>>>> Log the output from that daemon.
>>>> >>>>> Best, >>>> >>>>> Paul
>>>> >>>>> On Wed, May 26, 2010 at 4:00 PM, StuFF mc <m...@stuffmc.com> wrote: >>>> >>>>>> Since I'm not sure you can add images to this group, it's also >>>> >>>>>> here: >>>> >>>>>> http://emberapp.com/stuffmc/images/cpu-graph-rp
>>>> >>>>>> This graph shows how nuts the process is going (before to be killed >>>> >>>>>> I >>>> >>>>>> guess) after the daemons has been running for about a day or 2.
>>>> >>>>>> I haven't take the time yet to look at how God can help me - the >>>> >>>>>> last >>>> >>>>>> time I looked at God it wasn't trivial :(
>>>> >>>>>> But from the Feedzirra perspective (and thus, this is a question to >>>> >>>>>> the main dev), where could be the problem?
>>>> >>>>>> On Mon, May 24, 2010 at 8:50 PM, Samuel Lown <sam.l...@gmail.com> >>>> >>>>>> wrote:
>>>> >>>>>>> On 24 May 2010 20:01, StuFF mc <m...@stuffmc.com> wrote:
>>>> >>>>>>>> thanks for the tips. I had thought about spawn & god but what's >>>> >>>>>>>> the >>>> >>>>>>>> point of a daemon if the connection to Active Record goes away?!
>>>> >>>>>>> The DB can timeout a connection if its not being used or cut-out >>>> >>>>>>> on network >>>> >>>>>>> failure. This probably isn't a problem in your situation though. >>>> >>>>>>> It would be >>>> >>>>>>> if you were forking, as closing a fork will also close the DB >>>> >>>>>>> connection.
>>>> >>>>>>>> Also >>>> >>>>>>>> how come the memory isn't freed.
>>>> >>>>>>> You can try some profiling techniques if you want to discover >>>> >>>>>>> where your >>>> >>>>>>> memory is leaking. You're reading lots of external data and >>>> >>>>>>> passing it >>>> >>>>>>> through several libraries, there could be tiny leaks anywhere.
>>>> >>>>>>> I suspect most people give-up on this. Ruby makes programming >>>> >>>>>>> easier at the >>>> >>>>>>> cost of less-refined memory management, this is why god scripts >>>> >>>>>>> exist to >>>> >>>>>>> restart mongrel or unicorn processes to free-up trapped memory.
>>>> >>>>>>>> Can I call "something" in my loop to >>>> >>>>>>>> "clean up"? Do I have to?
>>>> >>>>>>> You can try forcing garbage collection, but I doubt that will >>>> >>>>>>> solve much:
>>>> >>>>>>> Seriously, if you're handling lots of data in a daemon during a >>>> >>>>>>> long period >>>> >>>>>>> of time, fork it, process, clean.
>>>> >>>>>>> Cheers, >>>> >>>>>>> sam
>>>> >>>>>>>> Cheers.
>>>> >>>>>>>> On Mon, May 24, 2010 at 5:48 PM, Samuel Lown <sam.l...@gmail.com> >>>> >>>>>>>> wrote: >>>> >>>>>>>> > Hi, >>>> >>>>>>>> > Check that its not running out of memory. god is useful for to >>>> >>>>>>>> > keeping >>>> >>>>>>>> > things running. >>>> >>>>>>>> > You might also want to checkout spawn and fork the process >>>> >>>>>>>> > before doing >>>> >>>>>>>> > a >>>> >>>>>>>> > loop and wait for it to complete. This will ensure your memory >>>> >>>>>>>> > is >>>> >>>>>>>> > release >>>> >>>>>>>> > after each loop, and automatically handle the ActiveRecord >>>> >>>>>>>> > database >>>> >>>>>>>> > connection (which you may also be loosing after a period of >>>> >>>>>>>> > time.) >>>> >>>>>>>> > Cheers, >>>> >>>>>>>> > sam >>>> >>>>>>>> > On 24 May 2010 10:39, StuFF mc <m...@stuffmc.com> wrote:
>>>> >>>>>>>> >> I don't think I'm doing anything wrong in the code itself. For >>>> >>>>>>>> >> your >>>> >>>>>>>> >> information I have 498 podcasts and currently 17961 episodes. >>>> >>>>>>>> >> I'm >>>> >>>>>>>> >> pretty sure the script runs for a few days before somehow >>>> >>>>>>>> >> dying.
>>>> >>>>>>>> >> Anybody else experiencing this?
>>>> >>>>>>>> >> Cheers.
>>>> >>>>>>>> >> -- >>>> >>>>>>>> >> You received this message because you are subscribed to the >>>> >>>>>>>> >> Google >>>> >>>>>>>> >> Groups >>>> >>>>>>>> >> "feedzirra" group.
