Comments on: Ruby Background Tasks with Starling http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/ Some random thoughts - Go big or stay home! Thu, 28 Aug 2008 22:34:43 +0000 http://wordpress.org/?v=2.5.1 By: Dave http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-229 Dave Tue, 22 Jul 2008 11:32:06 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-229 This is a pretty common Rails issues. When you have multiple instances of a model (one in the app and one in the workling in your case), and one instance saves, the other will not see the change. You have a couple of options. 1. Make sure to reload the model in your controller. This may work, but timing will be a problem. What if the workling is delayed? 2. Use a transaction and lock the model. Just be careful with locking multiple models, you could end up with a deadlock. If you do modify multiple models, make sure you always access them in the same order. This is a pretty common Rails issues. When you have multiple instances of a model (one in the app and one in the workling in your case), and one instance saves, the other will not see the change. You have a couple of options.

1. Make sure to reload the model in your controller. This may work, but timing will be a problem. What if the workling is delayed?

2. Use a transaction and lock the model. Just be careful with locking multiple models, you could end up with a deadlock. If you do modify multiple models, make sure you always access them in the same order.

]]> By: Raj http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-228 Raj Tue, 22 Jul 2008 04:09:48 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-228 Hi Dave, we are facing an issue while updating a model from workling.the issue is,i have a counter column in a table and i increment this from controller and decrement from workling.This counter is used for controlling number of workling to be triggered. The table in DB is showing the decremented value but still controller could not see the change in counter by the workling.Is this the expected behaviour of workling. Is there any way to access the changes made to model (db table) by the controller. Thanks in advance Kevin. Hi Dave,

we are facing an issue while updating a model from workling.the issue is,i have a counter column in a table and i increment this from controller and decrement from workling.This counter is used for controlling number of workling to be triggered.
The table in DB is showing the decremented value but still controller could not see the change in counter by the workling.Is this the expected behaviour of workling.
Is there any way to access the changes made to model (db table) by the controller.

Thanks in advance Kevin.

]]> By: Dave http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-227 Dave Wed, 16 Jul 2008 12:53:49 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-227 No, you're not missing anything. Certainly, if I was changing the workers then I would want them to empty out their queues before updating. In that case, I would do: 1. Stop my application to prevent new messages 2. run stats on starling to see how many items are left to process 3. When all queues are empty, restart workling 4. Start my application I suppose to be safest you would want this automated in the workling shutdown process. However, if you're queues tend to be long, and you haven't changed a worker, then all you really need to do is stop all the workling threads in between a message. The ultimate would be to have both options available. No, you’re not missing anything. Certainly, if I was changing the workers then I would want them to empty out their queues before updating. In that case, I would do:

1. Stop my application to prevent new messages
2. run stats on starling to see how many items are left to process
3. When all queues are empty, restart workling
4. Start my application

I suppose to be safest you would want this automated in the workling shutdown process. However, if you’re queues tend to be long, and you haven’t changed a worker, then all you really need to do is stop all the workling threads in between a message. The ultimate would be to have both options available.

]]> By: Aaron Gibralter http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-226 Aaron Gibralter Wed, 16 Jul 2008 01:34:43 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-226 Wouldn't you want a worker to finish off all of its queues before dying for consistency? For instance, if you deploy a new version of a site which changes the interaction between the app and the workers, a new worker picking up on an old queue might break things. Of course this can lead to memory strain when the queues are long and you're deploying a new app (new workers + old workers finishing off queues), but it seems like the only safe option besides stopping the queues early. Perhaps I'm missing something fundamental about workling/starling? Wouldn’t you want a worker to finish off all of its queues before dying for consistency? For instance, if you deploy a new version of a site which changes the interaction between the app and the workers, a new worker picking up on an old queue might break things. Of course this can lead to memory strain when the queues are long and you’re deploying a new app (new workers + old workers finishing off queues), but it seems like the only safe option besides stopping the queues early.

Perhaps I’m missing something fundamental about workling/starling?

]]> By: Dave http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-225 Dave Mon, 14 Jul 2008 15:11:18 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-225 I hadn't thought of using a special queue because I was concerned that if there are many methods (queues) in a worker, it might take a while to get to the stop message. I was working through some form of signaling mechanism so the process isn't just killed when you stop it. I have some of the code in there already to stop things, but daemon coding is not my area of expertise so I wasn't able to spend the time necessary to get the signal to work correctly. If you are up for suggesting a working scheme, please feel free to do so. I hadn’t thought of using a special queue because I was concerned that if there are many methods (queues) in a worker, it might take a while to get to the stop message. I was working through some form of signaling mechanism so the process isn’t just killed when you stop it. I have some of the code in there already to stop things, but daemon coding is not my area of expertise so I wasn’t able to spend the time necessary to get the signal to work correctly.

If you are up for suggesting a working scheme, please feel free to do so.

]]> By: Aaron Gibralter http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-224 Aaron Gibralter Sun, 13 Jul 2008 17:56:04 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-224 Hi Dave, Thank you for the response! Sorry I was a bit trigger happy with that post -- I should have read the rest of your blog first :). Anyway, as for the last point there (restarting workling), might it be possible to have each started instance of workling use a unique starling queue (perhaps prefixed by the timestamp of when workling starts up). That way, you can "signal" the workling process to shut down (set some sort of class variable?), and it will once it has exhausted its own starling queue. Plus, this would mean that for a new deploy, you could start a new workling right away to work with the new code. Just a few thoughts I guess... perhaps you've already considered them. Hi Dave,

Thank you for the response! Sorry I was a bit trigger happy with that post — I should have read the rest of your blog first :).

Anyway, as for the last point there (restarting workling), might it be possible to have each started instance of workling use a unique starling queue (perhaps prefixed by the timestamp of when workling starts up). That way, you can “signal” the workling process to shut down (set some sort of class variable?), and it will once it has exhausted its own starling queue. Plus, this would mean that for a new deploy, you could start a new workling right away to work with the new code. Just a few thoughts I guess… perhaps you’ve already considered them.

]]> By: Dave http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-223 Dave Tue, 08 Jul 2008 15:56:02 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-223 I haven't seen too many (outside of this series). Regarding your other questions, let me have go. The only external gems necessary for workling/starling are starling itself and the memcache-client. The memcached server is not required unless you use it for other things. One of the nice things about starling is that it is built on the memcached interface, and that means it is available from many languages and platforms. Plus, since so many apps use memcached for caching, it's nice not to have yet another protocol to deal with. Other than the occasional reboot for other reasons, I have never restarted starling or had it go into any form of bad state. It's been a rock. Even though I do have cap tasks for start/stop/restart of starling, those tasks are not part of any recipes. I restart workling whenever I restart my application. See part 3 (http://davedupre.com/2008/04/01/ruby-background-tasks-with-starling-part-3/) for a description of my cap tasks. Now, there is one minor issue with this scheme that I haven't got a good fix for yet. If you do stop the workling process, it may kill a worker in the middle of a task. This is due to how daemons is implemented (workling uses daemons). I've been in touch with the author to figure out a safe way to kill the process, but we haven't made much progress. What I want is a way to signal the pollers to exit their loops, but daemons makes that difficult. I suppose you could stop the application, wait for workling to get through everything in the queues, restart workling, then start your app. I haven't got to the point where this is a real problem yet, but I could see it coming. I haven’t seen too many (outside of this series).

Regarding your other questions, let me have go.

The only external gems necessary for workling/starling are starling itself and the memcache-client. The memcached server is not required unless you use it for other things. One of the nice things about starling is that it is built on the memcached interface, and that means it is available from many languages and platforms. Plus, since so many apps use memcached for caching, it’s nice not to have yet another protocol to deal with.

Other than the occasional reboot for other reasons, I have never restarted starling or had it go into any form of bad state. It’s been a rock. Even though I do have cap tasks for start/stop/restart of starling, those tasks are not part of any recipes.

I restart workling whenever I restart my application. See part 3 (http://davedupre.com/2008/04/01/ruby-background-tasks-with-starling-part-3/) for a description of my cap tasks.

Now, there is one minor issue with this scheme that I haven’t got a good fix for yet. If you do stop the workling process, it may kill a worker in the middle of a task. This is due to how daemons is implemented (workling uses daemons). I’ve been in touch with the author to figure out a safe way to kill the process, but we haven’t made much progress. What I want is a way to signal the pollers to exit their loops, but daemons makes that difficult. I suppose you could stop the application, wait for workling to get through everything in the queues, restart workling, then start your app. I haven’t got to the point where this is a real problem yet, but I could see it coming.

]]> By: Aaron Gibralter http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-222 Aaron Gibralter Tue, 08 Jul 2008 15:03:26 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-222 Are there any comprehensive resources on getting starling/workling up and running in a multi-stage environments? I'm still really confused as to what exactly is necessary for starling/workling. Do I need to set up a memcached server or is '$sudo gem install starling' enough? What capistrano tasks should I have? Should starling be restarted ever? When should workling be restarted? If I '$cap deploy' what happens to the current workling processes? Thanks so much in advance! Are there any comprehensive resources on getting starling/workling up and running in a multi-stage environments?

I’m still really confused as to what exactly is necessary for starling/workling. Do I need to set up a memcached server or is ‘$sudo gem install starling’ enough?

What capistrano tasks should I have? Should starling be restarted ever? When should workling be restarted? If I ‘$cap deploy’ what happens to the current workling processes?

Thanks so much in advance!

]]> By: Ram http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-216 Ram Fri, 20 Jun 2008 21:09:08 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-216 config.cache_classes is false and rest of my models, controllers don't require reloads. I even tried putting require_dependency in development.rb for my worker classes. Anyway, my current solution works - I basically have AsyncWorker < MyCustomBG < Workling::Base MyCustomBG has method_missing defined inside a class_eval based on the definition of Workling::Remote.dispatcher It's a hack - but works for now :) Thanks a lot! config.cache_classes is false and rest of my models, controllers don’t require reloads. I even tried putting require_dependency in development.rb for my worker classes. Anyway, my current solution works - I basically have
AsyncWorker < MyCustomBG < Workling::Base
MyCustomBG has method_missing defined inside a class_eval based on the definition of Workling::Remote.dispatcher
It’s a hack - but works for now :)

Thanks a lot!

]]> By: Dave http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-215 Dave Fri, 20 Jun 2008 13:43:06 +0000 http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/#comment-215 Interesting. Do you have: config.cache_classes = false in your config/environments/development.rb file? When running in synchronous mode, your worker classes should be exactly like any other classes and should be reloaded automatically. I haven't had this issue, but I tend to do the majority of my development and test in the console anyway. Wish I could be more help, but if you have cache_classes = false your worker classes should reload automatically. Do other classes reload for you, or is it only the worker classes? Interesting. Do you have:

config.cache_classes = false

in your config/environments/development.rb file?

When running in synchronous mode, your worker classes should be exactly like any other classes and should be reloaded automatically. I haven’t had this issue, but I tend to do the majority of my development and test in the console anyway.

Wish I could be more help, but if you have cache_classes = false your worker classes should reload automatically. Do other classes reload for you, or is it only the worker classes?

]]>