Patterns for an imperative problem

[Aaron Cruz]

I was just at baruco and all of the pattern talk reminded me of a problem that comes up with my projects fairly often.

I have a very imperative problem. Something needs to be done, then something else, then something else, constantly changing state from one step to the next.

I imagine there are common patterns for this but either I don't remember them or never learned them.

Example: Video Import

Steps

1. Look in directory for files in a heirarchy of folders
2. Organize by folder names
3. Group videos by suffix or prefix in filename
4. Persist videos
5. Send to workers to transcode
   

I am very interested in your answers.

--

bussi,

Aaron

[Ben Lovell]

Hey Aaron,

On 17 September 2013 17:00, Aaron Cruz <pferde...@gmail.com> wrote:

I was just at baruco and all of the pattern talk reminded me of a problem that comes up with my projects fairly often.

I have a very imperative problem. Something needs to be done, then something else, then something else, constantly changing state from one step to the next.

I imagine there are common patterns for this but either I don't remember them or never learned them.

Sounds to me like you're describing a pipeline - with each action dependent on the action prior. The elements of which can be composed and reused as required. I've recently delivered a similar solution in this exact architecture - where the individual elements of the pipeline are arranged as filters/mappers (this distinction can be significant, it was in my case) and could decide to veto further execution in the pipeline should the incoming message not satisfy the required precondition(s). 

I guess the closest design pattern would be pipes and filters as detailed by the EIP book: http://www.eaipatterns.com/PipesAndFilters.html

If you're disciplined enough to keep the individual steps very focused/cohesive this can be a very satisfying approach to a common problem and promotes a healthy amount of reuse.

As an aside - my implementation was based upon celluloid actors and was almost trivially easy to arrange individual pipeline steps as 'actors' (ideally based upon their work profile - being I/O or computational) and have them take advantage of any available cores for executing in parallel.

Lots to discuss - I'm happy to pair on something with you if you wanted to get something concrete in place...

Cheers,

Ben

 

Example: Video Import

Steps

1. Look in directory for files in a heirarchy of folders
2. Organize by folder names
3. Group videos by suffix or prefix in filename
4. Persist videos
5. Send to workers to transcode
   

I am very interested in your answers.

--

bussi,

Aaron

--
You received this message because you are subscribed to the Google Groups "Objects on Rails" group.
To unsubscribe from this group and stop receiving emails from it, send an email to objects-on-rai...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Aaron Cruz]

I can view this as two things, one I don't love but feels more like the pattern and the other I like more but isn't really a "pipe."

A pipe reads to me like a callback loop. In my example it may look like

# Pseudo-pseudo code - lots of missing stuff

class ImportsVideos

  def self.perform

    ReadDirectory.process do |results|

      SeparateFeeds.process(results) do |feeds|

        PullIntrosOutros.process(feeds) do |video_groups|

          PersistVideos.process(video_groups) do |videos|

            Transcoder.process(videos)

          end

        end

      end

    end

  end

end

One way to make this gross would be to use a defer system but that feels like overkill to me.

The other way I can think of doing this would be to have an object that holds the state of your "list" and iterate through a few filters that all mutate the global object

class VideoList

  attr_accessor :list

  def initialize

    @list = []

  end

end

class ImportsVideos

  def initialize list

    @list = list

  end

  def perform

    [

      ReadDirectory,

      SeparateFeeds,

      PullIntrosOutros,

      PersistVideos,

      Transcoder

    ].each do |filter|

      filter.process @list

    end

    @list

  end

end

list = VideoList.new

importer = ImportsVideos.new list

processed_list = importer.perform

The second is easier (for me) to read and uses filters but there is no unix pipiness to it, which I really like the idea of.

Your thoughts?

[Patrick Mulder]

An association that comes up to me is a "RunList" to provision a VM, e.g. http://docs-v1.vagrantup.com/v1/docs/provisioners/chef_solo.html#specifying_the_run_list

Not by coincidence, provisioning steps in Chef come from "cookbooks".

--

[Aaron Cruz]

Which reminds me of this

https://github.com/euruko2012/call-for-proposals/pull/83

video link - seems to be broken somehow :/

http://www.youtube.com/watch?v=i6pyhq3ZvyI

--

aaron

[Arne Brasseur]

A rule of thumb I find helpful for breaking up functionality is
"separate policy from mechanism". It's usually talked about in the
context of security or operating system resource allocation, but it's
also great when reasoning about OO design.

Each of your steps are generic mechanisms, behavior with a few
parameters that can be implemented in a clean, isolated way. After
that wiring them together becomes trivial, and gives you high level
code that specifies the policy of your program.

files = FileFinder.new.call(folders)
files = FolderOrganizer.new(by: :name).call(files)
grouped = VideoGrouper.new(by: [:name, :prefix]).call(grouped)
...

The policy is more likely to change than the mechanisms, so its good
if it's separate and descriptive. Implementing an actual pipeline
system would be worth it if the steps to take are themselves
configurable.

[Patrick Mulder]

Thanks very much for that pointer, indeed middleware in Vagrant looks very interesting: https://github.com/mitchellh/vagrant/blob/master/plugins/providers/virtualbox/action.rb#L52-L78

[Patrick Mulder]

Just come across this:  http://blog.jcoglan.com/2011/03/11/promises-are-the-monad-of-asynchronous-programming/

Translating to your import, the problem could be formulated as:

   pipe(dirname,

        [ readDir,

          organizeByFolder,

          groupFiles,

          persist,

          upload        ]);

Since one process depends on another, there must be a way to synchronize those. I am just looking whether https://github.com/jclem/q-defer or https://github.com/cotag/em-promise can help.

Anyone here has used such Pipes with Ruby?

Cheers,

Patrick

--

[Aaron Cruz]

Thanks! Those articles are great.

Now I'm off to overuse the crap out of this pattern.
--

aaron

[Alexandre de Oliveira]

Guys, imo hardcoded pipes in one object is no OO at all, but pure procedural code:

- the entire knowledge is monopolized by one object

- they aren't passing messages to each other (how can this be OO)

- you can't add new steps in the flow without pain (if you add a 4th step, you have to change 5th, 6th so and so)

For the sake of learning, I'd love to be proved wrong. So I wrote this small article for you guys to refute my points above: https://gist.github.com/kurko/6662513

--Alexandre

[Charles Hoffman]

maybe it's not supposed to be OO, maybe it's functional

--
[chuck hoffman]
[sounds, words, and code]
[what else is there?]
[http://hoff2.com]