Can we keep validation in memcached ?

[Fahim Patel]

Hi All ,

Can we validate request input data using memcache ? Actually I am facing scenario in which user uploading CSV file which can have hundreds of rows and to validate each row using ActiveRecord taking long time.

I have below question:-

1. So can we cache our validation rule in memcache ?
2. If it is possible to store validation rules in memcache but can I able to perform
validation check ?

[Peter Hickman]

Memcached is only useful for storing the responses and unless the validation rules are incredibly complicated using memcached is not going to speed anything up.

Simply not the right tool for the job, the real question is why are your validation rules so slow.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
To post to this group, send email to rubyonra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/10f8edbc-b78c-4cbe-9712-62e57b7105e9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Fahim Patel]

Thanks Peter to reply.

Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. 

Major issue is user will upload CSV file which can consist hundred's of rows.Through active record validating each row consuming lots of time.

That's why I am thinking to move validation rule to cache level. It will fast process.

Is this can be possible to validate data on cache level ?

[Fahim Patel]

Thanks Peter to reply.

Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. 

Major issue is user will upload CSV file which can consist hundred's of rows.Through active record validating each row consuming lots of time.

That's why I am thinking to move validation rule to cache level. It will fast process.

Is this can be possible to validate data on cache level ?

 

--
You received this message because you are subscribed to a topic in the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rubyonrails-talk/-h-dG0UWT5o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rubyonrails-ta...@googlegroups.com.

To post to this group, send email to rubyonra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/CALxYQy4x4-5JGcT6bLdq8w13BuJgo-9Vf225JEdnEiGKKUPReg%40mail.gmail.com.

[Josh Jordan]

Why do you think memcached is only useful for storing response data? That is simply not true. Memcache has been around longer than Rails, and is useful as a general purpose, fast, volatile data store. You can use it for anything you like, and it applies well in any case where there is deterministic output given a certain set of inputs.

[Fahim Patel]

Thanks for reply. 

@josh - So you are saying that we can validate request on memcache level also. Please correct me If I am wrong ?

On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote:

[Sur Max]

@Fahim

In my opinion memcached is not a suitable tool for the purpose you've mentioned. In fact it'd be waste of resources in such a case.

I would go for processing that thing in background using any of the available gem for processing the background job.

Showing an interim screen to the user mentioning that your uploaded file is being processed and process it in background with the help of ActiveRecord + DelayedJob.

[Tamara Temple]

On Nov 4, 2013, at 2:42 AM, Sur Max <sur...@gmail.com> wrote:

> @Fahim
>
> In my opinion memcached is not a suitable tool for the purpose you've mentioned. In fact it'd be waste of resources in such a case.
> I would go for processing that thing in background using any of the available gem for processing the background job.
> Showing an interim screen to the user mentioning that your uploaded file is being processed and process it in background with the help of ActiveRecord + DelayedJob.
>

I tend to agree. ActiveRecord does not do well for mass insert/updates, and better to offload this task and not keep the Rails stack busy with it.

I also wouldn’t hold up the user with a wait screen. I’d put a little indicator in an area that shows the processing is being done but does not restrict further use of the site, except for that new data. When the background processing is finished, the indicator can be removed.

>
> On Monday, 4 November 2013 00:26:08 UTC+5:30, Fahim Patel wrote:
> Thanks for reply.
> @josh - So you are saying that we can validate request on memcache level also. Please correct me If I am wrong ?
>
>
> On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote:
> Hi All ,
>
> Can we validate request input data using memcache ? Actually I am facing scenario in which user uploading CSV file which can have hundreds of rows and to validate each row using ActiveRecord taking long time.
>
> I have below question:-
>
> 1. So can we cache our validation rule in memcache ?
> 2. If it is possible to store validation rules in memcache but can I able to perform
> validation check ?
>
>

> --
> You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
> To post to this group, send email to rubyonra...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/70f5120b-b8f2-4726-8467-586edaf2873f%40googlegroups.com.

[Fahim Patel]

@sur and @tam thanks for reply....

@sur -  Currently I am using rescue ....I am using worker's but still I am thinking to use memcache for fast performance....What you say can I achieve validation on cache level.

@tam - I even showing processing image till process done....

Friends ....can i achieve fast performance through putting validation on cache level .....or I have to assume that this is max performance code....  

On Nov 4, 2013, at 2:42 AM, Sur Max <sur...@gmail.com> wrote: 

> @Fahim 
> 
> In my opinion memcached is not a suitable tool for the purpose you've mentioned. In fact it'd be waste of resources in such a case. 
> I would go for processing that thing in background using any of the available gem for processing the background job. 
> Showing an interim screen to the user mentioning that your uploaded file is being processed and process it in background with the help of ActiveRecord + DelayedJob. 
> 

I tend to agree. ActiveRecord does not do well for mass insert/updates, and better to offload this task and not keep the Rails stack busy with it. 

I also wouldn’t hold up the user with a wait screen. I’d put a little indicator in an area that shows the processing is being done but does not restrict further use of the site, except for that new data. When the background processing is finished, the indicator can be removed. 

> 
> On Monday, 4 November 2013 00:26:08 UTC+5:30, Fahim Patel wrote: 
> Thanks for reply. 
> @josh - So you are saying that we can validate request on memcache level also. Please correct me If I am wrong ? 
> 
> 
> On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote: 

> -- 
> You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. 
> To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com. 
> To post to this group, send email to rubyonra...@googlegroups.com. 

> To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/70f5120b-b8f2-4726-8467-586edaf2873f%40googlegroups.com.

On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote:

[Tamara Temple]

On Nov 4, 2013, at 4:32 AM, Fahim Patel <paf...@gmail.com> wrote:

> @sur and @tam thanks for reply....
>
> @sur - Currently I am using rescue ....I am using worker's but still I am thinking to use memcache for fast performance....What you say can I achieve validation on cache level.
>
> @tam - I even showing processing image till process done....
>

> Friends ....can i achieve fast performance through putting validation on cache level .....or I have to assume that this is max performance code….

Here’s what I don’t get: what could you possibly be caching here? If what’s happening is a user is uploading a CSV file that is going to inserted into table, what is going to be cached?

> To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/7902ebce-ee88-42f7-bd2c-cbe9b7374d18%40googlegroups.com.

[Fahim Patel]

Thanks @tamouse for  reply.

Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. 

Major issue is user will upload CSV file which can consist hundred's of rows.Through active record validating each row consuming lots of time.

That's why I am thinking to move validation rule to cache level. It will fast process.

Hope you understood situation ....

- show quoted text -

On Nov 4, 2013, at 4:32 AM, Fahim Patel <paf...@gmail.com> wrote: 

> @sur and @tam thanks for reply.... 
> 
> @sur -  Currently I am using rescue ....I am using worker's but still I am thinking to use memcache for fast performance....What you say can I achieve validation on cache level. 
> 
> @tam - I even showing processing image till process done.... 
> 

> Friends ....can i achieve fast performance through putting validation on cache level .....or I have to assume that this is max performance code….   

Here’s what I don’t get: what could you possibly be caching here? If what’s happening is a user is uploading a CSV file that is going to inserted into table, what is going to be cached? 

- show quoted text -

> To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/7902ebce-ee88-42f7-bd2c-cbe9b7374d18%40googlegroups.com. 

- show quoted text -

On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote:

[Sur]

Thanks @tamouse for  reply.

Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation. 

Major issue is user will upload CSV file which can consist hundred's of rows.Through active record validating each row consuming lots of time.

That's why I am thinking to move validation rule to cache level. It will fast process.

If I understood you correctly,

Validation rule resides inside a Ruby class i.e. a class inheriting ActiveRecord::Base, which is already there in memory to even be functional. It doesn't make any other sense to cache this class (or rules inside the class). Further, Rails automatically caches all the classes in production environment. Do you correctly understand this ?

In short, your say "I am thinking to move validation rule to cache level" itself is incorrect because the validation is already in memory and is already cached, you can not speed it up further by saying you want to cache it.

And for processing millions of rows with 15 columns each row, by any chance it surely is going to be a slow procedure even with memcached (unless you want to put a super computer in place for this task)... and the practical solution is to move this to background job.

or, I didn't understand you at all.

regards,

Sur

crimson9.com

[Fahim Patel]

Thanks @sur for reply.....

Yes you understood correctly my issue, and I even understood your solution....

let me cross check your solution and will update here.

If I understood you correctly,

Validation rule resides inside a Ruby class i.e. a class inheriting ActiveRecord::Base, which is already there in memory to even be functional. It doesn't make any other sense to cache this class (or rules inside the class). Further, Rails automatically caches all the classes in production environment. Do you correctly understand this ?

In short, your say "I am thinking to move validation rule to cache level" itself is incorrect because the validation is already in memory and is already cached, you can not speed it up further by saying you want to cache it.

And for processing millions of rows with 15 columns each row, by any chance it surely is going to be a slow procedure even with memcached (unless you want to put a super computer in place for this task)... and the practical solution is to move this to background job.

or, I didn't understand you at all.

regards,

Sur

crimson9.com

On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote:

[Tamara Temple]

On Nov 4, 2013, at 6:02 AM, Fahim Patel <paf...@gmail.com> wrote:

> Thanks @tamouse for reply.
> Actually my table consist more than 15 columns and each column have lots of validation like presence validation,format validation.
> Major issue is user will upload CSV file which can consist hundred's of rows.Through active record validating each row consuming lots of time.
>
> That's why I am thinking to move validation rule to cache level. It will fast process.

How do you move a validation rule to memcache?

>
> Hope you understood situation ....
> - show quoted text -
>
> On Nov 4, 2013, at 4:32 AM, Fahim Patel <paf...@gmail.com> wrote:
>
> > @sur and @tam thanks for reply....
> >
> > @sur - Currently I am using rescue ....I am using worker's but still I am thinking to use memcache for fast performance....What you say can I achieve validation on cache level.
> >
> > @tam - I even showing processing image till process done....
> >
> > Friends ....can i achieve fast performance through putting validation on cache level .....or I have to assume that this is max performance code….
>
> Here’s what I don’t get: what could you possibly be caching here? If what’s happening is a user is uploading a CSV file that is going to inserted into table, what is going to be cached?
> - show quoted text -
> > To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/7902ebce-ee88-42f7-bd2c-cbe9b7374d18%40googlegroups.com.
> - show quoted text -
>
> On Sunday, November 3, 2013 1:46:20 AM UTC+5:30, Fahim Patel wrote:
> Hi All ,
>
> Can we validate request input data using memcache ? Actually I am facing scenario in which user uploading CSV file which can have hundreds of rows and to validate each row using ActiveRecord taking long time.
>
> I have below question:-
>
> 1. So can we cache our validation rule in memcache ?
> 2. If it is possible to store validation rules in memcache but can I able to perform
> validation check ?
>
>

> --
> You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
> To post to this group, send email to rubyonra...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/1eb12dbd-4a58-4de4-9d4d-83bf8c3c4f0d%40googlegroups.com.

[Fahim Patel]

@tam - I am even not sure how can experiment it.  Collecting data and studying in progress.

@sur - I am still  cross checking.. If you provide any reference link or data to proof that validation already get cached.

Friends and @sur  please clear me on below question-

From rails app, when user insert data then each time data hitted on database for validation or validation is present in cache ? 

If validation get cached then we don't need move validation on cache.

[Dheeraj Kumar]

This is a great use case for a background job. Have you tried Resque?

If I have a user uploading thousands of rows, all of which need to be validated and processed/saved, I’d do it like this:

1. Let the user upload the file, save it somewhere. Trigger a Resque job.
2. Read the file one or more rows at a time, make a ActiveRecord or ActiveModel object out of them and run the validators. Collate the errors and include the row number.
3. If all the rows were valid, continue processing or saving.
4. If not, send an email to the uploader and/or show in his dashboard the failure message, along with a list of all errors with row numbers.

I’ve used this method in our app, and is working out great for our users.

-- 

Dheeraj Kumar

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
To post to this group, send email to rubyonra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/884cb08e-d516-4144-90de-701fbabd2497%40googlegroups.com.

[Fahim Patel]

@dheeraj - thanks for reply.I am using resque & redis. You  exactly got use case .

can you please give answer for below point:-

1. From rails app, when user insert data then each time data hitted on database for validation or validation is present in cache ? 

[Dheeraj Kumar]

First validate the data, and then insert it to the database. Till then, it’s read from the file and still in-memory.

I’m not bringing the cache into the picture at all. 

-- 

Dheeraj Kumar

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-ta...@googlegroups.com.
To post to this group, send email to rubyonra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-talk/213e0dea-5492-4476-8695-5fe284825d94%40googlegroups.com.

[Sur]

Well, this is a classic use case when we jump too early on Rails not understanding the Ruby (or programming in general) to at least a decent level.... let alone the meta-programming stuff.

One should get a good hold of Ruby at least to a level where s/he understands how the memory mapping is working behind the scenes... otherwise you can end up writing the piles of junk which is eating up all your server's memory.

@tam - I am even not sure how can experiment it.  Collecting data and studying in progress.

@sur - I am still  cross checking.. If you provide any reference link or data to proof that validation already get cached.

"If you provide any reference link or data to proof that validation already get cached."

I got a smile when I read the above line. :D

a humble suggestion:

before finding any proof, you gotta study more about programming and how the memory space is managed on the runtime.

Do you know what is the difference between a string and a symbol and why and where symbols are preferred over strings ?

try this:

a = "some string"

b = "some string"

b.object_id == a.object_id  (watch for the result of this expression)

sym_a = :some_symbol

sym_b = :some_symbol

sym_b.object_id == sym_a.object_id  (watch for the result of this expression)

Now, you don't need to go too far for finding the proof for the validation/class caching:

(let alone the Rails production environment, in one process of Ruby, if Ruby can't cache the class why would you even use it ?)

user_class = class User; end

another_user_class = class User; end

user_class.object_id == another_user_class.object_id  (watch for the result of this expression)

Note:- object_id is the method available to every single object in Ruby (remember every smallest/biggest entity in Ruby you have is an object)... and it returns the id of that object from the memory map.

Play as much as possible with the above snippets in IRB. IRB is interactive Ruby console, you can initiate it by typing "irb" in your terminal and then can type the above snippets to test there.

Try tasting the flavor of Ruby to a bit of the next levels.

regards,

Sur

crimson9.com

[Jordon Bedwell]

On Tue, Nov 5, 2013 at 12:07 AM, Sur <sur...@gmail.com> wrote:
> Do you know what is the difference between a string and a symbol and why and
> where symbols are preferred over strings ?
>
> try this:
>
> a = "some string"
> b = "some string"
> b.object_id == a.object_id (watch for the result of this expression)

> sym_a = :some_symbol
> sym_b = :some_symbol
> sym_b.object_id == sym_a.object_id (watch for the result of this
> expression)

Preach on about this bullshit when Ruby isn't primarily used for web
applications and when Ruby allows my system to reclaim memory, until
then lets leave the symbol argument bullshit out of this. Even if
they are constants they can lead to trouble if you just blindly state
they are better without explaining the dangers in a web application,
or people will repeat history, history people like me thought was
already common knowledge.

[Fahim Patel]

@sur - will check as you said.

@jordan - not understood your reply..

[Sur]

Preach on about this bullshit when Ruby isn't primarily used for web
applications and when Ruby allows my system to reclaim memory, until
then lets leave the symbol argument bullshit out of this.  Even if
they are constants they can lead to trouble if you just blindly state
they are better without explaining the dangers in a web application,
or people will repeat history, history people like me thought was
already common knowledge.

:)

well it'd be interesting to know why you're frustrated out of Symbol concepts of Ruby ?

I didn't say symbols are better than strings, in fact I asked that it's important to know for a developer that why and where they can be used!

when did I say Ruby is best/primary used for web ? I use it in chip programming and it's been great so far :)

It's an individual's choice to select a language/framework to achieve the solution to a problem, the main aim is to resolve a problem, now once decided to go with a language, one hast to know the limits and pros and cons of that language which certainly are there the way it's creators have made it to.

If someone doesn't know how the class is being loaded once in memory clearly means there is huge lack of knowledge about how memory mappings going on there. And getting insights of symbols is a great way to get to know more about this.

regards,

Sur

crimson9.com

[Jordon Bedwell]

On Tue, Nov 5, 2013 at 12:42 AM, Sur <sur...@gmail.com> wrote:
> well it'd be interesting to know why you're frustrated out of Symbol
> concepts of Ruby ?

It's hard to explain, fight the good fight long enough you might see
it or you might not, I won't force my views on you.

> I didn't say symbols are better than strings, in fact I asked that it's
> important to know for a developer that why and where they can be used!

It was implied (to me.)

> It's an individual's choice to select a language/framework to achieve the
> solution to a problem, the main aim is to resolve a problem, now once
> decided to go with a language, one hast to know the limits and pros and cons
> of that language which certainly are there the way it's creators have made
> it to.

That's a fallacy as most of the time people do not have this sort of
flexibility, just because you hear all about it on Twitter and from
CTO's and startup's does not mean this is the case most of the time,
if it was then none of us would still be on Rails 2.3 with Ruby 1.8.
While this is not me, I know more developers who get told what they
are going to use then telling what they would like to use and then
using it.

> If someone doesn't know how the class is being loaded once in memory clearly
> means there is huge lack of knowledge about how memory mappings going on
> there. And getting insights of symbols is a great way to get to know more
> about this.

I think this leaves a lot to be explained because you explicitly
mention classes and I do believe this "loaded once in memory" only
applies to the singleton and since classes are meant to be initialized
there could remain many copies of that object with their own state at
the end of the day. I could be wrong though.

[Tamara Temple]

> —

Wow, is this response ever out of context. The example Sur supplied had to do with how code is stored in memory, IN RESPONSE TO the question of whether code is in memory or not (see "caching").