GSoC 2017 - Make Daru more ready for integration with modern Web framework

[Ananyo Maiti]

Hi all,

I am Ananya Maiti a final year B.Tech student from NIT Durgapur, India. For the past few weeks I have been working on the SciRuby/daru repo in one of its issue. I want to apply for GSoC 2017 on the idea Make Daru more ready for integration with modern Web framework. I am proficient with Ruby, Ruby on Rails, C/C++ and previously worked as an GSoC 2016 student under PublicLab in its Expanded Q & A Project which is a written in Ruby on Rails. I will be discussing the rough points for the proposal regarding the idea in this thread soon. Hoping to work with you all.

Thanks,

Ananya Maiti

Github: @ananyo2012

Twitter: @MaitiAnanyo

[Sameer Deshmukh]

Have a look at this thread:

https://groups.google.com/forum/#!topic/sciruby-dev/nQKGrsqZ0gY

[Ananyo Maiti]

Yes I read through the thread and noted the points mentioned there. I am trying to understand the various features of Daru. I encountered a problem regarding the plot method. I am facing this error

NameError: uninitialized constant Nyaplot::Frame::IRuby

I installed iruby and also tried to run it in a terminal instance in iruby notebook. Still facing the same problem. Can you guess what might be the problem?

ᐧ

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Prasun Anand]

Hi,

Did you install Nyaplot?

Prasun

[Ananyo Maiti]

Yes I did install Nyaplot. In fact it was installed during bundle install . Well I managed to plot using the iruby notebook. But I had to add the 'iruby' and 'ffi-rzmq' gems as a dependency otherwise I was getting an error similar to mentioned in http://stackoverflow.com/questions/35999925/kernel-dies-when-creating-iruby-notebook. But running in irb gives the same error.

On Thursday, March 2, 2017 at 11:24:55 PM UTC+5:30, Ananyo Maiti wrote:

[Lokesh Sharma]

Hi 

Plotting is not supported in irb if I remember correctly. 

--

[Ananyo Maiti]

Well shouldn't we get a proper output even if it didn't support plotting? Like the Plot Object? As of now I am getting an error.

Also as I mentioned the iruby notebook was showing up an error until I included the 'iruby' and 'ffi-rzmq' gems. Should I open an issue for this?

Thanks and Regards,

Ananyo

 

ᐧ

[Lokesh Sharma]

On Mar 5, 2017 4:00 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Well shouldn't we get a proper output even if it didn't support plotting? Like the Plot Object? As of now I am getting an error. 

Yes that seems plausible. 

Also as I mentioned the iruby notebook was showing up an error until I included the 'iruby' and 'ffi-rzmq' gems. Should I open an issue for this?

Its weird that you had to require iruby to make it work. If that's the case then you can file an issue. 

[Ananyo Maiti]

Hi all,

Going through the repository I found there is a Daru::IO module that has importers for csv, excel, sql, activerecord and exporters for csv, excel, sql for dataframes. The project idea mentions

each importer should be independent class (probably subclass of some BaseImporter)

Can we define independent classes in the same IO module to achieve this?

For importing and exporting json data we can use the json gem which is pretty frequently used for Ruby implementation of json objects. Ruby hashes are analogous json objects. For Redis imports we can perhaps use the redis-rb gem.

I am still deciding on the web presentation and a clear way to implement it and will be discussing it soon.

Thanks,

Ananya

ᐧ

[Ananyo Maiti]

Also I was thinking of extracting RSS feed data which is a frequently used way of representing data in web formats. We can add importers for it. Ruby has a built-in RSS Class so we don't need to add any extra dependency. Thoughts on this?

Thanks,

Ananya

ᐧ

[Victor Shepelev]

Hey, I am glad you are willing to take on this project.

Some answers below.

> Can we define independent classes in the same IO module to achieve this?

Looks reasonable to me.

> For importing and exporting json data we can use the json gem

No need to think of this. Modern Ruby has bundled support for JSON handling.

> ...For Redis imports we can perhaps use the redis-rb gem.

Yes, there are easy to find gem for each target. But the more important things are:

1. Overall architecture (what is importer/exporter; what are they responsibilities; how they are integrated to daru). Question for meditate here: if, after your work is finished, somebody wants to add new importer, how hard would it be? How hard is to extract importers/exporters into separate gem(s) and still have them easy to discover and use?

2. Overall plan. What are you about to achieve, if you'll take an entire project? What importers/exporters are necessary to have, what are good to have, what changes in Daru architecture will be necessary and when you'll made them?

Once you'll have a clear vision of (1) and (2), everything else are just a details. 

> Also I was thinking of extracting RSS feed data which is a frequently used way of representing data in web formats.

Yes. But are there much examples of data which could be statistically processed, and are provided in RSS? What is a priority of this idea? Couldn't all importers/exporters considered into some larger categories, not just picked one-by-one?

Please, think on those topics. I believe it will allow you to move further.

And thanks for your efforts.

V.

[Ananyo Maiti]

Hi Victor,

Thanks for your feedback. After thinking over your questions and doing some research I came up with these answers.

Overall architecture (what is importer/exporter; what are they responsibilities; how they are integrated to daru). 

Thinking of the architecture for the importers/exporters module I made a directory structure

.
|
|----lib
|  |
|  |----daru
|  |  |
|  |  |----io
|  |  |  |
|  |  |  |----importers
|  |  |  |  |
|  |  |  |  |----importers.rb
|  |  |  |  |----dependencies.rb
|  |  |  |
|  |  |  |----exporters
|  |  |  |  |
|  |  |  |  |----exporters.rb
|  |  |  |  |----dependencies.rb
|  |  |  |  |

 We can break up the importers and exporters in separate modules under the IO module and use separate classes for each exporter/importer. Any dependencies required for the importers can go to a dependencies class in a separate file like the SQL datasource used by the from_sql method.

if, after your work is finished, somebody wants to add new importer, how hard would it be?

To extend more importer/exporters they can simply add more classes to the importer/exporter module along with their dependencies. With good documentation we can explain how to extend more importer/exporter classes. I have included the points of a generalized perspective in a extract below in this mail.

How hard is to extract importers/exporters into separate gem(s) and still have them easy to discover and use?

About a separate daru-io gem, I think it will be fairly easy to make with well written tests. We have to shift some dependencies from daru to the daru-io gem like the 'dbi' and 'activerecord' gems. With proper require statements it should be easy to extract from the IO module.  Basically a complete transfer of the Daru::IO module to a DaruIO gem. But should we first add the importers/exporters first and then devise the gem or the reverse. Thoughts on this ?

What importers/exporters are necessary to have, what are good to have, what changes in Daru architecture will be necessary ?

 I think for now the most important exporter/importer to have are the database importers like sql and activerecord and the json importer/exporter and we should prioritize those with well written tests. Since if we are to integrate with rails and other web frameworks accessing data from database and api calls will be indispensable. For changes regarding daru architecture we should focus on modularizing the  code for exporters/importers and the views so that it will be easier to make gems from those modules.

are there much examples of data which could be statistically processed, and are provided in RSS?

Ah I should have taken more time to think about this before proposing. Now on deeper thought I think it holds a low priority as RSS feeds are mostly used in blogs and forums which don't contain such statistical data. They are basically used by web crawlers and improving the SEO. But sometimes in future we can have it. We can use the feed data from multiple websites and use it for statistical analysis. Like 'I want to know the distribution of sites that have content on spectrometry'. But that will need a search library as well. So this may be a far fetched concept and can be postponed now.

Couldn't all importers/exporters considered into some larger categories, not just picked one-by-one?

Well importers are used to extract data from different sources and file formats. So each one has separate dependencies and library of extraction. So there is no specific path of implementation. But if we are to devise a generalized implementation method here is a way. We can first import the data from the source to a database. Then we can use the database importer to load the data into Daru Dataframe or Vector. Then we can have a generalized method of implementation and we just have to write the code for including the dependencies and importing it to the database. But I don't think that will be wise thinking of the efficiency as importing to database will add an extra step and increase the time complexity.

Some thoughts on web Presentation:

Till now I was thinking of a way to make the views dynamic and adding features like sorting, pagination. Can we use ActionView to implement dynamic templating features? Deciding this will solve the first half of that problem.

Finally 

Overall plan. What are you about to achieve, if you'll take an entire project?  when you'll made them?

Once I get the reply to these thoughts I will start drafting my proposal ASAP

Thanks,

Anany 

Ananya

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

On Mar 5, 2017 4:00 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Well shouldn't we get a proper output even if it didn't support plotting? Like the Plot Object? As of now I am getting an error.

Also as I mentioned the iruby notebook was showing up an error until I included the 'iruby' and 'ffi-rzmq' gems. Should I open an issue for this?

Thanks and Regards,

Ananyo

 

ᐧ

On Sun, Mar 5, 2017 at 3:00 PM, Lokesh Sharma <lokeshh...@gmail.com> wrote:

Hi 

Plotting is not supported in irb if I remember correctly. 

On Mar 5, 2017 2:20 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Yes I did install Nyaplot. In fact it was installed during bundle install . Well I managed to plot using the iruby notebook. But I had to add the 'iruby' and 'ffi-rzmq' gems as a dependency otherwise I was getting an error similar to mentioned in http://stackoverflow.com/questions/35999925/kernel-dies-when-creating-iruby-notebook. But running in irb gives the same error.

On Thursday, March 2, 2017 at 11:24:55 PM UTC+5:30, Ananyo Maiti wrote:

Hi all,

I am Ananya Maiti a final year B.Tech student from NIT Durgapur, India. For the past few weeks I have been working on the SciRuby/daru repo in one of its issue. I want to apply for GSoC 2017 on the idea Make Daru more ready for integration with modern Web framework. I am proficient with Ruby, Ruby on Rails, C/C++ and previously worked as an GSoC 2016 student under PublicLab in its Expanded Q & A Project which is a written in Ruby on Rails. I will be discussing the rough points for the proposal regarding the idea in this thread soon. Hoping to work with you all.

Thanks,

Ananya Maiti

Github: @ananyo2012

Twitter: @MaitiAnanyo

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

On Mar 5, 2017 4:00 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Well shouldn't we get a proper output even if it didn't support plotting? Like the Plot Object? As of now I am getting an error.

Also as I mentioned the iruby notebook was showing up an error until I included the 'iruby' and 'ffi-rzmq' gems. Should I open an issue for this?

Thanks and Regards,

Ananyo

 

ᐧ

On Sun, Mar 5, 2017 at 3:00 PM, Lokesh Sharma <lokeshh...@gmail.com> wrote:

Hi 

Plotting is not supported in irb if I remember correctly. 

On Mar 5, 2017 2:20 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Yes I did install Nyaplot. In fact it was installed during bundle install . Well I managed to plot using the iruby notebook. But I had to add the 'iruby' and 'ffi-rzmq' gems as a dependency otherwise I was getting an error similar to mentioned in http://stackoverflow.com/questions/35999925/kernel-dies-when-creating-iruby-notebook. But running in irb gives the same error.

On Thursday, March 2, 2017 at 11:24:55 PM UTC+5:30, Ananyo Maiti wrote:

Hi all,

I am Ananya Maiti a final year B.Tech student from NIT Durgapur, India. For the past few weeks I have been working on the SciRuby/daru repo in one of its issue. I want to apply for GSoC 2017 on the idea Make Daru more ready for integration with modern Web framework. I am proficient with Ruby, Ruby on Rails, C/C++ and previously worked as an GSoC 2016 student under PublicLab in its Expanded Q & A Project which is a written in Ruby on Rails. I will be discussing the rough points for the proposal regarding the idea in this thread soon. Hoping to work with you all.

Thanks,

Ananya Maiti

Github: @ananyo2012

Twitter: @MaitiAnanyo

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

ᐧ

Ananya

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

On Mar 5, 2017 4:00 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Well shouldn't we get a proper output even if it didn't support plotting? Like the Plot Object? As of now I am getting an error.

Also as I mentioned the iruby notebook was showing up an error until I included the 'iruby' and 'ffi-rzmq' gems. Should I open an issue for this?

Thanks and Regards,

Ananyo

 

ᐧ

On Sun, Mar 5, 2017 at 3:00 PM, Lokesh Sharma <lokeshh...@gmail.com> wrote:

Hi 

Plotting is not supported in irb if I remember correctly. 

On Mar 5, 2017 2:20 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Yes I did install Nyaplot. In fact it was installed during bundle install . Well I managed to plot using the iruby notebook. But I had to add the 'iruby' and 'ffi-rzmq' gems as a dependency otherwise I was getting an error similar to mentioned in http://stackoverflow.com/questions/35999925/kernel-dies-when-creating-iruby-notebook. But running in irb gives the same error.

On Thursday, March 2, 2017 at 11:24:55 PM UTC+5:30, Ananyo Maiti wrote:

Hi all,

I am Ananya Maiti a final year B.Tech student from NIT Durgapur, India. For the past few weeks I have been working on the SciRuby/daru repo in one of its issue. I want to apply for GSoC 2017 on the idea Make Daru more ready for integration with modern Web framework. I am proficient with Ruby, Ruby on Rails, C/C++ and previously worked as an GSoC 2016 student under PublicLab in its Expanded Q & A Project which is a written in Ruby on Rails. I will be discussing the rough points for the proposal regarding the idea in this thread soon. Hoping to work with you all.

Thanks,

Ananya Maiti

Github: @ananyo2012

Twitter: @MaitiAnanyo

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

On Mar 5, 2017 4:00 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Well shouldn't we get a proper output even if it didn't support plotting? Like the Plot Object? As of now I am getting an error.

Also as I mentioned the iruby notebook was showing up an error until I included the 'iruby' and 'ffi-rzmq' gems. Should I open an issue for this?

Thanks and Regards,

Ananyo

 

ᐧ

On Sun, Mar 5, 2017 at 3:00 PM, Lokesh Sharma <lokeshh...@gmail.com> wrote:

Hi 

Plotting is not supported in irb if I remember correctly. 

On Mar 5, 2017 2:20 PM, "Ananyo Maiti" <anan...@gmail.com> wrote:

Yes I did install Nyaplot. In fact it was installed during bundle install . Well I managed to plot using the iruby notebook. But I had to add the 'iruby' and 'ffi-rzmq' gems as a dependency otherwise I was getting an error similar to mentioned in http://stackoverflow.com/questions/35999925/kernel-dies-when-creating-iruby-notebook. But running in irb gives the same error.

On Thursday, March 2, 2017 at 11:24:55 PM UTC+5:30, Ananyo Maiti wrote:

Hi all,

I am Ananya Maiti a final year B.Tech student from NIT Durgapur, India. For the past few weeks I have been working on the SciRuby/daru repo in one of its issue. I want to apply for GSoC 2017 on the idea Make Daru more ready for integration with modern Web framework. I am proficient with Ruby, Ruby on Rails, C/C++ and previously worked as an GSoC 2016 student under PublicLab in its Expanded Q & A Project which is a written in Ruby on Rails. I will be discussing the rough points for the proposal regarding the idea in this thread soon. Hoping to work with you all.

Thanks,

Ananya Maiti

Github: @ananyo2012

Twitter: @MaitiAnanyo

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

ᐧ

[Ananyo Maiti]

Hi Victor,

Please have a look at the previous mail I sent discussing the solutions. It's just that I want to get started with the proposal.

Thanks,

Ananya 

ᐧ

[Victor Shepelev]

Hey, sorry for the late answer, I needed some time to consider all of your points.

Overall, your thoughts look reasonable, yet I have a few suggestions:

1. I am not sure that "all importers + all their dependencies" vs "all exporters + all their dependencies" is a good initial division. Many types of data (CSV, SQL, Excel, ...) could be both imported and exported, and I'd say that we can rather imagine something like IO::ActiveRecord::Importer & IO::ActiveRecord::Exporter (than IO::Importers::AR and IO::Exporters::AR). This way it even could be easily independently required, and even put into independent gem like `daru-io-activerecord` (or `daru-io-some-exotic-format`)

2. I think first we should change (carefully, not by "total rewrite") the internal architecture, and then probably separate io into independent gem (or, alternatively, add `daru-io` gem with "interesting yet less used formats")

3. Consider planning more useful formats (we already have ActiveRecord/SQL importers), some examples (I am NOT insisting they are most useful, you should do some additional research; also I am not saying you should propose tens of formats, just some reasonable plan):

a) for web apps: common logging formats; most common NoSQL DBs (I'd say, for example, ElasticSearch currently could be seen everywhere); specific timeseries databases (like Influx); most common user input formats

b) for analytics: what about Google BigQuery, for example?

c) for admins/devops: again, common log formats, anything else? (IDK, like crontab files or some profiler output)

4. For views: I am not sure ActionView is a good solution, it is pretty Rails specific (and even in Rails is not always used), we need something MUCH more lightweight. I've already wrote somewhere: look like chartkick https://github.com/ankane/chartkick library solves the task "view helper for charts, any web framework, any templating language". Note it has NO Ruby depenencies at all.

Good luck! I'll try to answer the upcoming letters faster.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev+unsubscribe@googlegroups.com.

[Ananyo Maiti]

Hey no worries. Thanks for the reply.

I will do a bit more research on the formats you mentioned. Few more questions from the points you discussed.

Making separate gem for each importer `daru-io-something` sounds like a overkill. I definitely support the idea you wanted to pass. Instead of keeping "all importers and their dependencies" we can modularize each importer and their dependencies separately in different directories. We can still support the format `IO::ActiveRecord::Importer`. Is it not reasonable to keep all importers/exporters in a single gem and extend it to add more exporters/importers?

For presentable charts we can definitely use chartkick. But the reason I proposed ActionView was not for the analysis but for templating. We do require a ruby templating engine right? For generating dynamic data in html format. There are other options too https://www.ruby-toolbox.com/categories/template_engines. Thoughts on this?

Thanks,

Ananya

[Victor Shepelev]

> Making separate gem for each importer `daru-io-something` sounds like a overkill. 

Yes, and I am not proposing to do this. What I am saying, it should be easy for independent developer to do such gem -- therefore each IO source (importer + exporter) should go in separate & independent module. 

> For presentable charts we can definitely use chartkick.

I am not insisting on it. But I am recommending you to look how chartkick solves the same task (providing unobtrusive, simple, framework-independent view helper).

>  We do require a ruby templating engine right? For generating dynamic data in html format. 

I'd say we should stay with ERB, it is not most fancy and modern one, yet it is part of standard library and absolutely enough for simple templates. Take a look at how we are already using it for HTML generation (IRuby-oriented): https://github.com/SciRuby/daru/tree/master/lib/daru/iruby/templates

[Ananyo Maiti]

Hi Victor,

Sorry for the inactivity. Had been engaged in a Conference for the past week. Here is what I thought going through the remaining questions

For NoSQL databases the most widely used databases are that of Redis (Column model), MongoDB(Document Model) and Cassandra(Key-value model). There are ruby drivers for each of these databases so integrating these with Daru won't be much of a problem. 

You mentioned about influxdb so I checked it out. It has a ruby-client influxdb-ruby so we can integrate it too. But I guess we can keep it for the end after we have done the basic importer/exporter structure.

Elasticsearch logs and other server logs come as plaintext files so it can be easily parsed using the Daru plaintext importer. But I can't understand what is its need? Will there be any useful data to represent from log files?

About Google BigQuery, it is an enterprise software so I am not sure whether we should consider supporting it.

I will start making my proposal using these ideas and make the timeline there.

Thanks,

Ananya

...

[Victor Shepelev]

Hey. Some answers, for where they are necessary:

> Elasticsearch logs and other server logs come as plaintext files so it can be easily parsed using the Daru plaintext importer. But I can't understand what is its need? Will there be any useful data to represent from log files?

About ElasticSearch, I've meant not its logs, but its API (data stored in search index).

About admin logs -- imagine the task of analyzing some error logs to make stats. Probably (but not necessary important), you'd want something like

Daru.from_text(File.read(some_log), pattern: "some_fancy_parcing_pattern")

It is just an idea, to show the point of what should be analyzed for preparing your proposal (not "what libraries we can use", in first place, but "what industry people may want to do with Daru").

> About Google BigQuery, it is an enterprise software so I am not sure whether we should consider supporting it.

Well, 

1. industry people use it a lot.

2. it has an extensive set of datasets suitable for experiments: https://cloud.google.com/bigquery/public-data/

Like, somebody could extract some public GitHub data: https://cloud.google.com/bigquery/public-data/github and analyze/visualise it with Daru, don't you think?

All in all, I am just showing the possibilities. What to include in your proposal is up to you ;)