Parsing old Microsoft Word format (.doc)

[Konstantin Pečaļka]

Hi all,

Looking for a way to extract text from .doc (not .docx which can be handled with xml libraries).

Website from this mission http://missions.opencorporates.com/missions/641 contains quite simple .doc files with mostly plain text wrapped in .doc. I can't find good python libraries to parse .doc (not tried to search the same for ruby), but tried catdoc utility on files from that site -- it works and extracts all text.

Is it possible to run external programs from scraper scripts on Turbot server? If yes, can catdoc utility be installed?

[Chris Taggart]

That looks like a good suggestion to add catdoc -- and looks pretty lightweight. An alternative might be poi  . Could you 

Because the bots run in Docker, we need to build the images in advance, with all the required libraries -- there's a balalnce between keeping them lightweight and functionality.

Could you possibly do a before and after sample with catdoc? i.e. input file and output file?

Thanks

Chris

[Chris Taggart]

Possibly http://tika.apache.org/ is a better alternative. 

Not sure current debian build of catdoc is best to use (see http://www.wagner.pp.ru/~vitus/software/catdoc/)

[Konstantin Mochalov]

Yes, catdoc is quite quirky. Yesterday compiled it under Mac OS and it crashes.

Tried catdoc, antiword and tika, here is outputs for one of .docs on minfin.gov.by:

https://gist.github.com/kolen/1e16aa2c2075a56968ca

Tika looks cool. It is somewhat heavyweight, however most flexible, can output xml and supports lots of formats (http://tika.apache.org/1.6/formats.html). Its binary has size of 30 Mb, however it does not require quirky compilation and requires only Java. I think this software would be useful for scrapers.

--
You received this message because you are subscribed to a topic in the Google Groups "OpenCorporates Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opencorporates-community/mqNkmP-hEm0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opencorporates-com...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Seb Bacon]

Thanks for doing this comparison Konstantin. It's one reason we've not
picked a particular solution yet for Turbot.

Tika feels like a good choice to me, for the reasons you outline. It's
used by various other projects, e.g. ElasticSearch, and is mature
software. We'll have a look at implementing that today.
-------------------------------------------------------
OpenCorporates :: The Open Database of the Corporate World
http://opencorporates.com
Blog: http://blog.opencorporates.com
Twitter: http://twitter.com/OpenCorporates

OpenCorporates is published by Chrinon Ltd, a company dedicated to
improving and publishing public data under an open licence that allows
and encourages reuse, including commercially. Registered in England,
number 07444723.


On 17 December 2014 at 07:47, Konstantin Mochalov

> You received this message because you are subscribed to the Google Groups
> "OpenCorporates Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Willem de Groot]

Would piping these .docs through an online service be acceptable?

https://translate.google.com/translate?hl=en&sl=ru&tl=en&u=http%3A%2F%2Fwww.minfin.gov.by%2Fupload%2Finsurance%2Flicens%2Fbagach.doc

[Seb Bacon]

On 20 December 2014 at 16:44, Willem de Groot <gwi...@gmail.com> wrote:
> Would piping these .docs through an online service be acceptable?
>
> https://translate.google.com/translate?hl=en&sl=ru&tl=en&u=http%3A%2F%2Fwww.minfin.gov.by%2Fupload%2Finsurance%2Flicens%2Fbagach.doc

As long as it was within the terms of service, and could be expected
to be reasonably reliable. I would expect the Google Drive API would
be the way to go with this -- I very much doubt it is within their ToS
to do otherwise programmatically.

This would have the convenience of a cloud service; on the other hand,
developers would have to manage their own API keys when in development
mode. Not a huge obstacle, so I do think this is worth considering.

Any other thoughts on this?

Thanks

Seb