reading the supermarket data project

[Yoni Sidi]

hi to all 

i decided to write open code to read the supermarket data. to make the data easier to get to for anyone who is looking to research or just compare prices before going shopping. anyone is welcome to fork and improve the code.

i started with mega and all the files in the other stores are supposed to be uniform layout.

this is the govt site with links to all the stores: http://www.economy.gov.il/Trade/ConsumerProtection/Pages/PriceTransparencyRegulations.aspx

this is the github repo i set up. https://github.com/yonicd/supermarketprices

the mega prices are there for the example so you can see what such a file has in it (a lot), please dont overload the repo with another daily data file.

enjoy

yoni

[Moran Koren]

Yoni , You rock!

--
You received this message because you are subscribed to the Google Groups "Israel R User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to israel-r-user-g...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Moran Koren,

[Tal Galili]

Totach.

----------------Contact Details:-------------------------------------------------------
Contact me: Tal.G...@gmail.com | 
Read me: www.talgalili.com (Hebrew) | www.biostatistics.co.il (Hebrew) | www.r-statistics.com (English)
----------------------------------------------------------------------------------------------

On Thu, May 21, 2015 at 6:46 PM, Yoni Sidi <yon...@gmail.com> wrote:

--

[Ephraim Goldin]

כל הכבוד.

אפרים

[EREZ Ben-Moshe]

Great! Will make it so much easier to use the data

‫נשלח מה-iPhone של ארז בן-משה

‫ב-21 במאי 2015, בשעה 18:46, ‏‏Yoni Sidi ‏<yon...@gmail.com> כתב/ה:‬

--

[Yoni Sidi]

For those interested.

I added to the git repo code to read the shufersal data. fetching the files thus far isnt possible since when i download the gz files directly with R it comes in as corrupt, but when i manually download the gz file is ok (anyone on solution?). 

i downloaded an example of each type of xml file.

i also laid the ground work to put the data on maps using shp files of israel.

the main idea is that the total data for a chain is ridiculously large (csv .5gb) so i made a function mega.distance.price that uses the price of 95 gas (downloaded from the transportation ministry) and the distance from user origin point to any store within a given distance the user is willing to travel. that will be added to the price of a product basket a user can choose.

one small note: the files between chains are really not uniform, different xml layouts, mixed formats of variables within a chain. not to say anything about access to the files which is really bad. in short the chains made a mess of this good law that was passed. 

yoni

To unsubscribe from this group and stop receiving emails from it, send an email to israel-r-user-group+unsub...@googlegroups.com.

[Yoni Sidi]

problem solved. full open access to mega and shufersal. enjoy.