Some WSL suggestions (?)

[Paolo]

Based on my recent usage of trunk version of wsl on Raspberry PI, I miss these (minor?) features:

1. less db updates for less SD stress
2. wsl script suspension of inverter polling during night.
3. Daily total generated KWh power read from inverter just before inverter shutdown and not from the sum of real time monitoring. In case of some failures during day we miss real total daily generated KWh even if inverter knows it (at least Aurora).
4. date + time instead of date only for log entries shown in misc.php page

[Marco]

Thanx for your feedback!

1. What do you mean less DB updates?

2. As far is a know, WSL isn't polling the inverter when its power-down. What give you the idea that WSL is still polling the device when its powered down?

3. I do not understand what you're trying to say here

4. If you hover the date it should show the time ;-) (same as "weather" on the index.php)

[Marco]

Regarding OWM weather data:

After some debugging,testing , code reviewing, API reading on OWM, my WSL is gathering weather data again.

We were polling;

http://www.openweathermap.org/data/2.5/find?units=metrics&cnt=1&mode=json&lat=XXX&lon=XXX

And the response of this URL was not alway good or the same.

I now found that we need to poll to:

http://api.openweathermap.org/data/2.5/find?units=metrics&cnt=1&mode=json&lat=XXX&lon=XXX

Looks like OWM changed there api slightly, because in the past it always worked....

Problem solved.

[edit]

Problem still not solved!

My WSL was receiving good data for about 1 hour and then it stopped.

I went searching again on the OWM site and found this on there tracker;

http://bugs.openweathermap.org/issues/264

Looks like there are more people with this problem....

looking at there tracker, it looks not good (144 of the 170 issues are still open) and maybe we need to find another service (except Wunderground because the work with api calls and we can't do that).

Issue tracking

• Bug: 144 open / 170

[Buu Tam Le]

Hi Marco,

Regarding Paolo's suggestion on point 3, I think he meant to have a way to tell what is the exact daily generated power. If I have understood correctly WSL calculates the total daily value by adding all known interval outputs of that day. The total daily value is not accurate if for any reason the RPI is not working correctly or missing a few intervals. I have seen this happens to me too, when my bluetooth dongle crashes it can't read the values for certain periods and it misses a few values. 

Beside WSL I use SMASpot with an addon which keeps track the generated output. If it misses an interval it will get the history from the inverter of the missing period. SMASpot generates a daily report file with its value, this way it is possible to see the overall value and the value at that interval, I have copied an example of it:

Date;          Time;     Overall value; value at that time  

2014-03-13 00:00:00;1152.138;0.000

.......

2014-03-13 11:15:00;1159.465;3.012

2014-03-13 11:20:00;1159.719;3.048

2014-03-13 11:25:00;1159.974;3.060

2014-03-13 11:30:00;1160.232;3.096

2014-03-13 11:35:00;1160.492;3.120

2014-03-13 11:40:00;1160.753;3.132

2014-03-13 11:45:00;1161.011;3.096

2014-03-13 11:50:00;1161.272;3.132

...........

2014-03-13 20:55:00;1173.319;0.000

This way I can see what my total daily value is: 1173.319 - 1152.138 = 21.181 kWh 

[Marco]

Ok and in the case that there were missing intervals, which figure(s) are not correct?

[Buu Tam Le]

I don't know which figures are not correct, but below is the interval output copied from the "Day" tab. For example the missing interval is 13:00:00, 12:15:00, 11:45:00, 11:30:00, 11:15:00, 08:30:00, 08:00:00 

BTW... just discover something concerning privacy, in the "Other" tab the inverter section show the GPS coordinates. This information shows where one lives. 

14:35:00

2645,00

14:30:00

2671,00

14:25:00

2738,00

14:20:00

2793,00

14:15:00

2806,00

14:10:00

2876,00

14:05:00

2936,00

14:00:00

2943,00

13:55:00

2916,00

13:50:00

3011,00

13:45:00

3023,00

13:40:00

3051,00

13:35:00

3072,00

13:30:00

3067,00

13:25:00

3138,00

13:20:00

3196,00

13:15:00

3170,00

13:10:00

3214,00

13:05:00

3226,00

12:55:00

3240,00

12:50:00

3256,00

12:40:00

3281,00

12:35:00

3312,00

12:30:00

3301,00

12:25:00

3293,00

12:20:00

3314,00

12:10:00

3337,00

12:05:00

3340,00

12:00:00

3371,00

11:55:00

3359,00

11:50:00

3324,00

11:40:00

3316,00

11:35:00

3307,00

11:25:00

3260,00

11:20:00

3293,00

11:10:00

3234,00

11:05:00

3265,00

11:00:00

3244,00

10:55:00

3210,00

10:50:00

3301,00

10:45:00

2781,00

10:40:00

1622,00

10:35:00

2012,00

10:30:00

2186,00

10:25:00

1844,00

10:20:00

740,00

10:15:00

1043,00

10:10:00

1218,00

10:05:00

785,00

10:00:00

1234,00

09:55:00

955,00

09:50:00

1091,00

09:45:00

644,00

09:40:00

740,00

09:35:00

609,00

09:30:00

496,00

09:25:00

515,00

09:20:00

509,00

09:15:00

519,00

09:10:00

675,00

09:05:00

565,00

09:00:00

481,00

08:55:00

495,00

08:50:00

636,00

08:45:00

538,00

08:40:00

419,00

08:35:00

390,00

08:25:00

1011,00

08:20:00

979,00

08:10:00

923,00

07:55:00

233,00

07:50:00

233,00

07:45:00

233,00

07:40:00

233,00

07:35:00

233,00

07:25:00

167,00

07:20:00

105,00

07:10:00

39,00

[Marco]

Ok, it could happen that intervals are missing. But you said that when this happens, also the daily totals change or are not the same as the inverter reports.

i just did a test on my own database and removed about 25 minutes of "intervals"

Action	id	SDTE	time	INV	I1V	I1A	I1P	I1Ratio	I2V	I2A	I2P	I2Ratio	GV	GA	GP	FRQ	EFF	INVT	BOOT	KWHT	pvoutput	dayNum	pvoutputErrorMessage	deviceId	I3V	I3A	I3P	I3Ratio	GV2	GA2	GP2	GV3	GA3	GP3	IP	ACP	pvoutputSend	pvoutputSendTime
	514350	20140314-13:18:00	1394799480	4	327.505	4.644	1521.024	66.904	304.232	2.473	752.424	33.096	228.355	9.691	2196.028	49.995	96.595	41.288	36.188	3127.362	1	73	0	4	0	0	0	0	0	0	0	0	0	0	0	0	1	1394799536
	514359	20140314-13:45:00	1394801100	4	316.477	5.098	1613.346	75.405	310.021	1.697	526.229	24.595	228.497	9.05	2074.432	49.987	96.955	40.835	36.566	3128.306	0	73	0	4	0	0	0	0	0	0	0	0	0	0	0	0	0

And before and after the removal of intervals, my figures didn't change;

About the GPS coordinates;

In Admin::General you could change the coordinates and also round them so the marker will move and the location is not that precise.

So its to the user which coordinates he "thinks" his house stand ;-)

i rounded my coords to 3 digits and that gives;

https://www.google.com/maps/preview?q=52.063,+4.509

Could you tell me, based on these figures, where my house stands?

[Buu Tam Le]

I don't know what happens exactly, but sometimes my daily totals are not the same as reported by the inverter. Due to incorrect daily totals the week, month, year and overall totals are incorrect too. This error can be fixed by correcting the daily totals wth the Yields option.

Regarding GPS coordinates reported by http://pony.frijmann.nl/websolarlog/, this inverter is located in the Ponyweide in Zoetermeer. Just click on the coordinates in the "Other" tab in the inverter section, a google map window will be opened.

The coorindates that you have provided is Snoeksloot 48, Zoetermeer.

[Marco]

The strange thing is; WSL works with the total production figure from SMAspot;

Energy Production:

        ETotal: 100.310kWh

And based on that value we calculate everything.

If you see this behauvior agian; please let me know and provide my your database so i could investigate it!

Pony.frijmann.nl is indeed located on the ponyweide in Zoetermeer. But know the question is; Where on the ponyweide is the installation located? I know that there are 3 installation in the area.

My installation is indeed located on Snoeksloot in Zoetermeer, but not on number 48 and number 48 is not near my house ;-) And again there are multiple installation in my and the surrounding streets.

So far is i could say; the GPS location are not such a big privacy issue

[Buu Tam Le]

Ok, will do..

True about the GPS coordinates, it is not a big privacy issue but it shows where you live nearby :) With this information if one wants to harm you, they will/can find your exact house based on your inverter output and public websolar page (if it is publicly accessible of course and the bluetooth password is still on default 0000). 

Speaking of the bluetooth passsword, does someone know how to change it for SMA inverters? 

@Paolo: sorry for kidnapping your topic :)

[Paolo]

Il giorno giovedì 13 marzo 2014 10:12:21 UTC+1, Marco ha scritto:

Thanx for your feedback!

My pleasure. :)

 

1. What do you mean less DB updates?

Just watch at wsl.db date and time. Mine changes about every 15sec even if during night period with inverter shut down and no one browsing site.

 

2. As far is a know, WSL isn't polling the inverter when its power-down. What give you the idea that WSL is still polling the device when its powered down?

I see aurora.exe running and my communication leds keeps blinking when my inverter is shut down. If I stop wsl.sh aurora.exe stops running.

 

3. I do not understand what you're trying to say here

See @Buum Tam Le msgs. He got the point. :)

 

4. If you hover the date it should show the time ;-) (same as "weather" on the index.php)

Nice, but try to hover with a tablet... I still prefer time shown by default. :)

BTW, my last update date is wrongly formatted on weather hover.

[Paolo]

Il giorno venerdì 14 marzo 2014 16:09:51 UTC+1, Buu Tam Le ha scritto:

True about the GPS coordinates, it is not a big privacy issue but it shows where you live nearby :) With this information if one wants to harm you, they will/can find your exact house based on your inverter output and public websolar page (if it is publicly accessible of course and the bluetooth password is still on default 0000). 

I agree 100%. Showing GPS position should be an option.

 

@Paolo: sorry for kidnapping your topic :)

You are welcome! :) 

[Paolo]

Il giorno venerdì 14 marzo 2014 15:12:40 UTC+1, Marco ha scritto:

Ok, it could happen that intervals are missing. But you said that when this happens, also the daily totals change or are not the same as the inverter reports.

I confirm same problem here with aurora inverter.

How do you get interval producion and total daily production in wsl?

I would get both from cumulative daily production reading that from inverter, getting interval power from actual cumulative - previous cumulative. Try this. If I start wsl after some hour in the morning it starts from 0 kwh production and not from cumulative power already in the inverter daily totals. So at the and of the day my daily power is wrong even if my inverter told it right at the last cumulative reading...

I think that this is a big issue to fix. Even if I'm not interested to high resolution partials I would like to get daily real cumulative power even if my hardware or wireless connection had some little fault. :)

Thanks for your patience and support. :)

Paolo.

[Marco]

Please come with examples where the the missing intervals could be linked to total values that are off.

WSL is working with tot TotalKWH value of all thirdparty reported software so its near impossible that WSL's TotalKWH is off in the month,year and/or overall value. It could be that there is a calculation problem, but then i still need examples to fix this issue. This because there are tons of calculations and its very time consuming to check them all and hopefully find a bug.

You have a point on when WSL is starting hours after sunrise. This is because we use the TotalKWH value of the inverter.

For example;

14-3-2014 19:10 ends with 1891.123 kWh

when for some reason WSL start 3 hours after sunrise the inverter could report;

15-3-2014 09:30 (inverter is 2 hours active) first interval with 1892.623 kWh then we miss 2,5kWh in the daily repost.

I could try to fix this by;

Check on the first save-interval of the day if there is a big (>100Wh) diff between end day yesterday and "start" of the new day. If there is a diff then we could create (on the fly) a dummy records on sunrise time with the totalKWH value where yesterday is ended with.

On the previous example we had;

14-3-2014 19:10 ends with 1891.123 kWh

15-3-2014 09:30 starts with 1892.623 kWh (missing 2,5kWh) 

we get after the fix;

14-3-2014 19:10 ends with 1891.123 kWh

15-3-2014 07:30 dummy start with 1891.123

15-3-2014 09:30 starts with 1892.623 kWh (no today reports 2,5kWh production again)

15-3-2014 09:35 interval with 1892.629 kWh (production 2,506kWh)

could this work?

[Paolo]

Please come with examples where the the missing intervals could be linked to total values that are off.

WSL is working with tot TotalKWH value of all thirdparty reported software so its near impossible that WSL's TotalKWH is off in the month,year and/or overall value. It could be that there is a calculation problem, but then i still need examples to fix this issue. This because there are tons of calculations and its very time consuming to check them all and hopefully find a bug.

1st, thanks again for your support. Forgive me for looooong post, I hope to help you to make a better WSL and not to bother you. :)

Your workaround about previous day "big diff" seems good. At least it minimizes some problems, but same problem could arise during day.

I'll try to explain my thougths about that.

Let's start from my aurora values:

# aurora -Y 8 -w 25 -e -a 2 /dev/ttyUSB0

Daily Energy               =      10.088 kWh

Weekly Energy              =     159.379 kWh

Monthly Energy             =     283.624 kWh

Yearly Energy              =     734.822 kWh

Total Energy               =    3830.669 kWh

Partial Energy             =    3830.566 kWh

We have a Daily Energy that resets automatically every day. I don't know if other systems have different counters, so please forgive me if I miss something.

Sampling Daily energy I could get interval average production production and cumulative day production.

At 12:00:

Daily Energy               =      10.620 kWh

Weekly Energy              =     159.910 kWh

Monthly Energy             =     284.156 kWh

Yearly Energy              =     735.354 kWh

Total Energy               =    3831.201 kWh

Partial Energy             =    3831.098 kWh

At 12:05:

Daily Energy               =      10.877 kWh

Weekly Energy              =     160.167 kWh

Monthly Energy             =     284.413 kWh

Yearly Energy              =     735.611 kWh

Total Energy               =    3831.458 kWh

Partial Energy             =    3831.355 kWh

Let's suppose my WSL was powered off from 0:00 till 12:00. My cumulative production till 12:00 will be 10.620KW and I would expect that my interval production graph will be a line starting from 0 to 10.620 from sunrise to 12:00 as it would be an average production of 10.620Kwh/sampling period every sampling period. A luxury aproximation could adapt the line to theorical production following the ideal curve.

I expect that today cumulative production at 12:00 will be 10.620KWh as inverter says.

Next sampling time (let's suppose 5 min, 12:05), cumulative production will be 10.877 as my inverter says as daily production and the interval production will be 10877-10620 = 0.245KWh.

Using Daily Energy I will never miss production in the same day.

Using a similar calculation I could get same exact calculation for week, month and year using respective totals. BUT only if every inverters stores them. My Aurora inverter does, I don't know about others.

About a real case (fulll day) with missing data, I'm trying to get log and data from my 12-03-2014 day but If I go to "day" tab of wsl and move to a different date I always get "today" in production History list under the graph.

I could get data from db. Which table should I query? BTW, I think that wsl.sh locks the db when it runs...

Some other issues. I'm having some refresh the mage and "not showing refreshing" in details page. Date formatting in this page is different from the others YYYY-MM-DD vs DD-MM-YYY and is difficult to know when page is refreshing changing date. Production is not visible too and Y-scaling does not dinamically changing.

Very long post, I hope it's clear. Thanks again for your patience. :)

[Marco]

Total Energy = 3831.201 kWh

Total Energy = 3831.458 kWh

WSL is working with these values and 3831.458 - 3831.201 gives 257Wh production, which is the same value you calculated.

The total Energy value always(!) is the value to calculate with and to prevent overal mis-calculations. It could happen that daily values are off for a few kWh, but i hope to fix soon.

Also every inverter manufacturer saved/stores the data different. Incidentally, it is so crazy that it may vary by model and thats almost unworkable for us. For example;

Old PowerOne models store only daily total data as total kWh and some new models have a bug that the report 1kWh as 0.1kWh and we need to fix this. PowerOne's won't report interval values and SMA's can report interval values. Then we have Diehl with a very nice and very different (to SMA and PowerOne) API that can be used. Delta Solivia have, as far as i known, non of either... 

So every manufacturer, every model, every revision, etc needs to be handled separately and need time/code to get the data.

The daily values i will take a look on and see if i can fix this (could you make a ticket for me?)! 

This evening i made a small piece of code that does;

//*

* here we do 2 checks;

* 1. is (current KWHT - last['KWHT']) > 0 

* 2. is (sunrise['time'] - last['time']) > 18000    (5 hours)

* 

* if these checks are good; we are probably started later with logging and so make a dummy record to fix the graph and daily production

*/

We then create a "dummy history record" with the last known KWHT (kWh total) value and the time of today sunrise minus 30 minutes.

[edit]

you mentioned that wsl.sh locks the database when its running. This is totally true and we could not fix this and this behavior is caused by the way SQLite works with databases. 

I'm working on a process to let the user move WSL from SQLite to MySQL. But this needs good testing, migrations scripts, changes in the Admin etc. Also i need to test how MySQL is performing on the Rpi and other low powered devices. 

MySQL vs SQLite;

MySQL needs more recourses then SQLite

MySQL works better with multi threats

MySQL could be a good option, when its configured to run on low powered devices. Unfortunately the basic config of MySQL is not that optimized for low-powered devices.... 

[Sparky Bartlett Jewell]

I see activity all night as well.  There aren't database entries added, but I do see activity and polling of the inverter (which is off).  When I add current monitoring at the service inlet, I'll have activity all day and night, so this isn't that much of a concern.  Though Flash memory will eventually wear out, I've got a sufficiently large enough card in my raspi that I'm not concerned about the life of the card.  I just need to back up to my NAS occasionally.

[Buu Tam Le]

Concerning writing data on the flash memory card, what I did was moving the entire WSL environment to my NAS share and running it from there. This way it won't produce a lot of write actions on the flash memory card and you don't have to make backup occasionally.

[Paolo]

The daily values i will take a look on and see if i can fix this (could you make a ticket for me?)!

Ok, I wll create it.

 

This evening i made a small piece of code that does;

//*

* here we do 2 checks;

* 1. is (current KWHT - last['KWHT']) > 0 

* 2. is (sunrise['time'] - last['time']) > 18000    (5 hours)

* 

* if these checks are good; we are probably started later with logging and so make a dummy record to fix the graph and daily production

*/

We then create a "dummy history record" with the last known KWHT (kWh total) value and the time of today sunrise minus 30 minutes.

I think that this is a good solution for late WSL wakeup. I assume that the remaining differencing logic also handle a stop during the day only causing missing sampling points (less detailed graph).

What about a "more than a day stop"? I think that some more checks are needed.

[Paolo]

Concerning writing data on the flash memory card, what I did was moving the entire WSL environment to my NAS share and running it from there. This way it won't produce a lot of write actions on the flash memory card and you don't have to make backup occasionally.

I'm using a Raspberry PI to save energy. It's a waste of energy to keep on both Raspberry and the NAS. I would have installed it all on the NAS. :)

[Paolo]

I see activity all night as well.  There aren't database entries added, but I do see activity and polling of the inverter (which is off).  When I add current monitoring at the service inlet, I'll have activity all day and night, so this isn't that much of a concern.  Though Flash memory will eventually wear out, I've got a sufficiently large enough card in my raspi that I'm not concerned about the life of the card.  I just need to back up to my NAS occasionally.

I think that all unwanted activity on the SD card should be avoided, if possibile, expecially for a service that works 24h. Why waste cpu, power and SD card durability for nothing? :)

I think I will open a ticket for it when I will be able to look at sources. Would like not to bother @Marco too much for his excellent work. :)

[Buu Tam Le]

On Tuesday, 18 March 2014 10:35:23 UTC+1, Paolo wrote:

Concerning writing data on the flash memory card, what I did was moving the entire WSL environment to my NAS share and running it from there. This way it won't produce a lot of write actions on the flash memory card and you don't have to make backup occasionally.

I'm using a Raspberry PI to save energy. It's a waste of energy to keep on both Raspberry and the NAS. I would have installed it all on the NAS. :)

True about. However RPI does not use that much energy which you basically can ignore it, the other thing is I can reboot my RPI when ever I want (maybe for testing purposes etc.) With a NAS I prefer not to do that because my NAS is connected to my server 24/7. For one who has his/her NAS 24/7 running I think it is better to write the WSL data on it to save the durability of the SD card.

[Marco]

Its very simple to say; WSL is wasting cpu, power and "destroying" my SD card...

Give me data, point me to power/CPU/SD-card wasting/destroying processes of WSL and i will take a look at it.

Also compare it to other hard/soft -ware solution for monitoring your system and compare it on power usage, features, support, 

Here in The Netherlands we have a small company called Plugwise. The sell smartplugs that can monitor your SolarSystem.

These plugs usage about 0.7-1.5Wh per grid fase. If you have a 3 fase inverter, you need 3 plugs of >€60 a piece a the plugs itself will use 2.1-4.5Wh.

With the plugs itself you can do nothing and you need a device that can deliver graphs/data. For this the have a device called a Stretch and that uses about 3-5Wh. Als i monitor my DutchSmartMeter and Plugwise also sells a solution for that and it calls a P1Smile. This is a seperate device that also usage 3-5Wh.

In the most positive scenario this solution uses about 8Wh and gives you VERY BASIC graphs with ZERO control over the system.

I used the plugs to track my power usage and smart switch some stuff. But the Stretch crashes very often and is also replace already in about 7 months. 

My Rpi uses about 5Wh and then i have;

1 Rpi, 1 powered USB-hub, 1 WiFi adapter, 1 expansion board for monitoring my Dutch SmartMeter

What does WSL in the night?!

There are running several processes to keep track on PVoutput data, clean the database, run checks etc.

You could find all defined processes in /path/to/websolarlog/scripts/server.php

It stand everyone free to use WSL, It stand everyone free to move to another solution, It stand everyone free to use to help with the development, It stand everyone free to speak, It stand everyone free to clone/modify/change/etc WSL......

Also try the command "top" on your console prompt and see what your host is doing over the day.

[gmk]

All,

valid points all around. simply said: yes the rpi is one of the most versatile low-cost, low-energy consumption devices around. Yes it is a known issue with the rPi that in the design fase tradoffs were made. Yes SD cards are known to crash, yes having USB and ethernet and SD cards on the same weak USB controller is a known issue with rPi. and there is nothing much marco and martin can do about that.

At the same time some valid concerns are voiced regarding IO and SD cards. I guess it would be a good thing if we could - in the future, because I believe Marco and Martin are keen on getting a stable v1.1. released - keep IO as a topic on the list.

However for now I would prefer to settle this by pointing out that there are several workarounds readily available of which the most common I list here:

1. using a USB stick/disk to store the WSL (or for that matter the OS) on (see google: raspberry pi install on USB disk)
2. using a network device such as a nas to store the WSL DB on (see google: mount network drive raspberry pi)

Marco, keep up the good work (I keep seeing day-to-day improvements - and they are killing my SD card ;-), and everyone else - thanks for your input

[Buu Tam Le]

Well said GMK! 

Marco and Martin are doing a great job with WSL, keep up the good work, thank you guys!

Wherever I can, I will give my input to make this software better and better :)

[Martin Diphoorn]

Im reading this post and i want to thank you all for helping us out with good testing.

About the the nightly checks. I think that Marco is wrong, we check always. And that is my fault.
Because i thought this was implemented. The idea was that we should slow down the checking.
So instead off every 2/3 seconds we check ones every hour to check if we can get data.

When we approaches the calculated sun up we switch to a faster checking.

This is however never implemented. And we should do this.

Someone wanted to raise a ticket for this. Please do so.

[Paolo]

Im reading this post and i want to thank you all for helping us out with good testing.

Thanks for your effort and feedback listening! I rarely find such responsiveness answering and fixing issues. :)

 

About the the nightly checks. I think that Marco is wrong, we check always. And that is my fault.
Because i thought this was implemented. The idea was that we should slow down the checking.
So instead off every 2/3 seconds we check ones every hour to check if we can get data.

When we approaches the calculated sun up we switch to a faster checking.

This is however never implemented. And we should do this.

Someone wanted to raise a ticket for this. Please do so.

Here it is: http://tracker.websolarlog.com/view.php?id=231

Thanks again,

Paolo.

 

[Paolo]

Its very simple to say; WSL is wasting cpu, power and "destroying" my SD card...

Give me data, point me to power/CPU/SD-card wasting/destroying processes of WSL and i will take a look at it.

I agree! It just takes some time to collect every items. I will try to make some more investigation and will give you some feedback if I will find something interesting that's worth fixing.

 

It stand everyone free to use WSL, It stand everyone free to move to another solution, It stand everyone free to use to help with the development, It stand everyone free to speak, It stand everyone free to clone/modify/change/etc WSL......

I'm trying to help, just forgive me if I bother too much. :)

[Paolo]

Its very simple to say; WSL is wasting cpu, power and "destroying" my SD card...

Give me data, point me to power/CPU/SD-card wasting/destroying processes of WSL and i will take a look at it.

I agree! It just takes some time to collect every items. I will try to make some more investigation and will give you some feedback if I will find something interesting that's worth fixing.

Based on my investigation, wsl cause some overloading of db writes, probably due to queserver database logic timing and REDbeansPHP handling of "beans". "iotop" shows that top io process is "php server.php" and it writes very often to wsl.db.

I read some of the wsl sources and I know that QueueItem table in wsl.sdb is continuosly written (and remains locked :) ) during queue handling but when I stop wsl I see that QueueItem table is is empty. Im supposing that QueueItem table should be essentially a "temp table", used to manage/track queue items/jobs/task but only when wsl is running. Moreover, I'm supposing that only 1 process thread (QueueItem.php + QueueServer.php) is managing the queue.

If my hypothesis are true, is it possible to take db writes to storage down and speed-up wsl "moving" QueueItem table/db in memory? I know that sqlite3 and redbeans should have this capability. For example, how about using more than a db: 1 for persistent data on storage and one in memory containing Queitem table and any other "temp" tables?

Ref.:

https://www.sqlite.org/inmemorydb.html

http://redbeanphp.com/manual3_0/multiple_databases

Let me know if I'm wrong.

Thanks,

Paolo.

[Martin Diphoorn]

Paolo,

i dont know why you think it is the queueitem table. Actually the queueItem table is almos always empty. We only use it for sending stuff from the frontend to the queueserver. Most used is when you do an update or test an device. So queueitem table cannot be that busy in writing.

Greetings,

Martin

[Paolo]

Il giorno martedì 25 marzo 2014 14:30:55 UTC+1, Martin Diphoorn ha scritto:

i dont know why you think it is the queueitem table. Actually the queueItem table is almos always empty. We only use it for sending stuff from the frontend to the queueserver. Most used is when you do an update or test an device. So queueitem table cannot be that busy in writing. 

You are right and I was wrong. :)

Further investigation led to an "overwrite load" regarding cache table. Marco is working on it.