WSL and Domoticz

[Peter]

I experienced that polling the smart meter from WSL and from Domoticz will result in all sorts of problems...

Now I'm looking for a (better) way to keep getting / using the data in both.

I'd like to keep having smart meter data in WSL so I can compare my production and consumption in detail.

I'd like to keep having smart meter data in Domoticz since it can be checked from smartphones and because it enables al sorts of triggering possibilities etc.

Now I see to ways of accomplishing this::

Either Domoticz should get the smart meter data from WSL, or WSL should (have a way to) get the smart meter data from Domoticz.

Or does anyone know of other ways to accomplish this?

Since both systems are absolutely great for their purposes and Domoticz is quite popular, I would like to suggest to develop this into an option in WSL. 

Of course I'll also propose this at Domoticz...

BTW, I don't want to overburden anyone with feature requests, questions and comments.

Please don't feel obliged to respond (right away). 

I'm just very enthousiastic about WSL can't stop diving deeper into all it's features!

Also, I must admit that I have not tried out PV monitoring in Domoticz yet, but I understood that it does not have a way to monitor my Omnik inverter (yet).

The beta has an option to get data from PVoutput, but this of course nowhere as good as monitoring the device from within my LAN.

[Peter]

Got Domoticz to show data from PVoutput, but if I'm not mistaking, it seems to be limited to PV data, so it doesn't display consumption data.

Also  Domoticz has no means to submit data to PVoutput.

I gues I need to script a query to the Domoticz database for the latest smartmeter data and output that to console, in order to read the data into WSL?

[Marco]

WSL does everything you want, but could not switch or interact with your Home automation.

I think the best way is to "force" Domoticz to gather the data from WSL.

Here is a link that you could use;

http://pathToWebSolarLog/websolarlog/api.php/Summary/4-8-2014

This link return an JSON with for example;

- Inverter production,

- smartmeter figures

- net/gros usage of the household kWh

- gas usage

- CO2 avoided/usage

- etc etc

This data can be used in Domoticz to do smart and automagische dingen ;)

Past March i bought an Vera3 and i'm also working on some cool stuff to make our home more automagisch :D

Recently i created my first 2 private plugins to switch an Ethernet Relayboard and gather all data from WSL to my Vera3 :)

Also i lost complete focus on PVo because i think it to "complicated" and not giving that much useful information on comparing neighboring systems.

For example;

My system contains 2 string of different types and counts of panels, 1 strings has more shaded then the other, 1 strings is performing better in spring and autumn, i could not create in PVO an system thats 90% OK with the system on my roof. I known that allot of people have the same issue and how could i compare my PVo system, that's not at least 90% matching with the real world, with other system that also are not matching the real world?!?!!?   :|

But that's a different discussion....

[Peter]

Hi Marco (and other interested sun worshippers),

I tried out some JSON (for the first time, I'm a noob on that) and managed to capture smartmeter energy data from Domoticz and output it to the console.

I will have to find a way to combine 2 JSON queries to output the smartmeter gas data also.

Further more It will need a procedure to use some variables for the server ip, port, username and password and device rid in set in an external file...

If necesary I could try and find out how to format and filter it down to the essentials. 

Couldn't something like that be used to create a new smartmeter communication in WSL?

That would be nicer than the other way round, since Domoticz allready has an easy to configure smart meter device, that has nice graphical representations, can comminicate with smartphone apps and most important, can be used to trigger all sorts of other "automagische dingetjes"   ;-)  within Domoticz.

script (so far)

#!/usr/bin/php

<?php

$json_string = file_get_contents("http://<username:password@>domoticz-ip<:port>/json.htm?type=devices&rid=<rid>");

$parsed_json = json_decode($json_string, true);

$STDOUT = fopen("php://stdout", "w");

        fwrite($STDOUT, print_r($parsed_json, TRUE));

        fclose($STDOUT);

$parsed_json = $parsed_json['result'][0];

$lastupdate = $parsed_json['LastUpdate'];

echo "Last Update ".$lastupdate."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['result'][0];

$counter = $parsed_json['Counter'];

echo "Imported ".$counter." kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['result'][0];

$countertoday = $parsed_json['CounterToday'];

echo "Imported today ".$countertoday."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['result'][0];

$counterdeliv = $parsed_json['CounterDeliv'];

echo "Exported ".$counterdeliv." kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['result'][0];

$counterdelivtoday = $parsed_json['CounterDelivToday'];

echo "Exported Today ".$counterdelivtoday."\n";

?>

 Output (so far):

Array

(

    [5MinuteHistoryDays] => 1

    [ActTime] => 1407255276

    [AllowWidgetOrdering] => 1

    [DashboardType] => 0

    [Latitude] => 52.450194

    [Longitude] => 4.808997

    [MobileType] => 0

    [TempScale] => 1

    [TempSign] => C

    [WindScale] => 1

    [WindSign] => m/s

    [result] => Array

        (

            [0] => Array

                (

                    [BatteryLevel] => 255

                    [Counter] => 2058.041

                    [CounterDeliv] => 1392.278

                    [CounterDelivToday] => 0.746 kWh

                    [CounterToday] => 3.981 kWh

                    [CustomImage] => 0

                    [Data] => 1234283;823758;437735;954543;0;330

                    [Favorite] => 0

                    [HardwareID] => 2

                    [HardwareName] => ISKRA ME382

                    [HaveTimeout] =>

                    [ID] => 1

                    [LastUpdate] => 2014-08-05 08:55:55

                    [Name] => Smart Meter Energy

                    [Notifications] => false

                    [Protected] =>

                    [SignalLevel] => 12

                    [SubType] => Energy

                    [SwitchTypeVal] => 0

                    [Timers] => false

                    [Type] => P1 Smart Meter

                    [TypeImg] => counter

                    [Unit] => 1

                    [Usage] => 0 Watt

                    [UsageDeliv] => 330 Watt

                    [Used] => 1

                    [XOffset] => 0

                    [YOffset] => 0

                    [idx] => 6

                )

        )

    [status] => OK

    [title] => Devices

)

Last Update 2014-08-05 08:55:55

Imported 2058.041 kWh

Imported today 3.981 kWh

Exported 1392.278 kWh

Exported Today 0.746 kWh

[Peter]

Can you help me with constructing a php / JSON query that will (allways) get all of the current smart meter data (so not "just" a summary of totals at the end of the day)?

Thanks!

[Peter]

It seems that simply leaving out date in the link will show the data for "now", right?

#!/usr/bin/php

<?php

//$date=date +"%d-%m-%y";

$today = date ('m-d-Y');

echo $today;

//$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Summar$

$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Summary/$

$parsed_json = json_decode($json_string, true);

$STDOUT = fopen("php://stdout", "w");

        fwrite($STDOUT, print_r($parsed_json, TRUE));

        fclose($STDOUT);

?>

[Peter]

The summary page at http://192.168.1.3/websolarlog/summary.php seems to miss the "totals" and the power levels of the smart meter (or am I overlooking something?), which I'd like to export to virtual meters in Domoticz:

- Total (lifetime) amount of imported electricity at low rate in kWh (1.8.1), 

- Total (lifetime) amount of imported electricity at high rate in kWh (1.8.2)

- Total (lifetime) amount of exported electricity at low rate, in KWh (2.8.1)

- Total (lifetime) amount of exported electricity at high rate in KWh (2.8.2)

- Total (lifetime) amount of gas used in m3 (beneath 24.3.0)

- Current power usage in kW (1.7.0)

- Currently power returns in kW

Can they be queried elsewhere or in another way?

(I'd like to read in the values of the smartmeter, not the outcome of any calculations over time or after combining the data with production data from the inverter, please)

Also the time value displayed seems to be taken from Open Weather Map. Can this be captured "internally" somehow, as the time of the last polling of the smart meter?

#!/usr/bin/php

<?php

//$date=date +"%d-%m-%y";

$today = date ('m-d-Y');

$time = time ('H-i-s');

echo "date: " . $today . PHP_EOL;

echo "time: " . $time . PHP_EOL;

//$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Summary/'. urlencode($today) . '");

$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Summary/07-08-2014");

//$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Summary/");

$parsed_json = json_decode($json_string, true);

$STDOUT = fopen("php://stdout", "w");

        fwrite($STDOUT, print_r($parsed_json, TRUE));

        fclose($STDOUT);

$parsed_json = $parsed_json['data']['weather'][0];

$time = $parsed_json['time'];

echo "Time ".$time."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['data']['metering'][0];

$lowUsage = $parsed_json['lowUsage'];

echo "Low Import Today ".$lowUsage."kWh"."\n";

echo "Lifetime Low Import ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['data']['metering'][0];

$highUsage = $parsed_json['highUsage'];

echo "High Import Today ".$highUsage."kWh"."\n";

echo "Lifetime High Import ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['data']['metering'][0];

$usageKWH = $parsed_json['usageKWH'];

echo "Total Import Today ".$usageKWH."kWh"."\n";

echo "Lifetime Total Import ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['data']['metering'][0];

$lowReturn = $parsed_json['lowReturn'];

echo "Low Return Today ".$lowReturn."kWh"."\n";

echo "Lifetime Low Export ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['data']['metering'][0];

$highReturn = $parsed_json['highReturn'];

echo "High Return Today ".$highReturn."kWh"."\n";

echo "Lifetime High Export ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['data']['metering'][0];

$returnKWH = $parsed_json['returnKWH'];

echo "Total Return Today ".$returnKWH."kWh"."\n";

echo "Lifetime Export Total ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json['data']['metering'][0];

$gasUsage = $parsed_json['gasUsage'];

echo "Gas Usage Today ".$gasUsage."m3"."\n";

echo "Gas Lifetime Usage ??? m3"."\n";

?>

[Martin Diphoorn]

Hi Peter,

the best way to support your needs is too extend the api.
If we extend the api then everyone can poll this info and use it.

I will think about it as security will become more important.

Martin

[Peter]

That would be just great!

Maybe the reverse is possible also: create some device and communication in WSL that get's its data externally by adding JSON queries.

Allthough that would create a risk to mess up the data(base)..

[karsten.k...@gmail.com]

Hi,

is there any documentation on the API?

I would love to get the current live data from the dashboard read into a variable that i can use externally in house automatisation

thanks

Karsten

[Martin Diphoorn]

At the moment there is no documentation for the api. The reason is that it is only used internally and we don't want angry people when we change something fundamentally.
However the live data is simple: api.php/Live

This will give you the data you need.

If you mis anything just tell me.


I'm just started looking at Domoticz and use it for my lights (kaku).
I would like to check if my solar panels are producing less then 100 watt and switch on the light.
Is that something you guys also want or have?

[karsten.k...@gmail.com]

yes - but i am currently building this based on openhab (www.openhab.org) - as it supports more devices/ connections than Domoticz.

I even have attached my ventilation system, stereo players, ... attached to it.

Cheers

Karsten

[karsten.k...@gmail.com]

Hi Martin, 

is there a command line call i can use for the api.php/Live?

just to be sure i would not need the way around the html request?

thanks

Karsten

[Peter]

Should have thought about the Live page. Perfect! 

Looks like api.php/Live  and JSON will get me just what I need for a virtual smart meter in Domoticz. Thanks!

Is there a page where I can query live data from the inverter in a similar way?

[Marco]

Maybe a domotica call could be a solution? I could also use this for my domotica system.

Basic figures suggest as current power, Grid Voltage, Grid failure, sun up/down, daily production, and???

[Peter]

Oh stupid me:

Production / inverter live stats can be queried from api.php/Live also,..

Daily usage / production figures can be queried from api.php/Sumary

Weather conditions and sun up/down can be found there also.

What else would you need from WSL for your domotica system? 

[Marco]

When I'm honest, I don't need WSL for my domotica system.

The only thing WSL could be help full, is the total produced figure so I could switch heavy duty device such as an electric boiler, laundry dryer, etc. But when I read my smartmeter with my Vera3 then I could also switch these devices based on the fact if the smartmeter is feeding to grid or taking from the grid.

But I think it's more efficient to create one api call with all the data (and one html call) and gather all the data and not make 3,4 or 5 html/json calls to my Rpi.

So which data should this call return?

Then Martin and I could take a look on this.

[karsten.k...@gmail.com]

I used http://txt2re.com/ to get the reg expressions needed for the integer value of Watts currently produced by my Diehl.

I anyway will use WSL as i think its a great project for my Diehl but for rule calculation in my home automation i will grep some data.

My perl script looks like attached.

In regards to your question - if the API return would be XML style - than it would be easier to grep tha data e.g. with XSLT templates. your structure is anyway already there - just a reformating of the output might be needed.

cheers

Karsten

[karsten.k...@gmail.com]

i did a XML based on the details i receive from my API return as sample. Will raise an enhancement request.

Cheers

Karsten

[Peter]

I see that on /websolarlog/api.php/Live the value for "liveUsage" will remain 0 and "liveEnergy" is negative, when my inverter produces more then I am consuming... 

I want to see and keep track of my "actual" usage and have this in Domoticz (and in WSL and PVoutput also by the way). The problem is that aparrently negative values are ignored in Domoticz and in PVoutput, so they the outcome of their calculations will be incorrect  

Wouldn't it be better make "live usage"show the "real" live usage, by doing a calculation based on production from the inverter (GP) minus returns liveReturn)?

Alternativly an extra value could be used (and graphed?)

[Peter]

Actually I'm most interested in displaying actual POWER levels.; how much power is my house consuming right now.

I guess that allthough GP is in Watts, liveUsage, liveEnergy and liveReturn are all in kWh..

How can I handle this? So far I have:

<?php

$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Live/");

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json [0]['data'];

$liveUsage = $parsed_json['liveUsage'];

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json [0]['data'];

$liveReturn = $parsed_json['liveReturn'];

$parsed_json = json_decode($json_string, true);

$parsed_json = $parsed_json [2]['data'];

$GP = $parsed_json['GP'];

//calculate actual energy consumption

if ($GP>$liveReturn) {

        $consumption=$GP-$liveReturn;

} else {

        $consumption=$liveUsage;

}

echo "current electricity consumption in the house is ".$consumption."W";

$json_string_37 = file_get_contents("http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=37&svalue=$consumption");

?>

Maybe I need an explanation on all the values?

[Marco]

Its with the available data in WSL very hard to tell whats the real(!) actual power usage of a home.

For example;

Your laundry dryer is running and its heating up and using 2kW and just after the last smartmeter telegram, the dryer stops heating up.

Then WSL thinks for 1 minute that your house uses >2kW.

WSL is polling your Solar system for every 10 seconds(?), so every 10 seconds the generated power is different.

My system could generate >3kW a hour and on a cloudy day this could change every few seconds from 500Wh - 3kWh.

So at 1 moment my house could return 1kW (2 uses, 3 solar power) to the grid and the other moment my house uses 1,5kW (2 uses, 0,5 solar power).

But in the real world my uses is returning almost all the power to the grid because my dryer is not heating anymore.

So because of the different refresh rates its nearly impossible to calculate/generate/etc a figure in WSL that 100% reflects the real world.

[Peter]

This discussion is reaching a point where it should have it's own topic ;-)

I understand that power (in Watts) over time will allways be averaged, but produced as well as consumed energy (in kWh) can be calculated acurately per interval: 

Apart from momentary power levels at the grid, so only when consumption is higher then production, the smartmeter telegrams will allways contain both timestamps and  imported and exported energy levels. While there is no import because consumption is lower than production, the power could / should be calculated as an average during one such interval,

Apart from momentary power levels, the inverter data also contains timestamps and produced energy volumes, both lifetime and "day up to now totals", so also production can be calculated acurately over the intervals,

While production volumes are higher than export volumes, consumption equals production minus export. This cannot be negative. 

If there is no export, consumtion equals production minus import.

If there is no production, consumption equals import.

So the actual consumption can allways be calcualated, graphed and submitted to PVoutput.

I understand that import and export can not be submited to PVo when the net flag is used, so it will calculate that from production and consumption data. If no or a negative consumption is submitted (which is what WSL does while production is higher) then PVo's calculated results on export, credit and debit's will be (completely) incorrect...  

Although a representation of produced or consumed power will never be 100% acurate it can be a a good approximization, by shortening the poll frequency. The smart meter could be polled every ten seconds also.

As stated, consumption volumes could be accurately calculated and displayed 24/7 nonetheless.

  

Toch?