can its-electric be a power data server?

[mwall]

can its-electric receive 'energy posting' data pushed from a TED
gateway, or does its-electric only poll a TED gateway?

i have a TED installation that is rather remote, has limited power,
and experiences rather extreme weather conditions. at this time i
cannot install a computer with its-electric on the same subnet as TED,
nor can i make the TED gateway accessible beyond the lan on which it
is installed. however i can make the TED gateway 'phone home' to an
arbitrary host on the internet.

i would like to use the 'energy posting' feature of TED to make my TED
gateway push data to its-electric running on one of my servers at a
different location.

can its-electric receive TED data pushed by a TED gateway? are the
data pushed by a TED gateway sufficiently detailed to keep its-
electric happy?

m

[Ron Knapp]

It doesn't support that ability that I am aware of, I have mine running on a plug computer,
it does run but not much else and the graphic display is slow, I am going to try to get something
going to store and forward the data so it-electric can display it on a remote computer. I have
other things running on the plug and it's takes almost half of the memory and the majority
of the cpu so want to come up with something light weight to just do the collection. Not sure
what constraints you have on the computer, the only problem with the plugs is probably heat
as they are passively cooled. The 5000 doesn't have much storage ability until you get to some
of the longer averages so ideally collection needs to be locally in case of internet/connection
failure you don't loose any data. Writing a listener for the third party posts is probably not that
hard, there are several web sites out there that support it since google shut down, but still have
that same problem if you loose connectivity you loose data. I am assuming you want the
detail data or you would just configure yours to post to the web sites that already exist.

Ron

> --
> You received this message because you are subscribed to the Google Groups "its-electric-software" group.
> To post to this group, send email to its-electr...@googlegroups.com.
> To unsubscribe from this group, send email to its-electric-sof...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/its-electric-software?hl=en.
>

[mwall]

i had not considered a plug computer - that might just work, and as
you pointed out it minimizes the potential for data loss. humidity
and low temperatures (and salt water spray) are the hostile
environmental factors.

in my scenario the plug would do two things: 1) use its-electric to
store more data than the TED gateway stores, and 2) phone home
periodically so that my existing monitoring system (nagios-based) can
keep tabs on things. i can probably do some simple scripting or java
coding to do (2).

i cannot connect to the TED server remotely because its internet
connection can only be initiated from the local lan, and that often
requires multiple re-tries.

thank you for the suggestion!

[Robert Tupelo-Schneck]

Mwall, there's currently no ability to read posted data. It probably
wouldn't be hard to add. Where are the docs for this? (Though I
don't foresee coding this soon.)

Ron (and mwall), you should be able to control how much memory the
its-electric Java process uses. Just use the -Xmx command-line
parameter to Java, e.g. -Xmx128M. In order to avoid out-of-memory
errors, you *may* have to tweak the size of the database cache. To do
this, create a text file called "je.properties" inside of the
its-electric database directory, with contents the line
"je.maxMemoryPercent=10". (Its-electric uses 40% of memory for the
database cache by default.)

Meanwhile, you should be able to tweak the its-electric options for
how frequently to import in order to reduce CPU usage.

Finally, you could have the plug only record data, and not serve it
(using the --no-serve option to its-electric). Then, you can just
periodically copy the database over the network to a machine which
*can* efficiently serve the data (on which its-electric would run with
the --no-record option). A cron job running rsync leaps to mind as a
possibility. Remember, you can hot-copy the its-electric database as
long as files are copied in alphabetical order. Existing database
files are never changed, only deleted, so rsync should in general only
be copying new data.

Let me know if you need any help with any of that.

Robert

[Ron Knapp]

This is model I am currently playing with
http://www.globalscaletechnologies.com/c-5-dreamplugs.aspx

It pulls a steady 7 watts with no external devices.

I upgraded it to debian squeeze, it has a 2GB root fs
so plenty of room for programs. If you can get it
hooked up at your remote site you could have it
keep a reverse ssh tunnel open to one of your
local servers and there-by give you incoming
access to the site to do whatever you want.

If you want to just use it as a reverse gateway then
the $99 sheeva would work great.

[Ron Knapp]

On Sep 6, 2011, at 10:21 PM, Robert Tupelo-Schneck wrote:

Finally, you could have the plug only record data, and not serve it
(using the --no-serve option to its-electric).  Then, you can just
periodically copy the database over the network to a machine which
*can* efficiently serve the data (on which its-electric would run with
the --no-record option).  A cron job running rsync leaps to mind as a
possibility.  Remember, you can hot-copy the its-electric database as
long as files are copied in alphabetical order.  Existing database
files are never changed, only deleted, so rsync should in general only
be copying new data.

This just occurred to me last night about running 2 instances one is

data collection, one serving, I will look at that. I had reduced

the memory parameter but it is more that it takes about 30 seconds

to server the data on connection.

[Robert Tupelo-Schneck]

Indeed, serving the data quickly seems to require a large memory cache.

Robert

[Ron Knapp]

Robert,

I have never worked with running java as a servlet so am on a learning curve,
but I believe you said that it would use up to 40% for cache and a max
of the memory parameter. After running 24 hours this is what I have

3448 root 20 0 526m 398m 292 S 45.9 79.4 1252:09 java

it is using 526m virtual, 398m real memory, with 45%cpu and 79.4% of total memory.

Is there something I am missing as this all seems to exceed the default limits
you described. It is also running my weather station software and some minor monitoring
software. Without it's electric it is less then 1% cpu and has like 394m of the 512m
of memory free.

Ron

[Robert Tupelo-Schneck]

Post the exact command line you use to start the its-electric Java program.

Robert

[Ron Knapp]

Pretty sure is one given in your example. It doesn't start out this bad, just there appears to be some leak or
memory creep over time.

I restarted it and after running about 30 minutes it is very reasonable

root 20 0 395m 111m 5060 S 0.3 22.2 30:34.73 java

395m Virt 111m real, .3% cpu and 22.2% ram

Just wondering if there is some sort of thrashing or some anomaly with the small memory footprint
going on that makes it continually suck up more resources at time passes.

exec /usr/bin/java -server -Xmx256m -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simpl
elog.defaultlog=debug -jar /opt/ted/its-electric-1.7.4.jar -g http://192.168.1.75 -p 8081 -m 4 -l stderr -u username /opt/ted/electricDb >> /opt/
ted/its-electric.log 2>&1 <<<password

Ron

[Robert Tupelo-Schneck]

I also use -server -Xmx256m on a computer with 4G of memory. After a
month of uptime it's only using 0.3% CPU and 4.6% memory.

I'd hope you could get away with using considerably less memory...

Let's see. When your its-electric gets busy, try running "jstack PID"
to see a thread dump, and send me the results.

Robert

[Ron Knapp]

Will do, I have some faster sd cards on order and going to move the system to
that when they come in, Class 6 so that should help on the database reads/writes,
root fs is on a class 2 sd card at 2MB/s and the database are on a usb drive at
around 4MB/s, the new cards will be 6MB/s.

[mwall]

i have been running its-electric for awhile now on three different
machines, monitoring two different ted5000 gateways. in each case,
its-electric seems to consume more and more memory as it runs. on the
dreamplug i have to kill its-electric periodically as it eventually
seems to consume all available memory. btw, although its-electric
definitely seems to leak, it looks like something else on the
dreamplug also leaks. when i monitor memory use on the dreamplug
(using nagios), free memory gradually decreases over time, even with
nothing running.

i am using its-electric 1.7

instance 1:
hardware: mac mini g4 with 512MB RAM
os: debian 5

java version "1.6.0_18"
OpenJDK Runtime Environment (IcedTea6 1.8.7) (6b18-1.8.7-2~lenny1)
OpenJDK Zero VM (build 14.0-b16, interpreted mode)

java -jar its-electric-1.7.4.jar -g http://192.168.32.151 -m 4 -d /
home/itse/db-house

its-electric has been running for 12 days. VSZ: 669956 RSS: 362080
cpu use is typically 10-15%. memory use is now 70% and continues to
grow.

instance 2:
hardware: dreamplug
os: debian 5

java version "1.6.0_18"
OpenJDK Runtime Environment (IcedTea6 1.8.7) (6b18-1.8.7-2~squeeze1)
OpenJDK Zero VM (build 14.0-b16, mixed mode)

java -jar its-electric-1.7.4.jar -g http://192.168.32.151 -m 4 -d /var/
itse/db-house --no-serve --voltage

its-electric has been running for 2 hours. VSZ: 656004 RSS: 63572
cpu use is typically 4-8%. memory use is now 12% and continues to
grow.

instance 3:
hardware: eepc
os: ubuntu 10.04
other details not available right now (the machine is on an island off
the coast of maine). it has been running since 10sep11. i'll provide
stats after i get to the island this weekend.

[Robert Tupelo-Schneck]

To reiterate, I have run its-electric for months at a time with no
memory issues.

Here are some things to try.

(1) Give Java an explicit memory limit, using an option to java like
-Xmx256M. (Or 128M, or 64M. Too low and you'll need to tweak
database parameters as I outlined in an earlier message; but I bet
128M would just work. I know 256M just works.)

(2) Use Oracle's Java instead of OpenJDK. Here's a site where someone
describes making that switch due to OpenJDK using too much memory:
http://wiki.apidesign.org/wiki/Virtualization

If those don't work... or if (1) doesn't work and you really need to
use OpenJDK for some reason... let me know.

For a record-only instance of its-electric, you could add the options
--no-serve --import-interval 0 --long-import-interval 1200 which has
its-electric ask TED for an hour of data every 20 minutes, and turns
off the data server and the more frequent small imports. You need (1)
or (2) above to fix your memory problem, but this is probably a good
idea just to make the thing work as little as possible.

Robert

[mwall]

after running for one month, the instance of its-electric running on
the eepc was using this much memory: VSS:761780 RSZ:359720

as for the dreamplug...

i wish i could use oracle's java, but i am running on an arm cpu and
as far as i can tell there is no oracle java for arm.

when i use the arguments '--import-interval 0 --long-import-interval
1200', its-electric quits immediately and complains that i am not
feeding it arguments that it understands. this is the full command:

java -Xmx128M -jar its-electric-1.7.4.jar -g http://192.168.2.50 -m 4 -
d /var/itse/db --voltage --no-serve --import-interval 0 --long-import-
interval 1200 2>&1 >> /var/log/its-electric.log &

i created a file called je.properties in the database directory, and
it contains a single line 'je.maxMemoryPercent=10'

my startup command looks like this:

java -Xmx128M -jar its-electric-1.7.4.jar -g http://192.168.2.50 -m 4 -
d /var/itse/db --voltage --no-serve 2>&1 >> /var/log/its-electric.log
&

at startup i now get:
VSS: 261556 RSZ: 68132

its-electric now uses much more cpu than it did before. it sits at
over 95% cpu for minutes at a time.

this is on a dreamplug with 512MB RAM running debian 5.

i'm afraid that the java virtual machine is simply too much overhead
for a plug computer with 'only' 512MB RAM.

[Ron Knapp]

I am running mine on a dreamplug as well using the open source jdk and debian squeeze.

After 5 days I have this.

1886 root 20 0 396m 271m 1740 S 2.0 54.2 1125:30 java

So using 54% of the 512M ram.

So 271m res and 396m Virt

My startup is

/usr/bin/java -server -Xmx256m -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simplelog.defaultlog=debug -jar /opt/ted/its-electric-1.7.4.jar -g http://192.168.1.75 -p 8081 -m 4 -l stderr -u username /opt/ted/electricD

My problem is drive space trying to only use flash and about full on my 8gb so have to come
up with something else. I have written a python program to dump all the data from the
unit into sqlite, the seconds table up to the month table. Just need to either come up with some
sort of graphics engine or somehow pipe it into its-electric. I am doubtfull that it could
feed the data fast enough to power an app like it-s electric but at least I have all the data
stored and I can even insert missing second records from the minute or even the hourly
data.

Still on the 1.7.4 version so I know he made some improvements with memory but haven't
had a chance to convert yet.

[Robert Tupelo-Schneck]

I really believe that OpenJDK could be made to work fine, even on your
Dreamplug, perhaps if we find the right options. The memory option
should keep the memory under control. Now it's the CPU. In
principle, its-electric doesn't need to be doing very much at all,
especially if it's not serving data; so if it's eating up all the CPU,
even on a tiny machine, something is going wrong.

I'd appreciate it if you send me your log. You probably need to add
options (to java) to get useful logging:
-Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog
-Dorg.apache.commons.logging.simplelog.defaultlog=trace

You could also do "jstack PID", a few times even, during high-CPU use.
This generates a thread dump.

Finally, the ability to turn off the short import was added in the
recent its-electric 1.8. If you need to use 1.7.4 you could try
"--import-interval 1200 --long-import-interval 1200", setting both to
20 minutes. That may go far to easing up on the CPU usage.

Robert

[Robert Tupelo-Schneck]

The memory improvements in 1.8 are mostly about the Javascript client,
not the Java server. However 1.8 does have an option to allow you to
delete old data from your database.

Robert

[mwall]

On Oct 10, 10:12 pm, Robert Tupelo-Schneck <schn...@gmail.com> wrote:
> I really believe that OpenJDK could be made to work fine, even on your
> Dreamplug, perhaps if we find the right options.  The memory option
> should keep the memory under control.  Now it's the CPU.  In
> principle, its-electric doesn't need to be doing very much at all,
> especially if it's not serving data; so if it's eating up all the CPU,
> even on a tiny machine, something is going wrong.

i have now installed its-electric 1.8 on two different computers.
'tupi' is the dreamplug mentioned earlier in this thread. 'burgos' is
an ionics 'nimbus' plug computer. in both cases, i start its-electric
with this command:

java -Xmx128M -jar its-electric-1.8.jar -g http://192.168.2.50 -m 4 -d
$DB --voltage --no-serve --import-interval 0 --long-import-interval

1200 2>&1 >> /var/log/its-electric.log &

the memory limit of 128M seems to be working. after a few days i see
roughly VS of 300000 and RSS of 135000.

when its-electric first starts, the cpu use is 70-90% for up to 10
minutes. i think this related to the size of the database. when
there are only 2 or 3 .jdb files, the cpu use drops off after a 30
seconds or so. when there are about 120 .jdb files, the cpu use is
sustained for quite awhile, but eventually drops off.

i am running into i/o issues (i think) with the plug platform. the
dreamplug is particularly bad - if i run nagios, wfrog, and its-
electric then the plug struggles to respond to pings and often drops
my ssh connections. the ionics plug is *much* better - i can run
nagios, wfrog, and its-electric and while this slows down the plug, at
least it still functions. both computers have 1.2G ARM cpu and 512M
RAM.

sorry but no logs yet - i had a short window in which to set
everything up, then it has to run unattended for the next 6 months.