Database problem
Ron
Ron
Robert Tupelo-Schneck
error message doesn't turn up anything particularly useful. Did
anything unusual happen around the time that this occurred?
You can probably recover the old data using the DbDump and DbLoad
back-end database commands.
You need to unzip the its-electric-*.jar file (a jar file is just a
zip file) in order to fish out the je-*.jar file. Then run something
like
java -cp je-*.jar com.sleepycat.je.util.DbDump -h
path/to/BAD/database-directory -r
This should create a 1.dump file for seconds resolution, 4.dump file
for 4-seconds resolution, etc..
Then you would do
java -cp je-*.jar com.sleepycat.je.util.DbLoad -h
path/to/GOOD/database-directory -n -s 1 -f 1.dump
to load the seconds database into the new database. (The -n option
means data from the new database will not be overwritten.) Repeat for
the other resolutions.
First do this on a *copy* of your new, good directory, because I
haven't tested this!
Let me know how this goes!
Robert
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/its-electric-software/-/5_oa2rwRkLEJ.
> To post to this group, send email to its-electr...@googlegroups.com.
> To unsubscribe from this group, send email to
> its-electric-sof...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/its-electric-software?hl=en.
>
Ron Knapp
011-09-10 21:09:53-0400,4,417,,
2011-09-10 21:09:54-0400,1,455,,
2011-09-10 21:09:54-0400,2,6,,
2011-09-10 21:09:54-0400,3,0,,
2011-09-10 21:09:54-0400,4,417,,
2011-09-10 21:09:55-0400,1,457,,
2011-09-10 21:09:55-0400,2,6,,
2011-09-10 21:09:55-0400,3,0,,
2011-09-10 21:09:55-0400,4,417,,
Trying the dump now.
Ron
Ron
Robert Tupelo-Schneck
larger... currently 17G... and probably gets more traffic to boot. No
problems at all.
The DbDump program can take -R instead of -r to more aggressively try
to recover a corrupted database.
Out of curiosity, what's the output of "java -version" for you?
So have you managed to recover most of your data?
Robert
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/its-electric-software/-/3bGwYXx2zbsJ.
Ron Knapp
OpenJDK Runtime Environment (IcedTea6 1.8.7) (6b18-1.8.7-2~squeeze1)
Yes once I restored the missing files. I think whatever happened that sucked up the
memory last week was cause or result of this data corruption. With all the missing
files restored I am sitting at 30% of memory by java and very low cpu usage even
on the arm.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17152 root 20 0 400m 152m 3944 S 4.0 30.4 127:21.18 java
load average: 0.32, 0.52, 0.39
mtrcycllvr
Thanks for posting this tutorial on how to salvage a database.. I probably sent you email directly by mistake this AM, Robert - I MEANT to simply post, but was in the "old" groups interface.. Silly me.. ;-)
Anyway, I have a question that you may or may not be able to answer?
The basic summary is that I corrupted my database by pointing two instances of its-electric at the same data repository by way of bleary eyes and fat fingers.. THEN I found out that the zfs snapshot I'd made didn't include the dataset that held the repository.. Double bad.
Your instructions were clear enough, however, so I created a new directory, created the dumps, then loaded the dumps into the new directory in the order you specified, highest granularity to lowest.
What I found was that the restore only SEEMED to reach into November of last year.. What was MORE interesting was that by the time I got to work, there was more data - suddenly I could see into January data.. Now, some eight hours later, ALL of the 1day and 8hour data seems to be present. However none of the higher resolutions seem to be there..
Clearly it APPEARS like there is some sort of indexing of the data going on in the back ground - or at least that's the only rational explanation I can come up with that fits the observed behavior.
While all that was going on, I was busy switching to the "-R" vs. "-r" for a higher level of robustness in DB recovery.. I don't think that's really required for my case, but I proceeded anyway. File sizes generated are SIMILAR, though not identical..
So, my question is this:
IS there some sort of back ground indexing going on, do you think? I am hesitant to perform another DB load of this freshly salvaged data if what I've got is actually "okay" and will continue (at a slower and slower pace) to rebuild itself.
You can view for yourself here:
http://www.mtrcycllvr.org/power-consuption/suburban-hell.html
The MAX link resolves quite nicely now (oh, and I was able to get rid of that rouge data point!! Thanks for that command line option!!) 1day and 8hour data all appears to be there at this time, but if you try any resolution above that, there appears to be nothing but a beginning and end point (excluding my fresh data being fed in since the morning of 2.29.2012)..
I think I'm going to simply let things sit for a bit - I have what I think are two sets of salvaged dumps at this point.. The 1day/8hour data took four to five hours to finish generating/indexing (whatever was/is) going on there. I expect higher resolutions would take X multiple times that period to achieve the same result? ie; 3 hour data might not show up for (2.5x5=12 hours or more) quite a long time?
Am I grasping at straws here, or are my guesses at least sort of on target, do you know?
Should I, after a good long wait, attempt to reload the higher resolution data, perhaps?
Bob, after your restore, did the data all start showing up immediately for you?
Thanks so much for your time and thoughts!
A few more details:
[ewt@helvetti:/nas/homepower]% zfs list nas/homepower/aloha
NAME USED AVAIL REFER MOUNTPOINT
nas/homepower/aloha 845M 3.08T 845M /nas/homepower/aloha
[ewt@helvetti:/nas/homepower]% zfs list nas/homepower/old
NAME USED AVAIL REFER MOUNTPOINT
nas/homepower/old 1.73G 3.08T 1.73G /nas/homepower/old
[ewt@helvetti:/nas/homepower]%
This makes me think I have NOT really restored all my data!! "old" is just a copy of the original *jdb files. aloha is the reloaded *.jdb database directory… That's a pretty big difference in bits..
Could the order I loaded them be significant?
dump file size comparison between "-r" and "-R":
[ewt@helvetti:/nas/homepower]% ls -l *.dump | sort -n -k 9
-rw-r--r-- 1 root root 665298204 Feb 29 11:19 1.dump
-rw-r--r-- 1 root root 169405838 Feb 29 11:19 4.dump
-rw-r--r-- 1 root root 45179210 Feb 29 11:19 15.dump
-rw-r--r-- 1 root root 11295236 Feb 29 11:19 60.dump
-rw-r--r-- 1 root root 2823374 Feb 29 11:19 240.dump
-rw-r--r-- 1 root root 754900 Feb 29 11:19 900.dump
-rw-r--r-- 1 root root 190464 Feb 29 11:19 3600.dump
-rw-r--r-- 1 root root 66786 Feb 29 11:19 10800.dump
-rw-r--r-- 1 root root 28158 Feb 29 11:19 28800.dump
-rw-r--r-- 1 root root 12750 Feb 29 11:19 86400.dump
[ewt@helvetti:/nas/homepower]% cd dump/first_pass
[ewt@helvetti:/nas/homepower/dump/first_pass]% ls -l *.dump | sort -n -k 9
-rw-r--r-- 1 root root 665282994 Feb 29 03:43 1.dump
-rw-r--r-- 1 root root 166394258 Feb 29 03:43 4.dump
-rw-r--r-- 1 root root 44372000 Feb 29 03:43 15.dump
-rw-r--r-- 1 root root 11093150 Feb 29 03:43 60.dump
-rw-r--r-- 1 root root 2773442 Feb 29 03:43 240.dump
-rw-r--r-- 1 root root 739708 Feb 29 03:43 900.dump
-rw-r--r-- 1 root root 185028 Feb 29 03:43 3600.dump
-rw-r--r-- 1 root root 61818 Feb 29 03:43 10800.dump
-rw-r--r-- 1 root root 23244 Feb 29 03:43 28800.dump
-rw-r--r-- 1 root root 7872 Feb 29 03:43 86400.dump
[ewt@helvetti:/nas/homepower/dump/first_pass]%
OS: Solaris 11 un-patched (Open Indiana on the next server!)
Server: Supermicro ATOM 525
Sum: Super rockin', ultra reliable, very low power, inexpensive NAS solution for the household data needs..
Robert Tupelo-Schneck
java -cp je-*.jar com.sleepycat.je.util.DbLoad -h
path/to/GOOD/database-directory -n -s 1 -f 1.dump
for seconds, but
java -cp je-*.jar com.sleepycat.je.util.DbLoad -h
path/to/GOOD/database-directory -n -s 4 -f 4.dump
java -cp je-*.jar com.sleepycat.je.util.DbLoad -h
path/to/GOOD/database-directory -n -s 15 -f 15.dump
and so on.
If you used -s 1 for all of them, only the seconds data would actually
have gotten included. Its-electric will recreate the coarser
resolutions, however that process is quite slow. I would have thought
that's all that's happening, to cause you to see "more data" start to
appear over time... it just reflects how far along its-electric is in
recreating coarser resolutions.
However, if you're not seeing any data for "in-between" resolutions,
something has gone wrong. Is that still the case? Let me know.
Robert
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/its-electric-software/-/yBL9hP1YPN0J.
Eric Timberlake
I was just able to restore the missing data. The DB still isn't QUITE right, but it's WAY better.
I think I need to take a closer look at the 1.8 distribution before inquiring any further - theres an anomaly when the graph first loads where scroll to present doesn't actually, but after a specific range has been request it's better, but "present" it still some offset of hours prior to the current time.
Anyway, THANKS so much for pointing out my blunder! Pretty silly error on my part.. ;-)
-ET-
Robert Tupelo-Schneck
pre-release to see if this bug is fixed?
Robert
Eric Timberlake
I'm not 100% clear there's a bug, but sure, I'm happy to test out now than I'm a little more comfortable with my use of zfs snapshots.. I honestly believe all my issues are 100% self inflicted and I was leaning towards a "re-export" of my existing database and recreating it one more time (knowing more fully what I'm doing) to see if that "zoom to present" issue clears up. I'm not satisfied I took all the correct steps at the correct time, as a matter of fact, we both know I was running the DB without major data points for some time..
I should mention that I've been using 1.8 in "test" mode with my remote meter without this issue at all, which is what leads me to believe there are still DB based anomalies vs. a bug in your code.
You can see the test version here:
http://test.mtrcycllvr.org/test/its-electric.html (my supposed retirement home - like I'll ever ACTUALLY be able to retire.. ;-) )
I just checked and the scroll to present function works perfectly there, but I've never migrated, or more importantly, never corrupted the DB filesystem like I did with the Suburban Hell DB. The only apparent data loss/corruption actually isn't - the power was actually out for several days after a storm. (1.18-1.20.2012)
I do plan, at some point with my copious free time, to do a small write up of how to use Solaris (and by inference Open Indiana) configuration of its-electric - how to configure as a service using SMF and the advantages of zfs compression and whatnot.. Some day.. ;-)
Anyway, long answer to a short question: YES, I'd be happy to test out a beta of 1.9.
Sincerely,
-ET-
Sea Monkeys1
Robert Tupelo-Schneck
.jar---possibly je-5.0.34.jar), and put a copy of that file wherever
you will be running the DbDump command. Then use the actual name,
like
java -cp je-5.0.34.jar com.sleepycat.je.util.DbDump -h
path/to/BAD/database-directory -r
Robert
>> > http://groups.google.com/group/its-electric-software?hl=en.
>> >
>
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> email to its-electric-sof...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Sea Monkeys1
>> > For more options, visit this group at
>> > http://groups.google.com/group/its-electric-software?hl=en.
>> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to its-electric-software+unsub...@googlegroups.com.
Robert Tupelo-Schneck
surely!) recreate the coarser resolutions.
Robert
>> >> > http://groups.google.com/group/its-electric-software?hl=en.
>> >> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "its-electric-software" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > Visit this group at
>> > https://groups.google.com/group/its-electric-software.
>> > For more options, visit https://groups.google.com/d/optout.
>
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Sea Monkeys1
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/its-electric-software?hl=en.
>> >> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "its-electric-software" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to its-electric-software+unsub...@googlegroups.com.
>> > To post to this group, send email to its-electr...@googlegroups.com.
>> > Visit this group at
>> > https://groups.google.com/group/its-electric-software.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to its-electric-software+unsub...@googlegroups.com.
Robert Tupelo-Schneck
Robert
>> >> >> > http://groups.google.com/group/its-electric-software?hl=en.
>> >> >> >
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "its-electric-software" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > Visit this group at
>> >> > https://groups.google.com/group/its-electric-software.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "its-electric-software" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > Visit this group at
>> > https://groups.google.com/group/its-electric-software.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Sea Monkeys1
>> >> >> > its-electric-software+unsub...@googlegroups.com.
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/its-electric-software?hl=en.
>> >> >> >
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "its-electric-software" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > email to its-electric-software+unsub...@googlegroups.com.
>> >> > To post to this group, send email to its-electr...@googlegroups.com.
>> >> > Visit this group at
>> >> > https://groups.google.com/group/its-electric-software.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "its-electric-software" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to its-electric-software+unsub...@googlegroups.com.
>> > To post to this group, send email to its-electr...@googlegroups.com.
>> > Visit this group at
>> > https://groups.google.com/group/its-electric-software.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to its-electric-software+unsub...@googlegroups.com.
Robert Tupelo-Schneck
see very recent data.
What do you see if you try a small "--export" from an earlier time range?
You may want to try starting over and loading only the dumped seconds,
and let its-electric slowly catch up on the other resolutions.
Robert
>> >> >> >> > http://groups.google.com/group/its-electric-software?hl=en.
>> >> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "its-electric-software" group.
>> >> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> >> > send
>> >> >> > an
>> >> >> > its-electr...@googlegroups.com.
>> >> >> > Visit this group at
>> >> >> > https://groups.google.com/group/its-electric-software.
>> >> >> > For more options, visit https://groups.google.com/d/optout.
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "its-electric-software" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > Visit this group at
>> >> > https://groups.google.com/group/its-electric-software.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "its-electric-software" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > Visit this group at
>> > https://groups.google.com/group/its-electric-software.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Sea Monkeys1
>> >> >> >> > its-electric-software+unsub...@googlegroups.com.
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/its-electric-software?hl=en.
>> >> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "its-electric-software" group.
>> >> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> >> > send
>> >> >> > an
>> >> >> > email to its-electric-software+unsub...@googlegroups.com.
>> >> >> > To post to this group, send email to
>> >> >> > its-electr...@googlegroups.com.
>> >> >> > Visit this group at
>> >> >> > https://groups.google.com/group/its-electric-software.
>> >> >> > For more options, visit https://groups.google.com/d/optout.
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "its-electric-software" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > email to its-electric-software+unsub...@googlegroups.com.
>> >> > To post to this group, send email to its-electr...@googlegroups.com.
>> >> > Visit this group at
>> >> > https://groups.google.com/group/its-electric-software.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "its-electric-software" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to its-electric-software+unsub...@googlegroups.com.
>> > To post to this group, send email to its-electr...@googlegroups.com.
>> > Visit this group at
>> > https://groups.google.com/group/its-electric-software.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "its-electric-software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to its-electric-software+unsub...@googlegroups.com.