Can someone explain how I turn this into data I can use?

[Dhowman]

I've read about a dozen posts on data logging and the manual and the base91 compression Dacian uses but I'm at a loss.  I seem to be able to grab the following over WiFi simply by pressing the "save log" button on the SBMS HTML monitoring web page, but have no idea what to do with this to see the uncompressed data:

undefined7*)'JX#TGPGTGTGSGTGQGSGL*j##-#1{#######1U###############%N(############################################################

7*)'J\#TGQGTGTGTGTGRGTGL*e##-#1m#######1S###############%N(############################################################

7*)'K##TGPGUGTGSGTGQGSGL*g##-#1u#######1U###############%N(############################################################

7*)'K(#TGPGUGUGSGTGRGSGM*g##-#2,#######1[#####'#########%N(############################################################

7*)'K,#TGRGTGUGSGTGQGRGL*g##-#2(#######1]#####)##(######%N(############################################################

7*)'K/#TGQGTGTGSGTGPGSGL*j##-#1{#######1U#####(##)######%N(############################################################

7*)'K4#TGRGVGTGSGTGQGSGM*j##-#1v#######1\########'######%N(############################################################

7*)'K8#TGPGUGUGTGTGRGSGL*j##-#20#######1_#####'##(######%N(############################################################

7*)'K;#TGQGUGTGSGSGQGSGL*g##-#1}#######1]#####'##)######%N(############################################################

.

.

.

Can someone help me out here?

Much appreciated.

thanks,

Howard

[Dacian Todea]

Howard,

This part     7*)'J\#TGQGTGTGTGTGRGTGL*e##-#1m#######1S###############%N(   is stored every time an update is made on the html page the rest will always be # that is zero as that other part is for the DMPPT450 only.

The manual will show the significance of all this and how to decompress

For example first value is for the Year that 7 at the beginning and if you check an ascii table you will see 7 is 55 in decimal then 55 - 35 = 20 meaning year 2020

Next is * that is for month so 42 - 35 = 7 so month 7

First cell voltage is GQ so you will have (71 -35)*91^1  + (81-35) = 3276+46 =3322mV

[Dhowman]

That much I understood.  But if you're pointing me to the manual to understand how to convert all those characters, I'm lost.

Is there a tool I can use to convert all those values?

[Dacian Todea]

You can see this formula in the user manual 

dcmp(p,s,d){xx=0; for (z=0;z<s;z++){xx = xx + ((d.charCodeAt((p+s-1)-z)-35)*Math.pow(91,z));}return xx;}

This is the same formula used in the html file to decompress and show the data.

There is currently no other tool just that html file that is written in javascript and you can edit that if you want just in a simple text editor.

[Dhowman]

Might help to know that all I need to do is take a full log file following a capacity test to see how individual cell voltages trended over time, so just need to bulk convert a file.

Won't ever need to convert in real time.

[Dave Festing]

If you want to write a Python3 script to do this I could send you something that would be a good start.

[Dhowman]

Appreciate it Dave but unless there's a Windows executable someone's written to unpack Dacian's data, it's a feature that doesn't exist for 99.5% of us.

No one's done this?

[Barry Timm]

I created a simple Excel spreadsheet to parse each character and convert it using the built in Excel formulas.
It works OK but only really suitable for one off analysis, not real time.

[Dhowman]

Thanks Barry.  Took a stab at just that solution.  It's not elegant and it took hours to figure it out (after a crash course in ASCII encoding ... god bless Wikipedia), but it seems to work.  Have a few questions though (see attached Excel file).

1. I just grabbed log file from the SBMS monitoring Web page "Save Log" button but it's giving me a row for each 5 sec page refresh so I presume this isn't a full log file once it gets past a certain point?

2. What's the simplest way to grab the log file with the 2 minute interval data (that starts rolling over every 2 yrs?) ?  I just imported the log file as "fixed width" data with one character per column on the "Encoded" tab of the spreadsheet.)

3. I think I understand how to parse each ASCII character to get its corresponding unencoded value and then how to roll adjacent # values to get the calculated values for everything up to and including ht2, but if someone wouldn't mind giving it a sanity check, that would be great.  (see yellow highlighted example:  row 3 on "Unencoded" tab ... replicated this logic for all records on the "Calculated" tab).

4. I'm clueless, however, on how to parse the remaining 66 characters in the row to get the Error Code data.  User Guide doesn't provide the level of detail for those that it does for the preceding fields so I am unable to ferret those out.

Any help (in the file?) on those data values that anyone can provide would be much appreciated.

Thanks,

Howard

[Dave Festing]

I don't know much about using formulas in Excel ... can you do logical AND, logical OR and Bit mask?

[Dacian Todea]

Howard,

1.2. I think you are interested in the SBMS0 internal data log that is saved every two minutes and that contains the data at fixed two minutes interval. That can be downloaded through USB all you need is a serial monitor program then select the month you want to download from the SBMS menu and push the Start Data Download that will automatically send all the stored data for that selected month and your serial monitor will capture that and you can save it to a text file.

You need to set the serial UART to ON and speed say at 115.2Kbps do download the internal stored data or you can just collect live data by selecting the refresh interval (default I think that is set to 30 seconds but you can set that as fast as 1 second or as slow as 4 minutes).

   

3.4. The user manual explains in details what each of the fields are and you only care about the first 59 characters it starts with "7*)" date and ends with "%N(" error code

This is from manual

var sbms="YMDHMS%%C1C2C3C4C5C6C7C8ITET+BATPV1PV2EXTAD2AD3AD4ht1ht2ERR"

Y ,M ,D ,H ,M ,S ,SOC,cel1 ,cel2 ,cel3 ,cel4 ,cel5 ,cel6 ,cel7 ,cel8 ,IT ,ET ,-Batt , PV1 , PV2 ,ExtLoad, ADC2, ADC3, ADC4, heat1 , heat2 , ERR

00,01,31,22,53,47,100,3414,3416,3444,3433,3426,3419,3420,3380,000,000,-002000,000000,000000,000000,000000,000000,000000,000000,000000,000000

Maybe you can only convert up to temperature is because there are groups of 3 characters after that so as an example here is how error code will be calculated

It is "%N(" in your data so

% =  37 -35 = 2

N   = 78 - 35 = 43

(  = 40 - 35 = 5

(2 * (91*91)) + (43* 91) + 5 = 16562 + 3913 + 5 = 20480

This is the number you will see under battery SOC as Stat: 20480

Then to convert this number to individual flags is also explained as you will convert this number to binary thus you will get a 15bit binary number and each bit is one of the 15 error flags with OV (OverVoltage) the least significant bit and DFET the most significant bit in the order that are listed in page 3 of the monitoring screen.

Not quite sure what is in your table but you seems to just have each character converted to decimal and subtract 35 from that but for example on cell voltages you do not have that converted to cell voltage in mV

So like at some point you have one of the cells "GP"

You converted that to 36 and 45 but to convert in mV you need 36*91 + 45 = 3321mV

[Dhowman]

So, an initial web search for serial port data loggers and screenshots of 'em ... their purpose seems to be to capture RT data, not browse to and download already logged data as a file, but I'll have to dive into that another day.

If you can recommend one that does what you describe, that might save me quite a bit of trial and error.

Regarding data calcs in that spreadsheet ... 

If the Error code is captured in the next three ASCII characters after the 3-character ht1 value, OK then, but there's no comma after "ERR" below and nothing that I could glean from your manual that explains what's going on with everything in yellow here, so I assumed everything that followed (in yellow) was related to error code data and flags.

Y ,M ,D ,H ,M ,S ,SOC,cel1 ,cel2 ,cel3 ,cel4 ,cel5 ,cel6 ,cel7 ,cel8 ,IT ,ET ,-Batt , PV1 , PV2 ,ExtLoad, ADC2, ADC3, ADC4, heat1 , heat2 , ERR

00,01,31,22,53,47,100,3414,3416,3444,3433,3426,3419,3420,3380,000,000,-002000,000000,000000,000000,000000,000000,000000,000000,000000,000000

Per your manual ... 

ERR - this is error code it will contain the information referring to the error flags on the SBMS.

There are 15 Flags (see monitoring page3 on SBMS) not all flags represent an error some are representing normal operation and if you consider this a 15bit binary number each bit representing one flag with “1” highlighted and “0” not highlighted and you start from the top left with OV flag and continue to right and then down to the last flag DFET and consider the OV as the least significant bit and DFET most significant bit then you get this binary number and convert to decimal that will be the value contained in this ERR Status and you can also see this number on the SBMS LCD just under the battery SOC value and also displayed as Status: in the html page. 

That's all there is.  I presume the yellow is related to "15 Flags" but there are 27 comma separated values in your list above and 63 remaining ASCII characters but nothing that i can glean that allows me to continue this exercise with that data to see which are on and which are off.

I've used the formulae you provided exactly as you described, but am displaying V and A instead of mV and mA:

So like at some point you have one of the cells "GP"

You converted that to 36 and 45 but to convert in mV you need 36*91 + 45 = 3321mV

Yup, that's what I've got for GP:

And the 3-ASCII char calcs (e.g. Ext Load = #1U):

 I'll calc the ERR code from the next 3 values after ht1 (same as I calc'd Ext Load above) but don't know what the rest are or how to calculate those (per above).

[Dacian Todea]

Howard,

What you marked with yellow is the compressed data shown as converted to decimal most is zero but shows how large of a number can be in that data.

The 59 ascii characters is all you care about and those will convert in to 27 numerical values as some are just one ascii character so can only represent a number from 0 to 90 like year or minutes then cells voltage is made of two ascii characters and with two you can represent a max value from 0 to 90 * 91 + 90 = 8280 and then with 3 ascii characters you have things like battery current and PV current and there you can represent a value from 0 up to (90 * 91 * 91) + (90 * 91) + 90 = 753570 thus the SBMS0 limitation of 750A (750000mA) when measuring current as more can not be encoded in 3 ascii characters.

The rest of the stuff after ERR will is always just zero You can see a bunch of ####### and those are only something you will care about if you had a DMPPT450 as they are related to that.

[Joe Herring]

Someone could make some money if they figured out how to parse and the present this data in an easy to understand format via windows app. I wish I could. I'd definitely be interested in it if there was one out there.

During my prototyping/testing days, it was more than once that I returned to the test site and just couldn't tell how low the battery went on any particular night. I could do some calculations based on the overall load captured, but sometimes it went days and there was just no way to tell.

We love the graphical interface on the SBMS0 (or SBMS120 that I own) but historical data is very:

1. Just plain interesting and cool

2. Could be useful to monitor:

• weather
• load
• battery voltage
• overall system trends
  

3. Troubleshoot possible issues with the system that can't easily been diagnosed by just what's happening this second. Or the past 24 hours.

I'm of the opinion that many if not most of the end users are people like me that researched online, poured over and over Dacian's instruction manuals, and watched countless YouTube videos to learn how to setup and maintain a system like this. We're intelligent enough to understand that there's not anything else out there, at this price-point or any price-point that I've seen that compares to Dacian's creation. Well don sir!

But on the same level, we are not familiar with base91. Some may have the time and desire to learn this for a one-time use, but others don't have the time, or in my case, no time AND really don't want to learn base91. (just sayin') I'm sure there are plenty of us out there that would love to have, and pay for, a plug and play solution. Something that we could access that would present us the data that is captured.

Cheers!

[Robert Muster]

Please don’t take this post too seriously. I know everyone who likes data wants records kept by their system. Before we had automation, data was always gathered manually.

I can’t understand why users have an issue with knowing how low the SOC of their battery gets every night and all the other associated data. That data is easy to get. It’s simple, really, just get out of bed before the sun comes up and look at the SBMS screen. If you want a record, either take out a pencil or start a spreadsheet and that’s it.

If you want all the other daily data for the 24 hours until dawn, turn off the midnight data reset in the Time and Date menu. You’ll be able to get the data for the 24 hours until dawn from page 2 of Monitoring and reset it when you take the data readings.

Yes I know this is time consuming and not automated and you’ll probably get fed up with it after a while, but you can do it if you really are interested.

For a while, I did it this way:
Got out of bed before dawn (as my elderly bladder persists I do), captured the screen as an image on my iPad I use as a system monitor, reset the SBMS data count on Page 2 and went back to bed. (There was a toilet stop in there somewhere.)

When I had the time later, I gathered the data from the images and recorded it in Excel then deleted the images. Got bored with collecting data after a few months, but I’ve always Wondered if there is an app you could run to automatically capture the screen at a set time.

Bob

[Barry Timm]

I agree, and don;t take this too seriously either, but while it is fairly easy to set an alarm just before sunrise to write down the soc, that's a VERY limited circumstance. 

What happens when one does not know the time of the lowest soc level? For example, when running an A/C during the day and you happen to be out sightseeing or grocery shopping, with a pet in the RV - a circumstance which happens all the time with tens of thousands of rvers.....

Up until recently, it was simply impossible for RVers to even boondock with an A/C keeping things cool for pets. Now that we have these awesome Lithium battery systems, and large, inexpensive PV arrays, it has become possible, but we are hesitant when a system does not provide the status of the battery in a simple manner in order to see how the system has handled that high daytime drain, while we're out. And yes, I know the standard answer is to look at the rolling charts of PV and LOAD and do a rough calculation (that only works even roughly if the loads and PV levels are fairly consistent while they're often NOT - but even that is a very, very weak excuse for what should be a very simple set of data produced. 

//Rant off/

[Kohala Jim]

With storage limitations, this encoding is absolutely justified, and likely what drove the logging firmware choices that Dacian made.

My guess is the external streaming is a direct mirror of what's internal to the SBMS0.

But, what are the limitations when dumping data said internal to the external world?

Might just consider streaming this externally in ASCII, HEX, or CSV format as a user option in a new SBMS0 firmware release?

[Joe Herring]

"just get out of bed before the sun comes up and look at the SBMS screen"

My specific situation didn't and still doesn't allow this as I don't live at the property that the system is installed at. It is land I am developing and with the help of Dacian's equipment and his willingness to share his knowledge, I've been able to stay off grid and power all my equipment with what the solar panels and batteries provide. It's really awesome!

I hope the comments I made didn't sound like a complaint as that was not how it was intended. A specific need I have is to be able to look at a week, a month, or longer in extreme cases, and see if my batteries SOC ever got to a point I wasn't comfortable with. I currently do not know of an easy way to do this. While on paper, I have the battery capacity needed to run a few small loads 24/7, I just can't bring myself to do it because of not knowing what my SOC was at a given point.

That is the main reason I posted my comment. The SBMS120 and SBMS0 could provide this data if it was possible for someone with limited engineering, software, electrical experience to just plug in a solution that would read and display the data that's already being collected.

No complaints just wishful thinking.

[Dacian Todea]

I will combine the answer to all of the above.

Yes the data compression was used due to limited internal storage as I wanted to have data stored for a full year internally. Also data transfer over serial will take minutes depending on speed and decompressed data like csv will take about 2.5 to 3x longer.

The decompression of live data is already done by the html page I provide written in javascript (so all anyone needs is a text editor to change that). The idea was to provide another html page that could download the SBMS logg data and display in a graphical format but that takes time and it never become the priority when producing the hardware was always priority.

The SOC is something that is calculated and not directly measured thus the reason it is not stored and presented as a graph. The 3 graphs showing the PV, Battery and Load data are taking 90% of the RAM memory this microcontroller can provide thus also the reason they are presented as 4minutes (1 second update), 4hours (1 minute update) and 24h (6 minute updates) to save memory or is you want maximize the amount of available memory while showing data logged at different intervals.

SOC since is calculated and very dependent on the user understanding of the system can provide a false sense of security.

I know that unfortunately around half of the installed SBMS do not provide an accurate SOC value either because the current shunts where not properly calibrated and or set or because the battery capacity is not introduced correctly in the parameter settings. Is quite a bit worse for those that have used cells.

The battery energy counter measures what gets out of the battery (of course same amount just slightly more needs to be put back in to battery at some point).

So say you have a 200Ah battery (real capacity that you tested to be true) and after sunset before going to sleep you reset the energy counters and look at the SOC say it is 70% then in the morning whenever you wake up even if it is after sunrise you can check those energy counters.

Say you see battery energy counter at 30Ah. The simplest calculation you can do is take the worse case and say PV panels did not produced anything up to the moment you wake up even if it is noon so 30Ah represents 15% of 200Ah thus SOC could have went 15% lower than it was before you went to sleep so 70% - 15% = 55%

Likely SOC was a bit higher than before sunset as the PV has started charging the battery and providing the loads directly since that time and if you want you can make more accurate calculation.

Usually at night you will have about the same systems running and you will have a good idea of what they will use until morning.

In normal conditions you will not have the battery below 50% in the morning as that will mean your battery is just undersized for your system no matter where you are in the world and how little cloudy days you have or how oversized the PV array is.

Also you should not see the battery below 30% more than one or twice a year when you did some mistake and forgot to turn off something that did not needed to be ON.

Keep in mind that when you see 30% with a new battery in about 15 years when battery ha degraded by 20+ percent this will mean just 10% SOC

My quality battery is just 3 years old and already 4.4% lower capacity than when was newly installed.

[Barry Timm]

I've bought a $500 Bogart Pentametric monitoring system that I'll use to determine and report SOC with 3 shunts input, for my SBMS0 system reporting.

This not only provides my required SOC reporting but also Internet access to the data, all external to the SBMS0. I'll leave the SBMS0 to managing the battery and loads/chargers, as I'm not smart enough to figure out how to extract the simple SOC data I need from the Electrodacus.

[David Burton]

This is a great conversation and it's exactly why I spend some time building out what was the best solution for me. I wanted to have all the historical data as well as access from the web, so on came a second device. I have a raspberry Pi plugged into my SBMS0 via USB and it collects the data feed every 30 sec and parses it with a python script, very similar the the JavaScript example in the manual. I then insert the data I to an open source database on the Pi (InfluxDB), then display it inside a beautiful open source graphing tool Grafana. Because all the data is being stored, any time I want a new graph, I can make it in seconds. This has been working for me for a while and I love it. I'm a lousy blogger, but here was a post I made about it with my code given away for free on GitHub.

https://familyburton.com/category/burton-passive-house/

[Joe Herring]

While I think this solution is still over my head as far my current capabilities are, I am somewhat familiar with linux command line. Albeit, not an expert by any means but I'm able to follow instructions if they're provided. And provided nothing goes wrong while following those instructions. (Still a tall-order for the person being asked to provide the instructions)

Plug in USB cable - check :-)

Setup a Raspberry Pi 3 B+ - confidence I could as long as google can fill in any gaps

Custom Python script - even with the code provided from the link, I don't know if I'd know how to implement

InfluxDB - install guide provided so I can probably do it

Grafana - install guide provided so I can probably do it

Setting up Grafana dash boards - you said you had to do some research to figure this out. If it was 5 minutes for you then 50 minutes for me is doable.

code updates with Grafana alerts - note sure I could or would need to do this step

This is the closest solution I've yet seen to accomplish what we're discussing and it looks like it's exactly what many of us are desiring.

Nice job David Burton!

I'm of the opinion that it's might be beyond my ability at this time but I am willing to bet that there are a number of SMBS owners out there that could take the knowledge you've shared with us and get a beautiful system up and running.

This could be setup and tested on a laptop running a linux VM yes?

Thanks for sharing!

[Kohala Jim]

On Monday, July 27, 2020 at 7:39:01 PM UTC-6, Dacian Todea wrote:

 Also data transfer over serial will take minutes depending on speed and decompressed data like csv will take about 2.5 to 3x longer.

 
Not a huge price to pay if someone is dumping this infrequently, and this data would be consumption ready

[David Burton]

I know I should add some more detail on the install and configure, I'm planning on creating a simple script to install and configure all software and and DB setup, this would reduce the setup to simply running a single script and then logging into the Grafana website hosted on a Pi.  I just added my current Grafana dashboard to my GitHub repo if anyone wants it.  I change my own dashboard regularlyas I'm often interested in different details.  Like last week I was doing capacity tests on new cells and wanted to keep an eye on cell delta as I was discharging them (and who wants to sit and watch).

Anyway, the dashboard JSON file is here: https://github.com/Burtond/Electrodacus

Next steps for my scripts and DB configuration is for my second SBMS.  I'll complete my design when I have the 48V system done for my whole house.

[Jim A]

Joe,  I am also trying to use David Burton's process using a Raspberry PI 3B+.  I believe I have all the software set up correctly, but I'm not sure about the USB connection for data transfer.  How did you set up up your UART parameters?

thx,  jim

[Jim A]

Dacian, On the SBMS120 can the USB connection be used for data transfer?

[Dacian Todea]

Jim,

Only the SBMS0 has USB with isolated data output.  The other models including the SBMS120 will require you to use a isolator on the TX line available on the 16pin connector and connect that to the Pi serial port (You need to add an opto isolator that is fast enough or a digital isolator else you risk damaging the SBMS120 and or the Pi).

[Jim A]

Maybe something as simple as this?

https://www.digikey.com/product-detail/en/sparkfun-electronics/BOB-09118/1568-1279-ND/5762429?utm_adgroup=General&utm_source=google&utm_medium=cpc&utm_campaign=Dynamic%20Search_RLSA_Cart&utm_term=&utm_content=General&gclid=Cj0KCQjwpZT5BRCdARIsAGEX0zmZPns3aeuDJChIIq9IJWUwekvlI58ta9QCqq6sgMcpPvJq2fOuStwaAmxMEALw_wcB

[Dacian Todea]

Jim,

Yes that should work but probably just at 9600bps so you need to set the speed on the SBMS to the lowest level 9.6kbps

[David Burton]

Well grr, 

Apparently I forgot to hit post on a very detailed description of some updates I have made and my session timed out and I lost it...very sad.  Here was the gist of it though.

I was able to streamline the Raspberry Pi configuration with a script that automated nearly all configurations.  There are some initial steps and final steps that need to be done, but I detailed them all in a post here: https://familyburton.com/solar-battery-testing-with-electrodacus-part-2/

Here are the highlights:

- Setup a new Raspberry Pi with Raspian Lite (>= 4Gb micro SD card), steps in the URL

- Run a few commands (they pull a python script from my github repo, https://github.com/Burtond/Electrodacus)

- Import my shared dashboard, also in the GitHub repo, but I highly recommend customizing as you can move graphs around and make it just how you like it :)

*** Don't forget to enable USART logging on your SBMS0, details in the manual, https://electrodacus.com/SBMS0/SBMS0.pdf

I was able to test all of the steps on a brand new Raspberry Pi with a 32GB micro SD card, confirmed 2GB was too small, but anything bigger should be fine.

[Barry Timm]

Many thanks, David. This is exceptionally nice of you to share. I will be ordering a Raspberry Pi as per your info, and give it a try.

Barry

[Gordon Bland]

Hi David, will a Raspbery Pi 4 2gb version be suitable? Or do you need the 4gb version as well as a large SD card? Thanks, Gordon.

[Jim A]

Hi David,  First off, thanks for all your time and effort put into helping us get up to speed on this.  I was really struggling with the first process you posted, but the recent automated process did the trick and worked seamlessly.  I'm very close but hung up on setting the data source.  I get the error message that it isn't receiving data... "network error: bad gateway (502)".  I'm using a SBMS0, the USB cable is connected, UART is enabled, and WIFI is disabled.  Could you help me sort through this?   thx,  jim

[Jim A]

Update:  With the help of Google and figuring out how to set up the Grafana Data Source, I found the problem.  The URL needs to be set at the IP address of the Pi.  It looked like to was set (at LocalHost), but needs to be changed. All is working well now.  Thanks again for posting this and setting up the automated setup!    jim

[David Burton]

I believe the Pi 4 would be just fine, but the micro SD card does need to be bigger than 2GB for the tools to all fit (at least that was my experience).