Electronic formats for SLM Data
sean_a...@hotmail.com
for many years. I note
that every SLM manufacturer whose equipment I have used has their own
propietary PC software.
This software will only download data from their own brand instruments
and serialise it to the PC
in their own propietary format. If one wishes to manipulate the data
outside this software (such as in a spreadsheet or database) the
propietary software may or may not be of assistance.
I have (so far) always found a way of doing it but often only due to my
previous experience.
Obviously the acoustics world does not need to be like this and
manufacturers could agree on
common electronic formats for data storage and then implement them, XML
would be an obvious choice for creating a format due to its
universality and extensibility. I think much noise data would lend
itself to XML which likes data to be stored in a structured
hierarchical manner.
Are there any initiatives within the industry to agree common
electronic formats for noise data?
Angelo Campanella
> Obviously the acoustics world does not need to be like this and
> manufacturers could agree on
> common electronic formats for data storage and then implement them, XML
> would be an obvious choice for creating a format due to its
> universality and extensibility. I think much noise data would lend
> itself to XML which likes data to be stored in a structured
> hierarchical manner.
The problem is that the SLM manufacturers have been doing this for over
20 years. What you see now is what they have all managed to get working
in their instruments and into the marketplace likely long before you
started in the work force (I think). That you now have a vista that can
comprehend their results in retorspect, is not surprising that you can
find the commonality and propose a codification. My major SLM was mature
before 1990; I still use it almost daily. I suspect that there
unfortunately is no funds available to develop to a standard i this
field. Rather, what I see happening is that each manufacturer includes
software with their hardware that exports the hard data (after being
DSP'd by their old tried and true firmware) in Excel form.
> Are there any initiatives within the industry to agree common
> electronic formats for noise data?
The acoustical field is far away from developing any such standard. You
are welcome to try, but you will have to beat your oun drum and blaze
your own trials in doing so. No one will stop you from doing it, if you
can. (There is vastly more enthusiasm for music and entertainment
standards; .wav and .mp3 and the like) tha here will ever be for
technical acoustics.
All above are my opinions...
Angelo Campanella
Herb Singleton
sean_a...@hotmail.com wrote:
> Obviously the acoustics world does not need to be like this and
> manufacturers could agree on
> common electronic formats for data storage and then implement them, XML
> would be an obvious choice for creating a format due to its
> universality and extensibility. I think much noise data would lend
> itself to XML which likes data to be stored in a structured
> hierarchical manner.
This is something I tried to do back in 2000 (see
<http://www.cross-spectrum.com/audio/articles/acoustic_xml.html> for
example) - I was inspired after reading an article in Sound & Vibration
magazine about various file formats used in acoustical analyses.
I agree that dealing with the various formats is a big pain - especially
since most acoustical consultants use a variety of manufacturers'
products, and if you do any kind of custom data analysis, you have to
deal with all the formats (heck, even if the products come from the same
manufacturers, you still have to deal with messy file formats - just
look at Larson-Davis products).
The biggest problem is that manufacturers just aren't interested - their
are some good reasons (memory constraints for example) and bad reasons
(lock-in), but everyone I talked with couldn't care less universality or
interoperability. Sad but true.
> Are there any initiatives within the industry to agree common
> electronic formats for noise data?
One effort is being spearheaded by IEST, see:
<http://www.sandv.com/downloads/0401alle.pdf>
I inquired about joining the working group, and the rep that I emailed
was receptive, but I got too busy to follow up.
There was one other effort I stumbled across a few years ago, but I
can't find the info right now - but I do remember that is was very
complex, so I didn't pay it much mind.
Herb
Angelo Campanella
> I agree that dealing with the various formats is a big pain - especially
> since most acoustical consultants use a variety of manufacturers'
> products, and if you do any kind of custom data analysis, you have to
> deal with all the formats (heck, even if the products come from the same
> manufacturers, you still have to deal with messy file formats - just
> look at Larson-Davis products).
Yes, I have; for over 20 years, and I simply live with it.
B&K essentially mortgaged the family farm in the 1980's to do this very
thing, and they lost that farm. They have not been the same
(financially) since.
Codifying acoustical data would be like codifying tire sizes for your
auto. While I was growing up, both the family auto and the light
delivery truck I drove used 6.00-16 tires. Then came the 15" and 14"
balloons, then the soft radials, and now we have 16" low profilers; all
for the same job; where the rubber meets the road. Codification? Heck, NO.
> The biggest problem is that manufacturers just aren't interested - their
> are some good reasons (memory constraints for example) and bad reasons
> (lock-in), but everyone I talked with couldn't care less universality or
> interoperability. Sad but true.
I do see a little codification: Most instruments now come with brief
software that will Dump ("Export") their instrument's results into an
EXCEL spreadsheet. To that extent (Excel) it's codified.
But I have to say that even I do not like that result. I prefer to use
their Routine (L-D's "TRANS.EXE") that exports a plain ascii file of
data in a visually recognizable format that I then condense and massage
with an ascii editor like EDIT or Notebook (either works fine) to
produce a file that I splice into an Excel spreadsheet for rendering
into report pages and graphs.
I don't think we can refine (codify) the process any more than that. We
professionals are obliged to do some work, and I do not complain.
For real time work, tighter connection to the instrument is desirable.
I have done that, and LD has cooperated as needed.
> One effort is being spearheaded by IEST, see:
> <http://www.sandv.com/downloads/0401alle.pdf>
>
> I inquired about joining the working group, and the rep that I emailed
> was receptive, but I got too busy to follow up.
I would not waste my time on it. My experience is that I work with what
I have; I develop a routine "that gets me there from here", then get on
with my life.
Angelo Campanella
Ken Plotkin
<a.camp...@att.net> wrote:
[snip]
> Codifying acoustical data would be like codifying tire sizes for your
>auto. While I was growing up, both the family auto and the light
>delivery truck I drove used 6.00-16 tires. Then came the 15" and 14"
>balloons, then the soft radials, and now we have 16" low profilers; all
>for the same job; where the rubber meets the road. Codification? Heck, NO.
[snip]
That's not a good example. Tire sizes are more complex than they used
to be, but the nomenclature has kept up. There are uniform codes for
aspect ratios, radials, speed rating, etc.
I share everyone's pessimism about there ever being a standard format.
It's not in the instrument manufacturers interest to do so. Nor in
the interest of some consultants. A few years ago, when we were
involved with noise in national parks, one of our guys worked out a
fairly straightforward device-independent data format.
Despite the customer (the Park Service) liking it, other consultants
working on the same projects proceded to unilaterally mung the format.
Essentailly, a uniform data format did not suit their competitive
needs.
Incidentally, the suggestion by the OP to base a data format on XML
seems to me to be a really bad idea.
Ken Plotkin
Herb Singleton
Ken Plotkin <kplo...@nospam-cox.net> wrote:
> Incidentally, the suggestion by the OP to base a data format on XML
> seems to me to be a really bad idea.
Why do you think so?
I know that XML has some drawbacks - file size for one (which is why I
don't advocate XML for internal device use, but I like it as a target
for ASCII translated data).
However, one thing that XML has in it's favor (IMHO) is that it's so
universally supported nowadays via free/pay toolkits, databases, OS's
and now MS Office, it's really easy to integrate XML data into various
workflows.
Herb
Chris Whealy
> Why do you think so?
>
> I know that XML has some drawbacks - file size for one (which is why I
> don't advocate XML for internal device use, but I like it as a target
> for ASCII translated data).
I know XML has become the de facto standard for data exchange, but this
does not make it the best solution. Lets face it, in recent years, the
browser has become the standard user interface client, but it is a
massive step backwards in terms of its graphical capability. Browsers
still rank way down the list of what is possible from a graphical user
interface.
> However, one thing that XML has in it's favor (IMHO) is that it's so
> universally supported nowadays via free/pay toolkits, databases, OS's
> and now MS Office, it's really easy to integrate XML data into various
> workflows.
Please don't think that just because something is popular and
convenient, that it is therefore the best tool for the job. Yes, XML
parsers are ten a penny, and yes implementation is very straight
forward, and yes they are widely supported; but there are far more
efficient ways of communicating a message between two system without
having to go through the format/parse cycle every time.
XML is horribly verbose; consequently, it has a very poor signal to
noise ratio. Non ASCII formats are far more efficient in this respect.
Also, because XML is rigidly hierarchical, you have to parse the entire
document (irrespective of size) before you know whether the document is
required or not. There is no possibility to just "read the header" to
find whether or not you should bother reading the rest of the document.
That fact means that in a high load situation, XML parsing becomes an
inescapable overhead.
The parsing process itself is relatively efficient, but the biggest
problem comes from the fact (particularly in Java), that there is no
effective mechanism for caching parsed objects. Further to this, Java
programmers are notoriously bad at considering the memory usage of their
programs. This leads to XML parsers being written that work fine for a
few large documents, but when they are placed under heavy load, the
complex inter-relationship that frequently exists between objects in the
parsed tree can lead to the Java garbage collector's runtime rising
quadratically with the heap size (Ouch!)
I have to use XML frequently in my job, but I must say from experience,
that its popularity far exceeds its utility; particularly when a high
volume of large documents need to be processed.
Chris W
--
The voice of ignorance speaks loud and long,
But the words of the wise are quiet and few.
---
Ken Plotkin
<moc...@ylaehw.sirhc> wrote:
>I know XML has become the de facto standard for data exchange, but this
>does not make it the best solution. Lets face it, in recent years, the
[snip]
Thanks for the great answer.
I was going pretty much on gut feel and experience with various
numeric data formats. Generally, the simpler the better. I've had a
much better time dealing with unadorned things like TechPlot format,
rather than supposed universal formats like NedCDF. Anyone who uses
USGS geographic data, and does not want to pay tribute to ESRI,
probably has great appreciation for the classic USGS file formats and
has absolute hatred for SDTS transfers.
Ken Plotkin
Herb Singleton
Chris Whealy <moc...@ylaehw.sirhc> wrote:
> Please don't think that just because something is popular and
> convenient, that it is therefore the best tool for the job.
Whoa there.... as a point of clarification, I never said (and I hope I
didn't imply) that XML was the "best" - and furthermore I don't believe
the popularity is any indication of quality (let's put it this way: I'm
a Mac user ;).
[snip, good arguments that XML processing is inefficient]
I agree that processing XML can be inefficient, but I see it as a
trade-off between efficiency and interoperability. I would argue that
most of the arguments you give against XML (particularly the ASCII-based
format and the hierarchical structure) are *good* things in that it
helps to maintain (and enforce) the structure of the data. I also like
ASCII-based file formats because it's much easier (IME) to recover
corrupt files. Like I wrote before, the large file size makes ASCII
unworkable for local device storage, but it's useful for data
interchange.
But for me, the issue comes down to being able to work with the data in
as many ways as possible on any platform that I chose -
interoperability. I've worked with XML data on Windows, Mac, Palm, and a
cell phone, and it was easy to do because there are free parsers for
each of those platforms (and trust me, I'm no programmer). I'm sure
there are more efficient ways to work with data, but can it be done with
widely available (and preferably free) toolkits, or do I have to
manually work with the bits at each platform? Serious question, if there
is something like this out there, I'd like to experiment with it.
> I have to use XML frequently in my job, but I must say from experience,
> that its popularity far exceeds its utility; particularly when a high
> volume of large documents need to be processed.
I don't know whether I should envy you or pity you ;)
FWIW, I use XML (and Expat) in a couple of custom programs, but with
relatively small sets of data. It has it's drawbacks, but again, the
wide availability of parsers makes it easy for me to work with the data
in different ways on different platforms. That's the part I *really*
like.
Herb
Herb Singleton
but the journal Nature just published an editorial about sharing
electronic data over the internet:
"A key technological shift that could change this is a move away from
centralized databases to what are known as 'web services'. These are
published interfaces that serve to simplify access to data and software
(for an example of such services in action, see
http://www.ebi.ac.uk/xembl/index.html). Until recently the preserve of
expert programmers, such interfaces now mean that anyone with even a
basic knowledge of programming can automate data processing and
analysis."
[...]
"In biodiversity research, for example, rather than creating centralized
monolithic databases, scientists could tap into existing databases
wherever the data are held, weaving together all the relevant data on a
species, from its taxonomy and genetic sequence to its geographical
distribution. Such decentralization also helps to solve the problem that
databases are often the fruits of individual or lab research projects
that are vulnerable to the vagaries of funding, and to people and labs
moving on to pastures new."
[...]
"Scientists may be justified in retaining privileged access to data that
they have invested heavily in collecting, pending publication ‹ but
there are also huge amounts of data that do not need to be kept behind
walls. And few organizations seem to be aware that by making their data
available under a Creative Commons licence (see
http://creativecommons.org/license), they can stipulate both rights and
credits for the reuse of data, while allowing its uninterrupted access
by machines."
Herb
Angelo Campanella
Herb Singleton wrote:
> "A key technological shift that could change this is a move away from
> centralized databases to what are known as 'web services'. These are
> published interfaces that serve to simplify access to data and software
> (for an example of such services in action, see
> http://www.ebi.ac.uk/xembl/index.html). Until recently the preserve of
> expert programmers, such interfaces now mean that anyone with even a
> basic knowledge of programming can automate data processing and
> analysis."
Socialism (enforced pooling) didin't work for societies, and it won't
work in the long run for data and knowledge.
That knowledge that the owner, in his or her wisdom should be shared,
should be.
That knowledge and data that are either trivial-local, or doubtful, or
critical to one cause or another should not be shared, as it causes too
much confusion and animosity.
Just my opinions.
Angelo Campanella