Open materials database?
Smári McCarthy
Hash: SHA1
Hi all,
Does anybody have any free('dom' & ' beer') materials databases in
computerised form lying around? Anything goes, just need a real example
of a materials dataset.
- Smári
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkqsICIACgkQ9cJSn8kDvvHDUwCg7OO+8NbSrDYudR+VjtoHzwaj
0JYAn28Z82HIoY7BpUKm/wTICePPZdd9
=pkBx
-----END PGP SIGNATURE-----
Bryan Bishop
> Does anybody have any free('dom' & ' beer') materials databases in
> computerised form lying around? Anything goes, just need a real example
> of a materials dataset.
No. However, I have many unfree data sets if you're interested in just
looking at examples.
Paul D. Fernhout
> Does anybody have any free('dom' & ' beer') materials databases in
> computerised form lying around? Anything goes, just need a real example
> of a materials dataset.
Well, OSCOMAK has some examples put in by Mike Harris:
http://www.oscomak.net/wiki/Category:Raw_Materials
Example:
http://www.oscomak.net/wiki/Solar_Grade_Silicon
http://www.oscomak.net/wiki/Silica
Some of that was draw from Wikipedia. We had some related discussions on the
OpenVirgle list about sources and licensing last year.
http://groups.google.com/group/openvirgle
http://groups.google.com/group/openvirgle/msg/b8628fa29049a623?hl=en
http://groups.google.com/groups/search?hl=en&q=openvirgle+materials+licensing
Still, while Mike did a great job getting things started, OSCOMAK is
undoubtedly not as comprehensive and detailed as you would like. It's just a
few entries that serve more as a proof-of-concept at this point, and
something to think about.
By the way, I hereby take this moment to redeclare that OSCOMAK content as
also under the CC-BY-SA 3.0 Unported license,
http://creativecommons.org/licenses/by-sa/3.0/
as made possible under the terms of the new GFDL 1.3 license, to remain
compatible both ways with Wikipedia. :-)
You should probably ask the open materials people for something better.
http://openmaterials.org/
There are some large sources of materials data sheets around the web, but
they tend to be proprietary in some way.
--Paul Fernhout
http://www.pdfernhout.net/
ben lipkowitz
> Smári McCarthy wrote:
>> Does anybody have any free('dom' & ' beer') materials databases in
>> computerised form lying around? Anything goes, just need a real example
>> of a materials dataset.
> You should probably ask the open materials people for something better.
> http://openmaterials.org/
i recall asking several months ago and getting no response:
http://groups.google.com/group/openmanufacturing/browse_thread/thread/92b7cb596061d785/8e0ab0cfba7baaf9#8e0ab0cfba7baaf9
I'm afraid that literally all material properties databases may be under
lock and key. However, it might be possible to simply rewrite the data in
a different format to circumvent copyright, assuming someone has access to
a database in the first place.
wikipedia contains some materials properties data copied from various
proprietary databases, i.e. http://en.wikipedia.org/wiki/Maraging_steel
but it is scattered, and not stanardized; you'd have to dig for it.
here's a nice list of possible database fields:
http://en.wikipedia.org/wiki/List_of_materials_properties
coefficient of friction is absent from the list, probably because it is a
function of two materials. I'm not sure how to organize data like that.
Good Luck
-fenn
PS: on an unrelated note, google groups searching doesn't seem to give any
results older than a month or so? wah!
Smári McCarthy
Hash: SHA1
need changing.
I'm currently working on a logical query language for this kind of
thing, and have been collaborating with several materials scientists
from Kenya, India, England and Iceland to launch an effort to catalogue
the world's materials in a freely available database. Hopefully we can
start to see results within the month.
As for querying; different materials will have different properties
registered. For concrete, for example, I doubt shear strength is a
commonly measured property, and while some materials might register
Brinnel hardness others may have Mohs hardness tagged (eventually it
would be good to find a conversion formula for that kind of thing)... so
what I'm working with is a list of materials which has a series of
tuples. How this goes into the database is a simple issue, but for
simplicity's sake I've made a language which supports queries such as:
materials ? 400 < thermalcapacity < 600 & meltingpoint > 400
It ain't SQL, it doesn't support joins and funky relational algebra, but
it's *sensible*, which is a nice change.
Interestingly, this language might make sense for objects too:
objects ? type ~ "lamp" & width < 20 & height < 200
I think adding units would make a lot of things easier... arr.
It's all in the git archive.
- Smári
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
1GUAn2UD6A9DT6Gs8FpebvVNkTIF4KbO
=WnJM
-----END PGP SIGNATURE-----
Bryan Bishop
> Okay. This is kind of what I expected. It sucks, but that just means it
> need changing.
Have you accessed other databases to see how they do it, even the
proprietary ones? There are certain lessons that can be learned from
the hundred-plus year old databases.
> I'm currently working on a logical query language for this kind of
> thing, and have been collaborating with several materials scientists
> from Kenya, India, England and Iceland to launch an effort to catalogue
> the world's materials in a freely available database. Hopefully we can
Why do you need a query language for material data representation?
> It's all in the git archive.
Hate to break it to you, but you haven't actually committed any
changes in many, many months. In fact, ever since the merge. Did you
check if it's actually there ? Maybe I missed it.
Paul D. Fernhout
> Okay. This is kind of what I expected. It sucks, but that just means it
> need changing.
>
> I'm currently working on a logical query language for this kind of
> thing, and have been collaborating with several materials scientists
> from Kenya, India, England and Iceland to launch an effort to catalogue
> the world's materials in a freely available database. Hopefully we can
> start to see results within the month.
Smári-
I'd suggest this is a very good topic to bring up with lawyer like at the
Center for the Study of the Public Domain at Duke.
http://www.law.duke.edu/cspd/
They could give you and the rest of your team good formal guidance on how
much you could draw from web available databases that claim to be
proprietary (given that scientific facts themselves supposedly usually can't
be copyrighted, only presentations of the facts, at least, as I understand
this). This might save you time and other troubles. I would think this is
*exactly* the sort of thing that the people at Duke would want to help with.
And if they can't help, then there are other places that might
(Berkeley/BOALT, Wikimedia foundation, FSF, Eben Moglen, etc.)
http://www.boalt.org/
http://en.wikipedia.org/wiki/Wikimedia_Foundation
http://www.fsf.org/
http://moglen.law.columbia.edu/
Anyway, what you are working on is a fantastic project, and it could really
move everything forward if it was comprehensive and definative.
I'd even suggest that maybe adding the data to Wikipedia in some machine
readable form might be one way to go, where everyone has easy access and
then the data could be slurped down into other formats? Although, here is a
case where one can wonder about how definitive Wikipedia would be as a
reference if one needed reliable figures on, say, melting points, if anyone
could edit them?
--Paul Fernhout
http://www.pdfernhout.net/
Smári McCarthy
Hash: SHA1
Hi,
Bryan Bishop wrote:
> Have you accessed other databases to see how they do it, even the
> proprietary ones? There are certain lessons that can be learned from
> the hundred-plus year old databases.
Yup. They generally aren't very well architectured as such (which is my
experience from working with old databases - the oldest database I've
worked with on a professional basis was created in 1703), and generally
either use the "loads of fields, many of them blank" model, or use
arbitrarily complex hashes.
> Why do you need a query language for material data representation?
I don't. SQL could suffice. However, SQL isn't always the best thing -
relational databases don't always make sense.
http://www.defmacro.org/ramblings/relational.html
Viewing materials as points in space and making a basic query language
to put arbitrary constraints on the space and look what's in it can be
lots lots faster than using relational databases, plus much more
sensible in management terms.
> Hate to break it to you, but you haven't actually committed any
> changes in many, many months. In fact, ever since the merge. Did you
> check if it's actually there ? Maybe I missed it.
I hate to break it to you, but the day after the merge I was met with
such arrogance that I decided to just keep on doing my own thing. Not
only was I bossed about, but I was accused of not knowing what the
project was about. When working on things out of passion I don't
appreciate that kind of attitude, and rather than argue about it I will
simply up and leave. I kind of assumed you would have figured that out
by now.
It was me who pushed for the merge originally, but in retrospect it was
a poor move, as although SKDB and TangibleBit are ostensibly two halves
of the same coin, we have very different notions about how to work on
projects. But please know that there are no hard feelings at all about
this, and I fully expect there will be a point in the future where SKDB
and TangibleBit will naturally align.
(Oh, and btw, it's not been "many, many months" since we tried the
merge. Don't exaggerate. It's been two months, barely, and for most of
that time I've been traveling around the world gathering information and
getting people involved. Thanks for providing such a great example of
the aforementioned arrogance though. :))
- Smári
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkqtnREACgkQ9cJSn8kDvvE8DwCcC4m2Qeqi5RAQZTOMsYO1vIN7
KFcAoKD8NkxGzakG/ecGRM+4E5oHBEVX
=eD/B
-----END PGP SIGNATURE-----
Smári McCarthy
Hash: SHA1
Thanks for that great suggestion Paul, I'll get on it right away. Those
Duke people seem to be the right people to talk to. I have been meaning
to mail Eben Moglen about this (I talk with him often on a variety of
matters) but haven't gotten around to it. I literally just got back from
my India/Afghanistan trip last Thursday, and am getting back into
"normal life" mode. :)
Cheers!
- Smári
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkqtng0ACgkQ9cJSn8kDvvG01gCeNqCM7fnbZjzE/nkV5dvLe91d
incAmQFnbE6yq/6pjboWmS2ORWOeS24B
=yVaf
-----END PGP SIGNATURE-----
Bryan Bishop
>> Hate to break it to you, but you haven't actually committed any
>> changes in many, many months. In fact, ever since the merge. Did you
>> check if it's actually there ? Maybe I missed it.
>
> I hate to break it to you, but the day after the merge I was met with
> such arrogance that I decided to just keep on doing my own thing. Not
None of the other thirty observers think that happened, so I am not
quite alone when I ask wtf? What are you talking about? I think this
was about the time when you were asking what was in some of the files
in the repository.
> only was I bossed about, but I was accused of not knowing what the
> project was about. When working on things out of passion I don't
Well, I did commit some code to the repository under your name, but
maybe that was too bossy, which I'm prepared to apologize for.
> appreciate that kind of attitude, and rather than argue about it I will
> simply up and leave. I kind of assumed you would have figured that out
> by now.
Not at all. This is all very surprising to me, Smari.
> It was me who pushed for the merge originally, but in retrospect it was
> a poor move, as although SKDB and TangibleBit are ostensibly two halves
> of the same coin, we have very different notions about how to work on
> projects. But please know that there are no hard feelings at all about
> this, and I fully expect there will be a point in the future where SKDB
> and TangibleBit will naturally align.
What are those different ideas? You seem to know the tools and how to
commit code, you seem to write good code, I really don't see the
problem.
> (Oh, and btw, it's not been "many, many months" since we tried the
> merge. Don't exaggerate. It's been two months, barely, and for most of
> that time I've been traveling around the world gathering information and
> getting people involved. Thanks for providing such a great example of
> the aforementioned arrogance though. :))
Ouch, Smari. Where is this attitude coming from? So I have a warped
perception of time, spend 14 hours a day writing code, two months to
me is a very long time.. to others it may be a drop in the bucket.
Paul D. Fernhout
> As for querying; different materials will have different properties
> registered. For concrete, for example, I doubt shear strength is a
> commonly measured property, and while some materials might register
> Brinnel hardness others may have Mohs hardness tagged (eventually it
> would be good to find a conversion formula for that kind of thing)... so
> what I'm working with is a list of materials which has a series of
> tuples. How this goes into the database is a simple issue, but for
> simplicity's sake I've made a language which supports queries such as:
>
>
> materials ? 400 < thermalcapacity < 600 & meltingpoint > 400
>
>
> It ain't SQL, it doesn't support joins and funky relational algebra, but
> it's *sensible*, which is a nice change.
>
> Interestingly, this language might make sense for objects too:
>
> objects ? type ~ "lamp" & width < 20 & height < 200
>
> I think adding units would make a lot of things easier... arr.
> ... SQL could suffice. However, SQL isn't always the best thing -
> relational databases don't always make sense.
>
> http://www.defmacro.org/ramblings/relational.html
>
> Viewing materials as points in space and making a basic query language
> to put arbitrary constraints on the space and look what's in it can be
> lots lots faster than using relational databases, plus much more
> sensible in management terms.
The sorts of issues raised in that defmacro essay raises about transaction
identifiers, undoing stuff, sparse data storage, and other AI-type isuses
are why I've spent about thirty years working on the Pointrel system
on-and-off. :-)
http://sourceforge.net/projects/pointrel/
Here is an example of use that which addresses some of those issues raised
in the defmacro comment on "Limitations of Traditional Model" (this version
is in Java):
http://pointrel.svn.sourceforge.net/viewvc/pointrel/trunk/Pointrel20090201/source/org/pointrel/pointrel20090201/test/TestSimpleUseage.java?revision=413&view=markup
Example (building a todo list):
"""
String newItem1 = "Play with the code";
Triple triple1 = session.addTripleForFields(contextTypeToUse, "test", "", \
"todolist", "", "entry", "", newItem1);
...
Iterator<Triple> iterator
session.iteratorForMatchingTriplesSortedByTimestamp(contextTypeToUse, \
"test", null, "todolist", null, "entry", null, null);
while (iterator.hasNext()) {
Triple triple = iterator.next();
System.out.println("### To do: " + triple.valueData);
}
"""
The transactions were put here:
http://pointrel.svn.sourceforge.net/viewvc/pointrel/trunk/Pointrel20090201/TestArchives/TestArchiveExamples/
Example (similar to the above, but not identical):
http://pointrel.svn.sourceforge.net/viewvc/pointrel/trunk/Pointrel20090201/TestArchives/TestArchiveExamples/PRF_01088d2cab39031d976ecaf8e3bd2d2186a78101660caf572f8f756d15a2a685_794.pointrel?revision=413&view=markup
"""
1 PointrelTransaction
2 version: 20090201.0.1
3 uuid: b4afe17b-42ae-43fb-bd33-a012b909d40a
4
5 Triple
6 uuid: c58349e4-b2ea-4ad2-b677-3c189307677c
7 timestamp: 2009-03-01T01:32:24.000Z
8 contextType: PointrelTransactionMetadata
9 contextData: b4afe17b-42ae-43fb-bd33-a012b909d40a
10 entityType: PointrelTransaction
11 entityData: b4afe17b-42ae-43fb-bd33-a012b909d40a
12 attributeType: PointrelTransactionMetadata
13 attributeData: timestamp
14 valueType: ISO8601Timestamp
15 valueData: 2009-03-01T01:32:24.000Z
16
17 Triple
18 uuid: 529eef47-7661-4546-9592-573854fd6aa3
19 timestamp: 2009-03-01T01:32:24.000Z
20 contextType:
org.pointrel.pointrel20090201.examples.ExtremelySimpleToDoList
21 contextData: test
22 entityType:
23 entityData: todolist
24 attributeType:
25 attributeData: entry
26 valueType:
27 valueData: improve database reloading
28
29 EndTransaction
"""
(The line numbers are not part of the data file.)
I also checked a couple earlier version into the OpenVirgle project that
emphasize different things:
http://code.google.com/p/openvirgle/
http://code.google.com/p/openvirgle/source/browse/#svn/trunk
Of course, my theory of querying is to just write the lookup function in
Java or Jython... :-) One could adapt one of the many query language
frontends to work with this backend otherwise, but that is to-be-done. But
you might adapt your system to it if you have working code that can work
with Java or Jython?
Still, it is a work in progress. So rough edges. Maybe gotchas. Poor
documentation. Bugs. A maintainer who might drop it for a "real job". Maybe
some big missing pieces or huge misconceptions on my part. And so on. :-)
The first version of OSCOMAK on the web was written using a version of the
Pointrel system, until I replaced it with a Halo Semantic MediaWiki (in
retrospect foolishly as far as my own motivation to improve the system).
The old code is here:
http://code.google.com/p/openvirgle/source/browse/#svn/trunk/PointrelExperimentOnOscomakCGI
And the data had a couple of materials added to it by Mike Harris:
http://code.google.com/p/openvirgle/source/browse/trunk/PointrelExperimentOnOscomakCGI/cgi-bin/.hiddenG95ht/archive_default.pointrel_database.xml
Example data:
"""
<P:transaction>
<P:triad
r="17168L"
s="test"
a="zirconia"
b="chemical_formula"
c="ZrO"
/>
</P:transaction>
<P:transaction>
<P:triad
r="17568L"
s="test"
a="electrolytes"
b="types"
c="zirconia"
/>
"""
You can see I've used all sorts of formats for the backend. :-) That one is
XML, the previous one was custom but UTF-8 text.
That CGI system was thrown together in a day or so with no security. :-)
It just happily took any input to create web pages that would store and
retrieve stuff from the Pointrel system. You can see an example of such a
page with a query at the end of the file at the last link. One could add
security like requiring a password to submit data, of course.
There are several versions of the Pointrel system implemented in different
languages with different data formats emphasizing different things. The
common aspect is a sparse representation of data using triples (triads)
usually with a context field, often with reified identifiers for triples and
transactions, with searches done by looking for matching content in the triple.
A simple example from first the web system (which uses a version of the
Pointrel system that is less complex/sophisticated than the one in Java above):
"""
from pointrel20030812 import *
electrolytes = Pointrel_allMatches("test", "electrolytes", "types", WILD)
print "Which types of electrolytes have the formula CeO?"
for electrolyte in electrolytes:
print electrolyte, Pointrel_lastMatch("test", electrolyte, \
"chemical_formula", WILD) == "CeO"
"""
But, the essentially all do pretty much the same thing. Though I'd recommend
looking at the lastest Java version if you were going to try it, even though
it is a little more complex. It has better support for synchronizing data
across several different types of backends (shared directories, cgi scripts,
ftp servers, some others), so it is designed to support peer communications
mediated usually through a relay server.
Essentially, the Pointrel System might be best understood as a database that
operates a lot like a wiki or mailing list. Like those, it has no built in
security, and anyone can post anything to it (who can access the server) and
can mess anything up. But, then anyone else can put things right. It's a
different vision of data storage than databases that came out of the need to
keep extremely accurate track of imaginary money. :-)
Somehow I think we've had this conversation before. :-)
http://groups.google.com/group/openmanufacturing/browse_thread/thread/5a6e78b9ac99d851/cb2421236bf9aa0b?hl=en&q=openmanufacturing+pointrel+smari#cb2421236bf9aa0b
http://groups.google.com/group/openmanufacturing/browse_thread/thread/3bd36851629847f0/268ecfe2b5c1610c?hl=en&q=openmanufacturing+pointrel+smari#268ecfe2b5c1610c
But, since I'm not using the system for anything mission critical now, I
can't expect others to. I keep thinking of switching my email over to it :-)
but I haven't.
--Paul Fernhout
http://www.pdfernhout.net/