Why Dart does not care about data?

[GT]

Why every new programming language focus on classes? C++ give us
classes, Java again give us the classes, J++ give us the classes, C#
give us the classes, Pascal give us the classes, and it looks like
everyone in the past 20 years is giving us the classes over and over
and over, but what about data?

Beside C#'s LINQ, no one else is trying to give us a better way to
work with data!

All what companies have been doing the past decades is to make sure
programming is structured and limited, you forgot a semicolon, too bad
compile error, it is all about how to limit the code and developer,
how can we make a class private, and then how can we go around this
with Reflection (C#), just a pointless loop of useless ideas!

Structure is nice, but data is the most important! Look at any
application out there, a graphics application for example needs some
kind of image data type that is partially in memory and mainly on the
hard drive, a data type that will allow you to easily modify that
image, or undo your steps, etc. all with the optimum memory, CPU and
disk usage.

A business application for example needs an easy way to structure the
customer information in memory, and embed the customer transactions
inside the customer object, store many of them in tables and sub
tables or an JSON / XML like structure in memory, or the hard drive or
a hybrid model, query them, modify them, and so on.

The examples are endless, data is everywhere, except in Google Dart,
that give us classes again, thank you!

Note: please name things correctly, a list is a list, a map is
something I use on my GPS, it is not a list! I call my car a CAR, not
a moving banana, and I call my fridge a FRIDGE, not an ice cream
wardrobe.

[Quildreen Motta]

On 12/10/11 10:40, GT wrote:
> Note: please name things correctly, a list is a list, a map is
> something I use on my GPS, it is not a list! I call my car a CAR, not
> a moving banana, and I call my fridge a FRIDGE, not an ice cream
> wardrobe.

If we ignore the fact that my other car is a cdr, and unless I just
don't get what you're talking about, map is a well known and widely
spread concept in programming <
http://en.wikipedia.org/wiki/Map_(higher-order_function) >, and it has
other several meanings. Words are not a 1:1 mapping.

[George Tarazi]

SQL is even "veeerrrry" more well known than map.

A modern programming language must focus on Data, the idea is to help
a developer not to restrict the developer!

All programming languages in the past many years have been in the same
useless loop, someone implements classes = restrictions, such as
private members, you write a lot of code, and then you realize that
although a member is private, you must call it directly from the
outside for a reason or another, then another development group tries
to resolve that problem by introducing reflection, which is a
workaround all limitations introduced by classes, and the loop goes on
(C# as an example).

The biggest problem today is not structure, structure is very nice,
but the most argent thing to be solved is complex data.

But complex data is a complex task, and structuring code is the
easiest thing to do.

The problem is still there, it is pretty complex to develop a server
side / client side application with today's programming languages, yes
it is very easy to create hello world, and it takes months and years
to develop a full business application, not because the business rules
are complex, just because the programming languages use prehistoric
ideas.

Google DART "we are reintroducing the dinosaurs!", good Job :-)

[Josh Leverette]

If a programmer uses well structured data and code, he can manipulate
vast quantities of data very efficiently and without too many lines of
code. Just because you're searching for some imaginary language that
is artificially intelligent and knows "hey! This is a financial
spreadsheet, I should go ahead and let the business know that profit
is gross minus expenses! They didn't even need to tell me, I'm just so
smart!" does not mean one exists. It is up to the programmer to
manipulate and understand the data, the language just provides useful
tools towards that end.

[GT]

Excel sheets manage data, and they don't look imaginary to me :-)

SQL server manages data, and it looks pretty real, Oracle as well

The SQL language is there and manages data, Microsoft LINQ to SQL
manages data in the code and it looks very real, we use it in
development :-) :-) :-) :-) :-)

I want to use well structured data, I even want more, I want to use a
complex very well structured data, and I want the programming language
syntax to be able to do so!

Instead, Dart introduced classes and interfaces! the same introduced
by C++ in 1983

[poltomb]

When did Excel spreadsheets and SQL servers become programming
languages?

[Adam Gray]

So what you're saying is that you'd really like something like Linq built in from the start or are you saying that google should be building a new type of datastore/database instead of attempting to provide a nice built-in alternative to javascript?  Your railing against object oriented programming doesn't seem to have any bearing whatsoever on what you're asking for and is making your request seem obtuse at best and trollish at worst.

[GT]

What I am trying to say is building applications, especially complex
business applications takes years, building games or graphics
applications takes years as well, building any application is taking
large amount of time, and if you happened to build such an
application, you would have noticed that most of the work is focused
on the data, loading it, saving it, caching it, searching it,
displaying it, editing it, researching it, querying it, etc, etc.
And if you happened to work at Google, and you have been given the
opportunity, budget, and resources to develop a new programming
language that can be used on the server and on the browser, then
maybe, just maybe, the first thing that you would focus on is how to
make it much simpler to work with data.

I am not against object oriented, but that thing was reintroduces so
many times in the past 30 years it just became prehistoric, JavaScript
5 is already object oriented.

[Luke Church]

Let me start of by saying that I like LINQ a lot, I use it fairly regularly, and it makes writing some types of business applications very nice.  

However, it's not without its own challenges. Every time an API is used to hide something like caching behavior from the user, it makes it easier to do the 'normal' thing, and harder to handle the edge cases, and often the error cases. When you're writing very widely deployed applications, the edges cases occur so often that you just have to deal with them.

Also, in my experience things like LINQ and SQL play nicely with nicely structured data. But life isn't usually that rosy, data formats change between versions, you have to do on the fly upgrades of applications whilst they're running, data gets dirty, constraints that you thought were valid turn out to trample on users' needs and have to be relaxed with breaking security. I have not had much success in getting LINQ - or indeed SQL like systems - to play with software like this. Protobuf on the other hand was designed out of just this world, as was BigTable.

So what we need really is a platform where we can play with both structured, semi-structured and down right messy data. The programming language has to live at this all embracing level. Optional typing provides an interesting set of tradeoffs for manipulating data like this, enabling more structure to be added when the structure is better node, and leaving it more flexible when it's not.

I think we can build good support for data integration on top of the platform, whereas we wouldn't have been able to do it the other way round. So, I think we now have a platform to build on. You say that you miss the features of LINQ, I hear you. Is there anything else that you would like to be able to do with data in your applications?

One of my interests has been that tooling has been hard for data processing languages, esp. in the face of messy data or data at scale. I think  Dart's approach may be a good place to start with dealing with that as well... But I'd be interested to hear your thoughts.

Best,

Luke

PS - You may be interested to see work being done in Oregon on adding type systems to spreadsheets in order to help fix their, somewhat notorious, error rates. (http://web.engr.oregonstate.edu/~erwig/UCheck/)

[GT]

More thoughts about data:

If you look at any business application, at the end when the
information is displayed to the end user, it is usually a complex data
structure, an example is a customer and his orders, or a customer and
his bank transactions, and other sub lists.

And if you look at the server side, it is always tables, and views on
these tables.

And in every business application, there is a thing that is repeating
over and over and over, you must hire many developers to load and save
the data from these tables, convert them to a complex structure, and
display them to the end user, take the user modifications and store
them to the database.

The server table structure does change by time, and the end user
complex data structure does change as well, and it is up to the
developers to find the best way to implement those changes.
This has been going for so many years, actually for decades, and
everyone knows that a full solution to the problem is to improve the
virtual machine, the programming language, the libraries and
algorithms that load and save data, and everyone is ignoring it.

Yes, by ignoring the problem we have created millions of software
development jobs, that do the same thing over and over and over,
plumbing things that could have been automated; and at the same time
is slowed down businesses who would have done much more if they had an
easy way to easily build powerful systems.

20 years ago, when I was reading about object oriented development, I
was all hope that in the next 10 or 15 years we will have mastered
calculations, statistics, and other algorithms, programming languages
will have full and integrated libraries for data, math, and
statistics.

And millions of software developers will focus for a cure of this
year's flue or find the best way to improve the human body,
nanotechnologies will be everywhere, your car will recognize you in
the same way a human recognizes another human, your house will know
everything about you, will take care of your health and will organize
itself, your garden will not only grow flowers but also grow many
other household items using programmed nano-organizams, and the list
goes on.

20 years later, millions of people are doing the same thing over and
over, mapping tables to user interface, and writing hundreds of
thousands of lines of code over and over for the same thing, all of
this over many years.

And Google comes and says, hey, we have object oriented programming
again, wow, Good JOB. At this speed we are all hope that after another
20 years they reintroduce SQL or LINQ or XPATH or something similar.

[Damien Pollet]

On 13 October 2011 15:30, GT <tar...@euzo.ca> wrote:
> The server table structure does change by time, and the end user
> complex data structure does change as well, and it is up to the
> developers to find the best way to implement those changes.
> This has been going for so many years, actually for decades, and
> everyone knows that a full solution to the problem is to improve the
> virtual machine, the programming language, the libraries and
> algorithms that load and save data, and everyone is ignoring it.

I think the guys at Gemstone / Maglev would probably have an objection here :)

--
Damien Pollet
type less, do more [ | ] http://people.untyped.org/damien.pollet

[Martin Turina]

I see this topic took place a considerable amount of time ago... so sorry for tackle it now again, but I just found it, and thought it is very interesting.  Data is in my opinion an unseen and important issue of today's IT.  It's something I tried to sort out in my head many times. I think I see what GT was trying to go.  I'm going to expose some abstracts (and maybe not that abstract) thoughts here, so be warned!

I can hardly think of an application that's not data-centric.  And the way in which data is structured is pretty much the same always... which is ABSTRACTION.  Even if you call it tables with rows and columns, or you call it classes or you call it object stores... to me is similar, the point is, you are going to end up having a collections of instances with a common set of properties.  Object Orientation gives you a plus, Class Inheritance, which in my opinion is another unavoidable concept.  When we go to eBay, we are accessing a database of bids, in YouTube you will find a collection of videos, IMDB, needless to say, movies information, and so on and so forth.  And what you are going to pretty much be willing to do with them, is listing, adding, changing, filtering, sorting, cross reference them, you want to go statistical on data, you want to do data mining, etc, etc.  You want to be able to manipulate data in the most intuitive and nonetheless complete and intricate ways possible. Seeing so much in common between application needs, it's striking that there still has to be one ad-hoc very complicated and costly implementation for each data set that needs to be represented and manipulated out there, no widely common way on how to do that. And then maintain, right?

So what I'm saying here? Create another SQL? Another database engine? Hmm... a database engine is designed to run underneath a computer program, but there is no real seamless integration.  Why do we still have to OPEN or CONNECT TO a database?  Do we have to open a file or folder to start using them?  Do we have to open a variable? (of course there are differences, a variable is ephemeral and runs in memory, data on db has constraints and simultaneous access requirements, you need to trigger transactions, but the same principle of data structure applies, right?) When programming over a database, there are many other things that make it obvious that you are dealing with something external to the programming language, it's a database API after all, not the programming language itself.  What I'm saying is the structure of data and its store format could be a well written standard.  How we access, and manipulate it, can be a set of tools running on top of it... that can't exist without the former.  And the programming language, which is one the cornerstone of those tools, today it has little or nothing to do with the data we want to store.  Maybe because just the programming language alone, wouldn't close the deal, as I will expand later.

I'm starting to use Dart, because I'm not that experienced on web developing and I'm turned to think it is going to make it easier for me.  What I want to do in principle is a price list (simple right?), of course that's the first milestone.  You can see very easily that my application IS ALMOST the data... or showing it up, anyways.  So first off,  I need to know HTML and CSS for the interface.  Say I'm not very enthusiast about Javascript, so I go learn Dart for the interface dynamics.  In my case I need offline access, so I added IndexedDB to the pile.  Then I need to put the information on a server.  Dart is still green on that regards, that I know of.  I can either go with PostgreSQL, or MongoDB or MySQL API's that I've seen out there, or use PHP and MySQL, or another combination.  I'm sure I will end up searching the Internet for examples, templates, adding things to my code based on other approaches.  In other words, copy/paste a good amount of stuff, so to speak.  Let me say what I think of copy/pase: is TECHNOLOGY OVERHEAD.  Every time you have to do that a lot, chances are there's something the technology is not doing the best for you.  And going back to my little project... you have HTML, CSS, Dart, IndexedDB, PHP, MySQL, and that's just the coding part, nothing about the infrastructure underneath that... needless to say, it feels like too much effort for a humble price list.  I think Dart can be great, but no, it was not aimed to go that further on innovation, in my opinion, and you will probably feel disappointed if you think it as such. In one hand, I would like to see a ground breaking change with this regards, someday.  In the other hand, if I'm going to invest time (maybe years) on this, then it better last!!  I'm IT engineer and I know how it is to waste a lot of time of my life on obsolete technologies.

Here is one of my fundamental questions: Why do we still store information in files?  Is it always has to be that encapsulated and 'mysterious'?  It sounds reasonable for a sound, a video, or an image which cannot be structured in terms of objects or tables with rows right?  Those are analogical data, it can only be understood by a decoder and available to human sense through the specific physical interface, so you have a beginning, a length, and everything in between is somebody else's business... so this is a FILE or BLOB, whatever you want to call it.  Then you will have the edge cases, the unconsidered, data that just wont fit the standard, a compiled binary, some sort of encrypted information, or simply a document with its proprietary structure.  So we create our ad-hoc data store, a consecutive stream of bits, which is only understood by our application... a FILE or BLOB again.  And to be more specific, I would say just a BLOB.  The difference between a file and a blob is that the blob is on top of a database.  In turns a database runs on top of a file system.  So a database is essentially a file, used by the db engine (put in terms of what most people can understand).  But anyways, that's the wrong way for all the rest, MOST INFORMATION.  If we do a SUBTLE SWITCH, instead of data on top of file systems, we do files (or blobs, if any) on top of data systems, (and by data system I mean very solid technology that can handle the concept of abstraction pretty well) things are going to converge towards a world where true data is widely used and comprehended in the same way.  And that's what languages are about.

** Sorry for this is getting waaay too long for an email, but I haven't completed my point yet.

Everything that's not analogical in nature and not an edge case, it can be represented using abstractions and instances (classes and objects, tables and rows, whatever you want to call it).  That's the rule, and with little effort you can see it everywhere we browse on the web (and the informational world in general).  Now, the average dude might know nothing about IT, but he will sure at least know what is a file, an extension and a folder.  For them, that is IT fundamentals.  WHAT IF, INSTEAD, they know what are classes, objects and properties (or call it another way if you are not fond of OOP, but abstraction is what is).  This is eventually what's going to be taught in schools, because is the atomic structure of information, no doubt about it.  What if in addition to the ability to make ambiguous search with different terms in our favorite search engine, we are also given with the possibility to do specific data oriented queries, sort of SQL like (maybe more user-friendly), directly from our Internet browser. Wouldn't it be nice that Chrome or Firefox in addition to be the language virtual machine and HTML renderer, etc, could also be a query interpreter that shows us data in the most fine-grained way possible and let us manipulate it (with all the security vicissitudes of course)? Browsers already include many tools for the local stored data, anyways.

Take this example, we are very picky with a color so we want to look for blue smartphones on the web.  Do we really want the websites relevant to the term "blue smartphones"? No... what we truthfully want too see is the list of BLUE SMARTPHONES, we want the actual thing!!

Something like

data://ebay.com smarphone: color = blue

Or you want a list of videos in which appears Jim Carrey, but you want to make sure is not that long

data://youtube.com video: (involved = 'Jim Carrey', duration < 5)

** These are hypothetical URL's showing a new imaginary "data" protocol, and imaginary query language.

Maybe the examples are not the greatest. The information can be or not that well structured, but for the sake of a getting matches it will better be.  Most probably even the user will contribute to that end.  It can seem to you that you don't need that level of structure on all your searches and data, or even looks silly now.... Let me tell you, it is very unlikely that you feel the necessity of something that doesn't even exist yet.  Some years ago a friend of mine, was convinced that he did not need Internet on his phone.  I told him, of course you don't need, you don't even know what's going to do for you.  You can imagine how dependent is of his full-internet-featured smartphone now.  Point is, once you achieve that level of tools, there's no going back.  

In addition you would be able to share URL's which holds a collection of things, data.  First you might think of, are FTP URL's, which have some analogies, although is not by far a fair comparison.  We want to manipulate data, not files.  Either when programming or generally speaking, that's what we ultimately want to do.  From the end user perspective we would still have access to the webpage, but say I'm tired of having to access data the way I've been imposed.  I have a shortcut, I can skip that and see the data directly, the way I would be used to, because I could do it everywhere else the same way.  Of course some webs already offer powerful ad-hoc tools to make refined searches, but is no as nearly good as it could be if it were a standard, besides you can't re-use that knowledge, it's just for that webpage, that domain.  In the other hand, then you want to do it for your own data-centric application, right? It is going to be a struggle, to say the least.

Imagine something impossible now.  Say we not only standardize data fundamentals, we start standardizing classes as well, 'federated classes' if you will.  I'm giving this very unlikely but I think illustrative example. You have contacts from Facebook and contacts from Google with the same footprint.  You list them all together like

data://google.com contact, facebook.com contact

The system will make you notice the domain they come from, plus all the properties/columns they have in common, you can sort them the way you want (given the correct indexation is set), see duplicated information and stuff.  Even if they don't have the same structure, the system would show you the properties from an upper class they descend from.  Every instance have a creation date, user, type, etc, at least.  See, when you take it to the Object level, it resembles a file system.

I will go a bit borderline.  Imagine if the standard go that spread, so we get to see cross features on HTML.  How powerful would it be to do

<table src="data://evernote.com note" rows="20">

  <colgroup>

    <col property="title">

    <col property="snippet">

  </colgroup>

  <tr>

    <th>Title</th>

    <th>Snippet</th>

  </tr>

</table>

** 'src', 'property' and 'rows' are imaginary.

In this example we have a list of notes of our Evernote notebook embed on HTML.  (one way or another everything can appear on screen, even a sound)  This way, you can embed data from a database on any HTML interface, that simply, no code.

Truth is, options are always good, for instance, skipping the presentation layer (interface, HTML) is to me, very appealing in more than one way.  And this is also mostly desirable on a business environment, where the tools and applications were developed in a very sucky way, many times due to that sense of urgency and only-needs-to-work philosophy.  Or take it to the basic, every Tom, Dick and Harry that simply want to make an inventory (something very trivial), today they need an spreadsheet, or maybe Microsoft Access, if they are pro's.  And from the developer point of view, it would simplify things, big time.  Developing learning curve, would be shorter, because you already know largely how the "data layer" works.  My off-line access issue on my website would possibly have a more straight forward approach,  I don't need 2 different database technologies, I would just cache objects, classes, etc, the same way I can today cache files on a web application (which by the way, is fairly easy to do, but not as useful as one might think, in my opinion).  Same way as files are files, whether on the server or on my PC, objects would be objects equally, assuming is a standard.  Other things that encompasses a database, will start to be closer to normal people. I'm not saying everyone would know what an index and a unique key is, but much more people would, for sure. Most people knows that a building is made of bricks, but the same does not apply to IT, and I think is because we still have not exposed clearly which is the fundamental brick of information (again, abstraction). If more people is closer, technology evolves faster.

The web is based on standards right? HTML might not be that 'state of the art' in comparison to other technologies, but if you need to do an interface, that's what most probably you are going to use (plus CSS, plus Javascript... or Dart ;) ).  And it does a good job, is easy and compatible.  And is surviving everything else, period.  In my opinion the current open data solutions out there go half the way down, being too generous, but mainly they are blind on one goal, which is to make data somewhat easy for normal people.  I think once we do the switch, we are going to see back in time and say 'Hey, where the hell was the data layer before??'

If you think about it, most people are still seeing information the same way we always did, back to DOS times (that's how further I can go given my age, anyways).  I might be dreaming more than John Lennon and Martin Luther King combined, but I still think that the change is most desirable.  Sort of IT 2.0, if you will.  It would not be in principle a dramatical game changer for websites itself (HTTP and HTML will stand the same), it would make it much much easier to access, manipulate, understand data and develop new tools, all across the web, and that indeed can be a game changer. So is not just a programming language, is also a data server and engine, a standard protocol that runs on top of TCP (just like SMTP, HTTP, FTP), a standard structure for the data, a couple of components embed on the browser, and possibly more.  Unless someone create all powerful open tools in a suite that support that new perspective, that's is not going to happen.  I would eagerly develop applications in a platform like this.  And that's what is about, making it appealing to developers to create a complete full-featured application from the ground up. So it has to fit economically in individuals and companies lives, with the risk that involves such a long shot approach.

[Andrei Mouravski]

I've tried to respond to this post in several different ways, but I've deleted everything because the comment itself is so philosophical, that I can only respond philosophically... and I hate philosophy. 

So I'm just going to provide a few helpful links and hope that you elaborate on your points with concrete examples of what you want, and what we can do to get there. 

---

Want to import a useful library? Try pub: http://pub.dartlang.org/

Want a generic database model? Try lawndart: http://pub.dartlang.org/packages/lawndart

Want components and data binding? Try web_ui: http://www.dartlang.org/articles/web-ui/

Want RESTful services? See: http://www.dartlang.org/articles/json-web-service/

Want an RIA that you can plug into? Try Rikulo: http://docs.rikulo.org/ui/latest/UXL/Fundamentals/UXL_Overview.html

---

See if you can boil your concerns to a few bullet points of examples and proposed solutions and I'll have more to say. :]

Andrei.

--
Consider asking HOWTO questions at Stack Overflow: http://stackoverflow.com/tags/dart
 
 

[Dirk Detering]

I stopped in the middle of this looooooong post.
(Btw Andrei, full quote ... ?   :-(  )
A pity this was not published on a blog, as it would be easier to further elaborate this thoughts, beyond Dart's context btw, as this is no Dart specific thing, I assume.
A blog post would be a better tool for web philosophers.

Am 18.03.2013 01:03 schrieb "Andrei Mouravski" <amour...@google.com>:
> and I hate philosophy. 

A pity again.

> So I'm just going to provide a few helpful links

No link to a Dart semantic-web lib?
Pehaps that is all he wants... :-/

> and hope that you elaborate on your points with concrete examples of what you want,

Yes! But perhaps not here...?

> and what we can do to get there. 

The ephasis is on _we_. Say: What can _Dart_ do, regarding its committed purpose, goals and current direction of development.
(Seeing the game oriented low-level performance related thread over there).

One thought that crossed my mind when I read the post was: Well, the FP world with its upcoming concepts of monads and arrows a.s.o. has to offer something. Abstract syntactic constructs like comprehensions or LINQ may be necessary.

But this all seems to be in the realm of R&D

[Ladislav Thon]

I'm absolutely sure that I didn't understand you, but I'm gonna answer anyway, providing some sort of disagreement as a food for thought.

> I can hardly think of an application that's not data-centric.

In the same sense, every application is a compiler -- it takes input, does some transformations and produces output. See the pattern? Start with too generic claims and you will end nowhere.

> Object Orientation gives you a plus, Class Inheritance, which in my opinion is another unavoidable concept.

Inheritance is evil. And by the way, there are object oriented languages that have no classes at all.

>  You want to be able to manipulate data in the most intuitive and nonetheless complete and intricate ways possible.

Complete and intricate data manipulation... that's called programming.

> Hmm... a database engine is designed to run underneath a computer program, but there is no real seamless integration.

Not necessarily. Check out MUMPS -- it's an ancient language that has one feature that no other language managed to steal yet. It's called global variables, and it means _persistent_ variables. Variables that start with ^ are simply stored on disk in a highly optimized sparse multi-dimensional hash table something. Some people call it "universal NoSQL", because it can accommodate all data models (key-value, tables, documents, graphs, what have you), and I honestly believe that this is the right way we should handle data.

>  Why do we still have to OPEN or CONNECT TO a database?

Because it's costly. Well, at least with SQL and other "modern" databases.

> Do we have to open a file or folder to start using them?

Yes.

> What I'm saying is the structure of data and its store format could be a well written standard.

Did we really reached that level of knowledge? Look at SQL, it's supposedly The Standard Way of dealing with data, yet it's ridiculously bad on so many levels.

> And the programming language, which is one the cornerstone of those tools, today it has little or nothing to do with the data we want to store.

Yeah, that's just stupid. We need to copy the MUMPS model.

> In other words, copy/paste a good amount of stuff, so to speak.  Let me say what I think of copy/pase: is TECHNOLOGY OVERHEAD.  Every time you have to do that a lot, chances are there's something the technology is not doing the best for you.

Google for "worse is better". That's how we ended up with things like Unix, C, SQL etc.

> I think Dart can be great, but no, it was not aimed to go that further on innovation

That's exactly right. Dart spent its "innovation budget" on solving some problems of current mainstream OO languages, not on saving the world. If it went that far, it would be very unlikely that it got popular.

> Here is one of my fundamental questions: Why do we still store information in files?

We don't, we use SQL. And the reason is ... because the world is crazy. There are people using Caché or GT.M (the two most popular implementations of MUMPS today), but vast majority went with SQL. Or MongoDB, which itself is a major improvement, but it's not good enough.

> In turns a database runs on top of a file system.

Not necessarily, some databases can run on top of raw disk partitions. But it doesn't really matter. You are all into abstraction, right, and then you go deep into implementation details. Data need to be stored somewhere.

> If we do a SUBTLE SWITCH, instead of data on top of file systems, we do files (or blobs, if any) on top of data systems, (and by data system I mean very solid technology that can handle the concept of abstraction pretty well)

And what exactly is the difference?

> things are going to converge towards a world where true data is widely used and comprehended in the same way.

Things are not going to converge. The world will always be a mess. Of course you can (try to) build an abstraction on top of that (like Yahoo Pipes, anyone?), but you can only go so far.

> The web is based on standards right?

Standards that are largely written by people that don't understands developers' needs. Which brings us to...

> If you think about it, most people are still seeing information the same way we always did, back to DOS times (that's how further I can go given my age, anyways).  I might be dreaming more than John Lennon and Martin Luther King combined, but I still think that the change is most desirable.  Sort of IT 2.0, if you will.  It would not be in principle a dramatical game changer for websites itself (HTTP and HTML will stand the same), it would make it much much easier to access, manipulate, understand data and develop new tools, all across the web, and that indeed can be a game changer.

XHTML 2, semantic web and all that stuff. Never heard of that? That's because noone was interested.

LT

[Giovanni Candido da Silva]

Database systems are born to handle some data problems like, storage, concurrency, transactions and relationships. But database systems are written in some language. I think that the language itself have not to do with databases, the language is the way you communicate with the no intelligent machine, and you teach that machine how to do some thing.

What you want is just too generic and is just too big to address on a language, but you can use the language to build what you want. This is why we have languages and databases, is about separate of concerns. 

Maybe I was too simplistic on my words, but you get the idea.

[Alex Tatumizer]

My understanding is that the complaint is of purely philosophical nature, and boils down to the question: why don't we have an AI system that understand my request, expressed in a slightly formalized language (attr1: value, attr2: value...)?

The answer is very simple. 

For a program to understand the meaning of your request , we need a formal definition of "meaning" - essentially, we need to formalize the meaning of term "meaning".

Such definition would necessarily be circular.

(Sorry to disappoint :-).

[Ladislav Thon]

Such definition would necessarily be circular.

If it would, we would never have programming languages with fully defined semantics. In other words, does the Dart language specification look circular to you? :-)

LT

[Alex Tatumizer]

 > does the Dart language specification look circular to you?

It has fully defined semantics in the same sense as phone book has fully defined semantics.

What's the semantics of the notion of "semantics" you use? :-)

[Ladislav Thon]

 > does the Dart language specification look circular to you?

It has fully defined semantics in the same sense as phone book has fully defined semantics.

What's the semantics of the notion of "semantics" you use? :-)

For a programming language, semantics is the definition of runtime behavior of programs. Or, you can also say that semantics is a set of constraints that each implementation must obey. And there's a bunch of other definitions. The important thing is: we have formal tools to define semantics of programming languages, so at least in the world of programming languages, semantics (= meaning) can be clearly and fully and unambiguously (and non-circularly) defined.

LT

[Martin Turina]

While I realized this could have been better material for a blog, this was no interest whatsoever to my circles, and I don't (currently) run a blog.

This is one of those things that stick to your mind, and I urgently needed to know what people on this forum think, before it grows and my life itself turns too philosophical.

Seeing the comments so far, my current feeling is that I haven't made my point, and I would need to elaborate A LOT more.

--

[Ladislav Thon]

This is one of those things that stick to your mind, and I urgently needed to know what people on this forum think, before it grows and my life itself turns too philosophical.

Seeing the comments so far, my current feeling is that I haven't made my point, and I would need to elaborate A LOT more.

Well, there are people that understand the importance of working with data. I personally imagine some sort of collection comprehensions on steroids (if it looks like SQL, that's a -1 to me), combined with dynamically creating "dynamic" objects for representing structured data. And of course persistent variables, but that's never going to fly :-(

But the main point is -- I think -- that you've lost a lot of people somewhere around the middle of your post. Sadly.

LT

[Dirk Detering]

Am 18.03.2013 15:14 schrieb "Ladislav Thon" <lad...@gmail.com>:
>>
>> Such definition would necessarily be circular.
>
>
> If it would, we would never have programming languages with fully defined semantics.

Hmm AFAIU he was talking about something like the meta-levels of MDA. There you have a MOF level where MOF is defined in its own terms coming to a circle in meta-meta.

[Alex Tatumizer]

Speaking of meaning of meaning, it's impossible to discuss the issue without referring to this rant :-)
Has anyone read it? Be honest, I will not tell anyone :-)

[Andrei Mouravski]

Hi Martin,

I think what you could do to help with people understanding your point is splitting it into a few discrete points, a TL;DR (too long; didn't read summary). I think you have a lot of interesting things to say, but it's getting lost in the elaboration.

Alternatively, if you have several discrete ideas, maybe spawn smaller threads about each point?

Andrei.

[Martin Turina]

I tried hard to set a reason, without diving into "how we should do a db engine", "what are we expecting from a pl", etc.  I guess at some point of my writing, things started to overflow a little bit.  I didn't initially think it was going to be that long.  I would have needed to reaccomodate some stuff and better use some headings, that way people can have an idea of what's coming next and create some expectation, so reading is more fluent.  It's funny I didn't use structure on a topic about structure, so to speak ;)

Thanks for the feedback, Andrei.  If I can elaborate something I consider worth of sharing, I will take the advice!

[Tobias Ritzau]

This reminds me of when I started to learn how to program. I couldn't understand why I had to draw lines and circles to be able to draw a car. I wanted a command where I could say: drawNiceCar() or possibly drawNiceCar(blue). That said, if look passed the ranting of object orientation (everything that is 30 years old has not gone bad...) I agree that in some cases (not speaking of Dart in particular) you have to copy and paste too much. Not too much to build great software, but enough to make it frustrating at least when you find something that you want to change in the copied and pasted code. I believe that most cases where I have run into this can be solved be the noSuchMethod() thingy, but i still find code like toString(), equals(), compareTo(), hashCode(), and possibly others being a real pain in the ass when you work with "data objects". You are best of generating them, and it would be great if the compiler could help out (instead of asking the IDE to generate all everytime you make a change). It would be really interesting if you can give examples of where you find to much copy-paste required, and let´s see how we can solve that. It could very well be that a lib is the solution. At least OO is not the problem ;)

-- Tobias

[Gen]

Invent your own programming language.
Then you will recognize that every language feature and trait has some cost and prevents other features or traits.
Including traits like performance, implementation in the browser, certain kinds of static program verification or potential for concurrency.
Besides manpower and budget for Dart is limited. The Dart team can not offer everything known and unknown in 1 or 2 years.

If I have well understood, the initial goal of Dart was to be like Javascript but with optional variable typing and cheaper and more automatic optimizations.
Your expectation was not, is not and can not be the goal for Dart.

Maybe someone will create a library like Linq for Dart.
But Linq is just one way among many ways to deal with data.
If you have found a solution that suits every possible case perfectly for everyone, I bet we would all love to know about it and have it integrated in Dart.   
For now, you are just critizising Dart for not offering this miracle solution which is unknown to the field of computer science.

[Ladislav Thon]

If you have found a solution that suits every possible case perfectly for everyone, I bet we would all love to know about it and have it integrated in Dart.   
For now, you are just critizising Dart for not offering this miracle solution which is unknown to the field of computer science.

I don't think that this is what happens here. Instead, I only see a frustration from the fact that no mainstram-ish programming language tries to do anything to make working with data easier. Well... except of C#. Which is a shame.

LT

P.S.: the collection library in Dart has all the LINQ-ish methods, so that's a good start.

[Gen]

That is probably because most mainstream languages are general purpose languages almost by definition.
Otherwise they are specific solutions for specific problems and not mainstream anymore.

And as already written, everything is limited by time, money and good ideas.

For the interested, there is a short discussion about programming in general and Mozart 2 in particular.
http://lambda-the-ultimate.org/node/4698

[Gen]

PS:

Programming is all about data.
Therefore Dart is all about data handling.

The discussion of classes in Dart or not is like the discussion about static types or not.
The question about how to create, find and manipulate data is the question how to program and what program to create for the given task or goal.

Am Donnerstag, 21. März 2013 09:48:59 UTC+1 schrieb Ladislav Thon:

[Alex Tatumizer]

Is there any mechanism of dynamic code generation in dart?

E.g. it's easy to produce dart source and put it in a file, but loading this file would be a problem: import is a static statement.
Any way around it?

[Ladislav Thon]

Is there any mechanism of dynamic code generation in dart?

E.g. it's easy to produce dart source and put it in a file, but loading this file would be a problem: import is a static statement.
Any way around it?

Not yet. Wait for "mirror builders".

LT

[Ladislav Thon]

Oh, but if you are fine with dynamically generated code that runs in new isolate, then you can have that, you can spawn an isolate that will run code loaded from given URI.

LT

[Alex Tatumizer]

>>Oh, but if you are fine with dynamically generated code that runs in new isolate, then you can have that, you can spawn an isolate that will run code loaded from given URI.

Hurrah!!!

Hopefully, URI includes also "data:" URI? Or no such luck?

[Ladislav Thon]

>>Oh, but if you are fine with dynamically generated code that runs in new isolate, then you can have that, you can spawn an isolate that will run code loaded from given URI.

Hurrah!!!

Hopefully, URI includes also "data:" URI? Or no such luck?

I don't think so. But I didn't try :-)

LT

[Justin Fagnani]

https://code.google.com/p/dart/issues/detail?id=4270

That's just Dartium. I'm really unsure how you would generate the URI for dart2js...

--

[Alex Tatumizer]

Good to know. Maybe some day it will work. I love dynamic code generation :-)

[ I love static code generation, too. Generating assembly code was my favorite device. Naturally, it applies only in cases where performance is a super-critical issue ]

[tar...@euzo.ca]

Wow, it is nice to see someone who understood the issue, you should post this to Google+

My understanding of data evolved a bit since the original post, in general, it will be nice if we have Standard Data Processing Instructions (SDPI), not just SQL, well SQL does a lot of processing, and some people hate it and they started the NOSQL that also does a lot of processing too :-)

And it is not required to be understood the same at every level, I mean the programming language can have a basic understanding of SDPI and a database server can have a complex understanding of it, a file system can understand it a bit differently there as well, etc.

SDPI will eventually collide with principals like the data must be separate from the code, I don’t know why people put such unrealistic things and they complain that there is no AI yet, or why is it hard to do voice recognition or image recognition; well the human brain is data and instructions together, the data evolves the instructions, and the instructions evolve the data, even the hardware itself is affected by the data and instructions.

My biggest problem the past many years is the lack of innovation, people say that to work with data in a programming language you need a database, but to work with memory you don’t, well that is not true, the memory management module is a kind of a database as well, a very simple one.

Software developers just invented too many limitations in the last two decades, such as the separation of data and code, the many useless layers of abstracting and separating things, and the security limitations, things are designed in a way that instead of using cars everyone must have a tank :-)

To whoever is designing languages and working with data, look at the nature, because it shows what is possible, the human eye is an object, it looks separate from the nerve, and yet the nerve is very integrated with the eye, the nerve is somehow separate from the brain and still very integrated with the brain, the blood moved freely between them, etc.

Every object knows a lot about many other objects, and every object brakes the separation rules when needed to achieve the total goal.

Good luck

On Sunday, March 17, 2013 5:50:39 PM UTC-4, Martin Turina wrote:

I see this topic took place a considerable amount of time ago... so sorry for tackle it now again, but I just found it, and thought it is very interesting.  Data is in my opinion an unseen and important issue of today's IT.  It's something I tried to sort out in my head many times. I think I see what GT was trying to go.  I'm going to expose some abstracts (and maybe not that abstract) thoughts here, so be warned!

I can hardly think of an application that's not data-centric.  And the way in which data is structured is pretty much the same always... which is ABSTRACTION.  Even if you call it tables with rows and columns, or you call it classes or you call it object stores... to me is similar, the point is, you are going to end up having a collections of instances with a common set of properties.  Object Orientation gives you a plus, Class Inheritance, which in my opinion is another unavoidable concept.  When we go to eBay, we are accessing a database of bids, in YouTube you will find a collection of videos, IMDB, needless to say, movies information, and so on and so forth.  And what you are going to pretty much be willing to do with them, is listing, adding, changing, filtering, sorting, cross reference them, you want to go statistical on data, you want to do data mining, etc, etc.  You want to be able to manipulate data in the most intuitive and nonetheless complete and intricate ways possible. Seeing so much in common between application needs, it's striking that there still has to be one ad-hoc very complicated and costly implementation for each data set that needs to be represented and manipulated out there, no widely common way on how to do that. And then maintain, right?

So what I'm saying here? Create another SQL? Another database engine? Hmm... a database engine is designed to run underneath a computer program, but there is no real seamless integration.  Why do we still have to OPEN or CONNECT TO a database?  Do we have to open a file or folder to start using them?  Do we have to open a variable? (of course there are differences, a variable is ephemeral and runs in memory, data on db has constraints and simultaneous access requirements, you need to trigger transactions, but the same principle of data structure applies, right?) When programming over a database, there are many other things that make it obvious that you are dealing with something external to the programming language, it's a database API after all, not the programming language itself.  What I'm saying is the structure of data and its store format could be a well written standard.  How we access, and manipulate it, can be a set of tools running on top of it... that can't exist without the former.  And the programming language, which is one the cornerstone of those tools, today it has little or nothing to do with the data we want to store.  Maybe because just the programming language alone, wouldn't close the deal, as I will expand later.

I'm starting to use Dart, because I'm not that experienced on web developing and I'm turned to think it is going to make it easier for me.  What I want to do in principle is a price list (simple right?), of course that's the first milestone.  You can see very easily that my application IS ALMOST the data... or showing it up, anyways.  So first off,  I need to know HTML and CSS for the interface.  Say I'm not very enthusiast about Javascript, so I go learn Dart for the interface dynamics.  In my case I need offline access, so I added IndexedDB to the pile.  Then I need to put the information on a server.  Dart is still green on that regards, that I know of.  I can either go with PostgreSQL, or MongoDB or MySQL API's that I've seen out there, or use PHP and MySQL, or another combination.  I'm sure I will end up searching the Internet for examples, templates, adding things to my code based on other approaches.  In other words, copy/paste a good amount of stuff, so to speak.  Let me say what I think of copy/pase: is TECHNOLOGY OVERHEAD.  Every time you have to do that a lot, chances are there's something the technology is not doing the best for you.  And going back to my little project... you have HTML, CSS, Dart, IndexedDB, PHP, MySQL, and that's just the coding part, nothing about the infrastructure underneath that... needless to say, it feels like too much effort for a humble price list.  I think Dart can be great, but no, it was not aimed to go that further on innovation, in my opinion, and you will probably feel disappointed if you think it as such. In one hand, I would like to see a ground breaking change with this regards, someday.  In the other hand, if I'm going to invest time (maybe years) on this, then it better last!!  I'm IT engineer and I know how it is to waste a lot of time of my life on obsolete technologies.

Here is one of my fundamental questions: Why do we still store information in files?  Is it always has to be that encapsulated and 'mysterious'?  It sounds reasonable for a sound, a video, or an image which cannot be structured in terms of objects or tables with rows right?  Those are analogical data, it can only be understood by a decoder and available to human sense through the specific physical interface, so you have a beginning, a length, and everything in between is somebody else's business... so this is a FILE or BLOB, whatever you want to call it.  Then you will have the edge cases, the unconsidered, data that just wont fit the standard, a compiled binary, some sort of encrypted information, or simply a document with its proprietary structure.  So we create our ad-hoc data store, a consecutive stream of bits, which is only understood by our application... a FILE or BLOB again.  And to be more specific, I would say just a BLOB.  The difference between a file and a blob is that the blob is on top of a database.  In turns a database runs on top of a file system.  So a database is essentially a file, used by the db engine (put in terms of what most people can understand).  But anyways, that's the wrong way for all the rest, MOST INFORMATION.  If we do a SUBTLE SWITCH, instead of data on top of file systems, we do files (or blobs, if any) on top of data systems, (and by data system I mean very solid technology that can handle the concept of abstraction pretty well) things are going to converge towards a world where true data is widely used and comprehended in the same way.  And that's what languages are about.

** Sorry for this is getting waaay too long for an email, but I haven't completed my point yet.

Everything that's not analogical in nature and not an edge case, it can be represented using abstractions and instances (classes and objects, tables and rows, whatever you want to call it).  That's the rule, and with little effort you can see it everywhere we browse on the web (and the informational world in general).  Now, the average dude might know nothing about IT, but he will sure at least know what is a file, an extension and a folder.  For them, that is IT fundamentals.  WHAT IF, INSTEAD, they know what are classes, objects and properties (or call it another way if you are not fond of OOP, but abstraction is what is).  This is eventually what's going to be taught in schools, because is the atomic structure of information, no doubt about it.  What if in addition to the ability to make ambiguous search with different terms in our favorite search engine, we are also given with the possibility to do specific data oriented queries, sort of SQL like (maybe more user-friendly), directly from our Internet browser. Wouldn't it be nice that Chrome or Firefox in addition to be the language virtual machine and HTML renderer, etc, could also be a query interpreter that shows us data in the most fine-grained way possible and let us manipulate it (with all the security vicissitudes of course)? Browsers already include many tools for the local stored data, anyways.

Take this example, we are very picky with a color so we want to look for blue smartphones on the web.  Do we really want the websites relevant to the term "blue smartphones"? No... what we truthfully want too see is the list of BLUE SMARTPHONES, we want the actual thing!!

Something like

data://ebay.com smarphone: color = blue

Or you want a list of videos in which appears Jim Carrey, but you want to make sure is not that long

data://youtube.com video: (involved = 'Jim Carrey', duration < 5)

** These are hypothetical URL's showing a new imaginary "data" protocol, and imaginary query language.

Maybe the examples are not the greatest. The information can be or not that well structured, but for the sake of a getting matches it will better be.  Most probably even the user will contribute to that end.  It can seem to you that you don't need that level of structure on all your searches and data, or even looks silly now.... Let me tell you, it is very unlikely that you feel the necessity of something that doesn't even exist yet.  Some years ago a friend of mine, was convinced that he did not need Internet on his phone.  I told him, of course you don't need, you don't even know what's going to do for you.  You can imagine how dependent is of his full-internet-featured smartphone now.  Point is, once you achieve that level of tools, there's no going back.  

In addition you would be able to share URL's which holds a collection of things, data.  First you might think of, are FTP URL's, which have some analogies, although is not by far a fair comparison.  We want to manipulate data, not files.  Either when programming or generally speaking, that's what we ultimately want to do.  From the end user perspective we would still have access to the webpage, but say I'm tired of having to access data the way I've been imposed.  I have a shortcut, I can skip that and see the data directly, the way I would be used to, because I could do it everywhere else the same way.  Of course some webs already offer powerful ad-hoc tools to make refined searches, but is no as nearly good as it could be if it were a standard, besides you can't re-use that knowledge, it's just for that webpage, that domain.  In the other hand, then you want to do it for your own data-centric application, right? It is going to be a struggle, to say the least.

Imagine something impossible now.  Say we not only standardize data fundamentals, we start standardizing classes as well, 'federated classes' if you will.  I'm giving this very unlikely but I think illustrative example. You have contacts from Facebook and contacts from Google with the same footprint.  You list them all together like

data://google.com contact, facebook.com contact

The system will make you notice the domain they come from, plus all the properties/columns they have in common, you can sort them the way you want (given the correct indexation is set), see duplicated information and stuff.  Even if they don't have the same structure, the system would show you the properties from an upper class they descend from.  Every instance have a creation date, user, type, etc, at least.  See, when you take it to the Object level, it resembles a file system.

I will go a bit borderline.  Imagine if the standard go that spread, so we get to see cross features on HTML.  How powerful would it be to do

<table src="data://evernote.com note" rows="20">

  <colgroup>

    <col property="title">

    <col property="snippet">

  </colgroup>

  <tr>

    <th>Title</th>

    <th>Snippet</th>

  </tr>

</table>

** 'src', 'property' and 'rows' are imaginary.

In this example we have a list of notes of our Evernote notebook embed on HTML.  (one way or another everything can appear on screen, even a sound)  This way, you can embed data from a database on any HTML interface, that simply, no code.

Truth is, options are always good, for instance, skipping the presentation layer (interface, HTML) is to me, very appealing in more than one way.  And this is also mostly desirable on a business environment, where the tools and applications were developed in a very sucky way, many times due to that sense of urgency and only-needs-to-work philosophy.  Or take it to the basic, every Tom, Dick and Harry that simply want to make an inventory (something very trivial), today they need an spreadsheet, or maybe Microsoft Access, if they are pro's.  And from the developer point of view, it would simplify things, big time.  Developing learning curve, would be shorter, because you already know largely how the "data layer" works.  My off-line access issue on my website would possibly have a more straight forward approach,  I don't need 2 different database technologies, I would just cache objects, classes, etc, the same way I can today cache files on a web application (which by the way, is fairly easy to do, but not as useful as one might think, in my opinion).  Same way as files are files, whether on the server or on my PC, objects would be objects equally, assuming is a standard.  Other things that encompasses a database, will start to be closer to normal people. I'm not saying everyone would know what an index and a unique key is, but much more people would, for sure. Most people knows that a building is made of bricks, but the same does not apply to IT, and I think is because we still have not exposed clearly which is the fundamental brick of information (again, abstraction). If more people is closer, technology evolves faster.

The web is based on standards right? HTML might not be that 'state of the art' in comparison to other technologies, but if you need to do an interface, that's what most probably you are going to use (plus CSS, plus Javascript... or Dart ;) ).  And it does a good job, is easy and compatible.  And is surviving everything else, period.  In my opinion the current open data solutions out there go half the way down, being too generous, but mainly they are blind on one goal, which is to make data somewhat easy for normal people.  I think once we do the switch, we are going to see back in time and say 'Hey, where the hell was the data layer before??'

If you think about it, most people are still seeing information the same way we always did, back to DOS times (that's how further I can go given my age, anyways).  I might be dreaming more than John Lennon and Martin Luther King combined, but I still think that the change is most desirable.  Sort of IT 2.0, if you will.  It would not be in principle a dramatical game changer for websites itself (HTTP and HTML will stand the same), it would make it much much easier to access, manipulate, understand data and develop new tools, all across the web, and that indeed can be a game changer. So is not just a programming language, is also a data server and engine, a standard protocol that runs on top of TCP (just like SMTP, HTTP, FTP), a standard structure for the data, a couple of components embed on the browser, and possibly more.  Unless someone create all powerful open tools in a suite that support that new perspective, that's is not going to happen.  I would eagerly develop applications in a platform like this.  And that's what is about, making it appealing to developers to create a complete full-featured application from the ground up. So it has to fit economically in individuals and companies lives, with the risk that involves such a long shot approach.

[Martin Turina]

On Mon, Mar 25, 2013 at 12:35 PM, <tar...@euzo.ca> wrote:

Software developers just invented too many limitations in the last two decades, such as the separation of data and code, the many useless layers of abstracting and separating things, and the security limitations, things are designed in a way that instead of using cars everyone must have a tank :-)

This exactly would be my first bullet.  And in my opinion, before going into the programming language we first need to address a data model that can be understood by our moms in 1 day, something human.  I'm not talking just to manage homogeneously structured data, but also complex data.  Something that covers most cases, something that can escalate. Once we can manage 'easy' data, data that becomes as massive as writing something on a notepad, we trigger awareness, maybe we can realize that we don't even need coding at all many times, or just leave coding for the interface dynamics.  And, also, I'm not taking about AI, I don't even care about AI at this point... 

I'm pretty sure I know in my head a data model that can serve this goal (better than the ones I know at least) .  I'm trying to design a simple database, to try to make a solid point, about this. Although there's a lot of things like B-Tree's and stuff about storing indexes, etc, that I need to struggle with first, to achieve that.