What is .history folder in .meta?

[Mahmoodreza Aarabi]

hi dear marcus
first thanks to this good package.
you know that i am using openmetadata vastly.

i created a location, and .meta is created, after that i add other groups and data sets to the directory, aotumatically added a hidden folder named ".history".
can you tell me more about it?

thanks

[Marcus Ottosson]

Hey Mahmoodreza,

The .history folder is where changes to metadata is kept. Whenever you make a change to an existing dataset, a record of its previous value is stored as part of its history, as well as the time of edit and who made the change. You can restore it later with om.restore()

History makes changes non-destructive and is used for persistent undo/redo as well as versioning, have a look at the specification for an overview of how it all works.

http://rfc.abstractfactory.io/spec/14

And at the examples for usage.

https://github.com/abstractfactory/openmetadata/samples/working_with_history.py

It is newly implemented, so let me know if you run into any trouble or if would prefer it disabled by default for any reason.

Best,

Marcus

[Marcus Ottosson]

Worth mentioning is the equivalent .trash folder which is where any groups or datasets you delete are stored. They will be retrievable via om.restore() too, but isn't at the moment.

[phili...@gmail.com]

I think this (.history) is a great feature. When we are talking about small amounts of data (config files etc.) I'd be happy to leave this history for ever but I can see that when we start to keep multiple versions of meta images and such we will want to easily remove history states with certain rules such as - over 6 months old etc. 

This is maybe more a function for "About" than openmetadata itself?

Anyway I think this would be a future wishlist thing rather than anything urgently required!

Philip  

[Marcus Ottosson]

Hey Philip, glad you like it.

Automatic clean-up of old history is something we've been in the talks about as it could easily become crowded in there. One thought was to make it such that only a certain number of historical events may exist at any point in time. For example, if there are 20 imprints of a certain set of metadata already, the next time you make a change, the oldest one would get discarded in favour of the new.

There could then be a setting some-place that governs the max-value.

Clean-up in time is certainly another option, but as you say, probably something better suited to more high-level logic, like a gui or separate background-process (such as a garbage collector).

About images, those wouldn't get tracked initially. At the moment, only simple data-types, like text, bools, floats etc get history. There are plans on the roadmap to allow for history of binary data; it would involve data de-duplication, either via simple hard-linking or analysing of chunks of data (like Dropbox). Binary data deserves history and versioning as much as the next guy, but its rather low on the todo list at the moment.

Ideally, I'd like history on at all times so that no data is ever lost. Disk-space is cheap, loosing data isn't.

--
You received this message because you are subscribed to the Google Groups "Open Metadata" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-metadat...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Philip Child]

I think a 20 (or whatever number) cutoff would be absolutely fine. It works for Photoshop!

The fact that you have this history at all is quite a luxury so I would not go any further.

Philip

[Marcus Ottosson]

Cool.

The idea is that for more persistent history, you would use versions instead.

>>> version = om.save(dataset)

>>> note = om.Dataset('note', parent=version)

>>> note.data = 'this version is important, don't lose it!'

>>> om.dump(note)

That way, you could maintain multiple variations of historic metadata, just like we would with Maya files and such.

Versions are an upcoming feature, I'll put it up here once it's ready.

--

Marcus Ottosson
konstr...@gmail.com

[Philip Child]

Well you are certainly building some decent foundations here. Having versions in addition to history states is very powerful.

P

[Mahmoodreza Aarabi]

very good points
thanks

but i have a recomendation, of course you see that and know it well,
when i have ".history" folder in ".meta" folder and i get contents of folder with om.pull(location), it give me all folders and files and .history as group.
you know?
now i want to know that this is the default or you can remove the name of folders that starts with [dot] in pull method?

and if you don't want remove it what is your reason?

thanks again

[Marcus Ottosson]

I struggled with this as well.

As I'm storing history in Open Metadata as Open Metadata, within a regular group with regular datasets, it made sense to not assign special treatment to it. In normal circumstance, you probably would never know it was there as you usually only grab what you are looking for. But in your case, as you are developing a GUI for it, you probably don't want it visible, unless you also wanted your GUI to display history.

Names starting with a dot is the naming convention used for hidden folders in OM. I see two ways of solving this problem; either OM hides them or you do.

OM could potentially hide them in one of two ways;

1. Add a om.pull(location, include_hidden=False) option

2. Keep om.pull() as-is and make Location and Group objects not return hidden folders.

With 1, hidden folders would never enter into an object to begin with which is probably fine. With 2, objects would know of hidden folders, but pretend like it doesn't. There could then be a special property on objects, like Location.history or Location.trash, replacing the om.history() and om.trash() methods.

What do you think?

--

You received this message because you are subscribed to the Google Groups "Open Metadata" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-metadat...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Marcus Ottosson]

About the hidden folders, in 0.4 hidden folders are not shown by default when getting the children of a location or group.

To access a hidden folder, or any folder, you can use brackets. :)

>>> dataset = group['child1']

>>> hidden = group['.hidden']

Let me know how that sounds

What do you think?

To unsubscribe from this group and stop receiving emails from it, send an email to open-metadata+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Philip Child]

Sounds fine to me. Did you find that tidier? Or was it just that most of the time hidden folders tend not to be directly accessed?

You received this message because you are subscribed to a topic in the Google Groups "Open Metadata" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/open-metadata/GS1qwqzVLXI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to open-metadat...@googlegroups.com.
Visit this group at http://groups.google.com/group/open-metadata.

[Marcus Ottosson]

I did, yes. I'm also seeing whether or not there ever is a need for them to show up along with regular children; so far they have not been missed.

Ideally anything hidden should remain in the background and only get accessed under certain circumstances, like when specifically asking for history, via om.history or in a UI when the user has specifically requested to browse history.

The indexing-syntax also blended nicely with overall grabbing of children. Its a syntax I borrowed from HDF5, so it should hopefully be rather future-proof.

To unsubscribe from this group and all its topics, send an email to open-metadata+unsubscribe@googlegroups.com.