Is there a fairly minimal workflow to import pages from Roam into TiddlyWiki?

[Yoni Balkind]

Hi All 

My intended workflow is to draft notes in Roam, and then if I deem certain notes worthy of publish, I publish them in my public-facing TiddlyWiki. 

However these "published" notes might became quite voluminous and if I update them in Roam I want them to update in my TW as well as it would be too laborious to manually maintain two versions of same notes. 

It seems to me that the way this might be achievable is to do regular JSON exports from Roam, and then to import these into TW, overriding existing Tiddly's with the same name.

The challenge is as follows:

• Roam does not allow me to export pages with tag X. So I need a mechanism to filter which pages get imported into TW as the JSON file will be full of pages I dont need.
• The JSON structure is different between Roam and TW, is there a workflow to edit the JSON structure of the Roam file to make it match TW? I suppose a simple search and replace in Notepad++ ? Or is it more complex than that? 

I'm not a programmer, and I'm new to TW, so perhaps there are some obvious tricks or well known tools/plugins that I'm missing. 

[Anne-Laure Le Cunff]

I was looking at the exact same thing yesterday and couldn't figure it out as the structure is quite different. Attaching two sample JSON files for people who want to have a look.

[TonyM]

I am not using roam but do you have other export choices?

json html text xlsx csv and now xml are possible formats to use into tiddly wiki.

Also can you map objects content and fields into tiddlywiki?, I expect so but this needs to be done.

Regards
Tony

[Mark S.]

If you don't have tags, what would you filter by?

How would you deal with "blocks" ? Break them out into tiddlers, or combine them into tiddlers?

[Saq Imtiaz]

@Yoni @Anne-Laure

Here is something to play with for importing from Roam to understand what is possible, though please don't expect too much as I've only had about ten minutes to work on this.

https://saqimtiaz.github.io/sq-tw/roam-import.html

How do tags work on Roam? Does it have tags? As far as I can tell from a quick glance, #tag is actually just a link to [[tag]]

should #xyz be turned into a tag on that tiddler?

Regards,

Saq

[Riz]

We could probably

1. treat the created time as title
2. Consider children to be tagged by their parent.

[Riz]

How deep does the levels go? Are there arbitrarily infinite levels? Like Children, grandchildren and so on?

[Yoni Balkind]

By "tags" I refer to every link within a page (ie within the strings of text of the children of a page). This links could be in the form of [[internal link]] or #internal-link (@Saq your understanding is correct)

So in my case I want to publish every page that has a link [[publish]] and/or #publish 

@Riz - I dont think we need to worry about children of children. Basically all children  (at any level) should just become text within the Tiddly.  

Pages already have titles so no need to treat the created time as the title. The child does not need a title, it is just a block of text if my understanding is correct. 

@Mark S, blocks should just be treated as normal body text.. so if there are 3 blocks (ie "children" in the JSON file) in a page those would all just go into the text block as paragraphs within the Tiddly. I suppose .. 

[Saq Imtiaz]

@yoni have a look at the link I posted and let me know what you think.

https://saqimtiaz.github.io/sq-tw/roam-import.html

Except for the tags bit does the rest work ok if using real data?

[Yoni Balkind]

Thanks @Saq.. I gave it a bash. It imports pages, but it fails to import the content within those pages (ie the "child" sections of the Roam JSON file).. Keep in mind my JSON file had ~1000 pages so after I dragged in the JSON file I ticked 7 pages to import. Maybe if had I ticked everything it would have also captured the "child" sections (ie the body text of each page)? 

I had the same experience when I tried to import the JSON file top my normal Wiki by the way. I'm not seeing yet what your plugin did differently? 

[Saq Imtiaz]

@yoni did you follow the instructions and rename the file from .JSON to .roam before importing?

[Yoni Balkind]

Ahh apologies, I had renamed the file but it was still saved as a JSON file in its properties.. I re-saved the file correctly and now it does seem to have worked. I am just checking the formatting and will report back shortly. 

[Yoni Balkind]

Okay so this is looking promising! Thanks @Saq

The import treats each child as a bullet point. This mimicks the way a page looks in Roam, but the idea of publishing on TW is that it reads more like a normal post, so I propose that you treat each child as just a paragraph (however nested children can still be treated as bulleted lists). 

I now see an additional problem. This might be too unique to my own use-case to warrant you catering for it, but let me tell you anyway in case you think its a broad enough problem..

Roam has no "tag" field per se.. Every internal link is a defacto tag.. So for example, the screenshot below shows a Roam note that has 4 different internal links. For my purposes, the two links in the top line operate as tags. I use these tags to help me locate pages that I've tagged as "evergreen notes" and "ready for publish".

The 2 links near the bottom of the page are simple internal links to cross reference other posts. Roam doesn't know the difference between all of these links, I just use them in such a way that the ones on top are "tags" for my purposes. 

Ideally, I'd want the import to ignore the two tags at the top. They are internal references that aren't needed in TW. I suppose IF you were to tackle this in your plugin, you could have a setting to ignore hashtag items. Or perhaps you could stipulate that should a user want to have certain lines ignored they should precede that line of text with "meta:" or something to that effect.. 

Final point, and again this might be my personal problem and not applicable to your plugin, but my JSON file has 1000 plus pages and I need a way to only import those pages that have certain tags (#ready for publish).. There is a probably some tool somewhere that would allow me to do this directly to the JSON file, but perhaps its something you want to consider for your plugin. 

[Rizwan Ishak]

I was having the same idea as Mark - treat each child as a separate tiddler. Because that is how usually things are in TW5 - smallest semantic unit and all. But I guess each child is a bullet point? 

JSON mapping would be much easier if you have a definite structure. So if the children doesn't have another level of children, it makes writing a generalized importer much easier.

--
You received this message because you are subscribed to a topic in the Google Groups "TiddlyWiki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tiddlywiki/cMsvhrOgoBg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/925db46b-24ad-4846-ade3-b56637aa173a%40googlegroups.com.

[Yoni Balkind]

The structure is definite in my opinion. A page in Roam should equate to a Tiddly. All the bullets (ie childs) in a Roam page should be grouped as normal text within the Tiddly. Insofar as child elements are nested in Roam, those should just translate to nested lists within the Tiddly. 

@Saq's implementation is fundamentally correct I would say. 

I'm quite confident that vast majority of people importing from Roam would expect it to work this way

[Saq Imtiaz]

@Riz: the number of nested children seems unlimited. In Roam the smallest semantic unit seems to be a bullet point, which is member of the childrens array in the JSON.

I think this is a case of needing to figure out what is most intuitive in terms of the end result the user would want, instead of what feels semantically the most correct. I suspect Roam users wont really want 5 pages to turn into 100 tiddlers in TW, even if we did create a parent tiddler for each page that transcluded all the tiddlers created from the bullet points. 

The tricky thing is, do you turn the children into bullet points, paragraphs? You could even make that a global configuration option but then I suspect users would want to decide on a per tiddler basis... 

The tag handling is also difficult because Roam doesn't really have tags, I think what we are referring to as tags are just links. So should we really be trying to create tags from [[link]] or #link? Or do we assume that the links/backlinks that will be generated are the proper representation for this?

@yoni I understand your use case and requirements. My approach is to start off step wise and try to put something together that would be useful to others wanting to import from Roam as well. That may mean that there wont be one "plugin" that does what you need, but rather a sequence of steps/plugins working together. 

To unsubscribe from this group and all its topics, send an email to tiddl...@googlegroups.com.

[Saq Imtiaz]

@yoni could export and share a few pages that contain both longer paragraphs and nested lists? I think Roam lets you export one page at a time as well.

Thank you.

[Rizwan Ishak]

@Saq. My thought was - if we turn children to just paragraphs, won't we lose all the metadata associated with it?

To unsubscribe from this group and all its topics, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/456b8926-cc41-4293-b9a2-bd686fbfac42%40googlegroups.com.

[Anne-Laure Le Cunff]

Oh, this is exciting! Thanks so much for looking into it, Saq.

I think this is a case of needing to figure out what is most intuitive in terms of the end result the user would want, instead of what feels semantically the most correct. I suspect Roam users wont really want 5 pages to turn into 100 tiddlers in TW, even if we did create a parent tiddler for each page that transcluded all the tiddlers created from the bullet points. 

I agree with this. I think one page in Roam = one Tiddler in TiddlyWiki.

The tricky thing is, do you turn the children into bullet points, paragraphs? You could even make that a global configuration option but then I suspect users would want to decide on a per tiddler basis... 

For this one I disagree with Yoni. It should be bullet points in TiddlyWiki as well, to stay as close as the Roam experience. It will feel more intuitive. Very easy in TiddlyWiki to remove bullet points by selecting all and removing the list styling.

The tag handling is also difficult because Roam doesn't really have tags, I think what we are referring to as tags are just links. So should we really be trying to create tags from [[link]] or #link? Or do we assume that the links/backlinks that will be generated are the proper representation for this?

Here I think, again in the spirit of staying as close as possible to Roam, to not even consider the question of tags. Whether #Title or [[Title]] these should all be converted to [[Title]] in TiddlyWiki. It may seem extreme but I think that's the cleanest way to respect the original structure from Roam and to keep things simple.

[Yoni Balkind]

@Saq see attached (with accompanying screenshot below from Roam interface)

The meta data attached to each "child" is invisible on the Roam front end, and I dont think users would expect this to survive the import.

As you can see from the above screenshot, the whole pages consists of bullet points. @Saq, you replicated this in your plugin which is a perfectly reasonable approach. 

But I think that the reason for bullet points in Roam is because each "block" is drag-able and can be referenced and behaves like a semantic unit. Since TW does not have this same granularity, I dont think it is required to display everything as bullet points. However, the nesting should be persevered in my opinion. To me personally the most logical behaviour is that the top-level child items are not bullet points, but their children are bullet points so that you can maintain the nesting in a way that matches the nesting in Roam.. So for my attached note, the first 3 children will not be bullet points, but the 4th child (beginning "Social Media..") should be a bullet point, and the next block a sub-bullet point. 

[Saq Imtiaz]

@Riz As far as I can tell the only metadata associated with the children is the created and modified dates. I can't really speak for how useful that information is and whether its worth the trade off for creating multiple tiddlers per page.

Roam users: do you need to know when each bullet was added or edited? Can you imagine a situation where you might? Does Roam even let you see this information?

Also, how do transclusions work in Roam? I have heard it mentioned but can't see any syntax mentioned for how to transclude content from one page to another in Roam. 

Doesn't dragging and dropping just copy or move the bullet point?

On Tuesday, May 5, 2020 at 1:08:14 PM UTC+2, Riz wrote:

@Saq. My thought was - if we turn children to just paragraphs, won't we lose all the metadata associated with it?

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/456b8926-cc41-4293-b9a2-bd686fbfac42%40googlegroups.com.

[Yoni Balkind]

For this one I disagree with Yoni. It should be bullet points in TiddlyWiki as well, to stay as close as the Roam experience. 

I think my proposal of top level item not being bullet points, but subsequent items being bullet points is best of both worlds. It respects the nesting levels but allows user to import a flat "essay" if they had no nesting.. But Anne-Laure's approach (which is what @Riz did in his prototype) is understandable and workable

...in the spirit of staying as close as possible to Roam, to not even consider the question of tags. Whether #Title or [[Title]] these should all be converted to [[Title]] in TiddlyWiki.

Agreed. But with some additional thoughts about how to use tags as reflected in my earlier post (the one beginning "Okay, this is looking promising..") 

[Saq Imtiaz]

One caveat with the current approach is line breaks in bullet points. The current quick hack replaces line breaks with <br>, which is ugly to say it midkly. Any thoughts on alternatives here are welcome.

The issue with paragraphs in TW is also that the automatic paragraph handling is odd and a little inconsistent, and paragraphs can't contain lists so there will be a not so insignificant gap between them.

[Anne-Laure Le Cunff]

Roam users: do you need to know when each bullet was added or edited? Can you imagine a situation where you might? Does Roam even let you see this information?

Nope, I don't care when a bullet point was added or edited.

I think my proposal of top level item not being bullet points, but subsequent items being bullet points is best of both worlds.

I think it feels more natural to just import it exactly as it looks in Roam, especially as your proposal assumes people mainly write essays in Roam, when really my bullet points are often actual bullet points, and I would be annoyed to have to add them back. But I'm so excited to have a solution at all that I'm not fussed if we drop the top-level bullet points even if I find that arbitrary.

Whatever you decide - this is exciting!

[Yoni Balkind]

how do transclusions work in Roam

You see the UID in the JSON file that each child item has. So in roam if you type that uid surrounded by double round brackets like this

((CH8ti2MnS))

Then Roam will transclude the contents of the item (The users doesn't actually know the UID but when you drag an item while holding a certain key combo then the effect is that the UID gets created with double brackets as shown above).. 

In the JSON file I presume you will simply see the UID surrounded by round brackets so will be difficult for you to implement the transclusion especially if the user does not import the whole DB.. I cant speak for all users, but I would not expect you transclusion to work in the import. I suppose in an ideal world you could make it work somehow when the UID is found, but just display the raw UID in brackets when the UID is not found... 

[Saq Imtiaz]

@anne-laure @yoni

Why not just import in markdown from Roam? With the markdown plugin installed from the plugin library, you can import markdown and it should just work.
This seems reasonable since content from Roam is going to be using markdown syntax for other things anyway. You can still get things like title autocomplete and support for widgets and [[link]] in TiddlyWiki using markdown if you need it.

What is to be gained by going the JSON route, Roam -> JSON -> TW ?

Also, it was instructive to compare the markdown and json respresentation of a page from Roam. In Markdown, a page is just a bullet list, and not separate pages.

Markdown syntax also solves the problem of line breaks in lists.

I can understand that Yoni might want to go via JSON to work in some extra tweaking of the text, but is there an advantage for the average user wanting to import content to TW from Roam?

Regards,
Saq

[Mark S.]

BTW, do we know if roam titles are unique?

[Yoni Balkind]

@Saq the problem with markdown import is that (as far as I can tell) we can only import one markdown file at a time. Roam exports each page as its own markdown file, so I'm not seeing a way to import in bulk? 

@Mark S yes Roam titles are unique but also case sensitive. So you can have two titles with the same name if they have different capitalisation. 

[Saq Imtiaz]

@Yoni you can drag and drop multiple MD files to import in bulk.

[Stian Håklev]

Hi all,

I'm not using TiddlyWiki, but I figured this would be useful to you. I've developed a tool called roam-export, which processes Roam JSON exports and does a lot of neat things, including resolving embeds, block-embeds, queries (almost done), exporting a subset of pages in Markdown etc. It's quite messy code right now, but I'm happy to help out or improve it if people need specific things - basically tell me which format you want the data in :) 

https://github.com/houshuang/roam-export

I'm using it for a Gatsby digital garden, with more details here: https://notes.reganmian.net/about

PS: I won't be monitoring this mailing list, so please email me directly sha...@gmail.com or @houshuang on Twitter if you want to discuss. Good luck - love the stuff you guys are all doing with TiddlyWiki! 

[Anne-Laure Le Cunff]

Thanks so much for sharing, Stian! I'll give a try. I know where to find you :)

[Alexander Kogan]

Hello,

I used your extension to import my notes from Roam. It worked quite well, but there are several things, that didn't work as expected. I tried to find the GitHub repository to enter an issue, but couldn't find it. Do you have one where I should post this?

For now I'll put it here:

One of the issues is quite severe, because I'd have to check and correct every tiddler manually. The bullet points are not indented correctly. For example if I have in Roam a structure like this:

* Meeting

** Project1 is fine

*** Participant1, Participant2

** Project2 is late

*** Participant3

** Project3

** Project4

*** Participant4

*** working

If I import with your TW I get:

* Meeting

** Project1 is fine

*** Participant1, Participant2

*** Project2 is late

**** Participant3

**** Project3

**** Project4

***** Participant4

***** working

So the indentation is not reset correctly.

The other issues are probably with how I used Roam which doesn't quite work for TiddlyWiki:

1. the #tags are not converted to links and TW also doesn't recognise them as references with blinks from Stroll (https://giffmex.org/stroll/stroll.html). That's solvable by tweaking Stroll though I guess?

2. In Roam there is a possibility to link pages in titles like this: [[[[Windows]] 10]]

That would create two pages:

- Windows

- [[Windows]] 10

Where you can click on Windows in the title of Windows 10 which would bring you to the Windows page.

But I didn't find a way to do something like this in TW yet. Fortunately I didn't have many titles like this and could resolve those manually.

I hope this feedback helps you.

Kind regards

Alex

On Tuesday, 5 May 2020 at 05:21:55 UTC+2 saq.i...@gmail.com wrote:

@Yoni @Anne-Laure

Here is something to play with for importing from Roam to understand what is possible, though please don't expect too much as I've only had about ten minutes to work on this.

https://saqimtiaz.github.io/sq-tw/roam-import.html

How do tags work on Roam? Does it have tags? As far as I can tell from a quick glance, #tag is actually just a link to [[tag]]

should #xyz be turned into a tag on that tiddler?

Regards,

Saq

On Monday, May 4, 2020 at 9:23:29 AM UTC+2, Anne-Laure Le Cunff wrote:

I was looking at the exact same thing yesterday and couldn't figure it out as the structure is quite different. Attaching two sample JSON files for people who want to have a look.

On Monday, May 4, 2020 at 7:54:05 AM UTC+1, Yoni Balkind wrote:

Hi All 

My intended workflow is to draft notes in Roam, and then if I deem certain notes worthy of publish, I publish them in my public-facing TiddlyWiki. 

However these "published" notes might became quite voluminous and if I update them in Roam I want them to update in my TW as well as it would be too laborious to manually maintain two versions of same notes. 

It seems to me that the way this might be achievable is to do regular JSON exports from Roam, and then to import these into TW, overriding existing Tiddly's with the same name.

The challenge is as follows:

• Roam does not allow me to export pages with tag X. So I need a mechanism to filter which pages get imported into TW as the JSON file will be full of pages I dont need.
• The JSON structure is different between Roam and TW, is there a workflow to edit the JSON structure of the Roam file to make it match TW? I suppose a simple search and replace in Notepad++ ? Or is it more complex than that? 

[Saq Imtiaz]

@Alexander this was a quick 10 minute hack that was ultimately abandoned due to a lack of interest and response from participants. There is no intention now of developing this further.

Your best option for importing from Roam is to do so with Markdown files, as discussed earlier in this thread.