Sandstorm Doc Deployment Update

[Adam Bliss]

On the last Office Hours call, I promised an update on the Docs situation. Here it is.

1. Asheesh (paulproteus@github) was kind enough to give me push access to the gitwebpages grain on alpha.sandstorm.io which hosts the content for https://docs.sandstorm.io . Jacob also has push rights to it. So in theory, he and I are both now able to push manual spot-fixes to the live docs.

2. However, it seems that nobody has ever pushed such manual spot-fixes before; all previous pushes appear to be autogenerated from the 'generate.sh' script in the sandstorm repo. 

3. Also, there appears to be no staging area for the docs, and since Asheesh didn't send me the .spk for his gitwebpages app, I wasn't able to create a new grain to push to as a staging area. For this and the previous reason, I'm hesitant to push to the live site for fear I might break something.

4. Asheesh said that he thinks the gitwebpages spk that's serving the live docs is up-to-date with the code in https://github.com/paulproteus/gitwebpages-sandstorm . However, I was unable to build a working spk from that code out of the box, due to some apparent bit-rot. I believe I have fixed those issues and will send a pullrequest.  In the meantime I have re-keyed, revved the version, and uploaded the spk here, in case anyone wants to try it:

https://drive.google.com/file/d/1K7ki_BQ7w-2uMZDXkp5IEX6tRRoKRrv4/view?usp=drivesdk

5. I used that spk to create a new grain on my own sandstorm instance, to serve as a staging-area to test out doc generation/push/hosting: 

https://7125z831x29zsn814csg.rapidraven.sandcats.io:6443/

(uptime not guaranteed, as this is running on my experimental Intel Compute Stick server in my house).

6. The sandstorm repo contains a doc-generation script, and instructions to run it. They also did not work for me out-of-the-box, apparently due to some more bitrot. I believe I have fixed the bitrot locally and will send a pullrequest. (There are a few lingering warnings I still want to investigate). In the meantime, I have deployed the generated docs to the above-linked staging grain. They look generally OK to me. If people generally agree, I can push this new set of generated docs to Asheesh's grain, and thus refresh the live docs to be current with sandstorm's master branch. Let me know if you feel strongly either for or against this action.

7. In addition to the gitwebpages grain, there's apparently a separate Davros grain for hosting static images. (E.g. the faq links to https://alpha-evgl4wnivwih0k6mzxt3.sandstorm.io/works-fine.png ). It looks like Asheesh was manually uploading new images here. I've asked for access. I haven't yet verified whether there are any new images that need to be uploaded in the current doc refresh.

8. One possible next step, now that I kinda understand how this all works, is to set up a github pipeline to generate & deploy the docs automatically. A second possible next step, discussed on the call, was to either pull the latest gitweb code into the gitwebpages fork, or else integrate the gitwebpages changes back into upstream gitweb sandstorm spk (Asheesh explained that this was undesirable because gitwebpages uses the deprecated webkey approach). A third possible next step is to rip out the gitwebpages stuff entirely, and simply push all the generated content (html and images) directly to the static davros grain. (I still don't really understand the point of using gitweb to publish static content. Is it supposed to make diffing the generated docs easier? We could have a separate repo for that, pushed to as part of the deployment pipeline, but the content doesn't have to get served out of there. Is it supposed to make deploying to the new site as easy as "git push"? That goal seems frustrated by the need to separately upload images to a davros grain; and in any case, once the github pipeline is set up, a simple "git push" will still be sufficient to deploy. Is there some other benefit that I'm missing?) I am happy to hear any and all thoughts regarding these or other next steps.

Cheers,

--Adam

[Jacob Weisz]

Some notes:

- Your test version of the docs is missing the link back to sandstorm.io. There is a custom theme in the sandstorm docs repo which adds it, and it would be good to get that adjustment re-added to your updated version.

- Not related to the basics of getting docs moving again, but I'd very much like to remove Google Fonts, as we've done with the core Sandstorm website. If the theme looks okay with Source Sans, we could pull https://sandstorm.io/fonts/sourcesans.css and not have people downloading multiple copies of cached Source Sans as they transition between the main website and docs.

- I do like having GitWeb able to act like GitHub Pages for people who'd prefer to manage such things with Git, and easily transition from a proprietary method to a modern one. While not as relevant for a smaller repo, for a larger amount of content, Git is pretty nice versus Davros, which prefers WebDAV for non-GUI access, which is somewhat broken. I am unsure what the programmatic way to push to Davros would be at this juncture.

- Speaking of that weird Davros grain hosting images... Why can't the images also be on the GitWeb Pages instance? Can we fix that? What benefits were there to hosting static images via a separate grain?

- If we can add GitWeb Pages functionality into the main GitWeb app, rather than publishing GitWeb pages separately, it should be implemented like Davros, where there is an Enable button for static publishing. If that's not practical at this time, IMHO, we should publish GitWeb Pages so people can find and use it.

--

  Jacob Weisz

  in...@jacobweisz.com

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CAEAeJWytdicHsqkoz9Rfw8WMgFPgMygSs5Lngxy%3DRb0xYmRP2Q%40mail.gmail.com.

[Adam Bliss]

Jacob, thanks for your email. Responses inline. 

On Wed, Jan 15, 2020, 16:38 Jacob Weisz <in...@jacobweisz.com> wrote:

Some notes:

- Your test version of the docs is missing the link back to sandstorm.io. There is a custom theme in the sandstorm docs repo which adds it, and it would be good to get that adjustment re-added to your updated version.

Good catch! Fixed. This was caused by a breaking change in the upgrade from mkdocs 0.17 to mkdocs 1.0. I read through their release notes and, as far as I can tell, I have handled all the breaking changes.

- Speaking of that weird Davros grain hosting images... Why can't the images also be on the GitWeb Pages instance? Can we fix that? What benefits were there to hosting static images via a separate grain?

I asked Asheesh and he said (1) there might be some issue with MIME types and (2) pushing images to a git repo tends to bloat it over time (since images do not delta-compress well). 

--

I have opened https://github.com/sandstorm-io/sandstorm/pull/3185 to fix the docs and https://github.com/paulproteus/gitwebpages-sandstorm/pull/1 to fix gitwebpages-sandstorm. I also uploaded my 0.0.7 release of gitwebpages-sandstorm to https://github.com/abliss/gitwebpages-sandstorm/releases .

So, any votes (yay or nay) on whether/when I should push this doc refresh to the live doc-serving grain? 

https://7125z831x29zsn814csg.rapidraven.sandcats.io:6443

Cheers,

--Adam

[Jacob Weisz]

A git repo shouldn't be bloating because of images unless those images are changing/being replaced, should it? Like, if you update an image, the git repo is going to contain the original and the new one. But a bunch of images nobody ever replaces aren't costing us anything extra.

How many images are we talking about? Are all of them even useful?

--

  Jacob Weisz

  in...@jacobweisz.com

[Jacob Weisz]

- Note that right now your grain is loading a home page that starts with:

This directory started out as a snapshot from the github.com/mkdocs/mkdocs repository on 2016-01-29.

It has the following changes:

• "Docs home" vs. "Project home" separation, per https://github.com/sandstorm-io/sandstorm/issues/1405
  

- Also... search stopped working. Something about pulling the modified theme back in might've re-introduced that breakage.

--

  Jacob Weisz

  in...@jacobweisz.com

--

You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/72da8c09-c1a8-48b2-8927-0e41681e1287%40www.fastmail.com.

[Adam Bliss]

Ugh, looks like you're right thanks. The theme was broken but since it wasn't being applied, I didn't notice. I'll try to update the theme.

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/d8a7a942-1b22-4531-8156-8cd1177a15c6%40www.fastmail.com.

[Dan Krol]

I'm curious what the problem is with Google fonts? Aren't some of them freely licensed?

To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CAEAeJWym%3DFiGZYwtR52snsALtXCtkuxkeZ1d1N3MVJsg8RTCxw%40mail.gmail.com.

[Jacob Weisz]

It forces my computer to make a call out to a Google service. Which I generally block client side of course.

 

Sent from my Windows 10 device

[Dan Krol]

Ah, well you can always download them and serve them locally.

[Jacob Weisz]

That’s basically how we switched sandstorm.io over. At the same time we unified the font choice a bit across multiple sandstorm.io sites. Docs wasn’t included there, but I'd like to either use the same fonts on Docs or host the fonts Docs is using on the same server as sandstorm.io.

 

Now that it looks like we have a refreshed and working Docs site in the pipeline here, I might test some PRs on changing that over myself too.