js-git

[Alexander Praetorius]

i'm following tim and the js-git project already for a long time...
https://github.com/creationix/js-git
in feature goals here: https://www.bountysource.com/teams/js-git/fundraiser
it says:

• Clone remote repositories to local storage over http, git, or ssh.
• Make and commit local changes offline.
• Manage tags and branches offline.
• Push changes back up to remote repositories.
• Serve git repositories over http, git, or ssh

is it already possible to checkout a repository from within a browser, say github, add a commit and push back the changes to github?
Are there any howtos, tutorials or other learning materials around jsgit?

on nodeschool.io i found https://github.com/jlord/git-it, but it seems to be just about normal git.

[Tim Caswell]

On Monday, December 29, 2014 9:02:19 AM UTC-6, serapath wrote:

i'm following tim and the js-git project already for a long time...
https://github.com/creationix/js-git
in feature goals here: https://www.bountysource.com/teams/js-git/fundraiser
it says:

• Clone remote repositories to local storage over http, git, or ssh.

The problem here is the protocols allowed by browsers.  I have fully implemented clone over https (smart git http protocol), but github refuses to enable CORS headers on their git endpoints or even tell me why they won't turn them on.  I've been asking literally for years now.  The normal git tcp protocol and ssh protocol require access to raw tcp, which a browser can't do alone.
 

• Make and commit local changes offline.

Yep, this works great as can be seen in the https://tedit.creationix.com/ website that works 100% in the browser.

• Manage tags and branches offline.

This also works great, but using low level APIs where the user manually creates the refs and tag objects if the tags are annotated.  I highly recommend reading the git internals chapter in the git book to understand how js-git currently works.  Most the APIs work at this low level currently. http://git-scm.com/book/en/v1/Git-Internals  I have not implemented diff and merge yet, everything works in terms of the git object graph.
 

• Push changes back up to remote repositories.

This is where I got stuck.  Since I can't speak any of the git protocols in a browser without involving an external proxy (https://github.com/creationix/git-proxy), I'm not sure how to proceed.  There are two major problems with using a proxy.

 1. My proxy will act as a man in the middle during your secure conversation with github.  I'll be speaking wss (websockets over tls) to the browser and https (http over tls) to the github server, but I'll be providing my own tls credentials to the browser and be able to see all traffic in the clear, including your credentials.  I don't want the ability to see someone's github credentials.  If the proxy was for read-only clones using the git protocol, it would be fine, except for point #2.

 2. My proxy will act as a bottleneck and cost me resources and make your experience dependent on my ability to keep it up.  It's simply much better for github to just add the CORS headers so that the browser can speak directly to github.  But they won't, so I got stuck here.  I ended up running out of time and money and had to work on other paying projects.

I did have an idea to implement TLS in the browser in pure JavaScript and letting the proxy only make TCP (without tls) connections.  It will still be a bottleneck and point of failure, but the proxy is simple to setup and many people could setup proxies to share the load.  Since the encryption would be end-to-end, the proxy in the middle wouldn't be able to see anything.  This can probably be done with the forge project combined with my http code.  Implementing http on top of a TLS stream in pure javascript is trivial compared to git pack protocol and tls protocol. 

I even had a crazier idea to implement ssh in pure ssh using the rsa primitives in forge.  You could generate ssh private key in your browser, upload it to github using the rest api or manually, and then speak ssh to clone and push using the pure js ssh implementation. But I ran out of time, sorry.
 

• Serve git repositories over http, git, or ssh

I have a couple of proof-of-concepts around this area, but since if you're creating a server, you can already access real git, it was lower priority.
One solution is to use the git-fs backend to js-git which uses the same filesystem format as real git.  Real git can do the network stuff and js-git will give you a read-write interface to the git repo locally.
 

is it already possible to checkout a repository from within a browser, say github, add a commit and push back the changes to github?
Are there any howtos, tutorials or other learning materials around jsgit?

I do have another route using the js-github project. It implements a js-git compatable object, but uses github's proprietary REST APIs to sync the local database with the remote git repos.  It's not as efficient and will likely cause you to reach your rate limit real fast if you try to do a full clone since every object is a separate http request.  It does, however, enable new workflows where you only download files on demand and treat the remote repo as the local database.  In this workflow, there is no clone, but it's more like a nfs or samba mount.
 

[Alexander Praetorius]

Any chance that you might continue the project in the future?
Would be quite awesome to host a jsgit repo on a server and check it out from within the browser, change it and push back.
If there was a nice step by step tutorial, i would for sure try to use it to build something and give feedback on it.

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/6ad5ec30-f081-4347-ada6-b46447fa5b95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Caswell]

If you're willing to speak a custom protocol, then it's pretty simple to host the git repo over secure websockets.  A simple protocol that allows remote execution of loadAs, saveAs, and the other js-git APIs would work great. You could authenticate using http headers (or add an authentication API to the websocket messages).

Another idea would be to tunnel the git tcp protocol over websocket and use the wss protocol for authentication much like git over https works.  You could even just use real git to serve the repo over https and add the CORS headers yourself.

I don't know when I'll be working on this again.  I have a full-time job now working on luvit stuff and am expecting my 4th child any day now.  I'll gladly guide and mentor anyone wanting to take on this work.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAN5%2BLUsxen3%3DwJsjOC0Eih_7mB1Ngo%2Bq2VcHteav_nimcdSMwQ%40mail.gmail.com.

[Alexander Praetorius]

Regarding: "You could even just use real git to serve the repo over https and add the CORS headers yourself"
Would that be possible with using jsgit to serve a repo?

Is it possible to "npm install jsgit", then "require('jsgit')" and have my server listen on a specific port in order to serve a repo and receive updates from a browser?
In other words:

(probably the analogy is lacking, but instead of configuring and using some http server, node can be used to build a custom one)

would it be possible to use jsgit to build my "custom git" which includes maybe some auth mechanism and an easy way to manage git hooks? maybe jsgit could be used to build a custom "version control" system?

(maybe that is not desirable - but if there are reasons for that, i would be interested to learn about them)

Otherwise, i can say, that solving the above, would be a very very interesting project that i would like to see happen and i would like to try to help if i can.

BUT: for now, i feel all this is a bit beyond my current skill level. From what you said above, i cannot comprehend how much effort it would be to make progress in this area nor do see if i would be able to learn and program it for real.
It would probably need a lot of guidance, ...kind of step by step instructions and how those steps relate to each other (the big picture).
...it's only afterwards, that i will be able to see if it would be possible for me to program that at all :-)

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAGkHjAW52dwi2EaCh23%2BYdSTRJKNzH0%3DVhV0LRDhAworsDzdDA%40mail.gmail.com.

[Tim Caswell]

I'm afraid js-git isn't far enough on the node side to have a pre-made server. The APIs are pretty low-level.

But setting up real git and adding CORS headers using something like a node or nginx proxy is pretty easy.

Just setup nginx to use the cgi program built into git <https://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html> and add cors headers into your nginx setup <http://enable-cors.org/server_nginx.html>  Then I do have a js-git sample that runs in a browser and speaks git smart http using XHR. <https://github.com/creationix/clone-test/blob/master/src/main.js>

The part I never finished is pushing back using pack protocol though.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAN5%2BLUvZX%2BU3NOyCanFGVob9WBUo6CTUizYkTozUooBHZrZMeQ%40mail.gmail.com.

[zladuric]

If it is solving your problem you want to achieve, did you hear about gitlab? Good a full github-like solution. If it is simple serving of repos, there are a few tutorials on the net about nginx for this.

If you want to solve the issue with node and JavaScript though, then I guess best bet is to clone js-git and continue where it stopped.

[P. Douglas Reeder]

On Monday, December 29, 2014 11:52:02 PM UTC-5, Tim Caswell wrote:

On Monday, December 29, 2014 9:02:19 AM UTC-6, serapath wrote:

Clone remote repositories to local storage over http, git, or ssh.

The problem here is the protocols allowed by browsers.  I have fully implemented clone over https (smart git http protocol), but github refuses to enable CORS headers on their git endpoints or even tell me why they won't turn them on.  I've been asking literally for years now.  The normal git tcp protocol and ssh protocol require access to raw tcp, which a browser can't do alone.
 

Are there other git hosters that do enable the CORS headers?

[Tim Caswell]

I'm not aware of any others that have core.  I asked all the major players a while back.

---

From: P. Douglas Reeder
Sent: ‎12/‎31/‎2014 4:34 PM
To: nod...@googlegroups.com
Subject: [nodejs] Re: js-git

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/2de35e20-cd7f-4a2c-a8dd-5e06e99bf0d2%40googlegroups.com.