Is there a way to download a snapshot of all clojuredocs content?

122 views
Skip to first unread message

Andy Fingerhut

unread,
Feb 23, 2011, 12:43:01 PM2/23/11
to ClojureDocs.org Discussion Group
Why I ask: I've just learned about clojuredocs.org and like what I
see. I found the API, and Lee's cd-client library on github.com that
lets someone show examples, comments, see also's, etc. from the live
web site. Very nice.

I thought it might be nice if snapshots of clojuredocs.org could be
downloaded in some form (e.g. a jar via Maven/Leiningen), and then cd-
client could be modified slightly to permit showing information based
upon that locally stored snapshot.

This would be in addition to all the cool things available now, not
instead of. Just something a person can refer to locally in case they
don't currently have internet access, or they pay based on usage, etc.

Thanks,
Andy

zkim

unread,
Feb 24, 2011, 10:57:20 PM2/24/11
to ClojureDocs.org Discussion Group
Andy-

I could probably throw together a nightly dump of some sort, something
like:

[{:ns "clojure.core"
:name "map"
:doc "..."
:version "1.2.0"
:examples [{:content "..."}
{:content "..."}]
:comments [{:content "..."}
{:content "..."}]}
:see-alsos [{:ns "clojure.core"
:name "first"}]
{:ns "clojure.core"
:name "reduce"
... }]

What were you thinking as far as modifications to cd-client to support
switching
between web / local?

-Zack

Andy Fingerhut

unread,
Feb 25, 2011, 2:40:29 AM2/25/11
to clojure...@googlegroups.com
I had not thought through the details on cd-client modifications yet,
but my initial thoughts are something like that when you load up cd-
client, it starts in either web / local mode based upon some saved
setting, e.g. in a $HOME/.cdclientrc file or something like that.
Without anything to specify otherwise, it would be in web mode.

After that, the user can change it to local mode with a function call,
e.g.

(cd-client.core/set-local-mode! "/home/me/my/local/copy-of-
clojuredocs.jar")

It wouldn't have to be a jar file, but it certainly could be. At that
point, future queries are local, unless you switch it back to web mode
with a similar function call like:

(cd-client.core/set-web-mode!)

I'm open to suggestions, though.

Andy

Andy Fingerhut

unread,
Mar 4, 2011, 3:26:54 AM3/4/11
to ClojureDocs.org Discussion Group
I have some local modifications to the cd-client.core library that
support:

(1) creating a snapshot of clojuredocs.org examples, see-alsos, and
comments remotely, via the API. Sorry, I probably should have asked
permission before sending thousands of API requests to the
clojuredocs.org server -- that is definitely not the most efficient
way to do this, but I wanted a big data set to test with after trying
out some small ones, and it did help me find a few more special
characters in symbol names that were not working with cd-client.core
that I was able to fix. See below for some (minor) strange behaviors
in the API that I don't know how to fix.

(2) Lets you switch between "web mode" and "local mode" for looking up
examples, see-alsos, and comments, as longs as you have a "clojuredocs
snapshot" stored locally. It is a bit of a hack right now, but I plan
to polish it a bit more. I haven't yet made searches work in local
mode, but there is no fundamental problem there other than deciding
exactly what kind of search strings you want to support, e.g. exact
match only, or regular expressions, etc.


I can't seem to find a string to put after the base URLs for the
examples, see-also, and comments API that can get back proper results
for these symbols:

..

I've tried _dot_dot .. and %2E%2E _dot_dot gets the closest but
returns no examples, even though this page has one:
http://clojuredocs.org/clojure_core/clojure.core/_dot_dot

/

I've tried _ / %2F but never get back any of the examples that
appear here: http://clojuredocs.org/clojure_core/clojure.core/_

+

I've tried + %2B but never get back the examples that appear
here: http://clojuredocs.org/clojure_core/clojure.core/+

Thanks,
Andy

Andy Fingerhut

unread,
Mar 7, 2011, 5:48:14 PM3/7/11
to ClojureDocs.org Discussion Group
I've polished things up a bit, and the results can be examined on my
github fork of cd-client, here:

https://github.com/jafingerhut/cd-client

The current behavior is exactly the same as the latest version of
dakrone's cd-client. My version adds the ability to switch to "local
mode" by specifying a file containing a snapshot of the clojuredocs
data. There is a sample small file test/partial-snapshot.clj in my
github repo that only contains clojuredocs info about 39 symbols
(those whose names contain the substring "let" -- just a small subset
I picked for testing purposes). Below is a brief example interaction
at the REPL.

I can send a pull request for these changes if you are interested.

user => (require '[cd-client.core :as c])
nil
user => (c/set-local-mode! "test/partial-snapshot.clj")
Read info on 39 names from local file
test/partial-snapshot.clj
nil
user=> (c/pr-comments let)

======================================== vvv
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Nota Bene: `let` in Clojure is like `let*` in Scheme -- each init-
expr has access to the preceding binding forms. (There is also a
`let*`, but it is more or less `let` without destructuring, and in
fact is the underlying implementation.)

*** Last Updated: 2011-02-07 01:52:53.0

======================================== ^^^
1 comment(s) found for clojure.core/let
nil


Andy
Reply all
Reply to author
Forward
0 new messages