A graph of the whole world....

[Nathann Cohen]

Hello !!!

Starting from a kind of troll related to the next-to-be tour for
Graphs in Sage ( I wanted to color the map of western europe, but the
majority of Sage's users do not live there ), I tried to build a graph
of the whole world by using your dear CIA's data ( available freely on
their website )

Do you think it would be interesting to add it to the graph generators
in Sage ? It could be available as graphs.WorldMap() :-)

If so, I have a small problem : how to include it ? I generated it by
parsing again and again several files, then building the final graph
through Python, but the only way I have now to share this graph is to
write it by g.write().

This Graph contains labels for the countries ( their names ) but also
their positions...

Nathann

[Jason Grout]

How big is the graph6 or sparse6 string? Can you just store that, along
with the vertex label and position dictionaries?

Jason

[Nathann Cohen]

5234 characters.. This may be the best solution indeed :-)

How can I know after that, though, the matching between vertices and
labels ?

[Jason Grout]

Nathann Cohen wrote:
> 5234 characters.. This may be the best solution indeed :-)
>
> How can I know after that, though, the matching between vertices and
> labels ?
>

What is the autmorphism group? You may be able to uniquely identify each
vertex... :)

I guess that's a bit harder question. It looks like sparse6 format just
uses the order of vertices returned by self.vertices(). The output is
always sorted, so you can just store the mapping by storing

map=dict(enumerate(self.vertices())

Then map[sparse_vertex_number]=original_vertex_name

(I haven't tried it, though...)

Another thing you could do is store it in the graph database that we
already distribute with Sage, or in an optional database that a user
could tap into. Both of those are a bit more work, but might open up
interesting future possibilities.

Thanks,

Jason

[William Stein]

On Thu, Aug 20, 2009 at 2:18 AM, Nathann Cohen<nathan...@gmail.com> wrote:
>
> Hello !!!
>
> Starting from a kind of troll related to the next-to-be tour for
> Graphs in Sage ( I wanted to color the map of western europe, but the
> majority of Sage's users do not live there ), I tried to build a graph
> of the whole world by using your dear CIA's data ( available freely on
> their website )
>
> Do you think it would be interesting to add it to the graph generators
> in Sage ?

Yes.

> It could be available as graphs.WorldMap() :-)

Yes.

>
> If so, I have a small problem : how to include it ? I generated it by
> parsing again and again several files, then building the final graph
> through Python, but the only way I have now to share this graph is to
> write it by g.write().

Why not just include a pickle of it? Make sure there is a doctest
that involves creating it, so that we can't release a version of Sage
if this pickle breaks.

And if you say you can't pickle it, then something is wrong with
pickling graphs, and we have a bigger problem to worry about!

In case you don't know, you can use pickle as follows:

sage: save(G, 'g.sobj')
sage: G = load('g.sobj')

William

>
> This Graph contains labels for the countries ( their names ) but also
> their positions...
>
> Nathann
> >
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Jason Grout]

Can we include binary data files with Sage? I thought for some reason
this was frowned upon.

If we can include pickles, then I agree, pickling it would be the
easiest option.

Jason

[William Stein]

It is frowned upon, but only if they can't easily be removed, and/or
easily recreated from some text form.

I've been meaning to add a script that takes an existing Sage tarball
and makes a "clean" binary-free tarball. This would mean removing the
pickle jar and all the hg repositories.

> If we can include pickles, then I agree, pickling it would be the
> easiest option.

We already include hundreds of pickles in the pickle jar, which is
used by doctesting to verify that new versions of sage don't break old
pickles.

-- William

[Jason Grout]

So really, the code to construct the graph should be there, but it
should just treat the pickle as a cached version of the output?

It sounds like this could be made more general, like a @cache_output
decorator that saves a pickle. We could ship Sage with or without those
pickles, or maybe we could just bundle all of the pickles into an spkg
that is a standard install.

Jason

[John H Palmieri]

Since countries change (names change, boundaries change, etc.), first
of all, you need to date this: say it's the world map as of
2009-08-20, or whatever, and probably give the source (since not
everyone agrees on names, boundaries, etc.). Then if it were possible
to automate its production (get some files from the CIA's website --
do these have a fixed URL? -- then process them with Python, etc., all
automatically), I see the whole thing as an optional spkg.

Can we access historical data, too? World map as of 1939-09-01, etc.?

John

[Nathann Cohen]

It took some time to build filters from the original files ( it is
HTML I had to parse, nothing as clean as xml, sadly .... ) and several
scripts...

I will look again at the sources to see whether I could find an
automatic way to do it... But the source is not clean at all, I had to
change a lot of thing by hand ( like Cuba having a boundary with the
"US base of guantanamo", and the "US base of guantanamo" has neither
Position nor even an entry in the list of its own.... ^^;

But I was sure there would be some easy-to-read database.. I'll try to
find this tomorrow :-)

Nathann

[Nathann Cohen]

Do you have any idea how it is possible to call the "load" function
from graph_generators.py ?
I cannot use "load" as the function is not defined yet, nor can I
directly write :
return sage.structure.sage_object.load("graph_world.sobj")

as sage is not defined at that time either... ^^;

Nathann

[William Stein]

On Fri, Aug 21, 2009 at 4:49 AM, Nathann Cohen <nathan...@gmail.com> wrote:

Do you have any idea how it is possible to call the "load" function
from graph_generators.py ?
I cannot use "load" as the function is not defined yet, nor can I
directly write :
        return sage.structure.sage_object.load("graph_world.sobj")

as sage is not defined at that time either...

You should *not* load the graph into memory when graph_generators.py is being imported.  I'm glad Sage doesn't let you do that, since it would be an inefficient waste of memory and time to lod that graph every time Sage starts up.

Just write code that is called only when the user actually types

   graphs.world***

which might cache its result if you want.

 -- William  

[Nathann Cohen]

Actually, it should not load the graph when graph_generator is loaded.
Anyway, I think that if it was the case, the error would be printed a
Sage's startup and not when I call the function, which is my current
situation :-/

When I remove all the docstrings, this functions is pretty simple :

def WorldMap(self):
return sage.structure.sage_object.load("graph_world.sobj")

I rebuild I start my branch without the slightest problem, but when I
type g=graphs.WorldMap()

I get :
sage: g=graphs.WorldMap()
---------------------------------------------------------------------------
NameError Traceback (most recent call
last)

/user/ncohen/home/.sage/temp/rebelote.inria.fr/4759/
_user_ncohen_home__sage_init_sage_0.py in <module>()

/usr/local/sage/local/lib/python2.6/site-packages/sage/graphs/
graph_generators.pyc in WorldMap(self)
2983 """
2984
-> 2985 return sage.structure.sage_object.load
("graph_world.sobj")
2986
2987
################################################################################

NameError: global name 'sage' is not defined

And I can not understand why, nor how I should do to bypass it :-/

Nathann

On Aug 21, 5:28 pm, William Stein <wst...@gmail.com> wrote:

[Robert Bradshaw]

When you start up Sage, a lot of things are imported automaticaly, but you
need to import them manually to use them from the library itself.

In this case you could do

from sage.structure.sage_object import load
return load("...")

- Robert

[Nathann Cohen]

Excellent :-)

I thought writing the whole path would be sufficient, but if I load
the function first then call it, it works at last :-)

My last problem is that the file is not found. I just wrote the
filename hoping it would try to find it in the directory of
graph_generator, but it seems to try to find it in the user's current
directory ^^;

Is there a variable defining the path of sage up to the current
branch ?

Thanks !

Nathann

[Simon King]

Hi!

On 22 Aug., 10:11, Nathann Cohen <nathann.co...@gmail.com> wrote:
> Is there a variable defining the path of sage up to the current
> branch ?

There is SAGE_ROOT, and if I am not mistaken then SAGE_ROOT+'/devel/
sage/' is a link to the current mercurial branch.

But there is also a folder SAGE_ROOT+'/local/'. Your package could
create a sub-folder (this is what distutils would do for local data of
a package, IIRC) and store all its data in that sub-folder.

Cheers,
Simon

[William Stein]

No, use the SAGE_DATA variable, which is defined in misc/misc.py:

sage: sage.misc.misc.SAGE_DATA

'/Users/wstein/sage/build/64bit/sage/data/'

Your graph should go somewhere in SAGE_ROOT/data/something...

 
William

[Nathann Cohen]

Done, and it works.... But now I wonder how to build the patch, as
SAGE/DATA is not monitored.. :-)

I could also just attach it to the Trac ticket, and let it be merged
to Sage, as it does not seem worth building a spkg....

Nathann

On Aug 22, 12:25 pm, William Stein <wst...@gmail.com> wrote:

[Simon King]

Hi William!

On 22 Aug., 12:25, William Stein <wst...@gmail.com> wrote:
[...]

> No, use the SAGE_DATA variable, which is defined in misc/misc.py:

Thank you, I didn't know about that one.
Is it for data that concern the whole Sage installation, or is it
specific to a single user (like DOT_SAGE)?

In the former case, probably it would also be the right place for my
group cohomology data base (that I currently put into SAGE_LOCAL/
pGroupCohomology/), wouldn't it?

But how can one tell setup.py that it is supposed to use SAGE_DATA?

'cause, if one has

setup(
name = "packagename",
...
data_files=[("Folder",
["File1",
"File2"]),
("Folder/Subfolder", [...]),
...
)

then the data files seem to be copied to SAGE_ROOT/local/Folder and
SAGE_ROOT/local/Folder/Subfolder, if I'm not mistaken. Note, in
particular, that they all go into SAGE_LOCAL, not SAGE_DATA

Best regards,
Simon

[William Stein]

On Sat, Aug 22, 2009 at 4:26 AM, Simon King <simon...@nuigalway.ie> wrote:

Hi William!

On 22 Aug., 12:25, William Stein <wst...@gmail.com> wrote:
[...]

> No, use the SAGE_DATA variable, which is defined in misc/misc.py:

Thank you, I didn't know about that one.
Is it for data that concern the whole Sage installation, or is it
specific to a single user (like DOT_SAGE)?

It is data that is global to the whole SAGE install. 
 

In the former case, probably it would also be the right place for my
group cohomology data base (that I currently put into SAGE_LOCAL/
pGroupCohomology/), wouldn't it?

Yes, definitely.  

 

But how can one tell setup.py that it is supposed to use SAGE_DATA?

'cause, if one has

setup(
 name = "packagename",
...
 data_files=[("Folder",
             ["File1",
              "File2"]),
             ("Folder/Subfolder", [...]),
...
)

then the data files seem to be copied to SAGE_ROOT/local/Folder and
SAGE_ROOT/local/Folder/Subfolder, if I'm not mistaken. Note, in
particular, that they all go into SAGE_LOCAL, not SAGE_DATA

I don't know any way to do what you want using setup.py.

William

[William Stein]

On Sat, Aug 22, 2009 at 3:57 AM, Nathann Cohen <nathan...@gmail.com> wrote:

Done, and it works.... But now I wonder how to build the patch, as
SAGE/DATA is not monitored.. :-)

I could also just attach it to the Trac ticket, and let it be merged
to Sage, as it does not seem worth building a spkg....

That makes sense to me.
 

Nathann

On Aug 22, 12:25 pm, William Stein <wst...@gmail.com> wrote:

> On Sat, Aug 22, 2009 at 1:56 AM, Simon King <simon.k...@nuigalway.ie> wrote:
>
> > Hi!
>
> > On 22 Aug., 10:11, Nathann Cohen <nathann.co...@gmail.com> wrote:
> > > Is there a variable defining the path of sage up to the current
> > > branch ?
>
> > There is SAGE_ROOT, and if I am not mistaken then SAGE_ROOT+'/devel/
> > sage/' is a link to the current mercurial branch.
>
> > But there is also a folder SAGE_ROOT+'/local/'. Your package could
> > create a sub-folder (this is what distutils would do for local data of
> > a package, IIRC) and store all its data in that sub-folder.
>
> No, use the SAGE_DATA variable, which is defined in misc/misc.py:
>
> sage: sage.misc.misc.SAGE_DATA
>
> '/Users/wstein/sage/build/64bit/sage/data/'
>
> Your graph should go somewhere in SAGE_ROOT/data/something...
>
> William