Network access removed

1440 views
Skip to first unread message

Andrey Novoseltsev

unread,
Mar 21, 2021, 10:57:12 PMMar 21
to sage-cell
Unfortunately, due to some people abusing SageMathCell servers access to the Internet from cells is going to be removed or severely limited.

If you really need such access, your options are:
- maintain your own server, e.g. behind a campus firewall
- use CoCalc, where you can enable such access
- explain exactly what kind of access you need and why - perhaps we will figure something out, but no guarantees on how long it will take, if ever

Elimboto Yohana

unread,
Mar 22, 2021, 4:29:24 AMMar 22
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Does this mean we will not be able to use SageMathCell integrated at https://www.tssfl.com/programming-and-computing-with-python-right-from-the-forum-6300?

Kind regards,

Elimboto
> --
> You received this message because you are subscribed to the Google Groups "sage-cell" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-cell/7c758ba4-74e5-4827-89a7-f68ea2713a58n%40googlegroups.com.
>

kcrisman

unread,
Mar 22, 2021, 7:32:24 AMMar 22
to sage-cell
Sorry to hear that, but given the history of such services it's probably inevitable - see sagenb.org :(  Thanks for keeping watch and keeping things going for such a vital service!

Jorge Garcia

unread,
Mar 22, 2021, 10:15:26 AMMar 22
to Andrey Novoseltsev, sage-cell

Andrey Novoseltsev

unread,
Mar 22, 2021, 10:45:56 AMMar 22
to sage-cell
On Monday, 22 March 2021 at 02:29:24 UTC-6 e...@tssfl.co wrote:
Hi Andrey,

Does this mean we will not be able to use SageMathCell integrated at https://www.tssfl.com/programming-and-computing-with-python-right-from-the-forum-6300?

Hi Elimboto,

Using SageMathCell on other servers of course will be possible, that's the whole point of it. But if your code within the cell is trying to access Internet, it will likely fail, as it does now. I hope we will allow some explicitly whitelisted services to function, e.g. getting files from github.

Thank you,
Andrey

Ingo Dahn

unread,
Mar 23, 2021, 2:32:55 AMMar 23
to sage-cell
This is indeed a severe restriction, and it is surprising that it is applied without prior notice.
For using real life data services with SageCell, as for my COVID-Site on https://dahn-research.eu/corona/International.html, it would be necessary to allow data access through pandas. Can that be enabled?
Could a SageCell server be packaged in a Docker container so that it can be installed the same way as CoCalc (pypi/sagecell on Dockerhub doesn't seem to be maintained) or could a SageCell server become part of a CoCalc Docker container?

Elimboto Yohana

unread,
Mar 23, 2021, 3:50:44 AMMar 23
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for clarifying. I would suggest whitelisting the following links:

1. Github: which is an ideal repository for sharing files. I have a number of codes that use data from GitHub (e.g., Iris and NLTK datasets). Some GitHub URLs are of the form https://raw.githubusercontent.com

2. Dropbox, www.dropbox.com: There are many cases where some data files I pushed (or by others) to Github could not be retrieved using urllib.request or similar methods. I end up getting errors or retrieving files that are corrupted. In this case, I found that Dropbox is great, I could upload and read any data file from it. Here are examples:


3. Google cloud console, https://console.cloud.google.com/, Google Drive, https://drive.google.com, Google docs, https://docs.google.com/ (e.g., for spreadsheets, https://docs.google.com/spreadsheets) -- I have been using Google Sheets API v4 and Google Drive API to access and automate Google spreadsheet tasks, see an example, https://www.tssfl.com/automating-the-google-spreadsheet-tasks-with-python-and-tssfl-open-discussion-forums-6402

4. Kaggle, https://www.kaggle.com -- useful for retrieving various data science and machine learning data files.

5. Astropy: This package depends on some data and information from http://data.astropy.org/tu and https://www.astropy.org/. For example, this is what I get after testing some sample code, "URLError: <urlopen error Unable to open any source! Exceptions were {'http://data.astropy.org/tutorials/FITS-images/HorseHead.fits': URLError(gaierror(-3, 'Temporary failure in name resolution')), 'http://www.astropy.org/astropy-data/tutorials/FITS-images/HorseHead.fits': URLError(gaierror(-3, 'Temporary failure in name resolution'))}"

6. Google API for Keras and TensorFlow: https://storage.googleapis.com/tensorflow/tf-keras-datasets/

7. MNIST dataset, http://yann.lecun.com/exdb/mnist/ for machine learning

8. However, the following URLs are of interest for cosmological or astrophysical data:

- NASA's High Energy Astrophysics Science Archive Research Center (HEASARC): https://heasarc.gsfc.nasa.gov/
- Data repository, http://cdsweb.u-strasbg.fr/
- Planck Satelite Legacy Archive for The European Space Agency, http://pla.esac.esa.int
- NASA JPL data repository, https://naif.jpl.nasa.gov/naif/

You can see various use case examples for the mentioned URLs at https://www.tssfl.com/programming-and-computing-with-python-right-from-the-forum-6300

Kind regards,

Elimboto

Andrey Novoseltsev

unread,
Mar 23, 2021, 11:28:40 AMMar 23
to sage-cell
On Tuesday, 23 March 2021 at 00:32:55 UTC-6 ingo...@gmail.com wrote:
This is indeed a severe restriction, and it is surprising that it is applied without prior notice.

This was not a planned change for which we could give advance notice - we were forced to act by the abuse of the servers which was possible due to our flexibility, which basically allowed you to do whatever you want.
 
For using real life data services with SageCell, as for my COVID-Site on https://dahn-research.eu/corona/International.html, it would be necessary to allow data access through pandas. Can that be enabled?

The goal is to allow access to certain IPs - if you can provide the list that works for your purposes, we can allow connection to them. The system for this whitelisting is not set up yet, but I hope to have it running in a couple of weeks. A quicker solution may be to set up your own server:
where you are free to impose any restrictions you want or none at all.
 
Could a SageCell server be packaged in a Docker container so that it can be installed the same way as CoCalc (pypi/sagecell on Dockerhub doesn't seem to be maintained) or could a SageCell server become part of a CoCalc Docker container?

I am sure it can be easily packaged, SageMathCell is just a service that runs on a machine with SageMath installed, but as I have zero experience with Docker it would be better if someone else did it, presumably someone who actually intends to use such a container.

Ingo Dahn

unread,
Mar 23, 2021, 2:10:47 PMMar 23
to Andrey Novoseltsev, sage-cell
Thanks, Andrey, for the update. I hope you can share the whitelist on the SageCell home page so that I can link to it from the Notebook Player.

To keep the current pages of me and my collaborators alive, access to subpages of the following two URLs would be needed.
https://opendata.ecdc.europa.eu/ (European Centre for Disease Prevention and Control)
https://opendata.dwd.de (Data from the German Weather Service).

But besides accessing these "official" data sources, restricting Internet access gives me yet another problem I am not sure how to solve:

Cells in some of my pages load a Python script from my server at https://mathtrek.eu, which defines some functions. I could put this code directly into some cell, but I'd rather like to hide it from the visitor  for the following reason.

I am encouraging the visitor to play around with the code in the SageCells of the page, thus modifying my data analysis. I do not assume that the visitor has experience with Sage but I hope to encourage her in this way to have a closer look at it. Revealing more complex code, she shouldn't tamper with, contradicts this - I am ready to do it, if it is the only way to keep the pages alive, but it doesn't make me happy.

Thanks for pointing me to the SageCell EVM, I'll have a look at it.

Best wishes
Ingo

Andrey Novoseltsev

unread,
Mar 23, 2021, 2:19:02 PMMar 23
to Ingo Dahn, sage-cell
On Tue, Mar 23, 2021 at 12:10 PM Ingo Dahn <ingo...@gmail.com> wrote:
Cells in some of my pages load a Python script from my server at https://mathtrek.eu, which defines some functions. I could put this code directly into some cell, but I'd rather like to hide it from the visitor  for the following reason.

I am encouraging the visitor to play around with the code in the SageCells of the page, thus modifying my data analysis. I do not assume that the visitor has experience with Sage but I hope to encourage her in this way to have a closer look at it. Revealing more complex code, she shouldn't tamper with, contradicts this - I am ready to do it, if it is the only way to keep the pages alive, but it doesn't make me happy.

The plan is to have github accessible, so if you host your modules there they will be available.

If you are using linked cells, it may be possible to hide "messy code", although you still have to press the button to evaluate it.

samuel....@gmail.com

unread,
Mar 24, 2021, 7:50:07 AMMar 24
to sage-cell
Andrey,

Sorry that the service was abused, forcing you to limit functionality.

Please post a prominent notice on sagecell.sagemath.org
so people who are surprised about the new behaviour know
what is going on.

There were already two related questions on sage-support:

- sage-support, 2021-03-23

- sage-support, 2021-03-23

Thanks a lot for maintaining this service!  --Samuel

matteo.mi...@gmail.com

unread,
Mar 24, 2021, 11:53:26 AMMar 24
to sage-cell
Hi, I'm sorry and I apologize with all you and with Andrey if Andrey's choice is related in some way to what I wrote in an another thread (Using physics in SageCell) in which I discussed the possibility of temporarily installing a python library with miniconda installed in a cell.

I ran some tests (about 8 attempts in about 3 or 4 days) last week by downloading the latest miniconda distribution into the current cell and trying to install a library (about 10-20 mb) without installing the dependencies; I observed the full directory tree created inside the cell and that the miniconda package in the cell took up a few hundred megabytes but at first I didn't bother because honestly I was completely convinced that any file or folder created or downloaded from the web remained in the cell for a short time, just the time for the execution of the script (which is at most a few minutes) or little more.

So, again, sorry if due to me (or maybe some other person who has performed similar tests on his own initiative) internet access has been removed from SageMathCell cells.

As far as I can tell, the ability to access files on the network from the cell or to download files is essential for a service like this. At the same time, however, the ability to access the web from the cell could be limited, as mentioned by Andrey.

Some ideas to restore the possibility of accessing the internet from the cell but preserving the functionality/integrity of the servers:

1-inform the users with a numeric field in web page of SageMathCell about the total free storage space of current used server, that depends from number of active cells in a certain time: in this way user, when opens a new cell, could see the % of available storage mem and maybe how many mb are "eaten" by own cell during calculation; maybe a limit of mb per cell could be introduced.  

2-limit the lifespan of the files/folders created in the cell to the script execution duration only, being able to download them automatically at the end of the script execution: for example, if I create files and folders in my cell with my script, at the end of the execution of the script the files and folders should be deleted automatically, while if at the beginning of my script I insert a code string such as, for example, "@download", at the end of the script execution a popup window may appear automatically, with limited duration, which allows user to download the entire contents of files and folders created in the current cell in own computer in the form of a compressed file. This function, if useful, should also be implemented in the sagecell-client.py script for whose use the service without a web browser.

3-the real revolution, if it wasn't already possible, would be to have a way that allows the server to access files and folders contained in remote locations (google drive, dropbox, github, ...) and to create files and folders directly in remote locations without physically creating such files and folders on the hard drives of the servers. For example, if I have a shared google drive folder that contains a suitably compiled python library for Linux (which is the operating system of the service), I would like to be able to import that library into a SageMathCell cell as I usually do, but instead of have the library saved in the cell, this is on in my remote folder (cloud) and the cell imports into the ram memory of the cell all files are required by that library, including dependencies (which are also in my remote folder, not in cell storage temp mem). But I'm not sure if it is technically possible, I don't know if python can import libraries that are located in a remote url.
For example, if my cloud working folder url is "https://drive.google.com/drive/folders/XXX/my_python_libs?usp=sharing" and inside the remote subfolder "my_python_libs" I have some files, folders and precompiled python libraries or pure-python libraries, the entire content should be able to be read or modified or imported by the script I run in cell if I set the full url of the shared folder "https://drive.google.com/drive/folders/XXX/my_python_libs?usp=sharing" as working directory. So, if in the script executed by cell the working directory (os.getcwd()) is "https://drive.google.com/drive/folders/XXX/my_python_libs?usp=sharing", with "os.listdir()" I'd like to see the full content (files and folder) inside os.getcwd() (that is: all the files and folders I have in my remote "my_python_libs" folder). In the same way I'd like to import anything from shared folder (from url) without first download files in cell.

Thank you
Regards
Matteo

Samuel Lelièvre

unread,
Mar 24, 2021, 12:09:47 PMMar 24
to matteo.mi...@gmail.com, sage-cell
2021-03-24 15:53 UTC, Matteo:
>
> Hi, I'm sorry and I apologize with all you and with Andrey if Andrey's choice
> is related in some way to what I wrote in an another thread (Using physics
> in SageCell) in which I discussed the possibility of temporarily installing
> a python library with miniconda installed in a cell.

An aggressive cryptocurrency mining attempt
would seem to be what forced the change.

https://groups.google.com/g/sage-support/c/DgPxSZdmqcY/m/1gOsNbNJBgAJ

Andrey Novoseltsev

unread,
Apr 1, 2021, 1:01:01 PMApr 1
to sage-cell
Most of the sites requested should be operational, please test and let me know your results! Exact IPs are here:

This one is behind Cloudflare, so does not have a stable IP and whitelisting the whole Cloudflare does not seem to be a good idea to me:

David Austin

unread,
Apr 1, 2021, 1:24:02 PMApr 1
to sage-cell
That's very helpful.  Thanks for your work on this. 

Michael Miller

unread,
Apr 1, 2021, 1:26:18 PMApr 1
to sage-cell
Can you whitelist https://sourceforge.net ?

Thanks!

Andrey Novoseltsev

unread,
Apr 1, 2021, 3:57:49 PMApr 1
to sage-cell
On Thursday, 1 April 2021 at 11:26:18 UTC-6 Michael Miller wrote:
Can you whitelist https://sourceforge.net ?
Done!

Elimboto Yohana

unread,
Apr 1, 2021, 5:36:59 PMApr 1
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for your great work. You have taken care of all URLs except where mentioned otherwise.

But I think due to some upgrade, most of the astrophysics codes that worked in the past are not working, the common error is

"AttributeError: module 'snappy' has no attribute 'compress'".

Another error I experienced is "AttributeError: partially initialized module 'fsspec' has no attribute 'utils' (most likely due to a circular import)". This error may be related to the first one.

It seems that the snappy package is missing, and I request installation: pip3 install python-snappy (see the discussion here https://github.com/dask/fastparquet/issues/459)

Can I request whitelisting our forum URL: https://www.tssfl.com? Sometimes we upload files that can be read directly from it.

Kind regards,

Elimboto


--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

William Stein

unread,
Apr 3, 2021, 10:21:24 AMApr 3
to Andrey Novoseltsev, sage-cell
Hi,

There's an active discussion on Hacker News right now about
cryptomining abuse of GitHub, with people working on other sites
talking about a wide range of things they do to combat the problem:

https://news.ycombinator.com/item?id=26678700

One comment has an estimate of how much money one of GitHub
cryptomining abusers might be making: "Judging by the hash rate, the
pool estimates a weekly income of 700$. It could be a multiple of this
if the culprit uses multiple wallets at once."

-- William
> --
> You received this message because you are subscribed to the Google Groups "sage-cell" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-cell/7c758ba4-74e5-4827-89a7-f68ea2713a58n%40googlegroups.com.



--
William (http://wstein.org)

Andrey Novoseltsev

unread,
Apr 5, 2021, 5:42:45 PMApr 5
to sage-cell
It seems that the snappy package is missing, and I request installation: pip3 install python-snappy (see the discussion here https://github.com/dask/fastparquet/issues/459)
Done!
Can I request whitelisting our forum URL: https://www.tssfl.com? Sometimes we upload files that can be read directly from it.
Done!

Elimboto Yohana

unread,
Apr 6, 2021, 5:10:31 AMApr 6
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for implementing my requests. But, re-installing the snappy package in the suggested way did not solve an issue. I tried to narrow down the source of the problem and found that importing either xarray or nbodykit.lab:

import nbodykit.lab
import xarray as xr

is the source of "AttributeError: module 'snappy' has no attribute 'compress'".

The previous versions had no issues.

Kind regards,

Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
Apr 16, 2021, 4:19:41 AMApr 16
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I just found that there are two Python-related snappy packages - both are useful packages, the first one is "for studying the topology and geometry of 3-manifolds, with a focus on hyperbolic structures" -- https://snappy.math.uic.edu/, see installation method, https://pypi.org/project/snappy/

This is one currently installed, you can verify this by:

import snappy
print(snappy.__version__)
print(snappy.__file__)
print(dir(snappy))

The second one is used by a number of Python packages for compression, "Python library for the snappy compression library from Google (http://google.github.io/snappy/)": https://pypi.org/project/python-snappy/ and can be installed as

sudo apt-get install libsnappy-dev (or similar method)
followed by
pip3 install python-snappy

It seems that you need to uninstall python-snappy and reinstall it following the order above, see https://stackoverflow.com/questions/48535799/module-snappy-has-no-attribute-decompress

Kind regards,

Elimboto

Elimboto Yohana

unread,
Apr 16, 2021, 7:20:58 AMApr 16
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Someone requested the installation of the Python Packages, pyforest (https://pypi.org/project/pyforest/) and lazypredict (https://pypi.org/project/lazypredict/):

pip3 install pyforest and pip3 install lazypredict

shall that be possible.

They are using them to study machine learning models.

Kind regards.

Elimboto

Andrey Novoseltsev

unread,
Apr 19, 2021, 4:16:22 PMApr 19
to Elimboto Yohana, sage-cell
I've added libsnappy-dev system package - can you please check that your code works as intended now? If not please send me the code that raises the error.

Andrey Novoseltsev

unread,
Apr 19, 2021, 4:17:59 PMApr 19
to Elimboto Yohana, sage-cell
I've installed pyforest

lazypredict requires fixed (and old) versions of many packages, so it breaks other installations. Those who need it should bug developers to use >= for versions, not == ;-)

Elimboto Yohana

unread,
Apr 20, 2021, 8:31:22 AMApr 20
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for installing pyforest.

I'm not clear how lazypredict can be used in the way you suggested.

For the Snappy package, I think the solution is to uninstall the current Snappy and install the one in question (python-snappy) using the suggested approach.

Python-snappy seems to impact many codes even a simple import as this one, import plotly.express as px will raise an error "AttributeError: module 'snappy' has no attribute 'compress'".

Or maybe it is time to consider Python virtual environment/virtualenvwrapper (https://www.tssfl.com/python-virtual-environments-virtualenv-5558) or something similar to separate various packages and their dependencies.

Kind regards,

Elimboto

Andrey Novoseltsev

unread,
Apr 20, 2021, 12:20:15 PMApr 20
to sage-cell
Hi Elimboto!

On Tuesday, 20 April 2021 at 06:31:22 UTC-6 e...@tssfl.co wrote:
I'm not clear how lazypredict can be used in the way you suggested.

My point was that its installation instructions are "wrong" and we can't install it until developers fix it to make it installable alongside other packages.
 
For the Snappy package, I think the solution is to uninstall the current Snappy and install the one in question (python-snappy) using the suggested approach.

Python-snappy seems to impact many codes even a simple import as this one, import plotly.express as px will raise an error "AttributeError: module 'snappy' has no attribute 'compress'".

Servers were rebuilt from scratch, I never uninstall packages. Does python-snappy conflict with snappy and so we have to drop one or the other?
 
Or maybe it is time to consider Python virtual environment/virtualenvwrapper (https://www.tssfl.com/python-virtual-environments-virtualenv-5558) or something similar to separate various packages and their dependencies.

Impossible in the current approach of having forked kernels ready and waiting for the user code. Besides it would require users to pick which environment they want to use and I seriously doubt that there is much point in implementing and supporting this. If someone's needs are so specific that they really need particular packages that do not play well with others, they probably should use local installation or alternatives like CoCalc, SageMathCell was intended as a simple way to run simple bits of code with as little configuration and choices as possible.

Having added quite a few packages over the years, my claim is that if some package cannot be installed and used alongside other packages - it is this package fault, not ours for not providing isolated environments.

Thank you!
Andrey

Andrey Novoseltsev

unread,
Apr 26, 2021, 10:57:42 AMApr 26
to Elimboto Yohana, sage-cell
Hi Elimboto,

Plain "snappy" package was installed because it was requested by another user before you asked for "python-snappy". Uninstalling the first and then installing the second does fix your problem, but obviously it makes the first package unavailable, so I am hesitant to make such a change. I have written to snappy developers and asked if they can come up with some idea to install packages simultaneously without using virtual environments, we'll see. The fastest solution for you may be to run your own server where you are in full control of installed packages, but while I tried to make initial installation simple, it does come with some maintenance overhead ;-)

Thank you!
Andrey

Elimboto Yohana

unread,
Apr 26, 2021, 4:55:48 PMApr 26
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for your response. I did not know that someone requested snappy, it is however useful. Frankly speaking, I prefer the SageMathCell. Let's wait for the reply from snappy developers.

Kind regards,
Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Andrey Novoseltsev

unread,
Apr 26, 2021, 10:36:38 PMApr 26
to Elimboto Yohana, sage-cell
The reply is that nothing can be done short of renaming one of the packages. Since you don't actually need python-snappy directly, perhaps there is a way for the packages that you care about to avoid using it?.. My understanding is that it is just some compression library, surely there are other available.

Andrey Novoseltsev

unread,
Apr 28, 2021, 11:05:07 PMApr 28
to Elimboto Yohana, sage-cell
On Tue, Apr 27, 2021 at 9:58 PM Elimboto Yohana <e...@tssfl.com> wrote:
NB: I'm facing issues connecting to Dropbox, this wasn't there before access restriction, may there be something wrong with Dropbox access/whitelisting?

Can you please post the command that does not work? It may be helpful if you also run host ... to see the IP address, although as far as I understand the resolution is done on our servers and geographic location should not matter...

Elimboto Yohana

unread,
May 3, 2021, 10:38:05 PMMay 3
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I use Google Drive API to perform operations on Google spreadsheets (e.g., here https://www.tssfl.com/automating-the-google-spreadsheet-tasks-with-python-and-tssfl-open-discussion-forums-6402), it turns out the problem is not Dropbox, but sometimes the Google API is slow, especially if using other browsers than Chrome.

Kind regards,
Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
May 17, 2021, 11:20:17 AMMay 17
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I will appreciate if you can install Google API Client Library for Python (https://pypi.org/project/google-api-python-client/): pip install google-api-python-client

It provides access to Google services, such as spreadsheets for data science.

Kind regards,

Elimboto

Andrey Novoseltsev

unread,
May 23, 2021, 6:57:45 PMMay 23
to sage-cell
On Monday, 17 May 2021 at 09:20:17 UTC-6 e...@tssfl.co wrote:
Hi Andrey,

I will appreciate if you can install Google API Client Library for Python (https://pypi.org/project/google-api-python-client/): pip install google-api-python-client

It seems to be installed - do you have problems using it? Perhaps it is due to network blocking.

Elimboto Yohana

unread,
May 30, 2021, 4:11:17 PMMay 30
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Can you please whitelist the Woldometers and Twitter URLs: https://www.worldometers.info/coronavirus/, www.twitter.com

They are useful for a number of analytics via web scrapping with Python -- Selenium, BeautifulSoup, etc.,

In addition, I request the installation of the following related library:

import tweepy (pip install tweepy) -- Python library for Twitter API.

Kind regards,

Elimboto

On Mon, Mar 22, 2021 at 5:57 AM Andrey Novoseltsev <novo...@gmail.com> wrote:
Unfortunately, due to some people abusing SageMathCell servers access to the Internet from cells is going to be removed or severely limited.

If you really need such access, your options are:
- maintain your own server, e.g. behind a campus firewall
- use CoCalc, where you can enable such access
- explain exactly what kind of access you need and why - perhaps we will figure something out, but no guarantees on how long it will take, if ever

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
Jun 1, 2021, 4:09:56 PMJun 1
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I also request the installation of the Python library bioinfokit: pip install bioinfokit (https://pypi.org/project/bioinfokit/0.3c/) for biological data analyses.

Kind regards,

Elimboto

Elimboto Yohana

unread,
Jun 2, 2021, 12:29:13 PMJun 2
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for implementing my request, packages are working fine. The only thing that does not work is the Worldometer website access, here is the error:
ConnectionError: HTTPSConnectionPool(host='www.worldometers.info', port=443): Max retries exceeded with url: /coronavirus/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7f2c5da8b0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

Kind regards,
Elimboto

Andrey Novoseltsev

unread,
Jun 2, 2021, 11:44:08 PMJun 2
to sage-cell
Hi Elimboto!

You are too fast - I was waiting for servers to finish updating before reporting that requests have been completed. Except for worldometers - they are behind Cloudflare and so don't have static IPs.

Best,
Andrey

Elimboto Yohana

unread,
Jun 11, 2021, 8:12:01 AMJun 11
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I would appreciate it if you can also install the following two Python libraries for Topological Data Analysis and R tidyverse packages.

1) scikit-tda: pip install scikit-tda

A high-performance topological machine learning toolbox in Python built on top of scikit-learn:

2) giotto-tda (https://arxiv.org/abs/2004.02551): pip install giotto-tda

3)   R tidyverse packages for statistics and data science (https://www.tidyverse.org/packages/): install.packages("tidyverse")

Kind regards,
Elimboto



On Thu, Jun 3, 2021 at 8:03 AM Elimboto Yohana <e...@tssfl.com> wrote:
Thank you Andrey. I find a workaround to read COVID-19 daily data from worldometers: pip install covid-daily (https://pypi.org/project/covid-daily/). I hope it works.

Kind regards,

Elimboto




--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Andrey Novoseltsev

unread,
Jun 13, 2021, 11:11:15 PMJun 13
to sage-cell
Hi Elimboto,

All should be available now!

Andrey

Elimboto Yohana

unread,
Jun 14, 2021, 6:07:35 AMJun 14
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you very much for installing the packages. I Should actually have noticed that covid_daily requires Worldometers URL to retrieve COVID data. As it is behind CloudFare, I experience the same error. Maybe there will be a way out in the future.

Kind regards,

Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
Jun 23, 2021, 1:16:03 PMJun 23
to Andrey Novoseltsev, sage-cell
Hi Andrey,

SageCell is getting steady use and a number of users via TSSFL Open Discussion Forums (www.tssfl.com) have been asking me about various packages, so I represent their requests, and as it deems fit please I request the installation of these packages:

1.  Latin Hypercube Sampling with Multi-Dimensional Uniformity library (https://pypi.org/project/lhsmdu/) :  pip install lhsmdu

2. PyTorch machine learning library (https://pypi.org/project/torch/): pip install torch

3. Transformers -- State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch (https://pypi.org/project/transformers/): pip install transformers

4. An updated EdgeDriver implementation for Selenium 3 with newly-added support for Microsoft Edge (Chromium) (https://pypi.org/project/msedge-selenium-tools/): pip install msedge-selenium-tools

5. Library for performing speech recognition, with support for several engines and APIs, online and offline (https://pypi.org/project/SpeechRecognition/): pip install SpeechRecognition

6. Twint -- An advanced Twitter scraping & OSINT tool (https://pypi.org/project/twint/): pip install twint

7. Spacy -- Industrial-strength Natural Language Processing (NLP) in Python (https://pypi.org/project/spacy/): pip install spacy

8. Cufflinks -- binds Plotly with Pandas to create graphs and charts of dataframes directly (https://github.com/santosjorge/cufflinks): pip install cufflinks

9. emoji -- Useful for textual data analyses that include emojis, e.g, WhatsApp group data (https://pypi.org/project/emoji/): pip install emoji

10. Lenstools for cosmology -- Toolkit for Weak Gravitational Lensing analysis (https://pypi.org/project/lenstools/): pip install lenstools

Kind regards,

Elimboto

Andrey Novoseltsev

unread,
Jun 25, 2021, 11:12:55 PMJun 25
to Elimboto Yohana, sage-cell
Hi Elimboto!

All should be available now, don't know if they require access to any sites.

Best,
Andrey

Elimboto Yohana

unread,
Jun 26, 2021, 4:26:23 AMJun 26
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you very much for installing the packages. I will test them one by one and let you know if there are any issues.

Kind regards,

Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
Jun 26, 2021, 5:37:00 AMJun 26
to Andrey Novoseltsev, sage-cell
I just found that twint requires nest_asyncio (to allow nested event loops) following the RuntimeError: This event loop is already running.

It can be installed as pip install nest-asyncio or pip3 install nest_asyncio ?, see https://pypi.org/project/nest-asyncio/.

Kind regards,

Elimboto

Elimboto Yohana

unread,
Jun 28, 2021, 9:44:30 AMJun 28
to Andrey Novoseltsev, sage-cell
I also found that wordcloud is useful for some NLP projects, and possibly the latest version can be installed as

git clone https://github.com/amueller/word_cloud.git

cd word_cloud

pip install .

Kind regards,

Elimboto

Elimboto Yohana

unread,
Jul 10, 2021, 5:41:22 AMJul 10
to Andrey Novoseltsev, sage-cell
Hi Andrey,

cufflinks -- connects Plotly with Pandas, won't work with SageCell for some old known problem "AttributeError: module 'snappy' has no attribute 'compress'".

import ripser (https://ripser.scikit-tda.org/en/latest/) also triggers this error: "ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject".

Kind regards,

Elimboto


Andrey Novoseltsev

unread,
Jul 11, 2021, 2:57:31 PMJul 11
to Elimboto Yohana, sage-cell
Hi Elimboto,

nest_asyncio and wordcloud will be installed via pip in a few hours

ripser problem is solved by upgrading numpy, but it seems that we are on an older version because of tensorflow that fixes it. It is a package that was installed by your request, so I am happy to remove it if you want to do so ;-) Personally I am not a fan of having packages that don't allow newer versions of dependencies.

Thank you!
Andrey

Elimboto Yohana

unread,
Jul 12, 2021, 3:36:23 AMJul 12
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for installing the packages. TensorFlow is more desirable, it is being used by more people than ripser. So, if the current version of Numpy suffices we can just leave it.

Kind regards,

Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
Jul 16, 2021, 7:31:46 AMJul 16
to Andrey Novoseltsev, sage-cell
Hi Andrey,

A request to install these Python packages:

1.

- pip install send_email (https://pypi.org/project/send_email/)

- pip install pretty-html-table (https://pypi.org/project/pretty-html-table/)

I have a scenario in which I need to send to emails personalized reports created from some dataset using Python. For example, converting an excel/Google spreadsheet to Pandas DataFrame, then creating an HTML table from the DataFrame and send each individual HTML report to the appropriate email using a combination of smtplib, ssl, email, send_email, and pretty_html_table packages. There are other packages close to send_email, such as sendmail (https://pypi.org/project/sendmail/) and python-sendmail (https://pypi.org/project/python-sendmail/) but I hope send_email will work as motivated here https://dev.to/siddheshshankar/convert-a-dataframe-into-a-pretty-html-table-and-send-it-over-email-4663

These  packages are useful for data science:

2. pip install autoviz (https://pypi.org/project/autoviz/) for automated data visualizations.

3. pip install drawdata (https://pypi.org/project/drawdata/) for drawing classification problem dataset in machine learning in a jupyter notebook -- hope it works with SageCell.

4. pip install cleantext (https://pypi.org/project/cleantext/), an open-source python package to clean raw text data.

Kind regards,

Elimboto

David Farmer

unread,
Jul 16, 2021, 8:33:00 AMJul 16
to e...@tssfl.com, Andrey Novoseltsev, sage-cell

How will we avoid abuse of send_email?

I don't see how the sage cell server, which is intended to allow
computation in the browser, has any business sending email.

Also, I do not like the request phrased as
"These  packages are useful for data science"

If you have a need for those packages, then explicitly say so.
It does not make sense to just install packages because theoretically
someone else might want to use them some time in the future.

Regards,

David Farmer
> To view this discussion on the web visithttps://groups.google.com/d/msgid/sage-cell/CACEZTmv3dRX0yme-bvVGWTgt%2BjBhU7k6%3Du7NzE%3DZ7hatsgciuw%40mail.
> gmail.com.
>
>

Jonathan Gutow

unread,
Jul 16, 2021, 8:54:06 AMJul 16
to David Farmer, e...@tssfl.com, Andrey Novoseltsev, sage-cell
I agree with David. I think send_mail is too easily abused to be part of sage-cell.

Regards,
Jonathan

Dr. Jonathan Gutow
Chemistry Department
UW Oshkosh
web: https://cms.gutow.uwosh.edu/Gutow
e-mail: gu...@uwosh.edu
Ph: 920-424-1326

On Jul 16, 2021, at 7:32 AM, David Farmer <far...@aimath.org> wrote:

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Elimboto Yohana

unread,
Jul 16, 2021, 9:49:13 AMJul 16
to David Farmer, sage-cell, Andrey Novoseltsev
Hi David,

I use the packages offline/locally, so I think they can be incorporated into SageCell and will be using them.

Is there an option to send emails in a safer way with SageCell? The motivation is, for example, getting data related to students' tests and exams or survey data (see a mock example here https://www.tssfl.com/tssfl-odf-odk-collect-and-google-drive-integration-to-collect-store-manage-process-and-analyze-data-6425) from Google spreadsheets, carrying computations with SageCell, and automatically attaching the scores/reports as HTML content and send them to each student. Because each student needs his/her own report.

It is now possible to share Google spreadsheets using the method explained here: https://docs.gspread.org/en/latest/user-guide.html#creating-a-spreadsheet from SageCell using gspread. So, I have been thinking in the same line of a possibility to share results of SageCell computations to emails.

Kind regards,

Elimboto

Jonathan Gutow

unread,
Jul 16, 2021, 10:17:41 AMJul 16
to David Farmer, sage-cell, Andrey Novoseltsev, e...@tssfl.com
Elimboto,

The problem is that the sage-cell server is publicly available. Making sendmail available would make it possible for anybody with network access to build a spambot that operates using the sage-cell server. Access to sendmail should be protected by a password at the very least.

I think you need your e-mail solutions to go through your password protected account.

Regards,
Jonathan

Dr. Jonathan Gutow
Chemistry Department
UW Oshkosh


From: sage...@googlegroups.com <sage...@googlegroups.com> on behalf of Elimboto Yohana <e...@tssfl.co>
Sent: Friday, July 16, 2021 8:49 AM
To: David Farmer <far...@aimath.org>; sage-cell <sage...@googlegroups.com>; Andrey Novoseltsev <novo...@gmail.com>
Subject: Re: [sage-cell] Network access removed
 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

David Farmer

unread,
Jul 16, 2021, 10:23:09 AMJul 16
to sage-cell

Dear Elimboto,

As I said before (although I am not the person who makes the
final decision), Sage Cells are for embedding computation in a
web page. It is not intended to be a full computing environment.

Maybe CoCalc is what you are looking for (https://cocalc.com/)?
The free account does not have network access, but you can try it for
free to see if it serves your purpose.

You can also install Sagemath on your computer, and do anything
you want and install any packages you want.

I think you are looking for a full-fledged co