Network access removed

854 views
Skip to first unread message

Andrey Novoseltsev

unread,
Mar 21, 2021, 10:57:12 PMMar 21
to sage-cell
Unfortunately, due to some people abusing SageMathCell servers access to the Internet from cells is going to be removed or severely limited.

If you really need such access, your options are:
- maintain your own server, e.g. behind a campus firewall
- use CoCalc, where you can enable such access
- explain exactly what kind of access you need and why - perhaps we will figure something out, but no guarantees on how long it will take, if ever

Elimboto Yohana

unread,
Mar 22, 2021, 4:29:24 AMMar 22
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Does this mean we will not be able to use SageMathCell integrated at https://www.tssfl.com/programming-and-computing-with-python-right-from-the-forum-6300?

Kind regards,

Elimboto
> --
> You received this message because you are subscribed to the Google Groups "sage-cell" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-cell/7c758ba4-74e5-4827-89a7-f68ea2713a58n%40googlegroups.com.
>

kcrisman

unread,
Mar 22, 2021, 7:32:24 AMMar 22
to sage-cell
Sorry to hear that, but given the history of such services it's probably inevitable - see sagenb.org :(  Thanks for keeping watch and keeping things going for such a vital service!

Jorge Garcia

unread,
Mar 22, 2021, 10:15:26 AMMar 22
to Andrey Novoseltsev, sage-cell

Andrey Novoseltsev

unread,
Mar 22, 2021, 10:45:56 AMMar 22
to sage-cell
On Monday, 22 March 2021 at 02:29:24 UTC-6 e...@tssfl.co wrote:
Hi Andrey,

Does this mean we will not be able to use SageMathCell integrated at https://www.tssfl.com/programming-and-computing-with-python-right-from-the-forum-6300?

Hi Elimboto,

Using SageMathCell on other servers of course will be possible, that's the whole point of it. But if your code within the cell is trying to access Internet, it will likely fail, as it does now. I hope we will allow some explicitly whitelisted services to function, e.g. getting files from github.

Thank you,
Andrey

Ingo Dahn

unread,
Mar 23, 2021, 2:32:55 AMMar 23
to sage-cell
This is indeed a severe restriction, and it is surprising that it is applied without prior notice.
For using real life data services with SageCell, as for my COVID-Site on https://dahn-research.eu/corona/International.html, it would be necessary to allow data access through pandas. Can that be enabled?
Could a SageCell server be packaged in a Docker container so that it can be installed the same way as CoCalc (pypi/sagecell on Dockerhub doesn't seem to be maintained) or could a SageCell server become part of a CoCalc Docker container?

Elimboto Yohana

unread,
Mar 23, 2021, 3:50:44 AMMar 23
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for clarifying. I would suggest whitelisting the following links:

1. Github: which is an ideal repository for sharing files. I have a number of codes that use data from GitHub (e.g., Iris and NLTK datasets). Some GitHub URLs are of the form https://raw.githubusercontent.com

2. Dropbox, www.dropbox.com: There are many cases where some data files I pushed (or by others) to Github could not be retrieved using urllib.request or similar methods. I end up getting errors or retrieving files that are corrupted. In this case, I found that Dropbox is great, I could upload and read any data file from it. Here are examples:


3. Google cloud console, https://console.cloud.google.com/, Google Drive, https://drive.google.com, Google docs, https://docs.google.com/ (e.g., for spreadsheets, https://docs.google.com/spreadsheets) -- I have been using Google Sheets API v4 and Google Drive API to access and automate Google spreadsheet tasks, see an example, https://www.tssfl.com/automating-the-google-spreadsheet-tasks-with-python-and-tssfl-open-discussion-forums-6402

4. Kaggle, https://www.kaggle.com -- useful for retrieving various data science and machine learning data files.

5. Astropy: This package depends on some data and information from http://data.astropy.org/tu and https://www.astropy.org/. For example, this is what I get after testing some sample code, "URLError: <urlopen error Unable to open any source! Exceptions were {'http://data.astropy.org/tutorials/FITS-images/HorseHead.fits': URLError(gaierror(-3, 'Temporary failure in name resolution')), 'http://www.astropy.org/astropy-data/tutorials/FITS-images/HorseHead.fits': URLError(gaierror(-3, 'Temporary failure in name resolution'))}"

6. Google API for Keras and TensorFlow: https://storage.googleapis.com/tensorflow/tf-keras-datasets/

7. MNIST dataset, http://yann.lecun.com/exdb/mnist/ for machine learning

8. However, the following URLs are of interest for cosmological or astrophysical data:

- NASA's High Energy Astrophysics Science Archive Research Center (HEASARC): https://heasarc.gsfc.nasa.gov/
- Data repository, http://cdsweb.u-strasbg.fr/
- Planck Satelite Legacy Archive for The European Space Agency, http://pla.esac.esa.int
- NASA JPL data repository, https://naif.jpl.nasa.gov/naif/

You can see various use case examples for the mentioned URLs at https://www.tssfl.com/programming-and-computing-with-python-right-from-the-forum-6300

Kind regards,

Elimboto

Andrey Novoseltsev

unread,
Mar 23, 2021, 11:28:40 AMMar 23
to sage-cell
On Tuesday, 23 March 2021 at 00:32:55 UTC-6 ingo...@gmail.com wrote:
This is indeed a severe restriction, and it is surprising that it is applied without prior notice.

This was not a planned change for which we could give advance notice - we were forced to act by the abuse of the servers which was possible due to our flexibility, which basically allowed you to do whatever you want.
 
For using real life data services with SageCell, as for my COVID-Site on https://dahn-research.eu/corona/International.html, it would be necessary to allow data access through pandas. Can that be enabled?

The goal is to allow access to certain IPs - if you can provide the list that works for your purposes, we can allow connection to them. The system for this whitelisting is not set up yet, but I hope to have it running in a couple of weeks. A quicker solution may be to set up your own server:
where you are free to impose any restrictions you want or none at all.
 
Could a SageCell server be packaged in a Docker container so that it can be installed the same way as CoCalc (pypi/sagecell on Dockerhub doesn't seem to be maintained) or could a SageCell server become part of a CoCalc Docker container?

I am sure it can be easily packaged, SageMathCell is just a service that runs on a machine with SageMath installed, but as I have zero experience with Docker it would be better if someone else did it, presumably someone who actually intends to use such a container.

Ingo Dahn

unread,
Mar 23, 2021, 2:10:47 PMMar 23
to Andrey Novoseltsev, sage-cell
Thanks, Andrey, for the update. I hope you can share the whitelist on the SageCell home page so that I can link to it from the Notebook Player.

To keep the current pages of me and my collaborators alive, access to subpages of the following two URLs would be needed.
https://opendata.ecdc.europa.eu/ (European Centre for Disease Prevention and Control)
https://opendata.dwd.de (Data from the German Weather Service).

But besides accessing these "official" data sources, restricting Internet access gives me yet another problem I am not sure how to solve:

Cells in some of my pages load a Python script from my server at https://mathtrek.eu, which defines some functions. I could put this code directly into some cell, but I'd rather like to hide it from the visitor  for the following reason.

I am encouraging the visitor to play around with the code in the SageCells of the page, thus modifying my data analysis. I do not assume that the visitor has experience with Sage but I hope to encourage her in this way to have a closer look at it. Revealing more complex code, she shouldn't tamper with, contradicts this - I am ready to do it, if it is the only way to keep the pages alive, but it doesn't make me happy.

Thanks for pointing me to the SageCell EVM, I'll have a look at it.

Best wishes
Ingo

Andrey Novoseltsev

unread,
Mar 23, 2021, 2:19:02 PMMar 23
to Ingo Dahn, sage-cell
On Tue, Mar 23, 2021 at 12:10 PM Ingo Dahn <ingo...@gmail.com> wrote:
Cells in some of my pages load a Python script from my server at https://mathtrek.eu, which defines some functions. I could put this code directly into some cell, but I'd rather like to hide it from the visitor  for the following reason.

I am encouraging the visitor to play around with the code in the SageCells of the page, thus modifying my data analysis. I do not assume that the visitor has experience with Sage but I hope to encourage her in this way to have a closer look at it. Revealing more complex code, she shouldn't tamper with, contradicts this - I am ready to do it, if it is the only way to keep the pages alive, but it doesn't make me happy.

The plan is to have github accessible, so if you host your modules there they will be available.

If you are using linked cells, it may be possible to hide "messy code", although you still have to press the button to evaluate it.

samuel....@gmail.com

unread,
Mar 24, 2021, 7:50:07 AMMar 24
to sage-cell
Andrey,

Sorry that the service was abused, forcing you to limit functionality.

Please post a prominent notice on sagecell.sagemath.org
so people who are surprised about the new behaviour know
what is going on.

There were already two related questions on sage-support:

- sage-support, 2021-03-23

- sage-support, 2021-03-23

Thanks a lot for maintaining this service!  --Samuel

matteo.mi...@gmail.com

unread,
Mar 24, 2021, 11:53:26 AMMar 24
to sage-cell
Hi, I'm sorry and I apologize with all you and with Andrey if Andrey's choice is related in some way to what I wrote in an another thread (Using physics in SageCell) in which I discussed the possibility of temporarily installing a python library with miniconda installed in a cell.

I ran some tests (about 8 attempts in about 3 or 4 days) last week by downloading the latest miniconda distribution into the current cell and trying to install a library (about 10-20 mb) without installing the dependencies; I observed the full directory tree created inside the cell and that the miniconda package in the cell took up a few hundred megabytes but at first I didn't bother because honestly I was completely convinced that any file or folder created or downloaded from the web remained in the cell for a short time, just the time for the execution of the script (which is at most a few minutes) or little more.

So, again, sorry if due to me (or maybe some other person who has performed similar tests on his own initiative) internet access has been removed from SageMathCell cells.

As far as I can tell, the ability to access files on the network from the cell or to download files is essential for a service like this. At the same time, however, the ability to access the web from the cell could be limited, as mentioned by Andrey.

Some ideas to restore the possibility of accessing the internet from the cell but preserving the functionality/integrity of the servers:

1-inform the users with a numeric field in web page of SageMathCell about the total free storage space of current used server, that depends from number of active cells in a certain time: in this way user, when opens a new cell, could see the % of available storage mem and maybe how many mb are "eaten" by own cell during calculation; maybe a limit of mb per cell could be introduced.  

2-limit the lifespan of the files/folders created in the cell to the script execution duration only, being able to download them automatically at the end of the script execution: for example, if I create files and folders in my cell with my script, at the end of the execution of the script the files and folders should be deleted automatically, while if at the beginning of my script I insert a code string such as, for example, "@download", at the end of the script execution a popup window may appear automatically, with limited duration, which allows user to download the entire contents of files and folders created in the current cell in own computer in the form of a compressed file. This function, if useful, should also be implemented in the sagecell-client.py script for whose use the service without a web browser.

3-the real revolution, if it wasn't already possible, would be to have a way that allows the server to access files and folders contained in remote locations (google drive, dropbox, github, ...) and to create files and folders directly in remote locations without physically creating such files and folders on the hard drives of the servers. For example, if I have a shared google drive folder that contains a suitably compiled python library for Linux (which is the operating system of the service), I would like to be able to import that library into a SageMathCell cell as I usually do, but instead of have the library saved in the cell, this is on in my remote folder (cloud) and the cell imports into the ram memory of the cell all files are required by that library, including dependencies (which are also in my remote folder, not in cell storage temp mem). But I'm not sure if it is technically possible, I don't know if python can import libraries that are located in a remote url.
For example, if my cloud working folder url is "https://drive.google.com/drive/folders/XXX/my_python_libs?usp=sharing" and inside the remote subfolder "my_python_libs" I have some files, folders and precompiled python libraries or pure-python libraries, the entire content should be able to be read or modified or imported by the script I run in cell if I set the full url of the shared folder "https://drive.google.com/drive/folders/XXX/my_python_libs?usp=sharing" as working directory. So, if in the script executed by cell the working directory (os.getcwd()) is "https://drive.google.com/drive/folders/XXX/my_python_libs?usp=sharing", with "os.listdir()" I'd like to see the full content (files and folder) inside os.getcwd() (that is: all the files and folders I have in my remote "my_python_libs" folder). In the same way I'd like to import anything from shared folder (from url) without first download files in cell.

Thank you
Regards
Matteo

Samuel Lelièvre

unread,
Mar 24, 2021, 12:09:47 PMMar 24
to matteo.mi...@gmail.com, sage-cell
2021-03-24 15:53 UTC, Matteo:
>
> Hi, I'm sorry and I apologize with all you and with Andrey if Andrey's choice
> is related in some way to what I wrote in an another thread (Using physics
> in SageCell) in which I discussed the possibility of temporarily installing
> a python library with miniconda installed in a cell.

An aggressive cryptocurrency mining attempt
would seem to be what forced the change.

https://groups.google.com/g/sage-support/c/DgPxSZdmqcY/m/1gOsNbNJBgAJ

Andrey Novoseltsev

unread,
Apr 1, 2021, 1:01:01 PMApr 1
to sage-cell
Most of the sites requested should be operational, please test and let me know your results! Exact IPs are here:

This one is behind Cloudflare, so does not have a stable IP and whitelisting the whole Cloudflare does not seem to be a good idea to me:

David Austin

unread,
Apr 1, 2021, 1:24:02 PMApr 1
to sage-cell
That's very helpful.  Thanks for your work on this. 

Michael Miller

unread,
Apr 1, 2021, 1:26:18 PMApr 1
to sage-cell
Can you whitelist https://sourceforge.net ?

Thanks!

Andrey Novoseltsev

unread,
Apr 1, 2021, 3:57:49 PMApr 1
to sage-cell
On Thursday, 1 April 2021 at 11:26:18 UTC-6 Michael Miller wrote:
Can you whitelist https://sourceforge.net ?
Done!

Elimboto Yohana

unread,
Apr 1, 2021, 5:36:59 PMApr 1
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for your great work. You have taken care of all URLs except where mentioned otherwise.

But I think due to some upgrade, most of the astrophysics codes that worked in the past are not working, the common error is

"AttributeError: module 'snappy' has no attribute 'compress'".

Another error I experienced is "AttributeError: partially initialized module 'fsspec' has no attribute 'utils' (most likely due to a circular import)". This error may be related to the first one.

It seems that the snappy package is missing, and I request installation: pip3 install python-snappy (see the discussion here https://github.com/dask/fastparquet/issues/459)

Can I request whitelisting our forum URL: https://www.tssfl.com? Sometimes we upload files that can be read directly from it.

Kind regards,

Elimboto


--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

William Stein

unread,
Apr 3, 2021, 10:21:24 AMApr 3
to Andrey Novoseltsev, sage-cell
Hi,

There's an active discussion on Hacker News right now about
cryptomining abuse of GitHub, with people working on other sites
talking about a wide range of things they do to combat the problem:

https://news.ycombinator.com/item?id=26678700

One comment has an estimate of how much money one of GitHub
cryptomining abusers might be making: "Judging by the hash rate, the
pool estimates a weekly income of 700$. It could be a multiple of this
if the culprit uses multiple wallets at once."

-- William
> --
> You received this message because you are subscribed to the Google Groups "sage-cell" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-cell/7c758ba4-74e5-4827-89a7-f68ea2713a58n%40googlegroups.com.



--
William (http://wstein.org)

Andrey Novoseltsev

unread,
Apr 5, 2021, 5:42:45 PMApr 5
to sage-cell
It seems that the snappy package is missing, and I request installation: pip3 install python-snappy (see the discussion here https://github.com/dask/fastparquet/issues/459)
Done!
Can I request whitelisting our forum URL: https://www.tssfl.com? Sometimes we upload files that can be read directly from it.
Done!

Elimboto Yohana

unread,
Apr 6, 2021, 5:10:31 AMApr 6
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for implementing my requests. But, re-installing the snappy package in the suggested way did not solve an issue. I tried to narrow down the source of the problem and found that importing either xarray or nbodykit.lab:

import nbodykit.lab
import xarray as xr

is the source of "AttributeError: module 'snappy' has no attribute 'compress'".

The previous versions had no issues.

Kind regards,

Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
Apr 16, 2021, 4:19:41 AMApr 16
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I just found that there are two Python-related snappy packages - both are useful packages, the first one is "for studying the topology and geometry of 3-manifolds, with a focus on hyperbolic structures" -- https://snappy.math.uic.edu/, see installation method, https://pypi.org/project/snappy/

This is one currently installed, you can verify this by:

import snappy
print(snappy.__version__)
print(snappy.__file__)
print(dir(snappy))

The second one is used by a number of Python packages for compression, "Python library for the snappy compression library from Google (http://google.github.io/snappy/)": https://pypi.org/project/python-snappy/ and can be installed as

sudo apt-get install libsnappy-dev (or similar method)
followed by
pip3 install python-snappy

It seems that you need to uninstall python-snappy and reinstall it following the order above, see https://stackoverflow.com/questions/48535799/module-snappy-has-no-attribute-decompress

Kind regards,

Elimboto

Elimboto Yohana

unread,
Apr 16, 2021, 7:20:58 AMApr 16
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Someone requested the installation of the Python Packages, pyforest (https://pypi.org/project/pyforest/) and lazypredict (https://pypi.org/project/lazypredict/):

pip3 install pyforest and pip3 install lazypredict

shall that be possible.

They are using them to study machine learning models.

Kind regards.

Elimboto

Andrey Novoseltsev

unread,
Apr 19, 2021, 4:16:22 PMApr 19
to Elimboto Yohana, sage-cell
I've added libsnappy-dev system package - can you please check that your code works as intended now? If not please send me the code that raises the error.

Andrey Novoseltsev

unread,
Apr 19, 2021, 4:17:59 PMApr 19
to Elimboto Yohana, sage-cell
I've installed pyforest

lazypredict requires fixed (and old) versions of many packages, so it breaks other installations. Those who need it should bug developers to use >= for versions, not == ;-)

Elimboto Yohana

unread,
Apr 20, 2021, 8:31:22 AMApr 20
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for installing pyforest.

I'm not clear how lazypredict can be used in the way you suggested.

For the Snappy package, I think the solution is to uninstall the current Snappy and install the one in question (python-snappy) using the suggested approach.

Python-snappy seems to impact many codes even a simple import as this one, import plotly.express as px will raise an error "AttributeError: module 'snappy' has no attribute 'compress'".

Or maybe it is time to consider Python virtual environment/virtualenvwrapper (https://www.tssfl.com/python-virtual-environments-virtualenv-5558) or something similar to separate various packages and their dependencies.

Kind regards,

Elimboto

Andrey Novoseltsev

unread,
Apr 20, 2021, 12:20:15 PMApr 20
to sage-cell
Hi Elimboto!

On Tuesday, 20 April 2021 at 06:31:22 UTC-6 e...@tssfl.co wrote:
I'm not clear how lazypredict can be used in the way you suggested.

My point was that its installation instructions are "wrong" and we can't install it until developers fix it to make it installable alongside other packages.
 
For the Snappy package, I think the solution is to uninstall the current Snappy and install the one in question (python-snappy) using the suggested approach.

Python-snappy seems to impact many codes even a simple import as this one, import plotly.express as px will raise an error "AttributeError: module 'snappy' has no attribute 'compress'".

Servers were rebuilt from scratch, I never uninstall packages. Does python-snappy conflict with snappy and so we have to drop one or the other?
 
Or maybe it is time to consider Python virtual environment/virtualenvwrapper (https://www.tssfl.com/python-virtual-environments-virtualenv-5558) or something similar to separate various packages and their dependencies.

Impossible in the current approach of having forked kernels ready and waiting for the user code. Besides it would require users to pick which environment they want to use and I seriously doubt that there is much point in implementing and supporting this. If someone's needs are so specific that they really need particular packages that do not play well with others, they probably should use local installation or alternatives like CoCalc, SageMathCell was intended as a simple way to run simple bits of code with as little configuration and choices as possible.

Having added quite a few packages over the years, my claim is that if some package cannot be installed and used alongside other packages - it is this package fault, not ours for not providing isolated environments.

Thank you!
Andrey

Andrey Novoseltsev

unread,
Apr 26, 2021, 10:57:42 AMApr 26
to Elimboto Yohana, sage-cell
Hi Elimboto,

Plain "snappy" package was installed because it was requested by another user before you asked for "python-snappy". Uninstalling the first and then installing the second does fix your problem, but obviously it makes the first package unavailable, so I am hesitant to make such a change. I have written to snappy developers and asked if they can come up with some idea to install packages simultaneously without using virtual environments, we'll see. The fastest solution for you may be to run your own server where you are in full control of installed packages, but while I tried to make initial installation simple, it does come with some maintenance overhead ;-)

Thank you!
Andrey

Elimboto Yohana

unread,
Apr 26, 2021, 4:55:48 PMApr 26
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for your response. I did not know that someone requested snappy, it is however useful. Frankly speaking, I prefer the SageMathCell. Let's wait for the reply from snappy developers.

Kind regards,
Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Andrey Novoseltsev

unread,
Apr 26, 2021, 10:36:38 PMApr 26
to Elimboto Yohana, sage-cell
The reply is that nothing can be done short of renaming one of the packages. Since you don't actually need python-snappy directly, perhaps there is a way for the packages that you care about to avoid using it?.. My understanding is that it is just some compression library, surely there are other available.

Andrey Novoseltsev

unread,
Apr 28, 2021, 11:05:07 PMApr 28
to Elimboto Yohana, sage-cell
On Tue, Apr 27, 2021 at 9:58 PM Elimboto Yohana <e...@tssfl.com> wrote:
NB: I'm facing issues connecting to Dropbox, this wasn't there before access restriction, may there be something wrong with Dropbox access/whitelisting?

Can you please post the command that does not work? It may be helpful if you also run host ... to see the IP address, although as far as I understand the resolution is done on our servers and geographic location should not matter...

Elimboto Yohana

unread,
May 3, 2021, 10:38:05 PMMay 3
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I use Google Drive API to perform operations on Google spreadsheets (e.g., here https://www.tssfl.com/automating-the-google-spreadsheet-tasks-with-python-and-tssfl-open-discussion-forums-6402), it turns out the problem is not Dropbox, but sometimes the Google API is slow, especially if using other browsers than Chrome.

Kind regards,
Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
May 17, 2021, 11:20:17 AMMay 17
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I will appreciate if you can install Google API Client Library for Python (https://pypi.org/project/google-api-python-client/): pip install google-api-python-client

It provides access to Google services, such as spreadsheets for data science.

Kind regards,

Elimboto

Andrey Novoseltsev

unread,
May 23, 2021, 6:57:45 PMMay 23
to sage-cell
On Monday, 17 May 2021 at 09:20:17 UTC-6 e...@tssfl.co wrote:
Hi Andrey,

I will appreciate if you can install Google API Client Library for Python (https://pypi.org/project/google-api-python-client/): pip install google-api-python-client

It seems to be installed - do you have problems using it? Perhaps it is due to network blocking.

Elimboto Yohana

unread,
May 30, 2021, 4:11:17 PMMay 30
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Can you please whitelist the Woldometers and Twitter URLs: https://www.worldometers.info/coronavirus/, www.twitter.com

They are useful for a number of analytics via web scrapping with Python -- Selenium, BeautifulSoup, etc.,

In addition, I request the installation of the following related library:

import tweepy (pip install tweepy) -- Python library for Twitter API.

Kind regards,

Elimboto

On Mon, Mar 22, 2021 at 5:57 AM Andrey Novoseltsev <novo...@gmail.com> wrote:
Unfortunately, due to some people abusing SageMathCell servers access to the Internet from cells is going to be removed or severely limited.

If you really need such access, your options are:
- maintain your own server, e.g. behind a campus firewall
- use CoCalc, where you can enable such access
- explain exactly what kind of access you need and why - perhaps we will figure something out, but no guarantees on how long it will take, if ever

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Elimboto Yohana

unread,
Jun 1, 2021, 4:09:56 PMJun 1
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I also request the installation of the Python library bioinfokit: pip install bioinfokit (https://pypi.org/project/bioinfokit/0.3c/) for biological data analyses.

Kind regards,

Elimboto

Elimboto Yohana

unread,
Jun 2, 2021, 12:29:13 PM (13 days ago) Jun 2
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you for implementing my request, packages are working fine. The only thing that does not work is the Worldometer website access, here is the error:
ConnectionError: HTTPSConnectionPool(host='www.worldometers.info', port=443): Max retries exceeded with url: /coronavirus/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7f2c5da8b0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

Kind regards,
Elimboto

Andrey Novoseltsev

unread,
Jun 2, 2021, 11:44:08 PM (13 days ago) Jun 2
to sage-cell
Hi Elimboto!

You are too fast - I was waiting for servers to finish updating before reporting that requests have been completed. Except for worldometers - they are behind Cloudflare and so don't have static IPs.

Best,
Andrey

Elimboto Yohana

unread,
Jun 11, 2021, 8:12:01 AM (4 days ago) Jun 11
to Andrey Novoseltsev, sage-cell
Hi Andrey,

I would appreciate it if you can also install the following two Python libraries for Topological Data Analysis and R tidyverse packages.

1) scikit-tda: pip install scikit-tda

A high-performance topological machine learning toolbox in Python built on top of scikit-learn:

2) giotto-tda (https://arxiv.org/abs/2004.02551): pip install giotto-tda

3)   R tidyverse packages for statistics and data science (https://www.tidyverse.org/packages/): install.packages("tidyverse")

Kind regards,
Elimboto



On Thu, Jun 3, 2021 at 8:03 AM Elimboto Yohana <e...@tssfl.com> wrote:
Thank you Andrey. I find a workaround to read COVID-19 daily data from worldometers: pip install covid-daily (https://pypi.org/project/covid-daily/). I hope it works.

Kind regards,

Elimboto




--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.

Andrey Novoseltsev

unread,
Jun 13, 2021, 11:11:15 PM (2 days ago) Jun 13
to sage-cell
Hi Elimboto,

All should be available now!

Andrey

Elimboto Yohana

unread,
Jun 14, 2021, 6:07:35 AM (yesterday) Jun 14
to Andrey Novoseltsev, sage-cell
Hi Andrey,

Thank you very much for installing the packages. I Should actually have noticed that covid_daily requires Worldometers URL to retrieve COVID data. As it is behind CloudFare, I experience the same error. Maybe there will be a way out in the future.

Kind regards,

Elimboto

--
You received this message because you are subscribed to the Google Groups "sage-cell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-cell+...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages