Div returns empty.

119 views
Skip to first unread message

Chaplin Gund

unread,
Sep 2, 2021, 12:20:30 PM9/2/21
to beautifulsoup

I am trying to get the players from a minecraft dynmap with help of webscraper, but it only prints the name of the div. How can i get the whole html content of the page/div?

```import requests from bs4
 import BeautifulSoup URL = "http://play.fantasycraftmc.eu:25570/?worldname=kingdom&mapname=flat&zoom=0&x=3904&y=64&z=2176#/" page = requests.get(URL) soup = BeautifulSoup(page.content, "html.parser") div = soup.find(id="mcmap") print(div)
```

facelessuser

unread,
Sep 2, 2021, 1:17:17 PM9/2/21
to beautifulsoup

You can print the full HTML by just printing the soup object.

import requests
from bs4 import BeautifulSoup
URL = "http://play.fantasycraftmc.eu:25570/?worldname=kingdom&mapname=flat&zoom=0&x=3904&y=64&z=2176#/"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser"
)
print(soup)

What is interesting is that you’ll see that the content you are after isn’t there. This is because the content is dynamically loaded with JavaScript. BeautifulSoup does execute JavaScript on the returned HTML. It may be useful to learn: https://github.com/SeleniumHQ/selenium.

➜  soupsieve git:(chore/upgrade-action-deps) ✗ python3 bug.py                                                                                
/usr/local/lib/python3.9/site-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.2) or chardet (4.0.0) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
<!DOCTYPE html>

<html lang="en">
<head>
<title>Minecraft Dynamic Map</title>
<meta charset="utf-8"/>
<meta content="minecraft, map, dynamic" name="keywords"/>
<meta content="Minecraft Dynamic Map" name="description"/>
<meta content="width=device-width, initial-scale=1.0, user-scalable=no" name="viewport"/>
<!-- These 2 lines make us fullscreen on apple mobile products - remove if you don't like that -->
<meta content="yes" name="apple-mobile-web-app-capable"/>
<meta content="black-translucent" name="apple-mobile-web-app-status-bar-style"/>
<link href="images/dynmap.ico" rel="icon" type="image/ico"/>
<script src="js/jquery-3.5.1.js?_=3.2-beta-2-483" type="text/javascript"></script>
<link href="css/leaflet.css?_=3.2-beta-2-483" rel="stylesheet" type="text/css"/>
<script src="js/leaflet.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/custommarker.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/dynmaputils.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/sidebarutils.js?_=3.2-beta-2-483" type="text/javascript"></script>
<!--<link rel="stylesheet" type="text/css" href="css/embedded.css" media="screen" />-->
<link href="css/standalone.css?_=3.2-beta-2-483" media="screen" rel="stylesheet" type="text/css"/>
<link href="css/dynmap_style.css?_=3.2-beta-2-483" media="screen" rel="stylesheet" type="text/css"/>
<!-- <link rel="stylesheet" type="text/css" href="css/override.css" media="screen" /> -->
<script src="version.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/jquery.json.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/jquery.mousewheel.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/minecraft.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/map.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="js/hdmap.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script src="standalone/config.js?_=3.2-beta-2-483" type="text/javascript"></script>
<script type="text/javascript">
                        $(document).ready(function() {
                                window.dynmap = new DynMap($.extend({
                                        container: $('#mcmap')
                                }, config));
                        });
        </script>
</head>
<body>
<noscript>
 For full functionality of this site it is necessary to enable JavaScript.
 Here are the <a href="http://www.enable-javascript.com/" target="_blank">
 instructions how to enable JavaScript in your web browser</a>.
</noscript>
<div id="mcmap"></div>
</body>
</html>
​

Chaplin Gund

unread,
Sep 2, 2021, 1:26:10 PM9/2/21
to beautifulsoup
i got that, but i want the whole html of the mcmap div.

Op donderdag 2 september 2021 om 19:17:17 UTC+2 schreef facelessuser:

facelessuser

unread,
Sep 2, 2021, 1:32:15 PM9/2/21
to beautifulsoup
You are going to have to look at https://github.com/SeleniumHQ/selenium and learn how to use that to get the full HTML content. Then you can scape it. Maybe there is someone here who will give assistance by showing an example. Or you can look at StackOverflow and maybe there is a post there with good examples: https://stackoverflow.com/search?q=how+to+use+selenium.
Reply all
Reply to author
Forward
0 new messages