Python-nvd3

0 views

Skip to first unread message

Delmiro Fain

unread,

Aug 3, 2024, 5:43:53 PM8/3/24

to canthortetoll

The following content describes a subset of the statistics features available in WiredTiger. For more complete information see the statistics page in the WiredTiger documentation. This content focuses on analyzing and reviewing the data generated by the statistics logging functionality.

By default statistics are disabled in WiredTiger - this is because tracking statistics has a performance overhead. WiredTiger has two modes for tracking statistics fast and all. Fast statistics are those that don't incur a significant performance overhead. The effect on performance of enabling statistics varies depending on the database and access characteristics. To enable statistics you should include either statistics=[fast,clear] or statistics=[all,clear] in the configuration string to wiredtiger_open.

Alternatively you can set the WIREDTIGER_CONFIG environment variable to include the relevant configuration string. Note that the environment variable is checked at wiredtiger_open time - changing the environment variable after WiredTiger is already running will not alter the behavior of WiredTiger.

WiredTiger provides two methods for accessing statistics: using a statistics: uri in the WiredTiger WT_CURSOR API and using the built in statistics logging functionality. The remainder of this document assumes the use of the statistics logging functionality as the tools provided parse the output that it generates.

Enabling statistics logging causes WiredTiger to generate files containing statistics information in the database directory. To enable statistics logging include statistics_log=(wait=30) in the configuration string to wiredtiger_open.

The wait option configures how often the statistics are written to the log file in seconds. Gathering and writing statistics has overhead - so this period should not be too small. The default location for statistics log files is in the WiredTiger database directory. The files are named WiredTigerStat.XX, where XX corresponds to the current hour. A new file is created for each hour. If your application runs for more than 24 hours and the file names wrap, the statistics logging output is appended to the already existing files. Statistics log file names are configurable. The output in the statistics log files is textual and human readable.

D3 is a standard for generating interactive graphs in a HTML page. There is a python program in the WiredTiger source tree at tools/wtstats.py that can be used to generate a D3 html page based on WiredTiger statistics log(s). In order to use the Python script you need to have Python installed and a recent version of the Python nvd3 library.

There are several optional arguments. Run with -h or --help to see them. For example, -o file or --output file redirects where the HTML output goes. Also --focus generates a chart that contains a focus slider to allow you to zoom in on particular time periods.

You can open the generated HTML document in your browser, and see the generated statistics. The keys in the graph are clickable - double clicking on one of the keys will show only that statistic. Single clicking will enable/disable a particular statistic.

The following steps take you through the process of generating a D3 graph of WiredTiger statistics based on the statistics generated from running an application included in the WiredTiger tree called wtperf. These instructions assume you have all the necessary tools to build WiredTiger, Python and python-nvd3 installed.

If you double click on the "btree cursor insert calls per second" key, it will show just that statistic in the window. Depending on the test you ran (I ran a modified version of the update-large-lsm.wtperf configuration), the graph may look something like:

From this particular statistic we can gather that it's unlikely the cursor insert rate that is a bottle neck in this particular test - since we could achieve an insert rate over double the steady state insert rate when required.

Let's Make a D3 Plugin showsyou how to create your own reusable plugin that you can use acrossmultiple visualizations as a separate JavaScript library. d3.jsversion 4 is now widely used so don't worry about the note at thestart of the article.

Make great-looking d3.js charts in Python without coding a line of JavaScriptcombines a Python backend with thepython-nvd3 library togenerate d3.js charts without having to hand-write the JavaScript code.If you are interested in a solution like this for your own visualizationsthen you should also check out Bokeh.

How to make a modern dashboard with NVD3.jsuses the NVD3.js library that works as anabstraction on top of d3.js to create charts. The post puts togetherseveral charts to show how to build a dashboard based on public JSONdata.

The trouble with D3is not a tutorial but it's an important read because it discusses whyD3 can be very difficult to learn: the learning curve depends on yourbackground. If you are a front-end developer you will likely have theeasiest time if you already understand JavaScript,SVG and the browser Document Object Model (DOM). Non-technical designersand analysts typically have the hardest time with using D3 because thenot only have to learn the tool itself but all the concepts andweb browser technologies that it is built upon.

Thisvisualization of d3.js modules and resourcesis a wonderful map for understanding how various parts of the libraryand the ecosystem fit together. The visual provides a useful map for whereto concentrate your learning depending on what type of visualization youare working to build.