taming the beast - asizeof refactoring
Robert Schuppenies
I was having a really hard time coming up with tests for asizeof. It was a
beast to me, which I just could not tame. As a last resort, I tried it with
devide and conquer: I refactored asizeof. I did it in a branch and just to see
how far I would get, but after half a day I am pretty happy with the results.
But since you are the owner of this beast, it is your call whether I proceed or
not.
The pros for me are:
- it is easier to understand
- it helps others who havn't been growing up with the code to maintain it
- it is better testable
- it allows better code reuse with other pympler components (e.g. printf)
Before you go and check out the code, please consider
- I have not changed the interface
- I had only half a day (-> it definitely needs more work)
- I did not have enough tests (-> something might be broken, although the test
suite which we had for asizeof before passes and the demos you included look good)
The current layout:
- asizeof: the core functionality
- common : code used by several modules
- compat : code required for Py3k compatibility
- demos : all the demos you provided
- sizes : code related to sizes of objects
- type_..: code related to type introspection
Again, there is more work left to be done. For example,
- the type_.. dependencies are ugly.
- sizes is too small to be self-sustainable
- I would like to put the print_ functions into a separate module
But I think it could be a pretty neat set of modules, providing a consistent
API at a package level.
The code can be find here:
http://code.google.com/p/pympler/source/browse/#svn/branches/asizeof-splitup
cheers,
robert
BEFORE REFACTORING
==================
$ wc pympler/asizeof/*.py
32 173 1054 pympler/asizeof/__init__.py
2595 10206 89078 pympler/asizeof/asizeof.py
2627 10379 90132 total
--------------------------------------------------------------------------------
$ sloccount pympler/asizeof/asizeof.py
Have a non-directory at the top, so creating directory top_dir
Adding /home/schuppe/workspaces/pympler/svn/trunk/pympler/asizeof/asizeof.py to
top_dir
Categorizing files.
Finding a working MD5 command....
Found a working MD5 command.
Computing results.
SLOC Directory SLOC-by-Language (Sorted)
2075 top_dir python=2075
Totals grouped by language (dominant language first):
python: 2075 (100.00%)
Total Physical Source Lines of Code (SLOC) = 2,075
--------------------------------------------------------------------------------
./run.py --test --coverage --cov-cmd=coverage.py test/asizeof/
Running unittests (64-bit Python 2.5.2)
=======================================
Python 2.5.2 (r252:60911, Jul 31 2008, 17:31:22)
64-bit [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)]
Test asizeof.flatsize() ... ok
Test asizeof.adict() ... ok
Test asizeof.asized() ... ok
Test asizeof.asizeof() ... ok
Test asizeof.asizesof() ... ok
Test asizeof.basicsize() ... ok
Test asizeof.itemsize() ... ok
Test asizeof.leng() ... ok
Test asizeof.refs() ... ok
Test sizing class objects and instances ... ok
Test integrity of sized generator ... ok
Test sizing methods and functions ... ok
----------------------------------------------------------------------
Ran 12 tests in 0.103s
OK
Name Stmts Exec Cover Missing
--------------------------------------------------------
pympler/__init__ 0 0 100%
pympler/asizeof/__init__ 16 14 87% 7, 22
pympler/asizeof/asizeof 1390 713 51% 179, 194, 210-211, 213,
217-218, 223, 237, 268-271, 277-279, 286-288, 297-303, 307-313, 338-341,
366-367, 376-383, 397-399, 410, 415-422, 437-443, 449-453, 458-462, 467-470,
475, 480-490, 501, 515-516, 518-520, 526-532, 537, 568, 574, 579, 584, 612-613,
619-622, 627, 637, 642, 647, 652, 658-661, 677-678, 683, 688, 693, 704,
710-715, 730, 736-741, 749, 757, 764, 772-775, 785-788, 793, 820-823, 861,
880-881, 883, 900-915, 937, 957, 963-971, 982-987, 996, 1004-1009, 1048, 1055,
1059, 1065, 1069, 1073, 1077, 1119-1120, 1130-1131, 1137-1138, 1142-1143,
1147-1148, 1152, 1155-1156, 1164, 1174-1175, 1180, 1184-1185, 1189-1190,
1193-1194, 1198-1199, 1204-1205, 1209-1210, 1214-1215, 1219-1220, 1223-1224,
1228-1229, 1233-1234, 1241-1242, 1247-1248, 1252-1253, 1258-1259, 1263-1264,
1268-1269, 1273-1274, 1283-1284, 1287-1288, 1291-1292, 1296-1297, 1301-1302,
1308-1309, 1312-1313, 1317-1318, 1342, 1346, 1361, 1363, 1369, 1376, 1378,
1380, 1382, 1385-1388, 1390, 1394, 1412-1420, 1423, 1428-1438, 1445-1452, 1476,
1522, 1532, 1537-1540, 1557, 1564, 1573, 1585-1586, 1596-1597, 1612-1613, 1633,
1644, 1653, 1674-1677, 1689-1713, 1734-1763, 1774-1794, 1803-1816, 1839-1841,
1847, 1849, 1854, 1861, 1867, 1873, 1914, 1925-1935, 2095-2097, 2186-2222,
2227-2557
pympler/gui/__init__ 0 0 100%
pympler/gui/garbage 25 0 0% 1-71
pympler/gui/graph 168 0 0% 7-310
pympler/metadata 9 0 0% 7-16
pympler/muppy/__init__ 7 0 0% 7-19
pympler/muppy/mprofile 47 0 0% 5-94
pympler/muppy/muppy 122 0 0% 1-238
pympler/muppy/refbrowser 192 0 0% 20-426
pympler/muppy/summary 136 0 0% 46-300
pympler/muppy/tracker 93 0 0% 12-245
pympler/process 72 0 0% 20-141
pympler/sizer/__init__ 4 0 0% 3-7
pympler/sizer/asizeof 1364 0 0% 141-2431
pympler/tracker/__init__ 1 0 0% 1
pympler/tracker/classes 200 0 0% 8-471
pympler/tracker/stats 394 0 0% 1-698
pympler/util/__init__ 0 0 100%
pympler/util/compat 23 0 0% 5-47
pympler/util/stringutils 22 0 0% 1-42
--------------------------------------------------------
TOTAL 4285 727 16%
AFTER REFACTORING
=================
wc pympler/asizeof/*.py
32 173 1054 pympler/asizeof/__init__.py
1262 5416 45755 pympler/asizeof/asizeof.py
94 325 2568 pympler/asizeof/common.py
68 223 1819 pympler/asizeof/compat.py
258 986 9699 pympler/asizeof/demos.py
64 285 2351 pympler/asizeof/sizes.py
184 699 5795 pympler/asizeof/type_common.py
187 640 6299 pympler/asizeof/type_repo.py
619 1894 19845 pympler/asizeof/type_spec.py
2768 10641 95185 total
--------------------------------------------------------------------------------
./run.py --test --coverage --cov-cmd=coverage.py test/asizeof/
Creating filelist for asizeof
Categorizing files.
Finding a working MD5 command....
Found a working MD5 command.
Computing results.
SLOC Directory SLOC-by-Language (Sorted)
2189 asizeof python=2189
Totals grouped by language (dominant language first):
python: 2189 (100.00%)
Total Physical Source Lines of Code (SLOC) = 2,189
--------------------------------------------------------------------------------
./run.py --test --coverage --cov-cmd=coverage.py test/asizeof/
Running unittests (64-bit Python 2.5.2)
=======================================
Python 2.5.2 (r252:60911, Jul 31 2008, 17:31:22)
64-bit [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)]
Test asizeof.flatsize() ... ok
Test asizeof.adict() ... ok
Test asizeof.asized() ... ok
Test asizeof.asizeof() ... ok
Test asizeof.asizesof() ... ok
Test asizeof.basicsize() ... ok
Test asizeof.itemsize() ... ok
Test asizeof.leng() ... ok
Test asizeof.refs() ... ok
Test sizing class objects and instances ... ok
Test integrity of sized generator ... ok
Test sizing methods and functions ... ok
----------------------------------------------------------------------
Ran 12 tests in 0.100s
OK
Name Stmts Exec Cover Missing
-----------------------------------------------------------
pympler/__init__ 0 0 100%
pympler/asizeof/__init__ 16 14 87% 7, 22
pympler/asizeof/asizeof 566 259 45% 190, 195-202, 212-218,
224-228, 233-237, 242, 247-253, 258, 272-280, 283, 288-298, 305-312, 336, 382,
392, 397-400, 417, 424, 433, 445-446, 456-457, 472-473, 493, 504, 513, 534-537,
549-573, 594-623, 634-654, 663-676, 699-701, 707, 709, 714, 721, 727, 733, 774,
785-795, 1034-1070, 1075-1224
pympler/asizeof/common 48 33 68% 6, 72-73, 75-77, 84-94
pympler/asizeof/compat 41 20 48% 23-26, 32-34, 41-43, 52-58,
62-68
pympler/asizeof/demos 190 46 24% 21-22, 27, 40, 46-49,
53-60, 66-67, 70-75, 78-81, 84-93, 96-102, 105-120, 123-133, 136-142, 145-158,
162-172, 175-177, 181-198, 201-230, 233-235, 238-240, 243-248, 251-258
pympler/asizeof/sizes 42 34 80% 16, 35-36, 38, 42-43, 48, 62
pympler/asizeof/type_common 110 87 79% 11-12, 37-38, 53, 66,
89-92, 130, 149-150, 152, 169-184
pympler/asizeof/type_repo 104 75 72% 11, 46, 52-60, 71-76, 85,
93-98, 137, 144, 148, 154, 158, 162, 166
pympler/asizeof/type_spec 385 238 61% 18-21, 52, 58, 63, 68,
96-97, 103-106, 111, 121, 126, 131, 136-139, 153, 158, 163, 174, 180-185, 200,
206-211, 219, 227, 234, 242-245, 255-258, 263, 281, 285, 300, 302, 308, 315,
317, 319, 321, 324-327, 329, 333, 346-348, 354-357, 371-373, 378-385, 407-408,
418-419, 425-426, 430-431, 435-436, 440, 443-444, 452, 462-463, 468, 472-473,
477-478, 481-482, 486-487, 492-493, 497-498, 502-503, 507-508, 511-512,
516-517, 521-522, 529-530, 535-536, 540-541, 546-547, 551-552, 556-557,
561-562, 571-572, 575-576, 579-580, 584-585, 589-590, 596-597, 600-601, 605-606
pympler/gui/__init__ 0 0 100%
pympler/gui/garbage 25 0 0% 1-71
pympler/gui/graph 168 0 0% 7-310
pympler/metadata 9 0 0% 7-16
pympler/muppy/__init__ 7 0 0% 7-19
pympler/muppy/mprofile 47 0 0% 5-94
pympler/muppy/muppy 122 0 0% 1-238
pympler/muppy/refbrowser 192 0 0% 20-426
pympler/muppy/summary 136 0 0% 46-300
pympler/muppy/tracker 93 0 0% 12-245
pympler/process 72 0 0% 20-141
pympler/sizer/__init__ 4 0 0% 3-7
pympler/sizer/asizeof 1364 0 0% 141-2431
pympler/tracker/__init__ 1 0 0% 1
pympler/tracker/classes 200 0 0% 8-471
pympler/tracker/stats 394 0 0% 1-698
pympler/util/__init__ 0 0 100%
pympler/util/compat 23 0 0% 5-47
pympler/util/stringutils 22 0 0% 1-42
-----------------------------------------------------------
TOTAL 4381 806 18%