Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

LOB space, with numbers

109 views
Skip to first unread message

Mladen Gogala

unread,
May 12, 2012, 9:58:57 PM5/12/12
to
In the last thread, I claimed that Oracle space management within the LOB
segments leaves a lot to be desired. I didn't have the numbers and this
thread is started to correct that mistake. Now, the numbers are here.
I loaded a bunch of binary files, 614 of them, into the database. Space
consumption was 2.3GB,

SQL> select bytes/1048576 from dba_segments
2 where owner='SCOTT' and
3 segment_type='LOBSEGMENT' and
4 segment_name='MUSIC';

BYTES/1048576
-------------
2368

After that, I deleted 47 files:

SQL> delete from scott.bin_files
2 where filename like '/home/mgogala/mp3/misc/The%';

47 rows deleted.


Those files were loaded in a separate table, defined exactly like the
original, to measure the space they consume.

SQL> select bytes/1048576 from dba_segments
2 where owner='SCOTT' and
3 segment_type='LOBSEGMENT' and
4 segment_name='MUSIC1';

BYTES/1048576
-------------
168

The space consumption of the files is 168 MB.

What will happen when we reload those files? The files that were deleted
are exactly the same so, as the theory goes, those files should fit
nicely into the same holes that were made by their removal from the
database. Of course, the order of the insert is not the same so we can
expect some space wastage, but not too much. The files being loaded range
in size from 2MB to 6.5MB:


SQL> select max(length(content)),avg(length(content)),min(length(content))
2 from bin_files_aux;

MAX(LENGTH(CONTENT)) AVG(LENGTH(CONTENT)) MIN(LENGTH(CONTENT))
-------------------- -------------------- --------------------
6212454 3620058.74 2030876

Elapsed: 00:00:00.01
SQL> select count(*) from bin_files_aux;

COUNT(*)
----------
47

Elapsed: 00:00:00.00

So, how much space will be consumed after we reload the data?

SQL> select bytes/1048576 from user_Segments
2 where segment_name='MUSIC';

BYTES/1048576
-------------
2496


So, after re-loading 168MB of data, the size of the LOB segment is
extended for a whopping 128MB. This is far too much. I would say that
there is definitely a bug with the space management within LOB segments.
This was for BASICFILE storage, which means that it is likely to
encounter the same issue with version 10G.
Now, let's try SECUREFILE storage. The bin_files DDL now looks like this:

CREATE TABLE "SCOTT"."BIN_FILES"
( "FILENO" NUMBER(4,0),
"FILENAME" VARCHAR2(256),
"CONTENT" BLOB
) SEGMENT CREATION IMMEDIATE
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
NOCOMPRESS LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
TABLESPACE "TEST_TBS"
LOB ("CONTENT") STORE AS SECUREFILE "MUSIC"(
TABLESPACE "TEST_LOB" DISABLE STORAGE IN ROW CHUNK 32768
CACHE NOCOMPRESS KEEP_DUPLICATES
STORAGE(INITIAL 106496 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE 0
BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)) ;

Off the bat, the load into the "SESCUREFILE" version is noticeably faster:

[mgogala@medo tmp]$ time ./test_lob
614 files loaded.

real 7m33.060s
user 0m5.751s
sys 0m4.416s

The best time with "BASICFILE" was 8 min 53 sec and with SECUREFILE LOB,
files were loaded in 7 min 33 seconds. Second, the LOB itself is larger
than the BASICFILE version:

SQL> select round(bytes/1048576,2) as MB from user_segments
2 where segment_type='LOBSEGMENT' and
3 segment_name='MUSIC';

MB
----------
2497.13

BASICFILE version consumed 2368MB of disk space while the SECUREFILE
consumed almost 2500MB. So, let's repeat the exercise:

SQL> delete from scott.bin_files
2 where filename like '/home/mgogala/mp3/misc/The%';

47 rows deleted.

Elapsed: 00:00:00.06
SQL> commit;

Commit complete.

Elapsed: 00:00:00.03
SQL> insert into bin_files
2 select * from bin_files_aux;

47 rows created.

Elapsed: 00:00:21.72
SQL> commit;

Commit complete.

Elapsed: 00:00:00.01
SQL> select round(bytes/1048576,2) as MB from user_segments
2 where segment_type='LOBSEGMENT' and
3 segment_name='MUSIC';

MB
----------
2497.13

Elapsed: 00:00:00.02
SQL>


Now this is a surprise! The segment hasn't grown at all! It is still
larger than the BASICFILE segment after the reload but at least an
exorbitant growth is not there. When I repeated the insert, the LOB
segment growth was moderate, less than the size of the batch that was
loaded:

SQL> insert into bin_files
2 select * from bin_files_aux;

47 rows created.

Elapsed: 00:00:31.75
SQL> commit;

Commit complete.

Elapsed: 00:00:00.00
SQL> select round(bytes/1048576,2) as MB from user_segments
2 where segment_type='LOBSEGMENT' and
3 segment_name='MUSIC';

MB
----------
2625.13

Elapsed: 00:00:00.04

Remember, BIN_FILES_AUX contains 47 MP3 files and the LOB segment size
168MB. The segment grew for exactly 128MB, just as the BASICFILE version.
My conclusion is that there definitely is a space management problem with
the BASICFILE version. I am not sure about the SECUREFILE version but the
initial allocation is noticeably larger. There is a definite speed
improvement, too. Also, loading speed was consistent with the SECUREFILE
LOB column, with 3 attempts, finishing in 7:33, 7:32 and 7:36 minutes.
With BASICFILE, the times were ranging between 8:53 and 12:10 minutes.
Of course, with SECUREFILE, defragmentation doesn't work:

SQL> alter table bin_files_aux modify lob(content) (shrink space);

Table altered.

Elapsed: 00:00:00.16
SQL> alter table bin_files modify lob(content) (shrink space);
alter table bin_files modify lob(content) (shrink space)
*
ERROR at line 1:
ORA-10635: Invalid segment or tablespace type

The "AUX" version stores the files beginning with "The" and the LOB
column is BASICFILE. I will try tomorrow with the manual space
management in the tablespace. I will not try compression because the
files being loaded are binary mp3 files and compression wouldn't be very
efficient.


--
http://mgogala.byethost5.com

Jonathan Lewis

unread,
May 13, 2012, 2:33:35 AM5/13/12
to
"Mladen Gogala" <gogala...@gmail.com> wrote in message
news:pan.2012.05...@gmail.com...
|
| SQL> select bytes/1048576 from dba_segments
| 2 where owner='SCOTT' and
| 3 segment_type='LOBSEGMENT' and
| 4 segment_name='MUSIC';
|
| BYTES/1048576
| -------------
| 2368
|
| So, how much space will be consumed after we reload the data?
|
| SQL> select bytes/1048576 from user_Segments
| 2 where segment_name='MUSIC';
|
| BYTES/1048576
| -------------
| 2496
|
|
| So, after re-loading 168MB of data, the size of the LOB segment is
| extended for a whopping 128MB.


But you're looking at segment size, not used block size, so there's room
for error in how much further the high water mark has moved. In this case
your figues is probably an UNDER-estimate.

I think you're using auto undo retention, and have the RETENTION keyword in
the LOB definition (from your previous thread. This means Oracle will NOT
overwrite a LOB until the amount of time is has been deleted is at least as
long as your auto undo retention time for the database - so your 168MB of
LOBs should have grown the lob segment by 168MB, not just 128MB.


| Now, let's try SECUREFILE storage. The bin_files DDL now looks like
this:
|
| LOB ("CONTENT") STORE AS SECUREFILE "MUSIC"(
| TABLESPACE "TEST_LOB" DISABLE STORAGE IN ROW CHUNK 32768
| CACHE NOCOMPRESS KEEP_DUPLICATES
| STORAGE(INITIAL 106496 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
| PCTINCREASE 0
| BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)) ;
|

There is no RETENTION keyword visible which I think means you have left it
to default, which means the same as for BASICFILES. In this case, though,
I'd guess that the space allocated already allowed enough space for the
168MB of extra data.

|
| Now this is a surprise! The segment hasn't grown at all! It is still
| larger than the BASICFILE segment after the reload but at least an
| exorbitant growth is not there. When I repeated the insert, the LOB
| segment growth was moderate, less than the size of the batch that was
| loaded:
|
| MB
| ----------
| 2625.13
|

That looks like one more extent of 128MB - again leaving us the possibility
that you first filled a previous extent and then added one more for the
overflow.
There is a small anomaly in the numbers, though, that might indicate a
timing problem, or might indicate that LOBs can add extents prematurely
when they are following a rapid growth pattern - I believe there's a
background asynchronous process that does some space management for
securefiles (but I'd have to check my notes to find out why I believe
that).


| Of course, with SECUREFILE, defragmentation doesn't work:
|

On the other hand you can specify RETENTION NONE with securefiles and not
use any of your space for retaining older versions.


--
Regards

Jonathan Lewis
http://jonathanlewis.wordpress.com
Oracle Core (Apress 2011)
http://www.apress.com/9781430239543



Mladen Gogala

unread,
May 13, 2012, 3:37:52 AM5/13/12
to
On Sun, 13 May 2012 01:58:57 +0000, Mladen Gogala wrote:

> The segment grew for exactly 128MB

This is because a new extent was allocated and it was 128MB in size. The
real problem is revealed by investigating with dbms_space. Even after
deletion, DBMS_SPACE did not show any partially filled or empty blocks,
only full blocks and unformatted blocks. That means that delete doesn't
free blocks. Here is the script which invokes DBMS_SPACE that I used:


set serveroutput on
declare
l_fs1_bytes number;
l_fs2_bytes number;
l_fs3_bytes number;
l_fs4_bytes number;
l_fs1_blocks number;
l_fs2_blocks number;
l_fs3_blocks number;
l_fs4_blocks number;
l_full_bytes number;
l_full_blocks number;
l_unformatted_bytes number;
l_unformatted_blocks number;
begin
dbms_output.enable;
dbms_space.space_usage(
segment_owner => upper('&owner'),
segment_name => upper('&segment_name'),
segment_type => upper('&segment_type'),
fs1_bytes => l_fs1_bytes,
fs1_blocks => l_fs1_blocks,
fs2_bytes => l_fs2_bytes,
fs2_blocks => l_fs2_blocks,
fs3_bytes => l_fs3_bytes,
fs3_blocks => l_fs3_blocks,
fs4_bytes => l_fs4_bytes,
fs4_blocks => l_fs4_blocks,
full_bytes => l_full_bytes,
full_blocks => l_full_blocks,
unformatted_blocks => l_unformatted_blocks,
unformatted_bytes => l_unformatted_bytes
);
dbms_output.put_line(' FS1 Blocks = '||l_fs1_blocks);
dbms_output.put_line(' FS2 Blocks = '||l_fs2_blocks);
dbms_output.put_line(' FS3 Blocks = '||l_fs3_blocks);
dbms_output.put_line(' FS4 Blocks = '||l_fs4_blocks);
dbms_output.put_line('Full Blocks = '||l_full_blocks);
dbms_output.put_line('Unformatted Blocks = '||l_unformatted_blocks);

end;
/

The test performed looks like this:

QL> @dbms_space
Enter value for owner: scott
old 17: segment_owner => upper('&owner'),
new 17: segment_owner => upper('scott'),
Enter value for segment_name: music
old 18: segment_name => upper('&segment_name'),
new 18: segment_name => upper('music'),
Enter value for segment_type: lob
old 19: segment_type => upper('&segment_type'),
new 19: segment_type => upper('lob'),
FS1 Blocks = 0
FS2 Blocks = 0
FS3 Blocks = 0
FS4 Blocks = 0
Full Blocks = 73562
Unformatted Blocks = 9160

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.01
SQL> delete from scott.bin_files
2 where filename like '/home/mgogala/mp3/misc/The%';

47 rows deleted.

Elapsed: 00:00:00.05
SQL> commit;

Commit complete.

Elapsed: 00:00:00.00
SQL> @dbms_space
Enter value for owner: scott
old 17: segment_owner => upper('&owner'),
new 17: segment_owner => upper('scott'),
Enter value for segment_name: music
old 18: segment_name => upper('&segment_name'),
new 18: segment_name => upper('music'),
Enter value for segment_type: lob
old 19: segment_type => upper('&segment_type'),
new 19: segment_type => upper('lob'),
FS1 Blocks = 0
FS2 Blocks = 0
FS3 Blocks = 0
FS4 Blocks = 0
Full Blocks = 73562
Unformatted Blocks = 9160

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.01

The block numbers before and after deletion are exactly the same. Either
DBMS_SPACE is buggy or space management within LOB segment is buggy.
Judging by the huge disk space consumption, the latter is the case.

--
http://mgogala.byethost5.com

Jonathan Lewis

unread,
May 13, 2012, 5:04:38 AM5/13/12
to

"Mladen Gogala" <gogala...@gmail.com> wrote in message
news:pan.2012.05...@gmail.com...
| On Sun, 13 May 2012 01:58:57 +0000, Mladen Gogala wrote:
|
| The real problem is revealed by investigating with dbms_space. Even after
| deletion, DBMS_SPACE did not show any partially filled or empty blocks,
| only full blocks and unformatted blocks. That means that delete doesn't
| free blocks.
|
| FS1 Blocks = 0
| FS2 Blocks = 0
| FS3 Blocks = 0
| FS4 Blocks = 0
| Full Blocks = 73562
| Unformatted Blocks = 9160
|
| The block numbers before and after deletion are exactly the same. Either
| DBMS_SPACE is buggy or space management within LOB segment is buggy.
| Judging by the huge disk space consumption, the latter is the case.

There is another interpretation - Oracle doesn't use the bitmap space
managed blocks for LOB segments in ASSM, beyond marking them as FULL when
they are first used. The LOBINDEX is (used to be) a two-part index, with
one part showing the lob chunks which were available for re-use, keyed by
the SCN at which the chunks were marked as deleted. When the chunk is
re-used the index entry is deleted.

I think you need to test something like:
set your auto_undo_retention to a very low value (say 30 seconds)
load your lobs into the table
delete a large number of rows
wait for a while (at least the undo_retention_time) - do a busy bit of
work on some other table in the database
insert the deleted lobs and see what happens to the space


There's a big difference between freelist management and ASSM when you look
at the lob segment headers, by the way, so you may also see some variation
in how things work if you switch.

Mladen Gogala

unread,
May 13, 2012, 12:37:14 PM5/13/12
to
On Sun, 13 May 2012 07:33:35 +0100, Jonathan Lewis wrote:

> On the other hand you can specify RETENTION NONE with securefiles and
> not use any of your space for retaining older versions.

Yes, that's the replacement for PCTVERSION.



--
http://mgogala.byethost5.com
0 new messages