Can't SELECT some records; Invalid range for HWM

[Reynaldo Dalisay]

We are running MariaDB Community Server:

- Server version 10.6.5-MariaDB-1:10.6.5+maria~focal mariadb.org 

- ColumnStore version 6.2.1

This is a single-node installation running on physical hardware (not a VM).

During the past couple of days, we started encountering the following errors when we try to execute a SELECT statement:

Example 1

select * from cvm_acct_csd_cs  where report_date >= '2023-06-27' limit 10;

ERROR 1815 (HY000): Internal error: MCS-2031: Blocks are missing.  Alter or drop table in progress?

Example 2

select * from cvm_acct_csd_cs  where report_date >= '2023-06-27';

ERROR 1815 (HY000): Internal error: An unexpected condition within the query caused an internal processing error within Columnstore. Please check the log files for more details. Additional Information: error in PrimitiveServer l

In addition, during this time, we've also been seeing these in the crit.log file:

$ sudo tail /var/log/mariadb/columnstore/crit.log
Jul 11 09:45:28 ods PrimProc[1621]: 28.543407 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 177011923, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 6655, lowfbo 6144, blocksReadAhead 512, range size -6037#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.586042 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 177011923, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 6655, lowfbo 6144, blocksReadAhead 512, range size -6037#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.594649 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 177011923, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 6655, lowfbo 6144, blocksReadAhead 512, range size -6037#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.675966 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 177015811, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 2559, lowfbo 2048, blocksReadAhead 512, range size -1941#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.681863 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 177015075, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 1535, lowfbo 1024, blocksReadAhead 512, range size -917#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.698983 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 176995909, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 3071, lowfbo 2560, blocksReadAhead 512, range size -2453#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.707531 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 176996707, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 3583, lowfbo 3072, blocksReadAhead 512, range size -2965#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.707861 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 177014351, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 1023, lowfbo 512, blocksReadAhead 512, range size -405#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.727965 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 177015811, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 2559, lowfbo 2048, blocksReadAhead 512, range size -1941#012        
Jul 11 09:45:28 ods PrimProc[1621]: 28.887976 |0|0|0| C 28 CAL0000: Invalid Range from HWM for lbid 176970752, range size should be <= blocksReadAhead: HWM 106, dbroot 1, highfbo 2559, lowfbo 2048, blocksReadAhead 512, range size -1941#012      

I've tried googling this error, but so far I've only seen a mention of disk space and trying to restart the server.  Our server has plenty of disk space, and we've already restarted the server twice.  Yet the issue continues.

What is happening, and how can we resolve this issue?

Appreciate any help you can provide.  Thank you.

[Allen Herrera]

sounds like a corrupted extent map after failed rollback or transaction
/var/lib/columnstore/data1/systemfiles/dbrm/

check your debug log and messages files on startup for more clues.

I hope someone in the community here can help you further, else I recommend opening a support ticket for further assistance if you have enterprise support.

--
You received this message because you are subscribed to the Google Groups "MariaDB ColumnStore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mariadb-columns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mariadb-columnstore/ed0f2f65-f8f3-45ea-b546-91214accfcdfn%40googlegroups.com.

--

Allen Herrera

Customer Engineer | MariaDB Corporation

+1.360.888.3938 | calendly.com/allen-herrera | allen.herrera@mariadb.com

[drrtuy]

Hi!

I agree with Allen. Looks like a failed rollback. There is a known issue with 6.2.1 that might corrupt metadata. You should either manually break systemd units dependencies, namely mcs-writeengine must not depend on mcs-exemgr or consider upgrade to more recent versions.

Regards,

Roman

вторник, 11 июля 2023 г. в 03:24:01 UTC+3, Reynaldo Dalisay: