Luminous Arc 3 English 82
Sourabh Doherty
Ceph now packages python bindings for python3.6 instead ofpython3.4, because EPEL7 recently switched from python3.4 topython3.6 as the native python3. see the announcement _for more details on the background of this change.
A health warning is now generated if the average osd heartbeat pingtime exceeds a configurable threshold for any of the intervalscomputed. The OSD computes 1 minute, 5 minute and 15 minuteintervals with average, minimum and maximum values. New configurationoption mon_warn_on_slow_ping_ratio specifies a percentage ofosd_heartbeat_grace to determine the threshold. A value of zerodisables the warning. New configuration optionmon_warn_on_slow_ping_time specified in milliseconds over-rides thecomputed value, causes a warningwhen OSD heartbeat pings take longer than the specified amount.New admin command ceph daemon mgr.# dump_osd_network [threshold] command willlist all connections with a ping time longer than the specified threshold orvalue determined by the config options, for the average for any of the 3 intervals.New admin command ceph daemon osd.# dump_osd_network [threshold] willdo the same but only including heartbeats initiated by the specified OSD.
The configuration value osd_calc_pg_upmaps_max_stddev used for upmapbalancing has been removed. Instead use the mgr balancer configupmap_max_deviation which now is an integer number of PGs of deviationfrom the target PGs per OSD. This can be set with a command likeceph config set mgr mgr/balancer/upmap_max_deviation 2. The defaultupmap_max_deviation is 1. There are situations where crush ruleswould not allow a pool to ever have completely balanced PGs. For example, ifcrush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 ofthe racks. In those cases, the configuration value can be increased.
In 12.2.11 and earlier releases, keyring caps were not checked for validity,so the caps string could be anything. As of 12.2.12, caps strings arevalidated and providing a keyring with an invalid caps string to, e.g.,ceph auth add will result in an error.
This release fixes the pg log hard limit bug that was introduced in12.2.9, A flag calledpglog_hardlimit has been introduced, which is off by default. Enablingthis flag will limit the length of the pg log. In order to enablethat, the flag must be set by running ceph osd set pglog_hardlimitafter completely upgrading to 12.2.11. Once the cluster has this flagset, the length of the pg log will be capped by a hard limit. Once set,this flag must not be unset anymore.
There have been fixes to RGW dynamic and manual resharding, which no longerleaves behind stale bucket instances to be removed manually. For finding andcleaning up older instances from a reshard a radosgw-admin command reshardstale-instances list and reshard stale-instances rm should do the necessarycleanup.
cephfs-journal-tool makes rank argument (--rank) mandatory. Rank isof format filesystem:rank, where filesystem is the CephFS filesystemand rank is the MDS rank on which the operation is to be executed. Tooperate on all ranks, use all or * as the rank specifier. Note that,operations that dump journal information to file will now dump to per-ranksuffixed dump files. Importing journal information from dump files isdisallowed if operation is targetted for all ranks.
This is the tenth bug fix release of the Luminous v12.2.x long term stablerelease series. The previous release, v12.2.9, introduced the PG hard-limitpatches which were found to cause an issue in certain upgrade scenarios, andthis release was expedited to revert those patches. If you already successfullyupgraded to v12.2.9, you should not upgrade to v12.2.10, but rather waitfor a release in which is addressed. Allother users are encouraged to upgrade to this release.
This is the ninth bug fix release of the Luminous v12.2.x long term stablerelease series. Although this release contains several bugfixes across all thecomponents, it also introduced the PG hard-limit patches which could causeproblems during upgrade when not all PGs were active+clean. Therefore, usersshould not install this release. Instead, they should skip it and upgrade to12.2.10 directly.
The bluestore_cache_* options are no longer needed. They are replacedby osd_memory_target, defaulting to 4GB. BlueStore will expandand contract its cache to attempt to stay within thislimit. Users upgrading should note this is a higher defaultthan the previous bluestore_cache_size of 1GB, so OSDs usingBlueStore will use more memory by default.
For the cluster that installed the broken 12.2.6 release, 12.2.7 fixed theregression and introduced a workaround option osd distrust data digest = true,but 12.2.7 clusters still generated health warnings like
12.2.8 improves the deep scrub code to automatically repair theseinconsistencies. Once the entire cluster has been upgraded and then fully deepscrubbed, and all such inconsistencies are resolved, it will be safe to disablethe osd distrust data digest = true workaround option.
Options rgw curl low speed limit and rgw curl low speed timeare added to control the lower speed limits and times below whichthe requests are considered too slow to be aborted and can helpmitigate data sync getting blocked during network issues
This is the seventh bugfix release of Luminous v12.2.x long termstable release series. This release contains several fixes forregressions in the v12.2.6 and v12.2.5 releases. We recommend thatall users upgrade.
v12.2.6 included an incomplete backport of an optimization forBlueStore OSDs that avoids maintaining both the per-object checksumand the internal BlueStore checksum. Due to the accidental omissionof a critical follow-on patch, v12.2.6 corrupts (fails to update) thestored per-object checksum value for some objects. This can result inan EIO error when trying to read those objects.
If your cluster includes BlueStore OSDs and was affected, deep scrubswill generate errors about mismatched CRCs for affected objects.Currently the repair operation does not know how to correct them(since all replicas do not match the expected checksum it does notknow how to proceed). These warnings are harmless in the sense thatIO is not affected and the replicas are all still in sync. The numberof affected objects is likely to drop (possibly to zero) on their ownover time as those objects are modified. We expect to include a scrubimprovement in v12.2.8 to clean up any remaining objects.
Notably, upgrading a cluster involves OSD restarts and as such mayincrease the risk of encountering this bug. For this reason, forclusters with erasure coded pools, we recommend the following upgradeprocedure to minimize risk:
This will cause an availability outage for the duration of the OSDrestarts. If this in unacceptable, an more risky alternative is todisable RGW garbage collection (the primary known cause of these radosoperations) for the duration of the upgrade:
This is the fifth bugfix release of Luminous v12.2.x long term stablerelease series. This release contains a range of bug fixes across allcomponents of Ceph. We recommend all the users of 12.2.x series toupdate.
Upgrading an MDS cluster to 12.2.3+ will result in all active MDSexiting due to feature incompatibilities once an upgraded MDS comesonline (even as standby). Operators may ignore the error messagesand continue upgrading/restarting or follow this upgrade sequence:
Reduce the number of ranks to 1 (ceph fs set max_mds 1),wait for all other MDS to deactivate, leaving the one active MDS,upgrade the single active MDS, then upgrade/start standbys. Finally,restore the previous max_mds.
This is the fourth bugfix release of Luminous v12.2.x long term stable releaseseries. This was primarily intended to fix a few build, ceph-volume/ceph-diskand RGW issues. We recommend all the users of 12.2.x series to update.
This is the third bugfix release of Luminous v12.2.x long term stable releaseseries. It contains a range of bug fixes and a few features across Bluestore,CephFS, RBD & RGW. We recommend all the users of 12.2.x series update.
This is the second bugfix release of Luminous v12.2.x long term stable releaseseries. It contains a range of bug fixes and a few features across Bluestore,CephFS, RBD & RGW. We recommend all the users of 12.2.x series update.
This is the first bugfix release of Luminous v12.2.x long term stablerelease series. It contains a range of bug fixes and a few featuresacross CephFS, RBD & RGW. We recommend all the users of 12.2.x seriesupdate.
Limiting MDS cache via a memory limit is now supported using the newmds_cache_memory_limit config option (1GB by default). A cache reservationcan also be specified using mds_cache_reservation as a percentage of thelimit (5% by default). Limits by inode count are still supported usingmds_cache_size. Setting mds_cache_size to 0 (the default) disables theinode limit.
The maximum number of PGs per OSD before the monitor issues awarning has been reduced from 300 to 200 PGs. 200 is still twicethe generally recommended target of 100 PGs per OSD. This limit canbe adjusted via the mon_max_pg_per_osd option on themonitors. The older mon_pg_warn_max_per_osd option has been removed.
Creating pools or adjusting pg_num will now fail if the change wouldmake the number of PGs per OSD exceed the configuredmon_max_pg_per_osd limit. The option can be adjusted if itis really necessary to create a pool with more PGs.
There was a bug in the PG mapping behavior of the new upmapfeature. If you made use of this feature (e.g., via the ceph osdpg-upmap-items command), we recommend that all mappings be removed (viathe ceph osd rm-pg-upmap-items command) before upgrading to thispoint release.
This is the first release of Luminous v12.2.x long term stable releaseseries. There have been major changes since Kraken (v11.2.z) andJewel (v10.2.z), and the upgrade process is non-trivial. Please readthese release notes carefully.
The new BlueStore backend for ceph-osd is now stable and thenew default for newly created OSDs. BlueStore manages datastored by each OSD by directly managing the physical HDDs orSSDs without the use of an intervening file system like XFS.This provides greater performance and features. SeeStorage Devices andBlueStore Configuration Reference.
6022c96aab