Path to making MariaDB optional

Dmitry Tantsur

unread,

Dec 8, 2021, 8:43:34 AM12/8/21

to Metal3 Development List

Hi folks,

For those who missed the early days of Metal3: we use mariadb in ironic because our early experiments show that using sqlite does not play well with the API-conductor separation. Indeed, although sqlite has WAL support, its concurrency is still based on exclusive locking. Inspector, being a monolithic service, does not suffer from this issue and still uses sqlite.

I'm now adding a new script [1] to Ironic that launches API and conductor in the same process with or without [2] the JSON RPC bus. This change paves a way towards disabling both mariadb and JSON RPC in Metal3 by default.

I have prepared a proof-of-concept PR [3] (that requires [1] and [2]) that (re)adds a /bin/runironic launcher, this time starting a combined API+conductor process. My thinking is to use Nginx to terminate TLS, just as we do for Inspector.

I would also like to split the mariadb container away from ironic-image. Or should we just drop support for it completely in the near future?

Thoughts, ideas, objections?

Dmitry

[1] https://review.opendev.org/c/openstack/ironic/+/819620

[2] https://review.opendev.org/c/openstack/ironic/+/820036

[3] https://github.com/metal3-io/ironic-image/pull/330

--

Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Kashif Khan

unread,

Dec 9, 2021, 3:37:59 AM12/9/21

to Dmitry Tantsur, Metal3 Development List

Hi Dmitry,

Thanks for the initiative.

Regarding the removal of mariadb and replacing it with Sqlite, how would it impact the performance of ironic?

Would it introduce any limitations regarding deployment on scale? I meant on the number of BMHs it can handle?

I think we can expect lower footprint in consumption of resources also for Sqlite?

Just thinking about, if it would bring any added limitations in addition to the advantages or not.

BR

Kashif

From: metal...@googlegroups.com <metal...@googlegroups.com> on behalf of Dmitry Tantsur <dtan...@redhat.com>
Sent: Wednesday, December 8, 2021 3:43 PM
To: Metal3 Development List <metal...@googlegroups.com>
Subject: [metal3-dev] Path to making MariaDB optional

--
You received this message because you are subscribed to the Google Groups "Metal3 Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metal3-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metal3-dev/CACNgkFyh%2B_XU_N6c_n-RxhT%3DAGHMgazOeMZc_%2Bui0sdH3s%2B9Gg%40mail.gmail.com.

Dmitry Tantsur

unread,

Dec 9, 2021, 7:56:13 AM12/9/21

to Kashif Khan, Metal3 Development List

Hi,

On Thu, Dec 9, 2021 at 9:38 AM Kashif Khan <kashi...@est.tech> wrote:

Hi Dmitry,

Thanks for the initiative.

Regarding the removal of mariadb and replacing it with Sqlite, how would it impact the performance of ironic?

I've got several people asking me this question. A precise answer is not possible without measurements, and I don't think we've anyhow measured the performance with MariaDB as well. There are several considerations at play:

1) MariaDB plays better with concurrent access, although WAL mode in SQLite should mostly make up for it.

2) MariaDB is designed for client/server interaction, single client usage may see a large overhead for networking.

3) There is an overhead of a separate MariaDB service, especially on a busy machine.

4) SQLite will likely keep the whole database mapped into the Ironic's memory.

And probably more.

Would it introduce any limitations regarding deployment on scale? I meant on the number of BMHs it can handle?

As a small experiment, I started an Ironic (combined, no RPC) instance with an SQLite file (in old, non-WAL mode - forgot about it) and the "fake" driver (all driver operations are stubbed). I used a Python script with a thread pool of 50 threads to enroll 10000 fake nodes (forgot about ports), modify them a bit and "deploy" (fake, so only state machine operations).

It took me roughly an hour, so nearly 3 nodes per second (could be affected by the fact I was logging to the console - my bad). No errors happened during that, Ironic still operates happily and within 200M RSS. The database file is just 9.4 MiB (would be somewhat larger for a real node, but still fit into memory). The API and my poor laptop were responsive during the process (I even had a video call at the same time!).

There is always room for improvement. As we scale up, it may make sense to make the power synchronization loop less frequent (it's a common recommendation for high-scale Ironic).

I think we can expect lower footprint in consumption of resources also for Sqlite?

Definitely. Not having another service is a large change.

Dmitry

Dmitry Tantsur

unread,

Dec 9, 2021, 8:52:04 AM12/9/21

to Metal3 Development List

On Thu, Dec 9, 2021 at 1:55 PM Dmitry Tantsur <dtan...@redhat.com> wrote:

Hi,

On Thu, Dec 9, 2021 at 9:38 AM Kashif Khan <kashi...@est.tech> wrote:

Hi Dmitry,

Thanks for the initiative.

Regarding the removal of mariadb and replacing it with Sqlite, how would it impact the performance of ironic?

I've got several people asking me this question. A precise answer is not possible without measurements, and I don't think we've anyhow measured the performance with MariaDB as well. There are several considerations at play:
1) MariaDB plays better with concurrent access, although WAL mode in SQLite should mostly make up for it.
2) MariaDB is designed for client/server interaction, single client usage may see a large overhead for networking.
3) There is an overhead of a separate MariaDB service, especially on a busy machine.
4) SQLite will likely keep the whole database mapped into the Ironic's memory.

And probably more.

Would it introduce any limitations regarding deployment on scale? I meant on the number of BMHs it can handle?

As a small experiment, I started an Ironic (combined, no RPC) instance with an SQLite file (in old, non-WAL mode - forgot about it) and the "fake" driver (all driver operations are stubbed). I used a Python script with a thread pool of 50 threads to enroll 10000 fake nodes (forgot about ports), modify them a bit and "deploy" (fake, so only state machine operations).

It took me roughly an hour, so nearly 3 nodes per second (could be affected by the fact I was logging to the console - my bad). No errors happened during that, Ironic still operates happily and within 200M RSS. The database file is just 9.4 MiB (would be somewhat larger for a real node, but still fit into memory). The API and my poor laptop were responsive during the process (I even had a video call at the same time!).

Got a bit more realistic experiment now. WAL enabled, logging into /tmp, creating a port with random MAC, fixed a small issue in handling fake nodes in Ironic.

10000 nodes created and tested in 26 minutes, more than 6 nodes a second. The first 100 nodes were created in 13 seconds, 1000 - in 159 seconds. The process was slowing down a bit, presumably because of the power sync loop. RSS was around 160 MiB. The resulting database is 12 MiB this time, still negligible.

Kashif Khan

unread,

Dec 9, 2021, 8:55:58 AM12/9/21

to Dmitry Tantsur, Metal3 Development List

Thanks a lot Dmitry. The number looks promising.

I would also really like to know the method of your scalability experiment. Can you may be post a doc or screen capture somewhere on your available time?

BR

Kashif

From: metal...@googlegroups.com <metal...@googlegroups.com> on behalf of Dmitry Tantsur <dtan...@redhat.com>

Sent: Thursday, December 9, 2021 3:51 PM

To: Metal3 Development List <metal...@googlegroups.com>

Subject: Re: [metal3-dev] Path to making MariaDB optional

To view this discussion on the web visit https://groups.google.com/d/msgid/metal3-dev/CACNgkFxtR6KQ%3DL2VC1e9GuSCjgAKeiWFRBpRoeU%3DcQVZ0qBaGw%40mail.gmail.com.

Nam Xuan Nguyen

unread,

Dec 9, 2021, 9:08:22 AM12/9/21

to Metal3 Development List

Hi Dmitry,

What is the benefit of running ironic in combined node compared to separation mode? And why didn't we run this mode at the beginning?

Also, if I understand correctly from your proposal, we should change to Sqlite only in combined mode, but keep using MariaDB in separation mode, i that true?

Best regards,

Nam

From: metal...@googlegroups.com <metal...@googlegroups.com> on behalf of Kashif Khan <kashi...@est.tech>
Sent: Thursday, December 9, 2021 3:55 PM
To: Dmitry Tantsur <dtan...@redhat.com>; Metal3 Development List <metal...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/metal3-dev/HE1P189MB0395CE8FB03A0D9151FE9CE390709%40HE1P189MB0395.EURP189.PROD.OUTLOOK.COM.

Zane Bitter

unread,

Dec 9, 2021, 9:39:24 AM12/9/21

to metal...@googlegroups.com

On 8/12/21 08:43, Dmitry Tantsur wrote:
> Hi folks,
>
> For those who missed the early days of Metal3: we use mariadb in ironic
> because our early experiments show that using sqlite does not play well
> with the API-conductor separation. Indeed, although sqlite has WAL
> support, its concurrency is still based on exclusive locking. Inspector,
> being a monolithic service, does not suffer from this issue and still
> uses sqlite.
>
> I'm now adding a new script [1] to Ironic that launches API and
> conductor in the same process with or without [2] the JSON RPC bus. This
> change paves a way towards disabling both mariadb and JSON RPC in Metal3
> by default.
>
> I have prepared a proof-of-concept PR [3] (that requires [1] and [2])
> that (re)adds a /bin/runironic launcher, this time starting a combined
> API+conductor process. My thinking is to use Nginx to terminate TLS,
> just as we do for Inspector.
>
> I would also like to split the mariadb container away from ironic-image.
> Or should we just drop support for it completely in the near future?

I guess my only concern here would be that if we wanted deployments to
be able to scale out to multiple conductors in future, that would
presumably require something similar to our current topology.

Other than that, I am all in favour of SQLite. It's a much better fit
for all current use cases.

> Thoughts, ideas, objections?
>
> Dmitry
>
> [1] https://review.opendev.org/c/openstack/ironic/+/819620
> <https://review.opendev.org/c/openstack/ironic/+/819620>
> [2] https://review.opendev.org/c/openstack/ironic/+/820036
> <https://review.opendev.org/c/openstack/ironic/+/820036>
> [3] https://github.com/metal3-io/ironic-image/pull/330

> <https://github.com/metal3-io/ironic-image/pull/330>

Dmitry Tantsur

unread,

Dec 9, 2021, 11:10:12 AM12/9/21

to Kashif Khan, Metal3 Development List

On Thu, Dec 9, 2021 at 2:56 PM Kashif Khan <kashi...@est.tech> wrote:

Thanks a lot Dmitry. The number looks promising.

I would also really like to know the method of your scalability experiment. Can you may be post a doc or screen capture somewhere on your available time?

I did a pretty simple thing actually.

1) Create a fake configuration file, like this:

[DEFAULT]
auth_strategy = noauth
debug = True
rpc_transport = none

enabled_hardware_types = fake-hardware
enabled_boot_interfaces = fake
enabled_deploy_interfaces = fake
enabled_management_interfaces = fake
enabled_power_interfaces = fake

[database]

# This reduces the time further from 26 to 20 minutes btw.

sqlite_synchronous = False

3) Checkout Ironic from git and cd to it.

4) Initialize the database:

sqlite3 ironic/ironic.sqlite "pragma journal_mode=wal"

tox -evenv -- ironic-dbsync create_schema

5) Start Ironic with the provided file:

tox -evenv -- ironic --config-file /path/to/fake/ironic.conf --log-file /tmp/ironic.log

6) Use https://paste.opendev.org/show/811581/ for testing.

Dmitry Tantsur

unread,

Dec 9, 2021, 11:12:01 AM12/9/21

to Nam Xuan Nguyen, Metal3 Development List

Hi,

On Thu, Dec 9, 2021 at 3:08 PM Nam Xuan Nguyen <nam.xua...@est.tech> wrote:

Hi Dmitry,

What is the benefit of running ironic in combined node compared to separation mode? And why didn't we run this mode at the beginning?

The combined mode only appeared this week, this is why. The benefit is that we only need one service, while in separation mode we launch ironic-conductor, then several copies of ironic-api (because mod_wsgi does not use eventlet, so we have to use real processes and threads).

Also, if I understand correctly from your proposal, we should change to Sqlite only in combined mode, but keep using MariaDB in separation mode, i that true?

Yes. SQLite is not really designed to be used from several processes, hence it only fits into the combined model well (another benefit of it).

If we decide to keep the separation mode available, I'd prefer to separate mysql into a new container image to avoid installing it for everyone.

Dmitry

To view this discussion on the web visit https://groups.google.com/d/msgid/metal3-dev/HE1P189MB05877ECAC752D5F6B4F2950FC2709%40HE1P189MB0587.EURP189.PROD.OUTLOOK.COM.

Dmitry Tantsur

unread,

Dec 9, 2021, 11:13:53 AM12/9/21

to Zane Bitter, Metal3 Development List

On Thu, Dec 9, 2021 at 3:39 PM Zane Bitter <zbi...@redhat.com> wrote:

On 8/12/21 08:43, Dmitry Tantsur wrote:
> Hi folks,
>
> For those who missed the early days of Metal3: we use mariadb in ironic
> because our early experiments show that using sqlite does not play well
> with the API-conductor separation. Indeed, although sqlite has WAL
> support, its concurrency is still based on exclusive locking. Inspector,
> being a monolithic service, does not suffer from this issue and still
> uses sqlite.
>
> I'm now adding a new script [1] to Ironic that launches API and
> conductor in the same process with or without [2] the JSON RPC bus. This
> change paves a way towards disabling both mariadb and JSON RPC in Metal3
> by default.
>
> I have prepared a proof-of-concept PR [3] (that requires [1] and [2])
> that (re)adds a /bin/runironic launcher, this time starting a combined
> API+conductor process. My thinking is to use Nginx to terminate TLS,
> just as we do for Inspector.
>
> I would also like to split the mariadb container away from ironic-image.
> Or should we just drop support for it completely in the near future?

I guess my only concern here would be that if we wanted deployments to
be able to scale out to multiple conductors in future, that would
presumably require something similar to our current topology.

Yes and no :)

MariaDB is a more natural choice for multiple conductors, I concur. But we'll need to run it in active/active (aka galera) configuration, which is a serious operational challenge.

What I'm pondering instead is an optional mode in Ironic where each conductor (or rather: each combined instance) will fully contain the information about the nodes it manages. That would mean that each Ironic will have its own SQLite database with its nodes. I haven't figured out how exactly it would work technically, but it's on my roadmap.

Dmitry

Other than that, I am all in favour of SQLite. It's a much better fit
for all current use cases.

> Thoughts, ideas, objections?
>
> Dmitry
>
> [1] https://review.opendev.org/c/openstack/ironic/+/819620
> <https://review.opendev.org/c/openstack/ironic/+/819620>
> [2] https://review.opendev.org/c/openstack/ironic/+/820036
> <https://review.opendev.org/c/openstack/ironic/+/820036>
> [3] https://github.com/metal3-io/ironic-image/pull/330
> <https://github.com/metal3-io/ironic-image/pull/330>

--
You received this message because you are subscribed to the Google Groups "Metal3 Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metal3-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/metal3-dev/35de2718-48da-bad3-ea3d-858a6737ee18%40redhat.com.

Zane Bitter

unread,

Dec 9, 2021, 12:04:31 PM12/9/21

to Dmitry Tantsur, Metal3 Development List

Wait, what? We wouldn't need a separate MariaDB for each conductor, they
could all connect to a single one.

> What I'm pondering instead is an optional mode in Ironic where each
> conductor (or rather: each combined instance) will fully contain the
> information about the nodes it manages. That would mean that each Ironic
> will have its own SQLite database with its nodes. I haven't figured out
> how exactly it would work technically, but it's on my roadmap.

That sounds cool :)

> Dmitry
>
>
> Other than that, I am all in favour of SQLite. It's a much better fit
> for all current use cases.
>
> > Thoughts, ideas, objections?
> >
> > Dmitry
> >
> > [1] https://review.opendev.org/c/openstack/ironic/+/819620
> <https://review.opendev.org/c/openstack/ironic/+/819620>
> > <https://review.opendev.org/c/openstack/ironic/+/819620
> <https://review.opendev.org/c/openstack/ironic/+/819620>>
> > [2] https://review.opendev.org/c/openstack/ironic/+/820036
> <https://review.opendev.org/c/openstack/ironic/+/820036>
> > <https://review.opendev.org/c/openstack/ironic/+/820036
> <https://review.opendev.org/c/openstack/ironic/+/820036>>
> > [3] https://github.com/metal3-io/ironic-image/pull/330
> <https://github.com/metal3-io/ironic-image/pull/330>
> > <https://github.com/metal3-io/ironic-image/pull/330
> <https://github.com/metal3-io/ironic-image/pull/330>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Metal3 Development List" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to metal3-dev+...@googlegroups.com

> <mailto:metal3-dev%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/metal3-dev/35de2718-48da-bad3-ea3d-858a6737ee18%40redhat.com

> <https://groups.google.com/d/msgid/metal3-dev/35de2718-48da-bad3-ea3d-858a6737ee18%40redhat.com>.
>
>
>
> --
> Red Hat GmbH, https://de.redhat.com/ <https://de.redhat.com/> ,

Dmitry Tantsur

unread,

Dec 10, 2021, 8:01:49 AM12/10/21

to Zane Bitter, Metal3 Development List

Then you'd have a single point of failure. That is, if the master with MariaDB goes down, the whole Ironic goes down.

Dmitry

--

Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,

Dmitry Tantsur

unread,

Dec 10, 2021, 8:03:29 AM12/10/21

to Metal3 Development List

Update:

1) We have https://github.com/metal3-io/mariadb-image. If you fix any MariaDB stuff in ironic-image, please also update that repo. I'll work on switching over to it, but that is likely to happen next year.

2) https://github.com/metal3-io/ironic-image/pull/330 works in my testing and is ready for review. Note that the RDO packages don't yet have the new executable, you need to use the source build if you want to try it.

Dmitry

Zane Bitter

unread,

Dec 10, 2021, 9:04:58 AM12/10/21

to Dmitry Tantsur, Metal3 Development List

On 10/12/21 08:01, Dmitry Tantsur wrote:
>
>
> On Thu, Dec 9, 2021 at 6:04 PM Zane Bitter <zbi...@redhat.com
> <mailto:zbi...@redhat.com>> wrote:
>
> On 9/12/21 11:13, Dmitry Tantsur wrote:
> >
> >
> > On Thu, Dec 9, 2021 at 3:39 PM Zane Bitter <zbi...@redhat.com
> <mailto:zbi...@redhat.com>

We have that already. It'll come back. (The BMO is also a single point
of failure in this sense.) The goal of scaling out here would be to
distribute the load, not for resiliency.

I guess one difference is that right now if the DB is restarted then the
conductor container is restarted along with it. Having the DB disappear
while the conductors are still running could get a bit interesting. That
said, if they just crash then that would give equivalent behaviour to now.

> > <mailto:metal3-dev%2Bunsu...@googlegroups.com
> <mailto:metal3-dev%252Buns...@googlegroups.com>>.

Mr. Mykola Dzham

unread,

Dec 10, 2021, 1:23:05 PM12/10/21

to Dmitry Tantsur, Nam Xuan Nguyen, Metal3 Development List

On Thu, 9 Dec 2021 at 17:12, Dmitry Tantsur <dtan...@redhat.com> wrote:
>
> Hi,
>

> On Thu, Dec 9, 2021 at 3:08 PM Nam Xuan Nguyen <nam.xua...@est.tech> wrote:
>>
>> Hi Dmitry,
>> What is the benefit of running ironic in combined node compared to separation mode? And why didn't we run this mode at the beginning?
>
>
> The combined mode only appeared this week, this is why. The benefit is that we only need one service, while in separation mode we launch ironic-conductor, then several copies of ironic-api (because mod_wsgi does not use eventlet, so we have to use real processes and threads).
>
>>
>> Also, if I understand correctly from your proposal, we should change to Sqlite only in combined mode, but keep using MariaDB in separation mode, i that true?
>
>
> Yes. SQLite is not really designed to be used from several processes, hence it only fits into the combined model well (another benefit of it).

Even though SQLite might be not so great to be used from several
processes, it still works. And you can share the same SQLite database
between several containers within a single pod - just put it into a
volume that will be mounted to all these containers.

> To view this discussion on the web visit https://groups.google.com/d/msgid/metal3-dev/CACNgkFyGzxxs1bQC3tSHZt55WY6KPLL9%3D-uyCb-x0ZK%3DtiwfyQ%40mail.gmail.com.

--
Mr. Mykola Dzham
+41 79 571 28 64

Reply all

Reply to author

Forward