[ANN] LIO userspace improvements

[Jerome Martin]

Hello Everyone,

For those of you who do not know me, I am Jerome Martin, the original
author of
targetcli, configshell and rtslib, the userspace tools currently used to
configure the LIO kernel SCSI target.

As some may have noticed, Datera recently opened up the licensing even
more for those tools, transitioning to an Apache v2 license. The project is
now available on GitHub (github.com/Datera), and this location will be the
primary repository for the development of the LIO userspace tools.

Today, I want to announce a new development track for those tools, and
open the
discussion on this mailling-list as to which direction this development
should
take.

You will find on the GitHub repository for rtslib a new branch, called
lio-config. This branch contains a prototype augmented rtslib API, as
well as a
proof-of-concept CLI tool (simply called lio) using it.

This new API uses a configuration-centric approach on top of legqacy rtslib.
When using the API or the lio CLI, you simply edit a configuration.
This is a drastic change from the targetcli approach.
Where targetcli is imperative, lio CLI is more declarative.

Instead of directly impacting the running system, you can carefully craft a
candidate configuration for it, by using CLI commands, but also by loading
and merging saved configuration snippets into the candidate configuration.

Via the API and CLI, you can issue simple statements, identical in form
to the
syntax of the configuration files themselves (they use the same parser),
that
are matched against the loaded configuration. For advanced search or set
operations, full regex support is available when writing configuration
statements, i.e. you can search for "fabric iscsi target .* tpgt [1-3]".
This gives a lot of expressive power that could be leveraged for a lot
of cool features.

Regarding day-to-day usage of the tools, the fundamental operations that
were missing in targetcli are there:

- Unlimited undo support, for all load/merge/set/clear/delete operations,
within a CLI session (and potentially accross sessions, to be
implemented),
along with full tagging of operations history within a session (you
can tell
where a specific object or value configuration comes from, when, etc.)

- A commit/rollback mechanism, simple to use and augmented with the
possibility
of listing the configuration changes, both against the running
system, but
also (via the API, soon in the CLI) against previously saved
configurations

- The system uses a policy engine that parses and apply policy rules.
Those use the same configuration format as configuration statements,
but describe instead expected value types, default values, available
attributes, the structure of the configuration, etc. This could easily be
extended to user-defined policy rules files, for simplifying recurring
administration tasks, like setting particular default values for
configuration
objects.

Enough rambling on supposed merits of that piece of code, I'll let you
all see
for yourself and give it a try. The implementation is not complete by any
means, it is intended as a Proof-of-concept for now. But this is a great
opportunity to discuss what features should be part of the future
development.

I will chime in later in March with a list of features that could be
implemented, so we could all discuss them and decide what is the most
interesting roadmap. For now, you can look at the # TODO tags in my code ;-)

Meanwhile, please write your comments, suggestions and any sort of
feedback on
the list. There are also packaging questions to discuss, as for now this is
implemented in the rtslib tree, and that might be suboptimal.
Package maintainers are welcome to send me their preferences.

Best,
--
Jerome Martin

[Jerome Martin]

Hi,

On 02/27/2014 03:14 PM, Sergej Roytman wrote:
> I have been finding the scripting mode that is shown here‹
>
> http://linux-iscsi.org/wiki/ISCSI#Scripting_with_RTSlib
>
> very useful. More convenient, for some things, than having to do a little
> configuration dance inside targetcli. Do the changes you describe extend
> that, or are they something different?
This is not a "scripting mode". This is just python code using the the
python rtslib API.
This does not change and is still available.
Both targetcli and the lio CLI are applications built on top of it, as
is the new config API.
Using that new config API from python, instead of:
> from rtslib import *
> backstore = IBlockBackstore(3, mode='create')
> so = IBlockStorageObject(backstore, "sdb", "/dev/sdb", gen_wwn=True)
> fabric = FabricModule('iscsi')
> target = Target(fabric,
> "iqn.2003-01.org.linux-iscsi.x.x8664:sn.d3d8b0500fde")
> tpg = TPG(target, 1)
> portal = NetworkPortal(tpg, "192.168.1.128", "5060")
> lun0 = tpg.lun(0, so, "my_lun")
> node_acl =
> tpg.node_acl("iqn.2003-01.org.linux-iscsi.y.x8664:sn.abcdefghijkl")
> mapped_lun = node_acl.mapped_lun(0, 0, False)
You could write:
> from rtslib.config import Config
> conf = Config()
> conf.set("storage iblock disk sdb path /dev/sdb")
> tpg = "fabric iscsi target
> iqn.2003-01.org.linux-iscsi.x.x8664:sn.d3d8b0500fde tpgt 1"
> conf.set(tpg + "portal 192.168.1.128:5060")
> conf.set(tpg + "lun 0 backend iblock:sdb")
> conf.set(tpg + "acl
> iqn.2003-01.org.linux-iscsi.y.x8664:sn.abcdefghijkl mapped_lun 0
> target_lun 0")
But that would be overkill, as you could just create a "snippet.lio"
config file like this:
> storage iblock disk sdb path /dev/sdb
> fabric iscsi target
> iqn.2003-01.org.linux-iscsi.x.x8664:sn.d3d8b0500fde tpgt 1 {
> portal 192.168.1.128:5060
> lun 0 backend iblock:sdb
> acl iqn.2003-01.org.linux-iscsi.y.x8664:sn.abcdefghijkl mapped_lun
> 0 target_lun 0
> }
And simply load it from the lio cli using "load snippet.lio" in config
mode, or "merge snippet.lio" if you want to merge that to the current
configuration. Then you just need to issue 'commit' when your
configuration is ready to be applied on the system and saved as the next
startup config.

But the lio cli is more practical for that kind of small snippets: you
enter the same amount of text, exactly the same statements, but get
autocompletion to help you type it :-) Also it will check the values
type as soon as you enter them, provide you with unlimited undo levels,
rollback to the last commited configuration, etc.

Best,
--
Jerome

[Andy Grover]

On 02/26/2014 06:35 PM, Jerome Martin wrote:
> Meanwhile, please write your comments, suggestions and any sort of
> feedback on
> the list. There are also packaging questions to discuss, as for now this is
> implemented in the rtslib tree, and that might be suboptimal.
> Package maintainers are welcome to send me their preferences.

Hi Jerome,

Good to see your email. Wow, a quantum leap!

Some things that might also be addressed include:

- There's a possibility that configuration operations by different users
of rtslib (maybe even the same user and app, but in a different window)
might be operating on configfs at the same time, and this could cause
big trouble. Establishing a convention for a fcntl-based lock that
accessors of configfs acquire might be enough to resolve this, or maybe
the commit/rollback feature might help?

- better support for configuring ALUA? I don't think this made it into
rtslib.

But overall, yeah, the direction you've started to lay out could be a
big step in improving the accessibility of LIO's capabilities. Looking
forward to see where it goes, and possible community development
opportunities.

Regards -- Andy

[Jerome Martin]

Hi Andy,

On 02/27/2014 08:15 PM, Andy Grover wrote:
>
> Good to see your email. Wow, a quantum leap!

Thanks!

>
> Some things that might also be addressed include:
>
> - There's a possibility that configuration operations by different
> users of rtslib (maybe even the same user and app, but in a different
> window) might be operating on configfs at the same time, and this
> could cause big trouble. Establishing a convention for a fcntl-based
> lock that accessors of configfs acquire might be enough to resolve
> this, or maybe the commit/rollback feature might help?

Yes. I know you are concerned about this (I saw your session patches)
and I have been thinking about how to handle this best. I have a range
of solutions in mind, but would like to discuss that on the ML to pick
the more appropriate. First of all, I plan to make the candidate
configuration persistent accross sessions. Then provide 3 options for
config mode:

1) shared - everyone edits the same config. This is ok, as every
operation is atomic: all load/set/merge/etc. ops are staged first,
checked in the stage and _then_ pushwed to the candidate config. Also,
we always consider the diff of the current config against the actual
running system just before commit, so the window to get out-of-sync is
small. If a diff still exists after commit, one would just have to redo
the commit. Which in fact could be automated I guess.

2) locked. You lock the access to the global candidate, no-one else can
enter config mode or do changes.

3) private. You enter config mode using your own private candidate.

Note that "info" gives you a full history of the operations that
affected the current candidate config, which you can undo if you want.

Of course, this is for the cli itself, and the configuration API. As for
accessing the configfs layer, either directly or through legacy rtslib
objects, there would always be a way of bypassing any mechanism in
place, so I would prefer to have sane synchronisation in the upper levels.

At the API level though, having a lock might be the most pragmatic way
to handle the access to rtlib lower-layers.

> - better support for configuring ALUA? I don't think this made it into
> rtslib.

Yes, this is on my todo list, as well as NPIV support, and access to PR
and sessions information.

>
> But overall, yeah, the direction you've started to lay out could be a
> big step in improving the accessibility of LIO's capabilities.

That's encouraging, thanks.

> Looking forward to see where it goes, and possible community
> development opportunities.

Me too, and I sincerely hope that I'll get some help to make that a
great configuration and admin environment :-)

Regards,
--
Jerome

[Jerome Martin]

Hi Ritesh,

On 02/28/2014 08:33 AM, Ritesh Raj Sarraf wrote:
>
>> And simply load it from the lio cli using "load snippet.lio" in config
>> mode, or "merge snippet.lio" if you want to merge that to the current
>> configuration. Then you just need to issue 'commit' when your
>> configuration is ready to be applied on the system and saved as the next
>> startup config.

> You may also want to have a "validate snippet.lio", called internally,
> to validate the syntax.
>
> Almost all big projects, with large configuration files, do so.
Yes! :-)

Actually this is done on load, and before commit: checked against
installed policy (defaults, references to other objects in the config,
required attributes, syntax). More is to be implemented, as the verify()
(should call it validate, rally) command is lacking currently and does
not do a few things yet: check a portal IP exists on the server's
interfaces, files and devices path are valid, etc.

BTW Ritesh, as a package maintainer, what do you thinks would be the
best in the eventuallity we want to include rtslib + the lio cli only:
rename the rtslib package into say 'lio', including all API + the cli,
or split it in two packages (but the cli one would be tiny) ?

[Jerome Martin]

On 02/28/2014 10:00 AM, Ritesh Raj Sarraf wrote:
> Actually this is done on load, and before commit: checked against
> installed policy (defaults, references to other objects in the config,
> required attributes, syntax). More is to be implemented, as the
> verify() (should call it validate, rally) command is lacking currently
> and does not do a few things yet: check a portal IP exists on the
> server's interfaces, files and devices path are valid, etc.

> Well. Have them distinguished. :-)
>
> Validate() => To check the syntactical correctness of the data. (Is the
> IP Address in the correct syntactical format)
>
> Verify() => To check the authenticity of the data. (Is the IP Address
> existent on the server)

Interesting, I never thought of it this way explicitly, although it is
more or less what happens under the hood. I will take any opinion on the
topic as I have doubts myself re. the correct way to do things.

For now, at the time you import a configuration statement, you must go
through three step:

* The parser transforms it into an AST (parse_tree in the code). That
ensure that:
- All object names are recognized (target, fabric, lun, disk, etc.)
- Object identifiers (fileio, iscsi, etc.) and attribute names and
values fit the syntactic requirements.
- Tokens are categorized in object specifiers, attribute group names
and attributes

* Nested statements from the AST are imported, and in the process:
- Matched against the policy data - a statement MUST match one
policy rule, always. The rule gives the loader information about
expected types (int, bytes, iqn) and default values or absence thereof
- Checked for object references coherency
- All values are normalized to an internal canonical format, that
ensure repeatability of load/dump operations: you can load one config
saying "size 1.0M; enable Yes" and an other "size 1M; enable true" and
they will have the same representation once dumped (size 1.0MB; enable yes).
- Each statement ends up being loaded as a ConfigTree instance in
Config.current (which hold teh current, "candidate" configuration). That
includes a bunch of information that you can retrieve from the API or
the cli: type, default value, required/optional, source statement info
(operation type and date for loading that statement, source metadata
i.e. for load operations file name and modification time, pointer to the
policy rule covering that configuration node etc.)

All of the above happen when you load or merge a config file, issue a
single statement from cli with the set command, etc.

Then, triggered either by commit, or manually using the "commit check"
command, pre-commit consistency checks are performed.

* A for now dummy verify() _should_ check IP addresses, existence of
devices and file paths, etc.
* All objects are checked again for missing required values (this is
delayed so you can craft configs in a temporary state, fithout filling
all the bits in a strict order and in one go - gives your more
flexibility). I am thinking of moving object references (like which
backend a LUN should use) to that step BTW.


A couple of other things might happen here, but out of the top of my
head, that's the rough process.

Now, this whole system is open to discussion. Should I move the
reference check to pre-commit instead of enforcing that at config
statement load time ? Are there some checks that the user should be able
to relax for flexibility reasons ? The guideline I use for now is that
you should be able to compose a configuration one one system, dump it
and load it on a production system, and only commit it there. So all
system-specific checks should be done during pre-commit, before we
compute the commit execution plan, the non-system specific checks being
done at load time (hence the reason why object references are checked
early).

>> BTW Ritesh, as a package maintainer, what do you thinks would be the
>> best in the eventuallity we want to include rtslib + the lio cli only:
>> rename the rtslib package into say 'lio', including all API + the cli,
>> or split it in two packages (but the cli one would be tiny) ?

> Today, for the packages available in Debian, apart from targetcli,
> OpenStack (python-cinder) is the other suite that has a dependency on
> rtslib. Your library has been out for some time now, so I'm not sure who
> else have consumed it.
>
> As a package maintainer, it wouldn't cost me much as I can still derive
> both the packages from the same source archive (irrespective of the
> rename), provided the library interfaces remain the same.
>
> I'd suggest you merge them to one, name it as rtslib, and ship the tiny
> lio CLI along with it.

Mmmh. One problem here is the init sequence. With the lio cli, no need
for lio-utils anymore. The initscript will be included too. So that new
rtslib+configAPI+lio cli package would conflict with lio-utils, which is
required by the current targetcli, which also ull configshell BTW. So it
would make it impossible to install both targetcli+lio-utils and a
lio-cli enabled rtslib, which I think we should support at least during
some migration period, and for side-to-side debug/testing. I really
don't know how to best do it, or I should leave all of that up to
package maintainers?