Requesting feedback from users of Ledger's Python API

[Alexis]

Hello all,

I've been using Ledger's Python API extensively for many years now
and have worked around some of its quirks and limitations in odd ways.

In my mind Ledger's Python API (and Ledger itself) has a lot of
unused potential.

I believe one next step towards unlocking that potential is converting
Ledger's Python module to a Python package.

"What's the difference?" someone might ask.

A Python module is a single shared library compiled from Ledger's
C++ code that can be used by Python. The bridging from C++ to Python
code (and vice versa) is done via Boost.Python¹.

A Python package on the other hand is a bundle of Python modules and
pure Python files which in combination offer a richer and more
versatile API.
A Python package allows to extend the functionality of C++ native types
by adding new functions to them or provide entirely new classes written
only in Python. This would allow for easier exploration for
new features. And if a new feature proves useful but is slow in Python
it would likely be less tedious to port it to C++ than develop it in
C++ from the beginning.

The PR #2196² is a draft that introduces Ledger Python package.

While the PR breaks existing scripts the fix is as simple as changing
`import ledger` to `import ledger.core as ledger`.

I kindly ask people relying on or using Ledger's Python API to provide
feedback on how the Ledger Python is used.

How do you invoke Ledger Python?
(ledger python, python script, in-line journal python, other)

Which parts of the Ledger API do you use most?
(Examples would be helpful)

What features are missing from Ledger Python?

Which existing Ledger Python bugs are limiting?

Getting answers to these questions would help drive development
in a direction that is most useful for API users.


Best
Alexis

¹ https://www.boost.org/doc/libs/1_81_0/libs/python/doc/html/index.html
² https://github.com/ledger/ledger/pull/2196

[Daniele Nicolodi]

On 14/02/23 11:48, Alexis wrote:
> Hello all,
>
> I've been using Ledger's Python API extensively for many years now
> and have worked around some of its quirks and limitations in odd ways.
>
> In my mind Ledger's Python API (and Ledger itself) has a lot of
> unused potential.
>
> I believe one next step towards unlocking that potential is converting
> Ledger's Python module to a Python package.

Hello Alexis,

one of the reasons I switched to Beancount after a brief period using
Ledger was indeed the comprehensive Python API offered by Beancount (an
easier feat, given that Beancount itself is implemented in Python).

If you are interested in scripting i recommend having a look at
Beancount. Beancount syntax is not exactly the same as Ledger syntax,
but there is the excellent ledger2beancount project to automate the
conversion.

Beancount (by choice) does not implement some of the features of Ledger,
but has support for plugins (written in Python) that allow to quite
easily fill the gap.

Another selling point for Beancount, if you are interested in analyzing
your ledger data in complex ways, is bean-query: an SQL inspired query
language for interrogating the ledger data an return in as tables.

Cheers,
Dan

[Alexis]

Hello Dan,

thank you for your reply.

At this point I'm too invested in Ledger's Python API with all the
scripts and the setup that I have and I'm unwilling to switch any
time soon, if ever :)

As a developer/Ledger contributor I'm interested in learning about
use-cases that folks have for a Python API.

Is there anything specific that you'd be willing to share in greater
detail? Any scripts, reports, or workflows would be interesting.


Thanks
Alexis

[Ryan Nowakowski]

On Tue, Feb 14, 2023 at 11:48:19AM +0100, Alexis wrote:
> While the PR breaks existing scripts the fix is as simple as changing
> `import ledger` to `import ledger.core as ledger`.

Is there a way to add the package but retain backward compatibility?

Alternatively and perhaps more interesting, you could create a separate
python package on PyPI that depends on ledger. That would allow you to
iterate more quickly than the current pace of ledger releases.
ledger as a tool is pretty stable whereas this new python package would
be less so as you quickly add features. Then once the python package
is more stable, you can merge it back into the ledger codebase. This is
the pattern that works well in the Django project. For an example of
this see Django's adoption of "South" as Django's db migration system.

> I kindly ask people relying on or using Ledger's Python API to provide
> feedback on how the Ledger Python is used.
>
> How do you invoke Ledger Python?
> (ledger python, python script, in-line journal python, other)

I use it as an import in scripts and ad-hoc in ipython.

> Which parts of the Ledger API do you use most?
> (Examples would be helpful)

I use it as a parser[1] to then be able to automatically edit my journal
via Python. I also use that same parser to import my ledger into some
Django models for easy editing via the web.

[1] https://gist.github.com/tubaman/0e80cec388d3d5d61e3f300e2477a9ae

> What features are missing from Ledger Python?

Documentation is pretty sparse. Most of my use comes from exploring the
API in ipython.

> Which existing Ledger Python bugs are limiting?

The one I filed a couple of weeks ago ;)

[Alexis]

Hello Ryan,

thank you for your detailed response, it is very helpful and
much appreciated.

> Is there a way to add the package but retain backward compatibility?

Yes, there is a way to add the package and retain backward compatibility.
I disliked the way I initially thought of, though, fortunately your
question made me think about it some more and I may have another idea,
that is backward compatible and offers a clean API.

> Alternatively and perhaps more interesting, you could create a separate
> python package on PyPI that depends on ledger. That would allow you to
> iterate more quickly than the current pace of ledger releases.

Interesting idea, thanks for the suggestions. I'll look into what is
needed to make it happen. Though I do like Ledger's Python API to be
so close to the native code. I need to dabble around with it to get
a better understanding of the pros and cons.

> I use it as an import in scripts and ad-hoc in ipython.

Anything you'd be willing to share?

> I use it as a parser[1] to then be able to automatically edit my journal
> via Python. I also use that same parser to import my ledger into some
> Django models for easy editing via the web.
>
> [1] https://gist.github.com/tubaman/0e80cec388d3d5d61e3f300e2477a9ae

Interesting. With only a rough look at the code are you aware of
Ledger's Data File Parsing Information¹?

> Documentation is pretty sparse. Most of my use comes from exploring the
> API in ipython.

What would you find most helpful?
A technical documentation? A tutorial? Something else?

> The one I filed a couple of weeks ago ;)

I assume you mean #2169.² I'll have another look.


Best
Alexis

¹ https://www.ledger-cli.org/3.0/doc/ledger3.html#Data-File-Parsing-Information
² https://github.com/ledger/ledger/issues/2169

[Ryan Nowakowski]

On Mon, Feb 27, 2023 at 05:11:46PM +0100, Alexis wrote:
> > Alternatively and perhaps more interesting, you could create a separate
> > python package on PyPI that depends on ledger. That would allow you to
> > iterate more quickly than the current pace of ledger releases.
>
> Interesting idea, thanks for the suggestions. I'll look into what is
> needed to make it happen. Though I do like Ledger's Python API to be
> so close to the native code. I need to dabble around with it to get
> a better understanding of the pros and cons.

Let me know if you need someone to bounce ideas off of.

> > I use it as an import in scripts and ad-hoc in ipython.
>
> Anything you'd be willing to share?

The gist below pretty much covers it.

> > I use it as a parser[1] to then be able to automatically edit my journal
> > via Python. I also use that same parser to import my ledger into some
> > Django models for easy editing via the web.
> >
> > [1] https://gist.github.com/tubaman/0e80cec388d3d5d61e3f300e2477a9ae


> Interesting. With only a rough look at the code are you aware of
> Ledger's Data File Parsing Information¹?

Yup, I'm using end_line in my gist above.

> > Documentation is pretty sparse. Most of my use comes from exploring the
> > API in ipython.
>
> What would you find most helpful?
> A technical documentation? A tutorial? Something else?

In priority order:

1. Comprehensive API documentation
2. A Quickstart guide

> > The one I filed a couple of weeks ago ;)
>
> I assume you mean #2169.² I'll have another look.

Yup! That's the one.

> ² https://github.com/ledger/ledger/issues/2169

[Alexis]

Hello Ryan,

On Mon, Feb 27, 2023 at 01:29:46PM -0600, Ryan Nowakowski wrote:
> On Mon, Feb 27, 2023 at 05:11:46PM +0100, Alexis wrote:
> > > Alternatively and perhaps more interesting, you could create a separate
> > > python package on PyPI that depends on ledger. That would allow you to
> > > iterate more quickly than the current pace of ledger releases.
> >
> > Interesting idea, thanks for the suggestions. I'll look into what is
> > needed to make it happen. Though I do like Ledger's Python API to be
> > so close to the native code. I need to dabble around with it to get
> > a better understanding of the pros and cons.
>
> Let me know if you need someone to bounce ideas off of.

Thanks for the offer, Ryan, much appreciated. I have a personal project
that I could use to get a feel for backwards compatible Ledger Python
API changes and also get acquainted with the process of creating
and maintaining a Python Package dependent on Ledger's Python Package
/ Module.

> The gist below pretty much covers it.

Great, I'll have a closer look.

> In priority order:
>
> 1. Comprehensive API documentation
> 2. A Quickstart guide

Ideally comprehensive API would be generated from code. Do you happen
to have any knowledge or experience with that?

In your mind is demo.py¹ a good starting point for a quickstart guide
that mainly needs some prose for explanation?
If not, is there an example of a good and helpful quickstart guide
that you've come across and could share?


Best
Alexis

¹ https://github.com/ledger/ledger/blob/master/python/demo.py

[Ryan Nowakowski]

[Alexis]

Hello all,

those who feel comfortable with installing Python packages from
source and installing Ledger's dependencies are invited to help
test the very first draft Pull Request¹ that allows the Ledger
Python package to be install via pip² (or other installation
methods supporting pyproject.toml³).

With Ledger's main dependencies installed (boost, boost-python, gmp,
mpfr, and gpgme) you can run the following (ideally from a Python
virtual environment) to install the Ledger Python package:

% /usr/bin/env CMAKE_BUILD_PARALLEL_LEVEL=$(sysctl -n hw.ncpu) \
pip install --verbose --no-dependencies \
git+https://github.com/afh/ledger.git@pypi

to test run:

% python ${PATH_TO}/ledger.git/python/demo.py

Head over to https://github.com/ledger/ledger/pull/2246 for
further the details and leave feedback and comments on the PR.

If you are interested in helping shape the future of Ledger
Python package API now is the time to chime in.


Best
Alexis

¹ https://github.com/ledger/ledger/pull/2246
² https://pypi.org/project/pip
³ https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml