Using Custom directives from importer script

[Mason H]

Hello,

I've recently been trying to write an importer script and was trying to use a Custom directive. There isn't a lot of documentation on this, and I haven't been able to figure out what the problem is from the source, and was wondering if anyone here had any ideas. I'm not sure if I'm using beancount.core.data.Custom wrong or if you're supposed to use a different object entirely.

Issue occurs when there is a beancount.core.data.Custom object in the list returned by the extract() function of a subclass of beancount.ingest.importer.ImporterProtocol. This is being ran by the 'bean-extract' command. The exception appears to be raised when printing the formatted beancount directives. I've written a test file which hopefully reproduces the issue on other machines below.

(To prevent an XYZ problem) I have a small number of rental properties which are run on a short term basis. I wanted to have a custom directive define a 'stay' so I could keep some relevant metadata for transactions (i.e. OTA platform, dates and length of stay, address of the property) , but I don't want to reference this metadata multiple times when there are multiple transactions involving a single stay. I was then hoping to link '^' all the transactions involving a single stay with a unique code, including this custom directive. I could do the same thing with the first transaction for a stay, but when 9/10 times I only want to know the property which a transaction was for, I don't want ten other metadata fields showing up in fava when I don't want them.

. . . At this point though I'm rather curious why I'm getting the exception, regardless of my use of the directive.

Other notes:

• I am running a python virtual environment for my beancount and fava executables, plus the modules used for importers.
• python version is 3.11.7. Binary is from Arch Linux repo.
  
• I am running the beancount version pulled from pip; Beancount 2.3.6 (git:d77540c4; 2023-10-05)
• (I'm unsure whether I should post this type of thing to github issues, mostly because I don't know if I'm doing something wrong or if this is a bug. Assuming I don't see an issue already I'm assuming I'm doing something wrong, so I'm posting this here).
• pdb identifies entry.values to be the last argument of the beancount.core.data.Custom, which is a list of arbitrary values.

Other questions involving custom directive:

• If I want to add links, tags to the directive, do I just append a str starting with '^' and '#' respectively to the values list?

Below is output with the exception:

;; -*- mode: beancount -*-
**** /home/mason/Documents/beancount/Test/test.txt

1970-01-01 * "Vendor" "Narration"
  Income:Example  10 USD
  Assets:Example

Traceback (most recent call last):
  File "/home/mason/.local/bin/bean-extract", line 8, in <module>
    sys.exit(extract_main())
             ^^^^^^^^^^^^^^
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/ingest/
scripts_utils.py", line 36, in extract_main
    return trampoline_to_ingest(extract)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/ingest/
scripts_utils.py", line 198, in trampoline_to_ingest
    return run_import_script_and_ingest(parser)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/ingest/
scripts_utils.py", line 246, in run_import_script_and_ingest
    return ingest(importers_list)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/ingest/
scripts_utils.py", line 140, in ingest
    args.command(args, parser, importers_list, abs_downloads, hooks=hooks)
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/ingest/
extract.py", line 243, in run
    extract(importers_list, files_or_directories, sys.stdout,
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/ingest/
extract.py", line 214, in extract
    print_extracted_entries(new_entries, output)
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/ingest/
extract.py", line 137, in print_extracted_entries
    entry_string = printer.format_entry(entry)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/parser/
printer.py", line 374, in format_entry
    return EntryPrinter(dcontext, render_weights, prefix=prefix)(entry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/parser/
printer.py", line 124, in __call__
    method(obj, oss)
  File "/home/mason/Documents/beancount/beancount-bin/lib/python3.11/site-packages/beancount/parser/
printer.py", line 345, in Custom
    for value, dtype in entry.values:
        ^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)

Below is a command, config file (green), and importer script (blue). The exception isn't raised when you comment out the line with data.Custom.

Where test.txt is an empty file:

$ bean-extract exampleimporter.config test.txt

# exampleimporter.config

import sys

testdir = '' # add full directory test.py is in here

sys.path.append(testdir1)
import test
CONFIG = [
    test.Importer()
    ]

#EOF

# test.py

from datetime import date as fdate
from beancount.core.data import EMPTY_SET
from beancount.ingest import importer
from beancount.core.number import D, ZERO
from beancount.core import data, account, amount, position

class Importer(importer.ImporterProtocol):
    def __init__(self, *args, **kwargs):
        pass

    def identify(self, file):
        return True
   
    def extract(self, file):
        ret = []
        meta = data.new_metadata(file.name, 1)
        date = fdate(1970,1,1)
        amnt = amount.Amount(D(10), 'USD')
        ret.append(data.Transaction(meta, date, '*', "Vendor", "Narration", EMPTY_SET, EMPTY_SET, [
            data.Posting('Income:Example', amnt, None, None, None, None),
            data.Posting('Assets:Example', None, None, None, None, None) ]))
        ret.append(data.Custom(meta, date, "example", ["value1", "value2", "value3"]))
        return ret

#EOF

[Daniele Nicolodi]

On 16/02/24 22:51, Mason H wrote:
> Hello,
>
> I've recently been trying to write an importer script and was trying to
> use a Custom directive. There isn't a lot of documentation on this, and I haven't
> been able to figure out what the problem is from the source, and was
> wondering if anyone here had any ideas. I'm not sure if I'm using
> beancount.core.data.Custom wrong or if you're supposed to use a
> different object entirely.
>
> Issue occurs when there is a beancount.core.data.Custom object in the
> list returned by the extract() function

[...]

> ret.append(data.Custom(meta, date, "example", ["value1", "value2", "value3"]))

The fourth field in a beancount.core.data.Custom() object is a list of
beancount.parser.grammar.ValueType() instances. ValueType() instances
are initialized with a value and a data type (a Python class).
Therefore, your code should read something like:

ret.append(data.Custom(
meta, date, "example", [
grammar.ValueType("value1", str),
grammar.ValueType("value2", str),
grammar.ValueType("value3", str),
])
)

An easy way to figure out this kind of things is to run the parser on
the Beancount format representation of the directive you want to obtain:

from beancount.parser import parser
entries, errors, options = parser.parse_string(
'''2024-02-17 custom "example" "value1" "value2" "value3"'''
)
print(entries)

Cheers,
Dan

[fin]

Mason H wrote:
> Hello,
>
...

> (To prevent an XYZ problem) I have a small number of rental properties
> which are run on a short term basis. I wanted to have a custom directive
> define a 'stay' so I could keep some relevant metadata for transactions
> (i.e. OTA platform, dates and length of stay, address of the property) ,
> but I don't want to reference this metadata multiple times when there are
> multiple transactions involving a single stay. I was then hoping to link
> '^' all the transactions involving a single stay with a unique code,
> including this custom directive. I could do the same thing with the first
> transaction for a stay, but when 9/10 times I only want to know the
> property which a transaction was for, I don't want ten other metadata
> fields showing up in fava when I don't want them.
>
...

> 1970-01-01 * "Vendor" "Narration"
> Income:Example 10 USD
> Assets:Example

why not just use the overall Account Root (or a parent part
of a subaccount) as a holder for some of the metadata? and
then you could do the occupancy flag as one of the values of
subaccount. like:


1970-01-01 * "Fredder Building" ""
address1: "101 Fredder St"
address2: "City, State, Country"
superphone: "(123) 567-9801"
Assets:Building:Fredder 1
Equity:Buildings -1
Assets:Apartments:Fredder 12
Equity:Apartments

1970-03-11 * "Apartment Rental" "Fredder, Apt 3"
Assets:Fredder:Apt3 1
Equity:Apartments

etc?


fin