luigi.format.TextFormat: How to preserve line endings/carriage return?

[Christoph Thiede]

Hi,

today I found out that luigi targets do not preserve original line endings when using a text format such as UTF8. Quick example:

```

import luigi

from unittest import TestCase

class TestTarget(TestCase):

    def test_target(self):

        target = luigi.LocalTarget("bar", format=luigi.format.UTF8)

        string0 = "foo\nbar\rbaz"

        with target.open('w') as stream:

            stream.write(string0)

        with target.open('r') as stream:

            string1 = stream.read()

        self.assertEqual(string0.encode(), string1.encode())

```

This test fails with AssertionError: b'foo\nbar\rbaz' != b'foo\nbar\nbaz'.

So here is my question: How can I preserve the original line endings but still use the UTF8 format? I see that I could `luigi.format.Nop` instead but this would be a pity because I would need to handle the formatting everywhere manually. Is there any option to use TextFormat but preserve original line endings?

Thanks in advance,

Christoph

[Tashrif B.]

Yeah, it would be:

with open(target,'w') as f:
  # write something
  f.write('hi')

 

with open(target) as f:
  # read back
  print(f.read())

-Tashrif

. 

On Tue, Feb 16, 2021 at 7:49 AM Christoph Thiede <christoph....@gmail.com> wrote:

Hi Tashrif,

thanks for your help. Unfortunately, this does not work for me:

>>> import luigi

>>> target = 'bar'

>>> target.open('w')

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

AttributeError: 'str' object has no attribute 'open'

Actually, I would have been very surprised if luigi would have added its own logic directly to Python's str class. Or am I probably misunderstanding your proposal?

Best,

Christoph

---

Von: Tashrif B. <tashri...@gmail.com>
Gesendet: Montag, 15. Februar 2021 18:48 Uhr
An: Christoph Thiede <christoph....@gmail.com>
Betreff: Re: luigi.format.TextFormat: How to preserve line endings/carriage return?

 

Hi Christoph,

How about:

target= 'bar'

with target.open('w') as stream:

I am suggesting this because I never created a target using luigi.localTarget(). Rather, I always used luigi.localTarget(bar) to check the existence of bar as created above (or by some other means).

Best,
Tashrif

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/luigi-user/0126ac32-d396-443a-9304-290e39917498n%40googlegroups.com.

[Christoph Thiede]

Ah, now I see.

Unfortunately, the built-in open() method does not support luigi's MockTargets which we use quite frequently in our tests, so this won't be an option for us ...

Best,

Christoph

---

Von: Tashrif B. <tashri...@gmail.com>
Gesendet: Dienstag, 16. Februar 2021 15:19 Uhr
An: Christoph Thiede <christoph....@gmail.com>; Luigi <luigi...@googlegroups.com>