kas 5.3 streams limit exceeded

[ORTIZ, ADRIAN]

Hello, 

We faced an issue in kas 5.3. 

While building, kas's stdout reader hit asyncio's 64 KiB per‑line default limit and crashed.

Recipe recipe cve-update-nvd2-native can generate a very long line.

25-Jun-2026 07:41:31    2026-06-25 07:41:31 - INFO     - NOTE: recipe cve-update-nvd2-native-1.0-r0: task do_rm_work: Succeeded

25-Jun-2026 07:41:32    2026-06-25 07:41:32 - ERROR    - Separator is found, but chunk is longer than limit

25-Jun-2026 07:46:41    Traceback (most recent call last):

25-Jun-2026 07:46:41      File "/usr/lib/python3.11/asyncio/streams.py", line 545, in readline

25-Jun-2026 07:46:41        line = await self.readuntil(sep)

25-Jun-2026 07:46:41               ^^^^^^^^^^^^^^^^^^^^^^^^^

25-Jun-2026 07:46:41      File "/usr/lib/python3.11/asyncio/streams.py", line 640, in readuntil

25-Jun-2026 07:46:41        raise exceptions.LimitOverrunError(

25-Jun-2026 07:46:41    asyncio.exceptions.LimitOverrunError: Separator is found, but chunk is longer than limit

25-Jun-2026 07:46:41    

25-Jun-2026 07:46:41    During handling of the above exception, another exception occurred:

25-Jun-2026 07:46:41    

25-Jun-2026 07:46:41    Traceback (most recent call last):

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/kas.py", line 245, in main

25-Jun-2026 07:46:41        kas(sys.argv[1:])

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/kas.py", line 219, in kas

25-Jun-2026 07:46:41        plugin.run(args)

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/plugins/build.py", line 107, in run

25-Jun-2026 07:46:41        macro.run(ctx, args.skip)

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/libcmds.py", line 119, in run

25-Jun-2026 07:46:41        _run_single(cmd)

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/libcmds.py", line 110, in _run_single

25-Jun-2026 07:46:41        command.execute(ctx)

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/plugins/build.py", line 154, in execute

25-Jun-2026 07:46:41        loop.run_until_complete(run_cmd_async(cmd, cwd=ctx.build_dir,

25-Jun-2026 07:46:41      File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete

25-Jun-2026 07:46:41        return future.result()

25-Jun-2026 07:46:41               ^^^^^^^^^^^^^^^

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/libkas.py", line 178, in run_cmd_async

25-Jun-2026 07:46:41        await asyncio.gather(*[asyncio.shield(t) for t in tasks])

25-Jun-2026 07:46:41      File "/opt/pipx/venvs/kas/lib/python3.11/site-packages/kas/libkas.py", line 107, in _read_stream

25-Jun-2026 07:46:41        line = await stream.readline()

25-Jun-2026 07:46:41               ^^^^^^^^^^^^^^^^^^^^^^^

25-Jun-2026 07:46:41      File "/usr/lib/python3.11/asyncio/streams.py", line 554, in readline

25-Jun-2026 07:46:41        raise ValueError(e.args[0])

25-Jun-2026 07:46:41    ValueError: Separator is found, but chunk is longer than limit

25-Jun-2026 07:46:41    Failing task since return code of [/tmp/runInDocker6682003822408205187.sh /apps/bamboo/remote-agent/agent-home/xml-data/build-dir/MAX-MLOB881-BFUSION/bamboo-build.sh] was 1 while expected 0

25-Jun-2026 07:46:41    Finished task 'Execute build' with result: Failed

Applying this patch fixed the issue(maybe 16mb is too much?):

--- a/kas/libkas.py

+++ b/kas/libkas.py

@@ -148,6 +148,9 @@ async def run_cmd_async(cmd, cwd, env=None, fail=True, liveupdate=False,

     try:

         process = await asyncio.create_subprocess_exec(

             *cmd,

+            # Workaround: raise asyncio's default 64 KiB StreamReader limit.

+            # Some bitbake stdout lines (e.g. cve-check summaries) exceed it.

+            limit=16 * 1024 * 1024,

             cwd=cwd,

             env=env,

             stdout=asyncio.subprocess.PIPE,

Thanks!

Saludos/Regards/beste Grüße

 

Adrián Ortiz Gutiérrez

adrian...@tkelevator.com

 

[MOESSBAUER, Felix]

On Thu, 2026-06-25 at 09:27 +0000, ORTIZ, ADRIAN wrote:
> Hello, 
>
>
> We faced an issue in kas 5.3. 

Hi,

unfortunately this will also affect kas 5.4 which just got released.

This is the culprit. Increasing the buffer just makes it less likely to
hit the limit, but IMHO is no proper fix.

We currently use .readline to forward the line to the logger (like
line-buffering). I'm wondering if we better just read up to a fixed
amount of characters and trim. Then we can seek until the next newline
character and start over.

> 25-Jun-2026 07:46:41               ^^^^^^^^^^^^^^^^^^^^^^^
> 25-Jun-2026 07:46:41      File "/usr/lib/python3.11/asyncio/streams.py", line 554, in readline
> 25-Jun-2026 07:46:41        raise ValueError(e.args[0])
> 25-Jun-2026 07:46:41    ValueError: Separator is found, but chunk is longer than limit
> 25-Jun-2026 07:46:41    Failing task since return code of [/tmp/runInDocker6682003822408205187.sh /apps/bamboo/remote-agent/agent-home/xml-data/build-dir/MAX-MLOB881-BFUSION/bamboo-build.sh] was 1 while expected 0
> 25-Jun-2026 07:46:41    Finished task 'Execute build' with result: Failed
>
>
> Applying this patch fixed the issue(maybe 16mb is too much?):
>
>
> --- a/kas/libkas.py
> +++ b/kas/libkas.py
>
> @@ -148,6 +148,9 @@ async def run_cmd_async(cmd, cwd, env=None, fail=True, liveupdate=False,
>      try:
>          process = await asyncio.create_subprocess_exec(
>              *cmd,
> +            # Workaround: raise asyncio's default 64 KiB StreamReader limit.
> +            # Some bitbake stdout lines (e.g. cve-check summaries) exceed it.
> +            limit=16 * 1024 * 1024,

Well... That means potentially 16MB ob junk on your terminal. While
this might not break the StreamReader, I'm not sure if it breaks
somewhere else (e.g. in the logger).

Felix

>              cwd=cwd,
>              env=env,
>              stdout=asyncio.subprocess.PIPE,
>
>
>
>
> Thanks!
>
>
>
>
>
> Saludos/Regards/beste Grüße
>
>  
>
> Adrián Ortiz Gutiérrez
>
> adrian...@tkelevator.com
>
>  

> --
> You received this message because you are subscribed to the Google Groups "kas-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kas-devel+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/kas-devel/AMBPR09MB83289199D03D80831D8E39AAFDEC2%40AMBPR09MB8328.eurprd09.prod.outlook.com.

[Jan Kiszka]

Indeed.

Is this really the way that yocto task is supposed to be used? Is it
reproducible with upstream yocto means, just triggered by a larger
downstream dataset? Or are there downstream changes involved?

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center