Can this code be improved?

[AJ R]

Hi,

So I rewrote ctypes code and use CFFI in ABI mode now. Can the code below be improved? Esp the part inside the while loop? I initially found that ctypes + cpython was fastest, then cffi + pypy, then ctypes + pypy (I did not try cffi + cpython). However, cffi + pypy became about 10% faster than any alternative after tweaking PYPY_GC_NURSERY. But, to be honest, this is really trial-and-error. On my test Windows VM, the amount of L1 cache is unknown. There's 8Gb RAM and 2 CPU cores. Increasing PYPY_GC_NURSERY from 4 to 8Mb helped massively. I have no idea what nursery size is best on a more powerful system. Or what other gc vars I should experiment with.

 def get_records(length):

     buffer = ffi.new("char[]", length)

     ffi_string = ffi.string

     c_get_record = dll_handle.XSortGetRecord

     while True:

         ret = c_get_record(buffer)

         if ret > 0:

             yield ffi_string(buffer)[:ret]

         elif ret == -1:

             error = XSortGetError()

             ffi.release(buffer)

             raise Exception(error)

         else:

             break

     ffi.release(buffer)

Thanks in advance!

Best wishes,

Albert-Jan

[Armin Rigo]

Hi,

On Sat, 6 Aug 2022 at 13:16, AJ R <sjeik...@gmail.com> wrote:
> yield ffi_string(buffer)[:ret]

We can't do anything more than guess from just a small function in
your code base. But I would guess that the line above unnecessarily
makes a (bytes or unicode) string, and then truncates it. I have no
idea about what you are really looking for, but the above line could
possibly be replaced with: "yield ffi_string(buffer, ret)". Or, if
you know that "ret" is exactly the length, you could do "yield
ffi.buffer(buffer, ret)[:]", which doesn't even scan the buffer for
zeroes.


A bientôt,
Armin Rigo

[Armin Rigo]

Hi again,

About PyPy's GC tweaks, maybe post a complete runnable example to the
pypy...@python.org mailing list. There are people there (including
me in the past) interested in knowing if we should tweak the default
value of these GC env vars; originally, we found them optimal, but
that was ten years ago. CPU performance and cache sizes have changed
since then.

A bientôt,
Armin Rigo

[AJ R]

Merci bien Armin, I just sent an email to pypy-dev.

Best wishes,

Albert-Jan