[erlang-questions] Binary pattern matching and optimization

[Vance Shipley]

I would like to have a function like this:

foo(<<_:Offset/binary, Foo:8, _/binary>>, Offset) ->
bar(Bin, Foo).

Unfortunately that doesn't work:

Eshell V5.6.5 (abort with ^G)
1> compile:file(codec).
./codec.erl:15: variable 'Offset' is unbound
error

I don't understand why the first version can't work but
neither do I understand much about writing compilers.

So instead I do this:

foo(Bin, Offset) ->
<<_:Offset/binary, Foo:8, _/binary>> = Bin,
bar(Bin, Foo).

This works fine of course but now I am trying to understand
the meaning of this (optional) compiler warning:

Warning: INFO: using a matched out sub binary will prevent
delayed sub binary optimization

How should I interpret this warning?

-Vance
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions

[Robert Virding]

2008/11/24 Vance Shipley <van...@motivity.ca>

I would like to have a function like this:

       foo(<<_:Offset/binary, Foo:8, _/binary>>, Offset) ->
               bar(Bin, Foo).

Unfortunately that doesn't work:

       Eshell V5.6.5  (abort with ^G)
       1> compile:file(codec).
       ./codec.erl:15: variable 'Offset' is unbound
       error

The order in which arguments are matched is not defined so there is no guarantee that Offset has a value when the binary is matched.

I don't understand why the first version can't work but
neither do I understand much about writing compilers.

So instead I do this:

       foo(Bin, Offset) ->
               <<_:Offset/binary, Foo:8, _/binary>> = Bin,
               bar(Bin, Foo).

This works fine of course but now I am trying to understand
the meaning of this (optional) compiler warning:

       Warning: INFO: using a matched out sub binary will prevent
       delayed sub binary optimization

How should I interpret this warning?

The compiler and VM is smart and optimises the matching of a binary where the last segment is a binary. This makes it more efficient to use a binary like a list and pick things off the front.

*BUT* for this to work the reference to the whole binary, Bin your case, must not be used after the match, then the compiler can reuse that binary reference and save work. You reuse Bin and the compiler is warning you that when you do this it can't do a good a job as possible.

Robert

[Vance Shipley]

Robert,

My thinking was that reading a big binary and then passing it
with offsets to various parsing functions would be easier on
the emulator than creating a new binary for each one. Is it
the case that the compiler will optimize away the creation of
these binaries?

-Vance

On Mon, Nov 24, 2008 at 08:10:38PM +0100, Robert Virding wrote:
} The compiler and VM is smart and optimises the matching of a binary where
} the last segment is a binary. This makes it more efficient to use a binary
} like a list and pick things off the front.
}
} *BUT* for this to work the reference to the whole binary, Bin your case,
} must not be used after the match, then the compiler can reuse that binary
} reference and save work. You reuse Bin and the compiler is warning you that
} when you do this it can't do a good a job as possible.

[Robert Virding]

2008/11/24 Vance Shipley <van...@motivity.ca>

Robert,

My thinking was that reading a big binary and then passing it
with offsets to various parsing functions would be easier on
the emulator than creating a new binary for each one.  Is it
the case that the compiler will optimize away the creation of
these binaries?

No, I don't think so. It optimises the following case:

foo(<< ... , Rest/binary>>, ... ) ->
    ...
    bar(Rest, ...);

The important thing is that the old reference to the binary is not used again and can be reused by the compiler. If you have a big binary and want to pass off chunks of it to different functions is a different problem. Then it is probably better to pass the whole binary and an offset. However, doing something like:

foo(Offset, Bin, ...) ->
    << _:Offset/binary, Stuff, _/binary>> = Bin,
    ...
    foo(Offset + N, Bin, ...)

and skipping over the beginning each time is most likely not the best way to step over bits of your chunk. I would do something like:

foo(Offset, Bin, ...) ->
    << _:Offset/binary,TheGoodBit/binary >> = Bin,
    foo1(TheGoodBit, ...).

and foo1 as my first example above which picks bit off the head of the bin.

Robert

[Bob Ippolito]

That used to be true, but it's supposed to be faster in R12+ to do it
in the "obvious" way.

[David Mercer]

Follow-up question to the converse problem: assembling binaries.  If binaries are best handled like lists while reading them, is it also true that binaries are best assembled back-to-front like lists, putting the new binary chunk onto the beginning of the binary instead of the end?

 

---

[Vance Shipley]

David,

That seems to be well documented in the efficiancy guide:

http://erlang.org/doc/efficiency_guide/binaryhandling.html#4.2

In R12B appending to binaries is optimized.

-Vance

On Mon, Nov 24, 2008 at 04:04:26PM -0600, David Mercer wrote:
} Follow-up question to the converse problem: assembling binaries. If
} binaries are best handled like lists while reading them, is it also true
} that binaries are best assembled back-to-front like lists, putting the new
} binary chunk onto the beginning of the binary instead of the end?

[David Mercer]

That is very good indeed. I kind of felt a pang of guilt in posting my
questio, because I thought I should test before asking, so I could see for
myself, but really, I ought to have looked in the docs. Thanks, Vance.

David

> -----Original Message-----
> From: Vance Shipley [mailto:van...@motivity.ca]
> Sent: Monday, November 24, 2008 16:11
> To: dme...@alum.mit.edu
> Cc: Erlang Questions
> Subject: Re: [erlang-questions] Binary pattern matching and optimization
>