Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Binary pattern matching and optimization
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  8 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Vance Shipley  
View profile  
 More options Nov 24 2008, 1:08 pm
From: Vance Shipley <van...@motivity.ca>
Date: Mon, 24 Nov 2008 13:08:32 -0500
Local: Mon, Nov 24 2008 1:08 pm
Subject: [erlang-questions] Binary pattern matching and optimization
I would like to have a function like this:

        foo(<<_:Offset/binary, Foo:8, _/binary>>, Offset) ->
                bar(Bin, Foo).

Unfortunately that doesn't work:

        Eshell V5.6.5  (abort with ^G)
        1> compile:file(codec).
        ./codec.erl:15: variable 'Offset' is unbound
        error

I don't understand why the first version can't work but
neither do I understand much about writing compilers.

So instead I do this:

        foo(Bin, Offset) ->
                <<_:Offset/binary, Foo:8, _/binary>> = Bin,
                bar(Bin, Foo).

This works fine of course but now I am trying to understand
the meaning of this (optional) compiler warning:

        Warning: INFO: using a matched out sub binary will prevent
        delayed sub binary optimization

How should I interpret this warning?

        -Vance
_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Robert Virding  
View profile  
 More options Nov 24 2008, 2:10 pm
From: "Robert Virding" <rvird...@gmail.com>
Date: Mon, 24 Nov 2008 20:10:38 +0100
Local: Mon, Nov 24 2008 2:10 pm
Subject: Re: [erlang-questions] Binary pattern matching and optimization

2008/11/24 Vance Shipley <van...@motivity.ca>

> I would like to have a function like this:

>        foo(<<_:Offset/binary, Foo:8, _/binary>>, Offset) ->
>                bar(Bin, Foo).

> Unfortunately that doesn't work:

>        Eshell V5.6.5  (abort with ^G)
>        1> compile:file(codec).
>        ./codec.erl:15: variable 'Offset' is unbound
>        error

The order in which arguments are matched is not defined so there is no
guarantee that Offset has a value when the binary is matched.

I don't understand why the first version can't work but

> neither do I understand much about writing compilers.

> So instead I do this:

>        foo(Bin, Offset) ->
>                <<_:Offset/binary, Foo:8, _/binary>> = Bin,
>                bar(Bin, Foo).

> This works fine of course but now I am trying to understand
> the meaning of this (optional) compiler warning:

>        Warning: INFO: using a matched out sub binary will prevent
>        delayed sub binary optimization

> How should I interpret this warning?

The compiler and VM is smart and optimises the matching of a binary where
the last segment is a binary. This makes it more efficient to use a binary
like a list and pick things off the front.

*BUT* for this to work the reference to the whole binary, Bin your case,
must not be used after the match, then the compiler can reuse that binary
reference and save work. You reuse Bin and the compiler is warning you that
when you do this it can't do a good a job as possible.

Robert

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Vance Shipley  
View profile  
 More options Nov 24 2008, 2:27 pm
From: Vance Shipley <van...@motivity.ca>
Date: Mon, 24 Nov 2008 14:27:30 -0500
Local: Mon, Nov 24 2008 2:27 pm
Subject: Re: [erlang-questions] Binary pattern matching and optimization
Robert,

My thinking was that reading a big binary and then passing it
with offsets to various parsing functions would be easier on
the emulator than creating a new binary for each one.  Is it
the case that the compiler will optimize away the creation of
these binaries?

        -Vance

On Mon, Nov 24, 2008 at 08:10:38PM +0100, Robert Virding wrote:
}  The compiler and VM is smart and optimises the matching of a binary where
}  the last segment is a binary. This makes it more efficient to use a binary
}  like a list and pick things off the front.
}  
}  *BUT* for this to work the reference to the whole binary, Bin your case,
}  must not be used after the match, then the compiler can reuse that binary
}  reference and save work. You reuse Bin and the compiler is warning you that
}  when you do this it can't do a good a job as possible.

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Robert Virding  
View profile  
 More options Nov 24 2008, 2:57 pm
From: "Robert Virding" <rvird...@gmail.com>
Date: Mon, 24 Nov 2008 20:57:52 +0100
Local: Mon, Nov 24 2008 2:57 pm
Subject: Re: [erlang-questions] Binary pattern matching and optimization

2008/11/24 Vance Shipley <van...@motivity.ca>

> Robert,

> My thinking was that reading a big binary and then passing it
> with offsets to various parsing functions would be easier on
> the emulator than creating a new binary for each one.  Is it
> the case that the compiler will optimize away the creation of
> these binaries?

No, I don't think so. It optimises the following case:

foo(<< ... , Rest/binary>>, ... ) ->
    ...
    bar(Rest, ...);

The important thing is that the old reference to the binary is not used
again and can be reused by the compiler. If you have a big binary and want
to pass off chunks of it to different functions is a different problem. Then
it is probably better to pass the whole binary and an offset. However, doing
something like:

foo(Offset, Bin, ...) ->
    << _:Offset/binary, Stuff, _/binary>> = Bin,
    ...
    foo(Offset + N, Bin, ...)

and skipping over the beginning each time is most likely not the best way to
step over bits of your chunk. I would do something like:

foo(Offset, Bin, ...) ->
    << _:Offset/binary,TheGoodBit/binary >> = Bin,
    foo1(TheGoodBit, ...).

and foo1 as my first example above which picks bit off the head of the bin.

Robert

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Bob Ippolito  
View profile  
 More options Nov 24 2008, 3:23 pm
From: "Bob Ippolito" <b...@redivi.com>
Date: Mon, 24 Nov 2008 12:23:32 -0800
Local: Mon, Nov 24 2008 3:23 pm
Subject: Re: [erlang-questions] Binary pattern matching and optimization
That used to be true, but it's supposed to be faster in R12+ to do it
in the "obvious" way.

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Mercer  
View profile  
 More options Nov 24 2008, 5:04 pm
From: "David Mercer" <dmer...@gmail.com>
Date: Mon, 24 Nov 2008 16:04:26 -0600
Local: Mon, Nov 24 2008 5:04 pm
Subject: Re: [erlang-questions] Binary pattern matching and optimization

Follow-up question to the converse problem: assembling binaries.  If
binaries are best handled like lists while reading them, is it also true
that binaries are best assembled back-to-front like lists, putting the new
binary chunk onto the beginning of the binary instead of the end?

  _____  

From: erlang-questions-boun...@erlang.org
[mailto:erlang-questions-boun...@erlang.org] On Behalf Of Robert Virding
Sent: Monday, November 24, 2008 13:58
To: Vance Shipley; Robert Virding; Erlang Questions
Subject: Re: [erlang-questions] Binary pattern matching and optimization

2008/11/24 Vance Shipley <van...@motivity.ca>

Robert,

My thinking was that reading a big binary and then passing it
with offsets to various parsing functions would be easier on
the emulator than creating a new binary for each one.  Is it
the case that the compiler will optimize away the creation of
these binaries?

No, I don't think so. It optimises the following case:

foo(<< ... , Rest/binary>>, ... ) ->
    ...
    bar(Rest, ...);

The important thing is that the old reference to the binary is not used
again and can be reused by the compiler. If you have a big binary and want
to pass off chunks of it to different functions is a different problem. Then
it is probably better to pass the whole binary and an offset. However, doing
something like:

foo(Offset, Bin, ...) ->
    << _:Offset/binary, Stuff, _/binary>> = Bin,
    ...
    foo(Offset + N, Bin, ...)

and skipping over the beginning each time is most likely not the best way to
step over bits of your chunk. I would do something like:

foo(Offset, Bin, ...) ->
    << _:Offset/binary,TheGoodBit/binary >> = Bin,
    foo1(TheGoodBit, ...).

and foo1 as my first example above which picks bit off the head of the bin.

Robert

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Vance Shipley  
View profile  
 More options Nov 24 2008, 5:10 pm
From: Vance Shipley <van...@motivity.ca>
Date: Mon, 24 Nov 2008 17:10:49 -0500
Local: Mon, Nov 24 2008 5:10 pm
Subject: Re: [erlang-questions] Binary pattern matching and optimization
David,

That seems to be well documented in the efficiancy guide:

   http://erlang.org/doc/efficiency_guide/binaryhandling.html#4.2

In R12B appending to binaries is optimized.

        -Vance

On Mon, Nov 24, 2008 at 04:04:26PM -0600, David Mercer wrote:
}  Follow-up question to the converse problem: assembling binaries.  If
}  binaries are best handled like lists while reading them, is it also true
}  that binaries are best assembled back-to-front like lists, putting the new
}  binary chunk onto the beginning of the binary instead of the end?

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Mercer  
View profile  
 More options Nov 24 2008, 5:30 pm
From: "David Mercer" <dmer...@gmail.com>
Date: Mon, 24 Nov 2008 16:30:16 -0600
Local: Mon, Nov 24 2008 5:30 pm
Subject: Re: [erlang-questions] Binary pattern matching and optimization
That is very good indeed.  I kind of felt a pang of guilt in posting my
questio, because I thought I should test before asking, so I could see for
myself, but really, I ought to have looked in the docs.  Thanks, Vance.

David

_______________________________________________
erlang-questions mailing list
erlang-questi...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »