Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
fast multithreaded option to accumarray or python equivalent of bincount
76 views
Skip to first unread message
Suresh
Sep 3, 2015, 6:02:13 PM9/3/15
to
Hi,
Is there an accumarray equivalent that is multi threaded.
I have read in the matlab central that the reason accumarray is not multithreaded is because it allows custom cell functions, etc and if only the two options are allowed, then it is easy to multi thread. Is there a function already that will do something like that? I was not sure if histcounts might work.
For example, python numpy bincount equivalent which only takes index and weight as arguments.
Thanks.
Suresh
Is there an accumarray equivalent that is multi threaded.
I have read in the matlab central that the reason accumarray is not multithreaded is because it allows custom cell functions, etc and if only the two options are allowed, then it is easy to multi thread. Is there a function already that will do something like that? I was not sure if histcounts might work.
For example, python numpy bincount equivalent which only takes index and weight as arguments.
Thanks.
Suresh
Michelle Hirsch
Sep 4, 2015, 10:35:18 AM9/4/15
to
Not answering the general question, but to your specific example - the discretize function (new in 15a) looks like it might do about the same thing as bincount though it doesn't have weighting capability.
http://www.mathworks.com/help/matlab/ref/discretize.html
"Suresh" <sur...@aps.anl.gov> wrote in message <msag10$pik$1...@newscl01ah.mathworks.com>...
http://www.mathworks.com/help/matlab/ref/discretize.html
"Suresh" <sur...@aps.anl.gov> wrote in message <msag10$pik$1...@newscl01ah.mathworks.com>...
Suresh
Sep 4, 2015, 11:15:10 AM9/4/15
to
Hi Michelle,
Yes, I saw that. But that does not weight like you said and so one cannot sum like accumarray. I do not know why there cannot be a multi threaded accumarray without the other cell options.
Suresh
"Michelle Hirsch" wrote in message <msca6n$t7$1...@newscl01ah.mathworks.com>...
Yes, I saw that. But that does not weight like you said and so one cannot sum like accumarray. I do not know why there cannot be a multi threaded accumarray without the other cell options.
Suresh
"Michelle Hirsch" wrote in message <msca6n$t7$1...@newscl01ah.mathworks.com>...
Steven Lord
Sep 4, 2015, 11:20:44 AM9/4/15
to
news:msccho$5u3$1...@newscl01ah.mathworks.com...
not.
You should care about whether ACCUMARRAY is fast enough for your
application. Is it? If not, show the ACCUMARRAY call you tested that's not
fast enough, explain how fast it's working and how fast you need/want it to
work, and perhaps someone can offer advice on how to improve its
performance.
*snip*
--
Steve Lord
sl...@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com
> Hi Michelle,
>
> Yes, I saw that. But that does not weight like you said and so one cannot
> sum like accumarray. I do not know why there cannot be a multi threaded
> accumarray without the other cell options.
In my opinion, you shouldn't care whether ACCUMARRAY is multithreaded or
>
> Yes, I saw that. But that does not weight like you said and so one cannot
> sum like accumarray. I do not know why there cannot be a multi threaded
> accumarray without the other cell options.
not.
You should care about whether ACCUMARRAY is fast enough for your
application. Is it? If not, show the ACCUMARRAY call you tested that's not
fast enough, explain how fast it's working and how fast you need/want it to
work, and perhaps someone can offer advice on how to improve its
performance.
*snip*
--
Steve Lord
sl...@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com
Suresh
Sep 4, 2015, 12:31:11 PM9/4/15
to
Hi Steven,
Thank you. I agree and trust me I did not care or know about it. I have a support ticket open and it might come to you as well from your team where it seems like multiple CPU machines take 4-5x longer to do accumarray than a single CPU machine. It could be just a problem in our machines, Richa Gupta is investigating this. The code I am running is pasted below, a sample test code. I am doing accumarray on matrices of 1e4 x 1e4 or some times even bigger. I have to do this on thousands of data sets and it takes 5-10 sec to do each of this. I was thinking or hoping that it would do in 1 sec as it is a vectorized operation. I am open to any new way of doing this.
Suresh
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
C=rand(1e4,1e4);
tic;
dim=size(C,1);
[z1,z2]=meshgrid(1:dim,1:dim);
h=z1-z2;
h=triu(h); %upper and lower triangular are symmetrical, so get rid of one
g=(h~=0);
C=C.*g;%get rid of the lower triangle in the two time
hh=h(g); %get the pixel indices with non-zero t1-t2
CC=C(g); %get the corr. values with non-zero t1-t2
g2full=accumarray(hh(:),CC(:)); %sums up values of C that belong to the same pixel indices as hh
index_count=histc(h(:),min(hh):max(hh)); %count the number of pixels in each parallel to diagonal
g2full=g2full./index_count;
toc;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
"Steven Lord" <Steve...@mathworks.com> wrote in message <msccs5$6m8$1...@newscl01ah.mathworks.com>...
Thank you. I agree and trust me I did not care or know about it. I have a support ticket open and it might come to you as well from your team where it seems like multiple CPU machines take 4-5x longer to do accumarray than a single CPU machine. It could be just a problem in our machines, Richa Gupta is investigating this. The code I am running is pasted below, a sample test code. I am doing accumarray on matrices of 1e4 x 1e4 or some times even bigger. I have to do this on thousands of data sets and it takes 5-10 sec to do each of this. I was thinking or hoping that it would do in 1 sec as it is a vectorized operation. I am open to any new way of doing this.
Suresh
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
C=rand(1e4,1e4);
tic;
dim=size(C,1);
[z1,z2]=meshgrid(1:dim,1:dim);
h=z1-z2;
h=triu(h); %upper and lower triangular are symmetrical, so get rid of one
g=(h~=0);
C=C.*g;%get rid of the lower triangle in the two time
hh=h(g); %get the pixel indices with non-zero t1-t2
CC=C(g); %get the corr. values with non-zero t1-t2
g2full=accumarray(hh(:),CC(:)); %sums up values of C that belong to the same pixel indices as hh
index_count=histc(h(:),min(hh):max(hh)); %count the number of pixels in each parallel to diagonal
g2full=g2full./index_count;
toc;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
"Steven Lord" <Steve...@mathworks.com> wrote in message <msccs5$6m8$1...@newscl01ah.mathworks.com>...
Sandor Toth
Sep 9, 2015, 6:07:09 AM9/9/15
to
Hi Suresh,
I cannot give a much faster solution, but there are some obvious improvements to your code. For example the line
index_count=histc(h(:),min(hh):max(hh));
is not necessary, it just produces constants:
index_count = ((N-1):-1:1)';
This spared me half the time.
Cheers,
Sandor
"Suresh" <sur...@aps.anl.gov> wrote in message <msch07$ffp$1...@newscl01ah.mathworks.com>...
I cannot give a much faster solution, but there are some obvious improvements to your code. For example the line
index_count=histc(h(:),min(hh):max(hh));
is not necessary, it just produces constants:
index_count = ((N-1):-1:1)';
This spared me half the time.
Cheers,
Sandor
"Suresh" <sur...@aps.anl.gov> wrote in message <msch07$ffp$1...@newscl01ah.mathworks.com>...
Suresh
Sep 9, 2015, 4:56:08 PM9/9/15
to
Hi Sandor,
Thank you for the tip. Yes, that helps by half almost. I was hoping to do at least parfor on a series that I have to do. May be I will post the code here to get some help.
Suresh
"Sandor Toth" wrote in message <msp0c7$1s$1...@newscl01ah.mathworks.com>...
Thank you for the tip. Yes, that helps by half almost. I was hoping to do at least parfor on a series that I have to do. May be I will post the code here to get some help.
Suresh
"Sandor Toth" wrote in message <msp0c7$1s$1...@newscl01ah.mathworks.com>...
0 new messages