Running Chains in Parallel

[Peter Li]

For the ensemble samplers, I want to run 4 chains in parallel using something like

for i in {1..4}

do

model stuff

done

and then 

bin/print samples*.csv

But I've noticed that when I run it like this the warmup/sample times are a lot slower and more variable than if I were to run the chains sequentially [which is slower in the sense that it would take longer than running them in parallel]. 

So should I be running them in parallel or not? And if I do run them in parallel, is it ok to use the warmup/sample times or should we just not bother with using the warmup time as a very rough estimate of time to convergence?

[Michael Betancourt]

This is all internal to Stan?

I advocate saving the warmup times for analysis use but not making any claim that they’re actually representative of convergence time.

[Ben Goodrich]

On Monday, July 21, 2014 5:16:35 PM UTC-4, Peter Li wrote:

But I've noticed that when I run it like this the warmup/sample times are a lot slower and more variable than if I were to run the chains sequentially [which is slower in the sense that it would take longer than running them in parallel]. 

Unless you choke the machine, I don't see how running the chains in parallel would be any different (assuming same seeds, different chain_id, etc.)

Ben

[Peter Li]

This is done via CmdStan.

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Li]

There might be some issue with how we're calling it from the python script then [we're not using PyStan], though I think I saw similar variations when I ran it straight from command line. It might be the seeds since I don't think we set them to anything particular, but I don't see why that would give so much variation in the warmup times. 

For instance, with the chains not in parallel from python script, here are the times [first column is warmup times and second column is sampling time]

0.783678  0.885639

0.742387  0.83084

0.749547  0.86031

0.756207  0.838067

0.745775  0.842676

0.764761  0.850028

0.768055  0.839632

0.75288  0.870942

0.858897  0.944903

0.809843  0.914556

0.804687  0.851096

0.766524  0.828348

0.746574  0.857063

0.749474  0.850166

0.742087  0.830808

0.798069  0.89224

0.742933  0.823869

0.751209  0.858057

0.776687  0.84336

0.759792  0.842849

And with them in parallel: 

0.597691  1.0708

0.713444  0.665822

0.684196  0.665878

0.624916  1.04938

0.698641  0.718136

0.699391  0.833642

0.612821  0.791303

0.60912  0.668759

0.616617  0.674189

0.650921  0.704704

0.596904  0.665802

0.624565  0.714502

0.614049  0.728062

0.588368  0.816542

0.615376  0.76874

0.597937  0.810229

0.652422  0.666215

0.689229  0.809552

0.685087  0.778758

0.618826  0.684638

0.608461  0.663496

0.618216  0.714233

0.614835  0.771172

0.648977  0.834636

0.641712  0.711322

0.610368  0.693231

0.763696  0.67053

0.662149  0.69607

0.592672  0.663624

0.624074  0.757671

0.639064  0.666944

0.66405  0.720346

0.62136  0.670739

0.825302  0.743411

0.743921  0.803981

0.609156  0.659802

0.710106  0.656647

0.73273  0.66441

0.623797  0.69142

0.605632  0.669921

0.603298  0.781241

0.615229  0.773051

0.622186  0.665223

0.624697  0.661309

0.785284  0.671883

0.612513  0.657957

0.59053  0.929375

0.809863  0.911549

0.598374  0.670436

0.668019  0.679614

0.628188  0.687355

0.662878  0.672412

0.733508  0.753928

0.645661  0.67135

0.61211  0.660012

0.585279  0.656249

0.59277  0.868712

0.599442  0.670595

0.691125  0.672769

0.709557  0.864766

0.778935  0.676523

0.602996  0.669012

0.78345  0.727614

0.594502  0.707861

1.01573  0.819883

0.861876  0.909337

0.692883  0.668624

0.603429  0.672616

0.605471  0.662827

0.693955  0.692049

0.623522  0.695205

0.636278  0.684141

0.874719  0.680954

0.875102  0.670081

0.607593  0.659166

0.602359  0.706636

0.589482  0.853998

0.778426  0.770332

0.602997  0.969123

0.777676  0.667156

0.952957  0.674299

0.689097  0.733402

0.896902  0.823293

0.637791  0.90986

0.634335  0.845136

0.600345  0.658835

0.610497  0.660167

0.601858  0.862338

0.598412  0.653359

0.599361  0.780881

1.23752  0.684686

1.14054  0.65795

0.596092  0.653298

0.779566  0.841736

0.854422  0.80732

0.59349  0.662399

0.789472  0.924911

0.611082  0.666006

0.766795  0.932733

0.61117  0.674341

0.619944  0.668492

0.589011  0.661336

0.600259  0.666862

0.599341  0.657231

0.601482  1.01739

0.615431  0.656904

0.602222  0.667574

0.606681  1.02465

1.08178  0.739739

0.593124  0.656235

1.11174  0.739863

0.610115  0.66125

0.596031  0.66485

0.609362  0.68656

0.652654  0.683552

0.658863  0.673524

0.845751  0.662661

0.629227  0.65739

0.818961  0.674049

0.587512  0.658294

0.633821  0.652965

0.592913  0.793561

0.607114  0.797768

0.600233  0.664658

0.650336  0.659084

0.591783  0.654945

0.628223  0.697549

0.60973  0.694768

0.608392  0.65488

0.596204  0.652189

0.602183  0.659904

0.605549  0.657289

0.958088  0.839779

0.602757  0.671771

0.700633  0.65799

0.728124  0.958023

0.63445  0.677797

0.603785  0.663802

0.594846  0.661701

0.625397  0.670384

0.960144  0.820582

0.960509  0.831616

0.598857  0.656216

0.595866  0.657784

0.610658  0.668074

0.606427  0.666942

0.973465  0.831595

1.00025  0.807463

0.599415  0.656312

0.641213  0.664504

0.64714  0.672113

0.618192  0.673983

0.648594  0.684856

0.598659  0.673714

0.659204  0.669133

0.595698  0.655237

0.594727  0.860245

1.02491  0.712247

0.613171  0.658791

0.872942  0.668935

0.599237  0.772748

0.785144  0.768729

0.62866  0.66597

0.760299  0.673786

0.592204  0.65871

1.22186  0.686413

0.590571  0.658009

1.21272  0.694511

0.687343  0.794538

0.598759  0.707302

0.649764  0.751757

0.773405  0.670742

0.618047  0.81328

0.639972  0.876423

0.632165  0.719442

0.704336  0.671975

0.613949  0.764183

0.597424  0.666084

0.590628  0.648594

0.613488  0.779567

0.711157  0.744664

0.72839  0.737017

0.605812  0.674695

0.599369  0.655844

0.606356  0.997305

0.615591  0.65638

0.618006  1.00055

0.60058  0.664266

0.59125  0.784964

0.781347  0.678257

0.596865  0.724429

0.77456  0.777435

0.589736  0.658607

0.60705  0.717325

0.622246  0.71582

0.592864  0.655241

0.614851  0.669302

0.628067  0.655383

0.596526  0.665958

0.595481  0.667608

0.708863  0.669689

0.660463  0.667472

0.750974  0.698039

0.817925  0.666944

0.593328  0.655391

0.632381  0.702547

0.710047  0.663558

0.711189  0.682252

0.607239  0.658848

0.612053  0.677998

0.685533  0.669617

0.665863  0.663789

0.603736  0.671723

0.610035  0.675279

0.609933  0.677062

0.598557  0.66724

1.10999  0.740121

0.639911  0.661064

1.00876  0.810529

0.59053  0.652622

0.63558  0.677229

0.627901  0.655001

0.593647  0.654607

0.589375  0.653292

0.609691  0.701528

0.631859  0.666746

0.611215  0.665065

0.620574  0.684429

0.774571  0.823635

0.802912  0.815041

0.596559  0.657887

0.595917  0.657523

0.787945  0.664074

0.659986  0.667999

0.711969  0.678659

0.614391  0.673891

1.1847  0.711173

0.695763  0.95624

0.592487  0.656563

0.901201  0.675858

0.615547  0.664736

0.601913  0.667781

0.745583  0.719924

0.757982  0.709661

0.622386  0.667305

0.592905  0.665389

0.62422  0.869335

0.608137  0.874809

0.621123  0.727223

0.603422  0.680443

0.608317  0.702648

0.615453  0.667633

0.593947  0.657415

0.954298  0.828428

0.985649  0.812605

0.643591  0.662419

0.598637  0.668022

0.643878  0.800507

0.600019  0.991304

0.662608  0.844711

0.794482  0.901382

0.596977  0.66117

0.591429  0.656515

0.790452  0.909321

0.595494  0.658114

0.733337  0.677136

0.741212  0.676084

0.595706  0.668568

0.809957  0.917494

0.625624  0.660528

0.800091  0.666099

0.60003  0.960977

0.917909  0.713679

0.618637  0.66847

0.602026  0.66773

0.87578  0.728431

0.592976  0.801842

0.837252  0.925287

0.712949  0.846103

0.634861  0.659033

0.616906  0.669205

0.597953  0.658793

0.589715  0.654288

0.603331  0.656785

0.645041  0.762934

0.595196  0.656626

0.602207  0.657089

0.64677  0.756652

0.602257  0.840552

0.630707  0.668852

0.631629  0.666346

0.64281  0.837776

0.641143  0.673755

0.624811  0.663557

0.592387  0.659311

0.599759  0.658881

0.6968  0.980756

0.605442  0.674602

0.744516  0.947595

0.607208  0.662356

0.789474  0.681735

0.63246  0.664448

0.696967  0.657024

0.85315  0.664481

1.08841  0.756464

1.00145  0.807883

0.600967  0.658107

0.657877  0.66506

0.761784  0.675274

0.590239  0.657353

0.727792  0.672994

0.595921  0.668406

0.652746  0.682286

0.641443  0.707461

0.629631  0.679489

0.602648  0.721459

0.599644  0.666789

0.710303  0.670753

0.711868  0.671049

0.592283  0.657479

0.598826  0.682675

0.615352  0.697076

0.605689  0.668297

0.619765  0.684671

0.878268  0.873647

0.812876  0.900283

0.627008  0.664704

0.597485  0.661482

0.59893  0.661244

0.6059  0.677419

0.642033  0.673918

0.633658  0.672493

0.853182  0.834638

0.601008  0.659085

0.617641  0.833684

0.840805  0.661194

0.587907  0.6541

0.63022  0.65922

0.619255  0.680468

0.639753  0.694394

1.02484  0.680958

0.592447  0.664491

0.607311  0.66132

1.03517  0.679638

0.909275  0.699125

0.909331  0.709788

0.600435  0.748144

0.592388  0.658882

0.732243  0.663285

0.606043  0.8116

0.612194  0.822774

0.72879  0.6657

0.637603  0.655182

0.590042  0.921684

0.634462  0.920587

0.606313  0.653157

0.704819  0.901872

0.635001  0.934363

0.589439  0.650566

0.608092  0.67416

0.719159  0.662617

0.59961  0.672514

0.596082  0.659588

0.717744  0.667679

0.627443  0.947374

0.666985  0.943325

0.620022  0.659583

0.592315  0.65762

0.589077  0.655423

0.593529  0.660252

0.641399  0.854672

0.624281  0.857168

0.594556  0.664021

0.730394  0.660446

0.792796  0.919272

1.03863  0.805379

0.905495  0.768707

0.899088  0.708092

0.67898  0.716576

0.653151  0.71194

0.733053  0.676923

0.609823  0.655153

0.613553  0.74747

0.815358  0.683212

0.635173  0.877788

0.76517  0.780158

0.663897  0.702008

0.706606  0.740431

0.839285  0.674683

0.775595  0.794336

0.786837  0.713167

0.855678  0.748269

0.811681  0.814562

0.7613  0.85795

0.6667  0.768252

0.769685  0.797838

0.701163  0.718342

0.671282  0.811985

0.721736  0.910261

0.713435  0.889012

[Ben Goodrich]

On Monday, July 21, 2014 11:03:25 PM UTC-4, Peter Li wrote:

There might be some issue with how we're calling it from the python script then [we're not using PyStan], though I think I saw similar variations when I ran it straight from command line. It might be the seeds since I don't think we set them to anything particular, but I don't see why that would give so much variation in the warmup times. 

Oh, with different seeds or other parameters, there have always been non-trivial runtime differences.

Ben

[Peter Li]

So if I run the model for 100 runs, 4 chains each, the 4 chains in each run should have the same seed, but from run to run they should have different seeds?

--

[Ben Goodrich]

On Monday, July 21, 2014 11:56:31 PM UTC-4, Peter Li wrote:

So if I run the model for 100 runs, 4 chains each, the 4 chains in each run should have the same seed, but from run to run they should have different seeds?

Yeah, just make sure you specify the chain ID.

Ben

[Michael Betancourt]

The chain id modifies the RNG so that parallel chains start with different seeds.  So when running in parallel this is okay,

seed=1 id=1

seed=1 id=2

seed=1 id=3

seed=1 id=4

but this is not,

seed=1 id=1

seed=1 id=1

seed=1 id=1

seed=1 id=1.

The former should be equivalent to something like,

seed=1

seed=2

seed=3

seed=4

and if it’s not then there’s something going wrong with the striding and the RNG.

[Bob Carpenter]

It's easy to choke a machine when you're doing unbuffered
I/O, because it's basically sequential.

Or when you run memory-intensive processes, because
the memory bus capacities aren't nearly as high as the CPU capacities.

And then there's paging issues. If you hit four processes and
only have four cores, you may run into process-switching issues
with programs other than what you're running.

Safest thing to do is run one process w/o anything else CPU or
memory or network-intensive running (no e-mail, no browsers, etc.)

We really need to get those servers we can dedicate to testing!

- Bob