wal directory increasing in size

[Alex Ilievski]

Hi

We have noticed over time the wal directory is increasing in size and never decreasing even after the jobs are being deleted.

Why is this happening and how can we decrease the space being used?

Cheers

Alex

[Keith Rarick]

On Wed, Sep 10, 2014 at 10:44 PM, Alex Ilievski <al...@myguestlist.com.au> wrote:
> We have noticed over time the wal directory is increasing in size and never
> decreasing even after the jobs are being deleted.

What version of beanstalkd are you using?

[Alex Ilievski]

Version 1.9

- Alex

[Keith Rarick]

How many files are in this directory?
Can you send the output of 'ls -l'?

[Alex Ilievski]

280 files taking up 2.8GB at the moment. We recently moved the beanstalkd server to a new host. On the old host it was at ~25GB.

Output of ls -l

total 2836480

-r-------- 1 c_btalk c_btalk 10485748 Sep 10 16:42 binlog.100

-r-------- 1 c_btalk c_btalk 10485688 Sep 10 16:53 binlog.101

-r-------- 1 c_btalk c_btalk 10485533 Sep 10 17:03 binlog.102

-r-------- 1 c_btalk c_btalk 10485731 Sep 10 17:14 binlog.103

-r-------- 1 c_btalk c_btalk 10485748 Sep 10 17:26 binlog.104

-r-------- 1 c_btalk c_btalk 10485521 Sep 10 17:36 binlog.105

-r-------- 1 c_btalk c_btalk 10485740 Sep 10 17:38 binlog.106

-r-------- 1 c_btalk c_btalk 10485737 Sep 10 17:48 binlog.107

-r-------- 1 c_btalk c_btalk 10485367 Sep 10 17:55 binlog.108

-r-------- 1 c_btalk c_btalk 10485645 Sep 10 18:01 binlog.109

-r-------- 1 c_btalk c_btalk 10485706 Sep 10 18:07 binlog.110

-r-------- 1 c_btalk c_btalk 10485754 Sep 10 18:19 binlog.111

-r-------- 1 c_btalk c_btalk 10485683 Sep 10 18:32 binlog.112

-r-------- 1 c_btalk c_btalk 10485693 Sep 10 18:43 binlog.113

-r-------- 1 c_btalk c_btalk 10485704 Sep 10 18:54 binlog.114

-r-------- 1 c_btalk c_btalk 10485690 Sep 10 19:07 binlog.115

-r-------- 1 c_btalk c_btalk 10485712 Sep 10 19:19 binlog.116

-r-------- 1 c_btalk c_btalk 10485699 Sep 10 19:32 binlog.117

-r-------- 1 c_btalk c_btalk 10485717 Sep 10 19:44 binlog.118

-r-------- 1 c_btalk c_btalk 10485723 Sep 10 19:55 binlog.119

-r-------- 1 c_btalk c_btalk 10485710 Sep 10 20:06 binlog.120

-r-------- 1 c_btalk c_btalk 10485713 Sep 10 20:17 binlog.121

-r-------- 1 c_btalk c_btalk 10485718 Sep 10 20:30 binlog.122

-r-------- 1 c_btalk c_btalk 10485753 Sep 10 20:42 binlog.123

-r-------- 1 c_btalk c_btalk 10485757 Sep 10 20:54 binlog.124

-r-------- 1 c_btalk c_btalk 10485711 Sep 10 21:07 binlog.125

-r-------- 1 c_btalk c_btalk 10485686 Sep 10 21:20 binlog.126

-r-------- 1 c_btalk c_btalk 10485745 Sep 10 21:34 binlog.127

-r-------- 1 c_btalk c_btalk 10485727 Sep 10 21:47 binlog.128

-r-------- 1 c_btalk c_btalk 10485696 Sep 10 21:59 binlog.129

-r-------- 1 c_btalk c_btalk 10485715 Sep 10 22:11 binlog.130

-r-------- 1 c_btalk c_btalk 10485691 Sep 10 22:23 binlog.131

-r-------- 1 c_btalk c_btalk 10485742 Sep 10 22:36 binlog.132

-r-------- 1 c_btalk c_btalk 10485678 Sep 10 22:45 binlog.133

-r-------- 1 c_btalk c_btalk 10485728 Sep 10 22:53 binlog.134

-r-------- 1 c_btalk c_btalk 10485760 Sep 10 23:02 binlog.135

-r-------- 1 c_btalk c_btalk 10485758 Sep 10 23:09 binlog.136

-r-------- 1 c_btalk c_btalk 10485722 Sep 10 23:19 binlog.137

-r-------- 1 c_btalk c_btalk 10485717 Sep 10 23:26 binlog.138

-r-------- 1 c_btalk c_btalk 10485618 Sep 10 23:31 binlog.139

-r-------- 1 c_btalk c_btalk 10485735 Sep 10 23:38 binlog.140

-r-------- 1 c_btalk c_btalk 10485722 Sep 10 23:46 binlog.141

-r-------- 1 c_btalk c_btalk 10485741 Sep 10 23:53 binlog.142

-r-------- 1 c_btalk c_btalk 10485699 Sep 11 00:01 binlog.143

-r-------- 1 c_btalk c_btalk 10485760 Sep 11 00:08 binlog.144

-r-------- 1 c_btalk c_btalk 10485693 Sep 11 00:16 binlog.145

-r-------- 1 c_btalk c_btalk 10485692 Sep 11 00:23 binlog.146

-r-------- 1 c_btalk c_btalk 10485718 Sep 11 00:31 binlog.147

-r-------- 1 c_btalk c_btalk 10485740 Sep 11 00:38 binlog.148

-r-------- 1 c_btalk c_btalk 10485733 Sep 11 00:46 binlog.149

-r-------- 1 c_btalk c_btalk 10485727 Sep 11 00:53 binlog.150

-r-------- 1 c_btalk c_btalk 10485684 Sep 11 01:02 binlog.151

-r-------- 1 c_btalk c_btalk 10485755 Sep 11 01:08 binlog.152

-r-------- 1 c_btalk c_btalk 10485680 Sep 11 01:16 binlog.153

-r-------- 1 c_btalk c_btalk 10485740 Sep 11 01:23 binlog.154

-r-------- 1 c_btalk c_btalk 10485699 Sep 11 01:30 binlog.155

-r-------- 1 c_btalk c_btalk 10485685 Sep 11 01:36 binlog.156

-r-------- 1 c_btalk c_btalk 10485723 Sep 11 01:41 binlog.157

-r-------- 1 c_btalk c_btalk 10485683 Sep 11 01:47 binlog.158

-r-------- 1 c_btalk c_btalk 10485726 Sep 11 01:52 binlog.159

-r-------- 1 c_btalk c_btalk 10485718 Sep 11 01:58 binlog.160

-r-------- 1 c_btalk c_btalk 10485580 Sep 11 02:02 binlog.161

-r-------- 1 c_btalk c_btalk 10485728 Sep 11 02:07 binlog.162

-r-------- 1 c_btalk c_btalk 10485726 Sep 11 02:12 binlog.163

-r-------- 1 c_btalk c_btalk 10485685 Sep 11 02:17 binlog.164

-r-------- 1 c_btalk c_btalk 10485707 Sep 11 02:22 binlog.165

-r-------- 1 c_btalk c_btalk 10485494 Sep 11 02:27 binlog.166

-r-------- 1 c_btalk c_btalk 10485751 Sep 11 02:32 binlog.167

-r-------- 1 c_btalk c_btalk 10485682 Sep 11 02:37 binlog.168

-r-------- 1 c_btalk c_btalk 10485745 Sep 11 02:43 binlog.169

-r-------- 1 c_btalk c_btalk 10485727 Sep 11 02:48 binlog.170

-r-------- 1 c_btalk c_btalk 10485726 Sep 11 02:54 binlog.171

-r-------- 1 c_btalk c_btalk 10485743 Sep 11 03:00 binlog.172

-r-------- 1 c_btalk c_btalk 10485756 Sep 11 03:05 binlog.173

-r-------- 1 c_btalk c_btalk 10485740 Sep 11 03:11 binlog.174

-r-------- 1 c_btalk c_btalk 10485748 Sep 11 03:17 binlog.175

-r-------- 1 c_btalk c_btalk 10485712 Sep 11 03:22 binlog.176

-r-------- 1 c_btalk c_btalk 10485729 Sep 11 03:26 binlog.177

-r-------- 1 c_btalk c_btalk 10485759 Sep 11 03:32 binlog.178

-r-------- 1 c_btalk c_btalk 10485744 Sep 11 03:37 binlog.179

-r-------- 1 c_btalk c_btalk 10485680 Sep 11 03:42 binlog.180

-r-------- 1 c_btalk c_btalk 10485711 Sep 11 03:48 binlog.181

-r-------- 1 c_btalk c_btalk 10485720 Sep 11 03:54 binlog.182

-r-------- 1 c_btalk c_btalk 10485678 Sep 11 03:59 binlog.183

-r-------- 1 c_btalk c_btalk 10485691 Sep 11 04:04 binlog.184

-r-------- 1 c_btalk c_btalk 10485757 Sep 11 04:10 binlog.185

-r-------- 1 c_btalk c_btalk 10485685 Sep 11 04:16 binlog.186

-r-------- 1 c_btalk c_btalk 10485701 Sep 11 04:21 binlog.187

-r-------- 1 c_btalk c_btalk 10485754 Sep 11 04:27 binlog.188

-r-------- 1 c_btalk c_btalk 10485702 Sep 11 04:33 binlog.189

-r-------- 1 c_btalk c_btalk 10485538 Sep 11 04:35 binlog.190

-r-------- 1 c_btalk c_btalk 10485688 Sep 11 04:38 binlog.191

-r-------- 1 c_btalk c_btalk 10485442 Sep 11 04:40 binlog.192

-r-------- 1 c_btalk c_btalk 10485730 Sep 11 04:43 binlog.193

-r-------- 1 c_btalk c_btalk 10485733 Sep 11 04:48 binlog.194

-r-------- 1 c_btalk c_btalk 10485756 Sep 11 04:53 binlog.195

-r-------- 1 c_btalk c_btalk 10485698 Sep 11 04:57 binlog.196

-r-------- 1 c_btalk c_btalk 10485709 Sep 11 05:02 binlog.197

-r-------- 1 c_btalk c_btalk 10485692 Sep 11 05:07 binlog.198

-r-------- 1 c_btalk c_btalk 10485742 Sep 11 05:13 binlog.199

-r-------- 1 c_btalk c_btalk 10485729 Sep 11 05:18 binlog.200

-r-------- 1 c_btalk c_btalk 10485728 Sep 11 05:23 binlog.201

-r-------- 1 c_btalk c_btalk 10485721 Sep 11 05:29 binlog.202

-r-------- 1 c_btalk c_btalk 10485708 Sep 11 05:34 binlog.203

-r-------- 1 c_btalk c_btalk 10485743 Sep 11 05:39 binlog.204

-r-------- 1 c_btalk c_btalk 10485694 Sep 11 05:45 binlog.205

-r-------- 1 c_btalk c_btalk 10485703 Sep 11 05:50 binlog.206

-r-------- 1 c_btalk c_btalk 10485697 Sep 11 05:55 binlog.207

-r-------- 1 c_btalk c_btalk 10485755 Sep 11 06:01 binlog.208

-r-------- 1 c_btalk c_btalk 10485752 Sep 11 06:06 binlog.209

-r-------- 1 c_btalk c_btalk 10485700 Sep 11 06:11 binlog.210

-r-------- 1 c_btalk c_btalk 10485757 Sep 11 06:17 binlog.211

-r-------- 1 c_btalk c_btalk 10485730 Sep 11 06:22 binlog.212

-r-------- 1 c_btalk c_btalk 10485754 Sep 11 06:27 binlog.213

-r-------- 1 c_btalk c_btalk 10485730 Sep 11 06:33 binlog.214

-r-------- 1 c_btalk c_btalk 10485697 Sep 11 06:38 binlog.215

-r-------- 1 c_btalk c_btalk 10485693 Sep 11 06:43 binlog.216

-r-------- 1 c_btalk c_btalk 10485710 Sep 11 06:49 binlog.217

-r-------- 1 c_btalk c_btalk 10485687 Sep 11 06:54 binlog.218

-r-------- 1 c_btalk c_btalk 10485759 Sep 11 06:59 binlog.219

-r-------- 1 c_btalk c_btalk 10485682 Sep 11 07:04 binlog.220

-r-------- 1 c_btalk c_btalk 10485704 Sep 11 07:08 binlog.221

-r-------- 1 c_btalk c_btalk 10485709 Sep 11 07:13 binlog.222

-r-------- 1 c_btalk c_btalk 10485692 Sep 11 07:18 binlog.223

-r-------- 1 c_btalk c_btalk 10485730 Sep 11 07:23 binlog.224

-r-------- 1 c_btalk c_btalk 10485739 Sep 11 07:28 binlog.225

-r-------- 1 c_btalk c_btalk 10485715 Sep 11 07:34 binlog.226

-r-------- 1 c_btalk c_btalk 10485497 Sep 11 07:38 binlog.227

-r-------- 1 c_btalk c_btalk 10485708 Sep 11 07:43 binlog.228

-r-------- 1 c_btalk c_btalk 10485518 Sep 11 07:44 binlog.229

-r-------- 1 c_btalk c_btalk 10485398 Sep 11 07:46 binlog.230

-r-------- 1 c_btalk c_btalk 10485703 Sep 11 07:48 binlog.231

-r-------- 1 c_btalk c_btalk 10485726 Sep 11 07:50 binlog.232

-r-------- 1 c_btalk c_btalk 10485710 Sep 11 07:55 binlog.233

-r-------- 1 c_btalk c_btalk 10485759 Sep 11 08:01 binlog.234

-r-------- 1 c_btalk c_btalk 10485696 Sep 11 08:06 binlog.235

-r-------- 1 c_btalk c_btalk 10485742 Sep 11 08:09 binlog.236

-r-------- 1 c_btalk c_btalk 10485732 Sep 11 08:14 binlog.237

-r-------- 1 c_btalk c_btalk 10485715 Sep 11 08:19 binlog.238

-r-------- 1 c_btalk c_btalk 10485744 Sep 11 08:24 binlog.239

-r-------- 1 c_btalk c_btalk 10485678 Sep 11 08:30 binlog.240

-r-------- 1 c_btalk c_btalk 10485678 Sep 11 08:35 binlog.241

-r-------- 1 c_btalk c_btalk 10485716 Sep 11 08:39 binlog.242

-r-------- 1 c_btalk c_btalk 10485696 Sep 11 08:45 binlog.243

-r-------- 1 c_btalk c_btalk 10485682 Sep 11 08:50 binlog.244

-r-------- 1 c_btalk c_btalk 10485745 Sep 11 08:55 binlog.245

-r-------- 1 c_btalk c_btalk 10485739 Sep 11 09:01 binlog.246

-r-------- 1 c_btalk c_btalk 10485740 Sep 11 09:06 binlog.247

-r-------- 1 c_btalk c_btalk 10485695 Sep 11 09:09 binlog.248

-r-------- 1 c_btalk c_btalk 10485679 Sep 11 09:11 binlog.249

-r-------- 1 c_btalk c_btalk 10485706 Sep 11 09:14 binlog.250

-r-------- 1 c_btalk c_btalk 10485684 Sep 11 09:19 binlog.251

-r-------- 1 c_btalk c_btalk 10485654 Sep 11 09:23 binlog.252

-r-------- 1 c_btalk c_btalk 10485735 Sep 11 09:27 binlog.253

-r-------- 1 c_btalk c_btalk 10485686 Sep 11 09:32 binlog.254

-r-------- 1 c_btalk c_btalk 10485729 Sep 11 09:36 binlog.255

-r-------- 1 c_btalk c_btalk 10485692 Sep 11 09:41 binlog.256

-r-------- 1 c_btalk c_btalk 10485684 Sep 11 09:46 binlog.257

-r-------- 1 c_btalk c_btalk 10485713 Sep 11 09:50 binlog.258

-r-------- 1 c_btalk c_btalk 10485689 Sep 11 09:55 binlog.259

-r-------- 1 c_btalk c_btalk 10485707 Sep 11 10:01 binlog.260

-r-------- 1 c_btalk c_btalk 10485751 Sep 11 10:06 binlog.261

-r-------- 1 c_btalk c_btalk 10485714 Sep 11 10:09 binlog.262

-r-------- 1 c_btalk c_btalk 10485696 Sep 11 10:13 binlog.263

-r-------- 1 c_btalk c_btalk 10485681 Sep 11 10:17 binlog.264

-r-------- 1 c_btalk c_btalk 10485729 Sep 11 10:19 binlog.265

-r-------- 1 c_btalk c_btalk 10485729 Sep 11 10:23 binlog.266

-r-------- 1 c_btalk c_btalk 10485755 Sep 11 10:26 binlog.267

-r-------- 1 c_btalk c_btalk 10485482 Sep 11 10:30 binlog.268

-r-------- 1 c_btalk c_btalk 10485427 Sep 11 10:32 binlog.269

-r-------- 1 c_btalk c_btalk 10485759 Sep 11 10:35 binlog.270

-r-------- 1 c_btalk c_btalk 10485730 Sep 11 10:39 binlog.271

-r-------- 1 c_btalk c_btalk 10485694 Sep 11 10:43 binlog.272

-r-------- 1 c_btalk c_btalk 10485685 Sep 11 10:46 binlog.273

-r-------- 1 c_btalk c_btalk 10485730 Sep 11 10:49 binlog.274

-r-------- 1 c_btalk c_btalk 10485714 Sep 11 10:51 binlog.275

-r-------- 1 c_btalk c_btalk 10485653 Sep 11 10:53 binlog.276

-r-------- 1 c_btalk c_btalk 10485747 Sep 11 10:55 binlog.277

-r-------- 1 c_btalk c_btalk 10485693 Sep 11 10:58 binlog.278

-r-------- 1 c_btalk c_btalk 10485687 Sep 11 11:01 binlog.279

-r-------- 1 c_btalk c_btalk 10485734 Sep 11 11:03 binlog.280

-r-------- 1 c_btalk c_btalk 10485688 Sep 11 11:05 binlog.281

-r-------- 1 c_btalk c_btalk 10485725 Sep 11 11:07 binlog.282

-r-------- 1 c_btalk c_btalk 10485722 Sep 11 11:10 binlog.283

-r-------- 1 c_btalk c_btalk 10485747 Sep 11 11:13 binlog.284

-r-------- 1 c_btalk c_btalk 10485740 Sep 11 11:16 binlog.285

-r-------- 1 c_btalk c_btalk 10485723 Sep 11 11:19 binlog.286

-r-------- 1 c_btalk c_btalk 10485689 Sep 11 11:21 binlog.287

-r-------- 1 c_btalk c_btalk 10485696 Sep 11 11:24 binlog.288

-r-------- 1 c_btalk c_btalk 10485749 Sep 11 11:27 binlog.289

-r-------- 1 c_btalk c_btalk 10485729 Sep 11 11:30 binlog.290

-r-------- 1 c_btalk c_btalk 10485757 Sep 11 11:33 binlog.291

-r-------- 1 c_btalk c_btalk 10485609 Sep 11 11:35 binlog.292

-r-------- 1 c_btalk c_btalk 10485575 Sep 11 11:38 binlog.293

-r-------- 1 c_btalk c_btalk 10485739 Sep 11 11:40 binlog.294

-r-------- 1 c_btalk c_btalk 10485731 Sep 11 11:43 binlog.295

-r-------- 1 c_btalk c_btalk 10485706 Sep 11 11:46 binlog.296

-r-------- 1 c_btalk c_btalk 10485717 Sep 11 11:49 binlog.297

-r-------- 1 c_btalk c_btalk 10485420 Sep 11 11:51 binlog.298

-r-------- 1 c_btalk c_btalk 10485737 Sep 11 11:54 binlog.299

-r-------- 1 c_btalk c_btalk 10485686 Sep 11 11:57 binlog.300

-r-------- 1 c_btalk c_btalk 10485609 Sep 11 12:00 binlog.301

-r-------- 1 c_btalk c_btalk 10485411 Sep 11 12:01 binlog.302

-r-------- 1 c_btalk c_btalk 10485732 Sep 11 12:03 binlog.303

-r-------- 1 c_btalk c_btalk 10485728 Sep 11 12:06 binlog.304

-r-------- 1 c_btalk c_btalk 10485726 Sep 11 12:09 binlog.305

-r-------- 1 c_btalk c_btalk 10485741 Sep 11 12:13 binlog.306

-r-------- 1 c_btalk c_btalk 10485700 Sep 11 12:17 binlog.307

-r-------- 1 c_btalk c_btalk 10485714 Sep 11 12:21 binlog.308

-r-------- 1 c_btalk c_btalk 10485743 Sep 11 12:24 binlog.309

-r-------- 1 c_btalk c_btalk 10485735 Sep 11 12:27 binlog.310

-r-------- 1 c_btalk c_btalk 10485758 Sep 11 12:31 binlog.311

-r-------- 1 c_btalk c_btalk 10485686 Sep 11 12:35 binlog.312

-r-------- 1 c_btalk c_btalk 10485728 Sep 11 12:39 binlog.313

-r-------- 1 c_btalk c_btalk 10485687 Sep 11 12:43 binlog.314

-r-------- 1 c_btalk c_btalk 10485739 Sep 11 12:47 binlog.315

-r-------- 1 c_btalk c_btalk 10485695 Sep 11 12:51 binlog.316

-r-------- 1 c_btalk c_btalk 10485737 Sep 11 12:54 binlog.317

-r-------- 1 c_btalk c_btalk 10485690 Sep 11 12:59 binlog.318

-r-------- 1 c_btalk c_btalk 10485573 Sep 11 13:02 binlog.319

-r-------- 1 c_btalk c_btalk 10485507 Sep 11 13:05 binlog.320

-r-------- 1 c_btalk c_btalk 10485346 Sep 11 13:07 binlog.321

-r-------- 1 c_btalk c_btalk 10485359 Sep 11 13:09 binlog.322

-r-------- 1 c_btalk c_btalk 10485742 Sep 11 13:11 binlog.323

-r-------- 1 c_btalk c_btalk 10485682 Sep 11 13:13 binlog.324

-r-------- 1 c_btalk c_btalk 10485697 Sep 11 13:17 binlog.325

-r-------- 1 c_btalk c_btalk 10485705 Sep 11 13:21 binlog.326

-r-------- 1 c_btalk c_btalk 10485679 Sep 11 13:24 binlog.327

-r-------- 1 c_btalk c_btalk 10485696 Sep 11 13:28 binlog.328

-r-------- 1 c_btalk c_btalk 10485698 Sep 11 13:31 binlog.329

-r-------- 1 c_btalk c_btalk 10485760 Sep 11 13:34 binlog.330

-r-------- 1 c_btalk c_btalk 10485747 Sep 11 13:38 binlog.331

-r-------- 1 c_btalk c_btalk 10485759 Sep 11 13:42 binlog.332

-r-------- 1 c_btalk c_btalk 10485750 Sep 11 13:47 binlog.333

-r-------- 1 c_btalk c_btalk 10485755 Sep 11 13:51 binlog.334

-r-------- 1 c_btalk c_btalk 10485721 Sep 11 13:55 binlog.335

-r-------- 1 c_btalk c_btalk 10485739 Sep 11 13:59 binlog.336

-r-------- 1 c_btalk c_btalk 10485677 Sep 11 14:04 binlog.337

-r-------- 1 c_btalk c_btalk 10485704 Sep 11 14:08 binlog.338

-r-------- 1 c_btalk c_btalk 10485755 Sep 11 14:12 binlog.339

-r-------- 1 c_btalk c_btalk 10485717 Sep 11 14:17 binlog.340

-r-------- 1 c_btalk c_btalk 10485694 Sep 11 14:21 binlog.341

-r-------- 1 c_btalk c_btalk 10485707 Sep 11 14:25 binlog.342

-r-------- 1 c_btalk c_btalk 10485748 Sep 11 14:30 binlog.343

-r-------- 1 c_btalk c_btalk 10485720 Sep 11 14:34 binlog.344

-r-------- 1 c_btalk c_btalk 10485716 Sep 11 14:38 binlog.345

-r-------- 1 c_btalk c_btalk 10485677 Sep 11 14:42 binlog.346

-r-------- 1 c_btalk c_btalk 10485754 Sep 11 14:47 binlog.347

-r-------- 1 c_btalk c_btalk 10485679 Sep 11 14:51 binlog.348

-r-------- 1 c_btalk c_btalk 10485760 Sep 11 14:55 binlog.349

-r-------- 1 c_btalk c_btalk 10485715 Sep 11 15:00 binlog.350

-r-------- 1 c_btalk c_btalk 10485702 Sep 11 15:05 binlog.351

-r-------- 1 c_btalk c_btalk 10485406 Sep 11 15:07 binlog.352

-r-------- 1 c_btalk c_btalk 10485743 Sep 11 15:10 binlog.353

-r-------- 1 c_btalk c_btalk 10485733 Sep 11 15:14 binlog.354

-r-------- 1 c_btalk c_btalk 10485703 Sep 11 15:19 binlog.355

-r-------- 1 c_btalk c_btalk 10485760 Sep 11 15:23 binlog.356

-r-------- 1 c_btalk c_btalk 10485708 Sep 11 15:27 binlog.357

-r-------- 1 c_btalk c_btalk 10485686 Sep 11 15:32 binlog.358

-r-------- 1 c_btalk c_btalk 10485736 Sep 11 15:37 binlog.359

-r-------- 1 c_btalk c_btalk 10485744 Sep 11 15:40 binlog.360

-r-------- 1 c_btalk c_btalk 10485725 Sep 11 15:43 binlog.361

-r-------- 1 c_btalk c_btalk 10485716 Sep 11 15:48 binlog.362

-r-------- 1 c_btalk c_btalk 10485760 Sep 11 15:51 binlog.363

-r-------- 1 c_btalk c_btalk 10485760 Sep 11 15:44 binlog.364

-r-------- 1 c_btalk c_btalk 10485760 Sep 11 15:49 binlog.365

-r-------- 1 c_btalk c_btalk 10485479 Sep 10 16:03 binlog.89

-r-------- 1 c_btalk c_btalk 10485361 Sep 10 16:04 binlog.90

-r-------- 1 c_btalk c_btalk 10485523 Sep 10 16:05 binlog.91

-r-------- 1 c_btalk c_btalk 10485712 Sep 10 16:05 binlog.92

-r-------- 1 c_btalk c_btalk 10485456 Sep 10 16:06 binlog.93

-r-------- 1 c_btalk c_btalk 10485458 Sep 10 16:07 binlog.94

-r-------- 1 c_btalk c_btalk 10485552 Sep 10 16:10 binlog.95

-r-------- 1 c_btalk c_btalk 10485748 Sep 10 16:13 binlog.96

-r-------- 1 c_btalk c_btalk 10485741 Sep 10 16:18 binlog.97

-r-------- 1 c_btalk c_btalk 10485714 Sep 10 16:26 binlog.98

-r-------- 1 c_btalk c_btalk 10485567 Sep 10 16:32 binlog.99

-rw------- 1 c_btalk c_btalk        0 Aug  7 10:47 lock

[Keith Rarick]

Thanks! This is interesting. What command line flags are you
using? (In particular, I'm wondering if you are using -n)

Also, can you send the output of a stats command on this
beanstalkd process?

[Alex Ilievski]

We are running the server with the following

beanstalkd -l 192.168.192.75 -p 11300 -b /home/c_btalk/wal/

Stats:

OK 1033

---

current-jobs-urgent: 0

current-jobs-ready: 7818

current-jobs-reserved: 0

current-jobs-delayed: 208605

current-jobs-buried: 2

cmd-put: 1534498

cmd-peek: 0

cmd-peek-ready: 129

cmd-peek-delayed: 120

cmd-peek-buried: 120

cmd-reserve: 0

cmd-reserve-with-timeout: 44655544

cmd-delete: 1318096

cmd-release: 38274507

cmd-use: 1534891

cmd-watch: 44655544

cmd-ignore: 44655544

cmd-bury: 3

cmd-kick: 24

cmd-touch: 8957

cmd-stats: 2

cmd-stats-job: 167

cmd-stats-tube: 438961

cmd-list-tubes: 226718

cmd-list-tube-used: 0

cmd-list-tubes-watched: 0

cmd-pause-tube: 0

job-timeouts: 9

total-jobs: 1534498

max-job-size: 65535

current-tubes: 15

current-connections: 21

current-producers: 1

current-workers: 19

current-waiting: 19

total-connections: 584077

pid: 587641

version: 1.9

rusage-utime: 12501.878425

rusage-stime: 10511.711976

uptime: 3041405

binlog-oldest-index: 89

binlog-current-index: 367

binlog-records-migrated: 141570

binlog-records-written: 41314838

binlog-max-size: 10485760

id: 9bb9e653af9e05a1

- Alex

[Keith Rarick]

Ok, with over 210,000 jobs currently in the system, it seems
quite possible they could be taking up over 2MB, depending
on how big the average job is. What is your mean or typical
job size? (You can find this by running command peek-ready
and measuring the size of the output.)

[Keith Rarick]

> Ok, with over 210,000 jobs currently in the system, it seems
> quite possible they could be taking up over 2MB, depending

My apologies, I misread. 2.8GB (in a few hundred segments)
is way more than it should be. This is probably a bug.

[Keith Rarick]

Did you put close to 2GB of jobs in this process at any one time?

This could be integer overflow finding the ratio of used to dead wal space.

[Alex Ilievski]

No we've just migrated all jobs from the previous server and there wasn't that much data.

I had a peek into the oldest binlog and can see job data from both deleted jobs and existing jobs.

I'm not sure how binlogs are cycled or re-used but it looks like they aren't being compacted or deleted.

- Alex

[Keith Rarick]

What's your average job size? Is it around 10 kbytes?

[Alex Ilievski]

It is < 1 kbytes. A typical job would look like this.

{

    "priority":4669,

    "message_id":"540fcc177304e",

    "campaign_id":"540faf7c82928",

    "client_id":"4f8d28131142c",

    "patron_id":"4ff2593dcb938",

    "username":"wolflane",

    "seed":false,

    "seed_email":""

}

- Alex

[Keith Rarick]

Ok, with a 225 byte average job, I'd expect 210,000 current jobs to
result in between 100 and 200 MB on disk. Something is definitely
strange here.

> --
> You received this message because you are subscribed to the Google Groups
> "beanstalk-talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to beanstalk-tal...@googlegroups.com.
> To post to this group, send email to beansta...@googlegroups.com.
> Visit this group at http://groups.google.com/group/beanstalk-talk.
> For more options, visit https://groups.google.com/d/optout.

[michele zuppala]

Hi,

I've little experience with Beanstalk then correct me if I was wrong, but I've notice that compaction try to move usable jobs from oldest binlog to current binlog. When a binlog has no more usable jobs, beanstalk remove it from filesystem, otherwise binlog stay in place and new jobs are added on an eventually new binlog.

This is very good with jobs that is produced and consumed quickly.

But with long delayed jobs, small binlog size, and high load, it's possible that compaction sparse usable jobs among several files (that seems to be your case) because current binlog is too small and compaction fail to move jobs around files.

In my opinion if you really use long delayed jobs, you can try to :

- increase binlog size

- or split delay deleting e resubmitting the same job more frequently in order to force it to move away from oldest binlog.

Cheers

Michele

[Keith Rarick]

On Thu, Sep 11, 2014 at 2:21 PM, michele zuppala <ard...@gmail.com> wrote:
> I've little experience with Beanstalk then correct me if I was wrong, but
> I've notice that compaction try to move usable jobs from oldest binlog to
> current binlog.

It moves existing jobs. Doesn't matter if the jobs are ready,
delayed, or buried.

> When a binlog has no more usable jobs, beanstalk remove it
> from filesystem, otherwise binlog stay in place and new jobs are added on an
> eventually new binlog.

This is correct.

> This is very good with jobs that is produced and consumed quickly.
>
> But with long delayed jobs, small binlog size, and high load, it's possible
> that compaction sparse usable jobs among several files (that seems to be
> your case) because current binlog is too small and compaction fail to move
> jobs around files.

Compaction tries to keep the ratio of wasted space to used space
below 2:1. When the ratio is higher than that, it'll move one or more
jobs after every user write operation. The higher the ratio, the more
jobs it moves each time. In other words, it works harder to try to
converge faster when necessary.

This ratio is simply used space (including all delayed, buried, and
ready jobs) to wasted space (deleted jobs and migrated records).

If the usable jobs are spread sparsely among several files, that
would imply there is a lot of wasted space, and the existing
compaction code should be working harder to migrate more jobs.

If that's not happening, it's simply a bug.

> In my opinion if you really use long delayed jobs, you can try to :

There should be no need for tricks like these to manage disk
space when you have many delayed jobs, or for any other access
pattern.

You should simply expect your disk usage to scale linearly (both
upwards and downwards) with the number of jobs in the system at
any given time. I think this is a reasonable and intuitive rule of
thumb: your disk usage is however much space you need to hold
all the jobs in the system, with a constant factor of overhead.

Again, this is the intended behavior, and it's looking like there is
a bug where it doesn't work properly in this case.

[ilmich]

Thanks Keith for this excellent explanation.

What do you think about adding statistical information of compactor (current waste space ratio, number of failed/completed compactions.. and so on) in a separate command (like stat-binlogs) or by extending current stats command!?

Cheers

--
You received this message because you are subscribed to a topic in the Google Groups "beanstalk-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/beanstalk-talk/k2r2ZJKoRFM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to beanstalk-tal...@googlegroups.com.

[kian.m...@gmail.com]

Hi Keith,

I may be running into this issue as well.  On a particular node (1 of 6), I'm seeing the binlog directory has grown to 27GB and around 2500 files.  The other 5 nodes have more reasonable binlog sizes (~5GB, 500 or so files).

 - Beanstalk 1.9 on Debian Linux

 - The host in question has binlog-records-migrated value of 0

 - Our jobs are around 250-400 bytes each

 - These nodes handle ~200 jobs per second each

I'm investigating high CPU utilization & disk I/O spikes in beanstalkd on production so I'm happy to help track this down in any way I can.

Here is memory usage:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                        

 4691 root      20   0 6149m 6.0g  660 S     5 25.3  77:36.33 beanstalkd     

Here is disk usage:

$ du -hs /var/lib/beanstalkd/

27G /var/lib/beanstalkd/

Here is file count:

$ ls -al /var/lib/beanstalkd/ | grep binlog | wc -l

2692

Here is the output of stats-tube:

--------------------------------------------------------

OK 1013

---

current-jobs-urgent: 0

current-jobs-ready: 10364940

current-jobs-reserved: 95

current-jobs-delayed: 0

current-jobs-buried: 3490

cmd-put: 8247649

cmd-peek: 0

cmd-peek-ready: 1799

cmd-peek-delayed: 26

cmd-peek-buried: 26

cmd-reserve: 4213336

cmd-reserve-with-timeout: 10993828

cmd-delete: 8785013

cmd-release: 0

cmd-use: 8251143

cmd-watch: 1655

cmd-ignore: 1655

cmd-bury: 58066

cmd-kick: 59

cmd-touch: 0

cmd-stats: 12867

cmd-stats-job: 944

cmd-stats-tube: 1101846

cmd-list-tubes: 157294

cmd-list-tube-used: 0

cmd-list-tubes-watched: 4

cmd-pause-tube: 0

job-timeouts: 1482

total-jobs: 8247649

max-job-size: 65535

current-tubes: 7

current-connections: 153

current-producers: 0

current-workers: 152

current-waiting: 37

total-connections: 8343737

pid: 4691

version: 1.9

rusage-utime: 1352.356517

rusage-stime: 3288.077492

uptime: 173047

binlog-oldest-index: 31746

binlog-current-index: 34353

binlog-records-migrated: 0

binlog-records-written: 17146841

binlog-max-size: 10485760

id: f5912079375bea2d

--------------------------------------------------------

What other information would be useful to provide?

Thanks for your time.

Kian