Bug#884229: 500 error/Internal Server Error

[Nicholas D Steeves]

Package: packages.debian.org
Severity: normal

Dear Debian WWW team,

I'm not sure where the website went wrong, but I found a 500 error/Internal Server Error on packages.debian.org for one of my packages.

https://packages.debian.org/sid/elpa-find-file-in-project

However, these pages are fine:
https://packages.qa.debian.org/f/find-file-in-project.html
https://tracker.debian.org/pkg/find-file-in-project

Thank you for your hard work,
Nicholas

[Laura Arjona Reina]

Hello

I can reproduce the problem with some other packages (I tried several from the
list https://packages.debian.org/unstable/main/newpkg and got the error in the
links to buster and sid).

I also tried
https://packages.debian.org/stretch-backports/kernel-image-4.14.0-0.bpo.2-arm64-di
with the same results (error 500).

Thanks to some help in the IRC channel we've found that only one of our wo
machines serving packages.debian.org fails:

https://packages-pkgmirror-csail.debian.org/sid/elpa-find-file-in-project
gives the 500 Internal Server Error
but
https://packages-picconi.debian.org/sid/elpa-find-file-in-project
works.

If I can be of any help to solve this, just tell. I'll try to look at the logs
later, but I'm not sure I have the corresponding permissions (will try).

Thanks
--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

[Cyril Brulebois]

Control: tag -1 patch

Hi Laura,

Thanks for mentioning this on #-www today!

Laura Arjona Reina <lar...@debian.org> (2018-01-24):

From the apache error log on the mirror:
> mod_fcgid: stderr: Can't call method "get_document" on an undefined value at ../lib/Packages/Search.pm line 264.
> End of script output before headers: dispatcher.fcgi

Looking at the code, that's a method call on a xapian query result.

Looking at the xapian directory on picconi:
> pkg_user@picconi:/srv/packages.debian.org$ ls -l files/db/xapian/
> total 302384
> -rw-r--r-- 1 pkg_user pkg_maint 1974272 Jan 24 15:56 docdata.glass
> -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 15:53 flintlock
> -rw-r--r-- 1 pkg_user pkg_maint 138 Jan 24 15:56 iamglass
> -rw-r--r-- 1 pkg_user pkg_maint 181952512 Jan 24 15:56 position.glass
> -rw-r--r-- 1 pkg_user pkg_maint 80699392 Jan 24 15:56 postlist.glass
> -rw-r--r-- 1 pkg_user pkg_maint 44998656 Jan 24 15:56 termlist.glass

Looking at the xapian directory on the mirror:
> pkg_user@pkgmirror-csail:/srv/packages.debian.org$ ls -l files/db/xapian/
> total 537584
> -rw-r--r-- 1 pkg_user pkg_maint 1974272 Jan 24 09:48 docdata.glass
> -rw-r--r-- 1 pkg_user pkg_maint 0 Jan 24 09:45 flintlock
> -rw-r--r-- 1 pkg_user pkg_maint 28 Nov 8 09:26 iamchert
> -rw-r--r-- 1 pkg_user pkg_maint 138 Jan 24 09:48 iamglass
> -rw-r--r-- 1 pkg_user pkg_maint 1699 Nov 8 09:30 position.baseA
> -rw-r--r-- 1 pkg_user pkg_maint 1725 Nov 8 09:30 position.baseB
> -rw-r--r-- 1 pkg_user pkg_maint 111788032 Nov 8 09:30 position.DB
> -rw-r--r-- 1 pkg_user pkg_maint 181944320 Jan 24 09:48 position.glass
> -rw-r--r-- 1 pkg_user pkg_maint 1254 Nov 8 09:30 postlist.baseA
> -rw-r--r-- 1 pkg_user pkg_maint 1263 Nov 8 09:30 postlist.baseB
> -rw-r--r-- 1 pkg_user pkg_maint 81616896 Nov 8 09:30 postlist.DB
> -rw-r--r-- 1 pkg_user pkg_maint 80707584 Jan 24 09:48 postlist.glass
> -rw-r--r-- 1 pkg_user pkg_maint 54 Nov 8 09:30 record.baseA
> -rw-r--r-- 1 pkg_user pkg_maint 56 Nov 8 09:30 record.baseB
> -rw-r--r-- 1 pkg_user pkg_maint 2523136 Nov 8 09:30 record.DB
> -rw-r--r-- 1 pkg_user pkg_maint 691 Nov 8 09:30 termlist.baseA
> -rw-r--r-- 1 pkg_user pkg_maint 703 Nov 8 09:30 termlist.baseB
> -rw-r--r-- 1 pkg_user pkg_maint 44883968 Nov 8 09:30 termlist.DB
> -rw-r--r-- 1 pkg_user pkg_maint 45006848 Jan 24 09:48 termlist.glass

For files dated Jan 24, the timestamps don't match, but that's probably
just a sync waiting to happen, and that doesn't explain the issues which
have been happening for so long. I suspected the stray files instead,
dating back to November. I've created an “old-files” directory and moved
them there, and the mirror seems to be behaving properly now.

I'm tagging this bug report with “pending” for the time being, to give
people a chance to comment. But I suppose it should be safe to remove
those old files entirely?

FTR, that would mean cleaning this:

230M /srv/packages.debian.org/files/db/xapian/old-files/


Cheers,
--
Cyril Brulebois (ki...@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[Olly Betts]

Yes. The directory on pkgmirror-csail actually had two different
versions of the database using different database backends, so the
files which aren't in the directory on picconi are essentially an
old version of the same database.

That alone shouldn't cause the search to fail though - it should just
open one or the other (it looks like both Xapian 1.2.x and 1.4.x will
open the old one in this rather odd situation).

Xapian's replication should cleanly handle the database being replicated
switching backends so I'm guessing the replication here is using rsync
(without --delete-after) or similar? That might also explain why the
old database is broken as the databases aren't safe to rsync if an
update is in progress - if the last rsync of the old database happened
while an update was in progress, it could have been left broken, which
would typically result in particular searches failing.

Updating with rsync can also break searches on the mirror while the
rsync is in progress even if the master isn't updating, so perhaps we
should switch to using Xapian's replication? That's already being
used for the main website search (I set it up with weasel at Debconf
15).

Happy to assist with that, though I'll probably need ssh access to
pkgmirror-csail (seems I currently only have it for picconi). My
Debian username is "olly".

Cheers,
Olly

[Cyril Brulebois]

Hi Olly,

Olly Betts <ol...@survex.com> (2018-01-26):

> Yes. The directory on pkgmirror-csail actually had two different
> versions of the database using different database backends, so the
> files which aren't in the directory on picconi are essentially an
> old version of the same database.
>
> That alone shouldn't cause the search to fail though - it should just
> open one or the other (it looks like both Xapian 1.2.x and 1.4.x will
> open the old one in this rather odd situation).
>
> Xapian's replication should cleanly handle the database being replicated
> switching backends so I'm guessing the replication here is using rsync
> (without --delete-after) or similar? That might also explain why the
> old database is broken as the databases aren't safe to rsync if an
> update is in progress - if the last rsync of the old database happened
> while an update was in progress, it could have been left broken, which
> would typically result in particular searches failing.
>
> Updating with rsync can also break searches on the mirror while the
> rsync is in progress even if the master isn't updating, so perhaps we
> should switch to using Xapian's replication? That's already being
> used for the main website search (I set it up with weasel at Debconf
> 15).
>
> Happy to assist with that, though I'll probably need ssh access to
> pkgmirror-csail (seems I currently only have it for picconi). My
> Debian username is "olly".

Looking around without knowing much about pkg stuff:
- picconi has: conf/pushpdo.mirror → csail pkgmirror-csail.debian.org pkg_user
- those bits are read by bin/pushpdo, which calls rsync this way:
rsync -e "ssh -i ${MKEYFILE} -${MPROTO} ${SSH_OPTS}" -av --stats "${MIRRORPATH}" ${MUSER}@${MHOSTNAME}:/does/not/matter >"${LOGDIR}/pushpdo.${MLNAME}.log"

I suppose help is welcome to possibly fix/improve that, but I'm no pkg
expert (I just happened to have helped one or twice in the past, and to
still be in the pkg_maint group…)

[Alex ARNAUD]

On Wed, 24 Jan 2018 17:32:45 +0100 Cyril Brulebois <ki...@debian.org> wrote:
> Control: tag -1 patch
>
> Hi Laura,
>
> Thanks for mentioning this on #-www today!

Hello all,

I'm also experiencing such issue on this page:
https://packages.debian.org/experimental/firefox-esr

Best regards,
Alex.

[Andrey Gursky]

Dear maintainers, server administrators and web masters,

a couple of weeks ago package sites from experimental suite started to
fail opening with error 500 Internal Server Error. Because this failure
is not temporary anymore but became unfortunately stable, I'd like to
ask you to check the situation.

Thanks,
Andrey

[Thomas Lange]

Currently the URL
https://packages.debian.org/sid/elpa-find-file-in-project
works and gives correct results.

--
regards Thomas

[Nicholas D Steeves]

Hi Thomas and everyone reading this,

On Wed, Jul 17, 2019 at 11:04:48PM +0200, Thomas Lange wrote:
> Currently the URL
> https://packages.debian.org/sid/elpa-find-file-in-project
> works and gives correct results.
>

Unfortunately the error is transient...within my DDPO it seemingly
randomly flips between packages from week to week.

Regards,
Nicholas

[Nicholas D Steeves]

Hi Laura,

Laura Arjona Reina <lar...@debian.org> writes:

> Hello
>
> Today I got mail about error 500 Internal Server Error happening in
> packages.debian.org, not related to this one, but I took the opportunity
> to review the situation about this bug too.
>
> 1. I cannot reproduce now the 500 Internal Server Error nor in picconi
> nor in pkgmirror-csail with any of the links provided in this bug
> report. Everything seems to work fine.
>
> 2.- I have checked both machines and the files under
> /srv/packages.debian.org/files/db/xapian/ are similar, so I guess
> synchronization is happening well.
>
> 3.- I have seen that the folder
> /srv/packages.debian.org/files/db/xapian/old-files/ still existed, so I
> have removed the old files there and the old-files folder too.
>
> So, closing this bug; if any issue happens again feel free to reopen or
> open a new bug (I guess we'd need to look at the issues in syncing both
> servers).
>

Thanks, that's a fair assessment. I confirm that my DDPO and Dashboard
looks good, with the exception of some git repo 404 errors that look
like gitlab/salsa bugs (off topic for this bug).

Best,
Nicholas

P.S. resending to b...@bugs.d.o, because I accidentally CCed bugs-done in
my last reply.