beets is super super slow (>24h to import 20 albums)

[Enrico]

Hey, I've been running beets on an hourly cron for about 6 months, everything used to be fine however I've noticed my imports becoming slower and slower, typically when more than one album is imported the process time is increased by an order of magnitude. Beets always gets through it with no probs, however I'd prefer if it could keep up. I currently have an import of about 25 albums which has been going for 36 hours.

My musiclibrary.blb is 655mb, log is less than 15kb, and state is 48kb (skips about 500 folders). An example of one of my config files is below, nothing too remarkable.

Any ideas?

# Beets configuration --------------------------------------------------------

directory: /music

library: /config/musiclibrary.blb

import:

# copy settings    

    move: no

    copy: yes

    incremental: yes

    resume: yes

# =====================   

    write: yes

    delete: no

    autotag: yes

    timid: no

    log: /config/beet.log

    quiet: yes  # enable with command line option

    quiet_fallback: asis

    none_rec_action: asis

    default_action: apply

    duplicate_action: remove

    singletons: no

    languages: [en]

    detail: no

    flat: no

    

# General --------------------------------------------------------------------

threaded: yes

timeout: 5.0

verbose: no

# use the release-date of the original (first) release of an album?

original_date: no

# on multi-disk releases, assign track numbers for the whole album.

# If "per disk", make sure tracknames do not collide ("paths" setting).

per_disc_numbering: yes

# files matching these patterns are deleted from source after import

#clutter: ["Thumbs.DB", ".DS_Store", "*.m3u", ".pls", "*.jpg", "*.m3u", "*.log", "*.cue"]

# files/directories matching one of these patterns are ignored during import

ignore: [".*", "*~", "System Volume Information"]

# Paths ----------------------------------------------------------------------

# Paths and filenames for music files

# relative to music directory

paths:

    #default: $albumartist/$album%aunique{}/%if{$multidisc,Disc $disc/}$track - $title

    #singleton: Non-Album/%if{$multidisc,Disc $disc/$artist - $title

    #comp: Compilations/$album%aunique{}/%if{$multidisc,Disc $disc/$track - $title

    #albumtype_soundtrack: Soundtracks/$album%aunique{}/%if{$multidisc,Disc $disc/$track $title

    

    comp:      '_Compilations/$album%aunique{}/%if{$multidisc,Disc $disc/}$track $title'

    singleton: '%bucket{$albumartist }/$artist/%if{$album,$album,(Single)}/%if{$multidisc,Disc $disc/}$title'

# force soundtrack to default format;

    albumtype:soundtrack: '%bucket{$albumartist }/$albumartist/$album %aunique{}/%if{$multidisc,Disc $disc/}$track $title'

    default:   '%bucket{$albumartist }/$albumartist/$album %aunique{}/%if{$multidisc,Disc $disc/}$track $title'

item_fields:

    multidisc: 1 if disctotal > 1 else 0    

# replace special characters in generated filenames

replace:

    '[\\/]': _

    '^\.': _

    '[\x00-\x1f]': _

    '[<>:"\?\*\|]': _

    '\.$': _

    '\s+$': ''

path_sep_replace: _

# filename for the album art

art_filename: cover  # results in "coverart.jpg"

max_filename_length: 0  # unlimited

# User Interface -------------------------------------------------------------

color: yes

list_format_item: %upper{$artist} - $album - $track. $title

list_format_album: %upper{$albumartist} - $album

time_format: '%Y-%m-%d %H:%M:%S'

terminal_encoding: utf8

ui:

    terminal_width: 80

    length_diff_thresh: 10.0

# Auto Tagger ----------------------------------------------------------------

match:

    strong_rec_thresh: 0.17      # match 90% or better for auto import

    medium_rec_thresh: 0.25

    rec_gap_thresh: 0.25

    max_rec:

        missing_tracks: medium

        unmatched_tracks: medium

    distance_weights:

        source: 2.0

        artist: 3.0

        album: 3.0

        media: 1.0

        mediums: 1.0

        year: 1.0

        country: 0.5

        label: 0.5

        catalognum: 0.5

        albumdisambig: 0.5

        album_id: 5.0

        tracks: 2.0

        missing_tracks: 0.9

        unmatched_tracks: 0.6

        track_title: 3.0

        track_artist: 2.0

        track_index: 1.0

        track_length: 2.0

        track_id: 5.0

    preferred:

        countries: []

        media: []

        original_year: no

    ignored: []

    track_length_grace: 10

    track_length_max: 30

# Plugins --------------------------------------------------------------------

plugins: [

    #fetchart,

    lastgenre,

    #chroma,

    scrub,

    discogs,

    #beatport,

    duplicates,

    #web,

    #extrafiles,

    #lyrics,

    #echonest_tempo,

    acousticbrainz,

    embedart,

    inline,

    bucket

    #mbsync,

    #mpdupdate,

    #mpdstats,

    #missing,

    #fromfilename

]

#pluginpath: /config/plugins/beatport.py

# Plugins Config -------------------------------------------------------------

    

lastgenre:

   # whitelist: ~/.config/beets/genres.txt

    #canoncical: ~/.config/beets/genres-tree.yaml

    #fallback: ''

    auto: yes

    source: album

embedart:

    auto: yes

    maxwidth: 800

replaygain:

    auto: yes

    overwrite: yes

    albumgain: yes

duplicates:

    checksum: no  # expensive

    tiebreak: { items: [bitrate] }

    

bucket:

    bucket_alpha:

    - _

    - _Compilations

    - A

    - B

    - C

    - D

    - E

    - F

    - G

    - H

    - I

    - J

    - K

    - L

    - M

    - N

    - O

    - P

    - Q

    - R

    - S

    - T

    - U

    - V

    - W

    - X

    - Y

    - Z

    bucket_alpha_regex:

        _: ^[^A-Za-z]

        _Compilations: (\W|^)Various\sArtists(\W|$)

    bucket_year: []

    extrapolate: no    

[Adrian Sampson]

Wow, that’s weird! I don’t have any obvious ideas, and the right way to sort this out would be to do some proper profiling. Do you have any ideas about how we might reproduce the problem? Or would you be willing to break out a Python profiler and take a look?
https://docs.python.org/3.7/library/profile.html

Adrian

> --
> You received this message because you are subscribed to the Google Groups "beets" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beets-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Robbie Sands]

I could send you my appdata folder, which would allow you to test beets in a similar environment. I'm running beets in a docker container and my python knowledge is minimal, so unfortunately my ability to test is limited. My guess is it's either due to the size of the music library or the state file. Starting with a fresh install yields quick imports. Unfortunately it's still running and I'm hesitant to stop it as I don't want to miss any new music, so immediate testing is limited.

[Enrico]

I switched from copy to move and removed the incremental and resume functions, seems to be working as normal now. If you have any ideas let me know. I'll try starting the copy function on a fresh directory and see if the problem ramps up again over time.

> To unsubscribe from this group and stop receiving emails from it, send an email to beets...@googlegroups.com.

[Adrian Sampson]

Weird! Well, if you sort out whether the problem seems to be caused by the state file or something else, that might help explain it. I wouldn’t be *terribly* surprised if something we do with the state file doesn’t scale well as it grows.

A

> To unsubscribe from this group and stop receiving emails from it, send an email to beets-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/beets-users/7e099c81-8ca5-46b6-83b4-021c326fcba6%40googlegroups.com.

[Enrico]

Ok it's been running for 48h now processing a folder with just over 500 album folders, however it's only processed about 12% of the content. Beets is using: move, not incremental, not resume. Checking the system, it's not like beets is doing much, plugins are minimal. I might remove a few more plugins and reboot (who knows?!) to see if there's any change.

plugins: [

    lastgenre,

    scrub,

    discogs,

    duplicates,

    acousticbrainz,

    embedart,

    inline,

    bucket

]

> To unsubscribe from this group and stop receiving emails from it, send an email to beets...@googlegroups.com.

[Enrico]

Ok I finally got an error:

Traceback (most recent call last):
File "/usr/bin/beet", line 11, in
load_entry_point('beets==1.4.7', 'console_scripts', 'beet')()
File "/usr/lib/python3.6/site-packages/beets/ui/__init__.py", line 1256, in main
_raw_main(args)
File "/usr/lib/python3.6/site-packages/beets/ui/__init__.py", line 1243, in _raw_main
subcommand.func(lib, suboptions, subargs)
File "/usr/lib/python3.6/site-packages/beets/ui/commands.py", line 943, in import_func
import_files(lib, paths, query)
File "/usr/lib/python3.6/site-packages/beets/ui/commands.py", line 913, in import_files
session.run()
File "/usr/lib/python3.6/site-packages/beets/importer.py", line 329, in run
pl.run_parallel(QUEUE_SIZE)
File "/usr/lib/python3.6/site-packages/beets/util/pipeline.py", line 445, in run_parallel
six.reraise(exc_info[0], exc_info[1], exc_info[2])
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/beets/util/pipeline.py", line 312, in run
out = self.coro.send(msg)
File "/usr/lib/python3.6/site-packages/beets/util/pipeline.py", line 194, in coro
func(*(args + (task,)))
File "/usr/lib/python3.6/site-packages/beets/importer.py", line 1516, in plugin_stage
task.reload()
File "/usr/lib/python3.6/site-packages/beets/importer.py", line 846, in reload
item.load()
File "/usr/lib/python3.6/site-packages/beets/dbcore/db.py", line 433, in load
assert stored_obj is not None, u"object {0} not in DB".format(self.id)
AssertionError: object 409301 not in DB

[Enrico]

Seems like everything has sped up with new beets update.

Thanks