SoulseekQt lowmem edition phase 1 - user shares

2825 views
Skip to first unread message

Nir Arbel

unread,
Jan 31, 2018, 12:25:14 AM1/31/18
to soulseek-...@googlegroups.com
So I'm not entirely sure how much of a problem SoulseekQt memory consumption is for people out there, but it's something that comes up pretty regularly on the forum and I figured I'd try to push through the horror of trying to refactor some of the most sensitive, complicated parts of the client code and see how far I can get. 

To recap, SoulseekQt takes a lot more memory than Soulseek NS, the Windows-only, basically original Soulseek client that's slowly been phased out over the last few years. Being peer-to-peer software that ends up doing a lot of different things under the hood, the Soulseek client is a pretty complicated piece of software. When I decided to create a new cross-platform client using the cross-platform Qt framework, I felt like I've hit some sort of complexity ceiling. It primarily had to do with the different kinds of data managed by the client and their interrelationships, so I designed a data system, a sort of super map that can accommodate all of the client's data and makes it very easy to manage and navigate. That made it a lot easier for me to implement the more advanced features that set SoulseekQt apart from the older Soulseek NS.

The downside is that it ends up consuming a lot more memory. This isn't a problem in most areas of functionality, but there are three in particular where the difference might be noticeable:
  1. When you browse users with large shares.
  2. When you share a lot of files yourself.
  3. When you keep a lot of search windows open.
Of those, the first one is probably that one that affects the most users. Search result windows are limited to 10,000 results each by default, which isn't that much data. And most users don't share hundreds of thousands of files, which is the point at which memory consumption becomes excessive. Browsing a single user's (large) share can end up consuming hundreds of megabytes, and keeping multiple such share browse windows open can often spell trouble. Fortunately, it's the easiest of the three to phase the data system out of and replace with memory-efficient standard containers (essentially a whole bunch of b-tree maps). Easiest, but not easy. It took me a few days to get to the point where I can actually browse someone and download something from their share, but taking the data system out doesn't only reduce memory consumption, it also increases the possibility of data inconsistencies and makes bugs more likely, so bear in the mind. On the memory consumption side, things are looking much better. Browsing a share of about 450,000 files, memory consumption just for loading the share went from about 350MB to 100MB with the Windows build which is 32-bit, and roughly double those numbers with the macOS build which is 64-bit.

I'm not completely decided on 2 and 3 yet. Removing the data system from search results may not have that much of an effect. Removing it from your own shared files will definitely help you if you're sharing hundreds of thousands of files, but it's by far the hardest to do, since your own shared files are indexed in all sorts of different exotic ways, and the only way to save client data currently, such as cached audio attributes of files you're sharing that have been scanned, is via said data system. So it will be a big project. Not off the table, but at a time when Chrome on my desktop is taking 1.5GB with four tabs open, I have to wonder if it's worth the trouble. In the meantime, these are the Windows/macOS builds for phase 1:


Known issues:
  • The text in tab headers everywhere (e.g. Search, Rooms) on macOS is mangled due to a yet to be fixed bug in Qt. The newest version of Qt without this bug doesn't work as well with newer versions of macOS unfortunately, so it's a choice between better macOS compatibility and this UI problem right now. I filed a bug report with the Qt folks (https://bugreports.qt.io/browse/QTBUG-64658), will update when they fix this.
  • The 'Upload to user' function doesn't work at all right now. In truth the function should be redesigned entirely, as the existing implementation can't differentiate between files that were requested from the uploader and files pushed by the uploader which leads to problems such as not being able to delete unwanted downloads without them being re-queued over and over again. A new handshake protocol for pushed uploads will address this, but will limit the feature to pairs of users who are both using the updated build.

lim...@gmail.com

unread,
Feb 1, 2018, 6:37:00 AM2/1/18
to Soulseek Technical Discussion
Linux?

Nir Arbel

unread,
Feb 1, 2018, 2:40:19 PM2/1/18
to Soulseek Technical Discussion
Added link to 64-bit AppImage.

RandomTox

unread,
Feb 2, 2018, 5:09:33 AM2/2/18
to Soulseek Technical Discussion
The search results header doesn't display the text you searched for.  

Attached is an example.
Screen Shot 2018-02-02 at 10.06.14 am.png

majster

unread,
Feb 2, 2018, 12:36:35 PM2/2/18
to Soulseek Technical Discussion
Nir, can you update post on the website?

majster

unread,
Feb 2, 2018, 1:13:20 PM2/2/18
to Soulseek Technical Discussion


On Linux version 2018-1-30 looks like on attachment.

Wrong version posted?

Zrzut ekranu z 2018-02-02 18-47-31.jpg

Nir Arbel

unread,
Feb 2, 2018, 4:21:41 PM2/2/18
to Soulseek Technical Discussion
RandomTox, this is a known issue. I've added details at the bottom of the original post.

Nir Arbel

unread,
Feb 2, 2018, 4:45:24 PM2/2/18
to Soulseek Technical Discussion
Hey majster, yeah, I posted the wrong build, thanks for the heads up. Weirdly I didn't need to change the download link after I posted the correct build which isn't normal behavior as far as I know. Normally the download link becomes invalid whenever the file changes. Let me know if you're still getting the wrong build.

Regarding updating the website, do you mean the download page? If so I'd like to wait until this build is finalized. There's at least a few other things I want look into like keeping a stored copy of the compressed share in memory so it doesn't have to be packed/zipped every time someone browses your share. Which I think is causing "random" freezes for some users.

majster

unread,
Feb 3, 2018, 4:41:53 AM2/3/18
to Soulseek Technical Discussion

majster

unread,
Feb 3, 2018, 5:02:25 AM2/3/18
to Soulseek Technical Discussion
now it's all right

winte...@gmail.com

unread,
Feb 4, 2018, 9:46:17 AM2/4/18
to Soulseek Technical Discussion
Mac OS X Yosemite user here. macOS image won't mount, I get 'no mountable file systems' message.

Nir Arbel

unread,
Feb 4, 2018, 4:17:10 PM2/4/18
to Soulseek Technical Discussion
Agh, yeah, thanks, done.

Nir Arbel

unread,
Feb 4, 2018, 5:41:22 PM2/4/18
to Soulseek Technical Discussion
Winterfury, looks like a new Qt bug with versions of macOS older than High Sierra: https://bugreports.qt.io/browse/QTBUG-65844

I'll look into possible solutions later tonight.

Nir Arbel

unread,
Feb 5, 2018, 6:01:44 PM2/5/18
to Soulseek Technical Discussion
Linked macOS build should now mount.
Message has been deleted
Message has been deleted

ohannesya...@gmail.com

unread,
Feb 8, 2018, 4:52:37 PM2/8/18
to Soulseek Technical Discussion
Dear Nir,

Thank you very very much for this build.

There is visible performance in this version. So in Mac OS X Sierra it works very well. Except from known bugs, everything was great.

Thank you for your efforts. If I find any errors I will contact you.

ohannes

Nir Arbel

unread,
Feb 8, 2018, 5:03:22 PM2/8/18
to Soulseek Technical Discussion
Thank you ohannes, glad to hear!

lim...@gmail.com

unread,
Feb 9, 2018, 7:09:54 AM2/9/18
to Soulseek Technical Discussion
Please make an assembly for ARM (Raspberry Pi and others) or some kind of universal package "snap"

kinde...@gmail.com

unread,
Feb 9, 2018, 8:32:55 AM2/9/18
to Soulseek Technical Discussion
hey i'm just here to say thank you thank you thank you.
so much dedication you put into this, it's amazing.
i'm using slsk for so many years now and i'm so glad i can be a part of this community sharing and exploring new & old music.
again, thank you so much!

Doktor420

unread,
Feb 9, 2018, 10:22:49 AM2/9/18
to Soulseek Technical Discussion
that lowmem edition runs for 96 hours with only one crash and one reboot. it freeze much less often and  just a few sconds.
so, i'm pleased so far.
just one thing. the upload to other user don't work anymore.

Pavel

unread,
Feb 14, 2018, 2:29:25 AM2/14/18
to Soulseek Technical Discussion
Hello, Nir
If it's possible, could you please add 32-bit Linux appimage? It would be useful for running on old hardware and netbooks.

Nir Arbel

unread,
Feb 15, 2018, 2:32:29 PM2/15/18
to Soulseek Technical Discussion
Thank you kinderesser! Much appreciated!

Nir Arbel

unread,
Feb 15, 2018, 2:34:03 PM2/15/18
to Soulseek Technical Discussion
limfeer, I don't know much about Raspberry Pi, but looking for information online doesn't it have very little memory? Not sure 1GB would be enough for SoulseekQt.

Nir Arbel

unread,
Feb 15, 2018, 6:42:18 PM2/15/18
to Soulseek Technical Discussion
Pavel, I spent a few hours setting up a 32-bit Ubuntu virtual machine with all the different tools needed to build SoulseekQt... basically Qt isn't offering updated installers for 32-bit Linux anymore. I tried with the older installer and SoulseekQt built fine, but then I found out the AppImage deployment tool I need (linuxdeployqt) isn't offering a 32-bit build at all. I tried creating one myself, but it's failing with said older version of Qt. 

I might be able to get around all this eventually if I keep trying, but it seems all the people making the tools I need are leaving 32-bit Linux behind, so I think I'm going to have to give up on that one. 

Nir Arbel

unread,
Feb 15, 2018, 6:48:12 PM2/15/18
to Soulseek Technical Discussion
Doktor420, yeah, it's listed as a known issue in the top post. The upload file view is awkwardly implemented as a copy of the browse share view with the downloading bits replaced with uploading bits, and since the underlying data structures were eliminated, it won't work anymore. I'll have to adapt the browse share view to work for both.

hanf...@gmail.com

unread,
Feb 17, 2018, 4:30:51 AM2/17/18
to Soulseek Technical Discussion
Nir Arbel, Hello. Yes, it is limited in resources. Has 1 GB of memory. How much this limitation I do not know. I can only note that I'm considering the use of Raspberry Pi and SoulseekQt, as a mini server that will work 24/7. Since this is an excellent compact mini PC, I think for these purposes. But at the same time it will be busy only with this, there will not be other applications on it. Also, as an alternative, you could think of a console application without a graphical interface. This will save resources, but at the same time it will also allow sharing music. This topic has already been raised more than once:


subscr...@gmail.com

unread,
Feb 17, 2018, 8:12:08 AM2/17/18
to Soulseek Technical Discussion

Thanks for your reply, Nir. I guess, no need for deploying Appimage in this case, as long as regular build works as it should. If I'm not mistaken, Qt 32-bit can be installed on Ubuntu by adding ppa from here https://launchpad.net/~beineri (up to 5.10 version).

пятница, 16 февраля 2018 г., 4:42:18 UTC+5 пользователь Nir Arbel написал:

Nir Arbel

unread,
Feb 21, 2018, 3:31:09 AM2/21/18
to Soulseek Technical Discussion
hanfbees, I'm running into a lot of trouble getting a Raspberry Pi QEMU virtual machine to run with Internet access inside a Linux virtual machine, which I apparently need in order to cross-compile SoulseekQt to the Pi. I'll look into getting a Raspberry Pi 3, but it may take a while before I can get anything going. There may be other hurdles involved that I'm not aware of as of yet.
Message has been deleted

realies

unread,
Feb 22, 2018, 4:57:45 AM2/22/18
to Soulseek Technical Discussion
Please consider providing a non-appimage version for Linux. The client from 2016 works fine under musl distributions with a gcompat layer, but due to buffer overflow errors, it has to be updated. Unfortunately, it is not possible to run 2017/2018 releases the same way as the old one - http://ix.io/NPc

realies

unread,
Feb 22, 2018, 10:04:49 PM2/22/18
to Soulseek Technical Discussion
Just to note. This above log is from running SoulseekQt that been extracted on a glibc system and copied over to the musl system. This might be the wrong way for when the app image doesn't like running on musl. For more details - http://ix.io/EbF

mattdag...@gmail.com

unread,
Mar 7, 2018, 1:45:21 AM3/7/18
to Soulseek Technical Discussion
Long time user here (since 2003 or so), and first time forum poster. Let me first say I'm glad to see there's work being done to make the program more resource-efficient. Your efforts are very much appreciated, Mr. Arbel.

My situation is this: I'm sharing 600,000+ files on Soulseek NS on a Windows 7 PC with 16 GB of RAM and an intel i7 quad-core. I have to keep my shares restricted to userlist only, because if I open them to the entire network, memory usage gradually rises as random people try browsing my shares. This goes on until about 1GB is being used, at which point the client crashes (C++ error).  So, unless I keep my shares open to userlist only, the client can only stay running for about 1 or 2 days at most, and with each reboot, it's a 3 or 4 hours wait to index everything. Not the most convenient of situations; however, userlist only has proven to be a very effective workaround, and allows me to keep the program running for weeks on end.

I've been holding off on making the switch to Qt for these last few years because of the memory issue mentioned in the OP, so I'm very excited to hear you're now turning your attention to this. I'll be following this thread closely with eager anticipation for a 64 bit memory-efficient windows build. Thanks again.

tund...@gmail.com

unread,
Mar 9, 2018, 2:11:02 PM3/9/18
to Soulseek Technical Discussion
Running great on ubuntu 16.04.3 Linux kernel 4.13.x
Message has been deleted

psynature…

unread,
Mar 10, 2018, 12:27:15 PM3/10/18
to Soulseek Technical Discussion
would it be possible to have soulseek closing into the tray, instead app exit when closing window? this is an issue for appimage build. using kde and ubuntu based distro. (walk–around is to set a rule in kwin not allowing window to close (closeable – force NO), and also in kwin disabling it from appearing in task manager – skip taskbar)

fah...@gmail.com

unread,
Apr 24, 2018, 12:35:50 PM4/24/18
to Soulseek Technical Discussion
Thank you.
This made thing sooooooo much better now.. running on Ubuntu 18.04.

nongeneric

unread,
May 16, 2018, 3:10:20 PM5/16/18
to Soulseek Technical Discussion
thank you, Nir!

litb...@gmail.com

unread,
Jun 7, 2018, 2:56:24 PM6/7/18
to Soulseek Technical Discussion
There is a problem with this lowmem version - on my win7 it take same memory as the previous version (i'm sharing about 500 000 files and it takes about 900 MB memory) but it's not a problem. the problem is that after some time(about 10-30 hours) it begins to slow down my computer and also slow down my upload speed(from about 1MB/s to 50Kb/s) and i need only restart windows to back to normal, also it takes more cpu usage(10-15 %, instead of normal 2-3%). so i think it have some bugs...

psynature…

unread,
Jul 15, 2018, 4:21:29 AM7/15/18
to Soulseek Technical Discussion
after hours of „work” and queuing about 20/30 albums soulseek stops downloading and can't connect anymore. i restarted it just to discover all work wasn't saved  (regardless saving settings every 5 minutes). i don't have time and energy to reproduce, but did anyone else stumbled on this nasty issue? [kernel 4.17, ubuntu 16.10, kde 5.13]

tund...@gmail.com

unread,
Jul 15, 2018, 8:39:08 AM7/15/18
to Soulseek Technical Discussion
Nope everything works perfectly fine on Ubuntu 18.04 I'd try updating because 16.10 is not even supported anymore.

psynature…

unread,
Jul 16, 2018, 5:11:08 AM7/16/18
to Soulseek Technical Discussion
i appreciate feedback,but did You actually had 25+active downloads (plus some uploads)?
16.10 is LTS meaning it's supported until 2021, please don't spread misinformation.
Message has been deleted

tund...@gmail.com

unread,
Jul 16, 2018, 5:55:07 AM7/16/18
to Soulseek Technical Discussion
16.10 is NOT an LTS release! 16.04 is an LTS. My info is correct there boss. 16.10 again is not an LTS release. So before you tell someone there info is wrong I'd check yourself first.

tund...@gmail.com

unread,
Jul 16, 2018, 5:55:18 AM7/16/18
to Soulseek Technical Discussion

psynature…

unread,
Jul 20, 2018, 6:04:25 AM7/20/18
to Soulseek Technical Discussion
typing 16.10 thinking of aprils release… sorry, i might be brain damaged.

tund...@gmail.com

unread,
Jul 20, 2018, 2:12:42 PM7/20/18
to Soulseek Technical Discussion
Its all good man. When 18.04 came out i did and upgrade from 16.04. Ive never ran into the issue you have but i also never hace 20/30 albums downloading. Now uploading i ususally have that. But no crashes. Im gonna continue to leave my soulseek and computer run through the rest of the day and through the night.

mkalam...@gmail.com

unread,
Jul 21, 2018, 5:17:16 AM7/21/18
to Soulseek Technical Discussion
Mac OS X version seems to be working fine, with the exception that when I go to the file transfers window, and I tried to right click and open the folder where the files I wanted have been downloaded, it will not open a finder window, nor will it 'play file' either. I have had this problem since installing QT 11-21-2017 and have no idea why this problem has emerged. This now means that I have to go and search for the downloaded files manually or by doing an indexed searched through finder. Any ideas why this is happening?

Much appreciated for all your efforts,

Marko

david...@gmail.com

unread,
Nov 18, 2018, 8:37:34 AM11/18/18
to Soulseek Technical Discussion
On Windows 10, I crash daily now … 

faircity

unread,
Nov 27, 2018, 6:59:31 AM11/27/18
to Soulseek Technical Discussion
Nir Please confirm this is untrue: "* Reconnected at 10:59:01.

[11:20:30][SoulseekAdmin] Soulseek Anouncement by Nir: Due to an official legal complaint by the RIAA, server support will terminate on November 30th 11:59pm EST

Reply all
Reply to author
Forward
0 new messages