Linux performance during startup with lots of large workspaces
Martin Dorey
I have ~32 workspaces, many of them containing over 20 000 source files. Since I just upgraded to Debian Lenny, with Linux 2.6.26, from Etch with 2.6.18, starting Evergreen with a cold cache is more painful than ever. It's not so much the time that's painful, though that is painful:
2009-07-29T13:14:23.61-0700 Evergreen: Java 1.6.0_14 (VM 14.0-b16, runtime 1.6.0_14-b08)
...
2009-07-29T13:25:51.348-0700 Evergreen: Scan of workspace "3rdParty" took 11m19s; found 74169 files.
It's the machine being really quite unusable in the meantime that's really getting my goat. It could be that the performance hasn't got worse, though I think it has, because I'm now using both screens attached to the same computer, so I can't use the other computer to "do email" in the meantime.
Hot cache restart is no issue at all, really, being less than 30s:
2009-07-29T13:29:44.832-0700 Evergreen: Java 1.6.0_14 (VM 14.0-b16, runtime 1.6.0_14-b08)
...
2009-07-29T13:30:07.595-0700 Evergreen: Scan of workspace "t1" took 3.25 s; found 26160 files.
For many months now, I've been running with this, but it doesn't seem to help me now:
@@ -137,6 +139,8 @@ WorkspaceFileList.FileListUpdater.FileListUpdater WorkspaceFileList.FileListUpdater.doInBackground
// Don't hog the CPU while we're still getting started.
Evergreen.getInstance().awaitInitialization();
+ Thread.currentThread().setPriority(Thread.MIN_PRIORITY);
+
ArrayList<String> newFileList = scanWorkspaceForFiles();
// Many file systems will have returned the files not in alphabetical order, so we sort them ourselves here.
// Users of the list can then assume it's in order.
I think this made it more responsive, without increasing (and perhaps reducing) the elapsed time:
--- src/e/edit/WorkspaceFileList.java (revision 1688)
+++ src/e/edit/WorkspaceFileList.java (working copy)
@@ -17,7 +17,7 @@ WorkspaceFileList.fileList
private ArrayList<String> fileList;
private FileAlterationMonitor fileAlterationMonitor;
- private ExecutorService fileListUpdateExecutorService;
+ private static ExecutorService fileListUpdateExecutorService;
public WorkspaceFileList(Workspace workspace) {
this.workspace = workspace;
@@ -109,7 +109,9 @@ WorkspaceFileList.initFileAlterationMonitorForRoot
// We have one thread to check for last-modified time changes...
this.fileAlterationMonitor = new FileAlterationMonitor(rootDirectory);
// And another thread to update our list of files...
- this.fileListUpdateExecutorService = ThreadUtilities.newSingleThreadExecutor("File List Updater for " + rootDirectory);
+ if (fileListUpdateExecutorService == null) {
+ this.fileListUpdateExecutorService = ThreadUtilities.newFixedThreadPool(10, "File List Updater");
+ }
fileAlterationMonitor.addListener(new FileAlterationMonitor.Listener() {
public void fileTouched(String pathname) {
I changed the magic number to 4 and then I could read my email, happily enough, while Evergreen booted in some 13 minutes:
2009-07-29T13:40:41.25-0700 Evergreen: Java 1.6.0_14 (VM 14.0-b16, runtime 1.6.0_14-b08)
...
2009-07-29T13:53:39.761-0700 Evergreen: Scan of workspace "t1" took 2m41s; found 26160 files.
Hot cache is still under 30s:
2009-07-29T13:55:39.911-0700 Evergreen: Java 1.6.0_14 (VM 14.0-b16, runtime 1.6.0_14-b08)
...
2009-07-29T13:56:01.785-0700 Evergreen: Scan of workspace "t1" took 2.37 s; found 26160 files.
Well, that's me happy again.
Elliott Hughes
--elliott
Elliott Hughes, http://www.jessies.org/~enh/
Martin Dorey
Martin Dorey
10 minutes with a cold cache:
2009-08-03T10:18:13.533-0700 Evergreen: Java 1.6.0_14 (VM 14.0-b16, runtime 1.6.0_14-b08)
2009-08-03T10:28:09.387-0700 Evergreen: Scan of workspace "t1" took 1m12s; found 26141 files.
Just less than 30s with a hot cache:
2009-08-03T10:30:16.53-0700 Evergreen: Java 1.6.0_14 (VM 14.0-b16, runtime 1.6.0_14-b08)
2009-08-03T10:30:45.224-0700 Evergreen: Scan of workspace "t1" took 1.52 s; found 26141 files.
Job's a good un then.