Confusion matrix calculations

[Dominica Harrison]

Hello,
I started with 3,000 points across 318 images (~10 points per image). Then I added another 30 points per image (about 9,500 additional points) to try to improve the classifier. However, the overall accuracy remained around 56%, and the confusion matrix only shows n = 750.

Can someone explain how the confusion matrix works and why the accuracy didn’t change?
thank you
Dominica

[Stephen Chan]

Hi Dominica,

The confusion matrix only updates when a new classifier is saved or selected for the source.

Maybe you expected a new classifier training to run as a result of increasing annotations by 30 per image, but I suspect that no new training has actually run yet.
You can go to the front page of your source, and look for the "Last classifier saved" and "Last classifier trained" dates to see if this is the case. (It shows two different dates because sometimes a classifier is trained but not saved.)

Here's why I suspect this: to determine when it's time to train a new classifier, the main thing CoralNet checks is the change in the confirmed-image count since the last training. However, this logic can be a bit limited sometimes. In your case, it sounds like the last training had 318 confirmed images, while you currently still have 318 confirmed images. So, even though you added a lot more points per image, CoralNet doesn't realize that there is more data than the last training run, because all it does is look at the image count.

[Dominica Harrison]

Hello Stephen,

I think I found a solution! I combined the annotations from each run and made a new project using this methods you suggested to another person!

There isn't a particularly easy way to do this, but it's possible:

- Export the points/annotations as CSV format.

- Edit the CSV file to add more points for each image. You'll have to come up with the randomized point locations yourself. If you want to annotate the new points on CoralNet, fill in values for row and column, but not label.

- Go to Upload -> Upload Points/Annotations (CSV file) in your source, and upload your edited CSV file. You'll get a warning about all previous annotations being overwritten, but that's fine since the CSV file contains the same annotations.

thank you!

[Stephen Chan]

Good to hear you got the training going!

For future reference though, I should point out an easier 'trick' that could've worked in this case. Whenever you change your source's labelset, you'll reset the classifier history for the source (thus resetting the confirmed images requirement) and trigger a new training. You can make this happen by simply adding a label to your labelset (without having the intention to actually use that label), and then removing that same label from the labelset.

I should eventually improve the retraining logic, or implement some kind of  'force retrain' button, but until then I'll endorse the usage of this labelset trick.