Please help design a NILM competition!

[Jack Kelly]

Dear Energy Disaggregation mailing list folks,

Has disaggregation accuracy improved since the 1980s?  Which algorithms are most accurate for a given use-case?  Which (if any) use-cases are well served by NILM already?

It's pretty much impossible to answer any of these questions with confidence (unless you only consider the tiny number of algorithms for which you have access to executable code).  We can't directly compare published results across papers because, when testing the disaggregation accuracy of NILM algorithms, each paper uses different datasets, different metrics, different pre-processing, etc.

This means that we can't measure progress over time.  Nor can we decide which NILM algorithms are most promising and which might be dead-ends.

These are bad problems.  Let's work towards fixing them.

Some other machine learning communities have had great success running yearly competitions.  For example, the ImageNet "Large Scale Visual Recognition Challenge" has been running yearly since 2010.  Some regard this competition as having played a crucial role in the recent dramatic increase in the accuracy of image classification algorithms.

The idea of running a NILM competition has been rumbling around for several years. But designing and implementing a NILM competition is hard.  The community uses sample rates ranging from monthly to MHz.  No single metric is informative for all use-cases.  Collecting ground truth data (the power demand of individual appliances) is expensive and time-consuming.

Maybe we can pull this off.  The first step is to decide on a design which will work for everyone.

To give us something concrete to debate, we'll outline one way this could work.  This is not meant to be definitive!  Think of this as the DNA for a clumsy, inefficient animal 500 million years ago.  Together, we need to evolve this design into an elegant, efficient beast, well adapted to its environment.

Please shoot holes in this proposal!  What won't work for you?  What's impractical?  What's unfair?  What opens the competition up to cheating?  How can we make the competition more attractive to researchers?  How can we make the competition more informative for the community? How can we simplify the process?

The draft proposal is available on Google Docs. I've linked to a Google Doc rather than copying-and-pasting the proposal into this post so that we can update the proposal as the discussion develops. Please add your comments either to this mailing list discussion; or to the Google Doc (please sign your comment with your name; unless you deliberately want to be anonymous); or if you want to keep your comment private then email me.

Thanks,

(in no particular order) Jack, Mario, Oli, Stephen, Grant, Marco, Peter

[Jack Kelly]

I will be working full-time on the NILM algorithm competition from now until the end of December 2016.  The first step is to consult with the community to make sure that the NILM competition is acceptable to as many people as possible!  To systematically collect feedback about the design of the competition, I have drafted a survey about the design of the NILM competition.  I plan to launch the survey on the morning of Friday (30th September).  Before Friday, I'd be really eager to hear feedback on the survey itself.  For example: is the survey missing any vital questions?  Do some questions not provide sufficient options?  Do some questions not make sense?!

Please note that, prior to Friday, the aim is to get feedback on the design of the survey itself.  So please don't actually submit any answers yet!  I'll send another message to the group when the survey is ready to accept answers.

It's probably best to provide feedback about the survey in public, here on the Google Group.  If you want your feedback to be private then, by all means, email me directly at jack....@imperial.ac.uk!

And please do get in touch if you have feedback on any aspect of the proposed NILM competition.

[Jack Kelly]

Dear Energy Disaggregation folks,

The survey on the design of a NILM competition will be launched on Monday morning.  So please send your feedback on the design of the survey by Sunday night!  

Thank you!

Jack

[Jack Kelly]

Dear Energy Disaggregation folks,

Please help us to design a competition for energy disaggregation (NILM) algorithms by filling in this survey!

Thank you,

Jack

[Martin Chileshe]

Hi Jack, are you still on the NILM challenge?

[Jack Kelly]

Hi Martin, I'm no longer working on NILM, I'm afraid. I'm now building a non profit entirely focused on using open science to mitigate climate change. We're working on solar PV nowcasting. Http://openclimatefix.org 

--
You received this message because you are subscribed to the Google Groups "Energy Disaggregation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to energy-disaggreg...@googlegroups.com.
To post to this group, send email to energy-dis...@googlegroups.com.
Visit this group at https://groups.google.com/group/energy-disaggregation.
To view this discussion on the web, visit https://groups.google.com/d/msgid/energy-disaggregation/8e49ea17-24cb-4f0a-9c94-500b625bcdf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.