Subject: Re: [Nmlrg] I-D Action: draft-jiang-nmlrg-network-machine-learning-01.txt

From: Brian E Carpenter <[email-masked]>
Date: Tue, 3 May 2016 11:38:35 +1200

Hi,
...
> There's also a htmlized version available at:
> https://tools.ietf.org/html/draft-jiang-nmlrg-network-machine-learning-01

I think this is a very good draft to act as background for our discussions.
I am really a beginner in this area, so I have an elementary question.

First, a quote from the minutes of the 1st meeting:

> [Jeffrey ?? from SDNLabs] Should be careful about the black box model 
> prediction. It is some fundamental concern because we don't understand
> how it works. It is not reliable.
> [Albert] Fully agree. If we had a model, then we should use that to solve 
> the problem. But there are scenarios that we don't have all the information 
> (e.g. underlying, topologies), so we don't have the model, ML is suitable 
> for those scenarios. We're aware of that ML is not a solution for everything.

Second, a quote from the draft:

>       Unsupervised learning  The given input are not labeled.  It leaves
>       the machine learning mechanism itself to find structure in its
>       input and output.

Third, a note that it is well known that blind (unsupervised) data analysis
of large datasets will lead to a definite fraction of false correlations.
(If you are interested in a mathematical proof of this statement, see
https://www.researchgate.net/publication/297656557_The_Deluge_of_Spurious_Correlations_in_Big_Data )

So, finally to my question. How could we avoid false results from
unsupervised machine learning, which could be very harmful to a running
network? Or should we immediately conclude that unsupervised learning
is a bad choice for network machine learning?

Regards
   Brian Carpenter

---

From: "Liubing (Leo)" <[email-masked]>
Date: Wed, 4 May 2016 01:58:38 +0000

Hi Brian,

I think this is a very meaningful question that worth to be explored deeply.
Following comments is some quick response. Since I am also not an ML expert, the comments is for discussion rather than answers or conclusions.

> -----Original Message-----
> From: nmlrg [mailto:[email-masked]] On Behalf Of Brian E
> Carpenter
> Sent: Tuesday, May 03, 2016 7:39 AM
> To: [email-masked]
> Subject: Re: [Nmlrg] I-D Action:
> draft-jiang-nmlrg-network-machine-learning-01.txt
> 
> Hi,
> ...
> > [Albert] Fully agree. If we had a model, then we should use that to
> > solve the problem. But there are scenarios that we don't have all the
> > information (e.g. underlying, topologies), so we don't have the model,
> > ML is suitable for those scenarios. We're aware of that ML is not a solution
> for everything.

> Third, a note that it is well known that blind (unsupervised) data analysis of
> large datasets will lead to a definite fraction of false correlations.
> network? 

[Bing] As I understand, we are not able to totally "avoid" the false results, either for unsupervised or supervised learning; and I think the above mentioned "black box" also applies to both of the two. 
So far, the ML approach might mostly work as an assistant for human decision. I guess it is far from mature to let the ML make decision autonomically for most use cases.

> Or should we immediately conclude that unsupervised learning is a
> bad choice for network machine learning?

[Bing] I think it depends on what the application is and what the training data is. Supervised learning implies the training data contains a set of samples and the targeted result (e.g. classification), when new sample comes in, it could predict the result for it. I think it only applies part of the network use cases. 
Unsupervised learning should also have its own position in network usage. Typical usage is clustering and outlier analysis. E.g., detecting some anomaly in a set of events.

Best regards,
Bing

> Regards
>    Brian Carpenter
> 
> _______________________________________________
> nmlrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/nmlrg

---

From: Sheng Jiang <[email-masked]>
Date: Wed, 4 May 2016 06:16:02 +0000

Hi, Brian and Bing,

Before we discuss unsupervised learning, more important is the assessment of false results. Could we establish an evaluating system to measure a result into false or right? If yes, even the learning process, either supervised or unsupervised, we could feedback the assessed results back as new training data for incremental training. Then, even unsupervised learning may become supervised learning if there are sufficient feedback data. With a good evaluating system, a close control loop that supports autonomic management could be fulfill.

However, as we all know, for most complex issues, an complete evaluating system that does not make mistakes/faults are even more challenge.

Best regards,

Sheng

>[Bing] As I understand, we are not able to totally "avoid" the false results,
>either for unsupervised or supervised learning; and I think the above
>mentioned "black box" also applies to both of the two.
>So far, the ML approach might mostly work as an assistant for human decision.
>I guess it is far from mature to let the ML make decision autonomically for
>most use cases.
>
>[Bing] I think it depends on what the application is and what the training data
>is. Supervised learning implies the training data contains a set of samples and
>the targeted result (e.g. classification), when new sample comes in, it could
>predict the result for it. I think it only applies part of the network use cases.
>Unsupervised learning should also have its own position in network usage.
>Typical usage is clustering and outlier analysis. E.g., detecting some anomaly
>in a set of events.
>
>Best regards,
>Bing
>
>> Regards
>>    Brian Carpenter
>>