OWASP Mobile Top Ten 2015 Data Synthesis Document

[Jonathan Carter]

Hi Everyone,

As discussed at the last meeting, I have produced an initial data synthesis document based on data submitted by participants.  You can download the synthesis here:

https://www.owasp.org/images/b/b5/OWASP_Mobile_Top_Ten_2015_-_Synthesis_Document_v0.1.pdf

It's not complete but it's a good first start and we'll want to add key observational material from Paco and from the other vendors not finished off.  See you on tomorrow's call!

[Jason H]

Hey JC,

Thanks!

Attached are two more slides for more data sets I finished parsing (Denim, Secure Net) Please update to v0.2 with those. VC and WH are still in progress, hopefully finished tomorrow. 

--
You received this message because you are subscribed to the Google Groups "OWASP Mobile Top 10 Risks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to owasp-mobile-top-1...@owasp.org.
For more options, visit https://groups.google.com/a/owasp.org/d/optout.

--

- Jason Haddix
Director of Penetration Testing
http://vizualize.me/jhaddix

[Paco Hope]

All,

On the 24th, I sent the email below to the mailing list from my pa...@cigital.com address. I cannot find it on the online archives, I got no bounce back, and I got no copy of it to myself. Is there some moderation in action on the mailing list? How did it just get eaten by Google Groups? This is very distressing. I realised that I had received no conversation or discussion on it and I thought that was odd. Now I see that it simply did not get delivered.

I was sick with the flu all of last week, so my ability to chase non-functioning mailing lists was fairly limited.

Paco

All,

TL;DR: shiny pretty pictures and a wiki for collecting our thoughts: https://www.owasp.org/index.php/Projects/OWASP_Mobile_Security_Project_-2015_Scratchpad 

I stayed home sick today and spent the day in bed with my laptop analysing the data. If Dropbox is working automagically, then I saved all my analysis spreadsheets and such into the MTTData folder. If not, I’m happy to upload them.

First, I took 6 of the data sets that seemed to report raw vulnerabilities and I did my best to categorise them against the 2014 Top Ten. Obviously there may be some discrepancies, which is why I have saved it all in spreadsheets. Someone reclassifying things should be able to have pivot tables automatically update. There were some clumsy things, but largely it works. I didn’t do Arxan, IBM, or LEXSI because they were hard for me to coerce into the old top ten format. In the case of LEXSI, they reported percentages instead of raw vuln counts.

The data came out looking like this:

Column1	Bugcrowd	Cigital	Denim	HackLabs	Pure Hacking	White Hat	Totals	Bugcrowd Pct	Cigital Pct	Denim Pct	HackLab Pct	Pure Hacking Pct	White Hat Pct	Totals2
M4: Unintended Data Leakage	49	14	166	8	5	282	524	23.56%	6.28%	44.03%	34.78%	23.81%	30.10%	29.29%
M2: Insecure Data Storage	35	77	90	3	3	102	310	16.83%	34.53%	23.87%	13.04%	14.29%	10.89%	17.33%
Other	9	21	74	0	0	147	251	4.33%	9.42%	19.63%	0.00%	0.00%	15.69%	14.03%
M3: Insufficient Transport Layer Protection	26	20	1	5	6	89	147	12.50%	8.97%	0.27%	21.74%	28.57%	9.50%	8.22%
M10: Lack of Binary Protections	35	17	0	1	2	92	147	16.83%	7.62%	0.00%	4.35%	9.52%	9.82%	8.22%
M5: Poor Authorization and Authentication	29	33	18	0	0	55	135	13.94%	14.80%	4.77%	0.00%	0.00%	5.87%	7.55%
M1: Weak Server Side Controls	2	0	7	6	2	96	113	0.96%	0.00%	1.86%	26.09%	9.52%	10.25%	6.32%
M6: Broken Cryptography	7	32	0	0	0	37	76	3.37%	14.35%	0.00%	0.00%	0.00%	3.95%	4.25%
M9: Improper Session Handling	0	1	19	0	3	35	58	0.00%	0.45%	5.04%	0.00%	14.29%	3.74%	3.24%
M7: Client Side Injection	11	6	2	0	0	0	19	5.29%	2.69%	0.53%	0.00%	0.00%	0.00%	1.06%
M8: Security Decisions Via Untrusted Inputs	5	2	0	0	0	2	9	2.40%	0.90%	0.00%	0.00%	0.00%	0.21%	0.50%
Total	208	223	377	23	21	937	1789

The idea is to calculate the numbers 2 ways. First is just by raw numbers. The second is by percentages. The idea of looking at percentages is to see that, in fact, many contributors were approximately 23% M4, whether they had many contributions or just a few.

Running it both ways, though (you can see the graphs attached) did not yield dramatically different results. M7 and M8 are very rarely reported at all. The top 4 categories account for 3/4 of all findings reported. One of the conclusions I draw from this is that changing 3 or 4 of the least-used categories won’t bother many people. Those categories are barely used at all.

I’m starting to collect data on that scratch pad. If people don’t see my spreadsheets in the MTTData folder on Dropbox, let me know. I’ll figure out what’s gone wrong.

Paco

+Paco Hope, CISSP, CSSLP Principal Consultant, Cigital Build Security In

Mobile: +44 7985 419 802 Follow me: @pacohope LinkedIn: pacohope

[Paco Hope]

Here’s another email I sent on Sunday the 22nd. Again from pa...@cigital.com. Again, hasn’t shown up on the mailing list or in the Google groups. No wonder nobody is replying to me.

Can someone please look into what’s going on here?

Paco

[Jason H]

Yeah i did not receive these till this morning. There isn't any moderation on the group as far as im aware....

[Paco Hope]

All:

I have some tentative conclusions from the data. I’ve scribbled them on the wiki scratchpad. They include:

• 5 Conclusions Drawn From Data
  • 5.1 At Least One New Category Is Needed
  • 5.2 The Bottom 5 Categories account for 25% Or Less
  • 5.3 The Existing Buckets are Hard To Use
  • 5.4 Some Topics that Show Up But Are Hard To Place
    • 5.4.1 Code Level Findings

FYI,

Paco

[Jonathan Carter]

Please keep in mind that Paco's opinions are solely his alone and do not represent the group. We are currently adding additional data that has not been finalized quite yet.

--

[Colin Watson]

Classification is always a challenge, and there is never a single good answer.

When that other more well-known Top 10 was being discussed, I always wanted to know what the 11-20ish items were, to give transparency to the data collection and ranking. I think it is reasonable to know what they are for mobile? 

This is also very useful for the control viewpoint. One mobile control might address say many risk items e.g. 7, 11, 12, 13, 14, 15, 16, 17 and 27, and so might be the most useful single control, even though it only addresses one lower item of the "top ten".

Colin

[mut Tonny]

@Jonathan, if any of Paco's options are found fit for the Top Ten, then i see no reason why they are not includes on the list. Lets analyse all data sets and decides what to include on the Top ten and others we can have them on the 11-20th oist

[Jonathan Carter]

I'm not commenting on whether or not his analysis is valid. I'm simply pointing out this his opinions are based on an analysis that has not been agreed upon by the group. It is not the authoritative source of what the group believes to be true about the data presented thus far.

[Jim Manico]

•Everyone• speaks for themselves and everyones opinion is valid. Paco has been an objective OWASP'er for many years and I for one deeply value his opinion.

Regards,

--

Jim Manico

@Manicode

(808) 652-3805

On Mar 3, 2015, at 12:38 PM, mut Tonny <mutt...@gmail.com> wrote:

@Jonathan, if any of Paco's options are found fit for the Top Ten, then i see no reason why they are not includes on the list. Lets analyse all data sets and decides what to include on the Top ten and others we can have them on the 11-20th oist

--

[Jason H]

Jonathan and Paco,

I don't see anything wrong with what Paco did. We asked people to independently evaluate the data and he did. We all understand that any analysis right now is not the end result of the research, just more data points. He is also not disrespecting any company or data set by pointing out their particular methodology or categorization. 

We have all voiced our opinions about the data thus far. None of these opinions are the final wording of the project yet.

Everyone knows this is hard.  We've made tremendous progress in the last month. Let's try not to set it back interpersonal communication issues. 

[Jonathan Carter]

I genuinely don't think Paco did anything wrong in the sense that he did his own separate analysis with a select few data sets.  It presents as fairly polished and I don't want people to think that this was reviewed and consensus is there because it's definitely not.

This really isn't that big of a deal.  It might be coming off that way because of the fact that we're communicating via email.