PSA: Largely reduced WebRTC checkout size!

[Henrik Kjellander]

TL:DR; Standalone WebRTC checkouts no longer includes a full Chromium copy, saving ~9GB of download and disk space. For legal reasons, the Google Play Services SDK licence prompt is unfortunately back (once).

Ever since we moved from Subversion to Git for WebRTC, we've forced to download a copy of Chromium in order to be able to build WebRTC. The reason for that is that we don't have a dedicated team to work full-time on the build toolchain and to support the 5 platforms + all other build configurations. Instead, we've been using the excellent work by the Chromium project for that, to a high price in terms of download traffic and disk space.

Luckily, I've found time to work focused on addressing this longstanding issue (webrtc:5006) the past few weeks and it's now completed after 22 CLs...

 

From 177567c (#15754) there's no longer any Chromium checkout in your repo (or any sync_chromium.py or setup_links.py file). All external dependencies are now synced the traditional way using the DEPS file.

Too good to be true?

Well, yes... We're still checking out parts of Chromium, namely so called "Git subtree mirrors" of a couple of directories:

• base (75MB) - only to be used by Android unit tests (and the libevent dependency)
  
• build (34MB) - contains most of the build toolchain.
• testing (27MB) - test-related scripts
  
• third_party (3.2GB) - lots of third_party code and foremost the BUILD.gn files for all. This is the obvious next-step to look at.
• tools (123MB) - tools required for build and development.

These are synced down similar to regular dependencies defined in the DEPS file. They're auto-rolled similar to what we used to to with the chromium_revision using a script that runs periodically on a bot (every 3 hours).

The tools directory here is also the reason why there's a new tools-webrtc directory introduced recently, to contain our own tools (to avoid colliding with Chromium's tools).

What does this mean to the total checkout size? 

Here are some total numbers on fresh checkouts I measured:

• Linux: 15GB --> 6.4GB (-8.6GB)
  
• Linux (with Android): 25 GB --> 16 GB (-9GB). 8.4GB of the remaining 16GB is Android SDK+NDK.
  
• Mac (with iOS): 14 GB --> 5.6GB (-8.4GB)
  

What's the downside?

Unfortunately we'll get many new directories in tools/ and third_party/ that are irrelevant to WebRTC development (but might be in the future). We'll have to live with that for now until somebody untangles every dependency into these directories and either sets up Git mirrors for each one of them or is signing up on maintaining copies of them (that we could host in our own repo).

This also brings back the annoying Google Play Services SDK license confirmation dialog upon your first sync, however it should no longer have the problems described in webrtc:5578.

What do I need to do?

When you've synced past 177567c (#15754), the src/chromium dir will show up as unversioned in your checkout. Just delete it and reclaim that large amount of disk space (I've seen that Git repo grow to >100GB over time).

You might also get some stuff moved into a  _bad_scm directory in the checkout root. That can also be deleted to reclaim more space.

Questions? Problems? 

I hope this migration will be reasonably painless, but if you get syncing errors like:

Syncing projects: 100% (45/45) src/third_party/robolectric/robolectric   

src/third_party/android_tools/ndk (ERROR)

----------------------------------------

[0:00:23] Started.

----------------------------------------

Error: 45> 

45> ____ src/third_party/android_tools/ndk at 26d93ec07f3ce2ec2cdfeae1b21ee6f12ff868d8

45>  You have unstaged changes.

45>  Please commit, stash, or reset.

for this or other directories configured in the DEPS file, please just delete them and retry the sync.

Don't hesitate to reply to this thread or reach out to me directly if you have issues.

[Henrik Kjellander]

I discovered some issues with old symlinks staying around as well, blocking the sync, so if you experiences this, make sure to delete them.

[Alexandre GOUAILLARD]

Great job! Happy to get that in time for Xmass. thanks for finding the time to pushing it through.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAEtuHR06s_16CXGSdRXfG2GxFq6MJL6tdSBr-MbyPDBVCvBR8g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[jba...@webrtc.org]

This is great news, thanks a lot!

[Henrik Kjellander]

After dealing with some minor fixes after the change, all that's now broken is the Android trybots. Please follow https://bugs.chromium.org/p/chromium/issues/detail?id=676617 for progress on that.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/bf6431fb-7422-4783-a228-3fd13a7efd28%40googlegroups.com.

[radioman . lt]

A step in the right direction, well done.

[minhao zheng]

excuse me, can you make a webrtc repository on github? and make gn buid command  through github source?becanuse of great wall, we waste most time to get  and compile the source

在 2016年12月22日星期四 UTC+8下午6:11:59，Henrik Kjellander写道：

[Henrik Kjellander]

On Fri, Dec 30, 2016 at 9:08 AM, minhao zheng <zheng...@gmail.com> wrote:

excuse me, can you make a webrtc repository on github? and make gn buid command  through github source?becanuse of great wall, we waste most time to get  and compile the source

I'm afraid we cannot, since we're relying on a lot of customized stuff the Chrome infrastructure team has done on chromium.googlesource.com.

Even we WebRTC would be moved to GitHub, the majority of stuff downloaded via the DEPS file would still be from chromium.googlesource.com.

  

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/63914b01-d042-4068-a836-531febc170bd%40googlegroups.com.