Build CefSharp 39.0.0/3.2171.2069 From Source--Version Confusion/Missing Symbols

[shadet...@gmail.com]

I'm trying to recreate the current CefSharp 3.2171.2069 release (https://github.com/cefsharp/CefSharp) from source, including building CEF. I have vs2013(update 4) and vs2012(update 4) installed.

I've followed the directions at https://github.com/cefsharp/CefSharp/wiki/Building-CefSharp (to build cefsharp) which references https://github.com/cefsharp/cef-binary/wiki/Building-Cef-from-source.

Based on those instructions I've downloaded the the automate.git.py script and installed the chromium depot tools (included python 2.7.6).

I run the automate.git.py script: python automate-git.py --download-dir=<my-download-dir> --branch=2171
This downloaded commit 9be541e from 2015-04-13)

This runs to completion and creates the expected binary_distrib folder with
cef_binary_3.2171.51.g9be541e_windows32
cef_binary_3.2171.51.g9be541e_windows32.zip
cef_binary_3.2171.51.g9be541e_windows32_debug_symbols
cef_binary_3.2171.51.g9be541e_windows32_debug_symbols.zip
cef_binary_3.2171.51.g9be541e_windows32_release_symbols
cef_binary_3.2171.51.g9be541e_windows32_release_symbols.zip
(I also did a 64-bit build creating _windows64 versions)

I downloaded cef-binary from https://github.com/cefsharp/cef-binary (master branch) and installed the cef binaries (above) in the cef_binary_3.y.x_windowsXX folders, modified (version 3.2171.51) and ran the build.ps1 script. This ran to completion and generated the cef_redist.x{86|64} nuget packages and cef.sdk.

I downloaded CefSharp v39.0.0 from https://github.com/cefsharp/CefSharp/releases/tag/v39.0.0, installed the generated nuget packages and tried to build all.

I get an error building CefSharp.core that CefAllowCertificateErrorCallback is undefined (Error    1    error C2065: 'CefAllowCertificateErrorCallback' : undeclared identifier    e:\cefsource\cefsharp-39.0.0\cefsharp.core\Internals\ClientAdapter.h    97    1    CefSharp.Core).

In the cef.sdk package from nuget, the symbol is defined in packages\cef.sdk.3.2171.2069\CEF\include\cef_request_handler.h

In my build packages the file (cef_request_handler.h) exists but does not define the symbol. Nor is it in the cef_request_handler.h that came with the cef branch 2171 build.

I'm assuming I have the wrong versions of things but I can make heads or tales of the various repositories and references.

In the cef build (chromium\src\cef) there is a file CHROMIUM_BUILD_COMPATABILITY.txt that references 'chromium_checkout': 'refs/tags/39.0.2171.95'. I don't know where the .51 came from in my cef build and I don't know where 2069 came from in the current CefSharp build.

So... How do I build the equivalent of 39.0.0/3.2171.2069 from source?  Or what am I doing wrong?

[shadet...@gmail.com]

I can add some information...
2069 was the old SVN revision number.
The repository is now git based which is using hex commit numbers (i.e. 9be541e).
The .51 in my post is the number of commits in the branch.

Symbol CefAllowCertificateErrorCallback() seems to have been replaced by a more generic CefRequestCallback() at some point.

SVN release 2171.2069 seems to use the old call.  The latest commit in 2171 (and 2272) use the new call.

Is there any map/conversion from SVN release numbers to git commit numbers?

[tharibo]

Hi.

I'm trying to do the same, but i can't get CEF branch 2171 to compile.

See https://bitbucket.org/chromiumembedded/cef/issue/1633/cant-compile-branch-2171-using-automate.

Did you encounter any problem during CEF compilation yourself?

[Alex Maitland]

`Ceforum` is an excellent resource for such questions.

See http://magpcss.org/ceforum/viewtopic.php?f=6&t=12905&p=25597

If you've already gone to the trouble of building `CEF` then I'd probably just backport the required changes into `CefSharp`, take less time than waiting for another build.
(There isn't supposed to be breaking `API` changes on a release branch, just so happens there is in this case).

It should be fairly easy to backport the changes from `2272`, see https://github.com/cefsharp/CefSharp/commit/6abfb8c05488294c7f3eb7b2c42af6235a1ab2d9

[Alex Maitland]

The only branch I've compiled recently is `2272`, so I can't comment on the state of the `2171` branch specifically.

I have a quick look at the issue you opened, did you get a basic `2171` build working first?

[tharibo]

Thanks for your reply (I don't want to parasit this thread by the way).

I could only get somehow a 2357 build working. 

(Well, cef_client was working (and compatible with MP4), but when I tried to force the use of the build with CefSharp, I had a wild crash. I just figured out I should first make things right: building everything like the last release version, and then only asking questions if necessary).

[shadet...@gmail.com]

Struggled with it for a while...

Windows 7 Ultimate 64bit
Visual Studio 2013 community (update 4)
Windows SDK 8.1

Downloaded Depot Tools from https://src.chromium.org/svn/trunk/tools/depot_tools.zip.
Extracted to c:\depot_tools and added to the system PATH (advanced system settings -> Environment Variables).  I think this is mostly not used because the cef builds download their own depot_tools but it gets you python which you need to kick things off.
       
Build directory e:\cefsource
           
Downloaded automate-git.py from https://bitbucket.org/chromiumembedded/cef/raw/master/tools/automate/automate-git.py (into build dir).

Set enviroment vars (I added them to the system environment vars)
set GYP_GENERATORS=ninja,msvs-ninja
set GYP_MSVS_VERSION=2013
set DEPOT_TOOLS_WIN_TOOLCHAIN=0
Note! DEPOT_TOOLS_WIN_TOOLCHAN is not in the cef instructions but I could not build without it. Found the reference to use it somewhere or other.  This seems to be the magic.

From build dir: python automate-git.py --download-dir=/cefsource/2171 --branch=2717
This ran for a long time. 90-120 min setup/download and 45 minutes compile (6 hyperthreaded 3.7ghz cores).

That ran to completion and created e:\cefsource\2171\chromium\src\cef\binary_distrib\...

You should be able to re-run with a 64-bit build
python automate-git.py --download-dir=/cefsource/2171 --branch=2717 --force-clean --x64-build

[shadet...@gmail.com]

Did get a cef branch 2171 build (automate-git.py) for 32 and 64 bit. Used the generated binaries in cef-binaries master branch (https://github.com/cefsharp/cef-binary/tree/master). Was able to build there and generate local nuget packages.  When I tried to use the generated packages with my projects or with cefsharp 39.0.0 or 39.0.1 (https://github.com/cefsharp/CefSharp/releases) I got the described symbol errors.

[Alex Maitland]

I've actually rewritten most of the instructions at https://github.com/cefsharp/cef-binary/wiki/Building-Cef-from-source

If you have any comments/feedback you can edit the wiki directly, all you need is a `GitHub` account.

You can have the python script reuse the copy of depot tools you downloaded, I taken that approach with the updated tutorial, you shouldn't need to set the DEPOT_TOOLS_WIN_TOOLCHAIN in that scenario.

[tharibo]

OK, so I restarted fresh from an empty directory, following the updated instructions.

Instead of your commandline, I used this .BAT :

@echo off

set DEPOT_TOOLS_WIN_TOOLCHAIN=0

set GYP_DEFINES="branding=Chromium buildtype=Official proprietary_codecs=1 ffmpeg_branding=Chrome"

set GYP_GENERATORS=ninja,msvs-ninja

set GYP_MSVS_VERSION=2013

set Path=%PATH%;H:\cef\depot_tools

set http_proxy=http://<myproxyconfiguration>

set https_proxy=http://<myproxyconfiguration>

set BOTO_CONFIG=H:\cef\.boto

set NO_AUTH_BOTO_CONFIG=H:\cef\.boto

python automate-git.py --download-dir=. --branch=2171 --build-log-file 

The linked log file is a concatenation of the console and the build-debug.log generated by the compilation.

I still have the config.asm compilation error. And I have no idea where it comes from.

Any idea someone?

[Bill Tutt]

Here's the command line I use:

python automate-git.py --download-dir=c:\CEF --depot-tools-dir=C:\xxx\yyy\depot_tools --branch 2171  --build-tests --force-build

(or variations thereof, depending on circumstances I'll add --no-update as well)

Fyi,

Bill

[Bill Tutt]

Your GYP_DEFINES may or may not be causing the problem, I know I don't define any when I did the 2171 build for CefSharp.

I have automate-git.py, and this setupenv.bat file in some directory OTHER than the download directory parameter I pass into automate-git.py.

Here's the .bat file I used to setup the Chromium environment variables for the 2171 build:

@ECHO OFF
REM Determine the directory this file lives in:
SET SETUPENV_DIR=%~dp0

REM Add depot_tools onto path presuming it was extracted to the correct place:
SET PATH=%PATH%;%SETUPENV_DIR%depot_tools

REM Add CMake to path for CEF/CefSharp SLN generation
SET PATH=%PATH%;C:\Program Files (x86)\CMake\bin

REM Ensure that depot_tools doesn't try to install VS from inside Google:
SET DEPOT_TOOLS_WIN_TOOLCHAIN=0

REM Set default generation outputs from gyp/gn:
SET GYP_GENERATORS=ninja,msvs-ninja
SET GYP_MSVS_VERSION=2013

REM Change the CWD to the enlistment this is in
REM so that shortcuts will do the right thing.
cd %SETUPENV_DIR%

Fyi,

Bill

[tharibo]

Ok, next steps for me are to try :

1. No special flags inside GYP_DEFINES

2. Download dir elsewhere than '.'

Any other idea to try?

Thanks all for your support.

[shadet...@gmail.com]

I'm mostly answered my own questions and succeeded in building 39.0.0/3.2171.2069 with proprietary codecs (read mp3) support.

My build environment is windows 7 with visual studio 2013 community (update 4).

CEF branch 2171 development has gotten out of sync with CefSharp.
To build CEF 3.2171.2069 you need to build from commit 395cdea (2015-03-17).
See https://bitbucket.org/chromiumembedded/cef/branch/2171

After commit 395cdea symbol CEF_REVISION was changed to CEF_COMMIT_NUMBER (reflecting the change from SVN to GIT).  This will cause a compile error building the cef binaries for cefsharp.

As of commit 9be541e (2015-04-13) several methods get redefined (including CefAllowCertificateErrorCallback()) causing additional errors.

The same is true for branch 2272, where you will need commit bc845db (2015-03-12) to avoid the same problems (I didn't build 2272 but did check the commits for the same symbol problems).

I found the build process to be very fragile. A few time I seemed to get caught up in build errors that couldn't be cleared except by deleting everything and starting over.  At one point I was building on two machines (completely separate) and getting different errors.

You can install the depot_tools as per Building-Cef-from-source instructions, or the actual build script will download them automatically.  In any event you need python 2.7,x to run the various scripts, which is part of depot_tools. I downloaded them and installed them in c:\depot_tools and added c:\depot_tools to my path.  Alternatively you can just install python 2.7.x and make sure it's in your path.

Create the build location.

This is somewhat arbitrary.  I ran into problems at one point with too-long path names when I had the build directory buried deeper into the hierarchy.  However I was using a full path name for the --download-dir argument (see below).  I moved the build directory to the root of the drive and changed --download-dir to a relative value (no \'s) at the same time and haven't had a problem, so I don't know if the relative directory was enough to fix the problem.

mkdir e:\cefsource
cd cefsource
mkdir 2171
cd 2171

Get CefSharp Binary 'master' release from https://github.com/cefsharp/cef-binary. I just downloaded the .zip file (https://github.com/cefsharp/cef-binary/archive/master.zip) and extracted in e:\cefsource (creating a cef-binary-master dir)

Get CefSharp source, release 39.0.0, from https://github.com/cefsharp/CefSharp/releases/tag/v39.0.0. I just downloaded the zip file (https://github.com/cefsharp/CefSharp/archive/v39.0.0.zip) and extracted into e:\cefsource (creating a CefSharp-39.0.0) dir.

Download automate-git.py from https://bitbucket.org/chromiumembedded/cef/raw/master/tools/automate/automate-git.py

You need a couple of environment variables set. These can be in your default environment (Advanced System Settings -> Environment variables), set from the command line, our built into a script.  Your choice.

set GYP_GENERATORS=ninja,msvs-ninja
set GYP_MSVS_VERSION=2013
set DEPOT_TOOLS_WIN_TOOLCHAIN=0

For including proprietary codecs (i.e. mp3). NOTE! IMPORTANT! don't use quotes in the set GYP_DEFINES statement.  This is windows specific. Quotes will prevent GYP_DEFINES from being properly parsed and you will get nothing for your efforts (including errors or warnings).

set GYP_DEFINES=proprietary_codecs=1 ffmpeg_branding=Chrome

I could not get the compile to run without setting DEPOT_TOOLS_WIN_TOOLCHAIN=0. The directions on the cefsharp-from-source web site (and a comment from the author in this thread) indicate that you can with the --depot-tools-dir=... argument to automate-git.py (i.e c:\depot_tools) you can avoid the extra download and you won't need the DEPOT_TOOLS_WIN_TOOLCHAIN setting. I did not test this scenario.

Create a build script to save some typing

build-2171.bat:

    set GYP_GENERATORS=ninja,msvs-ninja
    set GYP_MSVS_VERSION=2013
    set DEPOT_TOOLS_WIN_TOOLCHAIN=0

    set GYP_DEFINES=proprietary_codecs=1 ffmpeg_branding=Chrome
    python automate-git.py --download-dir=2171 --checkout=395cdea %*

Create a local NuGet repository directory to hold the packages you will be creating.

    mkdir NuGet (assuming you are still in e:\cefsource)

build-2171

    This generates a 32 bit build (cef_binary_3.2171.2069_windows32) in 2171\chromium\src\cef\binary_distrib. This needs to be put over in e:\cefsharp\CefSharp-39.0.0 with the name cef_binary_3.y.z_windows32.
    There are several ways to do it.  I deleted the initial empty cef_binary_3.y.x_windows32, copied cef_binary_3.2171.2069 over, and renamed it.
    The binary_distrib dir will be cleared before each build so you need to copy the dir out before staring the 64 bit build.

build-2171 --force-clean --x64-build

    I'm not sure if --force-clean is necessary. Maybe just --force-build, but I've been using --force-clean.  Had a situation where git was complaining about checksums and had to use --force-clean --force-clean-deps to resolve.

    As above, copy/move cef_binary_3.2171.2069_windows64 to the CefSharp-39.0.0 dir.

build the cefsharp cef-binary packages

    in cef-binary-master dir...
    edit build.ps1

        Edit the $CefVersion line -> 3.2171.2069
        Edit the $CefPackageVersion = "3.2171.2069-mp3"

        I used 3.2171.2069-mp3 for the CefPackageVersion to distinguish from the offical release packages (mp3 = proprietary_codecs/mp3 support). This could be whatever you want but it is what will show up as the nuget package versions. Adding "-..." to the package version will change the package status from stable to pre-release in NuGet. This is covered a little more below.

    A note on running powershell.  The default on my windows 7 was to disallow running scripts completely.  This can be defeated by starting powershell as administrator and entering: 'set-executionpolicy unrestricted' (or bypass. unrestricted will allow then to run after a confirmation prompt. bypass should let them run without a prompt).  I set mine to bypass but it would still prompt to run until I went into properties for the script (build.ps1) and clicked on the unlock button.  There are no shortage of warnings about how vulnerable your system will become after doing this.

    Run cmd
    e:
    cd \cefsource\cef-binary-master
    powershell .\build.ps1 vs2013
    powershell .\build.ps1 nupkg-only

    copy Nuget\*.nupkg to e:\cefsource\NuGet

Build CefSharp

    in CefSharp-39.0.0

    The build.ps1 script here would not run successfully for me. I would get a slew of error messages resembling " E:\cefsource\CefSharp-39.0.0\CefSharp.Core\CefSharp.Core.vcxproj(27,3): error MSB4019: The imported project "E:\Microsoft.Cpp.Default.props" was not found. Co nfirm that the path in the <Import> declaration is correct, and that the file ex ists on disk." and unsuccessful build.  Fortunately you can easily build from the CefSharp3.sln file in vs2013.

    Open CefSharp3.sln in visual studio 2013. Right-Click the Solution CefSharp3 line in the solution explorer window and pick Manage NuGet Packages for Solution. Add a new package source pointing at the NuGet directory.

    Settings -> NuGet Package Manger -> Packages Sources

    First hit Restore on the top yellow line (if it's there) to restore the original packages. If you don't do this, the build will re-load them for you and wipe out the changes we will be doing below. I had to enable the nuget package source (Settings) to get this to work. They also won't get deleted when you load the new packages and you will end up with mysterious missing package errors.

    "+" in upper right--Adds a generic entry "Package source"
    Click on "Package source" in the Available package sources window
    Edit the name (whatever you want, I used local-cefsharp)
    Edit the Source: e:\cefsharp\NuGet
    Update
    Ok

    It's probably good to clear the package cache while you are in settings (click NuGet Package Manger on the left and Clear Package Cache at top middle), in case you've been trying things over and over with different versions like I was.

    Add your newly built packages to the project

    Expand the Online section on the left of the NuGet window
    Pick you new source (local-cefsharp) -- Says No items found
    Change "Stable Only" to "Include Prerelease" in the drop down at the top
        It appears that by adding "-,,," to the end of the cef version number it becomes
        a pre-release.  I.e. 3.2171.2069-mp3.  Without the -mp3 it's "Stable"
    Pick cef.redist.x64 -> Install
        Check only the following:
            CefSharp.OffScreen.Example
            CefSharp.Test
            CefSharp.WinForms.Exmaple
            CefSharp.Wpf.Example
        OK
        This will add the new package and remove the old one we just "restore"d
    Pick cef.redist.x32 -> Install
        ... same as x64
    Pick CEF.SDK -> Install
        Check only the following:
            CefSharp.BrowserSubprocess.Core
            CefSharp.Core
        OK
    Close NuGet

    BUILD -> Batch Build -> Select All -> Rebuild
    Should finish with 58 projects built.

    Edit build1.ps1
        Change $Version = "39.0.0-mp3"
        Change $RedistVersion (if you want). I used 3.2171.2069-mp3

        If $RedistVersion had -mp3 and $Version doesn't the nupkg build will complain about mixing stable and pre-release version.  It also helps make sure you are using the right versions.

    See notes about powershell from the previous section. Also this script expects powershell 3. My windows have version 2 and would throw errors on "-ErrorAction:Ignore" statements in the script.  As an alternative to updating to version 3 you can change Ignore to SilentlyContinue. I installed Powershell 3.

    powershell .\build.ps1 vs2013 (default is vs2012 + nupkg-only)
    powershell .\build.ps1 nupkg-only
                                                                      
That's all there is too it! Good Luck!

[tharibo]

Wow, that was a pretty comprehensive and useful post!

I could finally recompile CEFSharp v 39.0.2 with MP3/MP4 support by following your instructions thoroughly.

It took time, but I noticed that --branch and --checkout options are not compatible. You have to use one or the other (don't ask why I tried the two at the same time).

Thank you for your support, shadet...@gmail.com !

[Raimund Bauer]

I tried to follow your instructions, but automate-git.py failed with an error for me.
Last lines of the logfile are:

[7188/12497] ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64_x86\cl.exe" /nologo /showIncludes /FC @obj\third_party\WebKit\Source\platform\heap\blink_platform.CallbackStack.obj.rsp /c ..\..\third_party\WebKit\Source\platform\heap\CallbackStack.cpp /Foobj\third_party\WebKit\Source\platform\heap\blink_platform.CallbackStack.obj /Fdobj\third_party\WebKit\Source\platform\blink_platform.cc.pdb
FAILED: ninja -t msvc -e environment.x86 -- "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64_x86\cl.exe" /nologo /showIncludes /FC @obj\third_party\WebKit\Source\platform\win\blink_platform.SystemInfo.obj.rsp /c ..\..\third_party\WebKit\Source\platform\win\SystemInfo.cpp /Foobj\third_party\WebKit\Source\platform\win\blink_platform.SystemInfo.obj /Fdobj\third_party\WebKit\Source\platform\blink_platform.cc.pdb
j:\cefsource\chromium\src\third_party\webkit\source\platform\win\systeminfo.cpp(58) : error C3861: 'IsWindowsVistaOrGreater': identifier not found
LINK : warning LNK4075: ignoring '/INCREMENTAL' due to '/LTCG' specification

Generating code

Finished generating code

ninja: build stopped: subcommand failed.
ninja: warning: multiple rules generate gen/about_credits.html. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]

with a little bit more info visible on the console:

-------- Running "ninja -v -Cout\Debug cefclient" in "j:\cefsource\chromium\src"...
Traceback (most recent call last):
  File "j:\src\cef_automate-git.py", line 873, in <module>
    if options.buildlogfile else None)
  File "j:\src\cef_automate-git.py", line 59, in run
    stderr=subprocess.STDOUT, stdout=f)
  File "c:\prg\depot_tools\python276_bin\lib\subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja', '-v', '-Cout\\Debug', 'cefclient']' returned non-zero exit status 1

any idea what I'm doing wrong?

[tharibo]

Hi.

What instructions did you follow?

What did you do *exactly*?

--
You received this message because you are subscribed to a topic in the Google Groups "CefSharp" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cefsharp/BJLMXl9c204/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cefsharp+u...@googlegroups.com.
To post to this group, send email to cefs...@googlegroups.com.
Visit this group at http://groups.google.com/group/cefsharp.
For more options, visit https://groups.google.com/d/optout.

[Raimund Bauer]

The instructions of the parent-post in this thread here.
GYP Settings:

set GYP_GENERATORS=ninja,msvs-ninja
set GYP_MSVS_VERSION=2013
set GYP_DEFINES=branding=Chromium buildtype=Official proprietary_codecs=1 ffmpeg_branding=Chrome

and --checkout=395cdea as parameter to automate-git.py.

32bit build on Windows 7 professional.
Any other information I should provide?

[tharibo]

You should follow precisely the instructions from shadetree247 in his last post from this thread. It only worked for me when I was precise enough, and not disregarding an instruction or the other, or adding my own.

[ваня бахчеев]

I just finished rebuild of CEF# 41 with proprietary codecs thanks to your instruction (https://groups.google.com/d/msg/cefsharp/BJLMXl9c204/HMJlp8mZzF0J).

There is some changes which can be important if anyone try to repeat guide from shadetree247. Also please note chromium 41 cannot be built using VS2015 because some things deprecated or not available, you'll need VS2013.

1) Commit assumption for chromium is wrong — for CEF# 41.0.1 you must use bda8dc7

2) After chrome was built, instead of downloading cef-binary, clone repository https://github.com/cefsharp/cef-binary and checkout proper commit for your CEF# (41.0.1 — 8a2061b)

3) Download and install cmake, it is needed for build.ps1 for cef-binary

4) I have error The imported project "E:\Microsoft.Cpp.Default.props" was not found. To correct this, edit build.ps1 file and replace first $MSBuildExe string with

$MSBuildExe = join-path -path (Get-ItemProperty "HKLM:\software\Microsoft\MSBuild\ToolsVersions\12.0").MSBuildToolsPath -childpath "msbuild.exe"

12.0 is a version of VS, if you're going to compile with other version, replace 12.0 with your version.