windows build / recent openfst

[Miroslav Janosik]

Hello

I'm building latest version of Kaldi in Visual Studio 2015 and I have noticed that I can't build it with OpenFst 1.3.4 anymore (missing type DefaultCacheStore, it is present in OpenFst 1.4.1).

I have read here on forums that Kaldi should be compatible with latest version of OpenFst (1.6.1 mentioned here https://groups.google.com/d/topic/kaldi-developers/l3A1ISP4XLE/discussion, current latest would be 1.6.2 now), is that correct?

So I'd like to build OpenFst 1.6.2 in Visual Studio 2015 but I realized that it is a pain - it is not officially supported, there is no easy way to produce 'solution file' for it and so on. Last supported version I found was 1.4.1 - https://github.com/edobashira/openfstwin-1.4.1 with Visual Studio 2013 ) and there I'm getting strange error 'not enough disk space' when generating .lib file...

So my question is - is there any other developer using VS2015 + latest OpenFst to give me the working solution file? If not, would be someone interested in me publishing the solution file for it when I succeed?

  with regards
    Miro Janosik

[Jan Trmal]

There is a substantial amount of work to be done to make the openfst 1.6.2 compile under windows (and verify it works).

I converted the paul dixon's changes into a patch and then applied on 1.6.2 and resolved the rejections. the result compiles under VS2017 and the tests run fine. But this of course does not include the latest packages/modules

You can find it here
https://github.com/jtrmal/openfstwin/tree/openfstwin-1.6.2

you can test it -- I'd be very happy to include any PRs from you if you succeed.

IIRC, VS2015 had problems in the C++11 implementation and the compilation of 1.4.x was causing the compiler to crash, that was the reason I made the howto using openfst 1.3.4. I didn't test openfst 1.6.2 under VS2015 (and it has very very very low priority on my todo list, given the issues I had with openfst 1.4.1)

As I said before, I keep working on this, but my time (and patience) is limited, especially because I don't use windows and I do not have an easy access to a windows machine with reasonable specs).

y.

--
visit http://kaldi-asr.org/forums.html to find out how to join.
---
You received this message because you are subscribed to the Google Groups "kaldi-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Miroslav Janosik]

Hi Yenda

Thank you for all the information.

I managed to compile 1.4.1 from https://github.com/edobashira/openfstwin-1.4.1 in VS2015, I needed to:
1) turn off "Incremental Link-Time Code Generation (LTCG)" (https://stackoverflow.com/questions/31554559/possible-to-stop-generating-ipdb-iobj-files-by-visual-studio-2015) to avoid 'not enough space' error during linking
2) in \src\include\fst\compat.h change "typedef SSIZE_T ssize_t;" to "typedef __int64 ssize_t;"

Now I'm trying to compile 1.6.2 from your repository but I'm running into terrible problems that I can't jump over, something along these lines:

1>D:\SL\openfstwin-1.6.2\src\include\fst/fst-decl.h(86): error C2065: 'Arc': undeclared identifier (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)
1>D:\SL\openfstwin-1.6.2\src\include\fst/fst-decl.h(86): error C2923: 'fst::VectorState': 'Arc' is not a valid template type argument for parameter 'A' (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)
1>D:\SL\openfstwin-1.6.2\src\include\fst/fst-decl.h(83): error C2065: 'Arc': undeclared identifier (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)
1>D:\SL\openfstwin-1.6.2\src\include\fst/fst-decl.h(83): error C2923: 'std::allocator': 'Arc' is not a valid template type argument for parameter '_Ty' (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)
1>D:\SL\openfstwin-1.6.2\src\include\fst/fst-decl.h(86): error C3203: 'allocator': unspecialized class template can't be used as a template argument for template parameter 'M', expected a real type (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)
1>D:\SL\openfstwin-1.6.2\src\include\fst/vector-fst.h(711): error C3203: 'VectorState': unspecialized class template can't be used as a template argument for template parameter 'S', expected a real type (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)
1>D:\SL\openfstwin-1.6.2\src\include\fst/vector-fst.h(711): error C2371: 'fst::StdVectorFst': redefinition; different basic types (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)
1>  D:\SL\openfstwin-1.6.2\src\include\fst/fst-decl.h(187): note: see declaration of 'fst::StdVectorFst' (compiling source file ..\..\src\extensions\compact\compact16_acceptor-fst.cc)

I'm trying further to find if I can use VS2015 + OpenFst 1.6.2 for latest Kaldi. Maybe I will have to switch to VS2017 too.

 with regards
  Miro

Dňa štvrtok, 25. mája 2017 16:43:21 UTC+2 Yenda napísal(-a):

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-develope...@googlegroups.com.

[Jan Trmal]

check if you have C++11 standard enabled in the solution.

Also, I think it would be better if you contact me directly in case you run into problems.

y.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-developers+unsubscribe@googlegroups.com.

[Jan Trmal]

BTW, for a public notice, I just tried to compile using VS2017 and run into similar issues. I find that really confusing as I _was_ able to compile it. But perhaps I was accidentally compiling something else? Anyway, as of now, it does not compile.

The issue is as follows:

fst-decl.h contains declarations such as

template <class Arc, class Allocator = std::allocator<Arc>>

class VectorState;

but when I go to definition (in vector-fst.h), the template definition is

template <class A, class M /* = std::allocator<A> */>

class VectorState {

As you can see,  the difference is class A vs class Arc.

When I change Arc to A in the declaration, it compiles without issues.

I actually do not really understand what is going on (I suppose something with the name lookup but just idea). Notice that class Allocator vs class M is fine. 

y.

[Kirill Katsnelson]

On 170525 0743, Jan Trmal wrote:
> As I said before, I keep working on this, but my time (and patience) is
> limited, especially because I don't use windows and I do not have an easy
> access to a windows machine with reasonable specs).

Yenda: Let me see if I can make it work somehow. I think it would be an
interesting idea to maintain the buildable version as a branch in a
mirrored repository, avoiding the need of patching. Something that can
be pulled from github and compiled. I'll try to carve some time for that
next week!

On Thu, May 25, 2017 at 3:56 AM, Miroslav Janosik <miro.j...@gmail.com
<javascript:>> wrote:
> So my question is - is there any other developer using VS2015 + latest
> OpenFst to give me the working solution file? If not, would be someone
> interested in me publishing the solution file for it when I succeed?

Miro: When I compiled version 1.4.1 into my project (a decoder-only
recognizer, using the library, as a static .lib, and no executable
binaries), my goal was to take the original OpenFST files without
modifications. That worked. What I did was create a directory
(include-before) that preceded others on the compiler include path, to
contain (a) files missing in VS2015 and (b) those I needed to override
the behaviors. I also needed a trivial non-implementation of class
fst::MappedFile returning NULL from all methods (or use assert() if you
want), and added it to the project instead of the original
mapped-file.cc. This is never called, just to make the linker happy.

kkm@func:src/openfst/include-before (master)$ ls -FR
.:
dlfcn.h fst/ sys/ tr1/ unistd.h

./fst:
config.h lock.h log.h

./sys:
mman.h

./tr1:
unordered_map unordered_set

log.h and lock.h are in the category (b), implementation overrides, as I
needed a real multithread locking and custom logging), unordered_* to
bring the implementation into the namespace std::tr1 where it was
expected by the fst (probably not needed any more):

#pragma once
#include <../include/unordered_set>
namespace std { namespace tr1 {
using std::unordered_set;
using std::unordered_multiset;
} }

(and mutatis mutandis for unordered map); unistd.h and mman.h just
empty; dlfcn.h with a minimum non-implementation of DLL functions

#pragma once
const int RTLD_LAZY = 0;
inline void* dlopen (const char*, int) { return nullptr; }
inline const char* dlerror() { return "dlfcn unsupported"; }

I think I'll try to come up with something along the same lines for 1.6.
Let's see if I get anywhere.

-kkm

[Daniel Povey]

Thanks!

We could make the Windows version of OpenFst a separate download that we maintain on openslr.org.  Or personally I'm fine with applying patches, if that's easier.

For Kaldi itself, we can add any ifdefs we need.

[Jan Trmal]

Miro has actually done a great progress with compiling  openfst under windows. We noticed that the visual studio provides Clang compiler so lot of the issues went away (he had to write wrappers for some calls not available on win)

y.

[Kirill Katsnelson]

On Saturday, June 10, 2017 at 2:57:25 AM UTC-7, Yenda wrote:

Miro has actually done a great progress with compiling  openfst under windows. We noticed that the visual studio provides Clang compiler so lot of the issues went away (he had to write wrappers for some calls not available on win)

Yes, as an option. Never used it though. What I found was actually there is an apparent bug in MSVC compilers (tried all VS2013 through VS2017). This is a minimum sample I am reproducing the bug with. This code, apparently correct can be compiled with gcc on Linux and Intel icl and clang that comes with VS2017 on Windows, but not MSVC.

template<typename A> class F {};
template<typename U = int, typename Z = F<U>> class C;
template<typename Ux, typename Zx> class C {};
static C<> t1;

And this compiles with MSVC cl, but not the others--note the change in the default for typename Z in the line 2:

template<typename A> class F {};
template<typename U = int, typename Z = F<Ux>> class C;
template<typename Ux, typename Zx> class C {};
static C<> t1;

Indeed, matching template argument typenames in lines 2 and 3 works around the bug. So I do not know which way to go from here:

1. using clang looks like opening a whole new can of unfamiliar worms.
2. renaming all template parameter typenames in fst-decl.h is more maintenance than I hoped to achieve
3. ?????

I would go with (2) in hope that the bug might be fixed by MS, although I am not holding amazingly high hopes.

Opinions?

 -kkm

[Daniel Povey]

Kaldi should compile with clang in general, that's what gcc really is on a mac, under the hood.

Dan

[Dogan Can]

The compiler we use on Travis to build under Linux is also clang.

Cheers,

Dogan

--
visit http://kaldi-asr.org/forums.html to find out how to join.
---
You received this message because you are subscribed to the Google Groups "kaldi-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kaldi-develope...@googlegroups.com.

[Kirill Katsnelson]

On Monday, June 12, 2017 at 4:39:43 PM UTC-7, Dan Povey wrote:

Kaldi should compile with clang in general, that's what gcc really is on a mac, under the hood.

The main point is whether we should drop the support for MSVC (at least temporarily, because of this bug) and stick with clang on Windows, or hack up a windows version that can be compiled with MSVC (not affecting the ability to compile with clang and other compilers, naturally).

 -kkm

[Daniel Povey]

I think clang is a better bet; bugs on MSVC don't have a good track record of getting fixed.

[Kirill Katsnelson]

On Monday, June 12, 2017 at 4:56:47 PM UTC-7, Dan Povey wrote:

I think clang is a better bet; bugs on MSVC don't have a good track record of getting fixed.

I am torn really. I would not bet on MS' irrevocable commitment to the support of clang in VS either. Don't want to squeezed between one buggy and another outdated. But I would also rather not do extra work of hacking over and then supporting the hacked openfst. A lovely disposition.

[Kirill Katsnelson]

On Monday, June 12, 2017 at 4:56:47 PM UTC-7, Dan Povey wrote:

I think clang is a better bet; bugs on MSVC don't have a good track record of getting fixed.

Well, looks like clang produces compatible object files in Visual Studio. Should be the way to go.

By the way, I remember there was a patch for OpenFST 1.4 (Linux, not Windows) -- is this not needed any more?

 -kkm

[Daniel Povey]

I think whatever reason we had to need a patch went away.  But there is no reason why we can't provide Windows-specific patches.

Well... I suppose the tricky part is that the program 'patch' wouldn't exist on Windows.  So maybe we'd have to just store the entire modified OpenFst files.

[Kirill Katsnelson]

On Monday, June 12, 2017 at 8:02:13 PM UTC-7, Dan Povey wrote:

I think whatever reason we had to need a patch went away.  But there is no reason why we can't provide Windows-specific patches.

Well... I suppose the tricky part is that the program 'patch' wouldn't exist on Windows.  So maybe we'd have to just store the entire modified OpenFst files.

Yes, that's what I am planning to do: A parallel Windows branch.

 -kkm

[Miroslav Janosik]

Indeed, matching template argument typenames in lines 2 and 3 works around the bug. So I do not know which way to go from here:

1. using clang looks like opening a whole new can of unfamiliar worms.
2. renaming all template parameter typenames in fst-decl.h is more maintenance than I hoped to achieve
3. ?????

I would go with (2) in hope that the bug might be fixed by MS, although I am not holding amazingly high hopes.

Opinions?

 Hi

my opinion is that using most reliable and easiest way is the best, and I agree that going in a new direction (VS + clang) can bring new issues. But staying with VS compiler is not easy. It support for new C++ features was always lacking behind other compilers.

I tried (2) renaming the templates for few hours and compiling with VS2017 native compiler and I found that it is just on too many places + got more compiler errors, for example:

openfstwin-1.6.2\src\include\fst/extensions/linear/trie.h(295): error C2976: 'std::unordered_map': too few template arguments (compiling source file ..\..\src\extensions\linear\linear-classifier-fst.cc)
openfstwin-1.6.2\src\include\fst/extensions/ngram/nthbit.h(13): error C3861: '__builtin_popcount': identifier not found (compiling source file ..\..\src\extensions\ngram\bitmap-index.cc)

After that I tried to compile with VS2017 + Clang and although (after some changes) I could compile OpenFst library I don't think I have a solution yet as I have trouble linking applications using the library - they can't get exported symbols from lib. Even the test projects in the OpenFst.
I was trying to create a dummy empty Clang dll project in VS2017 and I can't get that work either! It will take me some time to find out what is wrong, it is either wrong installation of VS2017 or I have to learn more about Clang.

I have also adapted kaldi generate_solution.pl so it can generate VS2017 Clang project files, but again, I'm missing OpenFst export symbols and can't link. I also have run into some other issues during compiling which I can't explain - for example when including pthread.h it seems like compiler says "dllimport" inside "__declspec (dllimport)" is unknown keyword.

I will post all my changes online in few hours for your review.

  with regards
   Miro

[Miroslav Janosik]

I will post all my changes online in few hours for your review.

Here are my changes: http://mypage.sk/tmp/kaldi-clang-openFst-compilability.zip (1.7mb)

OpenFst change summary:
1) all vcxproj files were changed:
    1.1) to use v141_clang_c2 toolset instead of v141
    1.2) Removed incompatible DebugInformationFormat: EditAndContinue and DebugInformationFormat: ProgramDatabase
    1.3) Removed incompatible AdditionalOptions: /bigobj %(AdditionalOptions)
2) OpenfstWinLib.vcxproj - added few cc files that compilation was missing
3) openfst.def changed CONSTANT keyword to DATA
4) few minor changes to source code, larger change to mapped-file.cc

generate_solution.pl - ability to generate VS2017 solution for Clang compiler

[Kirill Katsnelson]

Miro, thanks for posting your changes. To me (I mean my main Kaldi-based decoding code) compiling openfst with clang only also proved a no-go. Since the MSVC bug affects the OpenFST headers (most of it is inline), use of clang infectiously bubbled up the chain... with new problems.

For clang to recognize __declspec (dllimport), enable the "Microsoft Compatibility Mode", and set "Microsoft Compiler Version" to 1900. I did not try much beyond that. So I am going to succumb to the broken compiler and rename the identifiers in fst-decl.h, so that the code will compile with MSVC as well.

Now, as far as creating and uploading a version that builds on Windows, there is so many ways it can be built: which compiler/toolset? static or dynamic library? debug or release build? I can add necessary property sheets and configurations to the build process, but that will be time consuming. I think that the first objective is to get us both going (I do not know if there are any more people struggling right now to build on Windows, if so, please chime in!).

I know my requirements, what are yours?
- do you need a DLL, or static library will work for you?
- do you need the binaries (e. g. fstcompile.exe) immediately?
- do you need any extensions, beyond libfst? NB Kaldi configures OpenFST to build Far (FST archive) and NGram extension libraries, but I do not see them used anywhere in the mainline Kaldi code.

 -kkm

[Daniel Povey]

We don't need those NGram or Far extensions of OpenFst.

As regards whether we use dynamic or static libraries when compiling Kaldi, I'm not sure-- Yenda would know-- but my vague recollection is that if you make it dynamic in Windows things can get complicated, due to how dll's work.  I may be wrong though.

Dan

[Kirill Katsnelson]

On Tuesday, June 13, 2017 at 7:00:51 PM UTC-7, Dan Povey wrote:

 if you make it dynamic in Windows things can get complicated, due to how dll's work.

Right, not recommended my MS themselves for C++ DLLs in general; dance on a razor's edge with such a heavily templatized library as OpenFST.  I am compiling subsets of FST and Kaldi into static libraries internally.

But the openfstwin-1.4.1 port does build a DLL though, so I am afraid there are some users out there who might be relying on that.

 -kkm

[Daniel Povey]

Having *anything* build on Windows would be a step forward at this point.

[Jan Trmal]

The openfst extension registration mechanism needs dynamic libraries, but AFAIK the mechanism was broken under win anyway (in openfstwin 1.3.4/1.4.1), so static libs might be all right. 

Y. 

[Miroslav Janosik]

Dňa streda, 14. júna 2017 3:13:41 UTC+2 Kirill Katsnelson napísal(-a):

Now, as far as creating and uploading a version that builds on Windows, there is so many ways it can be built: which compiler/toolset? static or dynamic library? debug or release build? I can add necessary property sheets and configurations to the build process, but that will be time consuming. I think that the first objective is to get us both going (I do not know if there are any more people struggling right now to build on Windows, if so, please chime in!).

I know my requirements, what are yours?
- do you need a DLL, or static library will work for you?
- do you need the binaries (e. g. fstcompile.exe) immediately?
- do you need any extensions, beyond libfst? NB Kaldi configures OpenFST to build Far (FST archive) and NGram extension libraries, but I do not see them used anywhere in the mainline Kaldi code.

I was using OpenFst 1.3.4 to build Kaldi last year as lib + dll and I was using release version, 64bit because I am building 64bit kaldi to avoid memory limitations on larger models.
I don't mind static library, it is ok for me. I don't need any tools or extensions, I see that Kaldi needs only "openfst64.lib" to compile.

I have found yesterday how to advance with missing exports so I will try to continue on "Clang path". I will let you know about my progress.

  Miro

[Kirill Katsnelson]

On Wednesday, June 14, 2017 at 1:08:21 AM UTC-7, Miroslav Janosik wrote:

I was using OpenFst 1.3.4 to build Kaldi last year as lib + dll and I was using release version, 64bit because I am building 64bit kaldi to avoid memory limitations on larger models.
I don't mind static library, it is ok for me. I don't need any tools or extensions, I see that Kaldi needs only "openfst64.lib" to compile.

I have found yesterday how to advance with missing exports so I will try to continue on "Clang path". I will let you know about my progress.

Cool, thanks. My requirements are pretty much same, and I am going down MSVC path. We'll converge. :)

 -kkm

[Kirill Katsnelson]

On Tuesday, June 13, 2017 at 7:20:49 PM UTC-7, Dan Povey wrote:

Having *anything* build on Windows would be a step forward at this point.

I am on it. Fixed compilation issues, worked around another likely bug in the MS compiler. To my taste, there is too much template-fu in openfst 1.6!

I am bringing my kaldi snapshot up to date. The one-year-old one does not compile with the new FST. Just want to make sure everything compiles, creates a graph and decodes before pushing my OpenFST changes up to GitHub. Hope to get it done in a few hours.

 -kkm

[Miroslav Janosik]

Dňa streda, 14. júna 2017 22:51:42 UTC+2 Kirill Katsnelson napísal(-a):

I am on it. Fixed compilation issues, worked around another likely bug in the MS compiler. To my taste, there is too much template-fu in openfst 1.6!

I am bringing my kaldi snapshot up to date. The one-year-old one does not compile with the new FST. Just want to make sure everything compiles, creates a graph and decodes before pushing my OpenFST changes up to GitHub. Hope to get it done in a few hours.

 -kkm 

That's great news! With Clang I'm able to build a core of OpenFst, and few of the basic Kaldi libraries. I'm getting into some conflicts when linking to non-clang project and I will have to resolve that too. If you manage to make it compile with VS-compiler I will be happier than with Clang.

Please if you can share your current work-in-progress I would be glad to test it out.
  Miro

[Kirill Katsnelson]

On Thursday, June 15, 2017 at 12:52:22 AM UTC-7, Miroslav Janosik wrote:

With Clang I'm able to build a core of OpenFst, and few of the basic Kaldi libraries. I'm getting into some conflicts when linking to non-clang project

Same here, this is where I dropped the clang line. I was finally able to compile OpenFST and run kaldi-based decoding with the VS2015 and 2017 compilers.

The VS-compilable port is in https://github.com/kkm000/openfst/win/1.6
Changes from the original https://github.com/kkm000/openfst/compare/orig/1.6...win/1.6

I did not build any extensions, and did not implement a replacement for mapped files (used only by extensions). DLL registration is not supported (neither it was in openfstwin). The included solution and project build a single static library, libfst.lib.

Please let me know if it worked for you, and what problems if any you had to address.

 -kkm

[Daniel Povey]

Thanks.  I'll wait till Yenda has time to deal with this though, as I am busy with some other stuff.

[Kirill Katsnelson]

On Thursday, June 15, 2017 at 3:13:06 PM UTC-7, Dan Povey wrote:

Thanks.  I'll wait till Yenda has time to deal with this though, as I am busy with some other stuff.

Sure. My main question is whether we need anything else (extensions and/or binaries).

[Daniel Povey]

Well, for Kaldi training scripts to run you'd definitely need some OpenFst binaries, such as fstcompile and the like.

I don't know if it's feasible to get Kaldi training scripts working on Windows-- the concern, I think, is more about runtime-- but I suppose it's a nice-to-have.

Dan

[Kirill Katsnelson]

On Thursday, June 15, 2017 at 3:25:06 PM UTC-7, Dan Povey wrote:

Well, for Kaldi training scripts to run you'd definitely need some OpenFst binaries, such as fstcompile and the like.

I don't know if it's feasible to get Kaldi training scripts working on Windows-- the concern, I think, is more about runtime-- but I suppose it's a nice-to-have.

 

Well, I did an end-to-end example once, we then had to add support for mapping between cygwin and OS filenames. Symbolic links are a pig to get working under Windows too. Having done the whole exercise, I would not really suggest that to anyone. But in any case, the support is there.

I'll look into adding binaries later. Outside of training, they are indeed useful for analysis of FST files under Windows.

 -kkm

[Miroslav Janosik]

Dňa piatok, 16. júna 2017 0:11:21 UTC+2 Kirill Katsnelson napísal(-a):

The VS-compilable port is in https://github.com/kkm000/openfst/win/1.6
Changes from the original https://github.com/kkm000/openfst/compare/orig/1.6...win/1.6

I did not build any extensions, and did not implement a replacement for mapped files (used only by extensions). DLL registration is not supported (neither it was in openfstwin). The included solution and project build a single static library, libfst.lib.

Please let me know if it worked for you, and what problems if any you had to address.

Hi
 oh, thank you. I am getting crazy making the Clang compilation work. I'm happy that your progress is further.

First, download paths: I can't access those github.com paths, so I downloaded from https://github.com/kkm000/openfst as I see that it contains changes from today.

I tried to compile it with VS2015 and it compiles fine, and it is very fast :) (side note: compiling with Clang seems to be very much slower than with VS compiler)

I wanted to compile libraries that I use:
kaldi-base.lib kaldi-cudamatrix.lib kaldi-decoder.lib kaldi-feat.lib
kaldi-fstext.lib kaldi-gmm.lib kaldi-hmm.lib kaldi-ivector.lib
kaldi-lat.lib kaldi-lm.lib kaldi-matrix.lib kaldi-nnet2.lib
kaldi-nnet3.lib kaldi-online2.lib kaldi-thread.lib kaldi-transform.lib
kaldi-tree.lib kaldi-util.lib

Out of these some did not compile, looks to me that only reason is this error:
1>...\src\fstext/context-fst.h(316): error C2244: 'fst::ContextFst<Arc,LabelT>::InitStateIterator': unable to match function definition to an existing declaration

I will try to fix that.
  Miro

[Miroslav Janosik]

Hello

so after changing \src\fstext\context-fst.h lines:

template <class A, class I> inline
void ContextFst<A, I>::InitStateIterator(StateIteratorData<A> *data) const {
 data->base = new StateIterator< ContextFst<A> >(*this);
}

to:

template <class Arc, class LabelT> inline
void ContextFst<Arc, LabelT>::InitStateIterator(StateIteratorData<Arc> *data) const {
  data->base = new StateIterator< ContextFst<Arc> >(*this);
}

I was able to compile in VS2015 all my needed kaldi-*.lib libraries. That looks great.
I can't say that I'm finished as I had to adapt my program and now I'm not sorting out linker errors (currently fst::ReadFstKaldi(std::string) and kaldi::WriteLattice ) but I hope to solve that soon and get to testing program with new version of kaldi + openfst.

 cheers
 Miro

Dňa piatok, 16. júna 2017 11:13:49 UTC+2 Miroslav Janosik napísal(-a):

[Kirill Katsnelson]

Thanks Miro! Yes the same weird pattern of name capture in template as that in DeterminizeFst. I'll apply your fix today.

 -kkm

[Daniel Povey]

BTW, if one if you guys (Kirill?) can come up with a single patch for the current version of OpenFst that resolves these issues, I can put you in touch with the OpenFst maintainers and we can see if they'll apply it.  I expect they will accept it.  Although Kirill, you may know those guys already since you were at Google.

[Kirill Katsnelson]

On Friday, June 16, 2017 at 1:42:10 PM UTC-7, Dan Povey wrote:

BTW, if one if you guys (Kirill?) can come up with a single patch for the current version of OpenFst that resolves these issues, I can put you in touch with the OpenFst maintainers and we can see if they'll apply it.  I expect they will accept it.  Although Kirill, you may know those guys already since you were at Google.

I'll look into this, a good idea. There is a couple of other (non-)issues yelled at by the Intel and MS compilers in the arena allocator. These are not immediately problematic, as they are related to potential memory leaks caused by exceptions thrown from constructors, and those constructors do not really throw; I want to see if they can be fixed anyway because the library is mostly header files, so the warning has to be disabled in any code including OpenFST and therefore Kaldi. I am somewhat idiosyncratic about compiler warnings.

I do not know any of them personally, I worked even in a different campus. Wish I did!

 -kkm

[Daniel Povey]

So where are we on this? Kirill, do you have some partial work that
you can make available as a PR or some other mechanism? I want to
make sure we lock in any gains you guys have made.

[Miroslav Janosik]

Hi

just yesterday I was able to finally compile everything I needed for my program to work.
FYI - my program now compiles not only VS2017 but also in VS2015. I'm compiling it as x64 Release.
I have also compiled and run some kaldi tests.
So I did not need to do any further modifications to Kaldi or OpenFst except those that I mentioned earlier to \src\fstext\context-fst.h

I could not find reason why the fst::ReadFstKaldi and kaldi::WriteLattice are not linked in. I think it is caused by my forward-declarations as I do not get the error in simple dummy project that calls fst::ReadFstKaldi. I solved it by a workaround - by copying bodies of these methods to my code.

These are libraries I use:

kaldi-base.lib kaldi-cudamatrix.lib kaldi-decoder.lib kaldi-feat.lib kaldi-fstext.lib
kaldi-gmm.lib kaldi-hmm.lib kaldi-ivector.lib kaldi-lat.lib kaldi-lm.lib
kaldi-matrix.lib kaldi-nnet2.lib kaldi-nnet3.lib kaldi-online2.lib kaldi-thread.lib
kaldi-transform.lib kaldi-tree.lib kaldi-util.lib

pthreadVC2.lib libopenblas.dll.a openfst64.lib

  Miro

[Daniel Povey]

So what did you have to do? You had to modify or create the VS build
files, whatever they call them? Is this something that we could or
should check in?
I'm not an expert on the Windows build, but I want to make sure that
we get the benefit of what you and Kirill have worked out.

Dan

[Kirill Katsnelson]

I have checked in my changes + Windows specific build files for VS as https://github.com/kkm000/openfst/tree/win/1.6

Thanks to GitHub magic, an archive canbe downloaded by just appending .zip or .tar.gz to the above URL.

I did not get to building executables or extensions. As for building Kaldi on windows, there is no specific changes needed, I believe, except maybe for documenting the process. Currently it is pointing to openfstwin-1.4.1. IIRC.

[Daniel Povey]

Hopefully @jtrmal can figure out what needs to be done. He's in the
Czech Republic though; I don't know if he has a Windows machine there.

[Jan Trmal]

Thanks a lot guys. Kirill's patches look great.

Miro, can you publish your changeset as well? I'm working on getting my hands on a windows computer, so I'll try to combine the diffs from both of you and figure out how to make everything working out together.

y.

--

[Miroslav Janosik]

Hello Jan

 I did not find out yet how to create correctly pull-request, so I'm sending my changes to kaldi as a archive file with changed files. I hope that it is usable. If it is not I can prepare the pull-request in following days (or weeks, vacations are starting).

 with regards
  Miro


Dňa utorok, 27. júna 2017 10:53:57 UTC+2 Yenda napísal(-a):

[Jan Trmal]

thanks, this is sufficient, I will look into it 

y.

--

[Kirill Katsnelson]

Hi Miro,

Regarding the change of isascii to __isascii, I am wondering what compiler (and toolset in Visual Studio) are you using? The function is indeed deprecated (and IsToken is UTF-hostile anyway), but I have a compatibility macro defined in my ctype.h:

#if !__STDC__
    // Non-ANSI names for compatibility
    #define isascii __isascii
 . . . 

The built-in macro  __STDC__ could be defined only for C (not C++) compilation.

 -kkm

[Daniel Povey]

I think you may be mistaken when you say IsToken() is utf-hostile. It
is designed to accept UTF-8, although if someone uses one of the
Unicode points for weird forms of space, it won't be correctly
recognized as space.

The comment may not make this very clear.

[Kirill Katsnelson]

Agree, it should do. A UTF-8 trailing byte cannot be isascii or 255 (they have two MSBits of 10), so the second term of && will be false. The lead byte is isascii, so the conditions on the left work, assuming the C locale.
But oh boy, did my brain sizzle figuring that out!

[Miroslav Janosik]

Dňa štvrtok, 13. júla 2017 0:34:11 UTC+2 Kirill Katsnelson napísal(-a):

Hi Miro,

Regarding the change of isascii to __isascii, I am wondering what compiler (and toolset in Visual Studio) are you using? The function is indeed deprecated (and IsToken is UTF-hostile anyway), but I have a compatibility macro defined in my ctype.h:

 

If I remember correctly I did this change when I used Clang compiler with VS2017. Though it could be just a warning or error, I'm not sure about that. I think you can safely revert this change.
   Miro

[Kirill Katsnelson]

Ah, got it. Thank you!

 -kkm

[Miroslav Janosik]

Hello

FYI I have noticed that OpenFst code on https://github.com/kkm000/openfst/ has recently changed and now Kaldi has problem to compile on my VS2015. It compiles on VS2017 fine (with built-in compiler).

First error that I encountered is:
\openfst-win-1.6\src\include\fst/string-weight.h(176): error C2039: 'const_reverse_iterator': is not a member of 'std::remove_reference<std::list<int,std::allocator<_Ty>>>'

 with regards
  Miro

[Daniel Povey]

Yes, we know that compiling on Windows is problematic right now.
I thought we were working on a solution-- Yenda, where are we with this?

[Jan Trmal]

It's still opened issue. I mostly wait until I get back to jhu (which should be on Sun). 

Y. 

[Kirill Katsnelson]

Miro, please see https://github.com/kkm000/openfst/issues/1 -- Someone reported the same issue. but I cannot reproduce this. I would appreciate if you help me track the differences, as it compiles on 2 different machines for me. What is your version of Visual Studio and the compiler? Are you sure you have the the update 3?

[Kirill Katsnelson]

Dan, I am on it.There is actually one change from Miro's diff that is pending. I'll get to it on Monday.

[Miroslav Janosik]

Hi Kirill

 so I am back in my office and version of my Visual Studio 2015 is "14.0.25425.01 Update 3". I use v140 for projects and compiler version is "19.00.24213.1 for x64".

so my VS is a little older than yours "25431".

 with regards
  Miro

Dňa pondelok, 24. júla 2017 5:41:40 UTC+2 Kirill Katsnelson napísal(-a):