Error: cannot feed the input to external filter

[Ken Ismert]

Hello list,

I'm so close to getting the clean/smudge filter setup I want.

But, when I try to stage about 60 sql files, about six of them (both ANSI and UTF-16) error out with the message "error: cannot feed the input to external filter"

I've tried this on two machines, and the problem seems to follow the files, and it doesn't matter what the clean filter is, the same error occurs.

I played around with a certain file (plain ANSI), and found things that can make it work:
* Changing CRLF to LF
* Deleting lines 12-17, or every line after 299
* Adding back CRLF or the removed lines makes the file fail again.

Certain things have no effect:
* Renaming the file
* Copy/paste contents to a new file, and trying to commit that
* Converting to UTF-8, UTF-16, or back to ANSI
* Deleting/remaking the repository

Platform:
* Windows XP SP3
* msysGit 1.8.1.2-preview20130201
* The files are mainly from Microsoft SQL Server Management Studio 2008 R2

Questions:
* Any idea what the problem might be?
* Is there an error log maintained by Git I can inspect?

Thanks,
-Ken

[Johannes Sixt]

Am 24.04.2013 20:25, schrieb Ken Ismert:
> I'm so close to getting the clean/smudge filter setup I want.
>
> But, when I try to stage about 60 sql files, about six of them (both
> ANSI and UTF-16) error out with the message "error: cannot feed the
> input to external filter"

Hard to say what's wrong when you show neither the filter nor the data
that you feed it.

One possibility is that the filter finds an "error" in the data and
exits before it has consumed all input.

-- Hannes

[Ken Ismert]

gitconfig (just for testing):

[filter "mixedtext"]
    clean = cat %f
    smudge = cat %f
    required

.gitattributes:

*.sql filter=mixedtext

Problem file attached.

Thanks,
-Ken

[Ken Ismert]

Hard to say what's wrong when you show neither the filter nor the data
that you feed it.

Problem file attached.

I just tried it on a third computer, with the same results. Deleting some portion of the end of the file makes it stage properly.

The file is a plain ANSI file, with only Tab, LF, CR and printable characters below 127.

I would like for someone to try this, and tell me if they are getting an error.

Thanks,
-Ken

[Steffen Jaeckel]

I just tried it and got the same error and some t&r got me that it has
something to do with a 8KB (8192byte) boundary!

> sjaeckel@T7400-003 ~/foo (master)
> $ ls -al aaa_bad.sql
> -rw-r--r-- 1 sjaeckel Administ 8193 Apr 26 09:56 aaa_bad.sql
> sjaeckel@T7400-003 ~/foo (master)
> $ GIT_TRACE=2 git add aaa_bad.sql
> trace: built-in: git 'add' 'aaa_bad.sql'
> trace: run_command: 'cat '\''aaa_bad.sql'\'''
> error: cannot feed the input to external filter cat %f
> error: external filter cat %f failed
> fatal: aaa_bad.sql: clean filter 'mixedtext' failed
> sjaeckel@T7400-003 ~/foo (master)
> $ ls -al aaa_bad.sql
> -rw-r--r-- 1 sjaeckel Administ 8192 Apr 26 09:56 aaa_bad.sql
> sjaeckel@T7400-003 ~/foo (master)
> $ GIT_TRACE=2 git add aaa_bad.sql
> trace: built-in: git 'add' 'aaa_bad.sql'
> trace: run_command: 'cat '\''aaa_bad.sql'\'''

and it has nothing to do with 'cat', I tried 'less' and it has the
same behavior...

HTH
Steffen
--
Steffen Jaeckel
mailto:steffen...@gmail.com

[Johannes Sixt]

Am 24.04.2013 22:56, schrieb Ken Ismert:
> On Wednesday, April 24, 2013 2:24:15 PM UTC-5, Johannes Sixt wrote:
>
> Am 24.04.2013 20:25, schrieb Ken Ismert:
> > I'm so close to getting the clean/smudge filter setup I want.
> >
> > But, when I try to stage about 60 sql files, about six of them (both
> > ANSI and UTF-16) error out with the message "error: cannot feed the
> > input to external filter"
>
> Hard to say what's wrong when you show neither the filter nor the data
> that you feed it.
>
> One possibility is that the filter finds an "error" in the data and
> exits before it has consumed all input.
>

> gitconfig (just for testing):
> [filter "mixedtext"]
> clean = cat %f
> smudge = cat %f

As I suspected: Your filter does not consume its standard input. Note
that clean and smudge filters do not have the liberty to ignore the
standard input. The filter can use the file argument only to find out
properties about the file, but the data must be taken from the stdin.
This is particularly important for the smudge filter: How can you smudge
the repository content when you read the data from the file??

-- Hannes

[Ken Ismert]

> Steffen Jaeckel:

> I just tried it and got the same error and some t&r got me that it
> has something to do with a 8KB (8192byte) boundary!

> Johannes Sixt:

> As I suspected: Your filter does not consume its standard input.
> Note that clean and smudge filters do not have the liberty to
> ignore the standard input. The filter can use the file argument
> only to find out properties about the file, but the data must be
> taken from the stdin. This is particularly important for the
> smudge filter: How can you smudge the repository content when
> you read the data from the file??
> -- Hannes

Steffen, Hannes:

Thanks for your testing and feedback.

So, in looking at Hannes' advice, and re-reading the gitattributes man page, I got it to work.

My original approach was this:

    clean = iconv -sc -f $(file -b --mime-encoding %f) -t utf-8 %f
    smudge = iconv -sc -f utf-8 -t $(file -b --mime-encoding %f) %f

That failed about 5% of the time because iconv was told to source its input from the given file, not stdin.
So, the fix is to drop the ending %f, which implicitly tells iconv to use stdin:

    clean = iconv -sc -f $(file -b --mime-encoding %f) -t utf-8
    smudge = iconv -sc -f utf-8 -t $(file -b --mime-encoding %f)

This command now works for a combination of utf-16, utf-8 and standard ansi files.

Thanks again for your help!

-Ken