[AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars

[Klaus Hofeditz ]project-open[]

Hi all 

at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS 
5.6.0 on CentOS release 5.3 (Final)

We suddenly encountered the problem that ns_returnfile can't find a
file which filname contains special chars such as 'umlaute'
(ä, ö, etc.)

---
convmv tells me that the file name is utf-8 encoded.
---

Config.tcl has the standard settings:

ns_param   HackContentType    1
ns_param   DefaultCharset     utf-8
ns_param   HttpOpenCharset    utf-8
ns_param   OutputCharset      utf-8
ns_param   URLCharset         utf-8

---
Following CentOS locale:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

---
File System is 'ext3'
---
Files have been created with " exec /bin/cp $filename $dest_path" out of
a tcl file or using 'touch/emacs' on OS level. Same results.  
---

We did have the problems using AOLserver 3.3
Anybody an idea what the cause might be?

Probably its better to always encode file names of 
uploaded filesto avoid trouble when creating multi-plattform 
applications. 

Thanks for your support!

Klaus 


 

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Peter Sadlon]

Just a guess here, but by default, TCL is compiled with Latin-1 encoding.  This causes some issues when you are trying to do certain things in utf-8, even if you set all possible TCL config variables to use the UTF-8 charset.  You could attempt to recompile TCL with


--with-encoding utf-8


But in the end, your final comment is correct, it is best to encode the filename, then you don't have to worry about a bunch of special cases.

---

Date: Thu, 22 Sep 2011 22:59:02 +0200
From: klaus.h...@PROJECT-OPEN.COM
Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not find file - file name contains special chars
To: AOLS...@LISTSERV.AOL.COM

[Jim Davidson]

Howdy,

Looking at the code, ns_returnfile passes the filename through to the core Ns_ConnReturnFile without any of the care that core Tcl does handling filenames.  You may be able to replace ns_returnfile with ns_returnfp, passing a file handle returned from Tcl's open command which should be smarter with filesystem encoding.

The downside is that would bypass some of the smarts within the underlying Ns_ConnReturnFile used by ns_returnfile:  It appears to set Last-Modified headers, optionally cache small files, mmap large files, etc.  All this may not matter if you're doing one-shot type work, e.g., returning a custom crafted file for just one user, just once.

-Jim

[Jeff Rogers]

Another thing you could do is to set tcl's default encoding to utf-8, so
that the filenames passed to Ns_ConnReturnFile are the same encoding as
what the core tcl commands do.

Set the default encoding with

encoding system utf-8

in some tcl file. It's possible this could have some side effects.

I'm trying to figure out what needs to be patched in the server for this
to do the right thing, but encoding is a messy business.

-J

Jim Davidson wrote:
>
> Howdy,
>
> Looking at the code, ns_returnfile passes the filename through to the
> core Ns_ConnReturnFile without any of the care that core Tcl does
> handling filenames. You may be able to replace ns_returnfile with
> ns_returnfp, passing a file handle returned from Tcl's open command
> which should be smarter with filesystem encoding.
>
> The downside is that would bypass some of the smarts within the
> underlying Ns_ConnReturnFile used by ns_returnfile: It appears to set
> Last-Modified headers, optionally cache small files, mmap large files,
> etc. All this may not matter if you're doing one-shot type work, e.g.,
> returning a custom crafted file for just one user, just once.
>
> -Jim
>
>
>
>
> On Sep 22, 2011, at 3:27 PM, Peter Sadlon wrote:
>
>> Just a guess here, but by default, TCL is compiled with Latin-1
>> encoding. This causes some issues when you are trying to do certain
>> things in utf-8, even if you set all possible TCL config variables to
>> use the UTF-8 charset. You could attempt to recompile TCL with
>>
>>
>> --with-encoding utf-8
>>
>>
>> But in the end, your final comment is correct, it is best to encode
>> the filename, then you don't have to worry about a bunch of special cases.
>>
>>
>>
>>
>>

>> ------------------------------------------------------------------------

>> Date: Thu, 22 Sep 2011 22:59:02 +0200
>> From: klaus.h...@PROJECT-OPEN.COM

>> <mailto:klaus.h...@PROJECT-OPEN.COM>

>> Subject: [AOLSERVER] Fwd: AOLserver 4.5.0 - ns_returnfile does not
>> find file - file name contains special chars

>> To: AOLS...@LISTSERV.AOL.COM <mailto:AOLS...@LISTSERV.AOL.COM>

>>
>> Hi all
>> at ]project-open[ we currently use AOLserver 4.5.0 with OpenACS
>> 5.6.0 on CentOS release 5.3 (Final)
>>
>> We suddenly encountered the problem that ns_returnfile can't find a
>> file which filname contains special chars such as 'umlaute'

>> (�, �, etc.)

>> AOLserver -http://www.aolserver.com/

>> To Remove yourself from this list, simply send an email to

>> <list...@listserv.aol.com <mailto:list...@listserv.aol.com>> with

>> the body of "SIGNOFF AOLSERVER" in the email message. You can leave
>> the Subject: field of your email blank.
>> -- AOLserver - http://www.aolserver.com/
>> To Remove yourself from this list, simply send an email to

>> <list...@listserv.aol.com <mailto:list...@listserv.aol.com>> with

[Klaus Hofeditz ]project-open[]

tx everybody for the very useful input!  

Using ns_returnfp is (for several reasons) not an option for us.

Recompiling tcl using

                --with-encoding utf-8

did not resolve the problem. This time I used AOLserver 4.5.1 / tcl8.5.10.

I gave it a try since some of our users do upload files using tools
such as WINSCP. Since these files would also need to be accessible
through the ]project-open[ file manager,  we need to come up
with a slightly more complex solution to detect files with special
chars.

./k

(ä, ö, etc.)

--
Klaus Hofeditz
Co-Founder ]project-open[
Dipl.-Inf.

]po[ is Germanys most popular Open Source PM/ERP System
more news at: www.project-open.org/rss/project-open-community.rss

Tel: +34 933 250 914
Cell: +34 650 157 499
Fax: +34 932 890 729
mailto:klaus.h...@project-open.com
http://www.project-open.com
SKYPE: klaushofeditz
YAHOO: klaushofeditz
ICQ: 207-101-803

[Jeff Rogers]

Klaus Hofeditz ]project-open[ wrote:

> I gave it a try since some of our users do upload files using tools
> such as WINSCP. Since these files would also need to be accessible
> through the ]project-open[ file manager, we need to come up
> with a slightly more complex solution to detect files with special
> chars.

I have a patch (not checked in yet) that will let ns_returnfile use the
same files as native tcl commands (by doing the same utf-8 to external
decoding that tcl does internally).

It would make sense to apply this to all file-handling commands. But
it would make more sense to replace these where possible (and simple)
with tcl versions that map to the builtin tcl commands?. I think the
docs even say that many of these date back to the time when tcl did not
include these functions. Would anyone have any objection to replacing
these c-coded commands with tcl ones?

The commands that could be easily replaced are ns_chmod, ns_cp,
ns_rename, ns_link, ns_mkdir, ns_rename, ns_rmdir, ns_symlink,
ns_unlink, and ns_ftruncate.

-J