FTP port not picking up large Flat files
PeterW
We use FTP to pick up files, debatch them using receive pipeline ]with
FlatfileDisassembler and pass the individual files to an orchestration.
If the Flat file only contains a few records, the ftp port picks it up.
We have a file containing 3652 lines which should debatch just fine.
Trouble is the FTP port will not pick it up. If we create a new file and
caopy and paste just a few rows from the flat file and make sure the end of
row has a CR LF, and place that in the FTP folder, then the FTP port does
pick it up.
Content is identical, only quantity is different.
Anyone have any ideas as to what is happening here?
cheers
--
PeterW
PeterW
following message:
The adapter "FTP" raised an ettor message. Details: "The remote file could
not be opened by the FTP server".
This is followed by another message:
"The adapter recovered from recent failures. The Event Log Error Threshold
count reset."
--
PeterW
PeterW
Temp folder is set for adapter. File (146KB) is well within Max file size.
File is shown as still on the FTP Server but both FTP server and BizTalk FTP
log shows the FTP transfer as complete.
The Temp folder contains the whole of the file. I have tried stopping the
host instance, deleting the file in the Temp Folder and then redropping the
file to the FTP directory on the ftp server. Same result.
Try with a file containing just 2 rows and all works. Only when file
contains the full 3652 rows does this happen.
My thoughts go to the receive pipeline as an issue, but surely the receiving
of the file would complete in its entirety before the pipeline comes into
play and so should be outside scope of what to look for.
PeterW
The pipeline just plain takes a lot of time to debatch the messages. It
took 21 minutes for the pipeline to debatch 3652 from the flat file. By
contrast all 3652 orchestration instances fired and completed with stored
procedures persisting data to the database in a total of under 2 minutes with
each instance taking around a quarter of a second.
The issues were the appropriate setting of idle timeout on the FTP server to
a large enough value to not timeout on largish files. I was somewhat
surprised to see the actual times it took for the pipeline to debatch. The
times are much greater than I would have expected based on the milliseconds
it takes to do so many things.
The other issue was the temporary folder location. we had a temporary
folder but this was a file loaction on the BizTalk Server and not the FTP
server. After resetting the FTP adapter temporary folder to one on the FTP
server and configuring the write permissions to the created folder, all
worked properly.
The document "Known issues with the FTP adapter" at
http://msdn2.microsoft.com/en-us/aa577610.aspx pointed me in the right
direction.
--
PeterW