Need help with error correction.

31 views
Skip to first unread message

Steve Rooke

unread,
Feb 5, 1991, 5:10:51 AM2/5/91
to
I have two sites, one sending a file, the other receiving. There is a lot
of corruption at the receiving end caused by line noise. I am not able to
use any standard form of error correction on the line but I can request
retransmission of the file, as many times as needed, over another link.

I need to be able to compair these files and reconstruct the original with
reasonable confidence. By that I mean that if two, or more, files have the
same text at a certain point then I am reasonably confident that the text
is OK.

I orignally thought of basing this on diff by trying to understand it's
output and collecting up the lines that are not different in each file but
the error rate may be high enough to cause some corruption on each line.
I expect the routine would have to look for substrings and be able to
re-sync when characters are lost/gained.

Has anyone tried to do this sort of thing before and how did you do it,
please? Solutions based upon diff would do, I guess, for a majority of
the time.

Thanks a lot,
Steve
--
Steve Rooke st...@gapos.bt.co.uk (...mcsun!ukc!gapos!steve) UK + 394 693595
BT, CSD/AS, Area 106, Anzani House, | "You roll the dice with your heart
Trinity Ave, FELIXSTOWE, Suffolk, UK | and soul, But some times you
#include <std/disclaimer> | just don't know." - Sam Brown

Greg Hunt

unread,
Feb 6, 1991, 9:28:29 AM2/6/91
to
In article <steve.665748651@paddy>, st...@gapos.bt.co.uk (Steve Rooke) writes:
> I have two sites, one sending a file, the other receiving. There is a lot
> of corruption at the receiving end caused by line noise. I am not able to
> use any standard form of error correction on the line but I can request
> retransmission of the file, as many times as needed, over another link.
>
> I need to be able to compair these files and reconstruct the original with
> reasonable confidence. By that I mean that if two, or more, files have the
> same text at a certain point then I am reasonably confident that the text
> is OK.
>
> Has anyone tried to do this sort of thing before and how did you do it,
> please? Solutions based upon diff would do, I guess, for a majority of
> the time.

When I've had file transmission problems, I've used sum(1) to produce
a checksum of the file on both the sending side machine and the
receiving side machine and compared the results. If they weren't the
same, then I knew that something got corrupted in the transmission and
I got the file again.

If the systems you're working with have sum(1) that might be an easy
thing to use. Also, sum(1) will work for any sort of file, it doesn't
just have to be text (which is the only thing diff(1) can look at).

--
Greg Hunt Internet: hu...@dg-rtp.rtp.dg.com
DG/UX Kernel Development UUCP: {world}!mcnc!rti!dg-rtp!hunt
Data General Corporation
Research Triangle Park, NC, USA These opinions are mine, not DG's.

Michael Meissner

unread,
Feb 6, 1991, 4:36:23 PM2/6/91
to
In article <1991Feb6.1...@dg-rtp.dg.com>
hu...@dg-rtp.rtp.dg.com (Greg Hunt) writes:

| When I've had file transmission problems, I've used sum(1) to produce
| a checksum of the file on both the sending side machine and the
| receiving side machine and compared the results. If they weren't the
| same, then I knew that something got corrupted in the transmission and
| I got the file again.
|
| If the systems you're working with have sum(1) that might be an easy
| thing to use. Also, sum(1) will work for any sort of file, it doesn't
| just have to be text (which is the only thing diff(1) can look at).

The only hitch is that sum(1) produces different results on System V
based systems and Berkeley based systems. I think sum -r on System V
gives the BSD behavior.

--
Michael Meissner email: meis...@osf.org phone: 617-621-8861
Open Software Foundation, 11 Cambridge Center, Cambridge, MA, 02142

Considering the flames and intolerance, shouldn't USENET be spelled ABUSENET?

Larry Wall

unread,
Feb 6, 1991, 6:42:58 PM2/6/91
to
In article <MEISSNER.9...@curley.osf.org> meis...@osf.org (Michael Meissner) writes:
: In article <1991Feb6.1...@dg-rtp.dg.com>

: hu...@dg-rtp.rtp.dg.com (Greg Hunt) writes:
:
: | When I've had file transmission problems, I've used sum(1) to produce
: | a checksum of the file on both the sending side machine and the
: | receiving side machine and compared the results. If they weren't the
: | same, then I knew that something got corrupted in the transmission and
: | I got the file again.
: |
: | If the systems you're working with have sum(1) that might be an easy
: | thing to use. Also, sum(1) will work for any sort of file, it doesn't
: | just have to be text (which is the only thing diff(1) can look at).
:
: The only hitch is that sum(1) produces different results on System V
: based systems and Berkeley based systems. I think sum -r on System V
: gives the BSD behavior.

Since this is cross-posted to comp.lang.perl, I suppose it's okay for me
to mention that you can emulate System V sum with

#!/usr/bin/perl

undef $/;
while (<>) {
print unpack("%32C*", $_) % 65535, " ", int((length()+511)/512), " $ARGV\n";
}

The Book, by the way, is wrong when it says you can emulate sum with "%16C*".
That is only guaranteed to work on files less than 256 bytes long (512 if
there are not eighth bits). Teach me to choose my test cases better...

No, I didn't have any sources to consult. The man page says sum does a
16-bit checksum, and it lies. It does modulo 65535 (not 65536). Ah well.

The above code will only work on files up to 2**24 bytes long or so.
Some machines may need to change the "%32C*" to "%31C*" until 4.0 comes
out, since some machines think that 1 << 32 == 1, GRRR! I won't mention
any names, because I don't want to get sun4's into trouble... :-)

Larry

Narayan S. Raja

unread,
Feb 6, 1991, 8:15:37 PM2/6/91
to

In article <1991Feb6.142829.20725@dg-rtp>, (Greg Hunt) writes:


< When I've had file transmission problems, I've used sum(1) to produce
< a checksum of the file on both the sending side machine and the
< receiving side machine and compared the results. If they weren't the
< same, then I knew that something got corrupted in the transmission and
< I got the file again.


I've also used sum for the same purpose.
However, according to the man page, sum
may give different checksums if sizeof(int)
is different on the two machines.


Narayan Sriranga Raja.

Leslie Mikesell

unread,
Feb 7, 1991, 11:19:58 AM2/7/91
to
In article <steve.665748651@paddy> st...@gapos.bt.co.uk (Steve Rooke) writes:
>I have two sites, one sending a file, the other receiving. There is a lot
>of corruption at the receiving end caused by line noise. I am not able to
>use any standard form of error correction on the line but I can request
>retransmission of the file, as many times as needed, over another link.

Unless one or both of the sites are truely arcane or mismanaged, you
should be able to get a version of kermit working to provide error
correction during the transfer. If you have a problem with getting
appropriate permissions for outbound calls with kermit, you might
consider using a PC as an intermediate, placing the calls into
both machines. Both the unix and PC versions can be script-driven
so you can probably make it run unattended.

Les Mikesell
l...@chinet.chi.il.us

Dale Worley

unread,
Feb 7, 1991, 1:54:29 PM2/7/91
to

When I've had file transmission problems, I've used sum(1) to produce
a checksum of the file on both the sending side machine and the
receiving side machine and compared the results. If they weren't the
same, then I knew that something got corrupted in the transmission and
I got the file again.

But you're forgetting that in this application there are so many
errors that one cannot expect that more than a few lines get through
without error. The probability that the the entire file gets through
without error is infinitesimal, and waiting for it to happen twice
would take forever.

Here's an idea: Break up lines into, say, ten-character lines. (In
fact, you are using newlines in the file to resynchronize the
line-breaking algorithm.) The line length should be chosen so that at
least 3/4 of the created lines have no errors in them. Then apply Gnu
diff or diff3 (for speed) to the resulting files. Since most of the
ten-character lines get through uncorrupted, diff should be able to
discern how the two files correspond. Then you can integrate the
output of one or more diffs to reconstruct the file.

Dale Worley Compass, Inc. wor...@compass.com
--
PHOTOVOLTAICS: safe and clean (but not cheap) electricity from the SUN.

Guy Harris

unread,
Feb 7, 1991, 4:13:46 PM2/7/91
to
>Some machines may need to change the "%32C*" to "%31C*" until 4.0 comes
>out, since some machines think that 1 << 32 == 1, GRRR! I won't mention
>any names, because I don't want to get sun4's into trouble... :-)

Or PCs and clones or other 386-based machines, or 3B{2,5,15}s, or
perhaps DECstations, or MIPS boxes, or MIPS-based SGI boxes, or....
SPARC is hardly unique in that regard; maybe MIPS's compiler generates
instructions to compensate for the fact that the shift count is taken
modulo 32, but other compilers don't.

Steve Rooke

unread,
Feb 8, 1991, 4:30:29 AM2/8/91
to
In article <steve.665748651@paddy>, st...@gapos.bt.co.uk (Steve Rooke) writes:
> I have two sites, one sending a file, the other receiving. There is a lot
> of corruption at the receiving end caused by line noise. I am not able to
> use any standard form of error correction on the line but I can request
> retransmission of the file, as many times as needed, over another link.
>
> I need to be able to compair these files and reconstruct the original with
> reasonable confidence. By that I mean that if two, or more, files have the
> same text at a certain point then I am reasonably confident that the text
> is OK.
>
> Has anyone tried to do this sort of thing before and how did you do it,
> please? Solutions based upon diff would do, I guess, for a majority of
> the time.

Thanks for all your initial replys about using file xfer protocols, kermit
and such, checksuming of the file at each end and splitting the file into
short line lengths to enable diff to at least produce some matches.

I guess I should have expanded on the real problem. As I stated, I cannot
use any standard xfer protocols which is due to the sending and receiving
equipment being dumb (ie LIKE telex [no flames please!]). The file is
then passed onto a U*IX system where it is checked and actioned upon.
The feedback path, for error retransmission, is by voice as, up to now,
a human has checked the file and pieced the contents together from a
number of transmissions.

As you can see I have no way of runing kermit (or the like), checksums
or altering the standard xmission of the file but I can request a resending
as many times as necessary. The only way, I can see, for rebuilding the
correct contents is to compair sets of files and select matching sub-strings
in them.

If you have any further ideas or some code fragments then please let me
know.

Thanks again,

Louis A. Mamakos

unread,
Feb 10, 1991, 11:49:56 AM2/10/91
to

This is (probably) not a perl solution, but you might investigate using some
sort of forward error correcting code, since the "cost" of retransmitting
parts of the data is "high."

louie

Reply all
Reply to author
Forward
0 new messages