Millionths of an inch???

[Richard BRackin]

I have been wanting to ask this question for a while so here goes...

When I type a dimension in the transform box like, let's say.... 3.389063 inches and the display truncates it to 3.391 does that mean the additional numbers are removed?

Or is the precision still there but the displayed number just rounds up to a thousandth inch?

thanks
Richard

[Phosphor]

Truncation takes place, just for the sake of the numbers fitting into user-friendly sized numerical entry fields, but accuracy is maintained out to quite a few places. How many, exactly, I'm not sure.

This came up about two months ago, and in that thread I responded that when I use HotDoor's CAD Tools plugin, it will keep track of (and display) accuracy down to 0.0001 pt. This works out to about 1.39 millionths of an inch. (I used 72.0 points/Inch for my calculation). Now, whether this maintained accuracy comes from the plugin or from Illustrator's internal calculations, I don't know. But if you need any more accuracy than that, I'd suggest moving to a real CAD application.

[Teri Pettit]

Illustrator preserves values to 0.0001 accuracy when files are saved. We don't write out more than that to save file space. All distances are saved in pts regardless of how they are entered and displayed.

Internally while the file is still open distances are preserved to the accuracy of a 32 bit floating point number expressed in points. An exact number of decimal places cannot be given because it is in the nature of floating points that they have more decimal places of accuracy on small numbers than they do on large numbers. That's what it means for the "point" to "float" - when it doesn't need as much room to represent the integer part, it moves over to give more memory to the fractional part. Near the center of the artboard (where the internal coordinates are in the range of about 8200 pts) I believe they have about 7 or 8 decimal places of accuracy, way up in the top left corner they have about 11 decimal places of accuracy, and way down at the bottom right corner of the pasteboard they have about 3 and a half decimal places of accuracy. (What it means for a number to have a fractional number of decimal places of accuracy is that adjacent representable numbers differ by that fraction. Remember, the program isn't putting the numbers in decimal form at all until it writes them out or displays them; internally they are binary.)

One way you can verify that even more than 4 decimal places of accuracy are used internally is to accumulate a bunch of transforms. Make a small vertical line, copy-paste-in-front so that you have two copies at the same place, select the top one, specify that you want to move it "0.00001 pt" horizontally, and then hit Cmd-D 100 times. Select both of them and you will see that they are 0.001 pt apart. (It would work even with 0.000001 pt, but I wouldn't want to ask you to hit Cmd-D 1000 times.)

Since options like stroke width always have small integer parts, they internally have extremely high precision while the file is open, although like all values they are rounded to 4 decimal places when saved.

[Phosphor]

Ooooo...hit me, baby with that geek speak.

Thanks, Teri. Much appreciated, as always.

[Richard BRackin]

Thanks for the explanation.

If I were to now say, "That's what I thought," a lightning bolt would strike me dead where I sit.

4 decimal points is plenty room for this job.

The real CAD app that I use says it goes out 16 decimal places but it can't do some of the things I need for our digital printing end of things.

Thanks to you both for clearing it up so quickly.

[Mike Sulzer]

Teri,

Let's think about this a minute. If 32 bits are used to represent a floating point number, then some of them are for the factional part and some are for the exponent. The number of each has to be fixed or you would need more bits to tell how many are used for each. But a 32 bit floating point number is self-contained; you can send it to anyone, and it has the same meaning as long as you know which format (there are two common one.) is used.

Here is the IEEE standard used on the mac. The PC reverses the order of the bytes.

From: <http://www.psc.edu/general/software/packages/ieee/ieee.html>

Single Precision
The IEEE single precision floating point standard representation requires a 32 bit word, which may be represented as numbered from 0 to 31, left to right. The first bit is the sign bit, S, the next eight bits are the exponent bits, 'E', and the final 23 bits are the fraction 'F':

S EEEEEEEE FFFFFFFFFFFFFFFFFFFFFFF
0 1 8 9 31

The value V represented by the word may be determined as follows:

If E=255 and F is nonzero, then V=NaN ("Not a number")
If E=255 and F is zero and S is 1, then V=-Infinity
If E=255 and F is zero and S is 0, then V=Infinity
If 0<E<255 then V=(-1)**S * 2 ** (E-127) * (1.F) where "1.F" is intended to represent the binary number created by prefixing F with an implicit leading 1 and a binary point.
If E=0 and F is nonzero, then V=(-1)**S * 2 ** (-126) * (0.F) These are "unnormalized" values.
If E=0 and F is zero and S is 1, then V=-0
If E=0 and F is zero and S is 0, then V=0

In particular,

0 00000000 00000000000000000000000 = 0
1 00000000 00000000000000000000000 = -0

0 11111111 00000000000000000000000 = Infinity
1 11111111 00000000000000000000000 = -Infinity

0 11111111 00000100000000000000000 = NaN
1 11111111 00100010001001010101010 = NaN

0 10000000 00000000000000000000000 = +1 * 2**(128-127) * 1.0 = 2
0 10000001 10100000000000000000000 = +1 * 2**(129-127) * 1.101 = 6.5
1 10000001 10100000000000000000000 = -1 * 2**(129-127) * 1.101 = -6.5

0 00000001 00000000000000000000000 = +1 * 2**(1-127) * 1.0 = 2**(-126)
0 00000000 10000000000000000000000 = +1 * 2**(-126) * 0.1 = 2**(-127)
0 00000000 00000000000000000000001 = +1 * 2**(-126) *
0.00000000000000000000001 =
2**(-149) (Smallest positive value)

[Adam Lane]

Your point is?

Internally, all of the precision issues have been thought out. A mixture of single precision float, double precision float and fixed point integers are used. But you don't need to worry about those details.

Externally, all the user needs to remember is that the software has 0.0001 pt accuracy across the range of the entire artboard.

Exactly how a number is stored also depends upon the type of file you store it into.

[Thurgood]

What fun it almost makes sense!

[Phosphor]

Somehow, Mike's geek-speak isn't as appealing as Teri's.

Call me a sucker for someone who knows wherefrom they speak.

:o)

[Teri Pettit]

Mike,

I'm not sure what point you're making. The full mantissa is not available for the fraction, and how much is needed for the integer part will vary based on the object's position and size. That's why I said "about" in all my estimates.

When you're transforming something, you have to add the move vector to the existing coordinates, so the precision at which you can make the transformation is affected by the precision you have available to represent the object, not just the precision you have available to represent the vector.

I'm assuming typical objects will have dimensions in the range of 10 to 1000 pts wide, so even if you're working in the upper left corner of the artboard you will have only about 17 to 20 bits for the fraction. Coordinates near the middle of the artboard will use 12 to 13 bits for the integer part and thus have 10 to 11 bits for the fractional part, and coordinates down at the bottom right of the artboard maybe extending a little into the gray area will use 14 to 15 bits for the integer part and thus have 8 to 9 bits for the fractional part.

Anyway, Adam and Phosphor are right that it doesn't make a whole lot of difference how much precision is available for calculations while the program is open, since what matters the most is what's permanent. The bottom line is that values are written to file as decimal text strings to 4 decimal places, and that if you're working with objects on a typical page spread, you get at least that much accuracy while you're working too.

[Mike Sulzer]

Teri,

I am sorry, I completely misunderstood the beginning of your first message on this topic to mean fractional accuracy rather than absolute accuracy. Thank you for replying so patiently.

[Richard BRackin]

After reading all this I will not worry about precision in Illustrator again. Obviously these folks have it covered!!!

Perhaps I should explain why I was asking.....
We were printing a trailer graphic that was about 640 inches long and 102 inches tall.

Since illustrator doesn't go past 227.54 inches we have to work at a percentage.

We generate application instructions telling the decal installation crews where exactly to install the graphics.
The graphics have an overlap of about 1/2 inch so we have to include that in the schematic.

The schematic is printed on 8.5 x 11 paper so we must work in considerable precision to keep us out of any trouble if the installers screw up and try to blame us.

We are working at just over 1 and a half percent of the full size print so that means I need a minimum 5 thousands of an inch accuracy just for the overlaps and Illustrator obviously gives me more.

I need more accuracy to determine precise length and position of each panel and Illustrator gives me that, too.

I guess I could work at a larger scale, but we would have to set a lot of things up differently to do so.

Thank you for the information. I understand the concept but I wouldn't want to take a test on it !!!!

Kindest regards
Richard

[Max Heim]

In Richard's situation I would be tempted to at least work at 4 times the size of a 8.5 x 11 page, and scale to 25% when printing! That would make some of the distances at least visible on screen at a reasonable enlargement...

I once used Illustrator for an architectural facade 114 feet wide and 65 feet high. Yeah, print scaling was an issue with that, too <g>. The funny thing was, my illustration was much more accurate than the contractor's CAD drawing, and eventually the client insisted we use it instead. In this case it was more or less a case of GIGO -- the contractors were misreading the features of the actual facade as seen from ground level, when compared to the almost century-old blueprints.

[nilk narf]

I wish I could speak geek!