Is empty string cached?
Farshid Lashkari
or does python use some global pre-created object? I know python does
this with integer objects under a certain value. For instance, in the
following code is a new string object created for each function call?
func(0,'')
func(1,'')
func(2,'')
func(3,'')
I tried the following commands in the interactive shell:
>> x = ''
>> y = ''
>> x is y
True
>> x = 'hello'
>> y = 'hello'
>> x is y
True
This leads me to believe that python does reuse existing strings, but
once the variables are removed, does the item still exist in the cache?
-Farshid
Bryan Olson
> When I pass an empty string to a function is a new string object created
> or does python use some global pre-created object? I know python does
> this with integer objects under a certain value. For instance, in the
> following code is a new string object created for each function call?
>
> func(0,'')
> func(1,'')
> func(2,'')
> func(3,'')
In this case, the language implementation may either create new
strings or re-use existing ones:
for immutable types, operations that compute new values
may actually return a reference to any existing object with
the same type and value, while for mutable objects this is
not allowed.
[http://docs.python.org/ref/objects.html]
[...]
> This leads me to believe that python does reuse existing strings, but
> once the variables are removed, does the item still exist in the cache?
Either; see the same reference page.
--
--Bryan
Steve Holden
>>> a = "hello"
>>> b = "h"+"ello"
>>> a is b
False
>>> c = "hello"
>>> b is a
False
>>>
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/
Farshid Lashkari
>
> >>> a = "hello"
> >>> b = "h"+"ello"
> >>> a is b
> False
> >>> c = "hello"
> >>> b is a
> False
> >>>
I don't understand the point of your last expression. Were you intending
this instead:
>>> c is a
True
However, the following commands add to my confusion:
>> a = 'wtf?'
>> b = 'wtf?'
>> a is b
False
So how are string literals cached? Is there an explanation somewhere? Is
it some freaky voodoo, and I should just assume that a string literal
will always generate a new object?
Thanks,
Farshid
Steve Holden
>>It takes far too little evidence to induce belief:
>>
>> >>> a = "hello"
>> >>> b = "h"+"ello"
>> >>> a is b
>>False
>> >>> c = "hello"
>> >>> b is a
>>False
>> >>>
>
>
> I don't understand the point of your last expression. Were you intending
> this instead:
>
> >>> c is a
> True
>
> However, the following commands add to my confusion:
>
> >> a = 'wtf?'
> >> b = 'wtf?'
> >> a is b
> False
>
> So how are string literals cached? Is there an explanation somewhere? Is
> it some freaky voodoo, and I should just assume that a string literal
> will always generate a new object?
>
I really don't understand why it's so important: it's not a part of the
language definition at all, and therefore whatever behavior you see is
simply an artifact of the implementation you observe.
Farshid Lashkari
> language definition at all, and therefore whatever behavior you see is
> simply an artifact of the implementation you observe.
I guess I should rephrase my question in the form of an example. Should
I assume that a new string object is created in each iteration of the
following loop?
for x in xrange(1000000):
func(x,'some string')
Or would it be better to do the following?
stringVal = 'some string'
for x in xrange(1000000):
func(x,stringVal)
Or, like you stated, is it not important at all?
Thanks,
Farshid
Steve Holden
>>> import dis
>>> print dis.dis.__doc__
Disassemble classes, methods, functions, or code.
With no argument, disassemble the last traceback.
>>> help(dis.dis)
>>> dis.dis(compile("""\
... for x in xrange(1000000):
... func(x,'some string')""", "", 'exec'))
1 0 SETUP_LOOP 33 (to 36)
3 LOAD_NAME 0 (xrange)
6 LOAD_CONST 0 (1000000)
9 CALL_FUNCTION 1
12 GET_ITER
>> 13 FOR_ITER 19 (to 35)
16 STORE_NAME 1 (x)
2 19 LOAD_NAME 2 (func)
22 LOAD_NAME 1 (x)
25 LOAD_CONST 1 ('some string')
28 CALL_FUNCTION 2
31 POP_TOP
32 JUMP_ABSOLUTE 13
>> 35 POP_BLOCK
>> 36 LOAD_CONST 2 (None)
39 RETURN_VALUE
>>> dis.dis(compile("""\
... stringVal = 'some string'
... for x in xrange(1000000):
... func(x,stringVal)""", "", 'exec'))
1 0 LOAD_CONST 0 ('some string')
3 STORE_NAME 0 (stringVal)
2 6 SETUP_LOOP 33 (to 42)
9 LOAD_NAME 1 (xrange)
12 LOAD_CONST 1 (1000000)
15 CALL_FUNCTION 1
18 GET_ITER
>> 19 FOR_ITER 19 (to 41)
22 STORE_NAME 2 (x)
3 25 LOAD_NAME 3 (func)
28 LOAD_NAME 2 (x)
31 LOAD_NAME 0 (stringVal)
34 CALL_FUNCTION 2
37 POP_TOP
38 JUMP_ABSOLUTE 19
>> 41 POP_BLOCK
>> 42 LOAD_CONST 2 (None)
45 RETURN_VALUE
>>>
It just boils down to either a LOAD_CONST vs. a LOAD_NAME - either way
the string isn't duplicated.
Farshid Lashkari
> the string isn't duplicated.
Great, that's exactly what I wanted to know. Thanks Steve!
-Farshid
Peter Hansen
> However, the following commands add to my confusion:
>
> >> a = 'wtf?'
> >> b = 'wtf?'
> >> a is b
> False
>
> So how are string literals cached? Is there an explanation somewhere? Is
> it some freaky voodoo, and I should just assume that a string literal
> will always generate a new object?
A few comments (which I hope are correct, but which I hope you will read
then mostly ignore since you probably shouldn't be designing based on
this stuff anyway):
1. What you see at the interactive prompt is not necessarily what will
happen in a compiled source file. Try the above test with and without
the question mark both at the interactive prompt and in source and see.
2. The reason for the difference with ? is probably due to an
optimization relating to looking up attribute names and such in
dictionaries. wtf? is not a valid attribute, so it probably isn't
optimized the same way.
2.5. I think I had a third comment like the above but after too little
sleep last night it seems it's evaporated since I began writing...
3. As Steve or someone said, these things are implementation details
unless spelled out explicitly in the language reference, so don't rely
on them.
4. If you haven't already written your code and profiled and found it
lacking in performance and proving that the cause is related to whether
or not you hoisted the string literal out of the loop, you're wasting
your time and this is a good opportunity to begin reprogramming your
brain not to optimize prematurely. IMHO. FWIW. :-)
-Peter
Farshid Lashkari
> then mostly ignore since you probably shouldn't be designing based on
> this stuff anyway):
Thanks for the info Peter. My original question wasn't due to any
observed performance problems. I was just being curious :)
-Farshid
Magnus Lycka
In this particular case, it's no big deal, since you use
a literal, which is something Python knows won't change.
In general, it's semantically very different to create an
object at one point and then use a reference to that over
and over in a loop, or to create a new object over and
over again in a loop. E.g.
for x in xrange(1000000):
func(x, str(5))
v.s.
stringVal = str(5)
for x in xrange(1000000):
func(x,stringVal)
This isn't just a matter of extra function call overhead. In
the latter case, you are telling Python that all calls to
"func(x,stringVal)" use the same objects as arguments (assuming
that there aren't any assignments to x and stringVal somewhere
else in the loop). In the former case, no such guarantee can
be made from studying the loop.
As for interning strings, it's my understanding that current
CPython interns strings that look like identifiers, i.e.
starts with an ASCII letter or an underscore and is followed
by zero or more ASCII letter, underscore or digit. On the other
hand, it seems id(str(5)) is persistent as well, so the current
implementation seems slightly simplified compared to the
perceived need. Anyway, this is just an implementation choice
made to improve performance, nothing to rely on.