Feb 26 2012

ARC Gotcha – Unexpectedly Short Lifetimes

Disassembly
One of our engineers was working on a project and wrote some code that crashed when running on a device:

    CGColorRef color = 
        [UIColor colorWithRed: 0.2  
                        green: 0.3  
                         blue: 0.4  
                        alpha: 1.0].CGColor;
    [[self.view layer]
        setBackgroundColor: color];

It looks reasonable. Could it be a toolkit bug? It’d be weird for something to be so obviously broken in a fundamental CoreAnimation call. It’s like the CGColor is pointing to garbage when it gets used. Almost as if the newly-created UIColor suddenly vanished and took the CGColor down with it.

What could be releasing objects automatically? ARC – that’s its job. From ARC’s point of view, the UIColor created with -colorWithRed:... was used to access the CGColor, and then is no longer needed. The UIColor wasn’t assigned to anything, therefore ARC decided that the object could be cleaned up at the semicolon at the end of the expression. Therefore it’s a candidate for disposal and got released. The disassembly, viewable in Xcode with Product->Generate Output->Generate Assembly file, shows that this is the case:

    // UIColor colorWithRed:...
    blx	_objc_msgSend  
    ...
    // Hang on to the object, bypassing the autorelease pool
    bl	_objc_retainAutoreleasedReturnValue  
    ...
    // -CGColor
    blx	_objc_msgSend
    ...
    // The UIColor goes away
    bl	_objc_release 

    // and then the layer calls happen

Ordinarily, you’d expect the UIColor to be in an autorelease pool, and so would survive until the end of the function. In this case, though, you can see ARC’s “avoid the autorelease pool” optimization is being used: if a method returns an autoreleased object, and the caller doesn’t otherwise need to hang on to it, ARC can avoid the trip to the autorelease pool. The method will return a retained object and objc_retainAutoreleasedReturnValue will dispose of it. That’s why the UIColor is not kept alive by the autorelease pool.

So how to fix this? One idea we tried was assigning the new UIColor to a pointer variable, thereby taking it out of that very short-lived expression. After that, get the CGColor out of it:

    UIColor *uicolor = [UIColor colorWithRed: 0.2 
                                       green: 0.3 
                                        blue: 0.4 
                                       alpha: 1.0];
    CGColorRef color = uicolor.CGColor;
    [[self.view layer] setBackgroundColor: color];

This seems to work – no crashes. Ship it!

Unfortunately, it’s a ticking bomb. The returned UIColor is being assigned to a strong-reference variable, but it doesn’t get used in the method after extracting the CGColor. ARC is within its rights to release the object between fetching the CGColor and setting the layer’s background color. ARC variables are released as soon as the optimizer decides that they are no longer referenced, so the compiler is free to release uicolor after fetching the CGColor. Current compiler implementations do seem to wait until the end of scope, but that’s not guaranteed. The LLVM Project’s ARC notes say: “By default, local variables of automatic storage duration do not have precise lifetime semantics.”

This means the compiler can do whatever it wants. The currrent Apple LLVM 3.0 compiler releases uicolor at the end of the method. If the optimizer, either now or in the future, decides that uicolor‘s lifetime is over, it’ll get released.

So, how to fix this? Three Four ways. You can annotate the local variable with the objc_precise_lifetime attribute if you want it to live til the end of the scope. Update: Thanks to TJ Usiyan for pointing out the NS_VALID_UNTIL_END_OF_SCOPE foundation wrapper for objc_precise_lifetime

You can explicitly retain the CGColor so it doesn’t float away, and release it later:

    CGColorRef color = 
        CGColorRetain ([UIColor colorWithRed: 0.2 
                                       green: 0.3 
                                        blue: 0.4 
                                       alpha: 1.0].CGColor);
    [[self.view layer] setBackgroundColor: color];
    CGColorRelease (color);

The uicolor is welcome to disappear because the CGColor will stay around.

You can put in a call to self at the end of the method:

    [uicolor self];

And finally you can also bounce through the uicolor whenever you need its CGColor. This keeps the variable in use so it won’t be released yet:


    UIColor *uicolor = [UIColor colorWithRed: 0.2 
                                       green: 0.3  
                                        blue: 0.4  
                                       alpha: 1.0];
    [[self.view layer] setBackgroundColor: uicolor.CGColor];

Of these, I like the last one better. Less code, less explicit management of object lifespan (which is why we have ARC in the first place), and also makes it very clear from where the CGColor was derived.

(Thanks to Mike Ash for exploring some of these ARC behaviors with me. Thanks to Juri Pakaste and Dave Dribin for the [uicolor self] trick.)

35 Comments

  1. Juri Pakaste

    A fourth option would be to add a call to [uicolor self] to the end of the method.

    No, not particularly neat, either.

  2. David

    The best solution is to just do [view setBackgroundColor:uiColor]. It doesn’t show off the gotcha, but it does solve the exact same problem :).

  3. Mark Dalrymple

    :-)

    The problem also showed itself when setting other layer/color properties like the border color. AFAIK there’s no UIColor equivalent API for that.

  4. David

    Very true. The one-liner versions shouldn’t have that problem however (that is definitely a long one liner however).

  5. Jacob Relkin

    Very informative article. Thanks for writing it up!

  6. Jens Ayton

    Surely the preferred fix is identical to the ticking time bomb as far as the ARC optimizer is concerned? (Tangentially, -CGColor ought to be marked __attribute__((objc_returns_inner_pointer)) to avoid this.)

  7. mani

    Are you sure the CGColorRetain(…) version is guaranteed to work? Isn’t the compiler allowed to release the UIColor after getting the CGColor but before putting the CGColor in the CGColorRetain call?

  8. Mark Dalrymple

    @Jens, the preferred fix is different from the time bombs because that there isn’t any stretch of time that the uicolor goes unused between getting its CGCclor, and passing it on to another call that retains the color.

    A common trick to think about race conditions in code is to pretend every semicolon is a five-second sleep. Then try to imagine all the hell that can break loose in that five seconds. The same technique can be used regarding ARC. In the fall-down-go-boom scenarios, there’s opportunity for ARC to release the object. In the “use uicolor.CGColor directly inside the call” case, there’s not an opportunity for ARC to decide to free the object, so it’s safe getting the CGColor out of the color and into the layer where it can then be retained.

    • bob

      I disagree with this.

      [obj message:(some expression)];

      can always be written as

      temp = (some expression);
      [obj message:temp];

  9. Mark Dalrymple

    @mani – If ARC is releasing stuff at that point in time (between getting the object values for function arguments and calling the function), then we’ll have much worse things to worry about :-)

    Section 3.2 of the ARC document says “In general, ARC does not perform retain or release operations when simply using a retainable object pointer as an operand within an expression” so it shouldn’t be touching the pointer at that time.

  10. Cory

    Don’t see it here, so I thought I’d mention another possible solution:

    [[self.view layer] setBackgroundColor:[UIColor colorWithRed: 0.2 green: 0.3 blue: 0.4 alpha: 1.0].CGColor];

    I assume this would work, although I haven’t tested it. Your chosen solution is cleaner, though.

  11. Mark Dalrymple

    @cory – works too. Kind of makes the expression large and unwieldily, though :-)

  12. Nikolai Ruhe

    @Jens The ARC documentation explains why the time bomb version is different from the preferred way: It states in various places that temporary objects are released after “the end of the full expression”. This should answer @mani’s question, too.

  13. Nikita Zhuk

    The last option ([[self.view layer] setBackgroundColor: uicolor.CGColor];) is certainly the shortest and it works correctly, but I think it’s a bit problematic from code readability and maintainability perspective. This style basically means that code with a local temporary variable for CGColor and without the temporary variable produces different results, so it discourages refactoring nested calls into more self-documenting temporaries (if method call becomes more complex than current one) and can be a ticking time bomb by itself if someone uses that refactoring without knowledge of UIColor internals. I prefer ‘self’ call myself, but objc_precise_lifetime seems like a good idea as well.

    This discussion reminds me of the one which ObjC community had when we got GC support in ObjC and were hit with interior pointer problems [1]. Since then a special objc_returns_inner_pointer annotation [2] was added to clang which should prevent these kinds of problems in ARC. However, for some reason CGColor isn’t annotated with it.

    [1] http://lists.apple.com/archives/objc-language/2009/Mar/msg00037.html
    [2] http://clang.llvm.org/docs/AutomaticReferenceCounting.html#misc.interior

  14. jazzbox

    I tried the comma operator. The disassembled code releases the UIColor obj at the end of the full expression:

    CGColorRef color;
    color = [UIColor colorWithRed: 0.2 green: 0.3 blue: 0.4 alpha: 1.0].CGColor,
    [[self.view layer] setBackgroundColor: color];

  15. Duncan Wilcox

    How is this not a bug in the UIColor CGColor API?

    Or you can view it as a bug in ARC that’s unaware of CF object retain counts.

    @Jens’s suggestion to use __attribute__((objc_returns_inner_pointer)) on the CGColor property seems the most appropriate, though there might be better solutions that require changes to ARC wrt the nature of CF objects.

    Either way I don’t think working around the problem in any way is a good solution. Why pollute the code and require the developer to know about this?

  16. Matt Nunogawa

    We solved this issue with the __autoreleasing qualifier. Look here for that trick:

    http://amattn.com/2011/12/07/arc_best_practices.html

  17. Jim

    A fifth solution:

    UIColor * __autoreleasing shadowColor = [UIColor colorWithRed:0.12 green:0.12 blue:0.12 alpha:1.0];

    Tagging the UIColor variable with __autoreleasing will explicitly add it to the autorelease pool and make sure it stays around until the end of the method.

  18. Jens Ayton

    Mark, I must say, your “trick” to think about race conditions is horribly, frighteningly wrong. Statements are nowhere close to atomic.

    Nikolai’s response does address my previous concern, although the possibility of someone turning it into the “ticking bomb” version while cleaning up the code does seem to loom unsettlingly.

  19. Mark Dalrymple

    @jim / @matt- thanks for the __Autoreleasing qualifier. I didn’t think to use that outside of return parameters.

    @jens it’s not a perfect race condition thought experiment by any stretch of the imagination, but it helps open the mind (and especially student’s minds who never been exposed to the concept before) to the kinds of things that can go horribly, horribly wrong.

    It’s come in *very* handy when explaining things like file system races and the need for atomic system calls, and i’ve used it to explain race conditions during code reviews. It’s also a good first-pass to catch the egregious mistakes before you dig down into the details.

  20. Briggs

    @jazzbox, the comma operator is a odd one though isn’t it? Doesn’t the side effect of this essentially set the value of the color variable to some unknown value, since the value of [[self.view layer] setBackgroundColor: color] is void? What if someone added a line of code after the assignment and used the color variable? The program would most likely crash, right?

    I’m no expert in c/objective-c, so I am posting here to learn something.

  21. Vicente Vicens

    I am sorry to say this but i think that this article is misleading and not addressing to the point correctly, and people is starting to get confused about ARC as it is spreading over the internet.

    There is not gotcha and no bug and no unexpectedly short life lifetimes. It is all very well explained in the Transitioning to ARC Release Notes.

    This line of code will always be useless:

    CGColorRef color =
    [UIColor colorWithRed: 0.2
    green: 0.3
    blue: 0.4
    alpha: 1.0].CGColor;

    And is just because:
    1. You are not keeping a strong reference to the UIColor
    2. The CGColorRef object is a Core Fundation type object and it is pointing to an autoreleased object

    And Apple documentation about ARC states very clear that:

    “The compiler does not automatically manage the lifetimes of Core Foundation objects; you must call CFRetain and CFRelease (or the corresponding type-specific variants) as dictated by the Core Foundation memory management rules (see Memory Management Programming Guide for Core Foundation).”

    So after that line of code gets executed, the UIColor will be deallocated, and of course the CGColorRef unless you keep a strong reference to one of them.

    And that was all the mystery of the disappearing object, no need to disassembly the code, just read (and understand) the documentation, Apple’s documentation ;-)

    https://developer.apple.com/library/ios/#releasenotes/ObjectiveC/RN-TransitioningToARC/Introduction/Introduction.html#//apple_ref/doc/uid/TP40011226

    Just remember that Cocoa objects and Core Foundation type objects are different things, and ARC treats them differently.

    Best to everybody,

    Vicente Vicens

  22. XMLSDK

    Let’s use a handy UIColor category to CFRetain the CGColor:
    https://gist.github.com/1983329

    Remember to CGColorRelease.

  23. Seth Kingsley

    I can’t think of a reason why correctly written reference counted code should crash under ARC. Imagine if instead you had to go through clever tricks to use the result of -[NSData bytes] after the last message was sent to that NSData instance within a local scope (I haven’t tried this, but if it doesn’t work, I’m going to scream).

    The semantics for an inner CG object property return should be at least as life-preserving as for an opaque pointer. Therefore, this /has/ to be a framework bug.

    There are a few places where I’ve added some code to quiet the static analyzer (for instance, a lack of noreturn on -[NSException raise:]), but I’ve clearly marked these places with comments, lest anyone try to optimize them out later. I suggest you do the same, instead of changing your coding habits due to framework breakage.

  24. Warren P

    The only thing that would fit in the “principle of least amazement” would be if all local variables automatically have scope that ends where the method ends. And that’s it. All of the above article gives me the same sick feeling in my stomach that every bad and failed architecture and framework gives me. That anybody has to know above that stuff to use ARC is crazy. Automatic, my keester.

    Warren

  25. arcfannot

    Warren P, totally agree.

    Reference counting is already automatic for any programmer willing to internalize a few simple rules.

    For programmers not so willing, sure, ARC lets them avoid the easiest and most obvious manual memory management cases (and avoid learning), but it quickly plunges them into a morass of ARC attributes and special cases far more difficult for them to understand than CFRetain/CFRelease.

    For experienced programmers, I’ll admit ARC is providing some interesting fodder for intellectual puzzles, such as presented here. Real productivity gains? Nil, if not already negative. Alas, that will not be generally recognized until the “ARC Considered Harmful” articles appear and ARC joins the trash heap with OS X Garbage Collection.

    ARC primarily exists because some of Apple’s more ill-considered block/closure-based methods make reasoning about memory management nearly impossible. But that’s another rant, and yes blocks do have their uses; see the fine post above. However, nesting function within function within method (what is this, Pascal???) just to avoid writing a standalone callback routine is not one of them. And the “blocks make code easier to read and maintain” meme was absurd from the get-go. See Apple’s WWDC 2011 recommended override of saveToURL:ofType:forSaveOperation:completionHandler: for a degenerate case of block-based band-aid-upon-nested-band-aid – programming reduced to magical incantation. Oops, I went ahead and ranted…

    Still, ARC makes me smile: consider how one workaround for its “optimization” of “avoiding the autorelease pool” is for the programmer to understand manual memory management and add explicit retain and release where none was ever needed before! Now that’s funny.

  26. Macy

    This gotcha should not be happening if Apple could make it right.

    Didn’t it mean that if one uses ARC, all nested method should be avoided, and variable must be assigned with __Autoreleasing qualifier from object of each return method ? And if so, Apple should mention against practice of nested method.

  27. Matthew Johnson

    Everyone, please take a look at Vicente’s comment. He clearly explains the situation. If you do not understand why this is the case you have probably misunderstood ARC’s rules regarding toll-free bridging and Core Foundation objects. These are not Objective-C objects and are not managed by ARC.

    • AJ

      @Matthew Johnson: Agreed. It’s best to set an internal warning bell that goes off whenever you are working with CF Objects in ARC code. Even if you aren’t well-versed in Core Foundation and ARC, you should at least learn to identify them as potential problem areas when you are debugging code. If you have mysterious crashes that you can’t figure out, CF code chunks are a good place to review.

  28. John

    hi Mark,
    could you please help me with this about CGColorRef?

    http://stackoverflow.com/questions/14783145/id-for-objects-of-cgcolorref-opaque-type

    Thanks

    • Interesting – I haven’t heard of the freequartz project before. It’s a mystery to me why they have that extra ID. I’m curious why they called it ‘nextID’ – was that derived from Apple’s Quartz somehow?

  29. John

    hi Mark,
    i think just free quartz called it next id and may be not apple. i could found out the reason for the ID(thread safe counter) field. they are used to skip the color transformations which were cached for the last graphics context operation. we know that the reference(memory address) of a released object could be used while creating another object. for the next graphics context operation, just checking the reference will not be enough that same color object is valid but we need to compare contents or some kind of hash value. so, we could use the unique identifier values for this purpose.

    however, an identifier could be in use for a single object and its reference, so it is enough to compare only ids.

    but, both refs and ids are used. i don’t think that apple overlooked such a simple and crucial thing. so, i try to find out the necessity of comparing refs.

  30. Haolan Qin

    It shouldn’t crash at all. I tested the code snippet and it didn’t crash. The reason is that [UIColor colorWithRed:green:blue:alpha:] returns an autoreleased object so it won’t be released right after the line is executed. To illustrate your point, you would have to call [[UIColor alloc] initWithRed:green:blue:alpha:]. Because the returned object is not autoreleased, ARC compiler will add a release statement right after the line.

    • Don’t forget that ARC has optimizations to bypass the autorelease pool.

      The main problem is that UIColor is returning what’s essentially an internal object. I believe more modern headers have the decoration that this is what’s happening.

Leave a Comment

Join the discussion. Do not worry, your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>