Skip to content

malloc: error for object 0x3: pointer being freed was not allocated #77404

Closed
Closed
malloc: error for object 0x3: pointer being freed was not allocated#77404
Assignees
iskakaushik
Labels
P0Critical issues such as a build break or regressionc: contributor-productivityTeam-specific productivity, code health, technical debt.c: crashStack traces logged to the consolec: flakeTests that sometimes, but not always, incorrectly passengineflutter/engine related. See also e: labels.

Description

@jmagman
jmagman
opened on Mar 5, 2021

backdrop_filter_perf_ios__timeline_summary flaked on unrelated PR 7e02cc3

[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [+2234 ms] Runner(923,0x16d98b000) malloc: *** error for object 0x3: pointer being freed was not allocated
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ] Runner(923,0x16d98b000) malloc: *** set a breakpoint in malloc_error_break to debug
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [  +62 ms] Process 923 stopped
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ] * thread #10, name = 'io.flutter.1.ui', stop reason = signal SIGABRT
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ]     frame #0: 0x00000001b3c6c84c libsystem_kernel.dylib`__pthread_kill + 8
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ] libsystem_kernel.dylib`__pthread_kill:
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ] ->  0x1b3c6c84c <+8>:  b.lo   0x1b3c6c868               ; <+36>
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ]     0x1b3c6c850 <+12>: stp    x29, x30, [sp, #-0x10]!
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ]     0x1b3c6c854 <+16>: mov    x29, sp
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ]     0x1b3c6c858 <+20>: bl     0x1b3c49f5c               ; cerror_nocancel
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ] Target 0: (Runner) stopped.
[backdrop_filter_perf_ios__timeline_summary] [STDOUT] stdout: [        ] ios-deploy exited with code 0

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket.appspot.com/8853575074050880048/+/steps/run_backdrop_filter_perf_ios__timeline_summary/0/stdout

Activity

  1. jmagman
    added
    c: contributor-productivityTeam-specific productivity, code health, technical debt.
    engineflutter/engine related. See also e: labels.
    team: flakes
    on Mar 5, 2021
  2. jmagman
    changed the title [-]malloc: *** error for object 0x3: pointer being freed was not allocated[/-] [+]malloc: error for object 0x3: pointer being freed was not allocated[/+] on Mar 5, 2021
  3. zanderso
    added
    c: crashStack traces logged to the console
    on Mar 8, 2021

  4. zanderso commented on Mar 8, 2021

    @zanderso
    zanderso
    on Mar 8, 2021
    Member

    Also on flutter_gallery_ios__transition_perf:

    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20flutter_gallery_ios__transition_perf/134/overview

  5. kf6gpe commented on Mar 8, 2021

    @kf6gpe
    kf6gpe
    on Mar 8, 2021
    Contributor

    /cc @kf6gpe

  6. chinmaygarde commented on Mar 8, 2021

    @chinmaygarde
    chinmaygarde
    on Mar 8, 2021
    Contributor

    @jmagman: Right now, errors like this are hard to trace because they have no backtrace. However, setting the appropriate environment variables (specifically MallocStackLogging) before running running the application should give us more detailed information. Can we enable this for our perf tests please?

    As an aside, we should enable the same for engine unit-tests and also enable asan instrumented builds.

  7. chinmaygarde
    added
    P2Important issues not at the top of the work list
    P1High-priority issues at the top of the work list
    and removed
    P2Important issues not at the top of the work list
    on Mar 8, 2021

  8. jmagman commented on Mar 8, 2021

    @jmagman
    jmagman
    on Mar 8, 2021
    MemberAuthor

    Can we enable this for our perf tests please?

    I don't know how adding the Malloc env variables would impact the benchmark numbers. This one is https://github.com/flutter/flutter/tree/master/dev/benchmarks/macrobenchmarks/ios. Feel free to edit anything that would be useful for the engine team to triage.

  9. chinmaygarde commented on Mar 8, 2021

    @chinmaygarde
    chinmaygarde
    on Mar 8, 2021
    Contributor

    I don't know how adding the Malloc env variables would impact the benchmark numbers.

    Negatively, and significantly so.

    It is probably not viable to enable this for the benchmarks. But it could be useful in de-flaking. Right now, there is very little information to work with.

  10. zanderso commented on Mar 8, 2021

    @zanderso
    zanderso
    on Mar 8, 2021
    Member

    If the malloc flags will be too big a hit to enable by default, then this will have to be repro'd locally with the flags. This sort of flake makes me nervous esp. when we aren't getting a backtrace to help localize it, so I am going to bump this to P2, and work on finding an owner. The priority can be dropped once we know where this is coming from, if appropriate.

  11. zanderso
    added
    P2
    and removed
    P1High-priority issues at the top of the work list
    on Mar 8, 2021

  12. iskakaushik commented on Mar 9, 2021

    @iskakaushik
    iskakaushik
    on Mar 9, 2021
    Contributor

    I was able to get a stack trace for this:

    Target 0: (Runner) stopped.
(lldb) bt
* thread #19, name = 'DartWorker', stop reason = signal SIGABRT
  * frame #0: 0x00000001cd52a414 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001eb084b50 libsystem_pthread.dylib`pthread_kill + 272
    frame #2: 0x00000001a8a03b74 libsystem_c.dylib`abort + 104
    frame #3: 0x00000001af3e149c libsystem_malloc.dylib`malloc_vreport + 560
    frame #4: 0x00000001af3e1740 libsystem_malloc.dylib`malloc_zone_error + 104
    frame #5: 0x00000001af3cc934 libsystem_malloc.dylib`free_list_checksum_botch + 40
    frame #6: 0x00000001af3c9944 libsystem_malloc.dylib`small_free_list_remove_ptr_no_clear + 1232
    frame #7: 0x00000001af3cbbcc libsystem_malloc.dylib`small_malloc_from_free_list + 156
    frame #8: 0x00000001af3ca06c libsystem_malloc.dylib`small_malloc_should_clear + 232
    frame #9: 0x00000001af3c6d38 libsystem_malloc.dylib`szone_malloc_should_clear + 136
    frame #10: 0x00000001af3d5518 libsystem_malloc.dylib`_malloc_zone_malloc + 152
    frame #11: 0x00000001021aadc8 Flutter`dart::TimelineEventEndlessRecorder::GetNewBlockLocked() + 32
    frame #12: 0x00000001021a9be8 Flutter`dart::TimelineEventRecorder::ThreadBlockStartEvent() + 316
    frame #13: 0x00000001021a8f04 Flutter`dart::TimelineBeginEndScope::EmitEnd() + 76
    frame #14: 0x00000001021a8d4c Flutter`dart::TimelineBeginEndScope::~TimelineBeginEndScope() + 36
    frame #15: 0x00000001020b90a0 Flutter`dart::IsolateMessageHandler::HandleMessage(std::__1::unique_ptr<dart::Message, std::__1::default_delete<dart::Message> >) + 3100
    frame #16: 0x00000001020ce0c8 Flutter`dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool) + 404
    frame #17: 0x00000001020cdc20 Flutter`dart::MessageHandlerTask::Run() + 604
    frame #18: 0x00000001021a85a4 Flutter`dart::ThreadPool::Worker::Main(unsigned long) + 380
    frame #19: 0x0000000102134104 Flutter`dart::ThreadStart(void*) + 260
    frame #20: 0x00000001eb083cb0 libsystem_pthread.dylib`_pthread_start + 320
(lldb)

  13. iskakaushik commented on Mar 9, 2021

    @iskakaushik
    iskakaushik
    on Mar 9, 2021
    Contributor

    More complete back-trace:

    (lldb) b malloc_error_break
Breakpoint 1: where = libsystem_malloc.dylib`malloc_error_break, address = 0x00000001af3d489c
2021-03-09 12:57:24.040437-0600 Runner[47817:12095973] fopen failed for data file: errno = 2 (No such file or directory)
2021-03-09 12:57:24.040455-0600 Runner[47817:12095973] Errors found! Invalidating cache...
Runner(47817,0x16bc63000) malloc: Incorrect checksum for freed object 0x106f07010: probably modified after being freed.
Corrupt value: 0xffffffffffffffff
Runner(47817,0x16bc63000) malloc: *** set a breakpoint in malloc_error_break to debug
Process 47817 stopped
* thread #22, queue = 'com.Metal.CompletionQueueDispatch', stop reason = breakpoint 1.1
    frame #0: 0x00000001af3d489c libsystem_malloc.dylib`malloc_error_break
libsystem_malloc.dylib`malloc_error_break:
->  0x1af3d489c <+0>:  pacibsp
    0x1af3d48a0 <+4>:  stp    x29, x30, [sp, #-0x10]!
    0x1af3d48a4 <+8>:  mov    x29, sp
    0x1af3d48a8 <+12>: nop
Target 0: (Runner) stopped

    For all threads: https://gist.github.com/iskakaushik/16dac01b01ec6395fb7cb3d5837ba232

  14. iskakaushik
    self-assigned this
    on Mar 9, 2021

  15. zanderso commented on Mar 11, 2021

    @zanderso
    zanderso
    on Mar 11, 2021
    Member

    Copying over from a chat thread: @iskakaushik will attempt an ASAN run, and the Skia fix here https://skia-review.googlesource.com/c/skia/+/381222 could be related.

  16. iskakaushik commented on Mar 16, 2021

    @iskakaushik
    iskakaushik
    on Mar 16, 2021
    Contributor

    I'm closing this to continue discussion and debugging efforts at: #78076 given that this is a P1 and this is a P2.

  17. iskakaushik
    closed this as completedon Mar 16, 2021

  18. github-actions commented on Aug 4, 2021

    @github-actions
    github-actionsbot
    on Aug 4, 2021

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  19. github-actions
    locked as resolved and limited conversation to collaborators on Aug 4, 2021
  20. flutter-triage-bot
    added
    P0Critical issues such as a build break or regression
    and removed
    P2
    on Jun 28, 2023
  21. flutter-triage-bot
    added
    c: flakeTests that sometimes, but not always, incorrectly pass
    on Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionc: contributor-productivityTeam-specific productivity, code health, technical debt.c: crashStack traces logged to the consolec: flakeTests that sometimes, but not always, incorrectly passengineflutter/engine related. See also e: labels.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions