After updating a remote dependency on pubspec.yaml to a local version, the debugger keeps referencing the remote version #4076

Describe the bug
When you're using a dependency and swap it for a local version of it, if you add a breakpoint in the local files, it will highlight and traverse the cached remote dependency files instead of the local ones.

I work with an app that uses a Flutter Widget Library maintained by my company. Sometimes I need to debug that library, which I can only do from the actual app. For that, I swap the remote dependency to point to a local version, which I can easily make changes to. However, any breakpoints I use will end up highlighting the cached files instead, making debugging harder, since the cursor will be all over the place after I make changes to the local files.

The problem seems to go away after some Hot Restarts or installing the app again, but it's really inconsistent and time-consuming.

I've only experienced in Flutter projects, but I didn't test to see if it also happens in any kind of Dart project.

To Reproduce

1. Add a remote dependency to pubspec.yaml and use something from it in your Flutter app, like auto_size_text.
2. Access the remote files via VSCode by inspecting a symbol exported from the package, which Dart caches in your machine.
3. Clone the dependency's repository to your machine.
4. Make sure VSCode is debugging with "Debug my code + packages".
5. Run the app.
6. While it's running, update the dependency in pusbpec.yaml to point to the local version you cloned, like so:

auto_size_text: #^3.0.0
    path: ../auto_size_text

7. Add a breakpoint to a file from the local version you cloned.
8. "Hot Restart" the app.
9. When the debugger hits the breakpoint, it will bring up, highlight and traverse the cached remote files instead of the local files.

Expected behavior
The editor should bring up the local version of the files instead of the cached remote files, since a breakpoint was put into the former and not the latter.

Screenshots
An example: I add a new line and a breakpoint to the local package:

Then, If I change the dependency to the local version and hit Hot Restart, the editor brings up the cached files. Note that the highlight
cursor is in the wrong position, since a new line was added in the local file:

Please complete the following information:

• Operating System and version: Windows 10
• VS Code version: 1.67.1
• Dart extension version: 3.44.0
• Dart/Flutter SDK version: Flutter 2.10.4 (stable) - Dart 2.16.2
• Target device (if the issue relates to Flutter debugging): Tested on Pixel 4 and Nexus 9 (API 29) Android Emulators.