Repository navigation
TextField should support apple pencil input by default. #61278
Description
Activity
Does anyone have any thoughts and/or direction on this?
I took a crack at implementing
UIIndirectScribbleInteractionDelegateto see how that might work. The upshot is, I can get text that users write, but it doesn't support all of the scribble interactions like scratching out text to erase it since I'm not giving the iPadOS an actual representation of the text in order for it to know what word(s) to erase.I'm imagining that we will need some kind of custom UITextField-ish representation of the EditableText from flutter.
@cbracken Your name is all over some TODOs in the fields I've been messing with trying to figure out how to implement this, so figured I would tag you to see if you have any thoughts.
I have been playing around in
FlutterTextInputPlugin.mm:
twinsunllc/engine@c5664a8So far, it seems that listening for the
TextInput.setEditableSizeAndTransformmethod call and setting the frame of the_activeViewand implementing- (CGRect)firstRectForRange:(UITextRange*)rangeso that it returns a rect roughly in the area of the given range are the minimum needed.There are still some funny issues (and you can't start scribbling till the field is focused), but that seems like a starting point. Do you have any thoughts on the right way to go about implementing
firstRectForRange? I imagine that needs to accurately transform the given range into a rectangle matching where that text is on the screen.Tagging @justinmc and @LongCatIsLooong as well since you have recent commits in that file.
CC @xster in case you have any existing plans for Apple Pencil support.
Do we have any existing text code that would help like this?https://github.com/flutter/engine/blob/63a78df571e88e5bf715645afb2a6b0fbf510706/lib/ui/text/paragraph.cc#L118
It would be pretty cool if we could get this working. Thanks @fbcouch for investigating.
@justinmc Thanks for pointing out that code. That actually helped me make a little more progress on this.
I couldn't figure out how to get access to the Paragraph objects from FlutterTextInput, so what I've done is add some glue code on the flutter side (meaning, in the flutter/flutter repo, not sure how you refer to this): twinsunllc@7b39621
That sends the selection boxes through the channel, and then used those on the engine side to implement some of the relevant methods: https://github.com/twinsunllc/engine/tree/feature/scribble
This kinda works, but there are still a few issues that I may need some help figuring out:
- The FlutterTextInput having a size seems to "consume" tap events so that the cursor can't be moved around with non-scribble methods (instead of being able to tap with the pencil or a finger, you have to draw a short line with the pencil similar to how you can make a line through a word to select it). I tried forwarding the touchesBegan, etc selectors to the superview, but that didn't help.
- There do seem to be some issues still remaining around managing the selected text. Somehow, I keep ending up with the last character selected in addition to whatever actual selection is there.
- Once text is selected, making any marks will (usually) replace that text rather than making a new selection. I suspect that is related to the previous item. Possibly some kind of issue keeping the FlutterTextInput selection synced up with the flutter side?
- Finally, scribble only works once the input is focused, which is not how it works natively
I'm going to ask about this on the discord channel, but figured it would be good to have my findings recorded here as well.
Sounds like you're getting somewhere. +1 to asking on Discord. You also might consider opening a draft PR to make it easier to take a look at your code.
The FlutterTextInput having a size seems to "consume" tap events
iirc currently
FlutterTextInputViewisn't expecting any touch events. Touch events are forwarded in theFlutterViewController: https://github.com/flutter/engine/blob/1411cb671be8ddf4f16f8d3bd96d7a86a43b6fd3/shell/platform/darwin/ios/framework/Source/FlutterViewController.mm#L934-L948Once text is selected, making any marks will (usually) replace that text rather than making a new selection.
I think
- [FlutterTextInputView setMarkedText:(NSString*)markedText selectedRange:(NSRange)]
does this so multistage input replaces the current selection:
https://github.com/flutter/engine/blob/1411cb671be8ddf4f16f8d3bd96d7a86a43b6fd3/shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm#L759-L767scribble only works once the input is focused
That might be because we try to keep as few
FlutterTextInputs around as possible. iirc if a text field gets unfocus and it has nothing to do with username or password (for autofill) then the correspondingFlutterTextInputgets removed from the view hierarchy, to prevent leaks.Thanks @LongCatIsLooong that is helpful – I pushed a couple of commits to that PR using
UIScribbleInteractionDelegateto disablesetMarkedTextduring scribbling, and pass a reference to theFlutterViewControllerto theFlutterTextInputto forward touch events. That seems to have fixed the issues with touch events and scribble-to-select replacing the current selection. Disabling theshowAutocorrectionPromptRectForStartwithinfirstRectForRangewhile scribble is in progress also got rid of the issue where the last character was already selected.So, I think the remaining issues here are:
- This only works after focusing on the input
- The selection doesn't bring up the selection handles
Between twinsunllc/engine#1 and twinsunllc#1 I have the selection handles working properly now, so the last thing is starting the interaction without having focus already. I think the right thing to do will be using
UIIndirectScribbleInteractionDelegateand trying to pass focus to the correct element...fortunatelyfunc indirectScribbleInteraction(UIInteraction, focusElementIfNeeded: Self.ElementIdentifier, referencePoint: CGPoint, completion: ((UIResponder & UITextInput)?) -> Void)has thatcompletionhandler, so I assume it is not on the UI thread and I could wait for a response to a message over a flutter channel. We will see, anyway.Reacted by Justin McCandlessYep, that approach seems to have worked. I have updated the draft PRs – I'm going to ask in discord to see if I can get some kind of preliminary review of the approach before rebasing onto master/adding tests (which I assume is necessary to open a real PR)
The engine PR has been merged and the framework PR is waiting for final review, then we can ship this.
Reacted by Christopher Boyd, Daniil Shumko, Barton Ip, hatano0x06, John Ryan, otozyon, Bruno D'Luka, Jackson Hu, Alex Rabin and Tim LehmannThis thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.- locked as resolved and limited conversation to collaborators
on Feb 11, 2022
related: #38135
not the complex implementation for pressure, tilt, etc.
but textfield should support apple pencil input by default.
figure 1 - text recignition on default text inputs.
more info about this feature
apple pencil text input works on web html inputs by default, i think it just needs to be recognized as input from ios.
Great thanks to the flutter team.