Repository navigation
Android Presentation API and iOS AirPlay #99
Description
Monday Jul 20, 2015 at 14:19 GMT
Originally opened as https://github.com/flutter/engine/issues/130
It would be really great if Sky could support the use case of having two independent display/view instances that can communicate together. This would make Sky very appealing to game developers who want to make multi-screen games (phone/tablet as controller and large TV/monitor as second screen).
Both Android and iOS have support for this use case today using the standard frameworks.
There are two issues that would need to be addressed to make this work:
- A simple way to connect the two Sky views/apps so that they can quickly pass messages (mostly it will be one way messages from the controlling view to the dumb screen view). A critical feature would be to have a way for the controlling view to know when the other view is available to receive messages (can't start the game until all views are initialized).
- Exposing the necessary API in Dart to query for external displays and to be able to create a view to send to the external display. For Android this would be exposing the Presentation API and for iOS there is a Multiple Display Programming Guide for iOS.
For inspiration, here is a cordova Presentation API plugin that implements all of the necessary plumbing for both Android and iOS:
https://github.com/fraunhoferfokus/cordova-plugin-presentation
Activity
FlutterIssues commentedon Nov 9, 2015
Monday Jul 20, 2015 at 18:00 GMT
FWIW, I designed the Android presentation API. Let me know if you have any
questions.
On Mon, Jul 20, 2015 at 7:19 AM Lex Berezhny [email protected]
wrote:
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 00:57 GMT
@vlidholt could be something for you to look at as part of gaming?
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 01:21 GMT
Fwiw, I am putting some thought into how the Sky Shell should evolve to
support Android features such as this. It will be nontrivial given the way
things are plumbed right now.
On Wed, Jul 22, 2015, 5:57 PM Ian Hickson [email protected] wrote:
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 01:33 GMT
My initial ticket implies that the two views are running in different isolates but I don't know if this is necessarily the case. I noticed some code recently where Android Activities (I think that's what they were) were getting instantiated and started from within Sky app. Seems like it shouldn't be much of a stretch to implement the Presentation API patterns then. But maybe I'm missing something, Sky has accelerated in development so much lately that it's been difficult keeping track... that's a good thing! 👍
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 01:52 GMT
The Sky code may run in isolates but the Android code shares the state of
one process. Currently native viewport is designed to attach to the
current Activity. This will need to be enhanced to allow viewport backed
by Dialogs.
I've started sketching out some thoughts around how the Android embedding
of Sky / Mojo may be improved going forward (partly for performance as well
as creating a clearer path for adding enhanced capabilities). So if you
can sit tight until I get back from Seattle we can chat about it next
week. :)
Now if you're thinking about the Cast story in general (not just
Presentations but also MediaRouter, MediaSession, etc.) then we have a long
way to go! The lifecycle considerations get quite interesting (because we
need to go beyond Activities...)
On Wed, Jul 22, 2015, 6:33 PM Lex Berezhny [email protected] wrote:
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 02:07 GMT
I've got an idea for an app that I want to build, a few Android devices, a Chromecast and I know Dart pretty well so I'm very interested in any workable solution that would let me use the Chromecast as a second screen from a Sky app.
Several months ago I attempted to get this all working on my own by using network sockets to communicate between two Dart isolates that I was starting from the Java/Dalvik side. But it was quite a bit of a hack and just wasn't really workable. I also hard coded the C++ Sky engine to start two Sky Views... like i said it was a monster :-)
I think it's best if I wait for someone who knows what they are doing to implement this part. I'll be happy to test and experiment once there is something working though.
My app idea is for an eye exercise program that helps with focusing and convergence insufficiency issues. Currently behavioral eye therapists use the "Hart Chart" for this but it's a little too low tech and gets insanely boring after the first couple of times and practically impossible to get a child to do it. I have eye issues and my daughter who is 9 also suffers from this, a digital version of the Hart Chart would be a life saver (literally). I have a prototype that I implemented using WebViews but communicating between WebViews is more than a bit slow on Android (because they run on different threads and you have to synchronize the messages between them, it's a mess)...
Anyways, I'm really looking forward to this feature being supported in Sky!
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 02:30 GMT
Cool. Well I can certainly help with this stuff (given I designed most of
the Android Framework parts). :)
Now that said, you might not even need to rely on Presentation for this use
case. You'll need to activate Chromecast mirroring for that to work anyhow
at which point perhaps you could just draw the content you want on the
device as usual and let it be mirrored over.
(Presentation is mostly interesting for showing different content on
different displays.)
Another option would be to develop a native Chromecast app and connect to
it from the phone. Note that the APIs involved in doing that are somewhat
different (still involves MediaRouter but not Presentation; instead you
talk to the Cast API in Google Play Services).
So lots of options depending on the desired user experience.
On Wed, Jul 22, 2015, 7:07 PM Lex Berezhny [email protected] wrote:
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 02:33 GMT
Could you create a "Hello World" Sky app where it would list available displays, then after picking a display it would show something on that display?
FlutterIssues commentedon Nov 9, 2015
Thursday Jul 23, 2015 at 02:37 GMT
Ideally the user shouldn't have to choose the display itself but rather the
intended device to project onto. That's handled by MediaRouter.
Now if you want to see a native Android app that does enumerate displays
and shows content on them look at the Presentation samples in
development/samples/ApiDemos in the Android tree.
Jeff.
On Wed, Jul 22, 2015, 7:33 PM Lex Berezhny [email protected] wrote:
Hixie commentedon Nov 9, 2015
(note: comment stream got truncated in migration)
eseidelGoogle commentedon Sep 20, 2016
This is not currently on our near-term planning. I suspect it's very likely we'll provide a plugin for this, but I don't think we need this open at this time to track that.
Remove mockito, markupsafe, pymock, roboelectric, JSR-305 and jinja2 …
github-actions commentedon Sep 6, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.