feat(graphics): DisplayMirror for screen mirroring and remote input - #394
jamesarich wants to merge 9 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
The failure is in generating This diff is five C++ files ( What it looks like instead: I have not proven it by reproducing locally, so treat that as a strong hypothesis rather than a finding. If it holds, it will redden every PR until Note #393, opened the same day, shows no |
61cfbf7 to
6870270
Compare
|
Correcting my own comment above: the timing evidence I gave for the I said the last green What still stands: this diff is five C++ files under I have re-triggered CI to get a second datapoint. Treat the cause as unknown until that comes back, not as diagnosed. |
|
As a small hint: InputDriver is a base class, you won't put any implementation into here. Same goes for DisplayDriver. Please take more effort into your design choices before pushing PRs. What you probably want is a DisplayMirror class and put all implementation there. |
Appreciate the first look - this is draft specifically because of my non-confidence in the design choices 😅 . |
|
CI resolved: So the |
|
Extracted a What is left in existing files is 8 lines:
Everything else is Two things fell out of the move rather than being carried across:
Still draft, still a PoC. Thanks for the pointer, it is a much better shape. |
jamesarich
changed the title
|
Though still draft but let me remark that the current design how the class is integrated is still bad as it introduces artificial dependencies which are not in line with the current architecture. Class DisplayMirror with its static methods should work behind the scenes with no dependencies to it. So this has to be removed (inversion of control): LGFXDriver is just one out of many other possible choices of subclasses of DisplayDriver (think of e.g. EInkDriver, eTFTDriver, AdafruitDriver) depending on the driver class which all could be mirrored to another display. Said that it make sense to add a protected generic flush Also note, that with your DisplayMirror header declaration |
|
Good direction, thanks @mverch67 |
DisplayDriver gains a static flush observer invoked from the LGFX flush callback before the in-place panel byte-swap, so observers receive native little-endian RGB565 dirty rects (LVGL thread; copy and return), plus a thread-safe requestFullRefresh drained in task_handler so a newly attached observer can synchronize the full frame. Both flush variants stay in sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lv_scr_act covers only the active screen; clock and notification overlays live on the top and system layers, so a full-frame sync for a flush observer must repaint those too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
InputDriver gains injectTouch/injectKey statics callable from any thread: events land in lock-free SPSC rings drained by two always-present virtual LVGL devices created in the (previously empty) base init — a pointer whose read callback holds PRESSED for a requested duration (taps and synthesized long presses) and a keypad attached to the default group, which init now guarantees exists even on boards with no physical input so injected keys can navigate widgets. Mirrors the flush-observer pattern on the output side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fixes for remote control. First: a slept panel swallowed injected input as a wake event, which is every event when nobody is physically at the device — inject* now raises a wake request drained on the UI thread (force wakeup when powersaving, otherwise reset the inactivity timer), distinct from toggleDisplay which would sleep an awake screen. Second: add a virtual encoder device, because LVGL moves focus between widgets on encoder rotation while keypad UP/DOWN go to the already-focused widget — so arrow keys silently did nothing while typed characters worked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Creating the default input group unconditionally would enrol every focusable widget on touch-only boards that have no focus concept, so injection is now opt-in: a host calls enableInjection() before init(), and init() is otherwise inert as before. The flush observer becomes atomic with a documented detach-quiesce contract, its comment records that the tightly-packed pixel guarantee is PARTIAL-render-mode only and that LGFXDriver is the only driver honouring it, the injector contract narrows to single-producer (the rings are SPSC), and encoder steps clamp instead of truncating through int8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
InputDriver and DisplayDriver are base classes, so the mirror had no business living in them. All of it - the flush observer, the wake and full-repaint requests, and the virtual pointer, keypad and encoder - now sits in DisplayMirror. Both base classes revert to master. What is left in existing files is two hook lines in LGFXDriver::display_flush and a DeviceScreen accessor for the display driver. DisplayMirror::start() replaces InputDriver::enableInjection(): a host calls it after DeviceScreen::init(), where LVGL is already up and the view has not built its widgets yet, so the ordering flag on InputDriver::init() is gone. The wake and repaint requests ride the LVGL read callbacks rather than DisplayDriver::task_handler, so the mirror adds no periodic work of its own.
It touches LVGL directly instead of hopping threads, so it has to run before the host's UI task exists. That was implicit in the ordering note; say it.
The driver framework no longer knows DisplayMirror exists. DisplayDriver gains a generic flush notification - a static FlushCallback plus a protected flush() - and DisplayMirror registers a lambda in start(). LGFXDriver drops the DisplayMirror include and calls the inherited hook instead, so any other DisplayDriver subclass gains mirroring by adding the same one line. The callback is static because the LVGL flush callbacks a subclass registers are plain C function pointers with no instance to hand back; every driver here already keeps a static self-pointer for that reason. Registered once in start() and never cleared: assigning a std::function while the LVGL thread may be calling it is not safe, so capture stays gated on the atomic observer and the render path stays a single acquire load. LVGL also leaves DisplayMirror's header. It is how the class is implemented, not part of what it offers, and firmware includes this header to reach start()/inject*(); the indev pointers, read callbacks and rings move into an anonymous namespace in the .cpp.
cc2796a to
cd7ef66
Compare
2 participants
Adds
DisplayMirror: a self-contained class that streams the rendered screen out to a host and feeds the host's input back into LVGL. Inert unless a host callsstart(), so no default build changes behaviour.This is the missing piece of the screen-mirroring stack. firmware consumes it today through a
symlink://override inplatformio.ini, because no released device-ui carries it. Until this lands and a pin bumps, the whole MUI/colour path in meshtastic/firmware#11681 compiles out (HAS_MUI_MIRRORis 0) and only the 1bpp mono path works.What this adds
include/graphics/DisplayMirror.handsource/graphics/DisplayMirror.cpp. All of it lives there.Frame capture.
setFrameObserver()takes a dirty-rect sink, called on every LVGL flush before the panel byte-swap, so it sees(x, y, w, h)and pixels as native little-endian RGB565 with rows tightly packed. Null by default, loaded with acquire semantics on the flush path.A full-repaint request. A host attaching mid-session needs a whole frame, not whatever changes next.
requestFullRefresh()is callable from any thread and invalidates the active screen pluslv_layer_top()andlv_layer_sys(): overlay content (clock, notifications) lives on those layers, so a client would otherwise see a stale overlay.Remote input.
start()registers three LVGL virtual input devices, a pointer, a keypad and an encoder, fed from 16-entry single-producer rings.injectTouch()takes a hold duration so long-press works.injectEncoder()maps rotation onto MUI's trackball semantics, since encoder rotation is what actually moves focus in a group.A panel wake. Injected input has to wake a slept panel and still act, or the first remote event is swallowed as a wake, which is every event when nobody is physically at the device. The wake and repaint requests are drained on the LVGL thread from the read callbacks, so the mirror adds no periodic work of its own.
What it touches outside itself
8 lines, in two files:
LGFXDriver::display_flush, oneDisplayMirror::onFlush(...)call per overload. The only place the pixels exist.DeviceScreen::getDisplayDriver(), a one-line accessor, so a host can hand the mirror the driver it needs for panel wake.InputDriverandDisplayDriverare untouched, identical to master.Scope and safety
start()nothing is allocated, no input device is registered, and a flush costs one acquire load.<atomic>is the only added include.FromRadioand honouring a remote arm would stream the screen to whatever local client happens to be attached.Feature stack
Five repos, and the release order runs top to bottom. Nothing below can ship until the piece above it lands.
DisplayMirror: frame capture + remote inputDisplayFrame/DisplayPalette/DisplayInfo+ the two admin verbscapture_displaytoolingdevice-ui#394 is the current blocker: without it the MUI/colour path compiles out of every committed firmware configuration, leaving only the 1bpp mono path.
Testing
Exercised on a LILYGO T-Deck via meshtastic/firmware#11681 with a local
symlink://override: live mirror plus D-pad, keyboard and touch control from Meshtastic-Android.That bench run predates the
DisplayMirrorextraction. The refactor itself is verified only by host-compiling the new translation unit against the pinned LVGL 9.3 and by CI here, so the T-Deck pass needs redoing before this leaves draft.