Widget bindings should be generated - Lots of widgets are missing

[benbucksch]

Svelte NodeGUI sounds just what I need for my application.

The docs list only a number of very basic widgets.
These seem to be implemented in React NodeGUI, and Svelte NodeGUI just bridges over the React implementations.

For my real world application, I need a much larger number of Qt widgets. Without more widgets, I cannot even make a feasability study whether Svelte NodeGUI is suitable for my application, because the current set of widgets is just too small to find out whether it would work. Theoretically, I could add the widgets myself, but there are too many missing.

However, I am faced with 5 layers of abstraction:

1. OS widgets ->
2. Qt ->
3. Node GUI ->
4. React NodeGUI ->
5. Svelte NodeGUI

This makes it difficult to add new widgets, because I need to understand a lot of intermediate wiring code, to add a widget. Worse, the mappings seem to be hand-written.

May I suggest a different approach? Why don't you take the Qt widget API descriptions in computer-readable form, and then use Swig or a custom-made software to generate the bindings, for NodeGUI and each of React/Vue/Svelte NodeGUI?

This would have the following advantages:

• All Qt widgets supported, in all variants of NodeGUI
• API up to date with new Qt versions
• SvelteGUI would no longer need to use React NodeGUI, eliminating at least one level of indirection. (compare Stop bundling source files from React NodeGUI #21)

[benbucksch]

@a7ul @shirakaba

[shirakaba]

The set of QWidgets supported by Svelte NodeGUI

The docs list only a number of very basic widgets.
These seem to be implemented in React NodeGUI, and Svelte NodeGUI just bridges over the React implementations.

Svelte NodeGUI supports all the widgets that React NodeGUI does (unless I've missed any, but I did look quite hard), plus QSvg. The full set is listed here:

svelte-nodegui/src/svelte-nodegui.ts

Lines 62 to 158 in 1ce01eb

	image: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").ImageProps,
	import("./dom/react-nodegui/src").RNImage
	>;
	animatedImage: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").AnimatedImageProps,
	import("./dom/react-nodegui/src").RNAnimatedImage
	>;
	view: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").ViewProps<any>,
	import("./dom/react-nodegui/src").RNView
	>;
	checkBox: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").CheckBoxProps,
	import("./dom/react-nodegui/src").RNCheckBox
	>;
	text: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").TextProps,
	import("./dom/react-nodegui/src").RNText
	>;
	dial: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").DialProps,
	import("./dom/react-nodegui/src").RNDial
	>;
	lineEdit: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").LineEditProps,
	import("./dom/react-nodegui/src").RNLineEdit
	>;
	window: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").WindowProps,
	import("./dom/react-nodegui/src").RNWindow
	>;
	progressBar: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").ProgressBarProps,
	import("./dom/react-nodegui/src").RNProgressBar
	>;
	comboBox: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").ComboBoxProps,
	import("./dom/react-nodegui/src").RNComboBox
	>;
	button: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").ButtonProps,
	import("./dom/react-nodegui/src").RNButton
	>;
	spinBox: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").SpinBoxProps,
	import("./dom/react-nodegui/src").RNSpinBox
	>;
	radioButton: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").RadioButtonProps,
	import("./dom/react-nodegui/src").RNRadioButton
	>;
	tab: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").TabProps,
	import("./dom/react-nodegui/src").RNTab
	>;
	menu: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").MenuProps,
	import("./dom/react-nodegui/src").RNMenu
	>;
	menuBar: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").MenuBarProps,
	import("./dom/react-nodegui/src").RNMenuBar
	>;
	plainTextEdit: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").PlainTextEditProps,
	import("./dom/react-nodegui/src").RNPlainTextEdit
	>;
	slider: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").SliderProps,
	import("./dom/react-nodegui/src").RNSlider
	>;
	systemTrayIcon: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").SystemTrayIconProps,
	import("./dom/react-nodegui/src").RNSystemTrayIcon
	>;
	svg: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").SvgProps,
	import("./dom/react-nodegui/src").RNSvg
	>;
	action: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").ActionProps,
	import("./dom/react-nodegui/src").RNAction
	>;
	boxView: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").BoxViewProps,
	import("./dom/react-nodegui/src").RNBoxView
	>;
	gridView: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").GridViewProps,
	import("./dom/react-nodegui/src").RNGridView
	>;
	scrollArea: SvelteNodeGUIProps<
	import("./dom/react-nodegui/src").ScrollAreaProps,
	import("./dom/react-nodegui/src").RNScrollArea
	>;
	}

The Svelte NodeGUI docs (which are incidentally just a copy-paste of the React NodeGUI docs, with some find-and-replace applied and manual code examples added) are evidently missing a large number of the supported widgets. Contributions are welcomed on that front.

For my real world application, I need a much larger number of Qt widgets. Without more widgets, I cannot even make a feasability study whether Svelte NodeGUI is suitable for my application, because the current set of widgets is just too small to find out whether it would work. Theoretically, I could add the widgets myself, but there are too many missing.

I'd say 24 widgets is a good start, to be fair – for example, React Native provides 21 cross-platform core components, plus two iOS-specific ones and two Android-specific ones. We've recently merged two PRs to add extra widgets (namely <svg> and <scrollArea>), so I think that the barrier to adding new widgets from NodeGUI is also quite low.

Adding support for new Qt widgets to Svelte NodeGUI

This makes it difficult to add new widgets, because I need to understand a lot of intermediate wiring code, to add a widget. Worse, the mappings seem to be hand-written.

I know it's missing the point, but I might as well explain the wiring code while we're on the subject. The wiring code in that file, index.ts, corresponds to the React custom renderer APIs. You generally don't need to touch much, if anything, in those files; just copy-paste them and make the naming consistent. The only things to think about would be shouldSetTextContent (true for any element that can accept text nodes as children) and createInstance (where you define the initial props for the element – usually just {} – and any post-initialisation you might want to do). And to tell the truth, even that bit of effort is all just to make it easy to port contributions back over to React NodeGUI – Svelte NodeGUI doesn't even refer to those index.ts files.

To add support for a Svelte NodeGUI component, the minimum is to create an React NodeGUI element following the pattern set out in the self-named files like RNComboBox.ts. The methods are:

• setProps: Just call the props setter that's defined at the top of the file. That setter internally may call setViewProps at the end of its execution as RNComboBox does here if the underlying NodeGUI component inherits the common view props, which they do if they implement RNWidget (implying that it's visual), but not if they only implement RNComponent (as RNAction does, implying that it's non-visual).
• appendInitialChild: Just call appendChild here. Technically this is a method to allow you to have a special appendChild implementation for the component at the base of a React element hierarchy, but the use-cases are pretty exotic.
• appendChild, insertBefore, removeChild: These correspond to the DOM methods of the same name. Either call throwUnsupported here to show that the component doesn't support child elements at all, or call the equivalent NodeGUI APIs for performing each of these DOM operations.

Automating bindings

May I suggest a different approach? Why don't you take the Qt widget API descriptions in computer-readable form, and then use Swig or a custom-made software to generate the bindings, for NodeGUI and each of React/Vue/Svelte NodeGUI?

Automating generating the bindings from Qt all the way to Svelte NodeGUI would of course be the holy grail (and I'll never disregard the idea), but I'll get into the challenges involved that I think are show-stoppers to automation. We actually use automation to a certain extent on Svelte Native and React NativeScript to map NativeScript typings to JSX, but even with that successful tool, we can see that it wouldn't transfer to mapping the runtime APIs to equivalent DOM methods due to the number of exceptional cases and lack of consistency.

I haven't looked into how Atul generates the NodeGUI bindings for the Qt widgets, but I wouldn't be surprised if a certain amount of it is automated. Though as there are various missing Qt APIs, maybe it is all a manual process.

Mapping the Qt widget APIs to DOM methods is the ultimate goal, and sadly I don't think this is realistically possible to automate, because Qt doesn't have a consistent API for managing the element tree.

For example, how would you add a child QWidget to QMainWindow? You'd need to look at the docs to determine that it's setCentralWidget(widget)... unless, of course, the child in question is a QMenuBar, in which case it's more likely that you want to call setMenuBar(menuBar). How would you reverse that operation (e.g. to support HMR or simply for swapping the child in an {#if}{/if} block)? Would you call setCentralWidget(null) and setMenuBar(null)? I've no idea; we'll have to check the docs to see. And in some cases, like in the NativeScript UI runtime, certain operations just like that are simply not reversible as they were only ever designed as imperative APIs rather than for declarative use. I wouldn't be surprised of similar cases in Qt widgets.

Setting attributes is also a pain. You will see various cases in RNView where the setters are not simple cases of calling set[propName](value) but involve spreading the values from an object, or calling imperative APIs. This may be partially due to how NodeGUI maps out those APIs, but again, as with NativeScript, I wouldn't be surprised that plenty of the APIs are too inconsistent to map automatically. One obstacle in particular is the existence of APIs like setMenuBar, as mentioned above – it's worded like an attribute setter, but functionally, it adds a child element. Qt doesn't distinguish between attribute accessors and hierarchy-modifying APIs. So I don't see how it could practically be automated.

What Qt needs is a consistent set of APIs for getting/setting attributes and inserting/removing widgets into a hierarchy, such as the DOM model. If that existed, then renderers like React NodeGUI wouldn't need to manually map each widget to consistent APIs. Unfortunately, unless a project like that exists and has been completed, then we're stuck doing things manually.

[benbucksch]

Hi Jamie, first off, thank you for your very detailed response! I appreciate it.

svelte-nodegui/src/svelte-nodegui.ts

Thanks. That's the list of widgets I had found and looked at.

I'd say 24 widgets is a good start, to be fair

Indeed, what you and Atul accomplished here is seriously impressive. I love the concept. This is no small feat, and that you two developers got this done, probably not even as full time job, really speaks to your competence. I've written similar frameworks in the past, and I know it's hard, and a lot of work. You can be very happy about your accomplishment. Respect.

What I try to do is also challenging: I would like to implement a mail client on a quality and usability level on par with (or better than) Thunderbird or Outlook, for all 5 major platforms (desktop and mobile). You can probably see easily that the widget list above is not even close to sufficient for a project of this nature.

I'm looking at https://doc.qt.io/qt-5/widget-classes.html , and a lot of them will be useful, but there are a number that are not yet supported and absolutely necesssary:

• TreeView, with columns, capable to scroll 1 million rows without delay or lag
• ListView
• Toolbar / button
• Splitter
• Tabs
• HTML view (no JS, no network, with data: images)
• HTML editor (ditto)
• Browser view (full HTML5 browser engine with JS, SVG, CSS etc.)

That's just the start, the absolute minimum. And that's just for the prototype - the final app likely will need more. It's going to take a while to map all these manually, with their entire API respective surface. I will not be able to do that, nor can I ask you to do it. Hence, my suggestion here.

Qt doesn't have a consistent API for managing the element tree. ...
Would you call setCentralWidget(null) and setMenuBar(null)?

Yes, I see what you mean. Thanks for the concrete example, that helps me to understand the problem. It might be possible to solve that generically, if the generator understands the types, that a <menu> -> QMenu fits to setMenuBar(QMenu), but I concur it's not going to be easy. And it's going to fail in a number of cases. I can see the problem.

Setting attributes is also a pain. ... Qt doesn't distinguish between attribute accessors and hierarchy-modifying APIs.
What Qt needs is a consistent set of APIs for getting/setting attributes

RNComboBox.ts looks very repetetive and straight-forward. Would it be a reasonable approach to create a declarative (e.g. plaintext) list of attributes per widgets, and then generate these setters and getters from that list, instead of manually writing these setters and getters?

Then, as a second step, one could see whether that list could be generated from the Qt header files. The header parser would consider not only the function names, but also their parameter types.

That's just an idea. What do you think?

[shirakaba]

What I try to do is also challenging: I would like to implement a mail client on a quality and usability level on par with (or better than) Thunderbird or Outlook, for all 5 major platforms (desktop and mobile). You can probably see easily that the widget list above is not even close to sufficient for a project of this nature.

That sounds like a nice fit for a Qt app indeed. I've not seen any prior art on deploying NodeGUI to mobile via Qt for mobile, however, so it's uncharted territory.

I'm looking at https://doc.qt.io/qt-5/widget-classes.html , and a lot of them will be useful, but there are a number that are not yet supported and absolutely necesssary:

True, those are missing from NodeGUI.

It's going to take a while to map all these manually, with their entire API respective surface. I will not be able to do that, nor can I ask you to do it. Hence, my suggestion here.

Indeed, the browser view alone probably has an enormous API surface. Yes, I agree that an automated process for producing bindings from Qt to NodeGUI would be ideal, though I don't have the expertise to help with that.

The mapping of APIs from Qt to NodeGUI is non-trivial and I don't know how realistic it would be to automate it. You have to determine how to marshal the Qt C++ values into equivalent JS abstractions, and also have to handle the memory management. Even doing this manually when trying to add the saveGeometry API, I didn't know what I was doing (I have no expertise in C++) and whether it was the right way to marshal a QByteArray into an equivalent Napi object.

It won't be possible to auto-generate bindings without first creating a way to marshal each and every Qt type into a Napi equivalent. This is an unrealistically large task (on the scale of building NativeScript, but for the Qt runtime instead of iOS/Android), so I think the train stops here, unfortunately.

Yes, I see what you mean. Thanks for the concrete example, that helps me to understand the problem. It might be possible to solve that generically, if the generator understands the types, that a

-> QMenu fits to setMenuBar(QMenu), but I concur it's not going to be easy. And it's going to fail in a number of cases. I can see the problem.

I ran into exactly this when building a React renderer for NativeScript, and there were so many edge cases that I think if we were going to automate the mapping from NodeGUI to React NodeGUI, it would be most prudent to just automatically map as many APIs as possible, but for any case where a QObject is a param, simply write out a stub saying "this API needs to be filled in manually" to be addressed in a post-processing step.

There are enough NodeGUI components still missing from React NodeGUI that there would be some benefit in writing a tool, using ts-morph, probably, to automate some of this. But there's still no way around the issue of mapping Qt to NodeGUI in an automated fashion, which unfortunately is the most important part for supporting the full Qt API surface.

RNComboBox.ts looks very repetetive and straight-forward. Would it be a reasonable approach to create a declarative (e.g. plaintext) list of attributes per widgets, and then generate these setters and getters from that list, instead of manually writing these setters and getters?

I don't know exactly how Atul wrote out the getters and setters, but it may be that he started with a similar process – maybe not automated, but nonetheless quick. With multi-line text editing, the task is pretty small if you're only concerned with doing one component. For a large batch of components, something like the ts-morph idea may have some value.

Then, as a second step, one could see whether that list could be generated from the Qt header files. The header parser would consider not only the function names, but also their parameter types.

Again, this comes back to the issue of needing a tool to marshal values from Qt to NodeGUI in order to be of any use. I suppose at least having the API stubs for contributors to fill in would be handy, though, so that we'd have a proper picture of how many APIs are missing. But that's a discussion to bring up with Atul as it concerns NodeGUI itself.