Modernise the toolchain, and rebuild the Qt frontend - #127
Modernise the toolchain, and rebuild the Qt frontend#127ludovicthenot wants to merge 10 commits into
Conversation
The repository did not compile past LDC 1.34: botan 1.13.6 used the
`delete` expression, removed from the language in D 2.109. Unblocking it
pulled a chain of updates, each revealing the next failure:
botan 1.13.6 -> 1.13.8 drops `delete`
memutils 1.0.10 -> 1.0.12 `defaultInit`, const `opEquals`
requests 2.1.2 -> 2.2.1
Provision -> HEAD upstream already fixed a non-lvalue
`headers()[...]` for ldc-1.41
dqt -> qt5 branch the pinned commit fails on D 2.112, and
dqt master is now Qt6-only
That left three genuine bugs in this repository:
- `cast(Blob[])` reinterpreted an array of classes as an array of
interface references. It only worked because these classes implement
Blob first, so the offset happened to be zero. Converted element by
element instead, which is also what the newer frontend requires.
- A stack-allocated QString was handed to `cpp_new!QPushButton`. The
temporary died before the button took it, crashing as soon as a
device connected and the tools tab was populated.
- `clearLayout` unmanaged widgets without reparenting them, so they
stayed children of the tab and kept painting at their old geometry,
drawing every tool button twice.
`sessionAdsid`/`sessionToken` are exposed to the server package because
`private` in D is module-private, and developersession.d cannot otherwise
read what it needs to persist a session.
Install verified end to end on Windows, signing included.
The Qt frontend existed but was never finished: the Install button only logged a line and disabled its tab, and there was no way to sign in. This makes it the usable frontend and drops dlangui as the Windows target. Interface A QSS stylesheet applied over the Fusion style, the only Qt style that honours QSS throughout. Lucide icons, bundled Poppins for the wordmark, read-only fields rendered as values rather than greyed-out inputs. The window keeps its native frame and only hides the system caption, via WM_NCCALCSIZE in nativeEvent, so Windows still owns resizing, snapping, the shadow and rounded corners. Qt's FramelessWindowHint was tried first and meant reimplementing all of that by hand, badly. WM_NCHITTEST reports the header row as the caption, decided on the widget hierarchy rather than on coordinates. Install sideloadFull already reported named stages and a percentage; nothing consumed them. It now runs on a worker thread and reports through a queued signal, since widgets may only be touched from the UI thread. Sign-in stays on the UI thread because two-factor needs a dialog. The stage label, percentage and a fading log are driven from that callback. Only stage changes create a log line: sideloadFull re-emits the same stage constantly while a step progresses. Saved session Apple returns an adsid and a token at sign-in, which are enough to keep using the developer portal, so the password is never stored. The token goes to the Windows Credential Manager, encrypted by DPAPI and tied to the Windows account. A stored token is not validated at startup, as it is only exercised on the first request; Account > Sign out clears it. Strings in the .ui must stay ASCII: dqt hands literals to QString without decoding UTF-8, and anything else comes out as mojibake.
Kept for the record rather than discarded. This was the first attempt at a dark interface, before the Qt frontend turned out to be the better target: dlangui is unmaintained, has no SVG support and cannot draw a rounded corner without hand-made nine-patch assets. Two dlangui traits worth remembering, both documented in the theme: multiline text is switched on by the style's maxLines, not by the widget class, and the file picker is its own drawn explorer, so the frontend calls the native Windows dialog instead.
Superseded by the Qt frontend, which is the one that got the interface, the install flow and the saved session. Keeping dlangui meant carrying a dead second UI: an unmaintained 2014 toolkit with no SVG support, no rounded corners without hand-made nine-patch assets, and GDI text rendering. Its CI workflow goes with it, since it built a subpackage that no longer exists, and dev.ps1 loses the -Frontend gui option. The dlangui entries left in the dub.selections.json files are stale but harmless; dub drops them on the next upgrade. The work itself stays in history, one commit back, if any of it is ever worth revisiting.
The Account menu gave no clue whether a session existed: "Sign out" sat next to a permanently greyed "Log-in" whatever the state was. It now opens with a status line, "Signed in as <id>" or "Not signed in", and the entries follow that state. "Log-in" was a stub disabled in the .ui with a "Not implemented yet" tooltip. Signing in works now, so it is wired and lets the account be set up without starting an install, which was the only path to it before. Manage App IDs and Manage certificates were never connected to anything: zero references in mainwindow.d. The portal API was already complete, so only the screen was missing. Both are the same dialog, differing in wording and in which two calls they make. Two details that matter in use: App IDs show the quota, because Apple caps free accounts at ten and only frees a slot when one expires, which is the usual reason installs suddenly stop working; and revoking a certificate is called out as breaking every app already signed with it. Requests run on the UI thread like sign-in. They are short and modal, and threading them would only produce a frozen dialog. The dialogs also needed styling the main window never exercised: QListWidget is a scroll area and kept Fusion's light viewport, and each dialog is its own native window, so the dark title bar has to be asked for on its own handle. Window flags are set before winId(), since changing them recreates the native window and discards the attribute.
A free Apple account signs with a certificate Apple expires after seven days, and sideloaded apps then stop launching. Sideloader installed and forgot; this makes it stay around and sign again. Tracking installs.json next to the configuration keeps the bundle id, the package path, the device and the install date. Renewing is just signing the same package again, so that is all it takes. Nothing secret goes in it: the credentials stay in the Credential Manager. It is keyed on bundle id plus device, so reinstalling replaces an entry rather than accumulating duplicates, and a corrupt file is discarded rather than fatal, since installing rebuilds it. Notification area dqt exposes no QSystemTrayIcon, so the icon is registered with Shell_NotifyIconW against the main window, which already overrides nativeEvent for the frame. Only the icon is native: the menu is a QMenu and follows the application stylesheet, which an HMENU could not. Closing the window now hides it, since renewal only happens while the process lives; Exit in the tray menu is the way out. Renewal runSideload is shared by a manual install and by renewal, which is the same operation repeated over the tracked packages. Sequential: the portal and the device both dislike concurrent sessions. Each package takes its slice of the progress bar so a multi-app renewal still reads as one run. An hourly timer renews once a signature is within two days of lapsing. Apple only re-issues near the end, so renewing earlier would waste one of the ten App ID slots for nothing. Autostart uses the per-user Run key, not a service: a service has no desktop session, and renewal needs the device plugged in and someone available to answer a prompt. The entry passes --tray so sign-in does not flash a window. Closes the practical gap behind upstream issues Dadoum#77 and Dadoum#106, already addressed earlier in this branch.
The tray menu indented only its checkable entry: Qt reserves an indicator column for those, and the stylesheet did not account for it. The column is now reserved on every item, checkable or not, which is what native menus do, and the indicator is styled rather than left to Fusion. The device subtitle claimed "(USB)" unconditionally. iDeviceInfo carries the transport that idevice_get_device_list_extended reports, so it now says Wi-Fi or USB for real, and "unknown" if the device left the list in between. It matters beyond cosmetics: over Wi-Fi an automatic renewal can run without anything being plugged in. Icons sat flush against their labels on the Browse and Install buttons. Qt exposes no icon-to-text spacing in QSS for either button class, and widening the icons' viewBox would have affected the tabs and the title bar where the spacing is already right, so the gap lives in the label. The leading spaces are commented in the .ui so they do not read as a typo.
The saved session never worked: signing in produced no Credential Manager entry, silently. The cause was size. The payload carried Apple's URL bag, which alone is some 13 KB, against the 2560 bytes CRED_MAX_CREDENTIAL_BLOB_SIZE allows. An isolated round-trip test had passed on a 664-byte payload, so it could not have caught this. The bag is fetched from a public, unauthenticated lookup, so there was no reason to store it. fetchUrlBag is now a function of its own, used both by login and by restore, and the stored payload is down to 305 bytes. Failures are no longer mute: an oversized secret reports both numbers, and any other refusal logs the Win32 error. A generic "refused" cost a diagnosis round-trip. That diagnosis was only possible once the logs were readable. On Windows everything went to OutputDebugString, which needs a debugger attached, and a GUI build has no console for stdout either — so every log file captured until now was empty, and any conclusion drawn from their emptiness was worthless. Logs now also go to a rotating file under the configuration directory. The icon is wired application-wide, so windows and dialogs inherit it and the tray reads it back through WM_GETICON instead of falling back to the shell default. The generated artwork carried an opaque square behind its rounded corners; it is flood-filled to transparency from the four corners only, leaving the dark areas of the drawing untouched, and shipped as a 16-to-256 .ico since Windows picks the size it needs. Verified: signing in writes the entry, and relaunching after Exit restores the account without asking again.
ludovicthenot
changed the title
About the red CIShort version: the failures look unrelated to this branch, and two of them The same commit tree gives different results run to run. Three pushes
Two CLI targets already fail on
Every failure is in The Qt workflow never ran at all — it has stayed queued on both What I can vouch for is local: LDC 1.42 with Qt 5.15.2 on Windows 11, Happy to look at the workflows if you want CI fixed — bumping the pinned |
|
I looked at the PR briefly, I think it should be good, but at the same time I am kind of writing a replacement for it in apple-crates/super-sideloader (Rust + gpui) so I am not sure if I would do anything more than very basic maintenance work here. Maybe the right solution would be for me to update the README to mention that there is a more up-to-date fork pointing to yours, and also mentioning that I made an alternative too? Also, I think you should update the README accordingly, with screenshots, and I would recommend you to set-up a corrected workflow for building all of the Q frontend binaries (and maybe a manual one to build the releases, for the main operating systems supported by D, aka x86_64 + arm64, Windows Linux and macOS), to have something that can be trusted by anyone. But that's only recommendations |
The requests library probes OpenSSL by filename and asks for libssl-3-x64.dll before libssl-3.dll. Sideloader ships the latter, so the first name escaped to %PATH% and matched whatever OpenSSL the machine happened to have installed -- Git's, a laptop vendor's media tool, Bonjour. The mismatched libssl and libcrypto pair then failed on the first HTTPS request, if it loaded at all. Restricting the DLL search path to the application directory makes the names we do not ship fail cleanly, so the probe falls through to the copy next to the executable. It runs as a CRT constructor because requests loads OpenSSL from a module constructor of its own, and D leaves the order unspecified between modules that do not import each other. ideviceactivation.dll imports libcurl.dll, which was resolving from %PATH% for the same reason and now has to travel with the package. Separately, the device card stayed on screen with an empty combo box and no subtitle when no device was connected: refreshView only runs on a selection change, which never fires for a picker that was empty all along. The empty state below it already explains what to do, so the card is now hidden until there is something to pick.
The root dub.selections.json still pinned the versions from before the toolchain move: botan 1.13.6, whose `delete` expressions the language removed in 2.109, along with eight other packages left behind. Only the frontend selections were updated at the time, because that is where the local build script runs from. Everything that builds from the root -- the CI jobs, and the plain `dub build` the README tells people to run -- kept resolving the old set and failed to compile.
|
Thanks for taking the time to look at it, and for being upfront about where the project is going. Pointing the README at this fork sounds right to me, and mentioning super-sideloader alongside it makes more sense than either of us pretending the D version is where the future is. If you do add a link, the branch to point at is https://github.com/ludovicthenot/Sideloader I went and did the two things you recommended: README — rewritten with screenshots of the Qt frontend on Windows, and a note at the top saying plainly that this is a fork, linking back here and to super-sideloader. Your first-person sections and your sponsor link are untouched, and there is a sentence saying so. The hard part of this project is yours and the tips should keep reaching you. The build workflow — the Windows job now produces a package that actually runs. It could not before: no native libraries at all, no https://github.com/ludovicthenot/Sideloader/releases/tag/v0.1.0 macOS I left alone. Both jobs fail at Two things in this PR are worth having regardless of what you do with the rest, since they affect your users and not just mine: The OpenSSL one is the interesting bug. The other is that the root Both are small and independent, so cherry-picking them and closing the rest would be a perfectly reasonable outcome if you would rather not carry the frontend work. Thanks again for the project. Two years of reverse engineering to make this possible on Linux is not a small thing, and it is the only reason any of the above was worth doing. |
2 participants
Modernise the toolchain, and rebuild the Qt frontend
This started as "get it to build on my machine" and grew. It is offered as
a whole because that is how it was written, but it splits cleanly and I am
happy to reduce it to any part you want — see Taking only some of this at
the end.
Everything was developed and tested on Windows 11 with LDC 1.42 and Qt
5.15.2. Sideloading was verified end to end on a physical iPhone (iOS 27),
signing included.
1. The repository no longer built on a current D toolchain
Fixes #77 (does not compile using latest ldc).
botan1.13.6 used thedeleteexpression, removed from the language inD 2.109, so nothing past LDC 1.34 could compile. Raising it revealed the
next failure, and so on:
botandeleteexpressionmemutilsdefaultInit, constopEqualsrequestsprovision645d56d87717ce1fheaders()[...]for ldc-1.41dqt6a44b55f82213e7aqt5branch, since dqt master is now Qt6-onlyautomemcachetoolsdxmlintel-intrinsicsunit-threaded2. Three real bugs, independent of the toolchain
cast(Blob[])reinterpreted class references as interface references.Blobis an interface, so an array cast is not a conversion: it skips theclass-to-interface pointer adjustment. It happened to work only because
these classes implement
Blobfirst, putting the offset at zero. Convertedelement by element in
sign.dandct_bypass.d.A stack-allocated
QStringwas passed tocpp_new!QPushButton. Thetemporary died before the button had taken it. This crashed the Qt frontend
as soon as a device connected and the tools tab was populated — every other
call site in that file already used
*cpp_new!QString(...).clearLayoutleft orphaned widgets visible.takeAtunmanages an itembut does not reparent its widget, so the widget stayed a child of the tab
and kept painting at its last geometry. Every tool button was drawn twice.
3. The Qt frontend
It existed but was unfinished: the Install button logged a line and
disabled its tab, and there was no way to sign in at all. It is now the
working frontend.
the menu instead of a permanently greyed "Log-in".
sideloadFullon a worker thread and reports through aqueued signal. The named stages and percentage it already emitted were
never consumed by anything; they now drive a progress bar, a stage label
and a fading log.
portal API was complete, so only the screens were missing. App IDs show
the quota, since Apple caps free accounts at ten and only frees a slot on
expiry — the usual reason installs suddenly stop working.
honours QSS throughout. The window keeps its native frame and only hides
the system caption through
WM_NCCALCSIZE, so Windows keeps ownership ofresizing, snapping, the shadow and rounded corners.
Two dqt-specific traps are worth recording, both commented in the code:
strings in
.uifiles must stay ASCII, because dqt hands literals toQStringwithout decoding UTF-8; and dqt's uic does not applygeometryor
minimumSizeto the root widget, so those must be set in code.4. Saved session
Addresses #106 (keychain/libsecret support for storing creds?) on
Windows only.
Apple returns an
adsidand a token at sign-in, and those are enough tokeep using the developer portal, so the password is never stored. The
token goes to the Windows Credential Manager, encrypted by DPAPI and tied
to the Windows account, with a "Sign out" entry to clear it.
SavedDeveloperSession,save()andrestore()live indevelopersession.dand are platform-neutral; only the storage backend isWindows-specific, so a libsecret or Keychain backend would slot in beside
it. The issue suggests a cross-platform C++ wrapper, which would fit here.
One finding worth passing on: the URL bag must not be persisted. It
runs to some 13 KB, against the 2560 bytes
CRED_MAX_CREDENTIAL_BLOB_SIZEallows, and the write fails silentlyotherwise.
AppleAccount.fetchUrlBagnow exists so a restored session canfetch it again — it is a public, unauthenticated lookup. The stored payload
is 305 bytes.
5. Renewal from the notification area
A free certificate expires after seven days and sideloaded apps stop
launching. Sideloader installed and forgot. It now tracks what it installed
(bundle id, package path, device, date, in
installs.json) and re-signsbefore expiry, from a tray icon.
The icon is registered with
Shell_NotifyIconWbecause dqt exposes noQSystemTrayIcon; the menu is aQMenu, so it follows the stylesheet.Renewal fires only within two days of expiry, since Apple re-issues only
near the end and renewing earlier burns an App ID slot for nothing.
6. Logging
On Windows everything went to
OutputDebugString, which needs a debuggerattached, and a GUI build has no console for stdout either. In practice a
released build could not be diagnosed at all — this cost me a real
debugging session before I noticed. Logs now also go to a rotating file
under the configuration directory.
Removed
The dlangui frontend, along with its CI workflow. It is unmaintained, has
no SVG support and cannot draw a rounded corner without hand-made
nine-patch assets. Keeping two UIs alive was not worth it. Say the word and
I will restore it — the removal is a single commit.
Taking only some of this
The commits are ordered so the reusable work comes first. If you want just
the part that unblocks everyone:
That one commit closes #77 and the three bugs above, and touches nothing
cosmetic. The redesign, the tray and the renewal are opinionated and I do
not expect them to be uncontroversial; drop them without hesitation.
Not addressed
verification, so I had no way to exercise the
secondaryAuthpath.libimobiledevice-win32builds (libplist 2.0). They work, but theplist_mem_freepath is only reachable underversion(NewPlist).