Roadmap to rebuilding this mess?

[knghtbrd]

TODO: Explain this crap… TL;DR: ✦'s for how hard it is

• ✦✦✦✧✧ adtpro
• ✦✦✦✧✧ netatalk2 (✦✦✧✧✧ for initial package)
• ✦✧✧✧✧ appletalk-dkms
• ✦✦✦✦✧ macipgw (✦✦✧✧✧ if I have a test subject)
• ✦✦✧✧✧ applecommander (maybe ✦✧✧✧✧ after talking to @a2geek)
• ✦✦✧✧✧ nulib2 (if a bug was fixed, ✦✧✧✧✧)
• ✦✧✧✧✧ a2server-common
• ✦✦✦✧✧ a2server-tools (potentially less. Depends on how much of a PITA some tools are and how important)
• ✦✧✧✧✧ a2server-samba
• ✦✦✧✧✧ a2server-appletalk (with a netatalk veteran, ✦✧✧✧✧)
• ✦✦✧✧✧(?) a2server-udev-rules (unless they work as is!)
• ✦✦✦✧✧ a2server-console-service (kmscon in sid may make this REAL easy!)
• ✦✦✧✧✧ a2server-shell-tools
• ✦✦✦✦✦ a2server-apple2gs-gsos-installer
• ✦✧✧✧✧ a2server-apple2-prodos8-installer
• ✦✧✧✧✧ a2server-apple2-*-installer (and friends…)
• ✦✦✦✦✧ a2server-apple2gs-spectrum-installer (…friends save one glaringly complicated exception!)
• ✦✧✧✧✧ a2server metapackage to install everything

[knghtbrd]

Spectrum's a pain in the ass because not only does it get updates, but the URLs to the pieces and what they are keeps changing. It's not hard, you just have to accept that every so often you're going to get a bug report that the installer fails because something changed and the old version is no longer available, or the filename has changed, or the website's been reorganized, or the phase of the moon changed, or … 🤪 Might resolve this by downloading a copy of whatever the current version is and put them somewhere I control and point the installer at those. I can refresh what I have when I notice new versions of stuff and update the installer when I download new files.

Or maybe I just ship spectrum pre-installed in the package and people just unpack it? Hmm. I should be able to do that with the others too, as long as I don't plan on putting them into Debian or anything. That might solve many problems. I'll think about it.

[NJRoadfan]

I have been working on some of this one and off for the past 2.5 years.

My main tree based on Ivan's 1.5.x dev tree is here: https://github.com/NJRoadfan/a2server/tree/njroadfan-main-dev

The main branch is based on netatalk 2.2.6 + manually applied patches (all patches are now part of the main netatalk 2.x branch). Here is a brief change log:
Tools
-Fix AppleDouble header bug in cppo.
-Make sure everything uses Python 3 as Python 2 is deprecated in new distros.

a2server-2-tools:
-Always install nulib2 and unarchiver from package repos.

a2server-3-sharing.txt:
-Added compatibility for Debian Buster and Bullseye, also tested with latest Ubuntu releases.
-Install CUPS and enable papd for network printing. Mac users in particular are asking for this.
-Properly re-build AppleTalk kernel module with Linux 5.x kernels.

a2server-5-netboot
-Add option to install GS/OS System 6.0.4.
-GS/OS install now includes LaserWriter driver to print.
-Fix some dead links for downloaded utilities.
-Install Marinetti 3.0d11 and latest link layer revisions.
-Added installation of Webber and Networker2.
-Updated Spectrum to install the latest Platinum version.

In progress is a move to the @rdmark's netatalk-2.x branch from 2.2.6: https://github.com/NJRoadfan/a2server/tree/new-netatalk-test

With this move to the "new" netatalk, I am slowly switching things over to systemd and tossing compatibility with ancient Debian releases in the trash. Upgrade installs from old A2SERVERs will likely no longer be possible. The biggest improvement is macipgw now has a proper systemd service and the NAT is setup using nftables vs. the now obsolete iptables. I still have to fix things up like the samba setup script.

The biggest thing I'm having trouble with is that I don't have a RPi to test any of this with. I have been doing everything in VirtualBox at the moment. There is still ALOT of cleanup that needs to be done. Also I have not tested pulling files from an alternate URL or local hard drive at all.

As far as testing setups goes, I can confirm everything works on Bullseye and Ubuntu Server 22.04LTS and 22.10. By that, EVERYTHING works. You get file sharing (AFP on Apple II, classic Macs, and modern MacOS/ASIP), Apple II netbooting, printing (both GS/OS and classic MacOS), and MacIP internet access (both Marinetti and classic MacOS via MacTCP and OT). I have tested this with emulated machines and real hardware via both Ethernet and Localtalk using a Shiva Fastpath as a bridge.

[NJRoadfan]

To update this:

A2SERVER is being moved to using Netatalk 4.0 (after basically hacking back in AppleTalk support). In addition, its switching Netatalk 3.0+ extended attributes to store the metadata and ._ files to store resource forks. This gains full interop with Samba and its vfs_fruit module, which is now correctly setup and working. A modern Mac can now safely connect to an A2SERVER share via SMB or AFP. Most all of Ivan's tools have been updated to support the "new" way of doing things with extended attributes as well.

Right now my biggest issue is the stupid ciopfs overlay file system. It uses an extended attribute to store the mixed case filename, but that EA can't be copied to a new file on the same share, so it will throw an error and upset Netatalk. Looking to migrate to cicpoffs after mostly rewriting it to work properly with Netatalk. Added bonus is that it targets libfuse3.

Other big improvements are:
-TashRouter for LToUDP support.
-Option to setup a TashTalk LocalTalk interface hat if running on a RPi.
-Stephen Heumann's AFPBridge, NetDisk, and SMB FST are installed on the NetBoot GS/OS install.

Since these improvements are "nice to have", I have released an intern 1.9.9 release with Netatalk 2.x with the above enhancements in addition to what was added in the previous post.

[knghtbrd]

Easy enough: get rid of ciopfs! If you're building a filesystem instead of using NOOBs to install, just format it with -O casefold. You can tune2fs a filesystem already formatted, but I think it may need to be not mounted at the time of modifying, and if you do this there will be something lost in terms of compatibility with 10 year old versions of Linux. I think that's a reasonable price to pay.

With casefold enabled in the filesystem, you can chattr +F an EMPTY directory to enable casefold. The contents of the directory shall now be case-retentive, but not case-sensitive. No FUSE, no fuss.

If you are using pi-gen to create your own installation that works with NOOBS, you need to modify the script that creates the root filesystem to add -O casefold.

Sorry, I've known this existed for literally years (since before it was available on Raspbian), but did not know it was a sticking point for you or I'd have spoken up sooner.

I've thought for some time now that the right way to approach netatalk was just to use the current version and fork out the appletalk network protocol stuff into its own project. You need one new appletalk program: A bridge between DDP and TCP/IP for AFP. That keeps the ugly and obsolete DDP crud out of netatalk's way, and means that there is a smaller security attack surface to worry about for people trying to maintain the DDP suite since they don't have to keep TCP-based afpd secure. (And because you can firewall DDP traffic behind a LAN where the realities of modern Internet security don't exist.)

The only thing missing from that solution one could still want would be a new authentication mechanism for netatalk which was modern in terms of not transmitting passwords in the clear, but could be implemented on Apple IIs and m68k Macs. But again, this was beyond my knowledge and I couldn't just google how to do it. (I tried.)

[rdmark]

From one perspective, the DSI (TCP/IP) layer is the one that's ugly and tacked on. ;)

In general, I agree that a modular design would be better. Ideally, with both transport layers broken out of afpd. And at the same time, remove the forking daemon design that contemporary macOS really really hates, and make afpd a multi threaded application. This is my pie in the sky rearchitecture of netatalk.

FWIW, @NJRoadfan did a good job isolating the DDP code in netatalk 4.0 behind the NO_DDP macro, so that when you build with -Dwith-appletalk=false there's in theory no trace of DDP in the resulting afpd binary. The code is pretty ugly and spaghetti-like, but it works. :)

And another FWIW: We have a pretty good setup with authentication methods now. The RandNum UAM provides a modicum of encryption for pre-OS8 Macs and Apple IIGS. It's only for some very specific scenarios (the Apple //e ProDOS client I think) that you have to use the ClearTxt UAM. For 99% of setups you'd have DHX2 for OSX, DHX for MacOS8-9, and RandNum for anything earlier.

[knghtbrd]

I was under the impression that of the non-clear options, one of them could not be distributed (legally) in binary form—not that I care about that anymore—and another was not very good at all.

I'll just say I haven't even looked at this stuff since before the pandemic. I was pretty disillusioned before we lost a couple of old timers in the past several years. Folks were just doing their own thing and it just stopped being fun for me. But I still know some stuff, especially about Debian and the Raspberry Pi 4 and older, so I'm happy to help share what I've figured out along the way. I'm just glad to see someone working on this stuff again.

[NJRoadfan]

The issue with the RandNum and DHX UAMs was that they relied on OpenSSL for their cipher functions. The Debian project in particular had a problem with this and would build Netatalk with those UAMs disabled. For some reason they had an issue with the mere fact it required linking to a "non-free" library.

This has all since been fixed now that RandNum and DHX have both been ported to 100% GPL compatible libgcrypt (DHX2 always used it) and OpenSSL has been completely jettisoned from the codebase. This is something that should have been done over a decade ago, but nobody seems to have bothered with it.

Regarding casefolding, I'm well aware that ext4 supports it. The problem is that it isn't user friendly (or script friendly) to enable it on demand. The tune2fs program requires that a drive be unmounted, which would require some pretty ugly "custom initramfs to activate and reboot" steps I'd like to avoid. For VM installs, the Debian installer offers no way to enable the option during the partitioning phase of setup (boo). For RPis, I'd prefer if a potential user can stick with a "stock" image. The official RPi SD card writer is pretty straight forward to use. Finally, less hacks makes supporting new distros easier. I already have Trixie support working.

[rdmark]

I think the OpenSSL debacle started with the FSF taking a strong stance against the so-called "SSLeay" license (named after the author, Eric Young) which influenced Debian decision making (my read anyways.) Eventually the linking policy must have been relaxed, because OpenSSL 1.1.1 was definitely included in Debian Buster. Things were good for a while.

Then shortly after the release of Debian Bookworm, we started getting netatalk bug reports that authentication was broken for OS9. Turns out, Bookworm shipped OpenSSL 3.0, a version of the library that had deprecated the specific combination of crypto we used for DHX (DHCAST with less than 512 byte modulo size) in a very insidious way. Namely, you could link with the library, but it would crash when it saw the "insecure" configuration. I filed a bug with the OpenSSL project, but was promptly shot down. Noone must be allowed to use "insecure" crypto.

At one point, I tried and failed to replace OpenSSL with LibreSSL wholesale. The two couldn't really coexist on the same machine, and OpenSSL was too deeply ingrained in the Linux ecosystem.

Then a rockstar with the internet moniker dgdga (ret.) showed up and evacuated us off of the creaky old Autotools to Meson, then onto wolfSSL as SSL provider with Nettle as crypto provider. This was the netatalk 3.2 release series. I even had a conference call with wolfSSL engineers at one point when dealing with some symbol conflicts with OpenSSL and Nettle. (They are very supportive of open source.)

And the latest chapter of the saga was @NJRoadfan who had the now-famous eureka moment of "wait, I think we onboard both RandNum and DHX to Libgcrypt and use one library for everything". This turned into the netatalk 4.0 release series.

This is where we're at in the netatalk crypto journey now. Unless Libgcrypt gets hijacked by Jia Tan, we should be set for a good long while. Or Apple releases AFP 4.0 with mandatory quantum safe crypto.

[knghtbrd]

That's right, it was the SSLeay license issue … which to be fair OpenSSL worked very hard to get that code out of OpenSSL if for no other reason than all of the actual algorithms it contained were dangerously insecure now anyway. That left some utility functions with more than one implementation, so that must be where things went. Again, years since I've looked at any of this stuff. 🙂

Also are you guys going to be reviving the other side of things as well with the shell account?