mkj’s musings

Advice from my first year of learning Morse code

2025-12-26T00:00:00+00:00

Learning Morse code has been a good struggle for me this past year. I’ve averaged somewhere between half an hour and an hour per day of drill for the whole year, and taken only a few days off here and there.

I tried and failed to learn Morse as a child, so wanting to learn it at all surprised me. I think that I have a lot more trouble learning Morse than most people, and it’s still fun. If today’s resources for learning Morse code had been available to me as a child, it would have been a lot easier.

I had a lot of questions when I started, and had to go hunting for information. This is my history of learning Morse code and the things that I’ve learned, many of which I wish that I’d learned earlier.

Preparing to Learn

Whatever tool you use, don’t learn visually. Learn to listen. Learn Morse like a spoken language. Use tools and/or classes that use “Farnsworth” timing, where the individual characters are sent fast, with lots of extra space between the characters in which to practice conscious recollection as you slowly build unconscious recognition.

Most successful learners use variations on the Koch method (invented by Ludwig Koch), starting with two characters at full speed, adding one new character as characters as you build conscious recollection. (Koch originally used no extra spacing; most successful learning programs use the Koch method with some Farnsworth spacing.)

Ignore everyone who confidently tells you that there is exactly one character speed to use so that you don’t count elements (dits and dahs) as you learn (I’ve seen figures from 13WPM to 20WPM to 30WPM touted as universal speeds at which this is true). Find out what works for you. Set it to whatever speed it takes that you personally do not in practice count dits. (If you aren’t having trouble initially distinguishing H and 5, consider setting the character speed faster.)

I did a lot of music as a kid, “counting out” very fast rhythms, and I therefore had to set the character speed to 35–45WPM to finally shut down my subliminal counting machine, while setting the Farnsworth spacing in the 5–8WPM range to give room for slow recall. After months of drill, I was finally able to reduce the character speed, and I’m finally down to being able to drill at 20WPM character speed with no Farnsworth spacing.

Find Learning Resources

Where to start depends on whether you prefer (or have schedule for) guided learning or self-directed learning, and whether you benefit from social pressure to stay on task. I am primarily an auto-didact and prefer self-guided learning.

Classes

I did not use classes because of my schedule and learning preferences, but here are two I know others have used.

Writing

I found Zen and the Art of Radiotelegraphy by Carlo Consoli, IK0YGJ, well after I started learning Morse. Also, I didn’t have space in my life to follow its recommendations for how to learn even if I’d seen it earlier, but still found it useful perspective. Worth a read.

Online Tools

I started with DJ5CW’s LCWO. I drilled characters in the “Lessons” for several months before doing anything else. I now regret that I waited months to start using its “Morse Machine” for drill. It would have been better to go quickly through the lessons until I could at least sometimes, with thought, recognize each character, and then use the Morse Machine to cement that learning.

After a while, I wanted to go portable and practice on my phone, where I found phone apps a bit better integrated than web apps. I used Morse Code for Android as a portable Morse Machine for a few months.

I have continued to use IZ2UUF Morse Koch CW to drill head copy for call signs, random words, phrases, a list of names I added to it, and common QSO phrases. I set it running when I get in my car, and follow along as traffic and conditions permit. (The downside is that ignoring it when traffic requires high attention is that I inadvertently practice dropping my attention from the Morse. For me, on balance, it’s been the key to me actually averaging well more than half an hour a day of real drill. My work commute is nearly 30 minutes each way, and I can usually get in 30 minutes on any shopping trip as well.)

I use Morse Walker to practice for POTA, SST, and Field Day, and I keep hoping that the original author accepts my PR adding Field Day suport. When I started, I set a max calling station limit of just one station to avoid pileups, but now I’ll set it as high as three. (I don’t see that there’s much value in going to more than three in Morse Walker.) My branch also adds display of how many stations were in the pileup next to how many attempts it took, just to make me feel less bad about not getting things the first time.

I sometimes play Morsle for fun, where I’ve set the practice mode to start at 45WPM (though I rarely succeed hearing a word at 45WPM first try!)

I’ve recently learned about Morose for instant character recognition training, and keep meaning to use it more when I’m at the computer, but tend to gravitate to Morse Walker.

I occasionally watch K4SWL’s POTA activation videos and try to copy his callers.

Listen On the Air

You can listen to the ARRL’s W1AW code practice sessions on the air or on the web.

Progress

Drill will almost certainly feel frustrating at times. If you find yourself frustrated, then for at least part of your consistent drill, use a tool that gives you numeric measures. I felt like I was getting nowhere for many weeks, except that my Morse Machine character drill speeds crept up week to week (though they wandered up and down day to day). That helped me keep going.

I felt really dense doing the Koch lessons of blocks of five random characters. I would remember a character for one block, and forget the same character 15 seconds later. Also, I would swap characters that invert the dits and dahs, e.g. L (• ━ • •) for Y (━ • ━ ━), K (━ • ━) for R (• ━ •), P (• ━ ━ •) for X (━ • • ━), A (• ━) for N (━ •), and so forth. I would freeze up for ten to fifteen seconds at a time, and for a bit, code was just noise flowing in one ear and out the other. It took a lot of practice to get better.

I am glad I waited to practice sending until I had drilled for months. But once I did, starting with a paddle (see below), where my thumb sends dit and my fingers send dah, really helped reduce inversion errors. I think it helped me associate sound and physical movement. I don’t think starting that sooner would have been good. First learn to listen, then use sending with a paddle to help you over a hump later. Many folks I’ve talked to have described similar troubles with confusing inverted pairs like that. If you experience this too, don’t worry about it, you’ll get there.

After several months, I started to slowly reduce the Farnsworth spacing while drilling until I had removed all of it, but that took many more months.

Copying arbitrary text is a different skill from recognizing individual letters, and you should expect to practice it separately after becoming good at recognizing letters. Trying to pay attention to a spoken conversation in which t h e - p a r t i c i p a n t s - s p e l l e d - o u t - e v e r y - w o r d - w o u l d - a l s o - b e - h a r d - s o - g i v e - y o u r s e l f - a - b r e a k - w h e n - i t - i s - h a r d - t o - f o l l o w - m o r s e - c o d e. Holding letters in your mind while assembling a word is itself a new skill. This is true whether it is a call sign or a natural language word.

Sending

The Morserino M32 Pocket is the latest version of the open source Morserino available for purchase.

There are at least web three sites for practicing CW online:

VBand market their own adapter for keys, and explicitly say that all other adapters are unsupported. However, the Morserino is able to work as a VBand adapter.
Vail has at least one open source adapter design for connecting a morse code key to USB ports, and because Vail supports VBand adapters as one of several options, Morserino will also work with Vail.
Morse Me Please does not allow connecting adapters. This makes it more useful for use from a mobile, but of more limited value for real practice.

While practicing sending, do not use Bluetooth. Bluetooth always adds some delay as it encodes and decodes, by design. You need practically instantaneous feedback from touch to sound. Use wired headsets or earbuds, or wired speakers, to hear while you pratice sending.

Get On The Air

Start with programs where you can respond to a caller. Start by responding to someone else who is calling CQ.

You don’t have to start by learning to pull call signs out of a “pile-up” where multiple people are sending at the same time.
You can work out the details of the exchange ahead of time as the more experienced operator works other calling stations, so that when they take your call you are just confirming your expectations, and have fewer surprises.

Expect to get confused. It’s OK. No one will hold it against you. If you just give up and stop sending, the other operator will give up and move on at some point. It turns out that “QSB” (fading signals) is a normal occurence much of the time, and dealing with a disappearing station is part of our normal experience on the air. If you disappear a few times because you are out of your depth at first, it’s OK. No sweat. No one is keeping a list of stations that have “ghosted” them to never talk to again, and if they are, you don’t want to talk to them anyway!

POTA

Parks on the Air (POTA) is a great way to get started. The normal vocabulary is small. Plenty of operators will slow down to match your speed. Call at the speed you would like to hear. Most commonly, they will add Farnsworth spacing rather than slowing down their character rate.

You can use the POTA site (or the Ham2k Polo logging app that integrates with it) to see who is “spotted,” with their frequency, call sign, and state. That makes at least part of the listening process to be confirming what you see, making it easier to get started.

You’ll want to at least recognize your own call sign, numbers, and state/province abbreviations. Also BK (back to you/me), ? (used alone to ask for a repeat, or after a call sign to ask whether it was copied correctly), R (roger, used to confirm after a question like whether they copied your call sign right), STATE?, CALL? (call sign?), AGN? (again), RST, TU (thank you), GM (good morning), GA (good afternoon), GE (good evening), UR (your), and know that “9” is often sent as “N” (the most common “cut number”). The “BK” is sent both at the end of a short “over” meaning “back to you” and at the beginning of the next over meaning “back to me”. It is not technically a “prosign” and thus should be sent as separate characters, not run together, but you can expect lots of operators to send it run together anyway. This is amateur radio (emphasis on amateur), and the “ham” moniker originally referred to being bad at sending Morse code. It’s OK.

You can listen to a few other callers until you are sure you have copied the state correctly before you try to call. Most of the time, you just need to be able to send your call sign, and then after they copy your call sign, something like “BK GA UR 5NN 5NN ST ST BK”. Don’t worry at first about the signal report. As you get more comfortable, you can start sending better signal reports. The rules encourage real signal reports, but there are plenty of operators who just send 599 (typically as “5NN”).

If they respond with the full “over” with what sounds like your call sign but you are not sure, or if they definitely responded to you, but got your call slightly wrong, you can respond with a hint, “BK GA UR 5NN 5NN ST ST DE YOURCALL BK” — you’ll usually hear them respond with something like “BK R R YOURCALL TU ST ES 73 DE THEIRCALL” to confirm they got it.

Here’s the best video I’ve found for how to operate CW while doing POTA

SST

Twice each week for an hour, the K1USN SST (Slow Speed conTest) is an opportunity for a short “exchange” at a relaxed pace; you should expect the other operator to respond at roughly your speed, and never go above 20WPM. Use pskreporter to find where K1USN is currently operating and listen in.

The conversation will follow a script that is described on the SST site.

Since states or provinces are provided by standard two-letter abbreviation, that’s easy to learn, and is a skill shared with POTA.

The main new challenge when you add SST is to learn to copy the other operator’s name. When you are first getting started, listen to multiple other QSOs until you copy their name and state. Practice sending their name without transmitting. Once you are ready, try calling to them in response to “CQ SST THEIRCALL”. Just send your call sign once, and wait for them to respond “YOURCALL THEIRNAME THEIRSTATE”. You respond “GE THEIRNAME YOURNAME YOURSTATE” and expect their response to be something like “GL YOURNAME TU THEIRCALL SST”. (“GL” is short for “Good Luck!”)

You can respond either after “CQ SST THEIRCALL” or directly after another operator’s QSP as soon as you hear “GL OTHERNAME TU THEIRCALL SST” — the “SST” at the end is an invitation for the next station to call.

Keys and Keyers

A “key” is the physical device that you use to form characters, and a “keyer” is an electronic device that forms the sounds of the characters using input from a key.

Keys

There are four or five basic kind of keys:

Straight key (the original). When using a straight key, the radio sends a continuous tone while you are pressing it, and not otherwise, so that you are completely responsible for the length of the dit, the length of the dah, and the lengths of all the spaces.
Sideswiper (also known as a “cootie”), which acts like a straight key, but you move it from side to side instead of up and down. There is a contact on each side, but they are electrically the same. It can reduce RSI (“glass arm”), but like the straight key, you are responsible for the length of dit and dah because it’s just two contacts instead of one.
Bug (vibroplex) where it makes dits mechanically when you press with your thumb, but you still are responsible for the length of dah by pressing with your fingers. (You could use it just like a straight key by pressing only with your fingers.)
“Paddles” require an electronic keyer to form the dit and dah; most modern radios have this built in, and you can also buy or build separate keyers that connect to a radio. They keyer is set to generate dits and dahs at a specific speed.

Paddles come in two variants:

Single paddle, where your thumb gives you a stream of dits as you push with it, and your fingers give a stream of dahs as you push the other way with them, but the paddle can’t move in both directions at the same time.
Double paddle, which is like the single paddle, except that (in the most common modes) if you squeeze the two paddles together, you get a stream of alternating dits and dahs. This is called iambic, or “squeeze,” keying. An alternating “dit-dah-dit-dah” pattern feels like the iambic “foot” in scansion (poetic rhythm), thus the name.

Lots of hams love iambic, double-paddle keys. Most Morse code contest winners use single-paddle keys because they are more accurate than double-paddle keys, and the contests penalize mistakes.

Some paddles include a switch so that you can switch which side is dit and which side is dah, either for left-handed sending or because you just want to send the other way.

Regardless of key selection, some hams learn to send with their non-dominant hand so that they can write in their log with their dominant hand while sending with their non-dominant hand.

Keyers

Keyers were originally analog electronics, but are now mostly digital. There are three common modes (A, B, and Ultimatic), and now with digital keyers it is also possible to have a “bug emulator” that makes paddles act more or less like a bug. The difference between A and B is not relevant to single-paddle keys, only double-paddle. The choice between modes is a matter of personal preference. Ultimatic is the original dual-paddle keyer, which when squeezed, repeats the element associated with last paddle contact made, rather than alternating between elements.

Getting Started

When you do get a key, I advise against starting with a “straight key” or “sideswiper” where you are responsible for timing the dit and dah. There are plenty of really bad “fists” on the air with horrible timing that makes them hard to copy.

A keyer will help you learn how the characters should sound by forming good dits and dahs.

Learn what good timing sounds like, including when you are sending, then do straight key, sideswiper, or even bugs later when you know what good code sound like. Experienced operators hear those syncopated keys as an accent, but it’s harder for new operators.

Why “Morse?”

“Morse” might better be called “Vail” or “Gerke” code. The original telegraph code, which used three element lengths (dot, short dash, and long dash), was designed by Samuel Morse’s collaborator Alfred Vail. Samuel Morse’s idea was to just send numbers and look them up in a code book; Vail suggested and implemented sending unique codes for each letter. The modern ITU-R International Morse Code we use today is a refinement of the “Hamburg” variation created by Friedrich Gerke around 1848 for the German railway, which changed from three symbols to two. The International Telegraphy Congress standardized what is now International Telecommunication Union (ITU) Morse code in 1865. The most recent addition to the standard is the @ symbol (• ━ ━ • ━ •) which was added 24 May 2004, on the 160th anniversary of the first public Morse telegraph transmission.

Farnsworth timing

The additional character spacing named for W6TTB Russ Farnsworth is apparently misattributed, but the chances that it will be widely renamed “Edison timing” seem slim.

Reflections on a year of ham radio

2025-12-24T00:00:00+00:00

When I was about twelve, I was captivated by the idea of “ham” (amateur) radio and wanted to become a ham radio operator to talk to people around the world. A lot of the learning was easy enough for me, except for two things: I had great trouble memorizing all the frequencies, and the way I was taught Morse code was by memorizing visual flash cards.

Back then, you were required to operate with only Morse code for a year as a “novice” before you were allowed to upgrade your license and use any other modes. You couldn’t talk on the radio until you had used Morse code for a year, and you needed to get better at Morse code before you could upgrade. You had to pass a 5 words per minute (WPM) test perfectly to become a Novice, and I think at that time it was 13WPM for General, and 20WPM for Amateur Extra.

I never got past 4WPM, and eventually gave up the dream and moved on to other things. Then when I experienced the early internet in college, I got to live my dream of talking to people around the world, so when the Morse requirement was dropped for ham radio in the US, I didn’t pursue a ham radio license.

When hurricane Helene arrived in NC, I was cut off from communication with one of my (adult) children, who was living in the affected area. Then, shortly after we regained contact, I learned that information was getting in and out of western North Carolina in large part by ham radio. It was too late to be able to be helpful for this disaster, but now I had a reason to get my license to be able to help next time.

I started studying every waking moment that I wasn’t working. The test no longer expected you to memorize every frequency in the band plan. A lot of the information made sense to me because of my other interests, too. I passed all three tests in one sitting, becoming, as I put it, “the world’s least qualified Amateur Extra” for at least a short time.

Other than wanting to be prepared and able to help in the next disaster, I had really no idea what I wanted to do with ham radio. I thought it might be a fun hobby to dabble in from time to time. I was confident of only one thing. I was not going to touch Morse code, my childhood failure.

Ham radio has been full of surprises for me.

Morse Surprise

I bought VHF/UHF radios (line-of-sight frequencies, not round-the-world frequencies) and tried participating in local “nets” where you check in at a fixed time, and the net controllers gives you a time to say whatever you want to say. It works as practice for when you need to have organized communications in an emergency (as well as being a social outlet), but as a pilot who trained carefully to use only the minimum number of words on the radio, I found myself “mic-shy” and checking into the net “in and out” — meaning “just record that I was here, I don’t have anything else to say.”

The nets were held using a “repeater” — our radios can’t all reach either other directly, but everyone can reach an antenna high up on a tower, and can hear what that radio transmits, so we transmit to the receiving radio on the tower using one frequency, and listen to the transmitting radio on the tower on another frequency, and can hear each other indirectly through it. By the rules of amateur radio, these repeaters must identify the license under which they are operating from time to time. Many of them do this by quietly and rapidly transmitting their ID in Morse code.

Two weeks into having my license, I broke down and decided I wanted to be able to understand the repeater IDs. I signed up on LCWO to learn Morse code. I studied lightly, off and on, for a couple months, but then in December 2024 started to get more serious. By January, I knew that I wanted to become good at using Morse code on the radio. By March, I bought an additional radio primarily for doing portable operations using Morse code.

Studying Morse code started out as “type 2 fun” — I was often frustrated by how slow I was learning while practicing, but still glad I was learning. As I learned, it turned into “type 1 fun” — fun and relaxing in the moment. I’m not yet fluent, but I keep seeing signs of (slow) progress. This is the hardest I’ve worked to learn any one thing in at least a decade. Probably the single hardest learning I’ve done since becoming an instrument-rated pilot. I think I’m actually really bad at learning Morse code. Most people who I talk to who have done the kind of intensive drill I have over the past year are far more fluent than I am. But now I go practice in order to relax. I even practice Morse code entirely in my mind to relax to go to sleep!

It’s really been like learning a language. When it flows best, I am not translating it; the dits and dahs themselves carry meaning, and translating into letters and words is extra work. I’ve never become fluent in another spoken language, just learned some rudiments of a few of them. So perhaps it’s not a huge surprise that it’s taking me a while to become fluent in Morse. But now I want to be fluent in Morse.

I’ll keep learning in the new year.

Family Surprise

My wife and one (so far!) of my (adult) offspring have joined me and gotten their licenses. My wife has joined me as an Amateur Extra licensee. Her special interests are being the “net controller” running nets for community service events like bike rides and teaching ham radio classes. We also sometimes go to parks together to both spend time together outdoors and also participate in the Parks on the Air (POTA) program, using portable radios in state and national parks to see who we can talk to on any particular day.

Community Surprise

I’m fortunate that my most-local ham radio club is a very active and friendly community. Raleigh Amateur Radio Society (RARS) has some sort of get-together most weeks, and a large variety of expertise and encouragement. Of its 500–600 members, probably around 100 are really active in one or another club activity. I didn’t expect the ham club to become one of my social anchors, but it has. I am there about every other week, sharing ideas with and learning from other hams.

Hobby Surprise

Ham radio is “ten hobbies in a trench coat masquerading as one hobby” and hams have wildly different interests. Some only want to operate radios. Some find designing and building things more interesting than actually talking to people over the air. I love making things; it’s why 3D printing and hobby machining have been so fun for me. I found that designing and making antennas has been an unexpected hobby. I joke that it’s my “antenna of the month club” hobby, but the joke is not far off reality. I think I have designed and/or made either a new antenna or a newly-designed part of or new feature for an antenna at least a dozen times this year. Some of them have turned into open source projects that I’ve published, like my QRP 6/10/12/15/17/20/30/40m inverted-V linked dipole. This goes beyond antennas; I also designed a 3D-printed single-lever paddle for sending Morse code, and have fixed things for fellow hams.

There is more to learn in this hobby than one person can learn in a lifetime, and since I started well more than halfway through a typical full lifetime, I can’t imagine ever becoming bored.

I’m looking forward to a new year of learning more.

Death of a fridge

2024-08-20T00:00:00+00:00

Eight days shy of nine years ago, I got lucky when my new fridge I had purchased arrived hours after the previous unit died.

Today, I got texts about thawed meat in the freezer, and by the time I got home, the fridge was up to about 60°F.

Since the display is showing nonsense for the fridge and freezer temperatures, it’s the control board (just like last time) with a one-year warranty, not the refrigeration system with a ten-year warranty, which means there’s no point in trying to repair it.

Our first fridge in this house, a GE, lasted a bit over ten years. This second, an LG, lasted slightly less. I’ve heard definitely do not buy a Samsung fridge from a lot of people, so what next? I got a recommendation for Whirlpool. Maybe we’ll try that?

But in any case, full-size fridge delivery is at best a few days out, and many are a week or a month or more out.

So while we work on finding a new full-size fridge that will work well for us and can be delivered in reasonable time, I found an inexpensive fridge not that much different from the one that Technology Connections covered except that it’s not retro-style red. So… I have now ordered the same thermostatic controller he did to make it work better, initially as an emergency fridge, and later as an occasional backup fridge.

Fedora Linux on Lenovo T16 gen2 Intel

2023-08-11T00:00:00+00:00

My 14” Lenovo T490 laptop broke, so I bought a 16” T16 Intel gen2 to replace it. I chose Intel instead of AMD at least because it was required for the highest resolution screen, and I think it mattered for memory configuration as well.

I’ve found the 14” to be a little limiting when doing FreeCAD work, and while doing parametric CAD it’s really nice to have a numeric keypad. I’m having a bit of trouble getting used to the offset keyboard on the T16, but I think it will end up being a good tradeoff in the end. And I could not buy the same screen on a T14 that I had on my T490.

I could not find any information online about running Linux on this hardware. It was released only a few months ago. So I thought I’d report that running Fedora 37 (at this time) everything so far seems to work, including WiFi and the webcam, which I guess isn’t one of the MIPI IPU6 cameras from Intel that do not have upstream support on Linux, and aren’t expected to for years. Sleep and resume work fine.

It appears to use the same motherboard as the T14 gen4, so there is a good chance but no guarantee that this will apply equally to those systems.

I don’t have the fingerprint reader, smart card, or WWAN to test Linux with.

I moved my existing SSD over from the T490 to the T16. I did have to configure in BIOS to trust third party certificates signed by Microsoft before secure boot would work. That was the only change I needed to make to boot Linux.

Lenovo did not offer this machine without Windows, so I have a Windows license for it. However, I never booted it on this system.

I’m sad that there’s no microSD slot any more. On the broken T490, I used a 1TB microSD as a local backup target. Now I need to figure out what to do for local backups, or whether I give up and say network backups are enough. I will try a low-profile microSD USB-A USB-3.0 adapter and see whether leaving it in place permanently is feasible.

Some hardware information:

$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 4: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 4: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 4: Dev 2, If 2, Class=Video, Driver=uvcvideo, 480M
    |__ Port 4: Dev 2, If 3, Class=Video, Driver=uvcvideo, 480M
    |__ Port 4: Dev 2, If 4, Class=Application Specific Interface, Driver=, 480M
    |__ Port 10: Dev 3, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 10: Dev 3, If 1, Class=Wireless, Driver=btusb, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/3p, 20000M/x2
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
# lspci
00:00.0 Host bridge: Intel Corporation Device a708 (rev 01)
00:02.0 VGA compatible controller: Intel Corporation Raptor Lake-P [Iris Xe Graphics] (rev 04)
00:04.0 Signal processing controller: Intel Corporation Device a71d (rev 01)
00:06.0 PCI bridge: Intel Corporation Device a74d (rev 01)
00:07.0 PCI bridge: Intel Corporation Device a76e (rev 01)
00:07.2 PCI bridge: Intel Corporation Device a72f (rev 01)
00:0d.0 USB controller: Intel Corporation Device a71e (rev 01)
00:0d.2 USB controller: Intel Corporation Device a73e (rev 01)
00:0d.3 USB controller: Intel Corporation Device a76d (rev 01)
00:14.0 USB controller: Intel Corporation Alder Lake PCH USB 3.2 xHCI Host Controller (rev 01)
00:14.2 RAM memory: Intel Corporation Alder Lake PCH Shared SRAM (rev 01)
00:14.3 Network controller: Intel Corporation Device 51f1 (rev 01)
00:15.0 Serial bus controller: Intel Corporation Alder Lake PCH Serial IO I2C Controller #0 (rev 01)
00:16.0 Communication controller: Intel Corporation Alder Lake PCH HECI Controller (rev 01)
00:1f.0 ISA bridge: Intel Corporation Device 519d (rev 01)
00:1f.3 Audio device: Intel Corporation Device 51ca (rev 01)
00:1f.4 SMBus: Intel Corporation Alder Lake PCH-P SMBus Host Controller (rev 01)
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-P PCH SPI Controller (rev 01)
00:1f.6 Ethernet controller: Intel Corporation Device 0dc6 (rev 01)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

Here’s an abbreviated summary of my custom order in case it helps:

Processor 13th Generation Intel® Core™ i7-1355U Processor
Total Memory 16 GB DDR5-5200MHz (Soldered)
- Augmented with “Crucial RAM 32GB DDR5 5200MT/s (or 4800MT/s) Laptop Memory CT32G52C42S5” — note that after adding this, the system blinks ESC/F1/F4 with a dark screen for a while as it adjusts to its new memory. It does finish booting after a couple minutes. This is documented in the HMM
Solid State Drive 256 GB SSD M.2 2280 PCIe Gen4 TLC Opal
- This is a Western Digital PC SN740, which I have removed but can replace if I need to return for service
- Replaced with the Samsung 970 EVO Plus pulled from my dying T490
Display 16” WQUXGA (3840 x 2400), OLED, Anti-Reflection/Anti-Smudge, Non-Touch, HDR 500, 100%DCI-P3, 400 nits
Graphic Card Integrated Intel® UHD Graphics
Camera 5MP RGB+IR with Microphone
Wireless Intel® Wi-Fi 6E AX211 2x2 AX & Bluetooth® 5.1 or above
Integrated Mobile Broadband No Wireless WAN
Ethernet Wired Ethernet
Fingerprint Reader No Fingerprint Reader
System Expansion Slots No Smart Card Reader
TPM Setting Enabled Discrete TPM2.0
Battery 4 Cell Li-Polymer 86Wh
Power Cord 135W USB-C Slim 90% PCC 3pin AC Adapter - US
- This is the only adapter supplied if the larger battery is selected, but it’s still possible to charge from a 65W adapter more slowly.

Deploying Mastodon on CentOS 9 and derivatives with podman

2022-11-12T00:00:00+00:00

Warning: May 2023 I have given up on using podman due to multiple permissions failures with SELinux that I never resolved. I have moved to only runnning Mastodon on docker.

This now documents what I did that did not work.

After setting up Maker Forums Social runnning Mastodon on docker, I volunteered to deploy Aviating.com Social as well. This time, instead of using docker, I built on podman.

Warning: 5 February 2023 Updates are now attempting to replace runc with Docker’s containerd.io package. Using docker-compose with podman looks like a bad idea at this point. This document now serves as a document of how I have deployed a system, but I will shortly be working on a new deployment strategy, based either on podman generate systemd to generate individual systemd unit files and adding dependencies, or podman generate/play kube to create a Kubernetes yaml file and then run it.

System installation and preparation follow my earlier guidelines, including firewall, sysctl, and SELinux configuration, and this post builds on that with similar instructions for using podman directly, making it easy to benefit from SELinux as well as the rest of the podman architecture.

Podman

I first tried to replace docker-compose with podman-compose. It took lots of failures before I gave up. I then found that you can use docker-compose with podman now. At the time of writing, docker-compose is not available in EPEL, so we instead use the docker-compose plugin and create a link. Then we have to enable the socket that docker-compose uses to manage podman. Finally, we want git available to have the Mastodon sources handy.

# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# dnf install docker-compose-plugin
# ln -s /usr/libexec/docker/cli-plugins/docker-compose /usr/local/bin/docker-compose
# systemctl enable --now podman.socket
# dnf install git

Application

Now you’re finally ready to start installing Mastodon itself. Make room for Mastadon to live in:

# mkdir -p /opt/mastodon/database/{postgresql,pgbackups,redis,elasticsearch}
# mkdir -p /opt/mastodon/web/{public,system,static}
# chown 991:991 /opt/mastodon/web/{public,system,static}
# chown 1000 /opt/mastodon/database/elasticsearch
# chown 70:70 /opt/mastodon/database/pgbackups
# cd /opt/mastodon
# touch application.env database.env
# semanage fcontext -a -t httpd_sys_content_t /opt/mastodon/web/public
# restorecon -R -v /opt/mastodon/web/public

Create /opt/mastodon/docker-compose.yml

version: '3'

services:
  postgresql:
    image: postgres:14-alpine
    env_file: database.env
    restart: always
    shm_size: 512mb
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'postgres']
    volumes:
      - /opt/mastodon/database/postgresql:/var/lib/postgresql/data:z
      - /opt/mastodon/database/pgbackups:/backups:z
    networks:
      - internal_network

#  pgbouncer:
#    image: edoburu/pgbouncer:1.12.0
#    env_file: database.env
#    depends_on:
#      - postgresql
#    healthcheck:
#      test: ['CMD', 'pg_isready', '-h', 'localhost']
#    networks:
#      - internal_network

  redis:
    image: redis:7-alpine
    restart: always
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    volumes:
      - /opt/mastodon/database/redis:/data:z
    networks:
      - internal_network

  redis-volatile:
    image: redis:7-alpine
    restart: always
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    networks:
      - internal_network

  elasticsearch:
    image: elasticsearch:7.17.4
    restart: always
    env_file: database.env
    environment:
      - cluster.name=elasticsearch-mastodon
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - ingest.geoip.downloader.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test: ["CMD-SHELL", "nc -z elasticsearch 9200"]
    volumes:
      - /opt/mastodon/database/elasticsearch:/usr/share/elasticsearch/data:z
    networks:
      - internal_network

  website:
    #image: localhost/mastodon:v4.0.0
    image: tootsuite/mastodon:v4.0.0
    env_file:
      - application.env
      - database.env
    command: bash -c "bundle exec rails s -p 3000"
    restart: always
    depends_on:
      - postgresql
#      - pgbouncer
      - redis
      - redis-volatile
      - elasticsearch
    ports:
      - '127.0.0.1:3000:3000'
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:3000/health || exit 1']
    volumes:
      - /opt/mastodon/web/system:/mastodon/public/system:z

  shell:
    #image: localhost/mastodon:v4.0.0
    image: tootsuite/mastodon:v4.0.0
    env_file:
      - application.env
      - database.env
    command: /bin/bash
    restart: "no"
    networks:
      - internal_network
      - external_network
    volumes:
      - /opt/mastodon/web/system:/mastodon/public/system:z
      - /opt/mastodon/web/static:/static:z

  streaming:
    #image: localhost/mastodon:v4.0.0
    image: tootsuite/mastodon:v4.0.0
    env_file:
      - application.env
      - database.env
    command: node ./streaming
    restart: always
    depends_on:
      - postgresql
#      - pgbouncer
      - redis
      - redis-volatile
      - elasticsearch
    ports:
      - '127.0.0.1:4000:4000'
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health || exit 1']

  sidekiq:
    #image: localhost/mastodon:v4.0.0
    image: tootsuite/mastodon:v4.0.0
    env_file:
      - application.env
      - database.env
    command: bundle exec sidekiq
    restart: always
    depends_on:
      - postgresql
#      - pgbouncer
      - redis
      - redis-volatile
      - website
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
    volumes:
      - /opt/mastodon/web/system:/mastodon/public/system:z

networks:
  external_network:
  internal_network:
  #  internal: true

(Note: I have not tested the pgbouncer configuration; that is copied from sleeplessbeastie’s guide, and I have left it commented out in case I later need to add pgbouncer to my configuration.)

You can choose whether to use major versions and auto-upgrade minor versions with docker-compose pull, or manually change minor versions, for redis and postgresql.

You will want to lock mastodon to a specific version, follow updates (ATOM), and observe update procedures called out in the release notes when upgrading Mastodon. There is no major version tag for Elasticsearch on docker, so regularly check Elastic Docker Hub especially for security updates to address Java security flaws as they are discovered.

Note: you cannot set the internal_network as an internal network and use firewalld. The default docker-compose.yml that comes with Mastodon sets:

networks:
  external_network:
  internal_network:
    internal: true

That internal: true doesn’t work with firewalld, which is why it is commented out in the docker-compose.yml here. If this is ever fixed, you may be able to re-add that additional restriction.

Choose or build image

All of the image: specifications in the docker-compose.yml file that start with tootsuite/ are the official builds, which you can use as-is.

However, it’s handy to have the source around, so I recommend you clone it.

# cd /opt/mastodon
# git clone https://github.com/mastodon/mastodon.git

If you are installing the Glitch-soc version of Mastodon, instead clone this repository:

# git clone https://github.com/glitch-soc/mastodon.git

Whichever version you install, this gives you an easy reference to all the files there, including the .env.production.sample file that you will want to reference when setting up your environment files. There are additional settings not included in this example application.yml file that you will want to consider for your deployment.

If you want to build an image from source, do this using a meaningful tag for the version you are actually using:

# cd /opt/mastodon/mastodon
# podman build --format docker -f Dockerfile --tag mastodon:v4.0.0

This will take a while depending on your hardware, but it could easily be 15 minutes. Having done that, modify the image: tootsuite/ lines in docker-compose.yml to instead reference image: localhost/mastodon:v4.0.0 (or whatever tag you provided at the time you built it.)

Pull images you did not build

Now download all the images required to compose your system.

# docker-compose pull

You’ll have to choose a registry source. I chose docker.io/* everywhere.

Secrets and configuration

It’s time to fill in application.env and database.env. You will want to start with what’s in the current .env.production.sample file in the Mastodon source for the version you are running. But for clarity, here’s a template application.env file:

# environment
RAILS_ENV=production
NODE_ENV=production

# domain
LOCAL_DOMAIN=your.server.fqdn

# redirect to the first profile
SINGLE_USER_MODE=false

# do not serve static files
RAILS_SERVE_STATIC_FILES=false

# concurrency
WEB_CONCURRENCY=2
MAX_THREADS=5

# pgbouncer
#PREPARED_STATEMENTS=false

# locale
DEFAULT_LOCALE=en

# email, not used
SMTP_SERVER=mailserver.invalid
SMTP_PORT=587
SMTP_LOGIN=mastodon
SMTP_PASSWORD=ifYouNeedId
SMTP_FROM_ADDRESS=notifications-noreply@your.server.fqdn


# secrets
SECRET_KEY_BASE=add
OTP_SECRET=add

# Changing VAPID keys will break push notifications
VAPID_PRIVATE_KEY=add
VAPID_PUBLIC_KEY=add

To generate values for SECRET_KEY_BASE and OTP_SECRET run this twice to generate two different keys:

# docker-compose run --rm shell bundle exec rake secret

To generate values for VAPID_PRIVATE_KEY and VAPID_PUBLIC_KEY:

# docker-compose run --rm shell bundle exec rake mastodon:webpush:generate_vapid_key

Here’s a template database.env file:

# postgresql configuration
POSTGRES_USER=mastodon
POSTGRES_DB=mastodon
POSTGRES_PASSWORD=generate1
PGPASSWORD=generate1
PGPORT=5432
PGHOST=postgresql
PGUSER=mastodon

# pgbouncer configuration
#POOL_MODE=transaction
#ADMIN_USERS=postgres,mastodon
#DATABASE_URL="postgres://mastodon:generate1@postgresql:5432/mastodon"

# elasticsearch
ES_JAVA_OPTS=-Xms512m -Xmx512m
ELASTIC_PASSWORD=generate2

# mastodon database configuration
#DB_HOST=pgbouncer
DB_HOST=postgresql
DB_USER=mastodon
DB_NAME=mastodon
DB_PASS=generate1
DB_PORT=5432

REDIS_HOST=redis
REDIS_PORT=6379

CACHE_REDIS_HOST=redis-volatile
CACHE_REDIS_PORT=6379

ES_ENABLED=true
ES_HOST=elasticsearch
ES_PORT=9200
ES_USER=elastic
ES_PASS=generate2

To generate keys for the generate1 and generate2 values, use this twice:

# openssl rand -base64 15

Bring up

With the environment files filled with secrets and keys, initialize those services. First get the static files ready to be served directly by nginx on the host. The two-stage copy is because SELinux appropriately prevents the container from writing directly to the final location, but cp outside the container is able to copy the files to where nginx will be able to see them.

# docker-compose run --rm shell bash -c "cp -r /opt/mastodon/public/* /static/"
# unalias cp
# cp -rf web/static/* web/public

Then bring up the data layer.

# docker-compose up -d postgresql redis redis-volatile
# watch podman ps

Wait for running (healthy), then Control-C and initialize the database.

# docker-compose run --rm shell bundle exec rake db:setup

Note that later, after each mastodon update, you will need to run all database migrations, and update your copies of static files.

# docker-compose run --rm shell bundle exec rake db:migrate
# docker-compose run --rm shell bash -c "cp -r /opt/mastodon/public/* /static/"
# docker-compose -f docker-compose.yml run --rm shell bash -c "cp -r /opt/mastodon/public/* /static/"
# unalias cp
# cp -rf web/static/* web/public

Some updates require additional migration steps. Read the release notes.

Handle HTTPS

Install certbot and the nginx plugin for certbot.

# dnf config-manager --set-enabled crb
# dnf install epel-release
# dnf install certbot python3-certbot-nginx

Get a key. You’ll have to answer a bunch of questions. Make sure you include the FQDN of the server!

# certbot --nginx

The plugin may have started nginx without stopping it:

# killall nginx

Now create /etc/nginx/conf.d/mastodon.conf and change mastodon.example.com everywhere to the fully-qualified domain name for your server:

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream backend {
    server 127.0.0.1:3000 fail_timeout=0;
}

upstream streaming {
    server 127.0.0.1:4000 fail_timeout=0;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d max_size=1g;

server {
  listen 80;
  server_name mastodon.example.com;
  location / { return 301 https://$host$request_uri; }
}

server {
  listen 443 ssl http2;
  server_name mastodon.example.com;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;
  ssl_session_tickets off;

  ssl_certificate     /etc/letsencrypt/live/mastodon.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/mastodon.example.com/privkey.pem;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 80m;

  root /opt/mastodon/web/public;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml image/x-icon;

  add_header Strict-Transport-Security "max-age=31536000" always;

  location / {
    try_files $uri @proxy;
  }

  location ~ ^/(system/accounts/avatars|system/media_attachments/files) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000" always;
    # SELinux does not allow http and container access to these files together
    # to fix this, set up an nginx container for static files and
    # proxy to that container for static files
    #root /opt/mastodon/web/;
    try_files $uri @proxy;
  }

  location ~ ^/(emoji|packs) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000" always;
    try_files $uri @proxy;
  }

  location /sw.js {
    add_header Cache-Control "public, max-age=0";
    add_header Strict-Transport-Security "max-age=31536000" always;
    try_files $uri @proxy;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://backend;
    proxy_buffering on;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_cache CACHE;
    proxy_cache_valid 200 7d;
    proxy_cache_valid 410 24h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;
    add_header Strict-Transport-Security "max-age=31536000" always;

    tcp_nodelay on;
  }

  location /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";

    proxy_pass http://streaming;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    tcp_nodelay on;
  }

  error_page 500 501 502 503 504 /500.html;
}

Comment out the default server block from /etc/nginx/nginx.conf because it conflicts with the https redirect in mastodon.conf:

#    server {
#        listen       80;
#        listen       [::]:80;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        error_page 404 /404.html;
#        location = /404.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#        location = /50x.html {
#        }
#    }

Now enable and start nginx and the certbot renewal:

# systemctl enable --now nginx.service
# systemctl enable --now certbot-renew.timer

Tootctl

The tootctl CLI tool is essential for administering Mastodon. Make it accessible via the Mastodon shell docker image with a simple shell script.

Contents of /usr/local/bin/tootctl

#!/bin/bash
docker-compose -f /opt/mastodon/docker-compose.yml run --rm shell tootctl "$@"

Then:

# chmod +x /usr/local/bin/tootctl

Start mastodon

More systemd unit files!

Contents of /etc/systemd/system/mastodon.service

[Unit]
Description=Mastodon service
After=podman.service

[Service]
Type=oneshot
RemainAfterExit=yes
StandardError=/var/log/mastodon.err
StandardOutput=/var/log/mastodon.out

WorkingDirectory=/opt/mastodon
ExecStart=/usr/local/bin/docker-compose -f /opt/mastodon/docker-compose.yml up -d
ExecStop=/usr/local/bin/docker-compose -f /opt/mastodon/docker-compose.yml down

[Install]
WantedBy=multi-user.target

Then run

# systemctl daemon-reload
# systemctl enable --now mastodon.service
# watch docker-compose -f /opt/mastodon/docker-compose.yml ps

Wait for running (healthy) before the next step. This will probably take 30 seconds to a minute.

Create admin user

# tootctl accounts create $admin-user --email admin-user@mail.invalid --confirmed --role Admin

Create new secure password, then disable registration during setup.

# tootctl settings registrations close

If you later want to open registrations, go ahead whenever you want to.

# tootctl settings registrations open

Cleanup

Contents of /etc/systemd/system/mastodon-media-remove.service

[Unit]
Description=Mastodon - media remove service
Wants=mastodon-media-remove.timer

[Service]
Type=oneshot
StandardError=/var/log/mastodon-media-remove.err
StandardOutput=/var/log/mastodon-media-remove.out

WorkingDirectory=/opt/mastodon
ExecStart=/usr/local/bin/docker-compose -f /opt/mastodon/docker-compose.yml run --rm shell tootctl media remove

[Install]
WantedBy=multi-user.target

Contents of /etc/systemd/system/mastodon-media-remove.timer

[Unit]
Description=Schedule a media remove every week

[Timer]
Persistent=true
OnCalendar=Sat *-*-* 00:00:00
Unit=mastodon-media-remove.service

[Install]
WantedBy=timers.target

Contents of /etc/systemd/system/mastodon-preview_cards-remove.service

[Unit]
Description=Mastodon - preview cards remove service
Wants=mastodon-preview_cards-remove.timer

[Service]
Type=oneshot
StandardError=/var/log/mastodon-preview-remove.err
StandardOutput=/var/log/mastodon-preview-remove.out

WorkingDirectory=/opt/mastodon
ExecStart=/usr/local/bin/docker-compose -f /opt/mastodon/docker-compose.yml run --rm shell tootctl preview_cards remove

[Install]
WantedBy=multi-user.target

Contents of /etc/systemd/system/mastodon-preview_cards-remove.timer

[Unit]
Description=Schedule a preview cards remove every week

[Timer]
Persistent=true
OnCalendar=Sat *-*-* 00:00:00
Unit=mastodon-preview_cards-remove.service

[Install]
WantedBy=timers.target

Now start them up

# systemctl daemon-reload
# systemctl enable --now mastodon-preview_cards-remove.timer
# systemctl enable --now mastodon-media-remove.timer

Backups

I chose to use restic. Use what you want, but here’s an easy-to-apply pattern using restic.

Contents of /opt/mastodon/backup-files

/etc/nginx
/etc/letsencrypt
/etc/systemd/system
/etc/fail2ban
/root
/opt/mastodon/database/pgbackups
/opt/mastodon/*.env
/opt/mastodon/docker-compose.yml
/opt/mastodon/database/redis
/opt/mastodon/web/system
/opt/mastodon/backup-files
/opt/mastodon/mastodon-backup
/var/lib/rpm
/usr/local/bin

Now run:

# dnf install restic

Contents of /etc/systemd/system/mastodon-backup.timer

[Unit]
Description=Schedule a mastodon backup every hour

[Timer]
Persistent=true
OnCalendar=*:00:00
Unit=mastodon-backup.service

[Install]
WantedBy=timers.target

Contents of /etc/systemd/system/mastodon-backup.service

[Unit]
Description=Mastodon - backup service
# Without this, they can run at the same time and race to docker-compose,
# double-creating networks and failing due to ambiguous network definition
# requiring `docker network prune` and restarting
After=mastodon.service

[Service]
Type=oneshot
StandardError=file:/var/log/mastodon-backup.err
StandardOutput=file:/var/log/mastodon-backup.log

WorkingDirectory=/opt/mastodon
ExecStart=/bin/bash /opt/mastodon/mastodon-backup

[Install]
WantedBy=multi-user.target

Contents of /root/backup-configuration will need some pieces filled in. Do keep in mind that you also need to store the restic password somewhere safe; if it’s only on this system and no where else, your backup is no better than a pile of random data. It’s important to start this after mastodon.service because otherwise mastodon and mastodon-backup will race to create the docker networks, will both succeed in creating the networks,and then be very confused by the duplicate networks. If this ever happens to you, docker network prune will remove duplicate networks and you can start over.

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
SERVER=
PORT=
BUCKET=
RESTIC_PASSWORD_FILE=/root/restic-pasword

/opt/mastodon/backup-init

#!/bin/bash
set -e
. /root/backup-configuration
restic -r s3:https://$SERVER:$PORT/$BUCKET init

/opt/mastodon/mastodon-backup

#!/bin/bash
set -e
. /root/backup-configuration

docker-compose -f /opt/mastodon/docker-compose.yml run --rm postgresql sh -c "pg_dump -Fp  mastodon | gzip > /backups/dump.sql.gz"
restic -r s3:https://$SERVER:$PORT/$BUCKET --cache-dir=/root backup $(cat /opt/mastodon/backup-files) --exclude  /opt/mastodon/database/postgresql
restic -r s3:https://$SERVER:$PORT/$BUCKET --cache-dir=/root forget --prune --keep-hourly 24 --keep-daily 7 --keep-monthly 3

Now start them up

# chmod +x /opt/mastodon/mastodon-backup /opt/mastodon/backup-init
# /opt/mastodon/backup-init
# systemctl daemon-reload
# systemctl enable --now mastodon-backup.service
# systemctl enable --now mastodon-backup.timer

Confirm that hourly backups are happening and accessible using

# restic -r s3:https://$SERVER:$PORT/$BUCKET snapshots
# restic -r s3:https://$SERVER:$PORT/$BUCKET mount /mnt

Search

Search will (as of this writing) fail to deploy until at least one local toot has been made. After the first toot on your server, initialize search:

# tootctl search deploy

Questions?

I’ll try to improve this post to address your questions if I know the answers. Feel free to reach out by comments on Mastodon.

Deploying Mastodon on CentOS 9 and derivatives

2022-06-04T00:00:00+00:00

I recently found myself setting up a Mastodon server. Elon Musk threatening to buy Twitter reminded me how fragile it is to live in walled gardens controlled by others. I know something of what it takes to run large-scale applications in the cloud. It’s my job. (Senior Director of Engineering, Platform, at Pendo.) I have some clue what it costs, and I’m not upset at being presented advertisements in Twitter. I even like being recommended content that I didn’t know that I wanted to find!

But I’m not satisfied with what The Algorithm shows me, and I don’t like how my timeline (yes, even after choosing chronological) mutates halfway through my reading a tweet that I wanted to finish but which I’ll never find again. It’s like reading a book when I am dreaming; as soon as I realize I’m reading a book, it fades away and I can’t read it any more. Dreams and Twitter both frustrate me this way.

Many folks don’t remember what it was like when a lot of email was not compatible across systems. Most really don’t understand what a gift it is that you can just ask someone for their “email address” and don’t have to ask them which email service you have to sign up with to talk to them. (The fact that people sometimes ask for your “gmail address” as if there were no other email provider demonstrates that keeping interoperating systems is not the low-energy state.)

I’m not boycotting twitter. ~~I’m still there.~~ But… Remember how as interoperable email started to win, the walled garden email providers eventually had to bridge with it to stay relevant to their uesrs? The more of us move content into the Fediverse, the more pressure there will be for Twitter — and similar networks — to bridge. Right now it’s still niche. But the only way for it to become more mainstream is for more of us to adopt it. (If you don’t remember, just take my word for it.)

And I like having more control over what I see, more options for how to interact. And I really appreciate open standards, like ActivityPub and ActivityStreams.

I am making two contributions to add straw to the camel’s back here: Running Maker Forums Social Mastodon and writing this document about how I deployed it on a modern OS in a way that I believe will make it easy to maintain on an ongoing basis.

Credit

This was heavily based on sleeplessbeastie’s How to take advantage of Docker to install Mastodon but adjusted to work on CentOS 9 derivatives like RHEL, AlmaLinux, and Rocky Linux, and a bit more opinionated, such as recommending a particular backup solution.

Installation

I started from AlmaLinux 9, and the configuration I used during installation was:

Set hostname to the fully-qualified domain name
Select “Custom Storage Configuration”, then “Create default configuration”
Decreased swap to 2GB — if you are swapping much, this system won’t work well anyway
At least 10GB root (my 15GB appears to be excessive)
At least 15GB /var/lib/docker or /var/lib/containers (for podman) (my 25GB appears to be excessive)
The rest of the space for /opt/mastodon
Software Selection
- Server (instead of the default server with GUI) on the left, then on the right add
- Guest Agents
- Debugging Tools
- Mail Server

Preparation

If you need to temporarily allow password login by SSH during installation, the first task after installation is to set up SSH keys and then disable password login. After confirming that key-based login works, modify /etc/ssh/sshd_config to say PasswordAuthentication no and then systemctl restart sshd.

Some of the containers need to reach out to the internet to do their jobs, and the server needs to respond to http and https requests.

# firewall-cmd --add-service http --add-service https --zone public
# firewall-cmd --zone=public --add-masquerade
# firewall-cmd --runtime-to-permanent

Elasticsearch is needed for search, and it needs to increase this setting:

# echo "vm.max_map_count=262144" > /etc/sysctl.d/90-max_map_count.conf
# sysctl --load /etc/sysctl.d/90-max_map_count.conf

Use memory more efficiently:

# echo 'vm.overcommit_memory=1' > /etc/sysctl.d/90-vm_overcommit_memory.conf
# sysctl --load /etc/sysctl.d/90-vm_overcommit_memory.conf

SELinux needs one change for this to work.

# setsebool -P httpd_can_network_connect 1

If you do not plan to use cockpit, you can further reduce footprint. I chose:

# dnf erase $(rpm -qa | grep cockpit) --exclude=jq,parted,gdisk

Install nginx on the host

# dnf install nginx

Installing docker

Now you have a choice:

Use docker? Read this post.
Use podman? ~~Read that post.~~ This is currently failing for me, so I am not recommending it at this time.

I used the CentOS Docker package instructions converting from yum usage to dnf usage:

# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# dnf install --allowerasing docker-ce docker-ce-cli containerd.io docker-compose-plugin

(I needed to include --alloweraseing because of a conflict with podman and buildah that are already installed; they need to be erased to install docker.)

# systemctl enable --now docker

Application

Now you’re finally ready to start installing Mastodon itself. Make room for Mastadon to live in:

# mkdir -p /opt/mastodon/database/{postgresql,pgbackups,redis,elasticsearch}
# mkdir -p /opt/mastodon/web/{public,system}
# chown 991:991 /opt/mastodon/web/{public,system}
# chown 1000 /opt/mastodon/database/elasticsearch
# chown 70:70 /opt/mastodon/database/pgbackups
# cd /opt/mastodon
# touch application.env database.env

Create /opt/mastodon/docker-compose.yml

version: '3'

services:
  postgresql:
    image: postgres:14-alpine
    env_file: database.env
    restart: always
    shm_size: 512mb
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'postgres']
    volumes:
      - postgresql:/var/lib/postgresql/data
      - pgbackups:/backups
    networks:
      - internal_network

#  pgbouncer:
#    image: edoburu/pgbouncer:1.12.0
#    env_file: database.env
#    depends_on:
#      - postgresql
#    healthcheck:
#      test: ['CMD', 'pg_isready', '-h', 'localhost']
#    networks:
#      - internal_network

  redis:
    image: redis:7-alpine
    restart: always
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    volumes:
      - redis:/data
    networks:
      - internal_network

  redis-volatile:
    image: redis:7-alpine
    restart: always
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
    networks:
      - internal_network

  elasticsearch:
    image: elasticsearch:7.17.4
    restart: always
    env_file: database.env
    environment:
      - cluster.name=elasticsearch-mastodon
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - ingest.geoip.downloader.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test: ["CMD-SHELL", "nc -z elasticsearch 9200"]
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    networks:
      - internal_network

  website:
    image: tootsuite/mastodon:v3.5.3
    env_file: 
      - application.env
      - database.env
    command: bash -c "bundle exec rails s -p 3000"
    restart: always    
    depends_on:
      - postgresql
#      - pgbouncer
      - redis
      - redis-volatile
      - elasticsearch
    ports:
      - '127.0.0.1:3000:3000'
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:3000/health || exit 1']
    volumes:
      - uploads:/mastodon/public/system

  shell:
    image: tootsuite/mastodon:v3.5.3
    env_file: 
      - application.env
      - database.env
    command: /bin/bash 
    restart: "no"
    networks:
      - internal_network
      - external_network
    volumes:
      - uploads:/mastodon/public/system
      - static:/static

  streaming:
    image: tootsuite/mastodon:v3.5.3
    env_file: 
      - application.env
      - database.env
    command: node ./streaming
    restart: always
    depends_on:
      - postgresql
#      - pgbouncer
      - redis
      - redis-volatile
      - elasticsearch
    ports:
      - '127.0.0.1:4000:4000'
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health || exit 1']

  sidekiq:
    image: tootsuite/mastodon:v3.5.3
    env_file: 
      - application.env
      - database.env
    command: bundle exec sidekiq
    restart: always
    depends_on:
      - postgresql
#      - pgbouncer
      - redis
      - redis-volatile
      - website
    networks:
      - internal_network
      - external_network
    healthcheck:
      test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]
    volumes:
      - uploads:/mastodon/public/system

networks:
  external_network:
  internal_network:
  #  internal: true

volumes:
  postgresql:
    driver_opts:
      type: none
      device: /opt/mastodon/database/postgresql
      o: bind    
  pgbackups:
    driver_opts:
      type: none
      device: /opt/mastodon/database/pgbackups
      o: bind    
  redis:
    driver_opts:
      type: none
      device: /opt/mastodon/database/redis
      o: bind    
  elasticsearch:
    driver_opts:
      type: none
      device: /opt/mastodon/database/elasticsearch
      o: bind    
  uploads:
    driver_opts:
      type: none
      device: /opt/mastodon/web/system
      o: bind    
  static:
    driver_opts:
      type: none
      device: /opt/mastodon/web/public
      o: bind    

(Note: I have not tested the pgbouncer configuration; that is copied from sleeplessbeastie’s guide, and I have left it commented out in case I later need to add pgbouncer to my configuration.)

You can choose whether to use major versions and auto-upgrade minor versions with docker compose pull, or manually change minor versions, for redis and postgresql. You will want to lock mastodon to a specific version, follow updates (ATOM), and observe update procedures called out in the release notes when upgrading Mastodon. There is no major version tag for Elasticsearch on docker, so regularly check Elastic Docker Hub especially for security updates to address Java security flaws as they are discovered.

Note: you cannot set the internal_network as an internal network and use firewalld. The default docker-compose.yml that comes with Mastodon sets:

networks:
  external_network:
  internal_network:
    internal: true

That internal: true doesn’t work with firewalld, which is why it is commented out in the docker-compose.yml here. If this is ever fixed, you may be able to re-add that additional restriction.

Choose or build image

All of the image: specifications in the docker-compose.yml file that start with tootsuite/ are the official builds, which you can use as-is.

However, it’s handy to have the source around, so I recommend you clone it.

# cd /opt/mastodon
# git clone https://github.com/mastodon/mastodon.git

If you are installing the Glitch-soc version of Mastodon, instead clone this repository:

# git clone https://github.com/glitch-soc/mastodon.git

If you want to build an image from source, do this using a meaningful tag for the version you are actually using:

# cd /opt/mastodon/mastodon
# docker build -f Dockerfile --tag mastodon:v4.0.0 .

This will take a while depending on your hardware, but it could easily be 15 minutes. Having done that, modify the image: tootsuite/ lines in docker-compose.yml to instead reference image: mastodon:v4.0.0 (or whatever tag you provided at the time you built it.)

Pull images you did not build

Now download all the images required to compose your system.

# docker compose pull

Secrets and configuration

# environment
RAILS_ENV=production
NODE_ENV=production

# domain
LOCAL_DOMAIN=your.server.fqdn

# redirect to the first profile
SINGLE_USER_MODE=false

# do not serve static files
RAILS_SERVE_STATIC_FILES=false

# concurrency
WEB_CONCURRENCY=2
MAX_THREADS=5

# pgbouncer
#PREPARED_STATEMENTS=false

# locale
DEFAULT_LOCALE=en

# email, not used
SMTP_SERVER=mailserver.invalid
SMTP_PORT=587
SMTP_LOGIN=mastodon
SMTP_PASSWORD=ifYouNeedId
SMTP_FROM_ADDRESS=notifications-noreply@your.server.fqdn


# secrets
SECRET_KEY_BASE=add
OTP_SECRET=add

# Changing VAPID keys will break push notifications
VAPID_PRIVATE_KEY=add
VAPID_PUBLIC_KEY=add

To generate values for SECRET_KEY_BASE and OTP_SECRET run this twice to generate two different keys:

# docker compose run --rm shell bundle exec rake secret

To generate values for VAPID_PRIVATE_KEY and VAPID_PUBLIC_KEY:

# docker compose run --rm shell bundle exec rake mastodon:webpush:generate_vapid_key

Here’s a template database.env file:

# postgresql configuration
POSTGRES_USER=mastodon
POSTGRES_DB=mastodon
POSTGRES_PASSWORD=generate1
PGPASSWORD=generate1
PGPORT=5432
PGHOST=postgresql
PGUSER=mastodon

# pgbouncer configuration
#POOL_MODE=transaction
#ADMIN_USERS=postgres,mastodon
#DATABASE_URL="postgres://mastodon:generate1@postgresql:5432/mastodon"

# elasticsearch
ES_JAVA_OPTS=-Xms512m -Xmx512m
ELASTIC_PASSWORD=generate2

# mastodon database configuration
#DB_HOST=pgbouncer
DB_HOST=postgresql
DB_USER=mastodon
DB_NAME=mastodon
DB_PASS=generate1
DB_PORT=5432

REDIS_HOST=redis
REDIS_PORT=6379

CACHE_REDIS_HOST=redis-volatile
CACHE_REDIS_PORT=6379

ES_ENABLED=true
ES_HOST=elasticsearch
ES_PORT=9200
ES_USER=elastic
ES_PASS=generate2

To generate keys for the generate1 and generate2 values, use this twice:

# openssl rand -base64 15

Bring up

With the environment files filled with secrets and keys, initialize those services. First get the static files ready to be served directly by nginx on the host:

# docker compose run --rm shell bash -c "cp -r /opt/mastodon/public/* /static/"

Then bring up the data layer.

# docker compose up -d postgresql redis redis-volatile
# watch docker compose ps

Wait for running (healthy), then Control-C and initialize the database.

# docker compose run --rm shell bundle exec rake db:setup

Note that later, after each mastodon update, you will need to run all database migrations, and update your copies of static files:

# docker compose run --rm shell bundle exec rake db:migrate
# docker compose run --rm shell bash -c "cp -r /opt/mastodon/public/* /static/"

Handle HTTPS

Install certbot and the nginx plugin for certbot.

# dnf config-manager --set-enabled crb
# dnf install epel-release
# dnf install certbot python3-certbot-nginx

Get a key. You’ll have to answer a bunch of questions. Make sure you include the FQDN of the server!

# certbot --nginx

The plugin may have started nginx without stopping it:

# killall nginx

Now copy dist/nginx.conf to /etc/nginx/conf.d/mastodon.conf, then

change example.com everywhere to the fully-qualified domain name for your server
uncomment the ssl_certificate and ssl_certificate_key lines

Comment out the default server block from /etc/nginx/nginx.conf because it conflicts with the https redirect in mastodon.conf:

#    server {
#        listen       80;
#        listen       [::]:80;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        error_page 404 /404.html;
#        location = /404.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#        location = /50x.html {
#        }
#    }

Now enable and start nginx and the certbot renewal:

# systemctl enable --now nginx.service
# systemctl enable --now certbot-renew.timer

Tootctl

The tootctl CLI tool is essential for administering Mastodon. Make it accessible via the Mastodon shell docker image with a simple shell script.

Contents of /usr/local/bin/tootctl

#!/bin/bash
docker compose -f /opt/mastodon/docker-compose.yml run --rm shell tootctl "$@"

Then:

# chmod +x /usr/local/bin/tootctl

Start mastodon

More systemd unit files!

Contents of /etc/systemd/system/mastodon.service

[Unit]
Description=Mastodon service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
StandardError=/var/log/mastodon.err
StandardOutput=/var/log/mastodon.out

WorkingDirectory=/opt/mastodon
ExecStart=/usr/bin/docker compose -f /opt/mastodon/docker-compose.yml up -d
ExecStop=/usr/bin/docker compose -f /opt/mastodon/docker-compose.yml down

[Install]
WantedBy=multi-user.target

Then run

# systemctl daemon-reload
# systemctl enable --now mastodon.service
# watch docker compose -f /opt/mastodon/docker-compose.yml ps

Wait for running (healthy) before the next step. This will probably take 30 seconds to a minute.

Create admin user

# tootctl accounts create $admin-user --email admin-user@mail.invalid --confirmed --role Admin

Create new secure password, then disable registration during setup.

# tootctl settings registrations close

If you later want to open registrations, go ahead whenever you want to.

# tootctl settings registrations open

Cleanup

Contents of /etc/systemd/system/mastodon-media-remove.service

[Unit]
Description=Mastodon - media remove service
Wants=mastodon-media-remove.timer

[Service]
Type=oneshot
StandardError=/var/log/mastodon-media-remove.err
StandardOutput=/var/log/mastodon-media-remove.out

WorkingDirectory=/opt/mastodon
ExecStart=/usr/bin/docker compose -f /opt/mastodon/docker-compose.yml run --rm shell tootctl media remove

[Install]
WantedBy=multi-user.target

Contents of /etc/systemd/system/mastodon-media-remove.timer

[Unit]
Description=Schedule a media remove every week

[Timer]
Persistent=true
OnCalendar=Sat *-*-* 00:00:00
Unit=mastodon-media-remove.service

[Install]
WantedBy=timers.target

Contents of /etc/systemd/system/mastodon-preview_cards-remove.service

[Unit]
Description=Mastodon - preview cards remove service
Wants=mastodon-preview_cards-remove.timer

[Service]
Type=oneshot
StandardError=/var/log/mastodon-preview-remove.err
StandardOutput=/var/log/mastodon-preview-remove.out

WorkingDirectory=/opt/mastodon
ExecStart=/usr/bin/docker compose -f /opt/mastodon/docker-compose.yml run --rm shell tootctl preview_cards remove

[Install]
WantedBy=multi-user.target

Contents of /etc/systemd/system/mastodon-preview_cards-remove.timer

[Unit]
Description=Schedule a preview cards remove every week

[Timer]
Persistent=true
OnCalendar=Sat *-*-* 00:00:00
Unit=mastodon-preview_cards-remove.service

[Install]
WantedBy=timers.target

Now start them up

# systemctl daemon-reload
# systemctl enable --now mastodon-preview_cards-remove.timer
# systemctl enable --now mastodon-media-remove.timer

Backups

I chose to use restic. Use what you want, but here’s an easy-to-apply pattern using restic.

Contents of /opt/mastodon/backup-files

/etc/nginx
/etc/letsencrypt
/etc/systemd/system
/etc/fail2ban
/root
/opt/mastodon/database/pgbackups
/opt/mastodon/*.env
/opt/mastodon/docker-compose.yml
/opt/mastodon/database/redis
/opt/mastodon/web/system
/opt/mastodon/backup-files
/opt/mastodon/mastodon-backup
/var/lib/rpm
/usr/local/bin

Now run:

# dnf install restic

Contents of /etc/systemd/system/mastodon-backup.timer

[Unit]
Description=Schedule a mastodon backup every hour

[Timer]
Persistent=true
OnCalendar=*:00:00
Unit=mastodon-backup.service

[Install]
WantedBy=timers.target

Contents of /etc/systemd/system/mastodon-backup.service

[Unit]
Description=Mastodon - backup service
# Without this, they can run at the same time and race to docker compose,
# double-creating networks and failing due to ambiguous network definition
# requiring `docker network prune` and restarting
After=mastodon.service

[Service]
Type=oneshot
StandardError=file:/var/log/mastodon-backup.err
StandardOutput=file:/var/log/mastodon-backup.log

WorkingDirectory=/opt/mastodon
ExecStart=/bin/bash /opt/mastodon/mastodon-backup

[Install]
WantedBy=multi-user.target

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
SERVER=
PORT=
BUCKET=
RESTIC_PASSWORD_FILE=/root/restic-pasword

/opt/mastodon/backup-init

#!/bin/bash
set -e
. /root/backup-configuration
restic -r s3:https://$SERVER:$PORT/$BUCKET init

/opt/mastodon/mastodon-backup

#!/bin/bash
set -e
. /root/backup-configuration

docker compose -f /opt/mastodon/docker-compose.yml run --rm postgresql sh -c "pg_dump -Fp  mastodon | gzip > /backups/dump.sql.gz"
restic -r s3:https://$SERVER:$PORT/$BUCKET --cache-dir=/root backup $(cat /opt/mastodon/backup-files) --exclude  /opt/mastodon/database/postgresql
restic -r s3:https://$SERVER:$PORT/$BUCKET --cache-dir=/root forget --prune --keep-hourly 24 --keep-daily 7 --keep-monthly 3

Now start them up

# chmod +x /opt/mastodon/mastodon-backup /opt/mastodon/backup-init
# /opt/mastodon/backup-init
# systemctl daemon-reload
# systemctl enable --now mastodon-backup.service
# systemctl enable --now mastodon-backup.timer

Confirm that hourly backups are happening and accessible using

# restic -r s3:https://$SERVER:$PORT/$BUCKET snapshots
# restic -r s3:https://$SERVER:$PORT/$BUCKET mount /mnt

Search

Search will (as of this writing) fail to deploy until at least one local toot has been made. After the first toot on your server, initialize search:

# tootctl search deploy

Questions?

I’ll try to improve this post to address your questions if I know the answers. Feel free to reach out by comments on Mastodon.

Delicata

2020-12-01T00:00:00+00:00

About a year ago, my father introduced me to delicata squash. So tasty, and so easy to prepare! There are lots of recipes, but just to show how easy this is, here’s mine:

Scrub any dirt off with a vegetable brush
Slice into appoximately 5mm rounds. Thicker to cook longer, or brown less, or brown more evenly; thinner to cook quicker and brown faster. Don’t peel. Don’t pit. Just slice.
Arrange slices on parchment paper or silicone mat on baking sheets. Keep the seeds in their slices.
Drizzle lightly with olive oil. Don’t drench them.
Generously dust with herbes de provence. I use about 1 tablespoon per squash, measured by eye and taste.
Bake in a fast oven (400-425°F, 200-220°C). Time depends on thickness and preference. 40 minutes to 1 hour will generally be sufficient to induce the maillard reaction. They cook slowly until enough water evaporates, then they brown farily quickly. Check for doneness every 5 minutes or so after 30 minutes in the oven.
Serve quickly when done. They cool off quickly.

Dangerous Pumpkin Chiffon Pie

2020-11-26T00:00:00+00:00

Part of my family heritage appears to be to risk illness on some special holidays. Chiffon is made with never-cooked egg whites, so there’s a real risk of salmonella from this pumpkin chiffon pie. But my grandmother made it, and I make it almost every year in her memory. My version uses less sugar, more pumpkin, and a bit more spice than hers, but it’s still the tradition.

Michael’s Pumpkin Chiffon Pie

Ingredients

1 9-inch graham cracker crust (optionally, 10-inch crust)
1/2 cup brown sugar
1 envelope unflavored gelatin
1/2 teaspoon salt
1 1/2 teaspoon cinnamon
1/2 teaspoon nutmeg (freshly grated if possible)
1/2 teaspoon powdered ginger
3 slightly beaten egg yolks
3/4 cup milk
1 3/4 cups canned or mashed cooked pumpkin
3 egg whites
1 tablespoon granulated sugar

Directions

Separate whites and yolks from 3 eggs (4 for 10” crust)
In a 1 to 2 quart saucepan, combine the brown sugar, gelatin, salt, and spices.
Combine the beaten egg yolks and milk; stir these into the brown-sugar mixture.
Cook this and stir until the mixture comes to a boil.
Remove it from heat; stir in the pumpkin. (It is OK to use a whole 15oz can.)
Chill the mixture until it mounds slightly when spooned. (Test it every now and then — don’t let it get too stiff.)
Beat room-temperature egg whites until soft peaks form; gradually add granulated sugar, beating to stiff peaks.
Fold the pumpkin mixture thoroughly into the egg whites, until no more streaks of white are visible.
Turn this mixture into the crust
Chill until firm.

Presentation

Garnish slices as you serve with whipped cream.

Mail Loop

2020-08-21T00:00:00+00:00

My package is “running late” and has been “out for delivery” quite a few times now. It wandered of to New Jersey for a while before returning to North Carolina.

Turns out that mail loops aren’t just a sendmail.cf problem.

August 21, 2020, 3:06 am Departed USPS Regional Destination Facility RALEIGH NC DISTRIBUTION CENTER Your item departed our RALEIGH NC DISTRIBUTION CENTER destination facility on August 21, 2020 at 3:06 am. The item is currently in transit to the destination.

August 21, 2020, 1:36 am Arrived at USPS Regional Facility RALEIGH NC DISTRIBUTION CENTER

August 20, 2020, 11:20 pm Departed USPS Regional Facility GREENSBORO NC NETWORK DISTRIBUTION CENTER

August 20, 2020, 1:25 pm Arrived at USPS Regional Facility GREENSBORO NC NETWORK DISTRIBUTION CENTER

August 20, 2020, 11:42 am Departed USPS Regional Facility GREENSBORO NC DISTRIBUTION CENTER

August 19, 2020, 9:41 pm Arrived at USPS Regional Facility GREENSBORO NC DISTRIBUTION CENTER

August 19, 2020, 7:23 pm Departed USPS Regional Facility RALEIGH NC DISTRIBUTION CENTER

August 19, 2020, 1:40 am Arrived at USPS Regional Facility RALEIGH NC DISTRIBUTION CENTER

August 18, 2020, 11:56 pm Departed USPS Regional Facility GREENSBORO NC DISTRIBUTION CENTER

August 18, 2020, 10:09 pm Arrived at USPS Regional Destination Facility GREENSBORO NC DISTRIBUTION CENTER

August 18, 2020, 3:34 pm Arrived at Post Office BOONE, NC 28607

August 18, 2020 In Transit to Next Facility

August 15, 2020, 4:43 am Arrived at USPS Facility NORTH WILKESBORO, NC 28659

August 15, 2020, 1:57 am Departed USPS Regional Facility GREENSBORO NC NETWORK DISTRIBUTION CENTER

August 14, 2020, 9:52 pm Arrived at USPS Regional Destination Facility GREENSBORO NC NETWORK DISTRIBUTION CENTER

August 14, 2020, 3:17 pm Arrived at USPS Regional Facility RALEIGH NC DISTRIBUTION CENTER

August 14, 2020, 6:12 am Departed USPS Regional Facility JERSEY CITY NJ NETWORK DISTRIBUTION CENTER

August 13, 2020, 8:34 pm Arrived at USPS Regional Facility JERSEY CITY NJ NETWORK DISTRIBUTION CENTER

August 13, 2020, 9:35 am Arrived at USPS Regional Facility RALEIGH NC DISTRIBUTION CENTER

August 13, 2020, 7:52 am Departed USPS Regional Facility GREENSBORO NC DISTRIBUTION CENTER

August 11, 2020, 11:20 pm Arrived at USPS Regional Facility GREENSBORO NC DISTRIBUTION CENTER

August 11, 2020, 9:45 pm Departed USPS Regional Facility RALEIGH NC DISTRIBUTION CENTER

August 11, 2020, 5:02 pm Arrived at USPS Regional Destination Facility RALEIGH NC DISTRIBUTION CENTER

August 11, 2020, 7:10 am Out for Delivery BOONE, NC 28607

August 11, 2020, 1:16 am Arrived at USPS Regional Destination Facility GREENSBORO NC DISTRIBUTION CENTER

August 10, 2020, 3:23 am Arrived at USPS Facility BOONE, NC 28607

August 8, 2020, 10:03 pm Arrived at USPS Regional Destination Facility GREENSBORO NC DISTRIBUTION CENTER

August 8, 2020, 9:41 am Arrived at USPS Facility BOONE, NC 28607

August 8, 2020, 7:10 am Out for Delivery BOONE, NC 28607

August 7, 2020, 11:30 pm Accepted at USPS Destination Facility BOONE, NC 28607

August 7, 2020, 12:08 am Departed Shipping Partner Facility, USPS Awaiting Item CONCORD, NC 28027 Shipping Partner: AMAZON

August 6, 2020, 1:42 am Departed Shipping Partner Facility, USPS Awaiting Item CHARLOTTE, NC 28214 Shipping Partner: AMAZON

August 5, 2020, 10:20 pm Picked Up by Shipping Partner, USPS Awaiting Item CHARLOTTE, NC 28214 Shipping Partner: AMAZON

An unexpected acquaintance

2020-02-09T00:00:00+00:00

The linuxcnc docs say

For (perhaps) more information than you ever wanted to know about stepper motors, search the web for “Jones on stepping motors”. A generous soul has given a great tutorial on all aspects of stepping motor operation.

So I read Jones on stepping motors and as I finished reading, saw:

The standard Linux line printer driver attaches the device /dev/lp0 to the standard parallel port found on most PCs, and /dev/lp1 and /dev/lp2 to the optional additional parallel ports. The line printer driver has many operating modes that may be configured and tested with the tunelp command. The default mode works, and the following tunelp command will restore these defaults:
`tunelp /dev/lp0 -i 0 -w 0 -a off -o off -c off`
This turns off interrupts with -i 0 so that the board need not deal with the acknowledge signal, and it uses a very brief strobe pulse with -w 0, sets the parallel port to ignore the error signal with -a off, ignores the status when the port is opened with -o off, and does not check the status with each byte output with -C off. The settings of the tunelp options -t and -c should not matter because this interface is always ready and thus polling loop iteration is never required.

It’s been a long time since I conceived of and wrote tunelp. I had nearly forgotten about it, but I’m glad I exported full control to user space when I did.

In a sense, I owed this to OS/2. OS/2 didn’t support the standard PC parallel port. It required that the interrupt line be connected. DOS didn’t require or use interrupts for the parallel port, so most systems didn’t connect to the interrupt line, making it impossible to print from OS/2 on those systems. Also, if my memory serves, it sent only one character per interrput. It was famously slow, and picky about printers.

When I re-wrote the Linux parallel port driver to use interrupts and not sit in a constant port-polling loop, I felt competitive. I wanted to go faster than OS/2, with lower CPU overhead, regardless of whether interrupts were available.

I succeeded.

And tunelp allowed both going faster than OS/2 and being able to make the system work with any printer, even ones that couldn’t sense short strobe pulses.

And now I learn that it also was handy for controlling stepper motors.