Introducing Chalkinator: Native desktop blogging app 7 Jun, 2020 Would you be interested in me creating a native (macOS maybe?) desktop app that would manage your blog, geared at developers? I’ve been struck by how, over the years, I’ve had so many great work opportunities that have come my way through tech blogging, and how so few developers do it - surely they should, and if it was easier, maybe they would? Please let me know, I really want to know if it’s worth investing time and effort into this, if it seems like an idea that could help people or if it would just be a waste of time and effort. So I’ve created a prototype (the screenshot was above) for what I’d like to call ‘Chalkinator’. Basically with it you’d set up your site, choose your template, manage your posts, and voila you have a tech blog. It would manage the hassles of organising your AWS/Cloudflare hosting so your blog would basically cost zero to run (unlike say Wordpress where reputable hosting costs a bit for a hobby blog), and wouldn’t slow down under load.

It wouldn’t be complicated like using Jekyll or Hugo or managing AWS hosting yourself, because I don’t think many people have endless hours they can waste getting their blog working.

I’ve used plenty of blogging platforms over the years. And when I’ve blogged for over 10 years it has become apparent that you need something that’ll stick around, otherwise you face painful migrations every few years. This is why I’d charge monthly instead of one-off, this is a win-win for customers and me: Customers get an app maintained in perpetuity, and I get to pay my bills in perpetuity.

It wouldn’t necessarily be only for developers, but it would be geared that way.

Code highlighting would ‘just work’ out of the box without any plugins. I imagine you’d write in Markdown and certain indents would automatically be treated as code.

macOS native perhaps, or maybe cross-platform - I’m not 100% clear on what would be best here to be honest, and would love to hear your suggestions. So what’s with the bizarre name? Well, the thesaurus came up with ‘chalk’ as a synonym for write; my kids love watching ‘Odd Squad’ where agents use their gadgets to solve all sorts of weird problems, and all the gadgets end with ‘inator’… and so, this crazy portmanteau was born :) Anyway I repeat, please let me know if this is a good idea or not. You can get in touch at [email protected] Thanks for reading :)

Flare: Open source 2-way folder sync to Backblaze B2 in Swift 28 May, 2020 Over the years, various syncing applications have bugged me with how often they spike to 100% CPU usage, their memory usage, pricing, and other issues. So I set out to write my own for fun! Spoiler: in the end, I learned you really should just use Dropbox 🤣 However, I did manage to make it work, and reliably enough that I’m using it for some personal files, so I’m open-sourcing it in case anyone wants to check it out. Flare The project is called Flare, which is in reference to the ‘blaze’ part of the backend service’s name. It is essentially a Dropbox clone that performs 2-way sync of files between a local folder and ‘the cloud’ (in this case, Backblaze B2). These files will then sync to many computers automatically. You could use it to keep a folder synced for members of a team, for instance. And because it uses Backblaze, your storage costs are miniscule. It’s written in Swift, available at github.com/chrishulbert/flare, and (more-or-less) works! You really shouldn’t use this for important use, it’s a hobby project! Features Uses zero CPU/RAM at idle because it only syncs hourly (and upon wake from sleep), and has no process running the rest of the time.

Even when running, uses little CPU/RAM due to simplicity and native code.

Syncs to Backblaze B2, which has extremely generous free tier and inexpensive pricing: If you used Flare and could live with its limitations, you could theoretically save a bucketload vs an office of people paying for a popular sync service.

If you delete a file, it will remain accessible in the Backblaze B2 admin panel for the duration of their garbage collection period (a week or a month, potentially). Limitations Both macOS and Windows (probably Linux too, I didn’t check) do not update the ‘last modified’ date of the tree of folders above a modified file. Eg, say you have a structure similar to foo/bar/yada/blah.txt and modify blah.txt. Foo, bar, and yada’s last modified date will unfortunately not change. In a slightly better case, if you rename blah.txt to blah2.txt, the immediate parent folder (yada) will change it’s last modified date, however grandparents and great-grandparents (foo and bar, respectively) will not. This is because those operating systems want to remain efficient. However this means that Flare cannot simply look at a last modified date and determine that it can skip an entire folder tree: it must still scan every subfolder and every file to see if anything has changed, every time it syncs. This is the biggest limitation of Flare. This could be resolved with an always-running agent, but then you’d also need a full-scan when launching the agent to catch any changes that occurred while it wasn’t running. However, for a hobby project I simply do not have the time to pursue this (I have children to chase!). Also, B2 does not give you folder last modified dates either, so you’d need a custom backend.

and modify blah.txt. Foo, bar, and yada’s last modified date will unfortunately not change. In a slightly better case, if you rename blah.txt to blah2.txt, the immediate parent folder (yada) will change it’s last modified date, however grandparents and great-grandparents (foo and bar, respectively) will not. This is because those operating systems want to remain efficient. However this means that Flare cannot simply look at a last modified date and determine that it can skip an entire folder tree: it must still scan every subfolder and every file to see if anything has changed, every time it syncs. This is the biggest limitation of Flare. This could be resolved with an always-running agent, but then you’d also need a full-scan when launching the agent to catch any changes that occurred while it wasn’t running. However, for a hobby project I simply do not have the time to pursue this (I have children to chase!). Also, B2 does not give you folder last modified dates either, so you’d need a custom backend. Flare handles subfolders very poorly. If you create a subfolder, it will sync fine. But if you delete that subfolder, it won’t have the smarts to know to ‘push’ that deletion to the server and ‘pull’ it down and apply it to other computers. IIRC this was due to the lack of metadata that could be reasonably stored with B2, which is totally understandable from their perspective. I really pushed B2 too far with all this.

Renames count as a deletion of the old file, and creation of a new file.

Huge files are simply thrown in the too-hard basket and ignored.

The macOS service that it used to schedule it is flaky: It launches Flare on wake from sleep only about half the time.

In general: What this project taught me is that 2-way sync is an exponentially more complicated problem than 1-way sync (although not an impossible problem: I got reasonably close to a useful solution!) and you should use one of the popular services for anything important, and be amazed when they ‘just work’. Installation These instructions are only for macOS. However, the app should compile fine on any Swift-supported platform if you see fit. Create a Backblaze B2 account and bucket.

Install homebrew if needed: brew.sh

Install flare: brew install chrishulbert/flare/flare

Configure it: flare configure

Schedule it to run hourly: flare schedule Musings Swift really won me over with it’s ability to model all the edge-cases of synchronisation in a way that wouldn’t compile unless I handled every one. A great example of this is the switch statement in ListingReconciliation.swift. This is how a good language fosters safe code, and I’m a huge fan. So: thanks, Chris Lattner. Thanks for reading, I hope this interests someone, and have a great week! Legals: I take no responsibility; give no guarantee/warranty for this project. Photo by Christopher Cambpell via Unsplash

Making a baby monitor out of a couple of ESP32s, an I2S microphone, and a small speaker 16 Apr, 2020 I’ve been struck by how many baby monitors have died over the span of our children. The things just seem to last about a year or two, and they cost a lot! So I’ve finally decided to make my own. Hardware I’m using a pair of ESP32 boards here, because they’re popular, common, and cheap. They have ESP-Now which allows you to communicate between them without Wi-Fi, an I2S audio input, and a DAC for audio output. You should be able to find them on ebay for under $10 each. For the microphone, I’m using an INMP441. It’s a 24-bit I2S non-PDM microphone which is, you guessed it, cheap. You should be able to find them for under $5 on ebay. The speaker I used is an 8Ω one I had lying around in a learn electronics kit. It’s very quiet. If you were doing this project for real, you should probably find a small amplifier board instead. Microphone wiring To connect the microphone to one of the ESP32s, I recommend reading about I2S on Wikipedia first. Here’s the connections: L/R (left/right) -> This goes to ground. According to the INMP441 datasheet this makes it the left channel.

WS (word select, aka LRCLK) -> This goes to pin D15 on the ESP32.

SCK (serial clock, aka BCLK/bit clock) -> D14.

SD (serial data, aka DOUT) -> D34.

VDD -> To +3.3V.

GND -> Ground. Speaker wiring To connect the speaker (or line-out) to the other ESP32: One wire to ground.

Another wire to D25 on the ESP32. Arduino I’m using Arduino to program the ESP32s, because it’s simpler than Espressif’s toolchain. Install Arduino, then follow these instructions to add ESP32 support. You’ll probably need to install drivers to suit whatever USB-UART bridge chip is on your boards. There should be two chips near the USB connector: one is power (not many pins), and one is the UART (lots of pins). Read the chip number, find the manufacturer’s page, and find and install the drivers. In my case, it was a Silabs CP2102. The Arduino settings that work for me are as follows, perhaps they will work for you: Tools > Board > ESP32 Wrover module

Tools > Port > /dev/cu.SLAB_USBtoUART

Tools > Programmer > ArduinoISP I find that I need to hold down the ‘boot’ button on my boards while programming them. ESP-Now setup You’ll need to find the MAC of the receiving ESP32. Here’s some Arduino code to do this: #include "WiFi.h" void setup() { Serial.begin(115200); WiFi.mode(WIFI_MODE_STA); Serial.println(WiFi.macAddress()); } void loop() {} Transmitter Here’s the code for the ESP32 which listens to the mic via the I2S input, converts to 8-bits, and transmits over ESP-Now: #include "WiFi.h" #include "esp_now.h" #include "driver/i2s.h" uint8_t receiverMAC[] = {0x11, 0x22, 0x33, 0x44, 0x55, 0x66}; <- Replace this with the MAC of your other board! void setup() { Serial.begin(115200); WiFi.mode(WIFI_MODE_STA); // Wifi (prerequisite for ESP-Now). // Setup ESP-Now first, because I2S uses it. Serial.println("Setup ESP-Now..."); if (ESP_OK != esp_now_init()) { Serial.println("esp_now_init: error"); return; } esp_now_peer_info_t peerInfo = {0}; memcpy(peerInfo.peer_addr, receiverMAC, sizeof(receiverMAC)); // TODO encrypt, by setting peerInfo.lmk. if (ESP_OK != esp_now_add_peer(&peerInfo)) { Serial.println("esp_now_add_peer: error"); return; } // I2S. Serial.println("Setup I2S..."); i2s_config_t i2s_config = { .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX), .sample_rate = 11025, .bits_per_sample = I2S_BITS_PER_SAMPLE_32BIT, // INMP441 is 24 bits, but it doesn't work if we set 24 bit here. .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT, .communication_format = i2s_comm_format_t(I2S_COMM_FORMAT_I2S | I2S_COMM_FORMAT_I2S_MSB), .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1, .dma_buf_count = 4, .dma_buf_len = ESP_NOW_MAX_DATA_LEN * 4, // * 4 for 32 bit. .use_apll = false, .tx_desc_auto_clear = false, .fixed_mclk = 0, }; if (ESP_OK != i2s_driver_install(I2S_NUM_0, &i2s_config, 0, NULL)) { Serial.println("i2s_driver_install: error"); } i2s_pin_config_t pin_config = { .bck_io_num = 14, // Bit Clock. .ws_io_num = 15, // Word Select. .data_out_num = -1, .data_in_num = 34, // Data-out of the mic. }; if (ESP_OK != i2s_set_pin(I2S_NUM_0, &pin_config)) { Serial.println("i2s_set_pin: error"); } i2s_zero_dma_buffer(I2S_NUM_0); Serial.println("Setup done."); } // This is used to scale the audio when things get loud, and gradually increase sensitivity when things go quiet. #define RESTING_SCALE 127 int32_t scale = RESTING_SCALE; void loop() { // Read from the DAC. This comes in as signed data with an extra byte. size_t bytesRead = 0; uint8_t buffer32[ESP_NOW_MAX_DATA_LEN * 4] = {0}; i2s_read(I2S_NUM_0, &buffer32, sizeof(buffer32), &bytesRead, 1000); int samplesRead = bytesRead / 4; // Convert to 16-bit signed. // It's actually 24-bit, but the lowest byte is just noise, even in a quiet room. // If we go to 16 bit we don't have to worry about extending a sign byte. // Quiet room seems to be values maxing around 7. // Max seems around 300 with me at 0.5m distance talking at normal loudness. int16_t buffer16[ESP_NOW_MAX_DATA_LEN] = {0}; for (int i=0; i<samplesRead; i++) { // Offset + 0 is always E0 or 00, regardless of the sign of the other bytes, // because our mic is only 24-bits, so discard it. // Offset + 1 is the LSB of the sample, but is just fuzz, discard it. uint8_t mid = buffer32[i * 4 + 2]; uint8_t msb = buffer32[i * 4 + 3]; uint16_t raw = (((uint32_t)msb) << 8) + ((uint32_t)mid); memcpy(&buffer16[i], &raw, sizeof(raw)); // Copy so sign bits aren't interfered. } // Find the maximum scale. int16_t max = 0; for (int i=0; i<samplesRead; i++) { int16_t val = buffer16[i]; if (val < 0) { val = -val; } if (val > max) { max = val; } } // Push up the scale if volume went up. if (max > scale) { scale = max; } // Gradually drop the scale when things are quiet. if (max < scale && scale > RESTING_SCALE) { scale -= 300; } if (scale < RESTING_SCALE) { scale = RESTING_SCALE; } // Dropped too far. // Scale it to int8s so we aren't transmitting too much data. int8_t buffer8[ESP_NOW_MAX_DATA_LEN] = {0}; for (int i=0; i<samplesRead; i++) { int32_t scaled = ((int32_t)buffer16[i]) * 127 / scale; if (scaled <= -127) { buffer8[i] = -127; } else if (scaled >= 127) { buffer8[i] = 127; } else { buffer8[i] = scaled; } } // Send to the other ESP32. if (ESP_OK != esp_now_send(NULL, (uint8_t *)buffer8, samplesRead)) { Serial.println("Error: esp_now_send"); delay(500); } } Receiver #include "WiFi.h" #include "esp_now.h" #include "driver/i2s.h" // Called when ESP-Now receives. void onDataRecv(const uint8_t *mac, const uint8_t *incomingRaw, int samples) { // Convert it from 8 bit signed to 16 bit unsigned with an 0x80 delta which is what the DAC requires. int8_t *incoming8 = (int8_t *)incomingRaw; uint16_t incoming16[ESP_NOW_MAX_DATA_LEN] = {0}; for (int i=0; i<samples; i++) { int32_t value = incoming8[i]; value += 0x80; // DAC wants unsigned values with a bias, not signed! incoming16[i] = value << 8; } // Forward it to the DAC. size_t bytesWritten=0; i2s_write(I2S_NUM_0, incoming16, samples * 2, &bytesWritten, 500); } void setup() { Serial.begin(115200); // Setup I2S first, because the ESP-Now listener uses it. i2s_config_t i2s_config = { .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_TX | I2S_MODE_DAC_BUILT_IN), .sample_rate = 11025 .bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT, .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT, .communication_format = I2S_COMM_FORMAT_I2S_MSB, .intr_alloc_flags = 0, .dma_buf_count = 4, .dma_buf_len = ESP_NOW_MAX_DATA_LEN * 2, .use_apll = false }; i2s_driver_install(I2S_NUM_0, &i2s_config, 0, NULL); i2s_zero_dma_buffer(I2S_NUM_0); i2s_set_pin(I2S_NUM_0, NULL); // ESP-Now. Serial.println("ESP-Now setup..."); WiFi.mode(WIFI_STA); if (esp_now_init() != ESP_OK) { Serial.println("Setup > ESP-Now error"); return; } esp_now_register_recv_cb(onDataRecv); Serial.println("Setup complete."); } void loop() {} Etc Things you might want to consider, because this project is far from perfect: Using an ESP-Now encryption key.

Making the ‘sender’ not transmit a packet if the sound level is quiet, which will likely extend the life of the radio circuitry.

Making the ‘receiver’ handle missed packets gracefully and treat them as quiet.

Some kind of amplifier, or I2S sound output from the receiver, for better sound quality.

Mounting it nicely! Thanks for reading, I hope this helps someone, and have a great week! Legals: I take no responsibility; give no guarantee/warranty for this project.

You can see older posts in the right panel, under 'archive'.