Barry Frost — Articles

App defaults in late 2023

Wed, 08 Nov 2023 00:00:00 GMT

Following the recent trend of posting your default apps, here are mine:

📨 Mail Client: Mimestream, Gmail app
📮 Mail Server: Google Workspace
📝 Notes: Reflect
✅ To-Do: Things, Reflect Tasks
🟦 Photo Management: iCloud Photos
📆 Calendar: iCloud Calendar
📁 Cloud File Storage: iCloud Drive
📖 RSS: Readwise Reader
🙍🏻‍♂️ Contacts: iCloud Contacts
🌐 Browser: Safari
💬 Chat: WhatsApp, Slack
🔖 Bookmarks: /bookmarks, Pinboard
📑 Read It Later: Readwise Reader
📜 Word Processing: Google Docs, Pages
📈 Spreadsheets: Google Sheets
📊 Presentations: Google Slides, Keynote
🛒 Shopping Lists: Notes (iCloud sharing)
📰 News: The Guardian, BBC News
🎵 Music: Spotify
🎤 Podcasts: Spotify
🔐 Password Management: 1Password

Creating a tunnel to localhost with Cloudflare

Sat, 11 Mar 2023 00:00:00 GMT

I found this handy tip via @wesbos. You can create a temporary tunnel to localhost on your machine using Cloudflare.

Even better, if you use Cloudflare for a domain’s DNS, you can set up a permanent, named tunnel with a subdomain. Here’s how it works:

$ cloudflared tunnel login
$ cloudflared tunnel create local 
$ cloudflared tunnel route dns local http://local.yourdomain.com
$ cloudflared tunnel run --url http://localhost:9999 local

I’ve previously used ngrok for free temporary URLs when developing against external APIs, but to create permanent tunnels was a paid feature. This is a nice, free alternative through a service I’m already using.

My serverless, headless, Micropub-powered, personal website

Tue, 13 Jul 2021 00:00:00 GMT

TL;DR This is my new personal IndieWeb website built using serverless AWS services, written in Node.js with the Architect framework. The backend is a Micropub server with a separate frontend that fetches posts using Micropub queries, rendering pages behind a CDN.

Look, I know it’s a developer cliché for the majority of posts on your blog to be about rewriting your own personal website software, but I find my website is the perfect place to try out new technology and then document what I’ve learned.

So, yes, this is a blog post about my new website. As well as a refreshed design, it’s completely different behind the scenes. The project is called Vibrancy. It’s massively over-engineered! But that’s the point: learn, have fun and enjoy slowly hacking away after the kids go to bed.

Goals

Instant updates. My website has over 10K posts and even the fastest static site generators take ~10 seconds to build and deploy so many files. I want the time between hitting create/update and the page refreshing to be instant.
Save money. My previous, low-traffic website cost $16/month on Heroku for a Hobby tier dyno, plus a PostgreSQL database with over 10K rows. I love Heroku, but that’s a bit much for my little website. I wanted my website to cost a few dollars on AWS, after the always free limits.
Use new (to me) technology:
- Serverless: My website doesn’t get much traffic and having an always-on server seemed wasteful. It seemed like a good use-case for trying small, stateless Node.js functions that can be called on-demand without reserving compute.
- AWS: While I’d used EC2 and S3 before, I wanted to experiment with other AWS services like Lambda, API Gateway, DynamoDB, SNS and CloudWatch that complement a serverless approach.
- Node.js: Ruby is a comfortable pair of shoes. I used it for previous versions of this website and it felt like time for a change. Node.js is a well-supported choice for Serverless.

Framework

I’m using Architect for both the backend and frontend apps. I spent some time prototyping with the Serverless Framework, but was left frustrated at its incomplete support for local development. This is where Architect shines:

Architect is an opinionated developer experience (DX) for building database backed web apps with AWS. We remove all the noise and friction to building serverlessly. We prioritize speed with fast local dev, smart configurable defaults and flexible Infrastructure as Code.

At its core is the app.arc manifest file and a file structure based around primitives for HTTP requests, events, queues, scheduled tasks, tables, static files and more. Each maps to an AWS service, for example tables corresponds to DynamoDB tables and queues to SQS. Architect then provides helpers to simplify working with each service. And by running arc deploy, it builds a SAM application that is magically deployed via CloudFormation to AWS.

Architect has been a breath of fresh air and reminds me of how natural Rails felt the first time I tried it. It deserves a proper article.

Backend

Vibrancy’s backend architecture, hosted using AWS.

Headless CMS

I like the concept of separating the management of my content from its display by adopting a headless CMS architecture. In theory, I‘m free to build multiple frontends that all use the same backend. There are plenty of excellent headless CMS options, but I built my own. Why? Well, I wanted to embrace the constraints of managing content exclusively via Micropub.

By implementing the Micropub server specification, Vibrancy has a mature, well-documented API for creating, updating, deleting, reading and querying posts. There is no admin system. Instead, using a Micropub client like Micropublish or Quill, I can log in (via IndieAuth) and manage my content.

Media endpoint

Vibrancy offers a Micropub media endpoint: a method to upload images and get a URL for use in posts. Cloudinary is used to upload, store and serve photos. Its ability to dynamically-resize images means I can upload one large file and then request different sizes on the fly, keeping frontend page sizes down. Images are then also uploaded to GitHub.

Background events

Using Architect’s event handlers, the server fires off asynchronous tasks in the background to avoid blocking page requests. Lambda functions are handled using SNS.

Syndication: if specified when I create a post, the backend will POSSE notes to Twitter and bookmarks to Pinboard.
Webmentions: when a post is created or updated, the server will send webmentions to any links using Telegraph.
Backup: all posts, webmentions and photos are also stored in a private git repo on GitHub.
Contexts: to display a snippet of the source of any bookmarks, RSVPs, replies, reposts or likes, the server uses Granary to fetch structured data in Microformats format and falls back to Open Graph metadata.
Push: the server uses Pushover to send me a push notification whenever a webmention is received.

Storage

DynamoDB: Content is indexed and stored in AWS’s DynamoDB. It’s a fast, low-latency database and its tables are a core primitive in Architect. I’m reasonably happy with it, but found the need to create extra tables a bit dirty when trying to combine filters with pagination. It was worth embracing its differences and learn its constraints to follow the Architect happy path.
GitHub: Content is also stored in a git repository on GitHub as a backup. If needed, I could regenerate the DynamoDB database from the repo. I’ve chosen to make the repo private because it’s possible some posts may be private or start off as drafts.
Cloudinary: As described above, photos are stored in Cloudinary and served from its CDN.

Frontend

Vibrancy’s frontend architecture, hosted using AWS and Cloudflare.

Micropub queries

The frontend website doesn’t have a database or any content files of its own. Instead, when a post is requested, a Micropub source query is made to the backend and the post is returned in Microformats 2 JSON format.

// Request for https://barryfrost.com/2021/07/a-post
GET https://api.barryfrost.com/micropub?q=source&url=https://barryfrost.com/2021/07/a-post
{
  "type": [
    "h-entry"
  ],
  "properties": {
    "published": [
      "2021-07-01T12:34:56Z"
    ],
    "content": [
      "This is my post. I use *Markdown* to mark up text."
    ]
  }
}

The frontend takes this JSON object and renders a page using Nunjucks templates. It also converts Markdown to HTML if needed.

Lists of posts are also fetched in the same way, but with additional parameters to filter results:

before takes an integer representing the epoch time of a post’s published timestamp. This method is used to paginate results.
limit is the number of posts to return in the response, defaulted to 20.

Like the backend, the frontend uses the Architect framework.

Style

This is the first project on which I’ve used Tailwind CSS. Initially it felt like heresy, but I quickly began to enjoy how fast it was to build solid, responsive layouts without needing to bounce back-and-forth between HTML and CSS files. Tailwind also provides a built-in dark mode with very little configuration.

I’m using inline SVGs for icons instead of an icon font to help further decrease page load times.

CDN

The frontend is not static. It queries and renders pages on demand. However, posts are cached and served using Cloudflare’s CDN (Content Delivery Network). I use very long s-maxage cache headers for posts which mean that requests are less likely to hit the frontend. If a post is updated Vibrancy sends a flush API request to Cloudflare.

Webmentions

Vibrancy fully supports webmentions for replies, reposts, likes or mentions from another IndieWeb website or service.

Receiving: Vibrancy currently uses webmention.io to receive webmentions, accepting a Microformats payload using a webhook. Next on my list is building my own receiver.
Sending: Webmentions are sent via Telegraph using a background event whenever a post is created or updated.
Backfeed: Using the magic of Bridgy, responses to my syndicated copies (e.g. tweets) are pulled back as webmentions.

Open source

Vibrancy’s source code is available on GitHub for both the backend and frontend using the MIT licence. Feel free to poke around, but I’m still actively developing and improving it and I wouldn’t recommend using Vibrancy for your own site just yet.

Building a webmention receiver

Sat, 15 May 2021 00:00:00 GMT

In today’s Webmentions Beyond Webmention.io online meet-up there was a request for people who were willing to implement their own receiver.

I’m interested. I’d like to build an endpoint that replaces what I currently use webmention.io for: receiving, storing and sending a webhook to my server with the parsed webmention.

It wouldn’t be a drop-in replacement. I’m also not sure I’m ready to host and run a service for others right now. But I think a receiver would be a useful building block and I’m willing to open-source it for self-hosting.

Let’s see how I get on.

Micropublish: IndieAuth updates and supported properties feature

Mon, 14 Dec 2020 00:00:00 GMT

Yesterday I pushed a new release of Micropublish to include recent updates for clients to the IndieAuth specification, as summarised in Aaron Parecki’s IndieAuth 2020 write-up.

The two biggest IndieAuth changes are the use of a OAuth 2.0 PKCE (Proof Key for Code Exchange) to secure authorisation and confirming the auth server if the original me value differs from that returned from the token endpoint. See the GitHub issue for more.

Additionally, Micropublish now has experimental support for queries for supported properties, for a supported post-type.

Thanks to Jamie Tanna and David Shanske for their help debugging and testing.

Micropublish v2.3.0

Mon, 12 Oct 2020 00:00:00 GMT

I’ve pushed a biggish update to Micropublish to bring it to v2.3.0. It includes some of the stable and experimental Micropub extensions discussed in the IndieWeb pop-up session I joined earlier this year.

Added

Changed

Make JSON the default post creation method
Bump kramdown from 2.1.0 to 2.3.0
Only show edit, delete or undelete controls if scope allows
Added draft scope to login form
Force post-status to draft when using (only) draft scope

Leaving Venntro

Tue, 03 Apr 2018 00:00:00 GMT

After ten happy, successful and highly educational years, I’ve decided to leave my role as Chief Technology Officer at Venntro this month. After so long with the company I felt the need to shake myself out of my comfort zone and look for a fresh challenge.

I’m very excited about joining a new company where I can use my experiences creating high performance development teams, building a large B2C subscription platform and leading technology for an award-winning and profitable fast-growth business.

If you’re looking for a CTO / VP of Engineering / Head of Development etc. in South-West London, North Surrey or along the M4 Corridor (or know of someone who is) I’d love to talk. I’m planning to first enjoy some time at home with our newborn, but I’ll be open to new opportunities from the end of May.

And of course, I wish founders Ross and Steve and the whole of Venntro continued success for the future. I’m confident the platform is in very good hands with Ian as Venntro’s new Head of Technology leading a great team of superb developers.

Making a looping lullaby box

Fri, 29 Dec 2017 00:00:00 GMT

My wife and I are expecting our second baby in the new year. Our first, now a toddler, loved his Ewan The Dream Sheep at night, especially its gentle harp lullaby music, but the short duration (20 minutes) meant we were frequently restarting the music to help him fully drift off.

I wanted a better solution this time around with a song that lasts for hours. All the off-the-shelf baby products I found were similarly limited. I looked through several cheap Bluetooth speakers, but I didn’t want to have to leave my phone connected all day/night long. The Anker SoundCore Mini, however, also accepts a microSD card so can be left to play independently. I ordered one from Amazon for a very reasonable £16.99 and a SanDisk 16Gb microSD card (with SD adapter to go in my Mac) for £6.50.

Next I needed a lullaby track lasting for at least 12 hours, thinking optimistically that the baby may sleep that long. Ignoring all the spammy-looking “free download MP3 baby lullaby” links on Google, I decided to make my own.

I first paid for and downloaded a gentle piano recording of Brahm’s Lullaby from iTunes on my Mac. I converted the M4A/AAC file to an MP3 in iTunes (File > Convert > Create MP3 Version) – the SoundCore doesn’t play AAC files – and then copied the MP3 file to my desktop as lullaby.mp3.

This created a single song lasting 150 seconds, but I needed this to loop. So I installed ffmpeg to process and convert the audio. I specified the LAME MP3 library to create a final MP3 file.

$ brew install ffmpeg --with-libmp3lame

The ~150s track played 300 times would result in approximately 12.5 hours of music. The following command specifies 300 loops, takes an input of the original lullaby and uses the LAME library at the same 190kbit/s quality of the original (qscale:a 2).

$ ffmpeg -stream_loop 300 -i lullaby.mp3 -codec:a libmp3lame -qscale:a 2 lullaby-loop.mp3

After a few minutes of processing I had a large (~1Gb) file ready to transfer via the SD card adapter to the microSD card and the SoundCore speaker. One extra I needed to add was a USB-A charger to power the speaker, but I had one spare lying around so that wasn’t an issue. It all works beautifully.

Now let’s see if this all pays off and the little monkey sleeps when they arrive!

How we make remote working work

Wed, 18 Oct 2017 00:00:00 GMT

Now that, according to a recent study, over half of employees work remotely at least once per week, remote working is no longer considered an exception. Companies that recognise the benefits and work hard to support effective remote working are rewarded with happy and productive people.

This post describes how we’ve structured our team and processes in the development team at Venntro. It follows a chat with our remote team (naturally, over Slack) about their experiences working remotely with us, and I’ve included their comments.

The 100% remote

We have six people in our development team who are 100% remote: four spread around the UK and two in New Zealand. Of the six, all but one began working with us in one of our offices first and then switched to remote later on. This early face-time with colleagues helped build strong relationships and absorb knowledge before switching to full-time remote.

The occasionally remote

It’s not just the fully-remote who work outside the office. Any of the regular office team can also opt to work from home. When someone is expecting a delivery, has a medical appointment or needs to take their car to be serviced, it’s fine to work remotely.

We also encourage anyone to work from home if they’re starting to feel a bit poorly and not well enough to travel, but not too sick to work. Not only does this help with recovery, it prevents the spread of early-stage germs to avoid others being infected.

How we make it work

Chat: Like many other development teams, we make heavy use of Slack. We chat in chatter, discuss work in team channels and share things we’ve found in channels like newbie and ux. It’s the glue that binds everyone together, whether co-located or remote.

Video: Almost as vital is Google Hangouts for video conversations. We use a form of Scrum to organise our teams: every stand-up, sprint planning session and retrospective involves a Hangout. The integration with Slack means anyone can kick off an ad-hoc video conversation from within a channel, alerting and bringing together the whole team. We sometimes also use Screenhero for screen sharing and collaboration.

Web tools: Just like our company’s product, the tools we use are web-based, so can be accessed from anywhere. Source code is hosted on GitHub and we use its Pull Requests to collaborate on features and code-review. Stories are created and prioritised in Jira, mock-ups are reviewed in InVision and our documentation written-up in Confluence. Google’s GSuite handles our company email, calendars and office documents.

Equipment: Almost everyone in the team uses a MacBook Pro with plenty of memory plugged in to one or more big displays. And this includes remote workers. We also provide external keyboards, trackpads, mice and any other peripherals to make their environment comfortable and productive.

Management: Line managers hold scheduled weekly 1:1s and regular appraisals with remote people over Hangouts, in addition to ad-hoc daily chats over Slack.

Team: For each and every team event we get together via a Hangout. Each Monday I run a 10-minute meeting updating the whole team on what’s happening. When someone sadly leaves the team we gather for presents and goodbyes. Even our Secret Santa presents are sent in advance and then opened synchronously with remotes!

Company: And don’t forget company events and celebrations. For anything involving the wider company we set up a wide-angle camera and broadcast over a Hangout. It’s just as important to include remote people to hear from the CEO and other senior management.

Face-to-face: Each month we invite the whole team (including remoters) to our head office in Windsor for our regular company meeting. There is often a charitable or social element to the day, e.g. a pub quiz or sports day, and so the team can mix with colleagues they wouldn’t generally interact. We also organise team workshops or tech talks on the same day which are broadcast/recorded for those unable to attend.

Remote benefits our people

A common theme in feedback from our remote people was about their better work-life balance, especially no longer requiring long commutes which “drained” them and meant their only downtime was at weekends.

While some people enjoy open-plan offices, others find working at home more productive: “I don’t miss loud phone calls and blaring radios from other departments” and “Fewer distractions and better focus for me.”

Being remote hasn’t damaged career prospects. Three of our team progressed to senior positions after going fully remote. Appraisals are still conducted and promotions awarded regardless of location.

Remote benefits our company

Companies that support remote work have 25% lower employee turnover than companies that don’t. We’ve retained several of our most experienced people through embracing remote. By insisting individuals travel to our offices I don’t believe that would have been possible.

We see fewer employee sick days: as one of our remoters commented, “My health has improved since going remote. Much less exposure to the communal germ pool.”

While it doesn’t exactly qualify as “following the sun”, our two NZ developers are online and available earlier in the day to help react to emergencies or catch up with overnight issues.

Challenges to overcome

Staying in the loop: Where possible, we try to kick off ad-hoc Hangouts to discuss issues with everyone in the team instead of an in-office conversation. We don’t always get it right when fire-fighting and this can be a frustration for those offsite.

Being ignored: Our remote people say it can also be frustrating to ping someone on Slack when it’s important and not get a response. “You can’t just go over and kick someone!” We encourage team-based public channels rather than private channels or direct messages so that others can also jump in to help.

Culture: With remote people, it’s important to work even harder on company and team culture. This can be tough, but by reinforcing the attitude to always consider remoters’ needs in everything you do, the feeling of remoteness can be reduced.

Trust: It’s clearly essential that people that work remotely are trusted to do so by their team and manager. Our approach has been to start off with a trial period working at home to allow both sides to feel comfortable and establish expectations early on.

Pros and cons: Inevitably some things can’t be replicated through tools or process: “I miss pub lunches and the pool table. Oh, and the coffee machine,” but, “I find the convenience of working from home makes up for the loss of in-office benefits.”

Tips and advice

When chatting to our remote people I finished by asking them for their own tips and advice for making a success of working remotely. Here are their suggestions:

Work out a routine ahead of time and stick to it. Don’t work near a TV. And you need a good chair. Also remember to take a lunch break away from the computer.

It’s important to have a quiet space free from distractions. Especially if you have children. And having the right equipment helps too (e.g. big monitor).

I think getting out of the house before or after work or at lunch time for some exercise is important.

I tend to take small breaks from the screen to grab a drink and talk to the guinea pigs. So maybe all remoters should have guinea pigs.

Using JSON in Postgres with Ruby and Sequel

Tue, 10 Oct 2017 00:00:00 GMT

With its fast, built-in support for JSON, it’s worth considering Postgres for storing and querying your JSON data. Instead of using a specialised document server, Postgres may be the right choice for your application.

As of Postgres 9.4 you can index data stored in a JSONB (binary JSON) column using GIN (“Generalised Inverted Index”) indexes. JSONB/GIN provides special operators to efficiently and rapidly query data.

And for Ruby developers, the Sequel gem offers a collection of convenient methods that make it easy to query Postgres JSONB columns.

Example usage

First, install the sequel_pg gem:

$ gem install sequel_pg

Next, create a table with a JSONB column and a GIN index and add some sample JSON data:

-- Create a table with a JSONB column
CREATE TABLE posts (permalink VARCHAR(255) PRIMARY KEY, data JSONB);

-- Add an index using GIN
CREATE INDEX posts_gin ON posts USING GIN(data);

-- Insert a row with a JSON document
INSERT INTO posts VALUES ('/2017/10/my-post', '{
    "title": "My post",
    "content": "This is a new post that I have created.",
    "category": [
        "ruby",
        "postgres"
    ],
    "published": "2017-10-10T16:05:10Z"
}');

And then in Ruby:

# Require the Sequel gem
require 'sequel'

# Include the Postgres JSON Operations extension
Sequel.extension(:pg_json_ops)

# Connect to your Postgres instance via your database URL
DB = Sequel.connect(DATABASE_URL)

# Ask Sequel to use the Postgres JSON extension with your database
DB.extension(:pg_json)

# Create a JSONB Operation object for the "data" jsonb column in our table
data = Sequel.pg_jsonb_op(:data)

# Find posts containing title "My post"
DB[:posts].where(data.get_text('title') => 'My post')

# Find posts where the category array has a "postgres" value
DB[:posts].where(data['category'].contains(['postgres']))

# Find posts where the first value in the category array is "ruby"
DB[:posts].where(data['category'].get_text(0) => 'ruby')

# Find posts which have a content key
DB[:posts].where(data.has_key?('content'))

# Find posts sorted by the date published
DB[:posts].order(data['published'])

Read the Sequel documentation for further methods supported by the pg_json_ops extension.

The right choice?

Postgres is a great solution for JSON in your application, especially if you are already using Postgres for structured data in other tables. It offers easy storage and, with GIN indexes and the Sequel gem, you get fast querying of data from Ruby.

Where it may not be the right choice is if your use-case requires frequent partial updates to JSON documents. You can of course retrieve the JSON blob, parse it, update the value in the hash, convert back to JSON and then update the column, but a dedicated document database like MongoDB may be a better option.