Last updated: May 2019 👉 livestreamed every last Sunday of the month. Join live or subscribe by email 💌

Ever wondered if the emails you send spark joy? You can ask!

About a year ago I started adding a little "Did you like this?" form at the bottom of emails sent to some 9,000 readers every week. The results have been wonderful ❤️

I now know what lands and what doesn't with my audience and it's made me a better writer. Here's an example where I wrote the same message in 2 different ways, sent to the same audience.

Two emails, same message, same audience, wildly different opinion results.



The power of writing 🤨 pic.twitter.com/XuqyrNjQRV — Swizec Teller (@Swizec) August 1, 2019

What a difference writing can make!

You know what makes data like this even better? A data visualization. With hearts and emojis and transitions and stuff!

So I fired up the monthly dataviz stream and built one 😛

An entire dataviz from scratch. Data collection and all. Doing one of these epic streams every last Sunday of the month.



Article coming soon pic.twitter.com/nZdWJPdyQt — Swizec Teller (@Swizec) July 30, 2019

Watch the stream here 👇

It's a little long. Just over 5 hours. You might want to fast-forward a few times, read this article instead. Think of it as a recap and full featured tutorial.

Next lastish-Sunday-of-the-month you can join live. It's great fun :)

[convertkit]

Here's how we approached this data visualization on the stream:

Collect data See what we find Design a visualization Build with React & D3

I'm not so good at design methodology so we're going to focus on building and data collection. Design happened through trial and error and a few ideas in my head.

You can try it out live, here

Full code on GitHub

Our data comes from 2 sources:

ConvertKit for subscribers, emails, open rates, etc. TypeForm for sentiment about each email sent

We never ended up using ConvertKit subscriber data so I'm not gonna talk about downloading and anonymizing that. You can see it in the stream.

ConvertKit and TypeForm APIs worked great for everything else.

ConvertKit calls the emails that you manually send to your subscribers broadcasts. There's no built-in export for broadcast data so we used the API.

Since there's no ConvertKit library I could find, we built our own following the docs. A few fetch() calls and some JavaScript glue code.

const fetch = require ( 'node-fetch' ) const { CK_KEY } = require ( './secrets.json' ) const fs = require ( 'fs' ) async function getBroadcasts ( ) { const page1 = await fetch ( `https://api.convertkit.com/v3/broadcasts?page=1&api_secret= ${ CK_KEY } ` ) . then ( res => res . json ( ) ) const page2 = await fetch ( `https://api.convertkit.com/v3/broadcasts?page=2&api_secret= ${ CK_KEY } ` ) . then ( res => res . json ( ) ) const broadcasts = [ ... page1 . broadcasts , ... page2 . broadcasts ] const result = [ ] for ( let broadcast of broadcasts ) { const stats = await fetch ( `https://api.convertkit.com/v3/broadcasts/ ${ broadcast . id } /stats?api_secret= ${ CK_KEY } ` ) . then ( res => res . json ( ) ) result . push ( { ... broadcast , ... stats . broadcast . stats , } ) } fs . writeFileSync ( 'public/data/broadcasts.json' , JSON . stringify ( result ) ) } getBroadcasts ( )

We make two API calls to get both pages of data. 50 results per page, just over 60 results in total. A real API wrapper would use some sort of loop here, but for a quick hack this is fine.

Then we take the list of broadcasts and fetch stats for each. API gives us the subject line, number of sends, opens, clicks, stuff like that.

We end up with a JSON file that contains all the email meta data we need for our visualization.

[ { "id" : 2005225 , "created_at" : "2019-01-14T18:17:04.000Z" , "subject" : "A bunch of cool things and neat little tips" , "recipients" : 10060 , "open_rate" : 22.24652087475149 , "click_rate" : 4.473161033797217 , "unsubscribes" : 32 , "total_clicks" : 993 , "show_total_clicks" : true , "status" : "completed" , "progress" : 100 } ,

TypeForm data is best scraped with their API. They support CSV exports but those work one by one. Manually going through all 60-some forms would take too long.

Scraping was pretty easy though – there's an official JavaScript API client :)

const { createClient } = require ( "@typeform/api-client" ) ; const fs = require ( "fs" ) ; const typeformAPI = createClient ( { token : < API token > } ) ;

Those few lines of code give us an API client. Documentation is a little weird and you have to guess some naming conventions from the actual API docs, but we made it work.

Fetching data happens in 3 steps:

Get list of workspaces, that's what TypeForm calls groups of forms Get forms from all workspaces Get responses to each form

async function scrapeData ( ) { const workspaces = await typeformAPI . workspaces . list ( { pageSize : 200 , } ) . then ( res => res . items . filter ( ( { name } ) => [ 'Post Emails' , 'Emails' ] . includes ( name ) ) ) const allForms = await Promise . all ( workspaces . map ( ( { id } ) => typeformAPI . forms . list ( { workspaceId : id , pageSize : 200 } ) . then ( forms => forms . items ) ) ) const forms = allForms . reduce ( ( acc , arr ) => [ ... acc , ... arr ] , [ ] ) . filter ( f => new Date ( f . last_updated_at ) > START_DATE ) const responses = await Promise . all ( forms . map ( form => typeformAPI . responses . list ( { pageSize : 200 , uid : form . id } ) . then ( res => ( { form : form . id , responses : res . items } ) ) ) ) fs . writeFileSync ( 'public/data/forms.json' , JSON . stringify ( forms ) ) fs . writeFileSync ( 'public/data/responses.json' , JSON . stringify ( responses ) ) }

A GraphQL API would make this much easier 😛

Again, this isn't the prettiest code but it's meant to run once so no need to make it perfect. If you wanted to maintain this long-term, I'd recommend breaking each step into its own function.

We end up with two JSON files containing all our sentiment data. The first question, "Did you like this?", is numeric and easy to interpret. The rest contain words so we won't use them for our dataviz ... altho it would be cool to figure something out.

Ok now we've got our data, time to fire up a new create-react-app, load the data, and start exploring.

$ create-react-app newsletter-dataviz $ cd newsletter-dataviz $ yarn add d3 react-use-dimensions styled-components

We can work with a basic CRA app, no special requirements. Couple of dependencies though:

d3 gives us simple data loading functions and helpers for calculating dataviz props

gives us simple data loading functions and helpers for calculating dataviz props react-use-dimensions or useDimension for short helps us make our dataviz responsive

or for short helps us make our dataviz responsive styled-components is my favorite way to use CSS in React apps

On the stream we did this part before scraping data so we had somewhere to install dev dependencies. 😇

We want to load our dataset asynchronously on component mount. Helps our app load fast, tell the user data is loading, and make sure all the data is ready before we start drawing.

D3 comes with helpers for loading both CSV and JSON data so we don't have to worry about parsing.

A custom useDataset hook helps us keep our code clean.

function useDataset ( ) { const [ broadcasts , setBroadcasts ] = useState ( [ ] ) useEffect ( ( ) => { ; ( async function ( ) { } ) ( ) } , [ ] ) return { broadcasts } }

The useDataset hook keeps one state variable: broadcasts . We're going to load all our data and combine it into a single data tree. Helps keep the rest of our code simple.

Loading happens in that useEffect , which runs our async function immediately on component mount.

function useDataset ( ) { const broadcasts = await d3 . json ( "data/broadcasts.json" ) . then ( data => data . map ( d => ( { ... d , created_at : new Date ( d . created_at ) } ) ) . filter ( d => d . recipients > 1000 ) . filter ( d => d . status === "completed" ) . sort ( ( a , b ) => a . created_at - b . created_at ) ) ;

Inside the effect we start with broadcasts data.

Use d3.json to make a fetch request and parse JSON data into a JavaScript object. .then we iterate through the data and:

change created_at strings into Date objects

strings into objects filter out any broadcasts smaller than 1000 recipients

out any broadcasts smaller than 1000 recipients filter out any incomplete broadcasts

out any incomplete broadcasts sort by created_at

Always a good idea to perform all your data cleanup on load. Makes your other code cleaner and you don't have to deal with strange edge cases.

function useDataset ( ) { let forms = await d3 . json ( "data/forms.json" ) ; const dateId = Object . fromEntries ( broadcasts . map ( d => [ dateFormat ( d . created_at ) , d . id ] ) ) ; forms = Object . fromEntries ( forms . map ( form => [ form . id , dateId [ dateFormat ( new Date ( form . last_updated_at ) ) ] ] ) ) ;

Then we load the forms data using d3.json again.

This time we want to associate each form with its respective email based on date. This approach works because I usually create the email and the form on the same day.

We make heavy use of the fromEntries method. It takes lists [key, value] pairs and turns them into key: value objects.

We end up with an object like this

{ dtnMgo : 2710510 , G72ihG : 2694018 , M6iSEQ : 2685890

Form id mapping to email id.

function useDataset ( ) { let responses = await d3 . json ( "data/responses.json" ) ; responses = responses . map ( row => ( { ... row , broadcast_id : forms [ row . form ] } ) ) . filter ( d => d . broadcast_id !== undefined ) ; setBroadcasts ( broadcasts . map ( d => ( { ... d , responses : responses . find ( r => r . broadcast_id === d . id ) } ) ) ) ;

Finally we load our sentiment data – responses.json .

Use d3.json to get all responses, add a broadcast_id to each based on the forms object, filter out anything with an undefined broadcast. Guess the "email and broadcast on the same day" rule isn't perfect. 🤷‍♂️

While saving data in local state with setBroadcasts , we also map through every entry and .find relevant responses. When we're done React re-renders our app.

Since we don't want users to stare at a blank screen while data loads, we create the simplest of loading screens.

function App ( ) { const { broadcasts } = useDataset ( ) ; if ( broadcasts . length < 1 ) { return < p > Loading data ... < / p > ; }

Fire up the useDataset hook, take broadcasts data out, see if there's anything yet. If there isn't render a Loading data ... text.

That is all ✌️

Since we're using a return, we'll have to make sure we add all hooks before this part of the function. Otherwise you fall into conditional rendering and hooks get confused. They have to be in the same order, always.

We render emails on a timeline with a combination of D3 scales and React rendering loops. Each 💌 emoji represents a single email. Its size shows the open rate.

Responsiveness comes from dynamically recalculating D3 scales based on the size of our SVG element with the useDimensions hook.

function App ( ) { const { broadcasts } = useDataset ( ) ; const [ ref , { width , height } ] = useDimensions ( ) ; const xScale = d3 . scaleTime ( ) . domain ( d3 . extent ( broadcasts , d => d . created_at ) ) . range ( [ 30 , width - 30 ] ) ; const sizeScale = d3 . scaleLinear ( ) . domain ( d3 . extent ( broadcasts , d => d . open_rate ) ) . range ( [ 2 , 25 ] ) ; return ( < svg ref = { ref } width = "99vw" height = "99vh" > { width && height && broadcasts . map ( ( d , i ) => ( < Broadcast key = { d . id } x = { xScale ( d . created_at ) } y = { height / 2 } size = { sizeScale ( d . open_rate ) } data = { d } / > ) ) } < / svg >

A couple steps going on here 👇

Get ref , width , and height , from useDimensions . The ref we'll use to specify what we're measuring. Width and height will update dynamically as the element's size changes on scroll or screen resize. xScale is a D3 scale that maps created_at dates from our dataset to pixel values between 30 and width-30 sizeScale maps open rates from our dataset to pixel values between 2 and 25 Render an <svg> element with the ref from useDimensions. Use width and height properties to make it full screen. When the browser resizes, this element will resize, useDimensions will pick up on that, update our width and height , trigger a re-render, and our dataviz becomes responsive 🤘 When all values are available .map through broadcast data and render a <Broadcast> component for each

The <Broadcast> component takes care of rendering and styling each letter emoji on our visualization. Later it's going to deal with dropping hearts as well.

We start with a <CenteredText> styled component.

const CenteredText = styled . text ` text-anchor: middle; dominant-baseline: central; `

Takes care of centering SVG text elements horizontally and vertically. Makes positioning much easier.

Right now the <Broadcast> component just renders that.

const Broadcast = ( { x , y , size , data } ) => { return ( < g transform = { `translate( ${ x } , ${ y } )` } style = { { cursor : 'pointer' } } > < CenteredText x = { 0 } y = { 0 } fontSize = { ` ${ size } pt` } > 💌 < / CenteredText > < / g > ) }

Render a grouping element, <g> , use an SVG transform to position at (x, y) coordinates, and render a <CenteredText> with a 💌 emoji using the size prop for font size.

The result is a responsive timeline.

Animating the timeline is a sort of trick 👉 change N of rendered emails over time and you get an animation.

We create a useRevealAnimation React hook to help us out.

function useRevealAnimation ( { duration , broadcasts } ) { const [ N , setN ] = useState ( 0 ) useEffect ( ( ) => { if ( broadcasts . length > 1 ) { d3 . selection ( ) . transition ( 'data-reveal' ) . duration ( duration * 1000 ) . tween ( 'Nvisible' , ( ) => { const interpolate = d3 . interpolate ( 0 , broadcasts . length ) return t => setN ( Math . round ( interpolate ( t ) ) ) } ) } } , [ broadcasts . length ] ) return N }

We've got a local state for N and a useEffect to start the animation. The effect starts a new D3 transition, sets up a custom tween with an interpolator from 0 to broadcasts.length and runs setN with a new number on every tick of the animation.

D3 handles the heavy lifting of figuring out exactly how to change N to create a nice smooth animation.

I teach this approach in more detail as hybrid animation in my React for DataViz course.

The useRevealAnimation hook goes in our App component like this 👇

function App ( ) { const { broadcasts } = useDataset ( ) ; const [ ref , { width , height } ] = useDimensions ( ) ; const N = useRevealAnimation ( { broadcasts , duration : 10 } ) ; { width && height && broadcasts . slice ( 0 , N ) . map ( ( d , i ) => ( < Broadcast

N updates as the animation runs and broadcasts.slice ensures we render only the first N elements of our data. React's diffing engine figures out the rest so existing items don't re-render.

This avoid-re-rendering part is very important to create a smooth animation of dropping hearts.

Each <Broadcast> handles its own dropping hearts.

const Broadcast = ( { x , y , size , data , onMouseOver } ) => { const responses = data . responses ? data . responses . responses : [ ] const hearts = responses . map ( r => ( r . answers ? r . answers . filter ( a => a . type === 'number' ) : [ ] ) ) . flat ( ) . filter ( ( { number } ) => number > 3 ) . length return ( < g transform = { `translate( ${ x } , ${ y } )` } onMouseOver = { onMouseOver } style = { { cursor : 'pointer' } } > < Hearts hearts = { hearts } bid = { data . id } height = { y - 10 } / > < / g > ) }

Get a list of responses out of data associated with each broadcast, flatten into a simple array, and filter out any votes below 3 on the 0, 1, 2, 3, 4, 5 scale. Assuming high numbers mean "I liked this".

Render with a <Hearts> component.

The <Hearts> component is a simple loop.

const Hearts = ( { bid , hearts , height } ) => { return ( < > { d3 . range ( 0 , hearts ) . map ( i => ( < Heart key = { i } index = { i } id = { ` ${ bid } - ${ i } ` } height = { height - i * 10 } dropDuration = { 3 } / > ) ) } < / > ) }

Create a counting array with d3.range , iterate over it, and render a <Heart> for each. The <Heart> component declaratively takes care of rendering itself so it drops into the right place.

const Heart = ( { index , height , id , dropDuration } ) => { const y = useDropAnimation ( { id , duration : dropDuration , height : height , delay : index * 100 + Math . random ( ) * 75 , } ) return ( < CenteredText x = { 0 } y = { y } fontSize = "12px" > ❤️ < / CenteredText > ) }

Look at that, another animation hook. Hooks really simplify our code 🥰

The animation hook gives us a y coordinate. When that changes, the component re-renders, and re-positions itself on the page.

That's because y is handled as a React state.

function useDropAnimation ( { duration , height , id , delay } ) { const [ y , sety ] = useState ( 0 ) useEffect ( ( ) => { d3 . selection ( ) . transition ( `drop-anim- ${ id } ` ) . ease ( d3 . easeCubicInOut ) . duration ( duration * 1000 ) . delay ( delay ) . tween ( `drop-tween- ${ id } ` , ( ) => { const interpolate = d3 . interpolate ( 0 , height ) return t => sety ( interpolate ( t ) ) } ) } , [ ] ) return y }

We're using the same hybrid animation trick as before except now we added an easing function to our D3 transition so it looks better.

The result are hearts dropping from an animated timeline.

[convertkit]

Last feature that makes our visualization useful are the titles. They create context and tell users what they're looking at.

No dataviz trickery here, just helpful info in text form :)

const Heading = styled . text ` font-size: 1.5em; font-weight: bold; text-anchor: middle; ` const MetaData = ( { broadcast , x } ) => { if ( ! broadcast ) return null return ( < > < Heading x = { x } y = { 50 } > { broadcast ? dateFormat ( broadcast . created_at ) : null } < / Heading > < Heading x = { x } y = { 75 } > { broadcast ? broadcast . subject : null } < / Heading > < text x = { x } y = { 100 } textAnchor = "middle" > ❤️ { heartRatio . toFixed ( 0 ) } % likes 📖 { broadcast . open_rate . toFixed ( 0 ) } % reads 👆 { broadcast . click_rate . toFixed ( 0 ) } % clicks 😢 { ' ' } { unsubRatio . toFixed ( 2 ) } % unsubs < / text > < / > ) }

We use some middle school maths to calculate the ratios we're showing, then render a <Heading> styled component twice and a <text> component once.

Headings show the email date and title, text shows meta info about open rates and such. Nothing fancy, but it makes the data visualization a lot better I think.

And so we end up with a nice dataviz full of hearts and emojis and transitions and animation. Great way to see which emails sparked joy 😍

Next step could be some sort of text analysis and figuring out which topics or words correlate to more enjoyment. Could be fun but I don't think we have a big enough dataset for proper sentiment analysis.

Maybe 🤔

Thanks for reading, ~Swizec

See a mistake? Suggest an edit