home

Julia Evans

http://jvns.ca/atom.xml
2020-04-02T14:26:31+00:00
https://jvns.ca/atom.xml


When debugging, your attitude matters

A while back I wrote What does debugging a program look like? on what to do when debugging (change one thing at a time! check your assumptions!).

But I was debugging some CSS last week, and I think that post is missing something important: your attitude.

Now – I’m not a very good CSS developer yet. I’ve never written CSS professionally and I don’t understand a lot of basic CSS concepts (I think I finally understood for the first time recently how position: absolute works). And last week I was working on the most complicated CSS project I’d ever attempted.

While I was debugging my CSS, I noticed myself doing some bad things that I normally would not! I was:

All of this was exactly as effective as you might imagine (not very effective!), and it was because of my attitude about CSS! I had this unusual-for-me belief that CSS was Too Hard and impossible for me to understand. So let’s talk about that attitude a bit!

the problem attitude: “this is too hard for me to understand”

One specific problem I was having was – I had 2 divs stacked on top of one another, and I wanted Div A to be on top of Div B. My model of CSS stacking order at the start of this was basically “if you want Thing A to be on top of Thing B, change the z-index to make it work”. So I changed the z-index of Div A to be 5 or something.

But it didn’t work! In Firefox, div A was on top, but in Chrome, Div B was on top. Argh! Why? CSS is impossible!!!

I googled a bit, and I found out that a possible reason z-index might not work was because Div A and Div B were actually in different “stacking contexts”. If that was true, even if I set the z-index of Div A to 999999 it would still not put it on top of Div B.

I thought “man, this stacking context thing seems really complicated, why is it different between Firefox and Chrome, I’m not going to be able to figure this out”. So I tried a bunch of random things a bunch of blog posts suggested, which as usual did not work.

Finally I gave up this “change random things and pray” strategy and thought “well, what if I just read the documentation on stacking order, maybe it’s not that bad”.

So I read the MDN page on stacking order, which says:

When the z-index property is not specified on any element, elements are stacked in the following order (from bottom to top):
1. The background and borders of the root element
2. Descendant non-positioned blocks, in order of appearance in the HTML
3. Descendant positioned elements, in order of appearance in the HTML

This is SO SIMPLE! It just depends on the order in the HTML! I put Div A after Div B in the HTML (as a sibling) and it made everything work in both browsers.

better attitude: “let’s learn the basics and see if that helps”

This whole stacking problem turned out to really not be that complicated – all I needed to do was read a very short and simple documentation page to understand how stacking works!

Of course, computer things are not always this simple (and even in this specific case the rules about what creates a new stacking context are pretty complicated.). But I did not need to understand those more complicated rules in order to put Div A on top of Div B! I only needed to know the much simpler 3 rules above.

So – calm down for a second, learn a few of the basics, and see if that helps.

watching people who know what they’re doing is inspiring

Another area of CSS that I thought was “too hard” for me to understand was this whole position: absolute and position: relative business. I kept seeing (and sometimes using!) examples where people made complicated CSS things with position: absolute but I didn’t understand how they worked. Doesn’t position: absolute mean that the element is always in the same place on the screen? Why are these position: absolute things moving when I scroll like the rest of the document? (spoiler: no, that’s position: fixed.)

But last week, I paired with someone who’s a lot better at CSS than me on some code, and I saw that they were just typing in position: absolute and position: relative confidently into their code without seeming confused about it!! Could that be me?

I looked up the documentation on MDN on position: absolute, and it said:

The element is removed from the normal document flow, and no space is created for the element in the page layout. It is positioned relative to its closest positioned ancestor… Its final position is determined by the values of top, right, bottom, and left.

So things with position: absolute are positioned relative to their closest positioned ancestor! And you just use top/bottom/right/left to pick where! That’s so simple!

that’s all!

I don’t really know why I started to believe that it was “impossible” to understand basic CSS concepts since I don’t believe that about computers in general. Maybe because I’d never really tried to do a more involved CSS project than “let’s arrange some divs in a grid with flexbox”!

But this attitude really got in the way of me writing the CSS I wanted to write! And once I let go of it and used my normal debugging techniques I was able to get a lot more things to work the way I wanted.

2020-04-02T14:26:31+00:00


Getting started with shaders: signed distance functions!

Hello! A while back I learned how to make fun shiny spinny things like this using shaders:

My shader skills are still extremely basic, but this fun spinning thing turned out to be a lot easier to make than I thought it would be to make (with a lot of copying of code snippets from other people!).

The big idea I learned when doing this was something called “signed distance functions”, which I learned about from a very fun tutorial called Signed Distance Function tutorial: box & balloon.

In this post I’ll go through the steps I used to learn to write a simple shader and try to convince you that shaders are not that hard to get started with!

examples of more advanced shaders

If you haven’t seen people do really fancy things with shaders, here are a couple:

  1. this very complicated shader that is like a realistic video of a river: https://www.shadertoy.com/view/Xl2XRW
  2. a more abstract (and shorter!) fun shader with a lot of glowing circles: https://www.shadertoy.com/view/lstSzj

step 1: my first shader

I knew that you could make shaders on shadertoy, and so I went to https://www.shadertoy.com/new. They give you a default shader to start with that looks like this:

Here’s the code:

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
    // Normalized pixel coordinates (from 0 to 1)
    vec2 uv = fragCoord/iResolution.xy;

    // Time varying pixel color
    vec3 col = 0.5 + 0.5*cos(iTime+uv.xyx+vec3(0,2,4));

    // Output to screen
    fragColor = vec4(col,1.0);
}

This doesn’t do anythign that exciting, but it already taught me the basic structure of a shader program!

the idea: map a pair of coordinates (and time) to a colour

The idea here is that you get a pair of coordinates as an input (fragCoord) and you need to output a RGBA vector with the colour of that. The function can also use the current time (iTime), which is how the picture changes over time.

The neat thing about this programming model (where you map a pair of coordinates and the time to) is that it’s extremely trivially parallelizable. I don’t understand a lot about GPUs but my understanding is that this kind of task (where you have 10000 trivially parallelizable calculations to do at once) is exactly the kind of thing GPUs are good at.

step 2: iterate faster with shadertoy-render

After a while of playing with shadertoy, I got tired of having to click “recompile” on the Shadertoy website every time I saved my shader.

I found a command line tool that will watch a file and update the animation in real time every time I save called shadertoy-render. So now I can just run:

shadertoy-render.py circle.glsl 

and iterate way faster!

step 3: draw a circle

Next I thought – I’m good at math! I can use some basic trigonometry to draw a bouncing rainbow circle!

I know the equation for a circle (x**2 + y**2 = whatever!), so I wrote some code to do that:

Here’s the code: (which you can also see on shadertoy)

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
    // Normalized pixel coordinates (from 0 to 1)
    vec2 uv = fragCoord/iResolution.xy;
    // Draw a circle whose center depends on what time it is
    vec2 shifted = uv - vec2((sin(iGlobalTime) + 1)/2, (1 + cos(iGlobalTime)) / 2);
    if (dot(shifted, shifted) < 0.03) {
        // Varying pixel colour
        vec3 col = 0.5 + 0.5*cos(iGlobalTime+uv.xyx+vec3(0,2,4));
        fragColor = vec4(col,1.0);
    } else {
        // make everything outside the circle black
        fragColor = vec4(0,0,0,1.0);
    }
}

This takes the dot product of the coordinate vector fragCoord with itself, which is the same as calculating x^2 + y^2. I played with the center of the circle a little bit in this one too – I made the center vec2((sin(iGlobalTime) + 1)/2, (1 + cos(faster)) / 2), which means that the center of the circle also goes in a circle depending on what time it is.

shaders are a fun way to play with math!

One thing I think is fun about this already (even though we haven’t done anything super advanced!) is that these shaders give us a fun visual way to play with math – I used sin and cos to make something go in a circle, and if you want to get some better intuition about how trigonometric work, maybe writing shaders would be a fun way to do that!

I love that you get instant visual feedback about your math code – if you multiply something by 2, things get bigger! or smaller! or faster! or slower! or more red!

but how do we do something really fancy?

This bouncing circle is nice but it’s really far from the super fancy things I’ve seen other people do with shaders. So what’s the next step?

idea: instead of using if statements, use signed distance functions!

In my circle code above, I basically wrote:

if (dot(uv, uv) < 0.03) {
    // code for inside the circle
} else {
    // code for outside the circle
}

But the problem with this (and the reason I was feeling stuck) is that it’s not clear how it generalizes to more complicated shapes! Writing a bajillion if statements doesn’t seem like it would work well. And how do people render those 3d shapes anyway?

So! Signed distance functions are a different way to define a shape. Instead of using a hardcoded if statement, instead you define a function that tells you, for any point in the world, how far away that point is from your shape. For example, here’s a signed distance function for a sphere.

float sdSphere( vec3 p, float center )
{
  return length(p)-center;
}

Signed distance functions are awesome because they’re:

the steps to making a spinning top

When I started out I didn’t understand what code I needed to write to make a shiny spinning thing. It turns out that these are the basic steps:

  1. Make a signed distance function for the shape I want (in my case an octahedron)
  2. Raytrace the signed distance function so you can display it in a 2D picture (or raymarch? The tutorial I used called it raytracing and I don’t understand the difference between raytracing and raymarching yet)
  3. Write some code to texture the surface of your shape and make it shiny

I’m not going to explain signed distance functions or raytracing in detail in this post because I found this AMAZING tutorial on signed distance functions that is very friendly and honestly it does a way better job than I could do. It explains how to do the 3 steps above and the code has a ton of comments and it’s great.

step 4: copy the tutorial code and start changing things

Here I used the time honoured programming practice here of “copy the code and change things in a chaotic way until I get the result I want”.

My final shader of a bunch of shiny spinny things is here: https://www.shadertoy.com/view/wdlcR4

The animation comes out looking like this:

Basically to make this I just copied the tutorial on signed distance functions that renders the shape based on the signed distance function and:

making the octahedron spin!

Here’s some the I used to make the octahedron spin! This turned out to be really simple: first copied an octahedron signed distance function from this page and then added a rotate to make it rotate based on time and then suddenly it’s spinning!

vec2 sdfOctahedron( vec3 currentRayPosition, vec3 offset ){
    vec3 p = rotate((currentRayPosition), offset.xy, iTime * 3.0) - offset;
    float s = 0.1; // what is s?
    p = abs(p);
    float distance = (p.x+p.y+p.z-s)*0.57735027;
    float id = 1.0;
    return vec2( distance,  id );
}

making it shiny with some noise

The other thing I wanted to do was to make my shape look sparkly/shiny. I used a noise funciton that I found in this github gist to make the surface look textured.

Here’s how I used the noise function. Basically I just changed parameters to the noise function mostly at random (multiply by 2? 3? 1800? who knows!) until I got an effect I liked.

float x = noise(rotate(positionOfHit, vec2(0, 0), iGlobalTime * 3.0).xy * 1800.0);
float x2 = noise(lightDirection.xy * 400.0);
float y = min(max(x, 0.0), 1.0);
float y2 = min(max(x2, 0.0), 1.0) ;
vec3 balloonColor = vec3(y , y  + y2, y  + y2);

writing shaders is fun!

That’s all! I had a lot of fun making this thing spin and be shiny. If you also want to make fun animations with shaders, I hope this helps you make your cool thing!

As usual with subjects I don’t know tha well, I’ve probably said at least one wrong thing about shaders in this post, let me know what it is!

Again, here are the 2 resources I used:

  1. “SDF Tutorial: box & balloon”: https://www.shadertoy.com/view/Xl2XWt (which is really fun to modify and play around with)
  2. Tons of signed distance functions that you can copy and paste into your code http://www.iquilezles.org/www/articles/distfunctions/distfunctions.htm

2020-03-15T13:08:06+00:00


Questions you can ask about compensation

Talking about pay is hard, and a lot of the time it feels like it boils down to “hello I would like more money please?”. But it’s totally possible to have a conversation about compensation without asking for more money at all!

When trying to understand (and let’s be honest – increase!) my pay, I’ve found it really useful to first understand the processes around compensation at the company I work for. Here are some questions you can ask. Your manager can probably answer many of these, but your colleagues might know too!

  1. Who makes decisions about raises? (is it at the discretion of the manager? Does the manager have a fixed budget they can give out? Is there a formula based on past performance evaluations?)
  2. When do we adjust salaries? (on the employee’s work anniversary? Right after a performance review?)
  3. Do we do market adjustments to give people raises if the industry salary for this job increases? What’s the process for that?
  4. Is there a salary range for my level? What is it approximately? (Also same question for total compensation and not just salary)
  5. Does the company actually stick to its salary ranges or does it often make exceptions? What’s the process for getting paid higher than the range? Who can decide to make an exception?
  6. Which other companies are we trying to be competitive with when we make job offers?
  7. How is compensation split between salary/equity/bonus? (at higher levels, will my pay be mostly equity? what do we aim for with bonuses?)
  8. Is it possible to get more vacation? (at this company, do you get more vacation after X years?)
  9. When are equity refreshes given? (do we give refreshes yearly? Only when someone’s initial stock grant is about to expire?)
  10. Who makes decisions about equity refreshes and how? (are they based on level? Performance? Who decides?)
  11. When do my stock options expire? (this one you should definitely have been told, but if your company has stock options set up like “they expire 3 months after you leave”, it’s possible for them to change their policy)
  12. Is on-call time compensated? How?
  13. How do bonuses work exactly? (is it tied to company performance? Individual performance? Level? All of the above? Are bonuses targeted to be a percentage of salary?)
  14. Is there a peer bonus system? (can people recommend their coworkers for cash bonuses?)
  15. Is there a learning budget? (for conferences / books / training?)
  16. Is it possible to take unpaid time off?

If you’re negotiating a job offer it can also be useful to ask about signing/relocation bonus and details about the stock options.

This is probably too many questions to ask all at once, and your manager may not even know the answers to all of these questions themselves. That’s okay! I definitely didn’t know the answers to all of these at my last job, but knowing even some of these answers is really helpful.

company policies can vary a lot

The reason this blog post is “questions to ask” and not “how compensation works” is that different companies have VERY different compensation policies. At some companies you can ask for a raise and just get it if you make a good case, and at other companies there are very strict rules about the salary bands for each level. And everything in between, and then apply that to every axis of compensation (salary, bonuses, equity, paid vacation, benefits). Regardless of what the “best” compensation policies are, it’s good to know what situation you’re in exactly.

And be careful of assuming you know the answers already!

why this is useful

If you know when and how decisions about compensation are made, it’s easier to figure out where to apply pressure, either individually (by making a case for yourself) or through collective action (by making specific demands as a group for something to be changed).

2020-02-18T20:46:39+00:00


New zine: Become a SELECT Star!

On Friday I published a zine about SQL called “Become a SELECT Star!”

You can get it for $12 at https://wizardzines.com/zines/sql. If you buy it, you’ll get a PDF that you can either read on your computer or print out. You can also get a pack of all 7 zines so far.

Here’s the cover and table of contents:

why SQL?

I got excited about writing a zine about SQL because at my old job I wrote a ton of SQL queries (mostly related to machine learning) and by doing that I learned there are a lot of weird things about SQL! For example – SQL queries don’t actually start with SELECT. And the way NULL behaves isn’t really intuitive at first.

It’s been really fun to go back and try to explain the basics of SQL from the beginning. (what’s the difference between WHERE and HAVING? what’s the basic idea with indexes actually? how do you write a join?)

I think SQL is a really nice thing to know because there are SO MANY SQL databases out there, and some of them are super powerful! (like BigQuery and Redshift). So if you know SQL and have access to one of these big data warehouses you can write queries that crunch like 10 billion rows of data really quickly.

lots of examples

I ended up spending a lot of time on the examples in this zine, more than in any previous zine. My friend Anton helped me come up with a fun way to illustrate them, where you can clearly see the query, the table it’s running on, and what the query outputs. Like this:

experiment: include a SQL playground

All the examples in the zine are real queries that you can run. So I thought: why not provide a simple environment where people can actually run those queries (and variations on those queries) to try things out?

So I built a small playground where you can run queries on the example tables in the zine. It uses SQLite compiled to web assembly, so all the queries run in your browser. It wasn’t too complicated to build – I just used my minimal Javascript/CSS skills and vue.js.

I’d love to hear any feedback about whether this is helpful or not – the example tables in the zine are really small (you can only print out small SQL tables!), so the biggest table in the example set has 9 rows or something.

what’s next: probably containers

I think that next up is going to be a zine on containers, which is more of a normal systems-y topic for me. (for example: namespaces, cgroups, why containers?)



Here’s a link to where to get the zine again :)

2020-02-03T22:38:43+00:00


PaperWM: tiled window management for GNOME

When I started using Linux on my personal computer, one of the first things I got excited about was tiny lightweight window managers, largely because my laptop at the time had 32MB of RAM and anything else was unusable.

Then I got into tiling window managers like xmonad! I could manage my windows with my keyboard! They were so fast! I could configure xmonad by writing a Haskell program! I could customize everything in all kinds of fun ways (like using dmenu as a launcher)! I used 3 or 4 different tiling window managers over the years and it was fun.

About 6 years ago I decided configuring my tiling window manager wasn’t fun for me anymore and switched to using the Ubuntu stock desktop environment: Gnome. (which is much faster now that I have 500x more RAM in my laptop :) )

So I’ve been using Gnome for a long time, but I still kind of missed tiling window managers. Then 6 months ago a friend told me about PaperWM, which lets you tile your windows in Gnome! I installed it immediately and I’ve been using it ever since.

PaperWM: tiling window management for Gnome

The basic idea of PaperWM is: you want to keep using Gnome (because all kinds of things Just Work in Gnome) but you also kinda wish you were using a tiling window manager.

It’s a Gnome extension (instead of being a standalone window manager) and it’s in Javascript.

“Paper” means all of your windows are in a line

The main idea in PaperWM is it puts all your windows in a line, which is actually quite different from traditional tiling window managers where you can tile your windows any way you want. Here’s a gif of me moving between / resizing some windows while writing this blog post (there’s a browser and two terminal windows):

PaperWM’s Github README links to this video: http://10gui.com/video/, which describes a similar system as a “linear window manager”.

I’d never heard of this way of organizing windows before but I like the simplicity of it – if I’m looking for a specific window I just move left/right until I find it.

everything I do in PaperWM

there are lots of other features but these are the only ones I use:

I like tools that I don’t have to configure

I’ve been using PaperWM for 6 months on a laptop and I really like it! I also really appreciate that even though it’s configurable (by writing a Javascript configuration file), it does the things I want out of the box without me having to research how to configure it.

The fish shell is another delightful tool like that – I basically don’t configure fish at all (except to set environment variables etc) and I really like the default feature set.

2020-01-05T22:33:27+00:00


2019: Year in review

It’s the end of the year again! Here are a few things that happened in 2019. I wrote these in 2015, 2016, 2017, and 2018 too.

I have a business instead of a job!

The biggest change this year is that I left my job in August after working there for 5.5 years and now I don’t have a job! Now I have a business (wizard zines).

This has been exciting (I can do anything I want with my time! No rules! Wow!) and also disorienting (I can do anything I… want? Wait, what do I want to do exactly?). Obviously this is a good problem to have but it’s a big adjustment from the structure I had when I had a job.

My plan for now is to give myself a year (until August 2020) to see how this new way of existing goes and then reevaluate.

I wanted to write some reflections on my 5 years at Stripe here but it’s been such a huge part of my life for so long that I couldn’t figure out how to summarize it. I was in a much worse place in my career 6 years ago before I started working there and it really changed everything for me.

!!Con

2019 was !!Con’s 6th year! It’s a conference about the joy, excitement, and surprise of programming. And !!Con also expanded to the west coast!! I wasn’t part of organizing the west coast conference at all but I got to attend and it was wonderful.

Running a conference is a ton of work and I feel really lucky to get to do it with such great co-organizers – there have been at least 20 people involved in organizing over the years and I only do a small part (right now I organize sponsorships for the east coast conference).

This year we also incorporated the Exclamation Foundation which is the official entity which runs both conferences which is going to make organizing money things a lot easier.

I understand how the business works a little better

Earlier this year I signed up for a business course called 30x500 by Amy Hoy and Alex Hillman. They’ve influenced me a lot this year. Basically I signed up for it because I had a business that had made $100,000 in revenue already but I didn’t really understand how the business worked and it felt like it could just evaporate at any point. So $2000 (the cost at the time of 30x500) was worth it to help me understand what was going on.

Amy and Alex both just the other day wrote 100-tweet threads that have some of the ideas that I learned this year in them: Alex on creating sustainable businesses and Amy on design.

I was hoping to build a system for selling printed zines in 2019 and I didn’t get to it – that’s probably my one concrete business goal for 2020. I tried out Lulu for printing in the hopes that I could experiment with print-on-demand but the quality was awful so it’s going to be a bit more work.

blog posts and things

In 2019 I:

The blog post I’m happiest to have published this year is definitely Get your work recognized: write a brag document. I’ve seen quite a few people saying that it helped them track their work and it makes me really happy. A bunch of people at my old job adopted it and it’s one of the non-engineering projects I’m most proud of having done there.

Publishing this post about my business revenue was also important to me – in the past I loved blogging, but I didn’t think it was possible to make a living by explaining computer things online. And I was totally wrong! It is possible! So I think it’s important to tell other people that it’s a possibility.

zines

I published 2 zines: Bite Size Networking and HTTP: Learn Your Browser’s Language. And wrote most of a third zine about SQL which should be out in January.

I made the same business revenue as in 2018 (which I was thrilled about).

published a box set of my free zines

In August I published a box set of all my free zines with No Starch Press (Your Linux Toolbox, it’s in Real Physical Bookstores!!) They did a fantastic job printing it: the quality is really really good. I’m very happy with how it turned out. (and if you do buy it and like it, leaving an amazon review helps me a lot).

And No Starch just told me last week they’ve sold 4000 copies so far and are looking to do a second printing!

Having a Real Traditionally Published Thing out is really cool, I could not have imagined 4 years ago that I could go to an actual bookstore and buy the little 16-page zine I wrote about how much I love strace.

The business aspect of it is interesting – because I’m so used to running a business where I sell my own zines, getting 10% in royalties instead of 100% feels strange. But printing and distribution are complicated! And it’s really cool that I can say “yeah, go to Barnes & Noble, they’ll have it”! And No Starch helped me a lot with picking a good title and cover art! And basically the whole traditional publishing ecosystem just works in a completely different way from what I’m used to :)

I think I’ll have a better sense for how to think about traditional publishing from a business perspective in a year or so after the book has been out for longer.

A big thing I learned from this project is that having zines that are printed in a higher quality way (not just on a home printer) is really nice.

what went well

some things that were good this year:

some things that are harder:

2019-12-27T21:21:43+00:00


"server" is hard to define

Somebody asked me recently what a server was, and I had a harder time explaining it than I expected! I thought I was going to be able to give some kind of simple pithy answer but it kind of got away from me. So here’s an short exploration of what the word “server” can mean:

a server responds to requests

A server definitely responds to requests. A few examples:

webserver:

Me: "please give me google.com" 
Server: "here is the HTML for that webpage"

bittorrent server:

Me: "I would like this chunk of the good wife season 2"
Server: "here are some of the  bytes from that .avi file!"

mail server:

Me: "can you send this email to julia@jvns.ca"
Server: "I sent it!"

But what is a server actually specifically exactly?

a server is a program

My first instinct is to say “a server is a program” because for example a “the wordpress server” is a PHP program, so let’s start with that.

A server is usually a program that listens on a port (like 80). For example, if we’re talking about a Rails webserver, then the program is a Ruby program that’s listening on a port for HTTP requests.

For example, we can start a Python server to serve files out of the current directory.

$ python3 -m http.server & 
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ..

and send requests to it with curl:

$ curl localhost:8000/config.yaml
baseurl: https://jvns.ca
disablePathToLower: true
languageCode: en-us
title: Julia Evans
author: Julia Evans
...

a server might be a virtual machine

But often when I talk about “a server” at work, I’ll use it in a sentence like “I’m going to SSH to that server to see what’s going on with it”, or “wow, that server is swapping a lot, that’s bad!“.

So in those cases clearly I don’t mean a program when I say “that server” (you can’t ssh to a program, though the ssh server that runs on the VM is itself a program!), I mean the AWS instance that the server program is running on. That AWS instance is a virtual machine, which looks like a computer in a lot of ways (it’s running an operating system!) but it isn’t a physical computer.

a server might be a container

Similarly to how your server might be a virtual machine, it could also be a container running in a virtual machine. So “the server is running out of memory” could mean “the container is running out of memory and crashing” which really means “we set a cgroup memory limit on this container and the programs in the container with that cgroup exceeded the limit so the Linux kernel OOM killed them”.

But containers make everything a lot more complicated so I think we should stop there for now.

a server is a computer

But also when you buy a server from Dell or some other computer company, you’re not buying a virtual machine, you’re buying an actual physical machine.

Usually these computers are in building datacenters. For example in this video you can see thousands of servers in a Google datacenter.

The computers in this datacenter don’t look like the computers in my house! They’re short and wide because they’re designed to fit into these giant racks of servers. For example if you search Newegg for 1U server you’ll find servers that are 1 “rack unit” high, and a rack unit is 1.75 inches. There are also 2U servers which are twice as high.

Here’s a picture of a 1U server I found on Newegg:

I’ve only seen a server rack once at the Internet Archive which is in what used to be a church in San Francisco, and it was really cool to realize – wow, when I use the Wayback Machine it’s using the actual computers in this room!

“the server” might be 1000 computers

Next, let’s say we’re talking about how Gmail works. You might ask “hey, when I search my email to find my boarding pass, does that happen in the frontend or on the server?”.

The answer is “it happens on the server”, but what’s “the server” here? There’s not just one computer or program or virtual machine that searches your Gmail, there are probably lots of computers and programs at Google that are reponsible for that and they’re probably distributed across many datacenters all over the world.

And even if we’re just talking about doing 1 search, there could easily be 20 different computers in 3 different countries involved in just running that 1 search.

So the words “the server” in “oh yeah, that happens on the server” mean something kind of complicated here – what you’re actually saying is something “well the browser makes a request, and that request does something, but I’m not really going to worry about what because the important thing is just that the browser made a request and got some kind of response back.”

what happens when I search my email for a boarding pass?

When I search for “boarding” in my email, the Javascript running on the frontend puts together this request. It’s mostly indecipherable but it definitely contains the word “boarding”:

{
  "1": {
    "1": 79,
    "2": 101,
    "4": "boarding",
    "5": {
      "5": 0,
      "12": "1577376926313",
      "13": -18000000
    },
    "6": "itemlist-ViewType(79)-5",
    "7": 1,
    "8": 2000,
    "10": 0,
    "14": 1,
    "16": {
      "1": 1,
      "2": 0,
      "3": 0,
      "7": 1
    },
    "19": 1
  },
  "3": {
    "1": "0",
    "2": 5,
    "5": 1,
    "6": 1,
    "7": 1
  }
}

We get a response back which is large and complicated and definitely contains search results from my email about boarding passes. Here’s an excerpt:

"your electronic boarding pass. You could also be asked to display this \nmessage to airport security. * PLEASE NOTE: A printable",
"the attached boarding pass to present at the airport. Manage your booking \nBooking Details Passenger: JULIA EVANS Booking",
"Electronic boarding pass is not offered for your flight. Click the link \nbelow to access the PRINTABLE VERSION of your boarding",
"Save time at the airport Save time at the airport Web version",
"GET YOUR BOARDING PASS IN ADVANCE > You can now check in for your flight \nand you will receive a boarding pass > allowing",
"Save time at the airport Save time at the airport Web version",
"Booking Confirmation Booking Reference: xxxxxx Date of issue: xxxxxxxxxxxx \nSelect Seats eUpgrade",
"your electronic boarding pass. You could also be asked to display this \nmessage to airport security. * PLEASE NOTE: A printable",
"your electronic boarding pass. You could also be asked to display this \nmessage to airport security. * PLEASE NOTE: A printable",
"Save time at the airport Save time at the airport Web version",
"house was boarded up during the last round of bombings. I have no spatial \nimagination and cannot picture the house in three",
"Booking Confirmation Booking Reference: xxxxxx Date of issue: xxxxxxxxxxxx \nSelect Seats eUpgrade"
"required when boarding a flight to Canada. For more details, please visit \nCanada.ca/eTA . - Terms and Conditions of Sale",
"Your KLM boarding pass(s) on XXXXXX To: [image: KLM SkyTeam] Boarding \ninformation Thank you for checking in! Attached you",
"Boarding information Thank you for checking in! Attached you will find your \nboarding pass and/or other documents. Below",
"jetBlue® Your upcoming trip to SEATTLE, WA on xxxxxxxxxxx Flight status \nBaggage info Airport info TAG",
"your electronic boarding pass. You could also be asked to display this \nmessage to airport security. * PLEASE NOTE: A printable"

That request got sent to 172.217.13.197:443, which corresponds to some edge server near me. There were probably many other computers involved in searching my email than just the first one who got my request, but the nice thing about this is that we don’t need to care exactly what happened behind the scenes! The browser sent a request, and it got search results back, and it doesn’t need to know what servers.

We can just say “it happens on the server” and not worry too much about the ambiguity of what exactly that means (until something weird goes wrong :)).

the meaning of “server” depends on the context

So we’ve arrived somewhere a little bit interesting – at first when I thought about the question “what’s a server?” I really thought there was going to be a single simple answer! But it turns out that if you look at sentences where we use the word “server” it can actually refer to a lot of different things in a way that can be confusing:

2019-12-26T10:11:29+00:00


How tracking pixels work

I spent some time talking to a reporter yesterday about how advertisers track people on the internet. We had a really fun time looking at Firefox’s developer tools together (I’m not an internet privacy expert, but I do know how to use the network tab in developer tools!) and I learned a few things about how tracking pixels actually work in practice!

the question: how does Facebook know that you went to Old Navy?

I often hear about this slightly creepy internet experience: you’re looking at a product online, and a day later see an ad for the same boots or whatever that you were looking at. This is called “retargeting”, but how does it actually work exactly in practice?

In this post we’ll experiment a bit and see exactly how Facebook can know what products you’ve looked at online! I’m using Facebook as an example in this blog post just because it’s easy to find websites with Facebook tracking pixels on them but of course almost every internet advertising company does this kind of tracking.

the setup: allow third party trackers, turn off my adblocker

I use Firefox, and by default Firefox blocks a lot of this kind of tracking. So I needed to modify my Firefox privacy settings to get this tracking to work.

I changed my privacy settings from the default (screenshot) to a custom setting that allows third-party trackers (screenshot). I also disabled some privacy extensions I usually have running.

tracking pixels: it’s not the gif, it’s the URL + query parameters

A tracking pixel is a 1x1 transparent gif that sites use to track you. By itself, obviously a tiny 1x1 gif doesn’t do too much. So how do tracking pixels track you? 2 ways:

  1. Sites use the URL and query parameters in the tracking pixel to add extra information, like the URL of the page you’re visiting. So instead of just requesting https://www.facebook.com/tr/ (which is a 44-byte 1x1 gif), it’ll request https://www.facebook.com/tr/?the_website_you're_on. (email marketers use similar tricks to figure out if you’ve opened an email, by giving the tracking pixel a unique URL)
  2. Sites send cookies with the tracking pixel so that they can tell that the person who visited oldnavy.com is the same as the person who’s using Facebook on the same computer.

the Facebook tracking pixel on Old Navy’s website

To test this out, I went to look at a product on the Old Navy site with the URL https://oldnavy.gap.com/browse/product.do?pid=504753002&cid=1125694&pcid=1135640&vid=1&grid=pds_0_109_1 (a “Soft-Brushed Plaid Topcoat for Men”).

When I did that, the Javascript running on that page (presumably this code) sent a request to facebook.com that looks like this in Developer tools: (I censored most of the cookie values because some of them are my login cookies :) )

Let’s break down what’s happening:

  1. My browser sends a request to https://www.facebook.com/tr/?id=937725046402747&ev=PageView&dl=https%3A%2F%2Foldnavy.gap.com%2Fbrowse%2Fproduct.do%3Fpid%3D504753002%26cid%3D1125694%26pcid%3Dxxxxxx0%26vid%3D1%26grid%3Dpds_0_109_1%23pdp-page-content&rl=https%3A%2F%2Foldnavy.gap.com%2Fbrowse%2Fcategory.do%3Fcid%3D1135640%26mlink%3D5155%2Cm_mts_a&if=false&ts=1576684838096&sw=1920&sh=1080&v=2.9.15&r=stable&a=tmtealium&ec=0&o=30&fbp=fb.1.1576684798512.1946041422&it=15xxxxxxxxxx4&coo=false&rqm=GET
  2. With that request, it sends a cookie called fr which is set to 10oGXEcKfGekg67iy.AWVdJq5MG3VLYaNjz4MTNRaU1zg.Bd-kxt.KU.F36.0.0.Bd-kx6. (which I guess is my Facebook ad tracking ID)

So the three most notable things that are being sent in the tracking pixel query string are:

now let’s visit Facebook!

Next, let’s visit Facebook, where I’m logged in. What cookies is my browser sending Facebook?

Unsurprisingly, it’s the same fr cookie from before: 10oGXEcKfGekg67iy.AWVdJq5MG3VLYaNjz4MTNRaU1zg.Bd-kxt.KU.F36.0.0.Bd-kx6.. So Facebook now definitely knows that I (Julia Evans, the person with this Facebook account) visited the Old Navy website a couple of minutes ago and looked at a “Soft-Brushed Plaid Topcoat for Men”, because they can use that identifier to match up the data.

these cookies are third-party cookies

The fr cookie that Facebook is using to track what websites I go to is called a “third party cookie”, because Old Navy’s website is using it to identify me to a third party (facebook.com). This is different from first-party cookies, which are used to keep you logged in.

Safari and Firefox both block many third-party cookies by default (which is why I had to change Firefox’s privacy settings to get this experiment to work), and as of today Chrome doesn’t (presumably because Chrome is owned by an ad company).

sites have lots of tracking pixels

Like I expected, sites have lots of tracking pixels. For example, wrangler.com loaded 19 different tracking pixels in my browser from a bunch of different domains. The tracking pixels on wrangler.com came from: ct.pinterest.com, af.monetate.net, csm.va.us.criteo.net, google-analytics.com, dpm.demdex.net, google.ca, a.tribalfusion.com, data.photorank.me, stats.g.doubleclick.net, vfcorp.dl.sc.omtrdc.net, ib.adnxs.com, idsync.rlcdn.com, p.brsrvr.com, and adservice.google.com.

For most of these trackers, Firefox helpfully pointed out that it would have blocked them if I was using the standard Firefox privacy settings:

why browsers matter

The reason browsers matter so much is that your browser has the final word on what information it sends about you to which websites. The Javascript on the Old Navy’s website can ask your browser to send tracking information about you to Facebook, but your browser doesn’t have to do it! It can decide “oh yeah, I know that facebook.com/tr/ is a tracking pixel, I don’t want my users to be tracked, I’m just not going to send that request”.

And it can make that behaviour configurable by changing browser settings or installing browser extensions, which is why there are lots of privacy extensions.

it’s fun to see how this works!

I think it’s fun to see how cookies / tracking pixels are used to track you in practice, even if it’s kinda creepy! I sort of knew how this worked before but I’d never actually looked at the cookies on a tracking pixel myself or what kind of information it was sending in its query parameters exactly.

And if you know how it works, it’s a easier to figure out how to be tracked less!

what can you do?

I do a few small things to get tracked on the internet a little less:

There are still lots of other ways to be tracked on the internet (especially when using mobile apps where you don’t have the same kind of control as with your browser), but I like understanding how this one method of tracking works and think it’s nice to be tracked a little bit less.

2019-12-18T10:23:49+00:00


Challenge: find Twitter memes with suffix arrays

This challenge is a mix of data analysis and using fun algorithms! It’s the second challenge in a a short series of programming challenge I’m writing with Julian. (the first one was to write a tiny fun window manager)

Twitter has a lot of memes. For example, if you search Twitter for Flight attendant: is there a doctor on this flight?, you’ll find a bunch of tweets making jokes like this:

Flight Attendant: is there a doctor on board?
Parent: *nudging* That should've been you
Me: Not now, this is serious
Parent: Not asking for a hacker to help, are they?
Me: AAAAAAAA\x00\xd0X?\xfc\x7fBBBBj\x0bX\x99Rfh-p\x89\xe1Rjhh/bash/bin\x89\xe3RQS\x89\xe1\xcd\x80
Parent:~#

or if you search as a kpop fan there are thousands of these:

me as a kpop fan 

- kpop fan age: 10 years
- first group ever stan: super junior
- current ult groups: iKON, X1, Day6
- number of albums: >20 
- concerts attended: 6
- lightsticks owned: 2

So! Suppose you have a million tweets from the last 2 days. How do you find the jokes / quizzes / memes people are playing with on Twitter?

Challenge: find the twitter memes in 1 million tweets

This is a pretty open ended challenge and you can do it any way you want. Here’s a SQLite database with 1.2 million tweets, collected from the twitter streaming api over 2 days. It’s 250MB (70MB compressed), it only has English tweets. It excludes retweets and many tweets that are generated by bots.

The challenge: find at least 5 Twitter memes using that dataset.

memes as common substrings

The idea here is that memes are substrings like “me as a kpop fan” that many different people are using. The tricky thing is that you don’t really know how long those substrings will be, and maybe you’re interested in phrases of different lengths.

You can probably do this challenge without using anything fancy (with a hashmap of phrases or something) but I think it’s a nice opportunity to play with a fun data structure: suffix arrays! So let’s talk about what those are.

suffix arrays: sort all suffixes

Suffix arrays sort all suffixes of a string. For example, here’s the suffix array for “plantain” which has the suffixes plantain, lantain, antain, ntain, tain, ain, in, n.

ain
antain
in
lantain
n
ntain
plantain
tain

Representing this as a list of strings would be very inefficient (quadratic space), so instead we replace each suffix with the index of its first character in the original string – [5,2,6,1,7,3,0,4].

5 (ain)
2 (antain)
6 (in)
1 (lantain)
7 (n)
3 (ntain)
0 (plantain)
4 (tain)

Here’s a real example of what a suffix array of 1 million tweets concatenated looks like. This is an excerpt from the middle of the suffix array, with some of the suffixes that start with A little.

...
 A little distracted for a bit ...what do i do w my life hon.........
 A little exercise I did this afternoon.  #comics #art #clip.........
 A little extra Christmas Cash on me! Good Luck to everyone!.........
 A little girl in Savannah, Ga., appears to be the 38th huma.........
 A little heavy on the smut t… https://t.co/nvoxE7SNjTI wa.........
 A little in state battle tonight. #nova vs #penn. two very .........
 A little kiss...” one more time I’m going to vomit. #TT.........
 A little late catching up on last nights @GoodDoctorABC. On.........
 A little less bling never hurt anyone! Next project...🎄 .........
 A little more intensity to augment their talent and a coupl.........
 A little more time, because I have never lived really  - Os.........
 A little mor… https://t.co/kcq3zf9jgeWe love MX ❤️<F0><9F><A7>.........
 A little over 50k! Can We Guess How Much Is In Your Account.........
 A little ray of joy &amp; light in the midst of these very .........
 A little refreshment… https://t.co/HgX8PmYwPIThank you @L.........
 A little respect goes a long way. .........
 A little salt in d country's troubled legal system“Grant................
 A little snow &amp; people lose all common senseromantic st...............
 A little sun for the soul @realfreewebcams https://t.co/3CB...............
 A little sunkissed moment for y’all. ...............
 ....

Again, this is actually represented by a bunch of integer indexes into a concatenated string of all the tweets, like [18238223, 1921812, ...] so it’s a LOT more memory efficient than actually repeating all those strings.

suffix arrays let you find common substrings!

So what does this have to do with Twitter memes? Well, we can basically

  1. concatenate all tweets into a big string
  2. make a suffix array of that string
  3. iterate through the suffix array and notice when you see a lot of repeated substrings, like here:
me as a kpop fan ✨kpop fan age: 15 y/o ✨first group ever stan: blackpink ✨current ult groups: btxt ✨number of albu… https://t.co/24diHX9sLm
me as a kpop fan ⭐k-pop fan age: 12 y/o ⭐first group ever stan: bts ⭐current ult gps: bts and txt ⭐number of albu… https://t.co/8R95roQXoE
me as a kpop fan ⭐k-pop fan age: 14 y/o ⭐first group ever stan: girls generation ⭐current ult gp: txt ⭐number of a… https://t.co/010hLuJscF
me as a kpop fan ⭐k-pop fan age: 14-16 y/o ⭐first group ever stan: bts ⭐current ult gps: bts txt ⭐number of albums… https://t.co/0fDcxZGRrh
me as a kpop fan ⭐k-pop fan age: 15 y/o ⭐first group ever stan: blackpink ⭐current ult gps: txt ⭐number of albums… https://t.co/d8zZL83TvV
me as a kpop fan 🌸 k-pop fan age: 12 years old 🌸 first group ever stan: bts 🌸 current ult gps: bts &amp; wanna one 🌸 n… https://t.co/22R1nJpwNX
me as a kpop fan 🌸k-pop fan age: 10 🌸first group ever stan: 2pm 🌸current ult gps: skz,got7,itzy,twice, 🌸number of… https://t.co/mAluaP2yxH
me as a kpop fan 🌸k-pop fan age: 11 yo 🌸first group ever stan: beast 🌸current ult gps: ateez 🌸number of albums:  1… https://t.co/qxtFHG9HDg
me as a kpop fan 🌸k-pop fan age: 11 🌸first group ever stan: bts 🌸current ult gps: bts and ateez 🌸number of albums:… https://t.co/mKXlkrBBtC
me as a kpop fan 🌸k-pop fan age: 13 (now im 19) 🌸first group ever stan: snsd 🌸current ult gps: nct day6 aoa mamam… https://t.co/8XyQ3r5hwz
me as a kpop fan 🌸k-pop fan age: 13 years 🌸first group ever stan: 2pm,suju,bigbang 🌸current ult gps: bts,tbz,ateez… https://t.co/Zs1nQQz6Lt
me as a kpop fan 🌸k-pop fan age: 14 (2005) 🌸first group ever stan: super junior 🌸current ult gps: exo, gfriend, rv… https://t.co/vgmhe2vFMY
me as a kpop fan 🌸k-pop fan age: 14 y/o 🌸first group ever stan: nct dream 🌸current ult gps: svt and,,*insert stan… https://t.co/I38Ui69PvL
me as a kpop fan 🌸k-pop fan age: 15 y/o 🌸first group ever stan: 5sos 🌸current ult gps: bts and 5sos also some ggs… https://t.co/61ZmRkzmdl
me as a kpop fan 🌸k-pop fan age: 15 y/o 🌸first group ever stan: bts 🌸current ult gps: SVT, GOT7, Day6 🌸number of… https://t.co/16SWb3mSPg
me as a kpop fan 🌸k-pop fan age: 18 🌸first group ever stan: suju &amp; soshi 🌸current ult gps: snsd &amp; izone 🌸number of… https://t.co/SmSBFqJnGk
me as a kpop fan 🌸k-pop fan age: 19 y/o marupok 🌸first group ever stan: APINK 🌸current ult gps: SEVENTEEN 🌸number… https://t.co/StYjxr6uq9
me as a kpop fan 🌸k-pop fan age: 19 🌸first group ever stan: SuJu 🌸current ult gps: SuJu, SF9, SKZ, VIXX, ONEUS, NO… https://t.co/2o2DulCY5b

As an aside, the reason I got interested in suffix arrays in the first place was actually not for finding Twitter memes at all but for search.

I’ve spent a lot of time using Nelson Elhage’s livegrep at work to search code. It creates a suffix array using the divsufsort library. He has a blog post Regular Expression Search with Suffix Arrays where he talks about some of the implementation details.

The reason suffix arrays work for fast search is basically that if you’re looking for the string A little, you can do a binary search over the suffix array to find every instance of A little in your dataset. Binary searches are extremely fast so every search is guaranteed to run very quickly (in less than a microsecond I believe). What livegrep does is more complicated than that because it does a regular expression search, but that’s the idea to start.

There’s another blog post How to use suffix arrays to combat common limitations of full-text search applying suffix arrays to searching through a patent database. In that example, like with code search, the patent officers want to search patents for exact strings.

How do you make a suffix array?

You can use an existing suffix array library, for example index/suffixarray in Go, which is what I used, or divsufsort. There are Python bindings for divsufsort.

If you’re more excited about the data structures/algorithms aspect of suffix arrays you can also implement a suffix array-building algorithm yourself! I did not do this but you can see an implementation of qsufsort here in Go. That implementation links to a paper. There are lots of algorithms for constructing suffix arrays –sais and divsufsort are a couple of others.

5 or so hours, 100 lines of Go

As always with these challenges, I did this one to make sure that it’s both doable in a reasonable amount of time and fun for at least one person (me).

I did this one in about 5 hours and 100 lines of Go using the suffixarray implementation in the Go standard library, with a bit of bash shell scripting to postprocess the results. This is a messy data analysis challenge – as an example of a messy thing, Spotify released their end-of-2019 results while I was building the dataset and so there are a lot of tweets generated by the Spotify app.

My results ended up looking something like this:

5  an Aries and that’s why I gotta 
5  an Aries and that’s why I am so 
5  an Aquarius and that’s why I 
5  AM SO PROUD OF YOU 
5  am not a fan of 
5  am I the only one who 
5  am going to have to 

Then I sifted through them pretty manually to find the Twitter memes.

suffix arrays are used in bioinformatics

This “find twitter memes using suffix arrays” approach is a silly thing but it does have some relationship to reality – DNA sequences are basically really long strings, and biologists need to find patterns in them, and they sometimes use suffix arrays to do it.

I looked up packages in Debian that use libdivsufsort and I found infernal:

Infernal (“INFERence of RNA ALignment”) is for searching DNA sequence databases for RNA structure and sequence similarities. It is an implementation of a special case of profile stochastic context-free grammars called covariance models (CMs). A CM is like a sequence profile, but it scores a combination of sequence consensus and RNA secondary structure consensus, so in many cases, it is more capable of identifying RNA homologs that conserve their secondary structure more than their primary sequence.

email me the twitter memes you find if you do this!

If you do this exercise, I’d love it if you emailed me (julia@jvns.ca) with the twitter memes you found and/or your code! I found about 8 but I’m sure there are more.

I’ll publish any solutions I get (unless you don’t want me to publish your solution – just let me know!).

Thanks to Julian for discussing suffix arrays and suffix trees and trigram indexes with me at length, and to Kamal who had the idea of using suffix arrays to find Twitter memes.

2019-12-05T10:41:54+00:00


Solutions to the tiny window manager challenge

Hello! Last week I posted a small programming challenge to write a tiny window manager that bounces windows around the screen.

I’ll write a bit about my experience of solving the challenge, or you can just skip to the end to see the solutions.

what’s a window manager?

An X window manager is a program that sends messages to the X server (which is in charge of drawing your windows) to tell it which windows to display and where.

I found out that you can trace those events with xtrace. Here’s some example output from xtrace (for the toy window manager which is just moving windows about)

000:<:02d8: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=560 y=8}
000:<:02da: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=554 y=12}
000:<:02dc: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=548 y=16}
000:<:02de: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=542 y=20}
000:<:02e0: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=536 y=24}
000:<:02e2: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=530 y=28}
000:<:02e4: 20: Request(12): ConfigureWindow window=0x004158e5 values={x=524 y=32}

you can run programs without a window manager

You technically don’t need a window manager to run graphical programs – if you want to start an xterm in a window-manager-less X session you can just run

xterm -display :1

and it’ll start the xterm. Here’s a screenshot of an X session with no window manager open. I even have 2 windows open! (chrome and an xterm). It has some major usability problems, for example I don’t think you can resize or move or switch between windows. Which is where the window manager comes in!

move a window with XMoveWindow

The challenge was to make the window bounce around the screen.

In the tinywm source they use XMoveResizeWindow to move and resize windows, but I found in the docs that there’s also a function called XMoveWindow. Perfect!

Here’s what it looks like. What could be simpler, right? And it works just the way I’d expect!

XMoveWindow(display, windowID, x, y)

Except…

problem: multiple XMoveWindows don’t work

I ran into a problem (which I got stuck on for a couple of hours) where when I ran XMoveWindow twice, it would only apply the last move.

XMoveWindow(display, windowID, 100, 200)
usleep(2000 * 1000); # sleep for 2 seconds
XMoveWindow(display, windowID, 300, 400)

I’d expect this to move the window once, wait 2 seconds, and them move it again. But that was not what happened! Instead, it would pause for 2 seconds and then move the window once (to the second location).

use xtrace to trace window manager events

I used xtrace to trace the events and found out that my ConfigureWindow events that XMoveWindow was sending were all being sent at the same time. So it seemed like X was batching the events. But why?

XSync forces X to process events

I didn’t know why this was happening, but I emailed Julian about it and he pointed me in the direction of XSync, which forces X to process all the events you’ve sent it. Sure enough, I used XSync and everything worked beautifully.

solutions

I asked people to email me if they completed the challenge, and 4 people did! Here are their solutions. All the solutions I got implemented more features than I did, so I’d encourage you to look at all the solutions if you’re interested in how to solve this problem!

Here’s a gif of Alexsey’s solution. Apparently XQuartz on a Mac performs better than Xephyr!

And Aldrin’s solution, with a great use of xeyes:

thanks!

Thanks to everyone who emailed me a solution, and if you write your own implementation I’d love to post it here too, especially if you write one that isn’t in C or Ruby! I’m julia@jvns.ca. (and if you write a solution but don’t want me to post it I’d still love to see it!)

2019-12-03T14:57:10+00:00


Challenge: Write a bouncy window manager

Hello! I’m writing a short series of programming challenges with Julian, and this is the first one!

the challenge

requirements

The goal here is to make a very silly Linux window manager that bounces its windows around the screen, like in the gif above.

anti-requirements

The window manager doesn’t need to do anything else! It doesn’t need to support:

It turns out implementing this kind of toy window manager is surprisingly approachable!

the setup: start with tinywm

All the instructions here only work on Linux (since this is about writing a Linux window manager).

starter kit: tinywm

Writing a window manager from scratch seems intimidating (at first I didn’t even know how to start!). But then I found tinywm, which is a tiny window manager written in only 50 lines of C. This is a GREAT starting point and there’s an annotated version of the source code which explains a lot of the details. There’s a Python version of tinywm too, but I wasn’t able to get it to work.

I did this challenge by modifying tinywm and it worked really well.

tools

documentation

Some useful references:

If you’re not comfortable writing C, there are also libraries that let you work with X in other languages. I personally found C easier to use because a lot of the window manager documentation and examples I found were for the Xlib C library.

my experience: 5 hours, 50 lines of code

To give you a very rough idea of the difficulty of this exercise: I did this in 4 or 5 hours this morning and last night, producing the window manager you see in the gif at the top of the blog post (which is 50 lines of code). I’d never looked at the source code for a window manager before yesterday.

As usual when working with a new library I spent most of that time being confused about various basic things about how X works. (and as a result I learned several new things about X!)

For me this challenge was a fun way to:

send me your solution if you do this!

I’ll post the solution I came up in a week. If you think this window manager challenge sounds fun and end up doing it, I’d love it if you sent me your solution (to julia@jvns.ca)!

I’d be delighted to post any solutions you send me in the solutions blog post.

2019-11-25T18:03:40+00:00


What makes a programming exercise good?

I’ve been thinking about programming exercises lately, because I want to move into teaching people skills. But what makes a good programming exercise? I asked about this on Twitter today and got some useful responses so here are some criteria:

it’s fun

This one is sort of self-explanatory and I think it’s really important. Programming is fun and learning is fun so I can’t see why programming exercises would need to be boring.

it teaches you something you care about

I don’t think this has to strictly mean “relevant to your job right this second” – people don’t just have jobs, we also want to make art and games and fun personal projects and sometimes just understand the world around us. But it’s important to know what goals the exercise can help you with and what it’s related to!

Some arbitrary examples:

it’s a challenge

I don’t know if this is everyone’s experience but I often start programming exercises and get bored quickly (“oh, I know how to do this, this is boring”). For me it’s really important for the exercise to teach me something I really don’t know how to do and that’s a little bit hard for me.

My favourite set of programming exercises is the cryptopals crypto challenges because they get harder pretty fast – by exercise #6, you’re already breaking toy encryption protocols, and by #12 you’re breaking an Actual Encryption Protocol (AES in ECB mode)!

you can tell if you succeeded

It’s easy to write exercises that are too vaguely specified (“write a toy tcp stack!“). But what does that mean? How much of a TCP stack am I supposed to write? Having test cases and clear criteria for “yay! you did it! congratulations!” is really important.

you can do it quickly

In less than 2-3 hours (an evening after work), say. It’s hard to find time to spend like 8 hours on an exercise unless it’s REALLY exciting.

I also think that giving some specific real-world benchmark data seems nice (“I did this from scratch in 97 minutes”).

the author believes in you

This is a bit fuzzier but very lovely – this person on Twitter wrote:

Similar to that, the writing is patient and gives me the impression that it believes in my ability to accomplish the task. … I learned a ton in the early days from Linux HOWTOs. Some gave me the sense that it was impossible to fail. Just follow the steps. It’s all there.

Especially if you’re doing a somewhat challenging exercise like we talked about above, I think it’s nice for the author to believe in your! (and of course it’s crucial that they’ve actually written the exercises so that they’re right and you can likely do the thing!)

it’s been tested

I read the (great) biography Dearie: The Remarkable Life of Julia Child recently and one thing that stood out to me is that she tested all of the recipes in Mastering the Art Of French Cooking. It took her years to write the book and test the recipes and make sure that American home cooks actually had access to all the ingredients and had the.

I don’t think all cookbook authors test their recipes, but I think testing really improves cookbooks.

I started writing some SQL exercises (like this prototype of one on GROUP BY) a while back, and at some point I realized the big thing holding me back was that I didn’t have testers! I couldn’t find out if people were actually learning from them or not!

This is a new thing for me because when I write blog posts I don’t test them (I barely even proofread them!). I just write them and publish and people often like them and that’s it! I said to Amy Hoy (who is amazing) on Twitter that I didn’t understand why you have to test exercises if you don’t have to test blog posts and she pointed out that people have much higher expectations for exercises than for blog posts – with the blog posts you maybe expect to learn 1-2 new facts, but with exercises you expect to actually develop a new skill!

Also, people are often investing a lot more time in exercises (especially if they have to set up a dev environment or something!), so it’s extra important to make sure that they actually work.

you won’t get stuck

It’s SO EASY to get stuck on some random irrelevant point in a programming exercise that’s totally unrelated to the skill you’re trying to learn. For example there might be an easily-avoidable mistake that you can make with the exercise and spend a lot of time debugging but it doesn’t actually teach you a lot.

it’s easy to get help

If you’re doing a challenging exercise, you might want to get help from your friends / colleagues / the internet!

Some things that can go wrong:

One obvious way to accomplish this is by letting people use the programming language they’re most comfortable in, because they probably already know how to Google for help in that environment.

no time-consuming setup required

Installing software is boring, and a lot of programming projects require installing software! A few things that can go wrong with this (though there are a lot more than this!)

This kind of thing is a huge waste of time and super demoralizing. And it’s not trivial to avoid! If you’re trying to teach someone a specific piece of software, often that software

A few options I’ve seen or used to manage this:

it’s easy to extend

@tef has this great talk on Scratch A million things to do with a computer! which explains the 3 ideas of Scratch:

It sucks when you start learning something and then learn that what you can do with the Thing is very limited! It’s exciting when you learn something and see “oh, wow, there are SO MANY POSSIBILITIES, what if I did X instead?”

that’s a lot of things!

The criteria we arrived at:

That seems pretty hard, but it seems like a good goal to aspire to! I’m going to keep very slowly working on exercises!

2019-11-20T20:33:20+00:00


How containers work: overlayfs

I wrote a comic about overlay filesystems for a potential future container zine this morning, and then I got excited about the topic and wanted to write a blog post with more details. Here’s the comic, to start out:

container images are big

Container images can be pretty big (though some are really small, like alpine linux is 2.5MB). Ubuntu 16.04 is about 27MB, and the Anaconda Python distribution is 800MB to 1.5GB.

Every container you start with an image starts out with the same blank slate, as if it made a copy of the image just for that container to use. But for big container images, like that 800MB Anaconda image, making a copy would be both a waste of disk space and pretty slow. So Docker doesn’t make copies – instead it uses an overlay.

how overlays work

Overlay filesystems, also known as “union filesystems” or “union mounts” let you mount a filesystem using 2 directories: a “lower” directory, and an “upper” directory.

Basically:

When a process reads a file, the overlayfs filesystem driver looks in the upper directory and reads the file from there if it’s present. Otherwise, it looks in the lower directory.

When a process writes a file, overlayfs will just write it to the upper directory.

let’s make an overlay with mount!

That was all a little abstract, so let’s make an overlay filesystem and try it out! This is just going to have a few files in it: I’ll make upper and lower directories, and a merged directory to mount the combined filesystem into:

$ mkdir upper lower merged work
$ echo "I'm from lower!" > lower/in_lower.txt 
$ echo "I'm from upper!" > upper/in_upper.txt
$ # `in_both` is in both directories
$ echo "I'm from lower!" > lower/in_both.txt 
$ echo "I'm from upper!" > upper/in_both.txt 

Combining the upper and lower directories is pretty easy: we can just do it with mount!

$ sudo mount -t overlay overlay 
    -o lowerdir=/home/bork/test/lower,upperdir=/home/bork/test/upper,workdir=/home/bork/test/work 
    /home/bork/test/merged

There’s was an extremely annoying error message I kept getting while doing this, that said mount: /home/bork/test/merged: special device overlay does not exist.. This message is a lie, and actually just means that one of the directories I specified was missing (I’d written ~/test/merged but it wasn’t being expanded).

Okay, let’s try to read one of the files from the overlay filesystem! The file in_both.txt exists in both lower/ and upper/, so it should read the file from the upper/ directory.

$ cat merged/in_both.txt 
"I'm from upper!

It worked!

And the contents of our directories are what we’d expect:

find lower/ upper/ merged/
lower/
lower/in_lower.txt
lower/in_both.txt
upper/
upper/in_upper.txt
upper/in_both.txt
merged/
merged/in_lower.txt
merged/in_both.txt
merged/in_upper.txt

what happens when you create a new file?

$ echo 'new file' > merged/new_file
$ ls -l */new_file 
-rw-r--r-- 1 bork bork 9 Nov 18 14:24 merged/new_file
-rw-r--r-- 1 bork bork 9 Nov 18 14:24 upper/new_file

That makes sense, the new file gets created in the upper directory.

what happens when you delete a file?

Reads and writes seem pretty straightforward. But what happens with deletes? Let’s do it!

$ rm merged/in_both.txt

What happened? Let’s look with ls:

ls -l upper/in_both.txt  lower/lower1.txt  merged/lower1.txt
ls: cannot access 'merged/in_both.txt': No such file or directory
-rw-r--r-- 1 bork bork    6 Nov 18 14:09 lower/in_both.txt
c--------- 1 root root 0, 0 Nov 18 14:19 upper/in_both.txt

So:

What happens if we try to copy this weird character device file?

$ sudo cp upper/in_both.txt upper/in_lower.txt
cp: cannot open 'upper/in_both.txt' for reading: No such device or address

Okay, that seems reasonable, being able to copy this weird deletion signal file doesn’t really make sense.

you can mount multiple “lower” directories

Docker images are often composed of like 25 “layers”. Overlayfs supports having multiple lower directories, so you can run

mount -t overlay overlay
      -o lowerdir:/dir1:/dir2:/dir3:...:/dir25,upperdir=...

So I assume that’s how containers with many Docker layers work, it just unpacks each layer into a separate directory and then asks overlayfs to combine them all together together with an empty upper directory that the container will write its changes to it.

docker can also use btrfs snapshots

Right now I’m using ext4, and Docker uses overlayfs snapshots to run containers. But I used to use btrfs, and then Docker would use btrfs copy-on-write snapshots instead. (Here’s a list of when Docker uses which storage drivers)

Using btrfs snapshots this way had some interesting consequences – at some point last year I was running hundreds of short-lived Docker containers on my laptop, and this resulted in me running out of btrfs metadata space (like this person). This was really confusing because I’d never heard of btrfs metadata before and it was tricky to figure out how to clean up my filesystem so I could run Docker containers again. (this docker github issue describes a similar problem with Docker and btrfs)

it’s fun to try out container features in a simple way!

I think containers often seem like they’re doing “complicated” things and I think it’s fun to break them down like this – you can just run one mount incantation without actually doing anything else related to containers at all and see how overlays work!

2019-11-18T13:34:29+00:00


Some notes on vector drawing apps

For the last year and a half I’ve been using the iPad Notability app to draw my zines. Last week I decided I wanted more features, did a bit of research, and decided to switch to Affinity Designer (a much more complicated program). So here are a few quick notes about it.

The main difference between them is that Notability is a simple note taking app (aimed at regular people), and Affinity Designer is a vector graphics app (aimed at illustrators / graphic designers), like Adobe Illustrator.

I’ve never used a serious vector graphics program before, so it’s been cool to learn what kinds of features are available!

Notability is super simple

This is what the Notability UI looks like. There’s a pencil, an eraser, a text tool, and a selection tool. That’s basically it. I LOVED this simplicity when I started using Notability, and I made 4 zines using it (help! i have a manager!, oh shit, git!, bite size networking!, and http: use your browser’s language).

Recently though, I’ve had a couple of problems with it, the main one being that text boxes and things drawn with the pencil tool don’t mix well. (In general Notability has been GREAT though and their support team has always been incredibly helpful when I’ve had questions.)

Affinity Designer is really complicated

Affinity Designer, by comparison, is WAY more complicated. Here’s what the UI looks like:

There are

I still don’t understand what all the tools do (what’s the difference between Pencil and Vector Brush? I don’t know!). But I’m pretty excited about this because (unlike with Notability) there are so many options that if I’m frustrated about something, 90% of the time there’s a way to do the thing I want!

switching from Notability to Affinity Designer is really easy

Switching to Notability wasn’t the best: I reverse engineered the file format to transfer some files over but the quality was never the best (probably because of problems with my script) and I ended up having to redraw a lot of them in practice.

With Affinity Designer, I can just

It’s not perfect – the vector paths it comes up with are kind of weird, probably because of the way the PDF is – but it’s very good! It makes me feel confident that if I need to make a small edit to something I made in the past I can just import the PDF!

what can a vector drawing app do?

here are a few things Affinity Designer can do that Notability can’t:

There are also a LOT more features that I’m not interested in but I’m pretty excited about those 6 things and it feels like an app that I won’t grow out of.

iPad apps are great

I’ve been exclusively using Linux for the last 15 years where the image editing/media tools aren’t always great (though I really like Inkscape and I hear good things about Krita!), so it’s really cool to have access to all these great iPad apps. And the prices seem pretty reasonable:

It doesn’t make me want a Mac (I like the Linux desktop experience!), but it’s nice to have access to a bunch of these great tools. And I think a lot of these art tools work better on an iPad than on a computer anyway since you can just draw on the screen :)

2019-11-18T10:47:41+00:00


Some research on shipping print zines

I’ve been doing some preliminary research on shipping printed zines, since Your Linux Toolbox is out now and a bunch more people have been asking about print copies of my other zines. I thought I’d write down what I’ve learned so far because it turns out shipping is pretty complicated!

My original question I was trying to answer was “can I ship a single zine anywhere in the world for less than $6 or so?“, so let’s start there.

Surprisingly the best single resource I found was this very extensive PCMag article on e-commerce fulfillment services.

why not use letter mail?

The most obvious way to send a zine inexpensively in the mail is with letter mail – letters smaller than 10” x 6” and under 60g or so can be sent by letter mail. My zines definitely fit that criteria, even when printed on nice paper. This would be really good because international package postage is EXPENSIVE, but sending a letter to Belgium only costs $2.13 according to USPS’s website.

The issue with this is the small print on that USPS page:

Value of contents can not exceed $0.00

So it seems like you’re not actually allowed to send things worth money via letter mail. Probably that’s related to customs or something. Or maybe letter mail is subsidized by the government? Not sure why.

Option 0: Ship zines myself

I’ve done this before and it was actually really fun to do once but I think this is pretty unlikely to be a good idea because:

a. cost: I live in Canada, almost everyone I sell zines to is outside of Canada, and b. availability: I’d like for people to be able to get shipments when I’m out of town / on vacation

Option 1: Amazon

One obvious answer to how to sell book-like things is “sell them on Amazon!”. Amazon actually has at least 3 different programs that you can use to sell books online (Amazon Advantage, Fulfilled By Amazon, Kindle Direct Publishing), and since Amazon is such a big thing I looked into all of them a little bit.

In general the forums on https://sellercentral.amazon.com seem to be a good way to understand how the various Amazon options work for people.

I wrote a lot about Amazon here but overall it doesn’t seem that great of an option – it’s really complicated, selling on Amazon’s website isn’t very appealing, and I think there would be a lot of additional fees.

Kindle Direct Publishing

Kindle Direct Publishing is a service where Amazon will take care of everything from printing to shipping. (It has “Kindle” in the same but they actually do printing as well). Brian Kernighan’s new Unix book is an example of a book published with KDP.

KDP won’t work for this project because they don’t support saddle stitching (stapling the zine), so I didn’t look into it too much. Here’s a link to their paper and cover options though.

Amazon Advantage

Amazon Advantage doesn’t do printing – you ship them books, and then they take care of shipping them to people. This seems great on its surface (“amazon just takes care of it!“).

advantages:

disadvantages:

The biggest issue for me here seems to be “you have to ship them books every week or two”, which seems like it could get very expensive if Amazon Advantage keeps asking you to ship them small quantities of zines.

Fulfilled By Amazon

Fulfilled By Amazon seems like the Amazon option that involves the least Amazon magic. Basically I’d ship books to their warehouses and then they ship the things from those warehouses.

how it works:

Option 2: Blackbox

Blackbox is a shipping company by the Cards Against Humanity folks. This is their Pricing PDF. I’m not 100% sure if I can work with them – the first time I filled out the form on their website saying I was interested they said they weren’t accepting new customers, but I think now they may be?

Here’s the summary:

Option 3: Shipbob

Shipbob is an shipping company for smaller ecommerce companies. Here’s the pricing PDF I found.

The main difference as far as I can tell between Shipbob and Blackbox is that Shipbox lets you include up to 5 items per order for free and Blackbox charges $0.70 per additional item in an order.

Shipbob advertises a 99.8% fulfillment accuracy rate which is pretty interesting – it means they expect 2 in 1000 orders to have a problem. That seems pretty good!

Option 4: Whiplash

Similar to the other two above. Their pricing page. Shipbob and Blackbox both include everything (shipping, packaging materials, and someone packing your order) in their fee, and Whiplash seems to charge separately for shipping.

Shipping 1 thing costs the same as shipping 5 things

Zines are pretty light – I just weighed some of my zines printed on high quality paper on my kitchen scale and they’re 40g each on average. Most of these shipping services seem to charge in increments of half a pound, so shipping 5 zines (about 200g) costs about the same as shipping 1 zine (about 40g).

This makes me think it would be more reasonable to focus on shipping packages of many zines – right now the 6 pack of all my zines costs $58 for a PDF collection, and $6-$12 shipping for something around that price seems super reasonable (and I could probably even do “free” shipping, aka pay the shipping costs myself).

The other thing that I think could be work well is shipping packages of 5 or 10 of the same zine so that a group of people can each get a zine and save on shipping costs.

this seems like it could work!

I still have no plan for how to print zines, but writing all this down makes me feel pretty optimistic about being able to ship zines to people. Even though shipping individual zines doesn’t seem that practical, I think shipping packs of 5-10 zines could be really reasonable!

Speaking of print – I printed a zine with Lulu last week and just got it in the mail yesterday. I didn’t think Lulu would be able to print it in the way I wanted, and they didn’t, so I’m really happy to know that and be able to move on to trying other non-print-on-demand printers.

2019-10-29T10:51:14+00:00


SQLite is really easy to compile

In the last week I’ve been working on another SQL website (https://sql-steps.wizardzines.com/, a list of SQL examples). I’m running all the queries on that site with sqlite, and I wanted to use window functions in one of the examples (this one).

But I’m using the version of sqlite from Ubuntu 18.04, and that version is too old and doesn’t support window functions. So I needed to upgrade sqlite!

This turned to out be surprisingly annoying (as usual), but in a pretty interesting way! I was reminded of some things about how executables and shared libraries work and it had a very satisfying conclusion. So I wanted to write it up here.

(spoiler: the summary is that https://www.sqlite.org/howtocompile.html explains how to compile SQLite and it takes like 5 seconds to do and it’s 20x easier than my usual experiences compiling software from source)

attempt 1: download a SQLite binary from their website

The SQLite download page has a link to a Linux binary for the SQLite command line tool. I downloaded it, it worked on my laptop, and I thought I was done.

But then I tried to run it on a build server I was using (Netlify), and I got this extremely strange error message: “File not found”. I straced it, and sure enough execve was returning the error code ENOENT, which means “File not found”. This was kind of maddening because the file was DEFINITELY there and it had the correct permissions and everything.

I googled this problem (by searching “execve enoent”), found this stack overflow answer, which pointed out that to run a binary, you don’t just need the binary to exist! You also need its loader to exist. (the path to the loader is inside the binary)

To see the path for the loader you can use ldd, like this:

$ ldd sqlite3
	linux-gate.so.1 (0xf7f9d000)
	libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf7f70000)
	libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf7e6e000)
	libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xf7e4f000)
	libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7c73000)
	/lib/ld-linux.so.2

So /lib/ld-linux.so.2 is the loader,and that file doesn’t exist on the build server, probably because that Xenial installation didn’t have support for 32-bit binaries (?), and I needed to try something different.

attempt 2: install the Debian sqlite3 package

Okay, I thought, maybe I can install the sqlite package from debian testing. Trying to install a package from a different Debian version that I’m not using is literally never a good idea, but for some reason I decided to try it anyway.

Doing this completely unsurprisingly broke the sqlite installation on my computer (which also broke git), but I managed to recover from that with a bunch of sudo dpkg --purge --force-all libsqlite3-0 and make everything that depended on sqlite work again.

attempt 3: extract the Debian sqlite3 package

I also briefly tried to just extract the sqlite3 binary from the Debian sqlite package and run it. Unsurprisingly, this also didn’t work, but in a more understandable way: I had an older version of libreadline (.so.7) and it wanted .so.8.

$ ./usr/bin/sqlite3
./usr/bin/sqlite3: error while loading shared libraries: libreadline.so.8: cannot open shared object file: No such file or directory

attempt 4: compile it from source

The whole reason I spent all this time trying to download sqlite binaries is that I assumed it would be annoying or time consuming to compile sqlite from source. But obviously downloading random sqlite binaries was not working for me at all, so I finally decided to try to compile it myself.

Here are the directions: How to compile SQLite. And they’re the EASIEST THING IN THE UNIVERSE. Often compiling things feels like this:

Compiling SQLite works like this:

All the code is in one file (sqlite.c), and there are no weird dependencies! It’s amazing.

For my specific use case I didn’t actually need threading support or readline support or anything, so I used the instructions on the compile page to create a very simple binary that only used libc and no other shared libraries.

$ ldd sqlite3
	linux-vdso.so.1 (0x00007ffe8e7e9000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbea4988000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbea4d79000)

this is nice because it makes it easy to experiment with sqlite

I think it’s cool that SQLite’s build process is so simple because in the past I’ve had fun editing sqlite’s source code to understand how its btree implementation works.

This isn’t really super surprising given what I know about SQLite (it’s made to work really well in restricted / embedded contexts, so it makes sense that it would be possible to compile it in a really simple/minimal way). But it is super nice!

2019-10-28T11:19:19+00:00


Your Linux Toolbox: a box set of my free zines

About a year and a half ago, No Starch Press got in touch with me about publishing a print box set of my zines. I have two kinds of zines right now:

This set is basically a really lovely box set of all of the free zines, plus Bite Size Linux :). Here’s what’s in the box:

I’m really happy to get these zines into print, and that I can still give away all of the zines in the box away for free on my website – I asked them to write it into my publishing contract that I could still give them away, and they did :)

what it looks like

Here are the front covers of the zines in the box. We got colour covers illustrated for all of them, done by Vladimir Kašiković.

We had the idea to make the back covers a rainbow and I’m delighted about it:

There’s this fun “this toolbox belongs to:” detail on the bottom:

where to get it

It’s in a bunch of physical bookstores, and online! Here are a bunch of links to places you could get it:

North America:

Europe:

Asia/Pacific:

More:

why I’m doing this: to learn about print!

I don’t necessarily expect to make a lot of money from this box set (I get 10% or less of each sale, vs 97% for sales of my other zines online) but that’s not my priority with this project – I did it because I love the free zines I wrote, I wanted to make a really nice print version of them, and I wanted to learn about how print works and how traditional publishing works! I’ve already learned a lot about how publishing works and it’s been super interesting.

People have been very excited about this print project so far which has been really nice to see! Next I want to make it possible for people to order print copies of my newer zines, and I’m trying to figure out how to do that now. (if you have a print company that you’ve really loved using, let me know!)

I’m super happy about the print quality and if you get the box set I really hope you like it!

2019-10-21T16:15:19+00:00


SQL queries don't start with SELECT

Okay, obviously many SQL queries do start with SELECT (and actually this post is only about SELECT queries, not INSERTs or anything).

But! Yesterday I was working on an explanation of window functions, and I found myself googling “can you filter based on the result of a window function”. As in – can you filter the result of a window function in a WHERE or HAVING or something?

Eventually I concluded “window functions must run after WHERE and GROUP BY happen, so you can’t do it”. But this led me to a bigger question – what order do SQL queries actually run in?.

This was something that I felt like I knew intuitively (“I’ve written at least 10,000 SQL queries, some of them were really complicated! I must know this!“) but I struggled to actually articulate what the order was.

SQL queries happen in this order

I looked up the order, and here it is! (SELECT isn’t the first thing, it’s like the 5th thing!) (here it is in a tweet).

(I really want to find a more accurate way of phrasing this than “sql queries happen/run in this order” but I haven’t figured it out yet)

In a non-image format, the order is:

questions this diagram helps you answer

This diagram is about the semantics of SQL queries – it lets you reason through what a given query will return and answers questions like:

Database engines don’t actually literally run queries in this order because they implement a bunch of optimizations to make queries run faster – we’ll get to that a little later in the post.

So:

confounding factor: column aliases

Someone on Twitter pointed out that many SQL implementations let you use the syntax:

SELECT CONCAT(first_name, ' ', last_name) AS full_name, count(*)
FROM table
GROUP BY full_name

This query makes it look like GROUP BY happens after SELECT even though GROUP BY is first, because the GROUP BY references an alias from the SELECT. But it’s not actually necessary for the GROUP BY to run after the SELECT for this to work – the database engine can just rewrite the query as

SELECT CONCAT(first_name, ' ', last_name) AS full_name, count(*)
FROM table
GROUP BY CONCAT(first_name, ' ', last_name)

and run the GROUP BY first.

Your database engine also definitely does a bunch of checks to make sure that what you put in SELECT and GROUP BY makes sense together before it even starts to run the query, so it has to look at the query as a whole anyway before it starts to come up with an execution plan.

queries aren’t actually run in this order (optimizations!)

Database engines in practice don’t actually run queries by joining, and then filtering, and then grouping, because they implement a bunch of optimizations reorder things to make the query run faster as long as reordering things won’t change the results of the query.

One simple example of a reason why need to run queries in a different order to make them fast is that in this query:

SELECT * FROM
owners LEFT JOIN cats ON owners.id = cats.owner
WHERE cats.name = 'mr darcy'

it would be silly to do the whole left join and match up all the rows in the 2 tables if you just need to look up the 3 cats named ‘mr darcy’ – it’s way faster to do some filtering first for cats named ‘mr darcy’. And in this case filtering first doesn’t change the results of the query!

There are lots of other optimizations that database engines implement in practice that might make them run queries in a different order but there’s no room for that and honestly it’s not something I’m an expert on.

LINQ starts queries with FROM

LINQ (a querying syntax in C# and VB.NET) uses the order FROM ... WHERE ... SELECT. Here’s an example of a LINQ query:

var teenAgerStudent = from s in studentList
                      where s.Age > 12 && s.Age < 20
                      select s;

pandas (my favourite data wrangling tool) also basically works like this, though you don’t need to use this exact order – I’ll often write pandas code like this:

df = thing1.join(thing2)      # like a JOIN
df = df[df.created_at > 1000] # like a WHERE
df = df.groupby('something', num_yes = ('yes', 'sum')) # like a GROUP BY
df = df[df.num_yes > 2]       # like a HAVING, filtering on the result of a GROUP BY
df = df[['num_yes', 'something1', 'something']] # pick the columns I want to display, like a SELECT
df.sort_values('sometthing', ascending=True)[:30] # ORDER BY and LIMIT
df[:30]

This isn’t because pandas is imposing any specific rule on how you have to write your code, though. It’s just that it often makes sense to write code in the order JOIN / WHERE / GROUP BY / HAVING. (I’ll often put a WHERE first to improve performance though, and I think most database engines will also do a WHERE first in practice)

dplyr in R also lets you use a different syntax for querying SQL databases like Postgres, MySQL and SQLite, which is also in a more logical order.

I was really surprised that I didn’t know this

I’m writing a blog post about this because when I found out the order I was SO SURPRISED that I’d never seen it written down that way before – it explains basically everything that I knew intuitively about why some queries are allowed and others aren’t. So I wanted to write it down in the hopes that it will help other people also understand how to write SQL queries.

2019-10-03T10:24:53+00:00


Zine revenue for 2019

I occasionally get questions like “Can you share what you’ve learned about running a business?” The most surprising thing I’ve learned is that it’s possible to make money by teaching people computer things on the internet, so I want to make that a little more concrete by sharing the revenue from the zine business so far in 2019. Here’s a graph of revenue by month (the last month is September 2019):

This adds up to $87,858 USD for 2019 so far, which (depending on what I release in the rest of this year) is on track to be similar to revenue for 2018 ($101,558).

Until quite recently I’d been writing zines in my spare time, and now I’m taking a year to focus on it.

how $30,000 for September breaks down

The most obvious thing in that monthly revenue graph above is that 2 months (September and March) have way more revenue than all the others. This is because I released new zines (Bite Size Networking and HTTP: Learn your browser’s language) in those months.

Here’s how the $30,000 for September breaks down:

This September was the month with the most sales ever, which is mostly because of individual humans who find the zines useful (thank you!!).

expenses

The main expenses are paying illustrators and an accountant, a mailing list, and various books I buy to learn how to do things better. They probably come out to about 10% of revenue or so, and then there are taxes after that.

giving away free copies has been great

With the HTTP zine, like many of my previous zines, I’ve been giving away one free copy for every copy that people buy, so that people can get it even if $12 is hard for them to afford. (if you can’t afford $12, here’s the link, there are about 70 available as I’m writing this). I’m pretty happy with this setup – we’ve given away 1358 copies so far. (I think of this as kind of a “sales” statistic too)

I think I want to automate the system to give away free copies a bit more soon (like by automatically updating the number of free zines available using the Gumroad API instead of periodically doing it manually).

hopefully this is a useful data point!

Writing about money on the internet is weird, so this will probably be the first and last zine revenue post, but I’m writing it down in the hopes that it’s a useful data point for others. I thought for a long time that you could only really make money from writing on the internet with ads or sponsorships, but it’s not true!

The goal of this isn’t to say “you should run a business” or anything, just that this is a thing that’s possible in the world and that many developers do really value good educational materials and are happy to pay for them (if you’re one of those people, thank you!)

2019-10-01T12:53:58+00:00


Notes on building SQL exercises

In the last couple of weeks I’ve been working on some interactive SQL exercises to help people get better at writing SQL queries. This is a pretty new thing for me so I thought I’d write a few notes about my process so far!

why SQL is exciting: distributed SQL engines

To me the reason why SQL is exciting is that a lot of companies are storing their data in distributed SQL databases (Google BigQuery, Amazon Redshift, Spark SQL, Presto, etc) that let you run a complicated query across a billion rows pretty quickly! They’re fast partly because they’re designed to run your query across possibly tens or hundreds of computers.

At my last job I wrote thousands of SQL queries to do data analysis while I was working on the machine learning team, mostly ad hoc queries to answer questions I had about our data. I learned a lot of fun tricks to make them faster / easier to write and I’ve never really talked about it!

So I think SQL is a really nice way to go from “I have this sort of complicated question about billions of rows of data” to “ok, that’s the answer, great, I can move on”.

why write exercises: knowledge != skills

This is the first time I’m really trying in earnest to write exercises to teach something, instead of just explanations of the thing. The reason I’m doing this is that I read Design for how people learn by Julie Dirksen and she makes the point that knowledge is different from skills.

She defines a “skill” as “something you have to practice”. And SQL is definitely something that you have to practice if you want to learn it! So I thought – SQL is a relatively simple skill (as programming/programming-adjacent skills go!), maybe I can make something interactive and relatively simple to help people improve their SQL skills!

It’s also, well, a challenge, and I like trying things I haven’t tried before.

how I’m doing it: start with a challenge

I started out doing these SQL exercises in kind of the obvious way: start out with easy exercises, and then make them harder and harder over time to introduce new concepts. But when I watched people trying it out, I noticed a problem – a lot of people already know some SQL, and sometimes they would go through all the exercises without learning anything at all! That’s no fun!

So I came up with a different structure for each section of the SQL exercises:

  1. Start with a “challenge” that tests the skill the section is trying to teach.
  2. If the challenge is too hard, move on to a bunch of easier exercises that teach you the skills you need to solve the challenge.

Since showing is easier than explaining: here’s a draft of a page teaching GROUP BY. Here’s a screenshot of what the initial “challenge” for basic group by looks like:

I think that challenge in particular isn’t very good yet (I have a lot of work to do!) but that’s the idea.

how I’m getting feedback: anonymously track responses

Early on I also realized that I needed to get feedback about which challenges people were finding hard / easy. Every time someone runs a query, I track

I’ve already learned a lot from this, for example:

So basically (in addition to making more exercises) I think I need to spend more time cataloguing where/how people are getting stuck in practice and helping make sure fewer people get stuck.

the tech stack

To build this, I’m using:

I also bought the Refactoring UI book to try to improve my web design skills a tiny bit. I think it’s helped a little so far.

Vue components let me really easily add new challenges/exercises to a page like this:

<Puzzle
   id="count-the-owners"
   title='Count the number of different cat owners'
   description="
   You can use <code>COUNT(DISTINCT column)</code> to count distinct values of a column. (you can also do <code>SUM(DISTINCT column)</code> or <code>AVG(DISTINCT column)</code> but I'm not sure why that would be useful.
   "
   answer= "
   SELECT count(distinct(owner)) AS num_owners
   from cats
   "
   v-bind:table_names='["cats"]'
   >
</Puzzle>

the goal: make something that’s worth $100 or so

What I’m working towards is making exercises & challenges that would help someone with beginner/intermediate SQL skills improve their SQL fluency enough that it’d easily be worth $100 to them. We’ll see if I can get there! I don’t know whether I’ll price it at $100, but that’s my goal for how useful it should be.

The person I have in mind is sort of (as usual) myself 6 years ago, when I’d heard of SQL and could write a basic query but if you gave me a table of VERY INTERESTING DATA I couldn’t really effectively use SQL to answer the questions I had about it.

2019-09-30T12:35:51+00:00


Page created: Fri, Apr 03, 2020 - 09:05 AM GMT