I recently had to move parts of an app from Heroku to Disco. The app runs under an enterprise contract on Heroku, that allows for a fixed amount of resources. Dyno units as they call them. The usage increased temporarily due to a data migration that’s taking a few months, and Heroku checked in to see if the size of the contract should be increased.
Since the enterprise contract with Heroku still runs for another 6 months, I decided to move only a few workers to Disco, and leave the rest on Heroku for now.
Since the app is connected to a Postgres database and RabbitMQ, it makes sense to put the new Disco server in the same datacenter as the app running on Heroku.
You can find the region on Heroku using:
heroku info <APPNAME>
One of the fields is the region:
...
Region: us
...
You can then find that region in the list of regions:
heroku regions --json
{
"country": "United States",
"created_at": "2012-11-21T20:44:16Z",
"description": "United States",
"id": "59accabd-516d-4f0e-83e6-6e3757701145",
"locale": "Virginia",
"name": "us",
"private_capable": false,
"provider": {
"name": "amazon-web-services",
"region": "us-east-1"
},
"updated_at": "2016-08-09T22:03:28Z"
}
In this case, it’s AWS us-east-1.
The project already uses Docker for various purposes: Docker Compose for development, stagings run on Disco, the CI builds use Docker images. It was pretty easy to create a minimal Dockerfile that only installed the minimum for the workers I was moving to Disco. And then, since the project has many Dockerfile with different names, I just had to specify which one to use in disco.json:
{
"version": "1.0",
"services":
{
"worker_general":
{
"command": "worker queues=default,rush"
},
"worker_background":
{
"command": "worker queues=background,rush"
}
},
"images":
{
"default":
{
"dockerfile": "Dockerfile.prod.worker"
}
}
}
Similarly, since the project already runs staging environments on Disco, we already had a disco.json with a different list of services.
Disco let’s you specify another file by setting the environment variable DISCO_JSON_PATH to the name of the file to use instead. E.g.
disco env:set \
DISCO_JSON_PATH='disco.staging.json' \
--project staging-one
I exported the environment variables from Heroku using:
heroku config --app app-name
I removed a few magic Heroku variables, like HEROKU_APP_NAME and manually edited the output with VS Code multi-cursors to get a command like this:
disco env:set \
VAR_ONE='value one' \
VAR_TWO='value two' \
VAR_THREE='value three' \
--project prod-workers
When creating the project, I specified the branch I wanted to use (production in this case, but it could be main or master). But to avoid making the branch dirty by pushing (or force pushing) many small tweaks to get everything right, I pushed to antoher branch, and manually deployed using
disco deploy \
--commit 788ec8b1a1429f18cf28cf4937a54d9a57f9ae8a \
--project prod-workers
And when everything looked good, I squashed all the commits and merged to the correct branch (production).
A few months ago, a prospect client asked for an estimate to build an MVP. That MVP would have access to users bank accounts and be able to perform real ACH transactions on their behalf.
The client wanted to build something extremely quickly to be able to prove the market fit of his idea, which really does make sense. So I crafted an estimate that would include the desired minimal feature set. He was surprised by the amount of time needed to build that MVP (a few weeks).
For many reasons, I think he is a smart guy with a great business idea. However, he had some misconceptions about what it means to build a web application.
For a lot of people, the difference between a website and a web application can be unclear, and that’s really understandable! I have to explain it from time to time to clients, friends and family members.
So, that client asked for some explanation. Why would it take so much time to develop his app?
First of all, let me explain some differences between a website and a web application. You can use both through your web browser, like Chrome, Safari or Internet Explorer, but conceptually, they are not the same.
… content that you consume, that you read. Here are some examples:
… software that you can use to do things. In most cases, you can log in. Here are some examples:
It’s not always super clear. Here are some examples where the line is blurry:
The lifecycle of a website is not the same as the lifecycle of a web application.
The lifecycle of the code base of a website is similar to a brochure. Some process (meetings, design, production, etc.) leads to a final product. The brochure is printed, ready to be read. The website is online ready to be read.
At some point in the future, it will get too old. A new one will be produced. The same process starts over (meetings, design, production, etc.). The new one is ready, the old one is discarded.
The code base of the website during his whole life will not change much. A new social network appears, we must add an icon. The client did not realize he needed a page that let people know how to get to the factory, a new page must be added, with a new link to that page on the home page. If the content has to be changed (more articles in a blog, new products in the catalog), the code base will rarely change. The changes will rather be done in the CMS without touching the code base.
You rarely find bugs in a website because the code is simple: it gets data from a database and shows it in a page, it affects how things are presented (the color for example), it performs simple animations. If there are any bugs, they are generally really simple to fix and deploy. Even someone unfamiliar with the code base will be able to do it easily.
The lifecycle of the code base of a web application is extremely different. It really is a software dressed like a website.
You can rarely foresee every aspect of a software before actually using it. There’s no solution, it’s like that. Software is intangible. Talking about it without touching it only make things worse. So, the way successful software is generally done is by delivering the minimal feature set that has value for the users, and then augmenting/changing that feature set as time goes.
It is rarely a better idea to create a new version of a software and discard the old one. For example, if you discover that your timesheet application would also need to do some accounting and that users should have the ability to add details about the hours worked, those features will be added to the existing application, instead of creating a new application.
This requires a code base that is evolvable.
In order to make the code base elvolvable, the first thing that comes to my mind is automated testing. There are many types of automated testing: unit testing, functional testing, integration testing… I won’t go into the details of each type. Some types of testing may be more appropriate than others depending on the project.
An automated test is code that will reproduce a use case in an automated
manner and verify the result is correct. It could be code that tests that when
feeding 10 to the function compute_taxes, it actually returns 1.52.
Or it could be remote controlling a real browser to simulate that a user logs
in, fills his daily timesheet then logs out, making sure that the changes were
saved correctly and the side effects (update a report, etc.) were done
properly.
To someone not familiar with writing software, it may seem like trivial test examples that do not really add value. I’ll give you some examples where those tests will come in handy.
Three years after the software has initially been deployed, and the
person that wrote that code does not work there anymore, the business
expands and the function compute_taxes must be updated to compute the
taxes according to the state where the product is sold. Instead of receiving
only 10, it will also receive NY or CA, and the function should return
the tax amount according to the state. You can update the existing tests to
include the state that was implied before (for example, CA), that make
sure the working case still works after the modification, then add new tests
to also test other states. The developer does not have to manually test each
cases. He just runs the tests, and that way, he is sure all the application
still works correctly.
A bug report comes in. The taxes for NY are not computed correctly. If the
code base is not covered by tests, the developer will look at the code, fix
the bug, and test the application manually to see if the bug from the report
he received is now fixed. It seems to be fixed, so he pushes his change in
production. But by fixing the bug, he also introduced another subtle bug,
that made the tax calculation wrong for every other state. If the code base
is covered by test, the developer can run the test suite and catch the error
right away. Also, since there was a bug in the first place, the developer
will write a new test to make sure this bug never comes back.
A new requirement is that the timesheet includes a new box to enter the overtime, separated from the regular time. The existing browser test should still work. It may look like a very simple change, but it’s often in those unexpected cases that something breaks somewhere else. In this example, loading the reports could make the application crash. The browser test will find it really fast.
As most bugs are discovered as the code is being written, instead of receiving bug reports, fixing them as they are introduced is much faster. Also, the test suite is there to catch other bugs that the fix could introduce.
To summarize, automated testing really helps to keep a code base evolvable.
When building a website, pre-made modules/plugins are often useful. Need to create an animated slideshow? There’s a jQuery plugin for that. Need a private section on your website where only authenticated users can upload files? There’s a WordPress plugin for that.
When building a web application, using those pre-made modules is often not the optimal solution for many reasons.
When building a website, it makes sense to use those modules because it makes it so much faster to ship! Not using it could mean 10x more time needed. Since you don’t need to maintain that code, you would rather code 9 new websites over time than investing time in the first one to make it really maintainable by writing custom code. The first one will be thrown away in 2-4 years anyway.
When building a web application, you can’t afford to lose the flexibility that pre-made module will take away from you. It could be acceptable in some cases, but probably never in core parts of the application. For example, if there is an animated slideshow in the application, it’s probably OK to use that jQuery plugin. Replacing it with another one in the future will be easy since there is nothing that really depends on it.
However, using pre-made “users” modules, or “administration area” modules, is often a really bad idea when building software, because it will lock you in using those modules forever, with their limitations.
When building websites, we sometimes have to access external APIs (data access points that software can use). Most of the time, for a website, it is very straightforward:
You get the pattern: read data from one access point, output it on a page in the desired format. Even better: there are often libraries or code examples that do the data-fetching for you. Copy-paste and tada!
When building a web application, the consumption of those APIs are generally not that straightforward.
Also, since those APIs are external services, it can be hard or impossible to write automated test suites (see above) that incorporates them. There are few reasons for that:
Since you can’t include them in your tests directly, you have to create something that behaves like them for the context of the test cases. It’s called a “mock”. During the tests, your application will talk with that mock instead of talking with the real API endpoint. It’s a mini-version of the external service. For example, if you mocked a payment provider, it could accept all credit card numbers except for one, that will be used for testing when the user enters an invalid number. The payments could always work for all credit cards but one number, where the payments always fail. Or it could refuse payments of 29.99 on one specific card number. Or it could only allow for 2 payments of 29.99 on one specific credit card number, then bounce the next ones.
Since you wrote that mock and you host it, you have complete control over it:
Those mocks are really needed and are really useful, but they also take time to write.
Deploying a website is generally done by uploading the new files to the server that hosts those files. The developer sees that it looks good on when he worked on it on his laptop, then he uploads it. Or even faster, some directly edit the files on the production server.
Deploying a web application is generally more involving. You push your work in a source code repository (think GitHub), a continuous integration (CI) server builds the source (more on that below), runs all the tests, and allows you to deploy if all the tests are green. Doing the setup of the CI server (how it pulls the code from the repository, how it builds, how it runs the tests, how it deploys) requires some time.
Also, there’s often a staging server to maintain, where a copy of the application is running fake accounts. That way, you can demo new changes for approval without deploying to the production server.
You must use those good practices when developing an application because otherwise, bugs that did not occur on the developer’s laptop emerge in production. The exact source of the bugs is always a surprise, but it includes:
When a website has been deployed, it rarely happens that the database schema will change. For example, it will rarely happen that, suddenly, you need captions in the pictures of the slideshow, while you did not need it when the website was first published. Also, when that happens, you can connect to the database, write the command that adds the new column, and execute it.
For an application, a change to the database schema is everyday life. But, unlike the website, you can’t just connect to the database and write some commands. It needs to be executed in a reproducible manner. Some reasons are:
This is why we need database migrations: a system to do database schema changes in a reproducible manner. The simplest form could be a folder containing SQL files. But it’s generally in the form of code that apply those migrations for you when you ask for it.
Doing the setup of the migration system and writing proper migrations is another task that requires time and is not needed when building a website.
When building a website, you may want to minify your resources. That means that
you take your JavaScript and CSS (style sheets) and use a minifier that will
reduce the size of those files by using many strategies, like change long
variable names like numberOfChildren to b and removing line breaks. Also,
there are new languages that developers sometimes prefer to use instead of
JavaScript and CSS, that will be compiled by a compiler to become JavaScript
and CSS. CoffeeScript and SASS are two examples. When you build a website,
building those resources is something that could be done manually. For example,
just before you deploy, you minify all your resources by using the command
line.
When building a web application, that process has to be automated. Some reasons are:
Doing the setup of the automated minification/compiling is something that also requires time and could be skipped when building a website.
Until now, I talked about a website vs a web application. I used the website comparison, but it could probably also be valid with a prototype application that you only want to use to discover if your idea has a market fit.
If you intend to throw away your prototype application right after you discovered if your idea has, or not, a market fit, you could consider prototype applications to have the same characteristics that a website would have. In theory that is true. You will never need to maintain the code, so you don’t have to bother about making it evolvable.
However, there are some more considerations before you can go that path.
First, it’s really easy to fall in the trap of thinking that you will discard your prototype application really soon. In practice, the code base rarely gets thrown away. A total rewrite is extremely rare. The shortcut you take now will stay with you for a really long time. It will cost you money and agility.
Second, it’s not always possible to do a quick prototype like that. I’m going back to the example I started with: the client wanted an application that would have access to users’ bank accounts, see transaction history and other data and perform ACH transactions. This is a case where the application will store really sensitive data and would be able to perform really sensible operations. A bug or a security breach would be a disaster for the users, the client, the developer and everyone in between. In those cases, you have to write reliable software. That means using the best practices mentioned above, like automated testing.
Like I was saying in the introduction, I’m currently pushing Grawl to bring it to an MVP level. In this case, I’m my own client. I would really like it to get there faster.
I’m currently writing the code that handles the credit cards. It seems quite simple at first, but a lot of use cases have to be covered by the automated tests. Otherwise, I could be charging too much to someone, leak private data, allow someone to use the application without paying the correct amount, etc.
It takes time.
But building that part properly is required. That’s not an optional feature.
This was a super long article, but I think writing a shorter one would not have covered enough to be taken seriously.
Building software the right way requires time, but everyone wins in the end.
]]>Idempotence is a mathematical concept that should be understood by all developers.
An operation is considered idempotent when doing it more than once is the same as doing it once.
For example, multiplying by 1 is idempotent.
x * 1 == x * 1 * 1
Multiplying by 0 is also idempotent.
x * 0 == x * 0 * 0
The key concept to remember is that applying the operation once can have side effects, but applying it more than once will not do anything more that what was already done the first time.
Assignment is idempotent.
x := 4
You can assign 4 to x as much as you want, x will still be 4. But assigning 4 to x one time is different than zero time.
HTTP verbs can be classified as idempotent or not.
DELETE is an idempotent verb. No matter how many times you do it after the first time, it will give the same result as the first time. For example, DELETE /users/4/contacts/3 could remove your contact with the ID 3. If you call it again, that contact has already been removed and nothing more should happen.
GET is also idempotent. In fact, it’s more than idempotent. It is considered a safe method. Safe methods can be compared to multiplying by 1. Doing it zero, once or more times should have the same effect. All GET does is get a resource. For example, you should never use normal links to delete resources.
POST is not idempotent. Every time you do it, you can expect a side effect to happen. For example, every time you POST a contact form, an email is sent.
When it comes to APIs, that concept is well understood by consumers and providers. Designing around it will result in least astonishment.
See Wikipedia: Safe Methods and Idempotent methods.
Let’s say you build a web app to manage events. You can add people to the invitees of an event. In other words, an event has many invitees. For quicker response time, you decided to send all emails from a worker. So when a user finalizes an event, a message is queued. A worker gets that message and sends the invite emails for that event.
This is a really common pattern, and if you are not doing it that way, you should really start to.
You realize that you had an SMTP problem and all your emails have not been sent for some time, and it’s not even showing up in the logs! You think “oh, well, I’ll just call the function to send the emails again”, but you don’t know for which events you should call that function.
Here comes idempotence.
For the email example, every time you send an email to an invitee, you can keep the datetime in the database, in the event invitee row. If it has already been sent (sent datetime is not NULL), don’t send it again. As easy as that. Also, you might want to check if the event is finalized. If not finalized, do nothing.
More generally:
send_event_emails(event_id)) that will queue the 5 other jobs (e.g. send_event_email_to_invitee(event_id, invitee_id)).You realize something went wrong? You can always call your function to send emails on all events. Still crashed when half of the emails were sent? Fix what was wrong and just call it again. Also, it’s easy to inspect what emails have not been sent yet. Bonus, you can do some intelligence with the datetimes (how many emails a day do we send? what are the peak hours?).
Also, some message queues don’t garantee that a message will be delivered only once. Amazon SQS is that way. Your workers should really only do idempotent tasks.
In the same spirit as the worker example above, when you do an SQL migration, do it in an idempotent way when possible.
For example, you decide to split the user table in two tables. One for basic informations (users) and one for all details that are not always important (profiles). You put a foreign key user_id in the profiles table. You have a migration that takes every row in users (SELECT * FROM users) and inserts a row in profiles with user data. You run it, and well, it crashes midway after 1 hour, because of some NULL value you didn’t think about. You fix it and run it again, but you then realize that some users have already been processed and have two profiles.
The idempotent solution: instead of SELECT * FROM users, you can just select the users that don’t have a profile row. That way you can run it as many times as you want. It will only process the few users that have not been processed yet. A big advantage of that method is that you can leave your app running in production while you do the migrations. When you are ready to deploy the new code that uses the profiles table, you can call your function again to make sure the latest users that signed up are also migrated. That example is not so great, because a user could change some information in the users table during the migration, but I guess you get the point.
You have an application where each user has many documents. They can search their documents by tags. Tags come from many sources. The title of the documents, the folders, actual tags, the names of the authors, etc. You decided to keep a table named tags where you keep all the tags for every documents, it looks like this: id (hash), tag (actual text of the tag), document_id (foreign key to the document). When you add an author to a document, it does an insert in the tags table. When you remove an author, it finds the good tag and removes it.
Someday, you see in the log there was an error every once in a while when inserting a tag because of an obscure character encoding issue. You fix the issue and deploy the new code. However, there are a lot of missing tags and you have no easy way to fix it manually.
Instead of just having functions of the type add_tag_for_new_author, you should have a function of the type update_tags_for_document. When you call that function, instead of just adding a tag for the author that was just added, it checks all the document, rebuilds the tags list and makes sure that the correct data is in the database. That way, the tags table is really managed as it should be: a cache. You could delete all rows from that table and just call update_tags_for_document on every documents. It takes 2s to update the tags for a document? Let the worker do it, queue a message.
If you were not aware of idempotence, I hope I convinced you to use it. Also, please note that I kept the examples simplistic for educational purpose.
]]>Some basic explanation of a quadratic Bézier curve. You have a
start point P1, an end point P2 and a control point C. Both the x and
y of each point on the path is dependant on t, which will vary from
0 to 1. t=0 at the begining of the curve and t=1 and the end of the curve.
You draw an imaginary line between P1 and C (P1C line) and
another one between C and P2 (CP2 line). For each value of t, you mark
an imaginary point on P1C and CP2. The point on P1C is at t of the line,
starting at P1, and the point on the CP2 line is at t of the line,
starting at C. Let’s call those points C1 and C2.
Then, you draw an imaginary line between C1 and C2, and you mark a real
point at t of the C1C2 line.
You repeat the procedure for every value of t, where t varies from 0 to 1.
All the points you marked on the line C1C2 are showing the quadratic Bézier
curve.
Here’s a little animation I made to explain it better:
t = 0
You can have a look at the source for a complete understanding, but here’s the important part, edited for clarity:
// constants
var CANVAS_WIDTH = 301;
var CANVAS_HEIGHT = 301;
var p1x = 20;
var p1y = 200;
var cx = 140;
var cy = 20;
var p2x = 280;
var p2y = 280;
// basic setup
var $t = $('#bezier-example-1-t span');
var animationCanvas = $('#bezier-example-1 .animation').get(0);
var animationContext = animationCanvas.getContext('2d');
animationCanvas.width = CANVAS_WIDTH;
animationCanvas.height = CANVAS_HEIGHT;
var curveCanvas = $('#bezier-example-1 .curve').get(0);
var curveContext = curveCanvas.getContext('2d');
curveCanvas.width = CANVAS_WIDTH;
curveCanvas.height = CANVAS_HEIGHT;
curveContext.strokeStyle = "#777";
curveContext.lineWidth = 2;
curveContext.beginPath();
curveContext.moveTo(p1x, p1y);
curveContext.stroke();
setInterval(updateDemo, 1000/30);
var t = 0;
var d = 1; // direction
function updateDemo() { // called 30 times/second to animate
if (t > 1 || t < 0) {
d *= -1; // change direction
curveContext.clearRect(0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
curveContext.beginPath();
}
t += 0.01 * d; // continue moving
$t.html(Math.round(t*100)/100);
// update values
var c1x = p1x + (cx - p1x) * t;
var c1y = p1y + (cy - p1y) * t;
var c2x = cx + (p2x - cx) * t;
var c2y = cy + (p2y - cy) * t;
var tx = c1x + (c2x - c1x) * t;
var ty = c1y + (c2y - c1y) * t;
animationContext.save();
// clear old sketch
animationContext.clearRect(0, 0, CANVAS_WIDTH, CANVAS_HEIGHT);
// draw new line
animationContext.beginPath();
animationContext.strokeStyle = '#aaa';
animationContext.lineWidth = 1;
animationContext.moveTo(c1x, c1y);
animationContext.lineTo(c2x, c2y);
animationContext.stroke();
// draw points on lines
drawPoint(animationContext, c1x, c1y, 2, '#0f0');
drawPoint(animationContext, c2x, c2y, 2, '#0f0');
// draw point on curve
drawPoint(animationContext, tx, ty, 3, '#f0f');
animationContext.restore();
// draw the new Bezier curve segment
curveContext.lineTo(tx, ty);
curveContext.stroke();
}
The code is on Github.
]]>