Typed Routes with TypeScript

Although Esper started out as an OCaml shop, we also use quite a bit of TypeScript to build our front-end. “Why We Use TypeScript” would probably be a long post in and of itself, but this post is about how we use TypeScript to implement a small, open-source library that implements statically-typed route checking.

If you don’t care about the why or how, you can check out said library on Github or download the typed-routes package on NPM.

Routing

By routing, we mean taking some action based on a file-like string path. The path in question is usually a URL, and routing can occur on a web app backend, by the frontend for a single page app, or even by a mobile app.

Probably the most popular way of representing routes in JavaScript-land is an Express-style path (which is typically implemented using the path-to-regexp package). An Express-style path looks like this: /path/to/:arg1/:arg2. It matches/path/to/abc/123, and it captures the third and fourth parts of that path inside a param object that looks like: { arg1: "abc", arg2: "123" }.

This works well enough, but it’s not ideal from a static type-checking perspective. In TypeScript at least, /path/to/:arg1/:arg2 is just a string. There isn’t much it can (currently) do to parse out the parameters. Nor is there much it can do to verify that arg2 is an integer.

Chainable Route Definitions

But there’s nothing saying we have to use Express-style paths. Or even (just) strings at all. That’s where typed-routes comes in. With typed-routes, you can write your route with a chainable interface like this:

You can then convert string paths to param objects like this:

And you convert param objects back to string paths like this:

The library also features optional parameters and wildcard / rest routes as well:

Type All the Things

The actual implementation isn’t anything fancy, but the typing takes advantage of mapped types, a relevatively recent addition to TypeScript.

Here’s what the type for our param method looks like. For the sake of this post, these aren’t the exact types being used by typed-routes but they help illustrate what’s going on.

A few explanations:

Our Route interface is defined with a generic type describing how the parameters passed to this type look like. So Route means “a route with a single string param named arg1”.

K is a generic type indicating the key we want to use to describe a given parameter. TypeScript supports string literal types, so it’s capable of recognizing that K is of type "arg1" or "myKey" or whatever is passed to the function.

T is a generic type indicating what the type of that parameter should be. It defaults to string is not provided or inferred.ParamType is another interface for an object that describes how to convert a string path part to another type and back again. It looks like this:

typed-routes provides a few basic types such as IntParam. There isn’t an integer type in JavaScript though, so IntParamreally has a signature of ParamType, although its implementation code ensures that the parsed value is always an integer.

So when we call param("arg1", IntParam), TypeScript infers K to be "arg1" and T to be of type number.

The last bit to explain is the return value: Route

. This is where mapped types comes in. [X in K]assumes that K is a union of string literals like "key1"|"key2"|"key3" and maps each string literal to some value X. Combined with the keyof operator, this lets us create new types using the keys of other types:

In the immediate case though, we know that K is just a single string literal that we want to use as the name of a param with type T. That is, if K refers to "arg1" and T referes to number, then { [X in K]: T} means { arg1: number }.

We use the intersection operator (&) to unite the previous param type P with the new addition. Taken altogether, having a return type of Route

 means that TypeScript understands the param method to return a new Route some param K referring to type T added to the original types. We can then chain these calls together and create arbitrarily long routes with whatever types we want.

We do something similar to implement optional types and rest types.

Note that we extend Route from OptRoute and OptRoute from RestRoute because want TypeScript to understand that a required parameter cannot follow an optional param and that no params can follow a rest / remainder param.

Not All Batteries Included

typed-routes only implements parsing a string path to get a typed params object and stringifying the params object back to path. It doesn’t actually implement routing (i.e. doing something based on a path) itself. It’s more of an alternate to path-to-regexp than anthing else.

But an actual router implementation itself isn’t too tricky. Here’s what a simple hashchange-based router for an SPA might look like:

Trade Offs

When you build a brand new app with TypeScript, it’s very tempting to try to type-check everything. But with web apps at least, you generally have two things working against you: (1) TypeScript is a superset of JavaScript, which comes with a lot of things that are not statically typed, and (2) you’re dealing with user input and/or distributed systems, where at least some of those systems are not things you control.

As such, we’ve occasionally found ourselves bending over backwards to write our code in a way that makes it easier to catch errors with TypeScript, as opposed to simply writing idiomatic JavaScript. This generally takes the form of wrapper functions or objects that have no real purpose from JavaScript’s perspective, but allow us to pass along additional type information. It’s fair to say we might have done that with typed-routes, insofar that it’s a bit more verbose and not as performant as simply defining a bunch of regular expressions.

In Esper’s case though, we encode a quite a bit of information in the routes and paths we pass around our single page app. That information is visible (and easily modifiable) via the URL. The approach outlined here allows us to be reasonably confident that we’re not dealing with malformed data when pulling params from our URL, and that’s a win for us.

Thanks for reading. We hope typed-routes is useful for you. PRs and feature requests are welcome.

A Programmer’s Survival Guide to Calendaring

Getting Started

You decided to build an application that reads, and maybe writes, calendar events. Congratulations! There are a few things you should know.

There are multiple calendar providers. The main players in the enterprise world today are Google and Microsoft. Google offers a single service, Google Calendar, with a single JSON/HTTP API. Microsoft offers also offers a cloud-based calendar service similar to Google Calendar, branded Outlook.com for free users, Office365 for paying users as well as some other names. They are accessible via a JSON/HTTP API. However Microsoft also offers on-premise email hosting, branded as Microsoft Exchange Server, to not be confused with Exchange Online which is yet another brand for Office365. These self-hosted mail servers are still common as of 2017 so there’s a good chance you’ll have to support those users. The API to talk to these servers is called ActiveSync (EAS) and is difficult to deal with. Other large calendar providers include iCloud Calendar and Yahoo Calendar, both supporting the CalDAV standard.

If you want to avoid having to deal with all of these different APIs (and we recommend avoiding ActiveSync especially), consider using a service that exposes a unified interface. We currently use Nylas. It also provides access to email. Another service that we haven’t tried is Cronofy.

None of these APIs is super fast. An API call takes around 1 second, so your client should minimize the number of sequential requests to calendar APIs in context where latency matters. The best ways to make requests faster are:

  • Have the client load and display event lists incrementally e.g., fetch and render a list of events “page by page”, each of these calls to your server corresponding to just one call to the upstream calendar server.
  • Run requests in parallel when possible e.g., when fetching lists of events from multiple calendars before merging them.

As a last resort, you may have to cache some event data. This is hard to do correctly because of synchronization problems. Many calendar providers offer an API (like Nylas’s delta API) that calls a webhook on your server when calendar data changes. We don’t recommend starting off with webhooks for most calendar apps though. Webhooks don’t always fire when you expect them to, and if your server misses or errors on any webhook calls, your local cache may be inaccurate. Our experience that retrieving a month’s worth of calendar events for most providers rarely takes more than a few seconds. So unless your application really needs to frequently read event data for a very long period of time, you might want to avoid webhooks at first.

Typical Calendar Structure

It is safe to assume that all the leading calendar providers expose calendar data as follows:

  • A user can be identified by their email address.
  • Each user has access to a set of calendars.
  • Each calendar has an owner.
  • Each calendar has a globally-unique identifier and a title.
  • A calendar holds a set of events.
  • An event has an owner, who is not tied to the owner of the calendar.
  • Each event has a globally-unique identifier.
  • Each event has a start time and an end time.

Universal Operations Supported by Calendar APIs

  • List calendars accessible by the user.
  • List events that are contained or overlap a given time window.
  • Fetch a single event from its ID.
  • An option to expand recurring events into individual instances or to only return the master of the series.

Various Gotchas

  • Permissions and visibility are managed differently from one platform to another.
  • Not every user sees the same data when requesting the same event.
  • Free-busy access is a read-only mode for accessing a calendar where all details except the start and end time of the events are hidden from the requesting user.
  • Different users may see different data for the same event. In Google Calendar for example, if user A shares an event with user B, then B can edit the description field for themselves but A will keep seeing the original version.
  • Bugs exist, such as malformed timezones or recurrence rules. Can you spot the error in the following recurrence rule?
    RRULE:FREQ=WEEKLY;UNTIL=20171208T235999Z;BYDAY=WE,FR
  • HTML event descriptions are used by some providers (Microsoft), making them hard or impossible to edit programmatically because it’s not clear what’s legal to edit. We haven’t managed to edit HTML event descriptions successfully with Nylas.
  • Calendar applications typically create events with an empty guest list. The calendar owner is added to the guest list automatically only after another guest is specified, resulting in two guests. It makes it hard or impossible to determine who’s the person involved in the event because (1) the calendar owner is not known unless the accessor is also the owner and (2) the event creator is available but they may be an assistant rather than the calendar owner.
  • The guest list of an event needs to be cleaned up. In Google Calendar, some items are not guests but so-called resources (such as meeting rooms) that should be filtered out.

Timestamps and Timezones

Timestamps

All of your machine-readable timestamps represented in text-based formats (JSON, logs, …) should use the RFC 3339 date-time format e.g., 2017-07-25T13:16:42.792-07:00. Such a timestamp represents solely an instant in time. Your application should treat 2017-07-25T20:16:42.792+00:00 and 2017-07-25T13:16:42.792-07:00 interchangeably (with the exception of guessing a time offset, see below). JavaScript:

> new Date().toISOString();
"2017-07-25T21:54:30.611Z"

Note that some applications out there show the number of seconds since January 1, 1970 UTC with simple decimal formatting; this gives us 1501013802 for the previous example. As you can see, a human can’t make sense of such date without tools, which is problematic when inspecting data and logs that are meant to be human-readable.

Timezones

In the context of calendaring, you must deal with timezones because people travel or have calls with guests far away. Luckily, it’s easy to do correctly.

First, you need to know the difference between a timezone and a time offset. A timezone is a region of the world associated with rules that allow deriving an offset with respect to universal time (UTC). Timezones have unique identifiers such as America/Los_Angeles and can also be designated by aliases such as US/Pacific. The former is the recommended form and has the structure Continent/Largest_City_In_Timezone. Three-letter codes such as PDT are ambiguous and should not be used in machine-readable representations of timezones.

Time Formatting

Formatting time should be done with the time offset that matches the location of the context. An offset can be obtained from a timestamp and a timezone. It cannot be obtained from a timezone alone because of possible Daylight Savings rules or because rules may change over time.

In any case, it should be made clear to the user which timezone is used.

  • For viewing the day’s agenda, the current location of the user determines the best time zone.
  • For scheduling future events in a remote location, the local time at that location is most useful.
  • For flights across timezones, it’s best to show the start time and the end time in their respective timezones, as well as the trip’s duration.
  • For a single in-person meeting, the time should be formatted according to local time. If the user lives in Los Angeles but attends a meeting in New York City, local time is New York’s time.
  • For a call involving guests in multiple timezones, use the organizer’s timezone. In this specific case, it may be useful to also display times in the user’s timezone at the time of the call.

While this is kind of nice in theory, in practice we often don’t have all the data needed to follow the suggestions above.

Obtaining an Appropriate Timezone

Your application that involves calendars should ask the user for their timezone during the setup phase. Timezones may be available from the calendar API, but they are associated with each calendar rather than with the user, so if a user has access to multiple calendars, this doesn’t tell us which of the timezones is correct. It’s a good idea to guess the timezone using tricks that may be available, but it requires user confirmation.

A timezone can be obtained from a location using a service such as the Google’s Timezone API. Here are ways to guess a user’s timezone:

  • From the user’s geographical coordinates, if available from the environment.
  • From the client’s IP address, if available; then use a service to resolve it into geographical coordinates and a timezone.
  • From the time offset used by the web browser to format dates. This solution involves computing the difference between UTC and local time. It’s taken care of by JavaScript libraries such as Moment.js and it’s suitable for shifting all timestamps to the browser’s timezone, which may change automatically as the user travels.

Asking the user to pick their timezone from a dropdown also works and it’s simple to implement, but it’s inconvenient.

An example of timezone identifier is America/Los_Angeles. Pacific Time is the name that locals would recognize, so it would be suitable for display in English. Its offset with respect to UTC is -8 hours in winter and -7 hours in the summer and the exact rules are defined in the IANA timezone database.

Timezone definitions are maintained by the IANA (Internet Assigned Numbers Authority), with official releases a few times a year. Those definitions are installed on Ubuntu under /usr/share/zoneinfo and they’re used by the localtime system call which expresses a time in seconds since 1970 into year, month, day, hour, minutes, and seconds… according to the globally-defined timezone.

Yes, localtime doesn’t take a timezone parameter. In order to format a local time for an arbitrary timezone, the default POSIX way is to set the TZ environment variable, call localtime, then optionally set TZ back to what it was. Environment variables being global to a process, this is not thread-safe, but at least it works without installing a special-purpose library.

In bash/sh you can therefore format the current date in the timezone of your choice as follows:

$ TZ=Africa/Abidjan date
Wed Jul 26 00:15:36 GMT 2017

For a date in RFC3339 format with millisecond precision and GNU date, the following function will work:

$ function date_() { date '+%FT%T.%3N%:z' "$@"; }
$ date_
2017-07-25T17:30:09.528-07:00

Note that date --rfc-3339=ns will almost work, except there’s no ms precision for milliseconds, only seconds or nanoseconds (see man date), and it prints a space instead of the T. ¯\_(ツ)_/¯

Here’s how to obtain the current time in Hong Kong, formatted the way we want:

$ TZ=Asia/Hong_Kong date_
2017-07-26T08:30:24.024+08:00

And this is for formatting “86400 seconds (= 24 hours) after January 1, 1970 UTC”, our current timezone being 8 hours behind UTC at that time of the year of the year on that year:

$ date_ -d @86400
1970-01-01T16:00:00.000-08:00

Anil Madhavapeddy – Unikernels: Functional Infrastructure with Mirage OS

OCaml_Meetup_05062015
Anil captivates the crowd with his talk on Mirage OS.

Last week, we had the privilege of hosting a very special guest speaker, Anil Madhavapeddy. You already know him as one of the authors behind “Real World OCaml: Functional programming for the masses”.

Anil showed us some neat developments from the Mirage OS project. Topics of interest included: an informal demonstration of OCaml Unikernels, a tutorial about operating system aspects, and a walkthrough of the TLS stack. Magnus Skjegstad gave us a demo of Jitsu which can spin up servers within milliseconds just-in-time as a request arrives. Anil also showed us the Bitcoin Piñata and invites programmers to tackle this challenge for a chance at free Bitcoins!

Screen Shot 2015-05-12 at 2.59.14 PM
Anil talks about Unikernels.

Below is a video of the entire talk for your enjoyment.

Those who came out to our event also had the chance to meet other OCaml enthusiasts, munch on some food, partake in some drinks, and bask in our uniquely painted officeWe’d like give a big thanks to Anil once again for making the long trip and being awesome as usual.

Thanks to everybody who came out and hope to see you at the next OCaml Meetup.

Maze navigation challenge

We’ll be running this friendly programming challenge continuously for hopefully many months or years. Keep your submissions coming!

At Esper we’re always looking to challenge ourselves in new ways. Some of us find that writing programs and running them is more fun than a bunch of other activities, including but not limited to watching tv and talking to people; although we sometimes do. Anyway. Maybe your current task is a bit boring and you’re looking for new ways to procrastinate. Seek no further, we’ve created this little programming challenge just for you. If you succeed, you’ll win this one amazing mug, which undoubtedly will make all your friends and colleagues jealous of your advanced computer hacking skills.

Esper_mock-up_front

Esper_mock-up_back

This is one mug, not two. Sorry, we will no longer ship the mug to destinations outside of the United States. We’re really sorry and we’re working on an alternate prize we can send to everyone.

Problem

You must write a program for a robot that navigates a closed maze in search for letters that were printed on the floor of the maze. The solution consists of the list of all letters, ordered alphabetically. The letters are capitals (A-Z) and the same letter may exist in multiple locations.

Example

The solution for the following maze is AABQ:

#############
#   A   #   #
# ## ## # # #
# Q# #   B  #
#  #A# # ####
#############

# symbols indicate walls, which are illegal positions. Your robot can navigate on all other cells vertically or horizontally but not diagonally. You can see the 4 neighbors cells (up, down, left, right) around your current cell.

The initial position of the robot can be any non-wall cell (empty or letter), inside the maze.

Navigation API

Your program will read cell information from stdin, then write a command on stdout, and loop.

Cell information is given as one line of 5 characters indicating the type of the current cell and the type of each of the neighbor cells in the following order: current, up, down, left, right. In the example, if the current position is the position of the letter Q, cell information will be given as:

Q#  #

A command is either one of the 4 navigation commands indicating which neighbor cell to move to (U: up, D: down, L: left, R: right) or the key for escaping the maze starting with the K prefix, e.g. KAABQ which terminates the game. Commands are terminated by a newline character (LF).

An attempt to move into a wall or giving a wrong key terminates the life of the poor robot.

Here is a very simple maze:

####
#AB#
####

A successful session for this maze, assuming a start position on cell marked with A is:

 input: A###B
output: R
 input: B##A#
output: KAB

Restrictions

The number of moves is limited to at most twice the number of moves required by our reference solution. This limit as well as the size of the maze remain undisclosed (edited 2015-03-17).

Your program may not consult files or network resources other than stdin and stdout as specified above. It must consist in human-readable source code written by yourself. We will build it and run it on Ubuntu 14.04 (64 bits). Please provide:

  • the list of packages needed to build and run your solution or your best guess
  • a shell script to build your solution in one shot
  • the command to run your solution

Recommendations

Make sure your solution is successful on the sample mazes on this page, and make sure to flush stdout after printing each command when your program runs non-interactively.

Submissions

Please send us your solution to programming@esper.com. All successful entries win the exclusive trophy mug!

About us

We are Esper, a still small company based in the San Francisco Bay Area, and we’re building a service to empower executives with professional executive assistants 24/7. If you share our passion and standards in software engineering and want to make a difference in the world, you should talk to us!

Algebraic Data Types

Algebraic data types are the fundamental building blocks of programs in ML-style languages like Haskell and OCaml. Since they play such an important role in these languages, it is well worth understanding how they work and where they come from—at first, the design may feel a bit arbitrary, but in reality it flows naturally from a reasonable starting point.

Algebraic data types are made up of two components: products (also known as structs, records or tuples) and variants (also known as tagged unions, sums or coproducts). Let’s take a look at each of these in turn, how they can be combined and how they’re related.

Products

Products are a construct that appears in most programming languages: it’s a type that contains multiple values. The simplest example in OCaml would be a pair (just like an ordered pair in mathematics):

# let pair = (1, 2);;
val pair : int * int = (1, 2)

This pair has the type int * int which means it has two members, both integers. In prose rather than code, this is often written as int × int because it corresponds to a cross-product of sets. Fundamentally, we can do two interesting things with a pair: get the first element or get the second element:

# let first (x, y) = x;;
val first : 'a * 'b -> 'a = <fun>
# let second (x, y) = y;;
val second : 'a * 'b -> 'b = <fun>

In OCaml, pattern matching makes the definitions pretty visual. The functions first and second are often abbreviated as fst and snd or π₁ and π₂ (π is used because these are projection functions).

Why We Use OCaml

INTRODUCTION

According to the TIOBE Programming Community index for June 2014, the following nine programming languages make up over 60% of the total popularity ratings: C, Java, Objective-C, C++, C#, PHP, Python, JavaScript, and Ruby.

Here at Esper, we have chosen to use the OCaml language for the server-side backend that implements most of our application’s functionality. OCaml includes many features that are not available in the more mainstream programming languages listed above, and we believe this gives us a competitive advantage. But even among Silicon Valley startups, OCaml is an unusual choice. The purpose of this post is to explain the benefits of OCaml and compare it to other languages. We hope to convince readers, especially other developers, to consider adopting OCaml for their projects as well.

 

PROGRAMMING LANGUAGE FEATURES WE LIKE

Here are some of the most important features of the OCaml language that we consider useful, or even essential, for building high-quality, reliable software:

First-class functions and immutable values

It is not always clear what people mean when they say “functional programming”, so we avoid this term and state directly what we want. Functions should be full-fledged values that can be created within other functions, stored inside data structures, passed as arguments, and returned as results. A function should automatically capture the relevant parts of its environment at the time of creation into a closure.

Most recent programming languages support these features, but they may be little used in idiomatic programs. One reason is that a function’s behavior is more difficult to predict when the values in its environment can be modified after its definition. Such modification could cause the function to produce different results when called with the same arguments, contrary to the expected behavior of a function. Mutability can be a useful tool in some situations, but it is often unnecessary and potentially harmful.

Meet Oblivion: Inline HTML templates for JavaScript

Edit (July 2017): We now use React with TypeScript.

So at Esper we made this tool to make our lives easier.

Oblivion is a preprocessor for JavaScript source files. It allows the programmer to embed raw HTML code into JavaScript with very few modifications. Only local variable names need to be inserted, allowing these HTML elements to be modified programmatically – and safely.

Have you ever wanted to copy-paste HTML code directly into a JavaScript string?

var s = '
&lt;div&gt;
  &lt;span class="example"&gt;Welcome!&lt;/span&gt;
&lt;/div&gt;
';

Oops, that’s invalid. JavaScript doesn’t support line breaks in string literals. Instead you would have to write something like:

var s =
 '&lt;div&gt;'
+'  &lt;span class="example"&gt;Welcome!&lt;/span&gt;'
+'&lt;/div&gt;';

This required more edits than we wanted, but why not. Now something we often do is modify child nodes based one some conditions. We use jQuery to produce a dom node from a string, maybe as follows:

var view = $(
 '&lt;div&gt;'
+'  &lt;span class="example" id="msg"&gt;&lt;/span&gt;'
+'&lt;/div&gt;';
);

var text = isNewUser(user) ? "Welcome!" : "Hello!";
$("#msg").text(text);