In Feb/March 2018, we converted most of our frontend codebase from JavaScript to TypeScript. We did this in one go, to a codebase of about 40k lines of JS/JSX code.
TypeScript has gained huge amount of popularity since 2018 and seems like an obvious choice today for many projects. It has taken great steps forward as a language, too, but the core features were there already years back. This post discusses why we did the change, how it was done, and what we learned on the way.
Alloy.ai’s frontend is fairly complex, and our users leverage the platform for making business-critical decisions. In 2018, we had only existed for a couple of years, had limited test coverage, and needed to build more safety nets around our frontend codebase. There are few compile-time safety nets with JS.
At runtime, we could leverage a couple of things. We used immutable.js to force immutability as a characteristic of all of our data structures. This hugely helped reason about what the shape of these structures is. However, there was also a major performance cost to using these structures – parsing the raw JSON as it came from the backend and converting it to immutable.js Map objects, for instance, is not a free operation. When you process and display large, tree-like data sets in the frontend, the performance penalty for processing some of these was dozens, sometimes hundreds of milliseconds, and there would be user-visible performance issues. However, this safety net was important, and it worked well.
Another thing we could do at runtime was to add custom prop types to our React components, for development-time verification. This also had a major impact on performance, but only at runtime and when running a development build. So as long as you were running tests, automated or manually, you’d be able to identify breakages with the data model.
Running data checks at runtime makes little sense though. For someone coming from a C++ / Java world, all of this seemed like a set of huge workarounds for the actual problem – lack of typing with the language itself. You shouldn’t need to have a huge JS library like immutable.js to work around the lack of this construct. Similarly, for nested types, which we have a lot of, you don’t want to do some kind of runtime conversion and processing if you can help it. In case of 3rd party APIs, you might need to do deep verification, but in case of Alloy.ai’s, it’s enough that we know what the version of the API is, and the language’s type system can do the rest.
In 2017 when we first started evaluating the need to switch the language, it felt like Flow and TS had more or less equal standing. It wasn’t immediately obvious at least to me which one to pick at the time, but the problems with the codebase started to peek their ugly head already. In early 2018, the situation seemed to be much more in TS’s favor, and the ecosystem was also more mature. The choice was way more obvious a bit later. Sometimes it’s worth it to wait it out.
As discussed above, we relied on immutable.js for strict immutability constraints, and on prop types-based typing. There is also a library called react-immutable-proptypes that brings these two worlds together in a neat way. We started using this library and as a result, we had “strict” types for most of the types we shuffled around in the frontend. Here’s a couple of examples of how that looked like:
const Attribute = ImmutablePropTypes.mapContains({
type: PropTypes.oneOf(['PRODUCT', 'LOCATION', 'TRANSACTION', 'DATE']),
id: PropTypes.number,
name: PropTypes.string.isRequired,
valueType: PropTypes.string.isRequired
});
const AttributeInstance = ImmutablePropTypes.mapContains({
attribute: Attribute.isRequired,
graphContext: PropTypes.oneOf(['ORIGIN', 'DESTINATION'])
});
As you can see from the above example, these prop-based types could be set up in a hierarchical fashion; an AttributeInstance would leverage the nested types. This was fairly powerful, actually – if changing the language is not an option for you, this is a solid approach and much safer than using e.g. PropTypes.object type.
At first, we explored getting rid of immutable.js on the outset together with this change, but that quickly proved unfeasible. We would have needed to rewrite the whole frontend application, and doing a language change as well as a massive refactoring would have been foolish. The ideal state would have been to not have to do the JSON -> Immutable. js conversion but it looked like we’d need to stick with that step.
We were using Immutable.Map initially but its keys are not possible to type in TypeScript in the same way that the immutable-proptypes allowed us to do. However, we discovered the Record type in immutable.js, which lends itself well to to be used as a basis for immutable, typed, map-like objects. Below is what we ended up doing:
interface IAttribute {
type: 'PRODUCT' | 'LOCATION' | 'TRANSACTION' | 'DATE';
id?: number;
name: string;
valueType: string;
}
export class Attribute extends Record({
type: 'PRODUCT',
id: undefined,
name: undefined,
valueType: undefined
}) {
static fromJS(value: IAttribute) {
return new Attribute(value);
}
static fromArray(array: IAttribute[]): List {
return T.fromArray(array, Attribute);
}
}
interface IAttributeInstance {
attribute: Attribute;
graphContext?: 'ORIGIN' | 'DESTINATION';
}
export class AttributeInstance extends Record({
attribute: undefined,
graphContext: undefined
}) {
static fromJS(value: IAttributeInstance): AttributeInstance {
return new AttributeInstance(value).merge({
attribute: Attribute.fromJS(value.attribute)
});
}
static fromArray(array: IAttributeInstance[]): List {
return T.fromArray(array, AttributeInstance);
}
}
There are few things to note in the above code snippet:
Every core type in Alloy.ai then followed the same convention, so you’d always have the static fromJS/fromArray methods available in these types, which helped create a consistent, predictable development experience. One of the initial goals was to keep consistent for another reason – to eventually have a code generator do this work for us.
The way this API was used was simple:
get('/api/foo').then(AttributeInstance.fromArray).then(attrInstances => ...);
and that’s it!
A convention-based approach like this was not ideal, but it worked fine for the hardest problem we had – keeping the data model consistent.
Our codebase was about 43 000 lines of JS code at the time when we did the language conversion. It’s not a huge codebase by any means, but it was also of size that was hard to refactor into using TypeScript in one go. However, the real challenge with migrating to a language like TypeScript is not turning on TS compilation or adding a transpiler – that’s an hour or three of work. The real work is actually doing the migration, but it doesn’t seem feasible to do that all at once.
What we did, however, was exactly that – doing most of the work in one go. There were some areas of code that we left out from the initial conversion pass, but about 90% of the codebase was done all at once. One of our core values is focus on what matters – doing the highest-priority work. From a product perspective, one could say you should rather work on the product than tinker with the codebase’s language and typing system. From an engineering perspective, however, you need to build a solid foundation for that product work to happen too.
As such, doing a three-week, one-off effort was a strategic choice for us, and it paid off. We put frontend development on pause for those three weeks for the most part to avoid rebasing pains, and got most of the way there.
What we included in the initial conversion was:
This is roughly what we excluded:
What happened after the initial conversion with these excluded items is interesting and one of the most powerful lessons learned in this conversion effort.
The pull request that contained the change was huge, because it touched most of the codebase: over +/- 15k line diff. Reviewing a PR of this size is not really possible, but it was still possible to review the key parts. What we did was:
To double down on the code reviews, it’s okay to take the time to review a fundamental change like this in detail. The whole team would need to adapt a new language, and the initial change is setting a standard for the team to follow. The change doesn’t have to be perfect, but it should be good.
Most of the syntactic changes with a language conversion like this were luckily mechanical and not that exciting. Regex search/replace goes a long way! Making each robotic change in a separate commit and adding a note of what was done was also helpful in minimizing both the actual work needed, as well as the need to review those parts in detail.
After an initial three weeks of intense, focused conversion effort, we pushed the TS conversion and it was great! Even at the time in 2018, TS 2.0 was a fantastic language, and now in 2021, it’s even better. I’d thoroughly recommend it to anyone. At this point, we started to work again on the Alloy.ai product using the fancy new language, and interjected some of the remaining conversion work in between.
It took us another 9 months to actually call the conversion complete.
The noImplicitAny flag prevents you from adding code that has no typing and for which typing cannot be inferred at compilation time. Having that on in a TS project is very important.
Specifically, it took us 9 months to be able to turn on the noImplicitAny check. We shipped with 2400 warnings, and reduced it in a matter of weeks to under 800 warnings. But after that, things stalled, as other priorities took hold. It wasn’t the case that things got substantially more complicated towards the end, but rather that we no longer allowed ourselves to focus on a single problem, ship it, and move on. Instead, we worked on adding typing information in a context switching mode, doing many other things in between.
Looking back as we completed all the work years ago, it was probably two weeks worth of extra work in total to actually fully complete the migration. This is not significant compared to the cost of context switching, and we should have absolutely planned this even more carefully – we would have sped up our time to complete the conversion, increased compile-time type safety, and sped up the work later.
There was nothing particularly complex about converting the redux store to be TS-typesafe, but it was scoped out initially because it was a clear enough architectural piece to leave as-is.
As a separate issue, a handful of our own components were really tricky to implement in a type-safe way and needed to be refactored first to avoid duck typing – different behaviors based on the shape of the object. TypeScript has this beautiful union type system, so we were able to leverage that later, but getting most of the way there meant that we had to leave a few of these harder nuts to crack until a few weeks (or months) later.
This was an issue in 2018, but probably not today. Some of the npm packages on which Alloy.ai’s frontend depends were missing type definitions at the time. This was also preventing us from turning on the implicit-any checks. This problem was solved in two ways: waiting for the support to drop into DefinitelyTyped, and writing our own type definitions for the npm packages that were missing them. There were only a few, so this was not a major problem.
We did not set this as an initial goal, and we didn’t get there either. This is still one of the key deficiencies of our frontend architecture. The path forward for getting the strict null checks to work is similar as the original implicit any journey. It’s still achievable, but our codebase is substantially larger now, three years later, and that makes achieving this goal much harder now than what it would have been, had we attempted this at the outset.
In short, while it might seem counterintuitive to put time aside to do all of this refactoring (and it might also feel exhausting to do all of it at once!), one of the biggest lessons learned for us was that we should’ve kept that focus for a bit longer. Else, you’ll end up revisiting this for months, or years, depending on the size of your codebase.
We love immutability in our code and immutable.js is a fantastic library in a JS-only codebase. However, with TypeScript, it sort of felt like it’s getting in the way. Below is an assorted collection of pitfalls we ran into.
Record has a few curious properties, as per its design:
filter() / map() / isEmpty() are not part of the API. In Java, it doesn’t make sense that you can filter the keys of your POJO, and it doesn’t make sense with TS either. However, if your mental model of the Records is that of a JS object (i.e. anonymous map), you need to adapt that thinking.
merge() ignores keys that are not part of the Record definition. This means that you cannot magically convert one object into another, which is good. However, it could be seen as equally magical that the extra keys are simply ignored silently.
It has default values for all of its properties. Should I use undefined as a default for everything? Null? Empty string, perhaps a numeric value? How does this work with strict null checks? Why does the default value get applied in case of undefined? So many questions. It’s an unfamiliar mental model for most engineers.
We need a conversion function as discussed above to convert raw JS payloads into Records, which is expensive.
For us, there was the JS Set and the immutable Set to consider. The same thing with the Map, too. The JS Sets and Maps are mutable by default, which is a no-no for us. Furthermore, the accidental mixing up of these container classes could cause typing issues or depending on your TS settings, cause the typing to silently fall back to any sometimes. This was mostly a namespacing issue as we’d import {Map, Set} from immutable, shadowing the JS types. Not a major problem, but rather a subtle one due to the nature of shadowing.
This was a big problem for us. We used the *In selectors from immutable.js for most of the nested object accesses because they shortened our code and were null-safe. However, these selector methods were not type-safe at the time and they’re still not typesafe today. TypeScript was not ready for this at the time either. With TypeScript 3.7, it supports the optional chaining operator, which would allow foo.get('bar')?.get('baz') and so on, allowing for type-safe single-line expressions where desired.
We don’t use immutable.js for our core data types today so we don’t need these operators either, but at the time, these were frustrating. There were also forks of immutable.js that added typing for a fixed number of arguments for these function variants. However, that brings us to the next point.
In 2018, immutable.js was in 4.0 RC-9, and had been in a release candidate state since autumn 2017. Today, in 2021, it’s still in release candidate state, and it’s fair to say that it’s likely to be unmaintained at this point. It’s a stable library, but it was already apparent at the time that if you ran into issues with it, you’d need to fork.
We built our core data model foundation on immutable.js but not having that library maintained, among other issues we ran into using it while working on the TS conversion all contributed to us moving away from it later on as a separate project.
Our JavaScript-to-TypeScript conversion was a necessary and a very successful project overall. We did the bulk of the work in one go, which was great, but the final completion of the language transition took longer than we hoped for. Getting that long tail done did not prevent us from using TypeScript – on the contrary, moving fast allowed us to start using it faster – but more careful planning of the remaining work could have helped us declare the work complete months sooner.
Since the initial transition, we’ve migrated away from immutable.js for all of our core types and into using raw TS types as the readonly affords us the immutability we need. We’re also using an OpenApi-based code generator to have a single source of truth for our data types. That code generator and the related migration is worth its own article.
Bottom line – TypeScript is a fantastic technology and we thoroughly recommend it!
The global confectioner mitigates waste, improves service levels and controls costs by connecting digital supply chain visibility with POS analytics.
Keep readingHow to take an iterative approach to digital supply chain transformation with real-time alerts that motivate teams to collaborate on issue resolution
Keep readingUnderstand how gaps between systems, teams and processes are keeping you constantly firefighting and hurting your supply chain resilience
Keep reading
