Year 2019 Review! 🏏

I wrote the following in my previous blog post and it applies in this one as well.

It’s been a while since I wrote my last blog post, more than an year now. This post is basically a summary of what I have been upto personally and professionally in last one year.

Professional Developments

Unlike 2018, when I worked for a number of companies, 2019 was much more straightforward, I worked for only one company named as Turner & Townsend.

TL;DR : I had a great time there, would highly recommend them.

The office is located right next to St. Paul’s. The above photo (of St. Paul’s) was taken on 20th Feb, 2019.

Turner and Townsend is a construction consulting company i.e. they basically do consulting for construction companies, they don’t construct themselves as per my understanding.

I worked for them as a contractor in their backend team. I was quite skeptical about joining them initially, as I knew they were not really a tech company and their might be a lot of bureaucracy and lack of understanding of tech. In spite of all this, I took a gamble.

Unlike my expectations it turned out a great place to work at, I learnt a lot and made some good friends. It was a small team of roughly 10-12 developers, couple of designers and a few product owners.

Work

They were building a couple of applications know as Benchmarking and Cost Control. The former is basically a tool that uses the data they produce to benchmark and model the space, time and cost of a project and latter is kind of a banking application, which is basically a ledger of all the financial changes during the execution of a construction project.

There were two streams of work that I was part of:

• Backend: which was mainly creating data models and APIs on top of them for the web application to display them on the frontend.

• Data Pipelining: Pulling the data from various sources such as csv, excel, sql servers from the business into the system. Since there was a continuous need for creating data models and doing it via SQLAlchemy and then writing usual APIs for those models was a lot of repetitive work. To cater for this we used an open source tool called genyrator (which was written in house) for generating models and APIs by just writing some abstract definitions.

We used Apache Airflow to create, schedule and monitor those data pipelines.

A screenshot of a pipeline, from Airflow’s dashboard.

One of the cool things about this application is that it was written in a very declarative way using functional programming, which made it easy to write new pipelines, read, extend and maintain. Only con is that is was sometimes hard to debug due to 100s of lambdas in the stack trace.

Example

To illustrate the concept mentioned above, consider a case, when you are writing an ETL pipeline that fetches the data (say a row) from csv to the database and you want to rename a column and round a float value in a key:

Lets us write a couple of functions that we will call as transformer, that will rename the key and round the value of a key. These function will take keys as argument and will return a function, which when called with a row of data (dict), will apply the above operations and will return us a new row.

def rename_key(key: str, new_name: str):
    def _rename_key(row):
        mutable_row = dict(row)
        mutable_row[new_name] = mutable_row.pop(key)
        return mutable_row
    return _rename_key

def round_value(key: str):
    def _round_value(row):
        mutable_row = dict(row)
        mutable_row[key] = round(mutable_row[key])
        return mutable_row
    return _round_value

Let us now define a PipelineConfig class that will take a data and a set of functions (Transformers) that we can pass to it.

import copy

from dataclasses import dataclass
from typing import List, Callable

@dataclass
class PipelineConfig:
    data: dict
    transformers: List[Callable]
    def run(self):
        mutable_data = copy.deepcopy(self.data)
        for transformer in self.transformers:
            mutable_data = transformer(mutable_data)
        return mutable_data

Now let us see our PipelineConfig in action

pipeline = PipelineConfig(
    data={'foo': 123, 'bar': 456.456},
    transformers=[
        rename_key('foo', 'foo_new'),
        round_value('bar')
    ]
)

pipeline.run()

Output:

{'bar': 456, 'foo_new': 123}

You can see in the output, that foo is renamed to foo_new and the value at bar is rounded to nearest integer.

The advantage of this is you can define transformers for every operation you want to perform and then you can reuse it and if your transformer are idempotent and commutative then you can combine any set of transformers to get your desired transformation.

What’s Next?

This year in January, I joined a fintech startup called as Aire as a contract Software Engineer. They help in credit assessment, I am working in the lender side of the business. More about this in future blog posts.

Personal Life

Cricket

• Played lot of cricket (same as last year).
• Volunteered for the ICC Cricket World Cup 2019 in the Sports Presentation Team.
• Saw a lot of Cricket World Cup matches live in the stadium.

This is a video of me in the spectators behind, clapping after Bumrah takes the wicket of Usman Khawaja.

Some post cards from cricket world cup 2019, that I collected travelling across various venues, along with a brooch and a medal of appreciation for volunteering.

• Saw the England’s Captain lift the world cup in the dramatic final of Cricket World Cup, live in the Stadium.

Concert & Events

• Saw Asha Bhonsle live in Wembley
• Saw Rahat Fateh Ali Khan live in Wembley
• Finally saw the Jashn-e-Rekhta
• Attended St John’s May Ball 2019 (University of Cambridge)
• Saw some really good live music events, the favorite was Folkklubs Ala Pagrabs in Riga, Latvia.

Travel

• Travelled to a few places: Kerala, Hamburg, Chicago, Cleveland, Istanbul, Paris, Luxembourg, Croatia, Edinburgh, York, Bristol, Bath, Riga, Hyderabad, Bangkok, Phuket.

Misc

• Moved back to London from Cambridge.
• Attended PyCon US 2019.
• Renewed my Passport.
• Done my desk setup (learnt some woodworking basics)

2020

The first quarter of 2020 is almost finished, here is a brief summary of what has happened. The gif below is a footage of few of us trying to enter the year 2020, apparently it’s quite similar for everyone, thanks to COVID-2019