It’s been long time since I posted something on my blog. Lately I have been pretty busy with interesting and/or funny things happening in my life. I have draft posts regarding those which I may make public one day.
This time I’ll be sharing LeetCode Outage post which I wrote for official company’s statement regarding downtime LeetCode faced recently. Feel free to reach me out to drop feedback or any question.
====
Last week, LeetCode experienced an issue due to which it was completely down for 2 hours 35 minutes and continued degraded service for another 47 minutes. We all at LeetCode apologizes for inconvenience you have faced. At LeetCode, we take pride in building highly available systems that provide delightful user experience, and we are aware of the trust you place in LeetCode. That being said, we certainly would like to share our learnings from post analysis our engineers have performed and steps we have taken to make our system more resilient and fail proof.
1:58 PM, Nov 19 PST
Partial recovery - 04:33 PM, Nov 19 PST
Full recovery - 5:20 PM, Nov 19 PST
LeetCode’s architecture looks some what like below -
Our architecture is designed to be scalable as well as fail-safe. Despite the layer of redundancies, there still exist possible cases where our architecture may fail. On Nov 17, there was an accidental removal of kubernetes’s internal cilium-operator
service from the cluster. Even though it’s non-critical service responsible for garbage collection of the network policies, but due to its bad recovery, it stopped that garbage collection. On 1:58PM Nov 19 PST, we hit the limit of network policy map, breaking all the networking and halting communication between all the components of the cluster that led to this down time.
1:58PM, Nov 19 PST
Internal monitoring systems started generating alerts. We discovered site is down. By 2:04PM, all the devops were triaging the systems to figure out why application servers were not serving any requests.
02:09PM, Nov 19 PST
We discovered that it’s due to DNS Server containers being in PENDING state in the cluster. We found that they were stuck with cilium(k8s’s networking layer) related errors.
2:21 PM, Nov 19 PST
We felt that it’s a major issue and debugging current cluster state would take lot of time. We decided to fire up new kubernetes cluster since user facing services in the cluster are all stateless.
2:57 PM, Nov 19 PST
Finally new cluster provisioning finished but suddently old cluster started serving requests. We were confused and were investigating why it started serving requests back.
3:07 PM, Nov 19 PST
Old cluster failed again. We started working on that newly created cluster again.
3:18 PM, Nov 19 PST
We tried to setup production services on new k8s cluster but due to difference in k8s cluster version, our automatic provisioning scripts were failing. We decided to fallback to restoring our old architecture which didn’t use kubernetes but wasn’t much scalable.
4:19 PM, Nov 19 PST
We restored our old architecture and site was up partially but due to large amount of traffic, latency was quite high and UX was not upto the mark. We started working back on creation of kubernetes cluster.
5:20PM, Nov 19 PST
New cluster provisioning finished and all the user facing services were setup in the cluster. We switched the traffic to new cluster and site was back to normal.
Built over top of BPF interface (Berkley Packet Filter), Cilium implements networking stack of k8s. To give basic overview and bare explanation of what it does is, it assigns IP to each container/service/endpoint - all components of k8s, transparently managing communication between different components at different layers etc. It maintains a policy map which contains all these kind of rules regarding address translation, packet dropping policies and more. On kubernetes cluster, we have cilium
and cilium-operator
services running. Former one is responsible for implementing this network functionality and later one is responsible for clearing up entries from that policy map as new nodes/pods/k8s_component gets added/removed. So, cilium-operator is just responsible for garbage collection.
When we accidentally deleted cilium-operator from the cluster, we had to restore them back from the other kubernetes cluster since this service is setup into the cluster by Cloud Providers. After restoring the service, cilium-operator pod’s status was changed to “Running” as well as “Ready” but since we weren’t very much aware of its internal working, things seemed fine to us.
Since the restored cilium-operator service was from newer version of kubernetes, it’s specification differed, due to which it wasn’t cleaning up that map. As time passed, new nodes were added, new pods were deployed - we hit the upper limit of the configured policy map i.e. 16384. As the limit was hit, cilium stopped adding new entries to it due to which we faced outage.
We know how much you rely on LeetCode for anything related to your career. We feel happy when you make one step closer towards your aim through LeetCode. We are really passionate about the availability our services. We have been continously learning from our mistakes and improving to deliver best UX to our beloved users.
]]>Recently I visited solo to Kheerganga. Yeah, this time I planned to go alone and it was simply an amazing experience!!
Been there around two years ago, I was easily able to plan it to be a weekend trip. Traditional refreshing air of hills, away from city life without any mobile networks, chirping of birds and soothing sound of waterfall on your trek from the forests, relaxing dip in the hot water springs after your 14KM trekking journey, laid on grass watching clear sky and stars at camping arena - simply put, you will feel at peace.
To visit kheerganga, usually people catch bus to Bhuntar and then taxi to Barshaini from where the trek begins. I left for Bhuntar at around 11PM from Chandigarh and reached there in around 7-8 hours. Make sure you don’t eat anything heavy or otherwise you will be puking everything out in bus. I prefer not to take my personal vehicle on long trips because taking night bus saves time and also your vehicle would thank you for not jouncing on those roads.
I had seldomly ridden bike in Chandigarh during my college years. Instead of taking taxi from Bhuntar, I decided to rent Bajaj Avenger 220. Even though I just had abstract ideas about riding bike, I maintained my confidence and gave it a shot. At first, I wasn’t comfortable and was having hard time since unlike plains, usage of front brakes on hills was causing bike to slip a lot but very soon I developed good muscle memory for my feet. Also later on I felt 220cc bike isn’t good enough to deliver required torque in hills. Though I had even ascended 125cc scooty with one pillion rider all the way up to Tosh in the past.
I was finally enjoying my ride, not for long though. After travelling few KMs from Bhuntar, there was lot of dust in the air due to unsealed roads. It was causing irritation in eyes as well as nose. I wrapped handkerchief on my face, covering up my nose. Though I wished I could do the same for my eyes.
I reached Kasol after my one hour journey. If I have to describe Kasol, I would say it’s “paradise for smokers”. Well I don’t smoke but my college friends are Marijuana lovers. So, I’m well aware of places around it. I washed my eyes throughly and then had breakfast at some good cafe. After that, I left for Barshaini which is again one hour journey, passing through Manikaran - famous religious place. Even though Manikaran to Barshaini is even more off-road but since very few vehicles go there, I didn’t face any issues. Weather was pleasant throughout the way to the Barshaini. Even though sunlight wasn’t harsh at all but I still got my uncovered arms well tanned. Ping me if you know good way to un-tan them :/
After reaching Barshaini, I started the trek at around 11:30AM. It seems like due to ongoing construction of dam at Barshaini, starting point for the trek has been changed since my last visit. There exist two different trekking paths to Kheerganga - through forest and through village. I choose to go from forest and come back via village. Forest trek was much more beautiful and calming as compared to village trek. Seems like it was also the one less travelled. Last time we weren’t able to experience forest trek due to my friend’s foot injury. Trek felt quite easy as compared to last time. May be I have gotten fit ;). I interacted with people from lot of different places and backgrounds. I met few foreigners too and most of them were professionals travelling and exploring different countries all over the world.
I was around 1KM away from camping arena when it started to drizzle. The sky was heavily filled with dark clouds. I rushed to reach the top and booked shared tents with other trek-mates. We ordered food and soon enough it started to rain heavily. We sat outside in open under the shed and enjoyed hot tea. After few minutes it completely stopped raining. Weather’s really funny up there. Then we went for taking bath in hot water spring where we sat for around 1/2 hour. All the muscle pain just vanished and I felt this much energetic that I could go back to Barshaini and again do the trek. Okay, I may have exaggerated a bit but it was damn relaxing. After that, we came back to the tent, enjoyed dinner, weather, talked about random stuff and finally collapsed to bed late night.
Next day, I decided to leave late and trek down slowly since my bus was at 9PM from Kullu which takes around 30 minutes from Bhuntar, giving me plenty of free time. I kept on stopping at every other cafe on the way back. Also the no. of cafes are higher on the village trek. I trekked down in 5 hours, reaching Barshaini at 3:00PM. Since it started to drizzle, I decided to wait in the cafe in Barshaini. When the clock ticked 5, I decided to leave for Kasol. I had two options - either to take taxi and tell bike owner to retrieve bike himself and charge me whatever he feels right or ride back myself in rain. I knew it was going to be hard due to rain but I still went with that. I took raincoat from cafe which was built from very thin polythene used for making plastic bags and then left for Bhuntar. Due to rain, road was muddy and pot holes which were all over the place, got filled with water. Since my bike was slipping a lot, I slowed down and drove more carefully. I remembered my friend’s words from the last time we visited Tosh - “Bansal, agar scooty girani pad gyi, toh khai ke dusre taraf girana XD” means “Bansal, in case you lose control and we are going to fall down from bike, make sure to fall away from the ditch”. It took me two hours to reach Kasol where I ate something and then left for Bhuntar.
On the positive side, due to rain, I got to see this beautiful double rainbow. Due to sun going down, my speed dropped even further. It took me another 1 and 1/2 hours to reach Bhuntar. I’m glad I didn’t take taxi, otherwise there was quite a chance that I would have missed my bus due to traffic jams everywhere. Since I was already late, I decided to board the bus from Bhuntar only, instead of Kullu. I boarded bus at 9:30AM and reached early morning at Chandigarh.
It was fun and I enjoyed it a lot!
Well, the title of the blog post may not sound technical but it’s actually a technical post. “Game of Thrones - S8E01” will be out tomorrow and so will be torrent seeders for pirating its episodes. If you have ever tried to seed HBO’s content (or any other television network) on your servers, you must remember getting messages about abuse from your server provider. Do you actually understand how it works ? It’s all because of P2P protocol. You must have heard about Cryptocurrencies, decentralized network, TOR, GOSSIP protocol, torrent clients, some video calling solutions, Spotify(till 2014) - they all use P2P protocols.
Let’s see Peer-to-Peer 101.
In traditional, client-server networking model, all the clients request services from the central servers. If central server is down, client won’t be able to access the resources. To overcome that, we need to introduce redundancies and design services such that we can make systems resilient.
Where as in P2P network, all the nodes are regarded as peers which are equally privileged and share resources amongst each other without the use of any centralized system.
When Internet was designed, I believe it was drafted mainly keeping “Client-Server” networking model in mind. But still, we can implement P2P network over OSI. The crux of P2P model is we have to architect our system in such a way that peers can act as “clients” as well as “servers” for other peers.
To achieve this, we can create a virtual network of nodes. Let’s try to build up a baseline P2P system like Bittorrent for sharing files.
Joining the network: Let’s say a peer wants to join our virtual network. It somehow knows which all other peers are in the network. It will broadcast its own identity to those peers and will ask them for theirs. It can be a simple ping message implemented in our protocol which will be sent by the peer (acting as client) to other peers (acting as server).
To get to know about those peers, we can save that information in foo.torrent
file which you usually download from the torrent websites.
Peer identity: If a peer joins and then leaves a network, you may want to preserve identity of the peer. May be you have some ranking algorithm for leeching score which can be used to generate foo.torrent
file.
To do this, you can create UUID and save it client-side. Ahem ahem, your centralised service storing all the torrent files can keep this list. There can be several use cases of uniquely identifying peers based on the problem you are modelling.
Data fetching: Now you have connected to all the peers who have data which you are interested in. How can you get that data ? Well, you may just define “FETCH” operation in your protocol in which you will mention the file hash (stored in the foo.torrent
). Then you will send it to one of the peers and get file from it.
Instead of doing this, what if you think of dividing whole file into n
parts and store hashes of each part into the same foo.torrent
file. Then you can request data from multiple peers. This way you may even request data from the peers who themselves are seeding some part from other peers. After you fetch one part, you can compute hash of that part and see if hash is correct or not. If hash isn’t correct, that means peer is sending invalid data and you can announce it in the network, by which based on ranking, network consensus will tell every client to block that specific peer. So, there are infinite optimisations and fixes you can apply to this protocol.
Also, if you understand computer networking well, you might be thinking all the peers would need public IP in order to communicate like this. Yes, that’s true, but there are some ways to over come this. One obvious way would be having a server acting as relay between these two peers. More complex but not globally supported technique is hole-punching. You can google it out to know how can two device behind the NAT communicate with each other.
Adding new data: Let’s say we want to add new episode to our network so that everyone who’s interested in it, can fetch it. Since we are relying on the central service to give us foo.torrent
, client application running on peer can upload list of all the files which are present locally. Central service will then create an index of that which can be presented to other users.
“Napster” - first music streaming service did something like this. But since it used to maintain index, just like we did in above given design, of all the files on the central server, company faced legal issues and it was finally shutdown.
To fix this, another design came up - Gnutella network which is fully distributed network unlike Napster’s implementation. Ever heard of Limewire ? I remember downloading song from there on a dial-up connection at 5KBps, sitting beside my dad, back when I was in 8th standard. And after it downloaded successfully, I remember happiness on my face to see that magic. What a time it was! Limewire was Gnutella client application. But it faced scaling issues with increase in number of users. You can read more about its protocol implementation on Wikipedia.
Base of more reliable and scalable systems is Kademila DHT. I may write about it some other time.
Back to the title of the blog, now I think you can guess how HBO can know about the peers who are pirating the content and then report it to service providers to take them down. To overcome this, people came up with the idea of private trackers.
That’s it for this post. Now go think how all usecases listed at the start of the blog can be implemented using P2P protocols.
Thanks for reading!
]]>Since the time I started using Macbook, I have been sadly away from KDE. That’s around 3 years from now. I have gained skills from contributing to KDE which I could have never achieved otherwise.
After long time, I decided to try out kde-connect on macOS. I knew it won’t work properly on macOS because of deep coupling with linux specific stuff but still I wanted to give it a try.
Few years back, I had tried to use homebrew-kde taps but that wasn’t really a good experience as there was no kdeconnect formula defined in it. I learnt that KDE starting maintaining new build system - Craft. After trying it out, I found that building KDE applications had never been this easy.
Steps to build kdeconnect:
Install craft on your system.
curl https://raw.githubusercontent.com/KDE/craft/master/setup/CraftBootstrap.py -o setup.py && python3 setup.py --prefix ~/CraftRoot |
Load craft specific scripts/tools into env.
source ~/CraftRoot/craft/craftenv.sh |
Build kdeconnect. It’s gonna build Qt and all other dependencies of the project. So, it will take around 10-15 minutes. Go brew some coffee in the meanwhile.
craft kdeconnect-kde |
After this is done, you can run cb kdeconnect-kde
to go inside kdeconnect build directory. Now try to run the kdeconnectd
inside bin
directory.
CRAFT: % cb kdeconnect-kde |
This error means it’s unable to find Qt Platform plugins. Those plugins are installed in the $(craftRoot)/plugins
directory. So, we need to set QT_PLUGIN_PATH
environment variable.
export QT_PLUGIN_PATH=$(craftRoot)/plugins |
Try running kdeconnect daemon again.
CRAFT: % ./kdeconnectd |
It’s because kdeconnect need dbus for interprocess communication. You can either use dbus from the Craft directory or install dbus using brew and then run it. I prefer second way since brew manages dbus daemon and injects all the required environment variables required to connect to the bus.
brew install dbus |
You can check if it ran successfully by using dbus-monitor
or dbus-test-tool echo
. Then try running kdeconnect again.
CRAFT: % ./kdeconnectd |
Hurray! Finally daemon started and it broadcasted identity datagram on the network. Now go and install Android application expecting all the plugins to work flawlessly. Well, as of now, all the plugins are non-functional due to incompatibility. I hope someday I’ll fetch some time and hack into the code to at least get the basic functionality working.
That’s it for now. Thanks for reading!
]]>Guess what ? Here’s much awaited(I hope so) technical post.
At LeetCode we conduct weekly contests during which we get week’s peak traffic. I had mentioned earlier in one of my blogpost about how one contest went wrong.
Here’s the crux of it:
Servers couldn’t handle the traffic. Mysql latency skyrocketed due to which all the uwsgi processes were I/O blocked waiting for the response from the mysql. Since uwsgi stopped responding, load balancer’s failover got triggered and it disabled the traffic from one of the kubernetes cluster, resulting in even worse. So, it was kinda deadlock situation which we cleared manually.
We have NewRelic APM configured through which we can monitor application performance and try to figure out bottleneck but in the production environment, there were just too many variables due to which it was difficult to narrow down the issue. Moreover, since contest is organised weekly, it’s not efficient to wait and watch how our changes would affect performance when traffic rises many folds.
So, it arose the need of setting up load testing framework which could help us find root cause. After checking up couple of options, I decided to give “Locust” a try since it seemed
simple to start with and scalable too. Overall I really loved the simplicity “Locust” provided. I modified our application code to support privileged users who can bypass certain security measures and could be used for load testing. Then I defined user behaviour using locust’s APIs and tried to test it. I was able to get quite good throughput from locust running on single instance but not good enough to simulate traffic as that of contest. So, it was time to scale up locust.
I encapsulated my locustfile in docker container, wrote helm charts and deployed it to new Digital Ocean’s kubernetes cluster. I was able to do it pretty quickly due to simplicity locust provides. I scaled up that cluster and was able to figure out bottleneck. We had to tune certain mysql parameters, clean up caching logic in application code and scale up our application servers. That’s it and we were able to perform much better during next contest.
You just have to define the user behaviour. An example from locust.io looks like:
|
From developer point of view using the locust framework, here’s how its basic architecture looks like:
But even with single node it was able to achieve quite good throughput. It truly gained my attention and I decided to check its internals.
So, the way it achieves that kinda performance is it is entirely written based off events on top of high performant network library gevent library.
I went through its code and here’s abstract summary from my findings. It uses ZeroMQ for task distribution among different node using ZeroMQ’s Dealer-Router pattern with MessagePack for application level serialisation / deserialisation of the messages. Based on the hatching amount and rate, Master would send message to all the clients to start hatching users. Though the distribution doesn’t take hardware specification of the slave node into the account.
for client in (ready + running + hatching): |
Then the slaves would receive the configuration and start hatching new users and execute tasksets defined by the user in locustfile.py
. Every spawned user will run independent of each other in separate greenlet(If you do not have idea about greenlets, then you can think of them as micro-threads which are pseudo-concurrently and can run on just couple of threads). The logic to execute the taskset is pretty obvious. On every event like user hatch, request complete/fail - they will keep reporting their status back to master through which master can track everything and aggregate it to show us all the stats. Fairly simple and straightforward, right ?
That’s it for now, thanks for reading!
]]>Few days back I was trying to push to the docker hub and docker push was failing at just 99.x%
. I tried to push multiple times but it was exiting with same error message: Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password
The message was quite confusing to me as it didn’t make any sense to give such error message after pushing upto 99.x%
. I tried to look it up on the internet and after visiting lot of links, I saw someone talking about machine’s time in some irrelevant thread. Then it clicked me that sometimes my linux VM’s clock, on which docker runs, gets off the wall clock. Even though ntp runs in the background but it won’t make a big jump to correct the time. So, I manually fixed the time and docker push succeeded.
But I was still wondering why does docker push needs clocks to be in sync ? Any guesses ?
The answer is authentication process. I was easily able to find docker docs which describes the authentication process. But why authentication needs time to be in sync ? I’ll answer this up in a while.I decided to go wild and debug the authentication process manually. From the error message Get https://registry-1.docker.io/v2/: unauthorized
, I saw that docker uses standard HTTPS protocol for authentication. So, I decided to use standard HTTPS MITM proxy to inspect the traffic. I never did MITM on OSX, so wasn’t aware of any good tools for it. Moreover the traffic was originating from my linux VM, that means regular HTTP proxy solutions like burpsuite or charles proxy won’t work. So, I started looking up other options by which I can setup transparent proxy.
squid proxy - I had used it once in past during my internship at ZScaler but it’s quite lengthy process to set it up given that I’m quite lazy.
mitmproxy - Somehow I came across it and I loved its simplicity. I found the doc to setup transparent proxy quite easy. But in this example, they have only mentioned the case in which test device is different than the host on which proxy server is running and test device needs to be configured to use proxy host as the default gateway for sending packets.
Well, I knew I have to do some magic with routing so that I can use the same device as a test device as well as proxying requests. But I wasn’t sure how exactly it has to be done. I tried to look up on the internet and realised I don’t have enough understanding of the linux ip stack. Obviously it’s quite easy to just copy-paste few commands to block/unblock specific ip or port using iptables but how can I route the traffic from the docker instance to proxy server listening on port 8080 - that isn’t possible without proper understanding of how packets go through linux networking stack.
While going through various links on StackOverflow, trying to learn this black magic, I came across linux-ip.net. It explained linux ip stack in quite good detail. So, I decided to step back and learn about it from the beginning. Then I recalled it’s the same site which I had added in my ToDo list a year ago for learning networking on linux.
I skimmed through the whole guide and checked things which seemed relevant enough to me. I might have understood just 20% of it but believe me it was more than enough. I was able to follow and understand all of those SO links.
Above diagram is of utmost importance in case you want to play with the packets using netfilter interface of linux. Now I knew that I have to add rule in OUTPUT
chain to send all the IP packets with destination port of 80/443
(HTTP/HTTPS) to address 127.0.0.1:8080
(on which mitmproxy was listening). With this mitmproxy
will be able to listen to all the traffic and show me everything.
iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination 127.0.0.1:8080 |
But after intercepting these packets, mitmproxy
needs to send those packets to original destination. So, it will change the destination address of the packets back to the original address. If that’s the case, then it would result into never ending loop in which my above given rules would send the packets sent by mitmproxy back to mitmproxy itself. So, I needed to do something to break this loop. I decided to change destination of packets sent by processes owned by a “specific” owner.
Basically I added these rules to achieve it:
iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --uid-owner 1000 -j DNAT --to-destination 127.0.0.1:8080 |
With this, I ran the mitmproxy as root user(uid = 0). So, my rules will only do DNAT-ing on the packets originated from processes running as vagrant
user(uid=1000). After installing the certs from the mitm.it
, I ran curl https://google.com
and was able to see intercepted traffic in the mitmproxy. But when I ran docker push
, I couldn’t see anything. Then I recalled one difference I had learnt between rkt
and docker
i.e. docker
daemon runs as a root which isn’t safe etc etc. So, basically docker client was sending request to docker daemon and since docker daemon was running as root, it won’t show up in mitmproxy as I was only intercepting packets for uid=1000
user. I changed the above rules to setup intercepting for root user instead.
iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --uid-owner 0 -j DNAT --to-destination 127.0.0.1:8080 |
I ran mitmproxy
as non root user and voila!! I was able to see docker’s intercepted traffic. Now, it was time to see the intercepted traffic.
So, first of all it sends request to the registry-1.docker.io/v2/
and it returns 401 response. Let’s take a look at the full response headers.
It’s returning Bearer realm
in the Www-Authenticate
header in accordance with OAuth format mentioned in https://tools.ietf.org/html/rfc6750#section-3. So, it tells that client has to first authenticate itself with auth.docker.io
and get access token from there using which registry-1.docker.io
will service requests from the client.
Now, let’s check the request and response associated with auth.docker.io
.
Request headers:
Response headers:
In the request headers, docker client has specified Authorization
header passing our credentials to the server and in the response, server returns us access_token
which we have to send back to the registry-1.docker.io
to get access to the resource. That access_token
has associated expires_in
and issued_at
fields which defines for how long this token is valid. These fields are same as nbf
, iaf
, exp
fields in jwt specification. Well, that’s the reason why authentication was failing. Though I’m still not sure why docker allowed push upto 99.x%
and then failed. Real world is complex for sure.
If you want read more about the authentication process of docker, you can check it out here.
That’s it for now, thanks for reading!
]]>I scraped the idea of staying for longer duration at my friend’s place in Gurgoan since I wasn’t able to work properly from there. But I decided to attend tech meetups in Delhi on weekends. This weekend I planned to attend two events which were related to kubernetes and networking.
I reached Delhi at around “10:25AM” (Kalka-Delhi Shatabdi never gets late). I was quite excited about attending Kubernetes Meetup at Grofers
since I had been working lately on devops related stuff in LeetCode. But I knew there’s still lot of stuff which I need to learn and grow. After reaching there, I got to know that the meetup had been cancelled at the very last moment due to event place issue. What the heck! I travelled for 6 hours just to find out this ? Meh! I was quite disappointed and opened meetup page. From there I got to know that due to place issue, the organiser decided to organise it at his place. I was bit relieved that I can still meet new people there. I went to his place and there were only two people who came there. Both of them were devops guy having 4-5 years of experience, one from Paytm and other one organiser himself from Grofers. So, it was win situation for me to just converse with them and get to know what they have been working on. We discussed about lot of different stuff.
When we were finished, I got call from good old friend, Abhishek, from high school with whom I wasn’t able to connect from years. He told me that he has been staying in the Delhi. What a coincidence! He was preparing for govt exam there. Yeah, he did engineering in CS which was enforced upon him by his parents. He didn’t enjoy that at all. We decided to catch up and I learnt about what he had been doing all these years. I stayed at his place and the worst happened. There were mosquitos all over his room and somehow mosquitos love my blood a lot. Whole night I spent killing them so that I can get some sleep. I murdered more than 15 of them and still there were more who would just come around my ears and create annoying sound and wouldn’t let me sleep.
Next morning, there was LeetCode contest which I had to monitor to make sure everything was alright and our servers can handle it. Well it went terrible, they couldn’t handle the traffic. Mysql latency skyrocketed due to which all the uwsgi processes were I/O blocked waiting for the response from the mysql. Since uwsgi stopped responding, load balancer’s failover got triggered and it disabled the traffic from one of the kubernetes cluster, resulting in even worse. So, it was kinda deadlock situation. We cleared that manually and site was back to normal after few minutes. Of course we decided to take some steps so that it doesn’t happen again in the future. I might write about it some other time.
After that I left for Gurgaon to get ready for the next meetup in which I just planned to meet new people. I reached there late by around 2 hrs. I already had missed two talks by that time. I got to know the event was running late by 1hr. After few minutes, there was lunch time where I get to know people and have a general talks, stories etc. The event was held at the “Social Cops” working place and I learnt about what that company exactly does.
I found some great posters of commandments there. Well photograph itself came up quite bad, so let me add them here as text:
coder != engineer
- Jumping directly to writing code more often than not leads to bad code and bad engineering. Think twice before writing anything. import lib
- Life is too short to reinvent the wheel.With great power comes greater responsibility
- Sometimes the toughest problems have the most simple solutions. No, machine learning is not the answer to everything.Fail fast and keep learning!
Know thy build
- It’s not enough to know what will work and when, also know what will not and when.#itisnotmagik
- Think about what is happening under the hood. “foobar” is not a silver bullet. Everything has its own tradeoffs. Measure whatever is measurable
- Data backed decisions give better results instead of randomly taken ones.Automate whatever is repetitive
- We built machines only for this.Premature optimisation is root of all evil
- Pick the right battles to fight. You can never reach the perfection.When in doubt, ask!
- Be well researched before doing any thing.I really love these and yeah, these^ are of great importance to become a good engineer.
That’s it for now. Thanks for reading!
]]>If you have read my last blog post, you know I attended WWC meetup related to Algo/DS and from there, I came back to my friend’s place at around 3AM. Next morning, there was hackathon in Zomato in which I got myself registered with my friend’s team. We hadn’t even thought about the idea which we were going to build the very next day. I was quite exited about the hackathon and couldn’t wrap around my head around it. I was constantly thinking what should we build and just couldn’t get any sleep. I started by trying to understand company’s business model. I learnt this skill while working in LeetCode, to not only think about the technical part but also think how those developments will affect the business. I researched about Foursquare, Yelp etc and I came up with few ideas.
As the clock ticked 5, I started feeling hungry. I ordered food using Zomato and then slept at around 6 after doing breakfast. Hackathon’s official time was 10AM and we woke up at around 10:30. We rushed to the office only to find out that both of the Mohit’s colleagues ditched us and we had to fight this code war alone.
Before Hackathon started, Zomato’s senior folks came and talked about something which I cannot recall because I was quite tired and kinda sleeping. Then one more guy came and started with something like “I wrote Facebook’s core system which is chat engine, blah blah!”. I didn’t really pay attention to any of that. Then he said, I'm author of Cassandra
. Suddenly I regained the consciousness and I was like, wait what did he say ? He wrote “Cassandra” ? I didn’t expect that person to be in Zomato. I thought he might have contributed to it for a while and must be exaggerating that. To verify the facts, I opened the author list and yes, he was indeed co-author of the Cassandra. I got to know that he was working as an advisor for the Zomato.
Then hackathon started and due to previous speech by Cassandra author, I was quite driven by the scale, reliable and production grade architecture. We made the mistake right there and tried to build up the whole product which could be useful for the company instead of a small prototype suitable for hackathon.
After the first round of evaluation, I wasn’t really feeling enthusiastic. Even though judgers liked the overall idea and the implementation but what we had originally thought would have taken quite long time. Both of us weren’t fluent in the frontend stuff. Since we were using Python stack and I was quite fluent with it, I handled the backend stuff and Mohit worked on the frontend. We were building a crowdsourcing engine by which we can make use of the Zomato’s large community to gather all the data about restaurants. So, instead of having Zomato employees manually going to each and every restaurant to collect all this, we thought of collecting it through users themselves just like Google does on “Google Maps”. I was just thinking about getting back to bed and get some sleep. But since Mohit worked at Zomato, he couldn’t just let it go since everyone would had higher expectations from him. We weren’t exactly able to build what we planned but we were still able to reach the barebones prototype level. All thanks to my inhuman speed ;)
Since sleep was getting upon my head, I forgot to click any photographs of the Zomato workplace.
After the hackathon, I drove back to the Chandigarh and reached in around 4.5 hours, yeah I drove crazily. I celebrated Lohri with my family and then finally ended up on my lovely bed for around 12 hours.
That’s it for now. Thanks for reading!
]]>Since you know I’m in Delhi around this time, I decided to go to meetups happening around here. I checked all the gatherings on meetups.com and decided to attend ones which I was interested in.
One of them was “DS/Algo group” organised by WWC which aligns with my interests quite nicely. Since it was just 5km away from me, I drove to that place.
The meetup was organised in a co-working space called “WeWork”. I found the ambience of the place quite good. I have never been to any good co-working space.
There were only 5-6 people in the meetup and we decided that everyone will read different articles(related to Graphs) and then share the knowledge among everyone. I was given “Kruskal’s Minimum Spanning Tree Algorithm”. Since I was out of touch from DS/Algo for more than 2 years, I wasn’t able to recall it. But I read about the algorithm on GeeksforGeeks.com and was able to catch it up perfectly within 10-15 minutes. After that, we discussed about it and lot of unrelated stuff as well. And at last, at around 10 O’clock, we decided to sign off after playing a match of foosball.
And not to mention, of-course I won that match. :)
Next day, I again went to the same event since I had a good experience a day before. After the event was over, I decided to stay at that place only for some more time to do some office work because I knew I’ll find the party going on at my friend’s place. I bought some “ready to eat” food from the WeWork market and stayed overnight there till 3AM. After driving back to the flat, I was quite excited for the Zomato Hackathon happening next morning. I couldn’t just sleep due to that excitement and was thinking about some idea whole night. So, yeah I did participate in the hackathon. I’ll write my experience about the hackathon in separate page.
It was a fun experience overall. I would love to attend those meetups in future whenever I can.
I’m attaching some images relates to the event and the co-working space.
That’s it for now. Thanks for reading!
Good bye! o/
Wait, what are you saying ? No, I don’t remember promising anything about creating a blog post every week. (/me hides under the table)
Well, lot of things have changed since the last time I blogged. My todo list is increasing at tremendous rate compared to its burning rate. Putting other things aside from that list, there’s long list of blogs-to-be-written.
Anyway, let’s talk about this blog post.
Yesterday, at around 1:00 AM, I was trying hard to get asleep. Then a random thought came to my mind about moving to Delhi for few weeks - to explore new things, to fight with my fears, to conquer something, to try out different things. Since these days at LeetCode, I was doing work which didn’t require multiple screens, I thought that I’ll be fine without my whole work setup. So, I called my friend, Mohit, immediately and asked him if I can stay at his place in Gurgaon for some time and he talked to his flatmates and I got the green signal.
We left for Delhi same day at around 6:00PM (thanks to him for delaying!) and we reached around 12:30AM today. Yeah, it took bit longer due to traffic jams as well as speed limit of 110kmph set on my car.
As we entered Gurgaon, I found roads to be bit confusing and ended up taking wrong turns multiple times. Earlier I didn’t know about concept of “Service Roads” which isn’t there in Chandigarh. Also, sorry delhi-ites, my first expression is lot of people here don’t know how to drive. They drive crazily as if they own the roads. No traffic rules, nothing! Seriously.
Also, Mohit told me that there’s Hackathon being conduced in Zomato this weekend. I might be participating in it. It will be my second hackathon till date. I had gone to my first hackathon last year around March as a mentor and it was the experience I could never forget. I slept like just 10 hours in overall span of 40 hours. You don’t believe it, right ? A guy like me who sleeps 8hrs+ regularly can go this insane. Well, I got bit carried away by coffee. ;)
That’s it for now. I’ll try to keep my readers updated with new stuff whenever I get time.
Good bye! o/
]]>As I promised that I’ll be writing blogpost every weekend after joining LeetCode for full time position, so here I go.
This week I worked on finishing up a major feature i.e. Improving filter’s functionality. It’s live right now. You can check it out on LeetCode.
LeetCode’s frontend is built upon ReactJs. I wrote component logic for the filter functionality and Hercy, my colleague, connected all the dots and improved its UI. Now state of all the filters is saved in the URL itself. So, user can share or bookmark the URL to get the same state of the filter.
Moreover earlier filtering questions using search query used to work at frontend side due to which user was able to filter questions only using Question ID or Question Title. Filtering question by problem description is not feasible on frontend due to obvious reasons.
So, we decided to move the search functionality to the backend. Now we index all the question content and user is free to filter question using problem slug, problem details etc. Right now search isn’t that powerful like stemming doesn’t work. We will try to improve indexing even further.
Other than that, Accepted Solutions Runtime Distribution page was quite slow. Those who don’t know what that is, after you get ‘AC’ for a submission, open submission detail and there you can see this graph.
Everyone thought it’s due to lot of submissions on the OJ, more than 100 million. But fortunately LeetCode backend is connected to newrelic, which is a performance monitoring tool. We saw a large no. of MySQL transactions happening on each GET request on submission detail page from the evening of 24th April. We checked the git commit history and found a new feature was deployed into that page at that time, which allows user to check other people’s submission with different runtime.
I fixed its performance by implementing on-demand submission retrieval which reduced SQL transactions by manyfold. Thanks to newrelic we were able to debug it out!
Moreover, I also learnt about how LeetCode maintains user’s session. We use JWT for that. I can’t reveal whole implementation due to security reasons.
If you don’t know about JSON Web Tokens, then don’t forget to check it out after reading this blogpost. It’s really nice concept. Using JWT, you don’t need server-side session to check if user is authenticated or not. Don’t get me wrong. You might need server-side session due to some other reason but till this point, to check if user is authenticated, you don’t really need server-side session. You can store JWT in user’s cookies and you are good to go without persisting any session information server-side. Isn’t it cool ?
Other than that, we use slack for communication but our team’s account is on free plan due to which there’s limit on message history size we can browse. It happened to me 3-4 times that I scrolled up to check some previous message containing some important information and found that I can’t check messages any further. Switching to some other messaging service wasn’t possible as we have configured different Slack bots which send us many notifications like newly created issues on our issue tracker, any commit/branch updates on our VCS, CI updates, server status updates etc.
So, I decided to use Slack API to create a simple app which will receive every message sent in our slack team and log it into files. It isn’t complete yet as I was only able to spare an hour or so, to work upon it. You can check its source code here. It’s fairly simple.
Well that’s it for now.
Thanks for reading!
Well, again no technical content in this blogpost.
Sorry about the last blogpost!
I was quite sad at that time and next day when I read I was like what the **** I was writing. But still I’m not going to remove it up. It’s true upto some extent ;)
Well back to this blogpost, 9th June was my last day at ZScaler.
I learnt a lot about Computer Networks in my last five months of internship at ZScaler. Even more important than that, I made some really nice friends there. I enjoyed a lot working there.
I sent an informal goodbye email to the beloved ZScaler family and got best wishes from everyone for my future endeavours.
Guess what ? I even got the farewell party from the mobile team.
I’m already feeling nostalgic about those happy moments I spent with my friends at ZScaler. Harman, Rishabh, Tushti - I’m gonna miss you guys a lot.
That’s all for now.
Thanks for reading!
This blog post addresses importance of work-life balance.
Today I did a small experiment on my friend at office who had a lot of tasks assigned to her.I intentionally messed around my friend and from what I saw, I can’t believe how much work stress contributes to anger, resentment, unhappiness. I’m really sad while writing this post about how work can affect person’s life.
I really feel bad that people have to do so much struggle, live hard life etc. and for what ? Just to do living.
I would be lying if I say I didn’t work hard to become a bit better than an average Software Developer. Although that’s a whole different story.
I ain’t saying that one shouldn’t work hard. I just feel that good work life balance is must in the life. Otherwise, with every tick of the clock, some part of you will keep dying with evil drivers like stress, depression and what not.
When I was in college, I never thought about this. I just wanted to join a company with interesting work and yeah of-course good pay check.
I learnt about work-life balance after joining ZScaler as an intern. From my personal experience, work-life balance in Mobile team at ZScaler is quite good. Although that might not be true for other teams but still it’s better than other companies where my friends are working right now.
When I get to know about work culture in companies like Jugnoo, Utrade etc. from my friends, I really feel bad for them. I would never want to join a company, even if they pay me higher than what other pays, where I’m forced to work 9+ hrs daily to meet deadlines or get myself fired if I fail to achieve weekly targets.
I wished world where people work for fun not just to do living. And when work is fun, you won’t feel like working at all. Well that sounds quite like an ideal world. By the way, even if it’s fun, don’t over do that as there’s no exception to old famous proverb : Excess of anything is bad.
Anyway, that’s just what I feel. You might have different views on this topic and if yes, feel free to shoot them out at me.
That’s all for now.
Thanks for reading!
Well, I’m really excited to tell you guys that I’ll be joining LeetCode as Software Engineer, full time remote position.
LeetCode LLC is an U.S. company locating at San Francisco Bay Area, providing a platform to prepare for the Tech interviews.If you want to know more about it, just head over to LeetCode.
I have been working there as a Part Time Intern from last few months. I really enjoyed tech stack, code quality, working environment etc.
Well, very less info is available about the company on Internet. You might not even find the CEO of the company.
LeetCode’s CEO, Winston Tang, started LeetCode as his hobby project and later on he started working on it for full time. He’s really nice guy who promotes learning, prefers quality work instead of just get this shit done
. Interested in joining LeetCode? If you are good at algorithms and have worked on interesting projects, just shoot me an email.
You might be wondering why I’m writing this post. Is there any compulsion applied by LeetCode for blogging just like open source programs like SoK ?
Well, no. After joining LeetCode, probably in June, I have planned to write one blog post every weekend, describing the crazy stuff I did that week and problems I faced in doing that shit. Although it’s just a plan, let’s see how it goes.
I didn’t use to like the Blogger’s UI and theme I was using. So, I decided to check other themes but didn’t find any good one. And then I decided to try other blogging frameworks. I was confused between Jekyll and Hexo. And finally this blog helped me in deciding what I was looking for.
Quoting from that blog :
I started feeling crazy. I never used ruby before (I had also problems installing it on windows) and after a little bit I realized that every piece of Jekyll is built in this language: from the css preprocessor (scss), to the templating engine (liquid).
The real thing here was that I found Jekyll too much disorganized (from the documentation to the plugin management). With Hexo is the complete opposite, while you won’t have a strong and big community ready to help you, you have everything already pre-installed with it, even a github deployment module that with a single command it pushes changes on it, making the story of “jekyll supported by Github” completely irrelevant.
Same thing when you want to install a theme: with jekyll usually you must follow several steps and very often you have to install manually the dependencies, while in hexo one command is enough to install also every sub-module for you.
So, I finally decided to use Hexo and I really found it quite simple to setup.
That’s all for now.
Thanks for reading!
I reached Rialta Hotel on 24th at night. So, I missed some kind of Spanish ceremony/tradition but my friend, Pinak has clicked some photographs and it really looked cool.
Next two days were for talks. I attended most of the talks and the one which I found most interesting was Plasma Mobile by Sebastian Kügler. We also had group photo over there. At the end of the second day, Akademy awards were given to VDG group for amazing looks of plasma 5, Millian Wolff for his great work on KDevelop, Albert Vaca for creating awesome application KDE Connect and Akademy Organizers for their work and efforts.
From next day, BoFs were conducted and I attended the ones I was interested in. We also had a BoF session on kdeconnect in which we discussed about keeping single code base and making it cross platform and discussed the problems which we might face in that. More details of that BoF can be found in kdeconnect Mailing Archives. I also shared my idea of creating Synchronization Plugin for kdeconnect which would sync the specific folder across the devices and I already have created it but I’m having problems in testing and debug it out.
Other than that, I worked on Plasma Mediacenter and did some to-do from Bhushan’s list like fake media source, plugin infrastructure etc. I was the only one from PMC developers. I wished Bhushan, Shantanu and others could have joined me. I wasn’t able to work much because my laptop charger didn’t plug into European style socket. So, I always had to borrow it from someone else whenever it was free.
We also had a night party somewhere, I don’t remember the name of the place. Everyone had drinks, I only took soft ones ;P . We also had food over there and then dance as well. I interacted with other people there, with whom I didn’t get chance earlier to meet because of arriving late night at Rialta on 24th. It was really amazing.
I learnt about the cool architecture of baloo from Vishesh, met awesome/entertaining guy Àlex Fiestas, got to know more about KDE Frameworks from David Faure and much more. I got opportunity to meet other people from KDE India community, talked to people in person with whom I had only conversed online before and made new friends over there :D
I would also like to share some thoughts about my journey to Spain. Spain is very good place and I really enjoyed my whole journey. It was overall very good experience. I went from Madrid Airport to Bus Station via metro and then took bus from there to A Coruña. During return journey from A Coruña to Madrid, I traveled on train. I didn’t know Spanish but people were really helpful and Google Translate also helped me a bit. One thing that surprised me in Spain was sunset time. While traveling to A Coruña from Madrid by bus, there was clear sunlight even at 10:00 PM. I checked timezone twice on my phone and then browsed Internet if there are two time zones in this country. I even asked time from one person, sitting near me who knew English, to confirm that my phone was showing correct time. So, Nights are shorter over there. I went to aquarium there and has clicked some photographs over there. I also visited Torre de Hércules but didn’t go to top of it as I was feeling damn cold and I didn’t want to stand in long queue to get over there. I also have created 180° view near Torre de Hércules. Both, Aquarium and Tower of Hercules, were part of day trip present in Akademy schedule. Weather was really nice in A Coruña. It was the first city where I saw coastline as well as mountains at single place. I had some problem there to find food as I am a vegetarian, um, not exactly. I am actually a Lacto-Ovo-Vegetarian. So, most of the food items which I saw had meat.
I would like to thank José Millán Soto as well as other organizers for organizing such a well planned event. Looking forward to Akademy 2016!
Thanks a lot to KDE e.V. for the sponsorship without which it would not have had possible for me to attend such an awesome summit.
]]>I have completed the implementation of both DLNA/UPnP Media Server and Client as standalone applications and I have merged these applications into the plasma-mediacenter too but some errors are yet to be resolved related to C wrappers in C++
So, I currently can’t show DLNA/UPnP working inside the plasma-mediacenter but as soon as those errors would be resolved, I will update about it and currently you can check these standalone applications working good :)
The basic working of DLNA/UPnP Media Client is somewhat like this. Whenever new media server is found on the network and user clicks on expand button of a container, “Browse” Action is invoked, passing various standard arguments like Object ID(container id), starting index etc. and binding callback function. When media server sends the response, call back function is called with arguments having response data from the server. The XML response from the server is parsed, which gives the all the data and meta-data of the media files.
This whole procedure is done for each container or directory present on the Server. The libraries which has been used in the implementation are gupnp-1.0 gtk+-2.0 gssdp-1.0 gupnp-av-1.0 libxml-2.0
Above whole explanation holds true for the standalone application but there are minute changes in this if we talk upnp in context of PMC. Whenever any media server is found, it is going to scan each and every directory recursively, get all the information of the each and every media and store it into the MediaLibrary of plasma mediacenter which is built everytime pmc starts. Although there is limitation of MediaLibrary that we cannot remove any media from it once added and if DLNA/UPnP media server goes offline then, it would show those media files but would not play them. So, It need some changes to support removing media too and that’s where my role comes in. I will make changes in MediaLibrary and would be looking further to contribute into KDE :D
UPnP Media Server
And regarding the working of DLNA/UPnP Media Server, it just has to give response to all the actions invoked by the client application. That response include all the information about the media, didl container or item etc. according to the arguments passed to it. The libraries which I have used in upnp server implementation are gupnp-1.0 gupnp-av-1.0 libsoup-2.4
I really enjoyed this program and would like to contribute to this awesome and supporting community.
And last but not the least, my mentors Shantanu and Bhushan are very helping and supporting, and it was really not possible for me to implement this without them. A big thanks to them :)
I have used GUPnP library for this as it is the best and most stable library available to implement UPnP. I have used gupnp-tools as the reference. Till Now I have successfully finished the very first step of my project, i.e. show up the files of the MediaServer. I am able to browse the files on the Media Server successfully and that media is accessible over HTTP protocol using the URLs which is terminal output.
Here’s the screen shots of the working application :
In the root of the tree in application, there’s Device name being shown up in the first column and respective IDs in the other column. So, we can expand it down and browse the files further present on the media server. As we keep expanding it down, we will be getting URLs of all the audio, video, image and text files of those directories as the output on the terminal.
As I have got the URLs of the media, now I just have to pass those URLs to the existing PMC backend which would automatically be opening up the media file over the HTTP protocol, process up the metadata and shows all the controls like play, pause etc. But as I have said previously, I would be doing that integrating in PMC stuff later on.
What’s Next :
Now I am gonna work on implementation of Media Server in PMC using the same library. Using that, the user would be able to play the media files from PMC to any other device.
Season of KDE participants are required to blog about the work progress report but I am really kinda lazy in that and this is my first blog post I have ever made in my life :p
At last I would like to thank my mentors Shantanu and Bhushan, who helped me in each and every problem where I got stuck.
]]>