A Wednesday

It’s just another Thursday afternoon in Kemang Timur for the agents and guards. Kemang Timur DSU, the place where faceoff battle was won.

17 hours to Go-Live: We reached the DSU at around 2 PM with a mission to add data in prod db, take driver-qr service to prod and fix a breaking contract in IMS and Transaction service due to contract change in largo (3rd party service). And then, we were all set for GoLive. We couldn’t sleep well the past day due to late night connecting flights. As soon as we entered, we got some context for a new requirement to be done — having replenishment service in the popup flow. We were thinking on the approaches to do so.

Largo is still not live even when I am typing the blog. So I added the toggle feature for Largo to log the request than actually making one along with fixing the breaking contract changes. Never trust third parties! Ashu was going with the breaking changes of transaction service and fixing the specs while Rishabh was moving the qr service to production till we got the security review call. Result: High vulnerability bug found at our end in cicilan being applied along with some low priority bugs.

13 hours to Go-Live: About the bug: We weren’t checking the driver id given with the one in our jwt. So we can apply cicilan for some other driver. And also multiple calls would result in multiple cicilan’s being created. The QR service is deployed on prod but it’s failing. Ashu is working on debugging it while Me and Rishabh paired for the security fix. Our Security Fix: Check the driver_id with the one in JWT and have a redis with ttl for key as 24 hours (any number greater than validity of jwt). So store the jwt token with driver id in redis and if it already exists then don’t do anything. Time taken for fix: 0.5 hours coding + 1 hour testing. Ashu narrowed the issue down to reddis connectivity. Me and Ashu started pairing for fixing it while Rishabh paired with Chetan for specs and testing the security flow. On going to the prod redis, we found the systemd redis-server was failing due to passrequire error. We changed the redis-configuration to not have passrequire key in prod as our redis instance does not has a password. Apparently we couldn’t find a way to add password for a redis instance by proctor. The backend for QR service was now working. And the security flow was also fixed now with a Good to go given from security guys. Yay!!! But not for long.

10 hours to Go-Live: Android app is not showing the driver qr code while the backed is live in prod. On seeing the code of Driver-service, we were trying to figure out the route. Driver app and GoHost don’t work like Agent app in which app has a backend and it’s the backend responsibility to route to wherever. We figured out that the kong ACL for driver qr is only on integration and nobody of us have access to Atlas team to submit PR and merge the pipeline! Param to the rescue. We deployed it with him to the Prod. He told security guys to run the pipeline and damn, was that the fastest PR merge by the security guys. Having a TL do a job does it really fast.

8 hours to Go-Live: Driver app QR code is working. Now our job is to add the data to the database and test the flow once. We went for dinner.

7 hours to Go-Live: We got to know that the driver team has toggled off this qr code feature in prod as this was giving error. Subhankar looped all the people in (Vishy, Rajeev, Anusha) and made sure that Driver team turn the toggle on. Raza, Andika, Gede were with us since late and were telling us the bugs and pinging us for whitelisting. We really need some whitelisting portal where they can do the same on the own.

5 hours to Go-Live: IMS is up and running with data, so are pricing and transaction. Discount service has a good amount of things to do in their rake task. I went for a nap while Rishabh and Ashu added discount service data via rake tasks. After about 2 hours of sleep I woke up to the data being added in the rake tasks but this process was very slow as the pipeline was breaking multiple times in production. Some listen error was coming. Ashu-Rishabh confirmed that this had been coming since past 10–15 days. The reason this was coming was because Rails was failing over watcher. As Prod didn’t have that much bandwidth for listening to every file. The reason there was watcher in the first place was because we were running the dev build on integration and production. We set the environment variable and added the production build started running which succeeded every pipeline request every made since them.The RAILS_ENV was not set in environment variables. This was to be set in Pricing, Transaction, Discount and Inventory Service. But then the logs started to disappear since the environment is now Production. Damn Rails! There is this environment variable to be set for logs to be logged to STDOUT. We removed this config and ran all the pipelines again. Never hate cats and logs!

2 hours to Go-Live: We added the prod url’s in knights-watch. Well, we have 7 internal services so this took some time. We kubetailed the logs in 8 terminal windows and were checking the flow now! Rishabh & Subhankar went for a power nap. Finally everything should work. Damn Central auth! It was giving 404 error in getting the office API call. We cloned the repo and tested it. Sadly everything was working and so was on integration. We dived in to the production database and analysed to see if there was any data which was not entered well. At least 404 is something missing. After debugging for an hour we discussed the possibility of having integration url on prod for another 20 minutes. You see how much sleep deprived were we! Running an integration url on a prod cluster. Well the reason was that Office was earlier a GET api but it was refactored to POST. But this commit was not deployed on production. Last production pipeline ran 72+ odd days ago.

45 minutes to Go-Live: Transaction service was not running the sidekiq. Some prod pipeline rerun things. We never reran the sidekiq prod pipeline for transaction. I turned the toggle on for IMS to log the request than to send it to IMS. Then flow was working. Finally everything was working. I turned the toggle off for IMS and went back to hotel. We have to come for the launch again! None of us woke for another 1.5 hours and here we are in the morning and witnessing wide driver smiles who were taken by surprise by the GoJek agents who went to their homes to give free attributes. Ask me, It was totally worth it. Not more than the image below though!