December, 8, 2016

Business hours check for a global business: The problem statement

Customer support teams of large global organizations have teams working in multiple geographies across multiple time zones. Each of these teams usually have their own set of business days and hours (in their respective time zones). Below is a Clojure map that represents an example business hours and days configuration for an org -

(def test-business-timings
  {:business-days [true true false true true false true] ;; Mon-Sun
   :business-hours [;; Monday
                    [{:from {:hours 16 :minutes 0}
                      :to {:hours 17 :minutes 0}
                      :timezone "Asia/Kolkata"}
                     {:from {:hours 13 :minutes 0}
                      :to {:hours 18 :minutes 0}
                      :timezone "America/Los_Angeles"}]
                    ;; Tuesday
                    [{:from {:hours 8 :minutes 0}
                      :to {:hours 18 :minutes 0}
                      :timezone "Pacific/Kiritimati"}]
                    ;; Wednesday
                    ;; Thursday
                    [{:from {:hours 1 :minutes 0}
                      :to {:hours 3 :minutes 0}
                      :timezone "UTC"}
                     {:from {:hours 10 :minutes 0}
                      :to {:hours 18 :minutes 0}
                      :timezone "Asia/Kolkata"}]
                    ;; Friday
                    [{:from {:hours 0 :minutes 0} ;;all day
                      :to {:hours 0 :minutes 0}
                      :timezone "US/Hawaii"}]
                    ;; Saturday
                    ;; Sunday
                    [{:from {:hours 9 :minutes 0}
                      :to {:hours 17 :minutes 0}
                      :timezone "US/Hawaii"}]]})

When a support ticket from a customer comes in, we need an automated way to check whether any of the global support teams are within business hours.

Approaching the problem

Which day of the week is it for the support teams?

Before making the check for business days, you need to determine what the current day of the week is (‘today’). You need to account for the possibilities that it might still be ‘yesterday’ in some timezone or may already be ‘tomorrow’ in some timezone. And a team in some timezone behind or ahead of the server time may be in business hours.

On this day and at this time, do I have an actively working support team somewhere or not?

When making a check for a specific business day, building the time interval for the specified business hours (for that day) needs to take into account both timezone and the notion of what is the current day.


An approach that didn’t fly

Convert all the input business hours to UTC before making the check. There are 2 problems with that approach -

  • Shifting the business hours to UTC could mean crossing the day boundary (ahead or behind). And this gets complicated by the fact that we need to factor in business days (not just hours). We’d have to also build a transformed list of UTC business days.
  • We can’t store the business hours and days configuration in UTC format because the next time the user edits them, they’ll need to see it in the time zones they originally used.

The one that did

The solution relies on the fact that no time zone in the world is more than a day apart from UTC. In other words, no matter what timezone you’re in, relative to the UTC day it could never be ‘day after tomorrow’ nor could it still be ‘day before yesterday’.

In order to solve this problem, we assume whatever is the current day of week per UTC to be our current day of week. And then we need to make the following 3 checks -

  • whether ‘today’ is a business day, and whether ‘right now’ falls in ‘today’s’ business hours
  • whether ‘tomorrow’ is a business day, and whether ‘right now’ falls in ‘tomorrow’s’ business hours
  • whether ‘yesterday’ is a business day, and whether ‘right now’ falls in ‘yesterday’s’ business hours


Before I dive into the actual code, I’d like to side step a bit and dwell on how using Clojure (and FP in general) makes the solution intuitive.

Functions are Fun. And natural.

I started out as a Java programmer and I clearly remember that in my first few months at my first job, my natural tendency was to write a lot of C-style functions. And the way to do that in Java is to use static methods and classes. And this immediately stands out as a code smell or bad practice in the object oriented Java world. You need to think about modeling things as ‘real world objects’ I was told. Whatever that meant. Of course I adapted, moved on and eventually became a ‘good’ Java programmer. Working with Clojure now has made me rediscover that long lost natural tendency. I now write code in the style that is most obvious from a logic perspective. Little functions that each do one thing. I’ve come full circle - from instinctively thinking that functions are good, to being told that functions are bad, to finally understanding that functions are good. Java 8 now has functions you may say. But the truth is that just ‘supporting’ or ‘allowing’ functions as first class citizens is very different from a being a functional language at the heart. Which Java 8 certainly isn’t.

IMHO - an important characteristic of good code is that it makes the code look simple or easy. Its like a good batsman (in cricket) makes batting look easy. And this is something that I find in abundance in the Clojure ecosystem.

Clojure’s awesome date time lib : clj-time

Clojure has a feature rich, intuitive, and expressive date time library called clj-time. It’s an excellent example of what an easy-to-use API should look like. And using such a lib adds to the simplicity and elegance of your code.

Lets write the code

We’ll walk through and build the solution from the bottom up. The intent is to highlight how functional programming is the most natural way of building a solution with small, coherent, re-usable and composable functions.

We first create a Clojure NS for the solution. The only external lib we need is the above mentioned date time lib.

(ns business-timings.core
  (:require [clj-time.core :as t]))

First off, three simple functions that tell us ‘today’, ‘tomorrow’, and ‘yesterday’ for a specific UTC date time instance. Each of these return a number in the range of 1-7 representing the day of the week.

(defn- today
  (t/day-of-week dt))

(defn- tomorrow
  (t/day-of-week (t/plus dt (t/days 1))))

(defn- yesterday
  (t/day-of-week (t/minus dt (t/days 1))))

Now, a couple of functions that allows us to read the business hours for a specified day of week, and tell us whether a given day is a business day or not.

(defn- business-day?
  [day business-timings]
  ;; decerement day for zero based index
  (get (:business-days business-timings) (dec day)))

(defn- timings-for-day
  [day business-timings]
  ;; decerement day for zero based index
  (get (:business-hours business-timings) (dec day)))

Lets now use all the above to create functions that can tell us the corresponding timings for today, tomorrow, and yesterday. And whether they are business days.

(defn- timings-today
  [business-timings dt]
  (timings-for-day (today dt) business-timings))

(defn- timings-tomorrow
  [business-timings dt]
  (timings-for-day (tomorrow dt) business-timings))

(defn- timings-yesterday
  [business-timings dt]
  (timings-for-day (yesterday dt) business-timings))

(defn- business-day-today?
  [business-timings dt]
  (business-day? (today dt) business-timings))

(defn- business-day-tomorrow?
  [business-timings dt]
  (business-day? (tomorrow dt) business-timings))

(defn- business-day-yesterday?
  [business-timings dt]
  (business-day? (yesterday dt) business-timings))

Next, once we’ve read the business hours for a day, we’ll need some functions to help us build the actual time intervals for those business hours. In other words, the date time instances corresponding to the start time and end time of the business hours (on a given day). Note - these start and end date-time instances will need to be created in the timezone used in the configuration.

(defn- business-hours-start-today
  [time-slot dt]
    (t/date-time (t/year dt) (t/month dt) (t/day dt)
                 (get-in time-slot [:from :hours])
                 (get-in time-slot [:from :minutes]))
    (t/time-zone-for-id (:timezone time-slot))))

(defn- business-hours-end-today
  [time-slot dt]
  (let [end-hour (get-in time-slot [:to :hours])
        end-minute (get-in time-slot [:to :minutes])]
      (if (every? zero? [end-hour end-minute])
        (t/date-time (t/year dt) (t/month dt) (t/day dt) 23 59 59 999)
        (t/date-time (t/year dt) (t/month dt) (t/day dt) end-hour end-minute))
      (t/time-zone-for-id (:timezone time-slot)))))

(defn- busines-hours-start-yesterday
  [time-slot dt]
  (t/minus (business-hours-start-today time-slot dt) (t/days 1)))

(defn- business-hours-end-yesterday
  [time-slot dt]
  (t/minus (business-hours-end-today time-slot dt) (t/days 1)))

(defn- business-hours-start-tomorrow
  [time-slot dt]
  (t/plus (business-hours-start-today time-slot dt) (t/days 1)))

(defn- business-hours-end-tomorrow
  [time-slot dt]
  (t/plus (business-hours-end-today time-slot dt) (t/days 1)))

Lets now use the above to create check functions for today, tomorrow, and yesterday. The check passes if the specified date time instance falls in any of the time slots configured for a particular day. As you may remember, the problem statement requires allowing multiple different time slots for the same day (each with its own timezone). We use clj-time’s within? function (which is timezone aware) to make this check. Hence we don’t need to shift the specified date time instance to the timezone used for the business hours.

(defn within-todays-business-timings?
  [business-timings dt]
  (when (business-day-today? business-timings dt)
    (let [time-slots (timings-today business-timings dt)]
      (some (fn [time-slot]
              (t/within? (business-hours-start-today time-slot dt)
                         (business-hours-end-today time-slot dt)

(defn within-tomorrows-business-timings?
  [business-timings dt]
  (when (business-day-tomorrow? business-timings dt)
    (let [time-slots (timings-tomorrow business-timings dt)]
      (some (fn [time-slot]
              (t/within? (business-hours-start-tomorrow time-slot dt)
                         (business-hours-end-tomorrow time-slot dt)

(defn within-yesterdays-business-timings?
  [business-timings dt]
  (when (business-day-yesterday? business-timings dt)
    (let [time-slots (timings-yesterday business-timings dt)]
      (some (fn [time-slot]
              (t/within? (busines-hours-start-yesterday time-slot dt)
                         (business-hours-end-yesterday time-slot dt)

And now the final check function that will essentially be the API for this NS. And it’ll use all the functions we have defined so far. The check passes if the specified date time instance falls in either today, tomorrow, or yesterday’s business hours.

(defn within-business-timings?
  [business-timings dt]
  (or (within-todays-business-timings? business-timings dt)
      (within-tomorrows-business-timings? business-timings dt)
      (within-yesterdays-business-timings? business-timings dt)))

(comment (within-business-timings? test-business-timings (t/now)))


Functional programming is not about coolness or the bleeding edge. Rather it is - in my humble opinion - the most natural and easy way to solve a problem. With the side benefit that its also cool, and the bleeding edge. For full code and tests, see here

Moiz Jinia's photograph
Moiz Jinia
Backend Clojure Dev @ Helpshift
October, 10, 2016

Today we are excited to to open source Herald - a load feedback and check agent for Haproxy.

Herald is the agent service for the agent-check feature in Haproxy. This feature allows a backend server to issue commands and control its state in Haproxy. This can be used for out of band health checks, or load feedback (which we explain below), and many other use cases.


At Helpshift we swear by Haproxy, which is an extremely powerful and versatile load balancer.

Haproxy is used to balance requests from a frontend (client facing interface) to a pool of backend servers. For simple http requests the roundrobin balancing algorithm works well. But for long lived connections, an even balancing is not achieved. The leastconn balancing method helps to a certain extent - here the requests are sent to the backend with least connections - but in a cluster of Haproxy nodes, this doesn’t work, as no state regarding the connections is shared between the clustered haproxy nodes.

To solve this, we need a mechanism to regulate traffic sent by haproxy, depending on the load condition of the respective backend server in realtime. We built Herald to solve this load feedback use case.

Load feedback

As the name suggests, the backend servers can send feedback to Haproxy and regulate its incoming traffic. The application on the backends must expose its load status (a metric such as rps) over some interface, such as http.

When agent-check is enabled, Haproxy periodically opens a tcp connection to herald running on the backend servers. The agent on the server queries the application via the load status interface, and replies back with a weight percentage, say 75%. This directs haproxy to reduce the traffic sent to this node by that percent difference. This percentage can keep changing as per the current traffic condition.

Besides regulating load percentage, the reponse can also be an Haproxy action, such as MAINT, DOWN, READY, UP etc. The Haproxy documentation here has more details on agent-check.


Herald is the Haproxy agent that runs on the backends. It has been designed to be generic, with a plugin architecture that along with load feedback can be used for other use cases as well.

The agent has two responsibilities:

  1. Respond to Haproxy agent requests, and
  2. Query backend application, and calculate the haproxy agent response

The following illustrates this simple architecture :

Herald Architecture


Herald plugins do the job of querying the application and interpreting the result. Plugins for file and http are available, others can be easily added.

Following features are provided out of the box by the plugin framework:

  • Response result cacheing
  • Json parsing and processing parsed data using python dict expressions
  • Arthmetic expressions on the result
  • Regex pattern matching on the result
  • Fallback in case of failures


Being entirely network and IO driven, we chose to use gevent, a coroutine based python networking library. Gevent uses greenlets, to spawn cooperatively scheduled tasks, which yield when blocked, such as when waiting for IO. The resulting code is very simple, and highly performant.

We have been using Herald in production for almost a year with no issues, on a cluster with 100+ instances (depending on autoscaling), serving requests from a cluster of 10+ load balancers. We have had 100% uptime on our load feedback and agent check system.

Code and Documentation

The source code is on Helpshift github page. Please follow the README for installation and configuration instructions. We hope Herald is useful to others. Feature and pull requests are most welcome.

Raghu Udiyar's photograph
Raghu Udiyar
DevOps @ Helpshift
November, 20, 2015


Testing business logic quickly becomes tricky, as applications grow and scale. Example based unit and integration tests, and exploratory tests become poor choices to check and verify a large state space. Also such methods are not well-suited to clearly describe the state space / transitions we want to test.

For quite some time I have been experimenting with DSLs to write our tests at Helpshift. I recently gave a talk on “Testing business logic using DSLs in Clojure” at Functional Conf this year. I have shared my learning in the talk. Hope you enjoy it. Feel free to ask me any questions if you have.



If this sounds interesting to you, we are hiring, join us!

Mayank Jain

Mayank Jain's photograph
Mayank Jain
Backend Engineering
April, 10, 2015

Google is phasing out OpenID in favour of Oauth 2.0 with a deadline on 20th April, 2015 - just 10 days from today. A lot of projects depend on google auth, and can’t easily move to another OpenID provider. I recently had to fix this issue with Jenkins and Gerrit.

Jenkins has a great plugin available for this, which was a piece of cake to install and configure. But it wasn’t so easy with Gerrit. Lot of gerrit users have been asking for Oauth support since May last year; we got that finally when David Ostrovsky wrote gerrit-oauth-provider plugin.

I’ve listed the steps I followed below :

  1. Oauth2 credentials

    Get these from Google Developers Console, note down the client id and client secret. Ensure the redirect url is set to /oauth i.e.

  2. Get the custom gerrit war file

    There are a few gerrit changes the plugin needs that haven’t been merged yet. A custom war is available here with the plugin. Download this gerrit-2.10.2-18-gc6c5e0b.war file to the new gerrit server.

  3. Backup current gerrit data

    Create tarballs of the data directories and dump postgres data (if postgres is being used)

    old-gerrit~$ tar czpf gerrit.tar.gz /srv/gerrit/gerrit old-gerrit~$ tar czpf repositories.tar.gz /srv/gerrit/repositories old-gerrit~$ pg_dump -xO -Fc reviewdb > reviewdb-$(date +%d-%m-%Y).pdump

  4. Restore data to new gerrit server

    gerrit:/srv/gerrit$ tar xzpf repositories.tar.gz gerrit:/srv/gerrit$ tar xzpf gerrit.tar.gz

  5. Restore pg data

    psql : ALTER USER gerrit WITH SUPERUSER; $ dropdb reviewdb $ createdb reviewdb -O gerrit $ pg_restore -O -d reviewdb --role=gerrit reviewdb-20-03-2015.pdump psql: ALTER USER gerrit WITH NOSUPERUSER;

  6. Run migrations

    Gerrit requires cascading migrations to be run for every major version released. For e.g to update from 2.5 to 2.10, we have to run the following

    $ sudo su - gerrit -s /bin/bash $ java -jar gerrit- init -d gerrit $ java -jar gerrit-2.9.4.war init -d gerrit $ java -jar gerrit-2.9.4.war reindex --recheck-mergeable -d gerrit For the custom jar migration be sure to configure the Oauth plugin

    ``` $ java -jar gerrit-2.10.1-4-a83387b.war init -d gerrit […] OAuth Authentication Provider

    Use Google OAuth provider for Gerrit login ? [Y/n]? Application client id : Application client secret : confirm password : Link to OpenID accounts? [true]: Use GitHub OAuth provider for Gerrit login ? [Y/n]? n

    $ java -jar gerrit-2.10.1-4-a83387b.war reindex -d gerrit ```

  7. Switch old gerrit domain name to the new server

    For automatic acount linking to work, the domain name must match the old server. Otherwise the OpenID accounts will not be linked with the new Oauth2 account.

  8. Start gerrit server and confirm everything works

    gerrit:/srv/gerrit$ ./bin/ start

Raghu Udiyar's photograph
Raghu Udiyar
DevOps @ Helpshift
March, 16, 2015


“Make integration easier. No this is too much work for the devs, make it even easier”

This has been a line which is constantly heard if you wander over to the Mobile SDK team here at Helpshift. This is something that we have tried to strictly adhere to. We have made rapid strides in that direction for both our iOS and Android native SDKs. Gradle and Maven support has made our Android integration as simple as adding a single line. The Helpshift Integration Assistant has turned the Helpshift iOS integration as easy as installing any software on your Mac.

But sadly, we have lacked that kind of sophistication with our Unity plugins. The Helpshift Unity integration is still tedious, confusing and fraught with errors when interacting with other Unity plugins. So we had to fix this situation.

Why it is important

Gaming has been one of our champion verticals when it comes to the Helpshift CRM solution. We have made quite a foray into the Gaming market and many of the top game companies, including Supercell, have been using our product for some time now. And a majority of the Game studios today use Unity for developing their games.

With a lot of adoption, comes a lot of different scenarios. We have had a ton of feedback from our customers about our Unity plugin. Customers didn’t like the fact that we had 2 separate plugins for iOS and Android. They ran into issues due to conflicts with other plugins. The variety of folder structures that game devs follow complicated matters for us even further. This meant a big roadblock for all important customers and too much time spent in integration which undermined the core philosophy of the SDK team.

Humble beginnings

A bit of history is always good to set things in the right context. About a year and a half back, when we had just released the 1.0 version of the Helpshift SDK, our CTO, bg came to me and gave me a challenge. “There’s this game development engine, Unity or something. Go and see if you can make our SDK work with a Unity game!” That led to a frantic 4 hour hack into making a Unity app and integrating our iOS SDK with it. The C# API was basically a single function back then which would launch the FAQs screen from our SDK. And integrating the plugin meant manually copying files over from one location to another. Since then, we kept making improvements piece-meal to the Unity plugin and the integration process too. But there were still gaping holes in the whole thing.

The improvements

One plugin to rule them all

As we found out to our woe, distributing 2 separate plugins for iOS and Android was not ideal. We had to unify the packaging as well as the Helpshift API. This was fairly easy to achieve by creating a common API wrapper and adding UNITY_IOS and UNITY_ANDROID conditionals in there.

The plugin should be a Unitypackage

This was one of the most frequent feedbacks that we kept getting. Unity offers an easy way to add files to the project -> the .unitypackage. We needed to distribute our plugin in the same way to make it easier for developers to add the Helpshift SDK. Turned out, it was easy to create a .unitypackage from the Unity Studio UI but we wanted to make it part of the release process which meant we had to do it via the command line. Fortunately for us, some google searching led us to a good starting point. Given below is the shell script that we use to create a .unitypackage from an existing Unity project.

# Exports the Helpshift unitypackage from given Unity project
# arg-1 : path of the Unity project where Helpshift is integrated
# arg-2 : path where file needs to be exported
# arg-3 : version number
if [ "$#" -ne 3 ] || ! [ -d "$1" ]; then
    echo "Usage: $0 <path_to_Unity_project> <output_path> <version_number>" >&2
exit 1


DIR_1=Assets/<dir containing your plugin files>
DIR_2=Assets/Plugins/Android # for android plugin related files
"$BIN_PATH/$BIN_FILE" -batchmode -projectPath "$PROJECT_PATH" -logFile export.log -exportPackage "$DIR_1" "$DIR_2" "$EXPORT_PATH/$EXPORT_FILE" -quit

Remove conflicts with other Unity plugins

One of the things that was a constant pain for us was the conflicts with Unity plugins like Facebook which used the XCode Editor for Unity project to modify the XCode project generated by Unity. We internally used the more humble mod_pbxproj python file which did everything we wanted to do and did it well. But the XCodeEditor project unfortunately messed around with the XCode project after we were done adding our files to it. Specifically, it removed any PBXVariantGroups which were present in the project. This meant our Localization files went for a toss and app could not see any strings.

No strings found

To get around this issue, we had to make sure that our integration ran after any conflicting plugins and that we adopted this integration method right from the start. To achieve this, we had to add the HelpshiftPostProcess.cs file which could take a PostProcessBuild attribute which could be set to a high value. The higher the value, the later in the process it would be run. This file should be a Monobehaviour type of script which lies in the Editor folder of your plugins subfolder. For example, Assets/Helpshift/Editor/HelpshiftPostProcess.cs

using System;
using UnityEngine;
using UnityEditor;
using UnityEditor.Callbacks;

using System.Diagnostics;
using System.Collections;
using Helpshift;
public class HelpshiftPostProcess : MonoBehaviour
  // Set PostProcess priority to a high number to ensure that the this is started last.
  public static void OnPostprocessBuild(BuildTarget target, string pathToBuildProject)
    const string helpshift_plugin_path = "Assets/Helpshift";
    // Current path while executing the script is
    // the project root folder.
    Process myCustomProcess = new Process();
    myCustomProcess.StartInfo.FileName = "python";
    myCustomProcess.StartInfo.Arguments = string.Format(helpshift_plugin_path + "/Editor/HSPostprocessBuildPlayer " + pathToBuildProject + " " + target);
    myCustomProcess.StartInfo.UseShellExecute = false;
    myCustomProcess.StartInfo.RedirectStandardOutput = false;

Make the Unity Android plugin integration easier

The current Unity android plugin involves manually adding entries to an existing AndroidManifest.xml file, combining our res folder with the existing res folders. It was all a pain and there was minimal isolation of the Helpshift files. Recently, Unity added support for integrating Android library projects present in the Assets/Plugins/Android folder. We decided to take advantage of this fact and distribute our plugin the same way. Now our Android integration just involves the Helpshift library project to the Assets/Plugins/Android folder and letting Unity take care of the rest.

Add Push notifications support

One of the difficult to achieve features for us was Push notification integration with Helpshift. There seemed to be too many different push plugins which each did things their own way. This led to many problems for our users to effectively use the excellent Push support which our backend systems provide. To get around this problem, we basically had to develop our own Unity push plugins which were built in to the SDKs. This was comparitively easy to do on the Android platform, but for iOS we had to resort to the black art of Method swizzling. But we had to make sure that there was a way for the app developers to switch this support off in the off-chance that the App store rejects the app. (We have contacted Apple support and made sure that the way we use swizzling is not something that is cause for rejection.)

To use the in-built push support, developers can simply add the below API:


Make the SDK configuration easy for developers

This was actually one of the interesting snippets of feedback that we got from our awesome customers. To quote - “If you could add like a GUI inspector to configure the Helpshift SDK, that would be really hot!” We figured, let’s give that a try. We had to go through some pretty unclear documentation about how to create a CustomEditor in Unity. Basically it involves creating an Object which derives from the Editor class and overriding the OnInspectorGUI method inside it. Then all we had to do was create the UI using the widgets in the EditorGUILayout class. To store the configuration values, we wrote a Scriptableobject which would hold all the values returned from the GUI. Saving the values could be done when the build was about to be created. We used the AssetDatabase class and it’s methods to achieve this result.

AssetDatabase.CreateAsset(instance, fullPath);

Just a footnote, make sure all your instance members are Serializable by adding the [SerializeField] annotation. If you want to add some MenuItems to the Unity editor, you can do so by adding code snippets like below to the Editor class.

[MenuItem("Helpshift/Edit Config")]
public static void Edit()
	Selection.activeObject = Instance;

[MenuItem("Helpshift/Developers Page")]
public static void OpenAppPage()
	string url = "";

The next step for us was actually passing along this configuration information to the XCode and Android projects. To do that we resorted to using JSON. The class which hold the config values would spit out the json representation of the configuration. These JSON files would then be added to the XCode project as simple data files and to the Android project as raw resources placed in the res/raw folder. Some wiring code to read them from the native SDKs and we were ready to roll.

The result

Helpshift Config Editor


Always listen to your customers

In conclusion, I would really love to thank all of our existing customers who gave us awesome constructive feedback and spurred us on to better our own standards. Hopefully all of you will love the changes that we have made, and also come up with even better suggestions to help us keep improving ourselves.

Exciting times ahead.

Rhishikesh Joshi's photograph
Rhishikesh Joshi
Mobile SDK team

Helpshift is an in-app mobile help desk designed to improve customer support efficiency by over 400% and reduce cost by over 70%. Our engineering team is committed to building a meticulously designed, solid SDK to improve customer retention for clients such as Flipboard, Supercell, and more . We are currently serving over 500 million app sessions weekly.

Learn more about us at