October, 10, 2016

Today we are excited to to open source Herald - a load feedback and check agent for Haproxy.

Herald is the agent service for the agent-check feature in Haproxy. This feature allows a backend server to issue commands and control its state in Haproxy. This can be used for out of band health checks, or load feedback (which we explain below), and many other use cases.

Background

At Helpshift we swear by Haproxy, which is an extremely powerful and versatile load balancer.

Haproxy is used to balance requests from a frontend (client facing interface) to a pool of backend servers. For simple http requests the roundrobin balancing algorithm works well. But for long lived connections, an even balancing is not achieved. The leastconn balancing method helps to a certain extent - here the requests are sent to the backend with least connections - but in a cluster of Haproxy nodes, this doesn’t work, as no state regarding the connections is shared between the clustered haproxy nodes.

To solve this, we need a mechanism to regulate traffic sent by haproxy, depending on the load condition of the respective backend server in realtime. We built Herald to solve this load feedback use case.

Load feedback

As the name suggests, the backend servers can send feedback to Haproxy and regulate its incoming traffic. The application on the backends must expose its load status (a metric such as rps) over some interface, such as http.

When agent-check is enabled, Haproxy periodically opens a tcp connection to herald running on the backend servers. The agent on the server queries the application via the load status interface, and replies back with a weight percentage, say 75%. This directs haproxy to reduce the traffic sent to this node by that percent difference. This percentage can keep changing as per the current traffic condition.

Besides regulating load percentage, the reponse can also be an Haproxy action, such as MAINT, DOWN, READY, UP etc. The Haproxy documentation here has more details on agent-check.

Herald

Herald is the Haproxy agent that runs on the backends. It has been designed to be generic, with a plugin architecture that along with load feedback can be used for other use cases as well.

The agent has two responsibilities:

  1. Respond to Haproxy agent requests, and
  2. Query backend application, and calculate the haproxy agent response

The following illustrates this simple architecture :

Herald Architecture

Plugins

Herald plugins do the job of querying the application and interpreting the result. Plugins for file and http are available, others can be easily added.

Following features are provided out of the box by the plugin framework:

  • Response result cacheing
  • Json parsing and processing parsed data using python dict expressions
  • Arthmetic expressions on the result
  • Regex pattern matching on the result
  • Fallback in case of failures

Performance

Being entirely network and IO driven, we chose to use gevent, a coroutine based python networking library. Gevent uses greenlets, to spawn cooperatively scheduled tasks, which yield when blocked, such as when waiting for IO. The resulting code is very simple, and highly performant.

We have been using Herald in production for almost a year with no issues, on a cluster with 100+ instances (depending on autoscaling), serving requests from a cluster of 10+ load balancers. We have had 100% uptime on our load feedback and agent check system.

Code and Documentation

The source code is on Helpshift github page. Please follow the README for installation and configuration instructions. We hope Herald is useful to others. Feature and pull requests are most welcome.

Raghu Udiyar's photograph
Raghu Udiyar
DevOps @ Helpshift
November, 20, 2015

Background

Testing business logic quickly becomes tricky, as applications grow and scale. Example based unit and integration tests, and exploratory tests become poor choices to check and verify a large state space. Also such methods are not well-suited to clearly describe the state space / transitions we want to test.

For quite some time I have been experimenting with DSLs to write our tests at Helpshift. I recently gave a talk on “Testing business logic using DSLs in Clojure” at Functional Conf this year. I have shared my learning in the talk. Hope you enjoy it. Feel free to ask me any questions if you have.

Talk

Slides

If this sounds interesting to you, we are hiring, join us!

Mayank Jain

Mayank Jain's photograph
Mayank Jain
Backend Engineering
April, 10, 2015

Google is phasing out OpenID in favour of Oauth 2.0 with a deadline on 20th April, 2015 - just 10 days from today. A lot of projects depend on google auth, and can’t easily move to another OpenID provider. I recently had to fix this issue with Jenkins and Gerrit.

Jenkins has a great plugin available for this, which was a piece of cake to install and configure. But it wasn’t so easy with Gerrit. Lot of gerrit users have been asking for Oauth support since May last year; we got that finally when David Ostrovsky wrote gerrit-oauth-provider plugin.

I’ve listed the steps I followed below :

  1. Oauth2 credentials

    Get these from Google Developers Console, note down the client id and client secret. Ensure the redirect url is set to /oauth i.e. http://gerrit.yoursite.com/oauth.

  2. Get the custom gerrit war file

    There are a few gerrit changes the plugin needs that haven’t been merged yet. A custom war is available here with the plugin. Download this gerrit-2.10.2-18-gc6c5e0b.war file to the new gerrit server.

  3. Backup current gerrit data

    Create tarballs of the data directories and dump postgres data (if postgres is being used)

    old-gerrit~$ tar czpf gerrit.tar.gz /srv/gerrit/gerrit old-gerrit~$ tar czpf repositories.tar.gz /srv/gerrit/repositories old-gerrit~$ pg_dump -xO -Fc reviewdb > reviewdb-$(date +%d-%m-%Y).pdump

  4. Restore data to new gerrit server

    gerrit:/srv/gerrit$ tar xzpf repositories.tar.gz gerrit:/srv/gerrit$ tar xzpf gerrit.tar.gz

  5. Restore pg data

    psql : ALTER USER gerrit WITH SUPERUSER; $ dropdb reviewdb $ createdb reviewdb -O gerrit $ pg_restore -O -d reviewdb --role=gerrit reviewdb-20-03-2015.pdump psql: ALTER USER gerrit WITH NOSUPERUSER;

  6. Run migrations

    Gerrit requires cascading migrations to be run for every major version released. For e.g to update from 2.5 to 2.10, we have to run the following

    $ sudo su - gerrit -s /bin/bash $ java -jar gerrit-2.8.6.1.war init -d gerrit $ java -jar gerrit-2.9.4.war init -d gerrit $ java -jar gerrit-2.9.4.war reindex --recheck-mergeable -d gerrit For the custom jar migration be sure to configure the Oauth plugin

    ``` $ java -jar gerrit-2.10.1-4-a83387b.war init -d gerrit […] OAuth Authentication Provider

    Use Google OAuth provider for Gerrit login ? [Y/n]? Application client id : Application client secret : confirm password : Link to OpenID accounts? [true]: Use GitHub OAuth provider for Gerrit login ? [Y/n]? n

    $ java -jar gerrit-2.10.1-4-a83387b.war reindex -d gerrit ```

  7. Switch old gerrit domain name to the new server

    For automatic acount linking to work, the domain name must match the old server. Otherwise the OpenID accounts will not be linked with the new Oauth2 account.

  8. Start gerrit server and confirm everything works

    gerrit:/srv/gerrit$ ./bin/gerrit.sh start

Raghu Udiyar's photograph
Raghu Udiyar
DevOps @ Helpshift
March, 16, 2015

Background

“Make integration easier. No this is too much work for the devs, make it even easier”

This has been a line which is constantly heard if you wander over to the Mobile SDK team here at Helpshift. This is something that we have tried to strictly adhere to. We have made rapid strides in that direction for both our iOS and Android native SDKs. Gradle and Maven support has made our Android integration as simple as adding a single line. The Helpshift Integration Assistant has turned the Helpshift iOS integration as easy as installing any software on your Mac.

But sadly, we have lacked that kind of sophistication with our Unity plugins. The Helpshift Unity integration is still tedious, confusing and fraught with errors when interacting with other Unity plugins. So we had to fix this situation.

Why it is important

Gaming has been one of our champion verticals when it comes to the Helpshift CRM solution. We have made quite a foray into the Gaming market and many of the top game companies, including Supercell, have been using our product for some time now. And a majority of the Game studios today use Unity for developing their games.

With a lot of adoption, comes a lot of different scenarios. We have had a ton of feedback from our customers about our Unity plugin. Customers didn’t like the fact that we had 2 separate plugins for iOS and Android. They ran into issues due to conflicts with other plugins. The variety of folder structures that game devs follow complicated matters for us even further. This meant a big roadblock for all important customers and too much time spent in integration which undermined the core philosophy of the SDK team.

Humble beginnings

A bit of history is always good to set things in the right context. About a year and a half back, when we had just released the 1.0 version of the Helpshift SDK, our CTO, bg came to me and gave me a challenge. “There’s this game development engine, Unity or something. Go and see if you can make our SDK work with a Unity game!” That led to a frantic 4 hour hack into making a Unity app and integrating our iOS SDK with it. The C# API was basically a single function back then which would launch the FAQs screen from our SDK. And integrating the plugin meant manually copying files over from one location to another. Since then, we kept making improvements piece-meal to the Unity plugin and the integration process too. But there were still gaping holes in the whole thing.

The improvements

One plugin to rule them all

As we found out to our woe, distributing 2 separate plugins for iOS and Android was not ideal. We had to unify the packaging as well as the Helpshift API. This was fairly easy to achieve by creating a common API wrapper and adding UNITY_IOS and UNITY_ANDROID conditionals in there.

The plugin should be a Unitypackage

This was one of the most frequent feedbacks that we kept getting. Unity offers an easy way to add files to the project -> the .unitypackage. We needed to distribute our plugin in the same way to make it easier for developers to add the Helpshift SDK. Turned out, it was easy to create a .unitypackage from the Unity Studio UI but we wanted to make it part of the release process which meant we had to do it via the command line. Fortunately for us, some google searching led us to a good starting point. Given below is the shell script that we use to create a .unitypackage from an existing Unity project.

#!/bin/bash
# Exports the Helpshift unitypackage from given Unity project
# arg-1 : path of the Unity project where Helpshift is integrated
# arg-2 : path where file needs to be exported
# arg-3 : version number
if [ "$#" -ne 3 ] || ! [ -d "$1" ]; then
    echo "Usage: $0 <path_to_Unity_project> <output_path> <version_number>" >&2
exit 1
fi

BIN_PATH=/Applications/Unity/Unity.app/Contents/MacOS
BIN_FILE=Unity

PROJECT_PATH=$1
EXPORT_PATH=$2
EXPORT_FILE=plugin-name.unitypackage
DIR_1=Assets/<dir containing your plugin files>
DIR_2=Assets/Plugins/Android # for android plugin related files
"$BIN_PATH/$BIN_FILE" -batchmode -projectPath "$PROJECT_PATH" -logFile export.log -exportPackage "$DIR_1" "$DIR_2" "$EXPORT_PATH/$EXPORT_FILE" -quit

Remove conflicts with other Unity plugins

One of the things that was a constant pain for us was the conflicts with Unity plugins like Facebook which used the XCode Editor for Unity project to modify the XCode project generated by Unity. We internally used the more humble mod_pbxproj python file which did everything we wanted to do and did it well. But the XCodeEditor project unfortunately messed around with the XCode project after we were done adding our files to it. Specifically, it removed any PBXVariantGroups which were present in the project. This meant our Localization files went for a toss and app could not see any strings.

No strings found

To get around this issue, we had to make sure that our integration ran after any conflicting plugins and that we adopted this integration method right from the start. To achieve this, we had to add the HelpshiftPostProcess.cs file which could take a PostProcessBuild attribute which could be set to a high value. The higher the value, the later in the process it would be run. This file should be a Monobehaviour type of script which lies in the Editor folder of your plugins subfolder. For example, Assets/Helpshift/Editor/HelpshiftPostProcess.cs

using System;
using UnityEngine;
using UnityEditor;
using UnityEditor.Callbacks;

using System.Diagnostics;
using System.Collections;
#if UNITY_IOS || UNITY_ANDROID
using Helpshift;
#endif
public class HelpshiftPostProcess : MonoBehaviour
{
  // Set PostProcess priority to a high number to ensure that the this is started last.
  [PostProcessBuild(900)]
  public static void OnPostprocessBuild(BuildTarget target, string pathToBuildProject)
  {
    HelpshiftConfig.Instance.SaveConfig();
    const string helpshift_plugin_path = "Assets/Helpshift";
    // Current path while executing the script is
    // the project root folder.
    Process myCustomProcess = new Process();
    myCustomProcess.StartInfo.FileName = "python";
    myCustomProcess.StartInfo.Arguments = string.Format(helpshift_plugin_path + "/Editor/HSPostprocessBuildPlayer " + pathToBuildProject + " " + target);
    myCustomProcess.StartInfo.UseShellExecute = false;
    myCustomProcess.StartInfo.RedirectStandardOutput = false;
    myCustomProcess.Start();
    myCustomProcess.WaitForExit();
  }
}

Make the Unity Android plugin integration easier

The current Unity android plugin involves manually adding entries to an existing AndroidManifest.xml file, combining our res folder with the existing res folders. It was all a pain and there was minimal isolation of the Helpshift files. Recently, Unity added support for integrating Android library projects present in the Assets/Plugins/Android folder. We decided to take advantage of this fact and distribute our plugin the same way. Now our Android integration just involves the Helpshift library project to the Assets/Plugins/Android folder and letting Unity take care of the rest.

Add Push notifications support

One of the difficult to achieve features for us was Push notification integration with Helpshift. There seemed to be too many different push plugins which each did things their own way. This led to many problems for our users to effectively use the excellent Push support which our backend systems provide. To get around this problem, we basically had to develop our own Unity push plugins which were built in to the SDKs. This was comparitively easy to do on the Android platform, but for iOS we had to resort to the black art of Method swizzling. But we had to make sure that there was a way for the app developers to switch this support off in the off-chance that the App store rejects the app. (We have contacted Apple support and made sure that the way we use swizzling is not something that is cause for rejection.)

To use the in-built push support, developers can simply add the below API:

HelpshiftSdk.registerForPush(<gcm-key-for-android/ignored-for-ios>)

Make the SDK configuration easy for developers

This was actually one of the interesting snippets of feedback that we got from our awesome customers. To quote - “If you could add like a GUI inspector to configure the Helpshift SDK, that would be really hot!” We figured, let’s give that a try. We had to go through some pretty unclear documentation about how to create a CustomEditor in Unity. Basically it involves creating an Object which derives from the Editor class and overriding the OnInspectorGUI method inside it. Then all we had to do was create the UI using the widgets in the EditorGUILayout class. To store the configuration values, we wrote a Scriptableobject which would hold all the values returned from the GUI. Saving the values could be done when the build was about to be created. We used the AssetDatabase class and it’s methods to achieve this result.

AssetDatabase.CreateAsset(instance, fullPath);

Just a footnote, make sure all your instance members are Serializable by adding the [SerializeField] annotation. If you want to add some MenuItems to the Unity editor, you can do so by adding code snippets like below to the Editor class.

[MenuItem("Helpshift/Edit Config")]
public static void Edit()
{
	Selection.activeObject = Instance;
}

[MenuItem("Helpshift/Developers Page")]
public static void OpenAppPage()
{
	string url = "https://developers.helpshift.com/unity/";
	Application.OpenURL(url);
}

The next step for us was actually passing along this configuration information to the XCode and Android projects. To do that we resorted to using JSON. The class which hold the config values would spit out the json representation of the configuration. These JSON files would then be added to the XCode project as simple data files and to the Android project as raw resources placed in the res/raw folder. Some wiring code to read them from the native SDKs and we were ready to roll.


The result

Helpshift Config Editor

Conclusion

Always listen to your customers

In conclusion, I would really love to thank all of our existing customers who gave us awesome constructive feedback and spurred us on to better our own standards. Hopefully all of you will love the changes that we have made, and also come up with even better suggestions to help us keep improving ourselves.

Exciting times ahead.

Rhishikesh Joshi's photograph
Rhishikesh Joshi
Mobile SDK team
October, 24, 2014

Background

Fifth Elephant (organised by hasgeek) is probably one of the best conferences focussed on Data in India (I am biased though as I am on the Editorial Panel that curates the talks :) I gave a talk on “How to build a Data Stack” at this years edition. It was well-received as many people came up to me after my talk and asked quite a few interesting questions about their stack. I build a Data stack at Akamai and then Inmobi. Now I am doing the same at Helpshift. Each time, the same questions were asked but the answers were different due to context and the maturity of the data ecosystem. I have refined these learnings in the talk. Hope you enjoy it. Send me any questions you have, as I am always curious of how people build their stack and the learnings from them.

Talk

Slides

If you’re interested in working in a fast paced team, we are hiring, join us!

Vinayak Hegde

Vinayak Hegde's photograph
Vinayak Hegde
Backend Engineering

Helpshift is an in-app mobile help desk designed to improve customer support efficiency by over 400% and reduce cost by over 70%. Our engineering team is committed to building a meticulously designed, solid SDK to improve customer retention for clients such as Flipboard, Supercell, and more . We are currently serving over 500 million app sessions weekly.

Learn more about us at Helpshift.com