Category Archives: Programming

Building a bi-directional interop bridge with WebView in Xamarin.Forms!

Let’s build an advanced communication bridge into your Xamarin.Forms WebView, talk to it, and let it talk back to us at will!? 😉 lol Yes let me show you how to pass data from Javascript environment to C# .NET run time in the Web page rendered inside your Xamarin.Forms WebView!

I’m talking about building a bi-directional communication tunnel with HTML/Javascript inside your WebView in Xamarin.Forms yo! 😀 buckle up your seatbelts!

So in my previous article, Talking to your WebView in Xamarin.Forms! I talked about, how to build a uni-directional C# .NET to Javascript environment in Xamarin.Forms WebView.

WebView in Xamarin.Forms..

In this article I’m going to take another step forward and allow the same functionality to occur the other way around as well… We’re talking about a two-way invoking between .NET run time and javascript run time in a Xamarin.Forms WebView!

Unfortunately this cannot be done by default in WebView.

Behold, Hybrid WebView!

This right here is a bit more advanced extension of the WebView with a bit of Xamarin Native magic! 😉 In order to establish an invoke bridge directly from HTML Javascript sandbox that its running inside the WebView, out to the .NET runtime, we need something more natively handled in Xamarin!

Basically we’re going to implement a device native script handler for the WebView which is going to handle the bridging between the Javascript and the .NET runtime handshake, in return giving us the opportunity to invoke calls from javascript into the .NET run time the Xamarin.Forms is execution on! 😉

Well that’s a very simplistic explanation, but there’s a whole article about it on Microsoft Xamarin Docs, Customizing a WebView if you’re interested! Since its already there, I wouldn’t be going into complete details of it, rather I would be explaining the improved implementation I have done on top of it for the Hybrid WebView.

Over there it focuses on loading Embedded HTML content, but I will extend my implementation to support for dynamic HTML content, allowing you to handle javascript loaded from a Web Source and support even run time generated javascript.

Invoking C# from Javascript in the WebView!

In order to do this, in par with Xamarin.Forms WebView, we need to implement a Custom Renderer for WebView, which we will refer to as HybridWebView.

HybridWebViewRenderer will be created across all the native platforms we intend to use our HybridWebView, in Android, iOS and Windows native environments each equipped with its own javascript handler to build a bridge on to our .NET run-time. 😉

We access the native WebViewRenderer properties and basically implement a special handler to listen to a certain pre-defined Javascript method execution. In this method which we add into the javascript that is rendered inside the WebView, we will define the parameters we need to use, in that way we can pass any number of parameters and data types as we want.

We’re going to intercept the execution of this javascript method inside our Hybrid WebViewRender, and then redirect it on to the .NET method we’ve subscribed to. So in the Hybrid WebView definition we will have an Action method that we bind to in our Xamarin.Forms level which we will subscribe to wherever we’re using this magical bits! 😉

Let the coding begin!

Let’s begin with HybridWebView Control in Xamarin.Forms! Here we;re adding an Action that we will subscribe to in order to receive data from Javascript invokes inside the WebView rendered content.

HybridWebView

namespace XFWebViewInteropDemo.Controls
{
    public class HybridWebView : WebView
    {
        private Action<string> _action;

        public void RegisterAction(Action<string> callback)
        {
            _action = callback;
        }

        public void Cleanup()
        {
            _action = null;
        }

        public void InvokeAction(string data)
        {
            if (_action == null || data == null)
            {
                return;
            }
            _action.Invoke(data);
        }
    }
}

 

InvokeAction is the method that will be used by the Native Renderer object to direct the invokes from javascript executions. Using the RegisterAction we can dynamically register the Action we need to subscribe to.  You can add any number of parameters as you wish in here, but you need to make sure to handle them in the native renderer as well.

Native Renderers…

We’re going to build native renderers for each platform we’re targeting, Android, iOS, and UWP (Windows). Basically all the renderers follow the same basic concept as we discussed before, but each of their implementation is going to be different based on the platform.

We need to make sure to handle the subscribe and unsubscribe of the Element Native properties and events properly in the renderer’s OnElementChanged() event.

We’re going to inject the javascript method that we’re going to listen to in the renderers as following.

private const string JavaScriptFunction = "function invokeCSharpAction(data){....}";

 

We will be defining this in each renderer, according to the native platform, and you can define the number of parameters you want to pass from this method as you wish. Right now we’re using a single parameter as you can see.  Every time a invokeCSharpAction() javascript method executes inside the WebView, it will get fetched by the Renderer and the following method call will occur.

((HybridWebView)Element).InvokeAction(value);

 

Up to the HybridWebView’s Action subscription on Xamarin.Froms run time, allowing our Action to fire up and retrieve the data coming in from javascript.

Alright now let’s get into details of each native renderer.

Android Renderer!

We’re going to use the Android’s WebViewRenderer to subclass our HyrbidWebViewRenderer.

github: /XFWebViewInteropDemo.Android/Renderers/HybridWebViewRenderer.cs

Like we discussed before for Android, we have the following script injection defined,

private const string JavascriptFunction = "function invokeCSharpAction(data){jsBridge.invokeAction(data);}";

 

For Android we need some extra bits of implementation, by creating a JavascriptWebViewClient that will set up listening to the execution of javascripts inside the WebView.

Then we have to create a JsBridge, which handles the interfacing with Javascripts, and fires up InvokeAction() method to redirect the execution flow up to the Xamarin.Forms level handlers.

Both those custom objects need to be set up in the HybridWebView in the renderer Element upon instantiation.

Control.SetWebViewClient
(new JavascriptWebViewClient($"javascript: {JavascriptFunction}"));
Control.AddJavascriptInterface
(new JsBridge(this), "jsBridge");

 

That’s pretty much it for the Android Renderer. Next on to iOS…

iOS Renderer!

For iOS we are going to use WkWebViewRenderer as the base renderer for our HybridWebView and in addition we have to implement IWKScriptMessageHandlder interface to handle the custom javascript execution monitoring that we target to handle.

github: /XFWebViewInteropDemo.iOS/Renderers/HybridWebViewRenderer.cs

We set up a WKWebViewConfiguration object in the constructor and we get access to the property WKWebViewConfiguration.UserContentController which allows us to set up our native bridge to Javascript execution firing up inside the WebView.

public HybridWebViewRenderer(WKWebViewConfiguration config) : base(config)
{
    _userController = config.UserContentController;
    var script = new WKUserScript(new NSString(JavaScriptFunction),
                   WKUserScriptInjectionTime.AtDocumentEnd, false);
    _userController.AddUserScript(script);
    _userController.AddScriptMessageHandler(this, "invokeAction");
}

 

Then for iOS, we have the following script injection defined using webkit API, accessing the invokeAction script that we attached and finally calling on the postMessage() method with the data parameters.

private const string JavaScriptFunction = "function invokeCSharpAction(data){window.webkit.messageHandlers.invokeAction.postMessage(data);}";

 

IWKScriptMessageHandler provides us with DidReceiveScriptMessage() method which we use to transfer the data up to the Xamarin.Forms level handler using, HybridWebView.InvokeAction(data).

Quite simple ans straight forward eh! next to Windows, or UWP as you might prefer.. 😉

UWP Renderer!

We use the Xamarin native WebViewRenderer for UWP or Windows platform.

github: /XFWebViewInteropDemo.UWP/Renderers/HybridWebViewRenderer.cs

The native default renderer grants us access to these two events NavigationCompleted and ScriptNotify. We need to make sure to subscribe to those events in our HybridWebViewRenderer in Windows as follows.

Control.NavigationCompleted += OnWebViewNavigationCompleted;
Control.ScriptNotify += OnWebViewScriptNotify;

 

NavigationCompleted, allows is to easily inject our javascript handler function, which is defined as follows for UWP or Windows,

private const string JavaScriptFunction = "function invokeCSharpAction(data){window.external.notify(data);}";

 

And then ScriptNotify, provides us the chance to redirect back the execution to Xamarin.Forms level handler using HybridWebView.InvokeAction(data).

Bingo, that completes the UWP or Windows Renderer!

Now that we’ve finished the setting up of our HybridWebView and its Native Renderer for Android, iOS and Windows, its time to consume it and taste it out! 😉

Let’s try it out!

Here’s we shall begin by consuming it in a XAML page in Xamarin.Forms!

<controls:HybridWebView
	x:Name="webViewElement"
	HorizontalOptions="FillAndExpand"
	VerticalOptions="FillAndExpand" />

github: /XFWebViewInteropDemo/HybridWebViewDemoPage.xaml

And then don’t forget to Subscribe to retrieve the data coming in from javascript inside our WebView using RegisterAction() method we created!

...
    // Subscribe for the data coming in from Javascript
    webViewElement.RegisterAction(DisplayDataFromJavascript);
}

private void DisplayDataFromJavascript(string data)
{
    Device.InvokeOnMainThreadAsync(() =>
    {
        ...
        // Do whatever you want with the data
        ...
    });
}
...

github: /XFWebViewInteropDemo/HybridWebViewDemoPage.xaml.cs

I’m just going to use the Main UI Thread’s help to execute any UI related stuff. And here’s a little demo HTML that I’m setting up in our Hyrbid WebView.

webViewElement.Source = new HtmlWebViewSource()
{
    Html =
        $@"<html>" +
        "<head>" +
            ...
            "<script type=\"text/javascript\">" +
                "function invokexamarinforms(){" +
                "    try{" +
                "        var inputvalue = 
document.getElementById(\"textInputElement\").value;" +
                "        invokeCSharpAction(inputvalue + '. This is from Javascript in the WebView!');" +
                "    }" +
                "    catch(err){" +
                "        alert(err);" +
                "    }" +
                "}" +
            "</script>" +
            ...
        "</head>" +

        "<body>" +
            "<div>" +
                "<input type=\"text\" id=\"textInputElement\" placeholder=\"type something here...\">" +
                "<button type=\"button\" onclick=\"invokexamarinforms()\">Send to Xamarin.Forms</button>" +
            "</div>" +
        "</body>" +

        "</html>"
};

github: /XFWebViewInteropDemo/HybridWebViewDemoPage.xaml.cs

As you can see I have a javascript function, invokexamarinforms() that will get invoked from a button call in the body. Once this method executes, it calls on the invokeCSharpAction() method that we defined in our Hybrid WebViews Native renderers.

In my javascript snippet I’m surrounding this call with a try catch in order to make sure the Native Renderer is properly implemented or not. Making sure this method is properly executes is a crucial step during debug if you run into any issues.

So let’s try out that sample code bits in action!

Time for some action! 😉

Hit that F5 yo! (well.. if you’re in Visual Studio! lol)

Side by side iOS, Android and UWP working like charm! 😉

As you can see in my simple Xamarin.Forms demo, I am demonstrating a simple C# .NET to Javascript call with data and Javascript to C# .NET call with data, a true bi-directional communication bridge!

Here we are typing some text in the Xamarin.Forms Entry element and sending it into the HTML inside the WebView. And then typing some text in the HTML Text Input element inside the WebView and click on HTML Button, and sending it to the Xamarin.Forms Label to be displayed, works like a charm!

I have shared the demo app code in my github as usual: github.com/XFWebViewInteropDemo

Extra tips!

Yep it’s that time, for some extra tips based on my experience with Xamarin.Forms Hybrid WebView! Although the extra tips that I already discussed in my previous article Talking to your WebView in Xamarin.Forms! still applies for this as well since we’re still extending from default Xamarin.Forms WebView, but apart from that…

Web Source, Embedded, Code HTML!? all same!

Doesn’t matter whatever the source of the HTML you’re setting in the Hybrid WebView, be it a web source directly from a URL, or loading an embedded HTML File, or even a code generated dynamic HTML content, it doesn’t make a difference.

The only thing that matters is the invokeCSharpAction() in your rendered HTML, so that the native renderers can pick it up and forward the execution to Xamarin.Forms .NET handlers!

Yes! extra parameters!

Even though I’m showcasing only a single parameter during this demo article, from javascript to C# .NET run time, you can easily extend this same implementation to pass any number of parameters as you wish! As I explained in the article make sure to define it in the following bits,

HybridWebView.InvokeAction(string data1, string data2)
private const string JavaScriptFunction = "function invokeCSharpAction(data1, data2){....}";
((HybridWebView)Element).InvokeAction(data1, data2);

And wire it all up, you’re good to go! 😉 I might share another article with some cool implementation with this in near future! 😀

Conclusion

You can easily build a communication bridge from C# .NET to javascript environment in Xamarin.Forms WebView! but the other way is not really possible out of the box!

That’s why we’re implementing this Hybrid WebView Control which allows us build a communication bridge from javascript to C# .NET environment directly during run time! 😉

So this concludes my bi-directional communication tunnel with HTML/Javascript inside your WebView in Xamarin.Forms yo!

Well that’s pretty much it!

Share the love! Cheers! 😀

Talking to your WebView in Xamarin.Forms!

Let’s build a communication bridge into your Xamarin.Forms WebView, and talk to it!? 😉 lol Yes let me show you how to pass data into the Web page rendered inside your Xamarin.Forms WebView!

I’m talking about building a uni-directional communication with Javascript inside your WebView in Xamarin.Forms yo! 😀 get your game face on!

WebView in Xamarin.Forms..

Xamarin.Forms provides a neat WebView that could render any Web HTML content efficiently similar to a browser inside your own Xamarin.Forms App.

Earlier there used to be a lots of issues that needed to be dealt with when to comes to rendering HTML content alongside Javascript inside the WebView, but with the recent update it has gotten far better with lots of features and facilities straight out of the box to be used! 😀

Invoking Javascript in the WebView!

Using the WebView straight out of the box, we can execute Javascript methods rendered inside the HTML content. Now I know this used to require a lot hacks and tricks, along side dealing with lot of run time exceptions.

But in the most recent updates of Xamarin.Forms, the WebView has gotten rock solid, and now even provides a dedicated method, EvaluateJavaScriptAsync() to invoking Javascript methods straight out of the box.

WebView.EvaluateJavaScriptAsync(String)

 

So now you can execute Javascript methods along with data parameters from you C# code in Xamarin.Forms using the default WebView control. EvaluateJavaScriptAsync() is an async method that lets you execute javascript and even await the call to response from the invoke as well.

var result = await webView
        .EvaluateJavaScriptAsync
            ("javascriptmethod('Hello world!')");

 

All you need to do is call the javascript method you’re targeting to invoke with or without the parameters you prefer using EvaluateJavaScriptAsync() in an asynchronous manner allowing you to await for a result back from the javascript into the .NET environment itself! Yep its that simple to talk to the HTML content in your WebView now! 😀

Let’s give it a try and establish a uni-directional communication with our WebView! 😉

Let the coding begin!

Here I have prepared a small demo where I’m loading some HTML content, along with a nice little javascript bits, into my WebView using HtmlWebViewSource as follows…

webViewElement.Source = new HtmlWebViewSource()
{
    Html =
        $@"<html>" +
        "<head>" +
            ...
            "<script type=\"text/javascript\">" +
                "function updatetextonwebview(text) {" +
                "    document.getElementById
                     (\"textElement\").innerHTML = text;" +
                "}" +
            "</script>" +
            ...
        "</head>" +

        "<body>" +
        ...
        ...
        "</body>" +

        "</html>"
};

Full code on github: /XFWebViewInteropDemo/DefaultWebViewDemoPage.xaml.cs

So here in my HTML content, I have a simple Javascript method, updatetextonwebview(text) where it takes in a value and set it to an HTML text element in the body. Pretty simple and straight forward.

string result = await webViewElement
        .EvaluateJavaScriptAsync
           ($"updatetextonwebview('{textEntryElement.Text}')");

 

And then I take a Text value from an Entry Element and pass it into the updatetextonwebview() javascript method using EvaluateJavaScriptAsync() of WebView.

Alright, let’s try it out!

Hit F5!

Well if you’re on Visual Studio, just hit F5 and watch the magic!

Side by side iOS, Android and UWP with Xamarin.Forms right out of the box! 😉

As you can see in my simple Xamarin.Forms demo, I have an Entry element which I type some text into and then I click on the Button which then going to pass that text data into the WebView’s javascript method. Then inside the javascript, it takes in the data and set it to a text label in the HTML body.

I have shared the demo app code in my github as usual: github.com/XFWebViewInteropDemo

No hacks, no work arounds, no custom renders, just straight out of the box in Xamarin.Forms! Works like a charm! 😉

Extra tips!

Yep it’s that time, for some extra tips based on my experience with Xamarin.Forms WebView!

-Track invoking of Javascript

Using WebView.EvaluateJavaScriptRequested event you can track the javascript invoke calls injecting into the WebView from your .NET code, basically an monitoring mechanism of all your javascript invokes from C# which will allow you to validate them or add extra details as you prefer on demand.

-Track Navigation inside the WebView

WebView provides a whole list of events to track the navigation now, along side back and forward navigation, redirects, and even refresh events.

WebView.Navigating

This event Triggers upon the beginning of the Navigation, allowing you to cancel it on demand. This event provides a WebNavigatingEventArgs object which provides you all the details about the navigation that is about to occur, direction, url endpoint and so on.

This also provides WebNavigatingEventArgs.Cancel property which allows you to cancel that navigation on demand. So yeah a lot of cool bits you can do with it!

WebView.Navigated

This event Triggers after a Navigation completes, providing you with the same details similar to Navigating event. In addition it gives WebNavigatedEventArgs.Result property which tells you whether the navigation was success or failure.

WebView.GoBackRequested | GoForwardRequested | ReloadRequested

Now these are some simplified events thats provided by WebView, allowing you to directly hook into GoBack, GoForward and Reload events when they occur. Although they do not provide facility to cancel those events like how we get in Navigating event. Just a quick easy way to monitor those events as they occur.

-Think Creative!

Most developers miss this point, if you can send a simple text string, then you can pass anything into the WebView’s javascript. Well… in disguise of a string of course!

  • Get device native data
  • Location GPS data
  • Proximity data
  • Internet Connectivity data
  • Captured File/Image data

Those are few examples, yes even an Image captured from the device camera can easily be sent as a byte array converted into a base64 string!  Imagination is the limit yo! 😉

Conclusion

Xamarin.Forms WebView has come a long way since the early days, into a complete mature sandbox environment to render any HTML Web Content inside your Xamarin.Forms app. And it provides lots of features to communicate, pass data back and forth, and even monitor and control the navigation happens inside itself.

This article basically focuses on uni-directional execution from C# .NET code to Javascript environment, while you can still await for the results from the javascript.

But there is no direct execution from Javascript environment to C# .NET yeah? So in my next article I’ll share how to build a bi-directional execution from C# .NET code to Javascript environment with Xamarin.Forms WebView. 😉

Well that’s pretty much it!

Share the love! Cheers! 😀

Xamarin.Forms Native HttpClientHandler for HttpClient…

Let’s make sure our Xamarin.Forms apps are properly configured with Native HttpClientHandler for optimum performance and better network security..

If you’re using the HttpClient for making Web API calls outside your app, you probably using the HttpClientHandler to set up various kinds of configuration for the HttpClient instance.

Now this HttpClient and Native HttpClientHandler applies directly for both Xamarin Native and Xamarin.Forms apps…

Although in this article I’m focusing on Xamarin.Forms, the same configuration set up can be used for any Xamarin Native apps as well.  By default in Xamarin you can use either the Managed HttpClientHandler which is fully maintained by .NET mono run time or the Native HttpClientHandler that maps itself to the Native run time configuration.

Why Native HttpClientHandler?

Thanks to the awesomeness of the Xamarin and the powerful Xamarin.Forms ability to map itself efficiently to the Native device environment, provides you with this facility to use the device Native Network Layer’s configuration in your Apps as well.

  • Using the Native HttpClientHandler provides you with a lot of advantages in terms of Network Communication Layer, which maps itself completely to the native properties and behaviors.
  • It provides your App with the in built default System native Security such as Transport Layer Security, TLS 1.2 features. This is basically built in for both Android and iOS system devices by default, which then we can leverage automatically on to our Xamarin.Forms app as well during run time.
  • This gives the user a peace of mind, in terms of the security of the network communication in the app while also giving the user the free of choice to let the app inherit itself the system configured security settings.
  • Defaulting to the Native Network configuration we can make sure our app is fine tuned for Security and Performance on the device native level and you do not have to spend extra time managing those bits manually.
  • Another great advantage is not needing to manually handle device Proxy Settings, allowing your Xamarin.Forms app to communicate through the device’s default network tunnel.

Well that’s pretty much a good list of reasons to make sure to set up our Xamarin.Forms apps to use the Native HttpClientHandlers eh! 😉

So what are they?

So below are the Native HttpClientHandlers available in Xamarin run time for each Platform, which applies for Xamarin.Forms as well.

AndroidClientHandler -AndroidClientHandler is the new handler that delegates to native Java code and Android OS instead of implementing everything in managed code. This option has better performance and smaller executable size.

NSUrlSessionHandler -The NSURLSession-based handler is based on the native NSURLSession framework available in iOS 7 and newer. This options has better performance and smaller executable size, supports TLS 1.2 standard.

WinHttpHandler -WinHttpHandler is implemented as a thin wrapper on the WinHTTP interface of Windows and is only supported on Windows systems. Provides developers with more granular control over the application’s HTTP communication than the default HttpClientHandler class.

So here as you can see, using the device native HttpClientHandlers provides you with the best of performance and security for your app compared to opting to use the Managed HttpClientHandler where you have to manually handle those optimizations yourself.

Although I must make a note here, in Windows or UWP Xamarin apps the default set up is the .NET Managed HttpClientHandler because the underlying native environment is Windows itself. But opting to use WinHttpHandler provides arguably better advantage according to many documentation, and also it’s in the same .NET stack! 😉

What no to do?

So before we get into “the how?”, let’s first make sure the bits not to do in our app project!

– not use “Managed”

So when it comes to Xamarin.Forms, by default when you create your project in Visual Studio the Native project nodes Properties are set up to use the Native HttpClient Handlers already. You can see this in both Android and iOS Project settings,

  • Android project node -> Properties -> Android Options -> Click “Advanced”
  • iOS project node -> Properties -> iOS Build

Do not set it to option to “Managed” HttpClientHandler in either of those settings, which will opt you out of Native HttpClientHandler.

– not use “new HttpClientHanlder()” 

If the above Settings check success, the next thing to consider is not instantiating HttpClientHanlder on its own as below,

HttpClientHandler httpClientHandler = new HttpClientHandler();
...
// setting up httpClientHandler settings
...
...
HttpClient httpClient = new HttpClient(httpClientHandler);

 

This is something you should not do, which will override your Native project property set up regarding the HttpClientHandler, and opt your HttpClient to use Managed HttpClientHandler instead, resulting you losing all the native goodness!

Next let’s see what to do?

What to do?

Here are the things you need to make sure to do instead.

– not using HttpClientHandler!?

Consider not using HttpClientHandler at all with your HttpClient, then you’re basically good to go, as long as you have set it up in your App Project Native settings. Not a joke though! lol 😛

Just use plain HttpClient instance out of the box! but make sure to do the following as well.

– set Native HttpClientHandler!

Go to the following settings in each of your Xamarin.Forms Native project nodes,

  • Android project node -> Properties -> Android Options -> Click “Advanced”

  • iOS project node -> Properties -> iOS Build

Make sure to set the Native Android and NSUrlSessionHandler those settings, to opt to use AndroidClientHandler and iOS NSUrlSessionHandler for your HttpClientHandler by default.

Well UWP or Windows project nodes doesn’t have such settings as it by defaults use .NET Managed HttpClientHandler.

A little demo!

Now if its all good, you should be able to see the following behaviors in action,

So this is a little Xamarin.Forms demo that I prepared to demonstrate the behaviors of Native HttpClientHandlers on Android, iOS and Windows UWP.

Here I’m demonstrating the Network access (blue color access granted and red color access blocked in run time) for a list of scenarios,

Now you can see how each device Native environment handles those endpoint calls, basically only allowing access to trusted secure web endpoints in the native network tunnel to go through.

Well that was quite simple eh! but we all know the real life requirements wouldn’t be so simple, what if we need to use the HttpClientHandler in code?

Yes we need access to the Native HttpClientHandler in code!

So then let me walk you through handling an advance implementation of the native HttpClientHandler with more customization added in code! 😉 

How to? Advanced set up of Native HttpClientHandler!

Yes as you can see in the Project Settings, it doesn’t really give you much options to customize your Native HttpClientHandler settings, or even override some of its behaviors at all. In a real life scenarios you would definitely need some more access to your HttpClientHandler use in code.

Compared to the Managed .NET HttpClientHandler where you easily have access to all its properties and behaviors.

But it is crucial for us to stick to the Native HttpClientHandler, so the solution would be to implement an access to the Native HttpClientHandler in our Xamarin.Forms code.

– Under the hood!?

Thanks to the awesomeness of Xamarin we have full access to those Native HttpClientHandlers in code as well, so that we can use them as however as we like. Let’s take a look under the hood of these Native bits shall we,

Android:

iOS:

Windows:

Now you can see that all these Native Handlers are extending from either HttpClientHandler or the HttpMessageHandler,

Drilling down further into HttpClientHandler we can see that its extending itself from HttpMessageHandler.

– Using em in code!

Let’s start by using our AndroidClientHandler in code to be used with HttpClient instance.

var androidClientHandler = new AndroidClientHandler();
... 
// setting up native httpClientHandler settings
...
...
HttpClient httpClient = new HttpClient((HttpMessageHandler)androidClientHandler);

 

And for iOS with the NSUrlSessionHandler.

var iosClientHandler = new NSUrlSessionHandler();
... 
// setting up native httpClientHandler settings
...
...
HttpClient httpClient = new HttpClient((HttpMessageHandler)iosClientHandler);

 

Then for Windows or UWP, opt to our WinHttpHandler.

var uwpClientHandler = new WinHttpHandler();
... 
// setting up native httpClientHandler settings
...
...
HttpClient httpClient = new HttpClient((HttpMessageHandler)uwpClientHandler);

On Windows or UWP make sure to install nuget package: System.Net.Http.WinHttpHandler to use WinHttpHandler which is a far better native option than default HttpClientHandler.

As you can see we’re casting them to HttpMessageHandler as a common ground object, since they all inherit from that base.

Now that we’ve got access to them in code, we can access all their properties and behaviors, and even override to customize them as we wish to.

– build the bridge to Xamarin.Forms!

Since the above bits are not directly accessible from Xamarin.Forms, we need to build the bridge that will allow us to access the Native HttpClientHandler instance in Xamarin.Forms environment directly.

Since I already created a common ground instance across all the native environments with the casting to HttpMessageHandler, this is much easier. Now there are plenty of ways leverage the access to this object up towards Xamarin.Forms layer, but here I’m going to showcase rather a simple implementation.

code on github repo: XFNativeHttpClientHandler/Services/HttpClientService.cs

Here I have a simple Service implementation in Xamarin.Forms where it maintains a Singleton object of itself, which contains a HttpClient object and HttpClientHandler object.

Given the HttpClientHandler is provided, I am instantiating my HttpClient() on demand during the run time as you can see below.

private HttpClientService()
{
    HttpClient = HttpClientHandler != null ?
        new HttpClient((HttpMessageHandler)HttpClientHandler) 
      : new HttpClient();
}

public static HttpClientService Instance
{
    get
    {
        lock (Padlock)
        {
            return _instance ?? 
                  (_instance = new HttpClientService());
        }
    }
}

 

So the setting up of the HttpClientHandler property happens in the each Native level’s execution start up point.

On Android: MainActivity.cs

protected override void OnCreate(Bundle savedInstanceState)
{
    ...
    
    var androidClientHandler = new HttpClientHandler();
    Services.HttpClientService.HttpClientHandler =               
                                     androidClientHandler;
    
    ...
}

 

On iOS: AppDelegate.cs

public override bool FinishedLaunching(UIApplication app, NSDictionary options)
{
    ...
    
    var iosClientHandler = new NSUrlSessionHandler();
    Services.HttpClientService.HttpClientHandler = iosClientHandler;

    ...
}

 

On Windows (UWP): MainPage.xaml.cs

public MainPage()
{
    ...

    var uwpClientHandler = new WinHttpHandler();
    Services.HttpClientService.HttpClientHandler = uwpClientHandler;

    ...
}

 

That’s it for the set up, then let’s use it in Xamarin.Forms code:

var result = await HttpClientService.Instance.
		HttpClient.GetAsync("https://google.com/");

 

Now this should provide you with a Xamarin.Forms Solution allowing you to be able to access all the Properties and Behaviors of Native HttpClientHandlers!

Yay! Access in Code!

So the whole point of access these Native HttpClientHandlers in code was to be able to customize their settings and behaviors according to our requirements eh! 😀

Go ahead and access those properties and behaviors that you wish to use…

Here’s me demonstrate a simple scenario, how to override HTTPS Certificate Validation while using our Native HttpClientHandlers!

Full gist: https://gist.github.com/UdaraAlwis/0787f74796d22c294b91be81ff162347

Things Keep in mind!

So when you’re accessing the Native HttpClientHandlers in code there are some things you need to absolutely keep in your mind, to make sure the performance and security is not compromised.

– Custom bits, only Native level!

All the custom configuration that you need to do should be done in Native Xamarin level code, accordingly to native properties and behaviors.

– One time init() only!

You should instantiate your Native HttpClientHandler instances from Native level only once, and they shouldn’t be altered later for consistency during run time across your app.

– HttpMessageHandler, keep it as it is!

Keep the HttpMessageHandler instance that we up-cast from the Native HttpClientHandler instances as it is after instantiation, to make sure you’re not overriding any native properties and behaviors that we set up or inherited prior.

– Release Build, watch out!

When you’re using the Xamarin Platform specific Project Settings, make sure those settings are propagated to Release mode as well. In .csproj file has separate configurations for Debug and Release build configurations, so make sure keep an eye out for those configuration during Release builds as well.

Conclusion

Whenever you need to use the HttpClientHandler along side HttpClient, in your Xamarin.Forms or Native Xamarin Apps, its best to use the Native HttpClientHandler. This can be easily configured in each Native Project Settings or we can even instantiate them in code to be used across our Xamarin.Forms app environment as I’ve explained in this article.

I have shared the demo app code in my github as usual: github.com/XFNativeHttpClientHandler

Well that’s it! 😉

Share the love! Cheers!

Stunning app Themes in Xamarin.Forms Shell projects!

Wanna build some awesome themes into your Xamarin.Forms Shell project? Change the App themes during the run time dynamically? Save the theme properties in User Settings? Then you’re at the right place! 😉

Themes in Xamarin.Forms Apps…

Given Xamarin.Forms is such an incredible cross platform mobile framework, it provides a lot of awesome features out of the box, but unfortunately it doesn’t straight away provide App Theme feature in your face. 😛  but…

It’s got all the bells and whistles you need to implement an awesome App Theme set up with ease, in combination of awesome Dynamic Binding, Static Binding and Style Properties giving you full control of how to handle implementing Themed Styles into your app. You just gotta put together all those bits and you can easily build App Themes functionality into the app.

This is actually quite powerful given the cross platform nature of Xamarin.Forms! Now there are plenty of documentation, articles and sample demos regarding this, so I’m not going to be repeating on that topic, but if you’re interested try out this office doc article from Microsoft: Theming a Xamarin.Forms Application Now that gives a very good step by step explanation on the topic for anyone to easily get started, but anyhow focusing back on this article…

Enters, Xamarin.Forms Shell Apps…

Then enters the recently introduced Xamarin.Forms Shell, a new paradigm of Xamarin.Forms App development. Now when I got started with this new awesomeness, I couldn’t find any direct articles or samples on how to implement App Themes in a Xamarin.Forms Shell projects.

So I had to figure it out myself, and that’s why I thought of sharing my experience with you guys. Now now, don’t get me wrong, since we’re still in the same context of Xamarin.Forms, the App Theming strategy is still the same as the official doc I shared about, but I may have found a better way and a bit of improvements on top of that. 😉

So let’s get started… Stunning app Themes in Xamarin.Forms Shell projects!

How to? in a nutshell!

Basically in a nutshell we’re going to define a set of Themes with Color Properties inside them that holds Color values unique to each Theme. Then we will build all a complete collection of Styles targeting all the components in our app such as Button, Labels and so on. Now in this case we need to stick to Dynamic Binding since we’re going to be switching between Themes during run time, so use of Xamarin.Forms DynamicResource binding all the way ah! 😉

Next we make sure all our UI elements are bounded to those Styles that we created before and still making sure to stick to Dynamic Binding. Now that we have established the binding chain from Theme Properties to Styles to UI Elements, we need to set up the default Theme to load in the ApplicationResources XAML node.

Once all that’s set up we shall be implementing a simple mechanism  to save the User’s App Theme preferences, with features such as Saving and Reloading those preferences on demand.

And that’s it! 😀

Sneak Peak!

Here’s a little sneaky peaky demo magic on Android and iOS.. 😉

If you’d like a sneak peak into the code before I get into the code details, you can find the whole demo project here in my github repo: https://github.com/UdaraAlwis/XFShellAdvThemeing

So in my demo code, I have used the default out of the box Xamarin.Forms Shell project template you get in Visual Studio 2019 latest update, so that you can easily familiarize yourself with the code and adopt the same implementation in your own code.

Set up of the project!

Let’s get started with the set up of the Project, but first should take a look into the structure we’re going to be implementing, where we need to first focus on the following aspects.

  • List of Themes for the App
  • List of properties in a given Theme
  • Styles built using the properties of Themes
  • Helper extension to change App Theme
  • Saving the User’s App Theme preferences

Now keeping those main aspects in mine, I’m going to assume that you have already set up your basic Xamarin.Forms Shell app project. I’m going to name my little demo as XFShellAdvThemeing, denoting Xamarin.Forms Shell Advanced Theme-ing! 😉

Alright there goes our Xamarin.Forms Shell App project, with strong base of MVVM baked in! You may have noticed I have added “Themes” folder, which will hold the App Themes that we’re going to create. And then a “Helpers” which will hold a simple extension method that we’re going to built for switching the App Theme selection by the user during run time.

Building the App Themes…

Time to define the App Themes, so first of all we need to be specific about what properties we are going to be using to each Theme, be it colors, fonts, images and so on, you need to be specific, and make sure all the Theme definitions follow the exact same format.

on github: /XFShellAdvThemeing/Models/Theme.cs

Here I have defined the list of Themes that I’m going to include in my App, and as for having the type of Enum, is for to be used later in the code to populate the data to the User. So make sure to add a record here every time you add a new Theme to your app.

This step is very crucial because you shouldn’t neglect this and try to change the app theme properties half way into the app development. If you’re working with a team,

…you need to define with the team designers together how the Themes of the app should behave, and what elements should be controlled over them.

For this demo I’m going to define the following list of properties inside each Theme.

  • Main Colors:
    • Primary Color
    • Accent Color
    • Secondary Color
  • Page Colors:
    • Page Background Color
    • Navigation Bar Color
  • Text Colors
    • Primary Text Color
    • Secondary Text Color
    • Tertiary Text Color

As you can see I have taken into consideration of having 3 set of main Colors, and as a back up since I’m going to be using native Navigation bar, a property to control its colors and the Page background as they render next to each other. Then finally the Colors for the Text inside the app.

Its very crucial you develop this kind of design centric thinking when you’re developing any mobile app…

So let’s create our beautiful little XAML snippets inside the Themes folder that are going to hold all the properties of each App Theme.

Just simply add a XAML page into the Themes folder and rename the parent node to ResourceDictionary of type and update the same in the code behind as well.

on github: /XFShellAdvThemeing/Themes/DarkTheme.xaml

There my first theme, LightTheme which holds the color values I need to customize in my app for Theme-ing! 😉 Feel free to add as many themes as you like following the same structure.

Defining the Styles…

Now these bits are the middle man between your UI elements and the Theme definitions. As I explained before we are going to create Styles targeting all the types of UI Elements that we’re using in the app, so that we can “Style” them with those! 😉 get it!? lol

on github: /XFShellAdvThemeing/App.xaml

You can define these Styles as a Global App resources or in Page levels as you wish, but I have added them into my App.xaml for this demo, such as Styles targeting Buttons, Labels, etc.

As you can see we are using DynamicResource binding to hold on to the Theme property values, so that we can update our Style property values dynamically in run time.

Now you might wonder why I have referenced Themes/LightTheme.xaml in the  global Resource Dictionary, well… that is to set the default theme as the Light App Theme that I just defined above. You can keep it as it is or watch me switch the App Theme dynamically during run time below… 😉

You also need to directly reference your Shell bound UI Elements using the Theme properties we defined earlier. Here I have added to to the same App.xaml global scope instead of keeping it in the AppShell.xaml scope, just for organizing the styles in one place.

Switching App Theme Dynamically…

Now of course we need to allow our App to be able to switch the Theme dynamically during run time according to User’s choice or some configuration built in. We can easily do this in Xamarin.Forms, using MergedDictionaries property by removing the existing Theme in memory Resources and switching to our choice of Theme Resources.

I have created a simple Helper extension, with the method SetAppTheme() which accepts the type of Them Enum value you need to use and returns the boolean result.

on github: /XFShellAdvThemeing/Helpers/ThemeHelper.cs

Like I said before this is where out little Helper extension comes into play, so just add this little snippet into your Helpers folder.

Based on the Theme Enum identifier value, we will be instantiating the Theme object, assigning it to the Resources in memory as you see above.

Now that’s all cool and stuff, but how about persisting this preferred theme selection?

Saving to User Preferences!

This can easily be done with the help of Xamarin.Essentials, which allows us to save Application context key values pairs using the Preferences API. Now I believe in Visual Studio when you create a new Xamarin.Forms Shell project by default it comes pre-installed with Xamarin.Essentials, otherwise make sure to add it to your project from nuget.

We are going to save the Selected App Theme settings with the key name “CurrentAppTheme” as below.

on github: /XFShellAdvThemeing/Views/ThemeSelectionPage.xaml.cs

And make sure to load it back to the app during the App’s launch event and call up on our little magic extension ThemeHelper.SetAppTheme() as shown here..

on github: /XFShellAdvThemeing/App.xaml.cs

You need to call that in the App() constructor invoke, so that we can load the saved App Theme settings instead of loading the default one that we set up in App.xaml resources.

 

Bingo! nice little App Theme Selection Page to our Xamarin.Forms Shell App! 🙂

Time for some action!

Here it is side by side iOS and Android,

 

Themes with more than Colors?

Now our App Themes aren’t always going to be as simple as a bunch of Color properties right?! It could even contain Fonts, Text Sizes, Images, Icons and so on. But if you’re wondering if that’s even possible in Xamarin.Forms, yes absolutely you can!

It’s basically no different than defining a Color property in your Theme.xaml file, just add the XAML node to the file and you’re good to go!

Make sure to give it a Key name value though, and reference it in your Styles as usual where applicable.

You can follow the same pattern for any kind of Theme property you want to add and basically you’re good to go! 😉

Github Repo: github.com/UdaraAlwis/XFShellAdvThemeing

Any property that you can reference usually from your XAML, you can easily include them in your App Theme and link the binding through Styles to your UI elements straight away.

Some Tips!

Here are some tips and tricks that might come in handy for you, during the whole shabang of “Theme-ing” your Xamarin.Forms Shell app projects.

More Unification! Less Repetition!

Make sure to avoid adding repetitive theme properties into the Theme XAML definitions, by unifying the Colors, Fonts, Icons you use in your app. As an example if you define a Text Color property in your Theme, make sure to use that only for Text Coloring Elements and Styles, try not to use them for other aspects.

So you can easily manage those properties in future and they’d be easy to understand for anyone to extend the properties. This is quite crucial when you maintain massive App Projects, and it wouldn’t affect the capability to grow the App code altogether.

Stubborn Native Elements?! Yikes!

Now as you probably know or don’t Xamarin.Forms doesn’t let you change the color values of your App’s few very native elements during run time. Such as,

– iOS/Android System Status Bar Colors

– Android UI Elements that contains horizontal bar such as Entry, Picker, etc. (Unless you’re using Xamarin.Forms Material Visual)

They require native Android/iOS level access to change during run time. So you need to have Custom Renderers or Native Bound Services that can be communicated through Xamarin.Forms layer during run time. Well that’s a blog post for another time! 😉

So here’s how it could be easily solved as you can see below,

My suggestion would be to maintain values of those UI elements as compatible as possible that could be matched with the Theme Colors you currently use. Such as light Gray, White or Black mostly.

You can easily set them up from Resources/values/styles.xml in your Android project.

And on iOS project’s Info.plist configuration.

And that’s pretty much it!

Share the love! 😀 Cheers!

I built a Google Forms Toolkit Library for .NET!

Watch me building the soon-to-be-famous (lol) my GoogleFormsToolkitLibrary for .NET C#, a nuget library that will help you easily access your Google Forms programmatically in dotnet, letting you load Field Question data, Submit Form data and so on! 😉

In my previous blog article, Programmatically access your complete Google Forms skeleton! where I shared about my adventure into working out a little hack to access your Google Forms page content programatically, I realized it was a solid implementation, and could be used with almost all kinds of openly accessible Google Forms.

So following up on that, this time I will be building a complete .NET library that will consist of all the awesome hacks and tricks that I built around playing with Google Forms myself, which I have also continuously written blog articles as well!

Backstory…

So far we have come across…

And then in my previous blog article, Programmatically access your complete Google Forms skeleton! I shared how we retrieving the following data on a given Google Form extensively,

  • Google Form Title, Description, Form ID
  • List of Question Fields

And in each Question Field,

  • Question Field Text
  • Question Type
  • If submitting Answer is mandatory or not
  • Available answer list (Multiple answer selection)
  • Question Field Identifier (Field Id) or Answer Submission ID

Then we analyzed the data structure of the FB_PUBLIC_LOAD_DATA_ script that we scraped out of the rendered HTML content, and even managed to parse it into a meaningful data structure which could be mapped to the data we are looking for.

So using that knowledge and expertise, let’s step up the game and build a solid library tool, which we can reuse easily in anywhere to access our Google Forms programmatically to perform all kinds of awesomeness as we wish! 😉

However if you haven’t gone through my previous blog articles in this Google Forms Hacks series, please do so before continuing to avoid any confusion, since I might not be diving into all the details.

Behold GoogleFormsToolkitLibrary v1!

So this library that I’m building will be on .NET Standards, which will allow you to use it in almost any kind of a .NET project. From Console apps, Desktop, Web and even to Mobile Apps in Xamarin!

It will provide you with the following awesome features!

  • Load information on your Google Form
  • Load Question Field data on your Google Form
  • Submit Form data to your Google Form

At least for now, I will be adding more features as I go along my journey of hacking around Google Forms! 😉

I will be publishing this library to Nuget, so that you can easily grab it into your .NET projects, also let’s add some fancy Test Driven goodness to it, using xUnit Tests! 😀

Project Solution Set up…

So I created a .NET Stanard version 2.0 library in Visual Studio, naming it GoogleFormsToolkitLibrary of course! 😉 The I added a Test folder into the Solution which will hold the xUnit Unit Test project, which I named .Tests!

Then don’t forget to add the HTMLAgilityPack and Newtonsoft.Json nugets to the Library solution which will allow us to scrape out the HTML content in a Google Form and use the Json content parsing logic!

Now the basic set up is done, let’s move on to the code!

Let the coding begin…

We need to model the data objects that we’re going to handle when it comes to the perspective of a Google Form. Basically following are the main entities of a Google Form as far as we have recognized in our previous blog articles!

  • A Google Form
  • A Google Form Question Field
  • The Google Form Field Type

Then as of the functionality I would be adding two public methods that we can call upon to execute.

  • Load Google Form Structure, data, and form fields
  • Submit form data to a given Google Form

Crafting the Models…

So we let’s start by building Model classes that will represent the entities we need.

GoogleFormsFieldTypeEnum

Let’s start with the simplest Model we could build that is the Google Form Field Type object, which will represent the type of a given Question Field. I basically described in detail about this entity and how I build the entity model myself through trial and error in my previous blog article:

Let’s create an enum model class giving it the name  GoogleFormsFieldTypeEnum.

We’re adding the Description tags to hold the human readable value of each type.

GoogleFormField

Now this fella right here is a big deal, that is the Google Form Question Field object, which will represent a single Question Field in a given Google Form. A whole list of these fields comprises in every Google Form.

Let’s create a model class giving it the name  GoogleFormField.

As you can see we have created properties inside this model that will hold all the values that we discussed previously, that could consist in a given Google Form Question Field.

GoogleForm

Now here’s the top entity, representing a whole Google Form page content. Let’s create a model class giving it the name  GoogleForm.

Pretty straight forward set of properties that would usually contain in any Google Form as you can see above. Now make sure to add those Model classes to the Models directory in your project.

Alrighto, we got the model classes done, then let’s move on to the actual functional coding!

Implementing the core features…

We’re going add two main functionality to this library, for now as following! 😉

LoadGoogleFormStructureAsync

Now this method will intake a string parameter that will carry your Google Forms link, and it will return an object of type GoogleForm that we define above. Which obviously contains the given Google Form’s generic information and Question Field list data including Question Type, Answer Options, Submission Id, etc. Now that’s quite a handful eh! 😉

Task<GoogleForm> 
LoadGoogleFormStructureAsync(string yourGoogleFormsUrl)

So this fella will make Http Async call to load the rendered HTML of the Google Form into memory, run through my magical data scraping algorithm, build the GoogleForm object and return it! Sounds pretty simple but don’t let the implementation confuse you, take a look here! 😉

https://gist.github.com/UdaraAlwis/dcb473f9f07d5024376f1289c1b08ace

Please feel free to take a look at the full implementation on Gist link.

This method is more of an extension of the ScrapeOffFormSkeletonFromGoogleFormsAsync() method that I shared with you all in my last blog post, which I gotten into in depth details about explaining each step of the code.

So to reduce repetition I’m not going to repeat the same here, rather I would share the important bits to explain that are improvements on top of the previous implementation.

One of the differences I would say is that now we’re injecting the data that we scraped out into our model objects, as you can see we’re initializing out GoogleForm and GoogleFormField data objects here, and then loading the GoogleForm data object values.

Then inside the loop we’re loading the GoogleFormField object values as well, and then each object will then be added to the QuestionFormFieldList property of the GoogleForm. That’s pretty much it! 😉

SubmitToGoogleFormAsync

This method will intake a string parameter that will carry your Google Forms link, and  a Dictionary object containing the Form field answer submission id and answer values mapping. As a result it will return a boolean value denoting the success or failure of the submission task.

Task<bool> SubmitToGoogleFormAsync
(string yourGoogleFormsUrl, Dictionary<string, string> formData)

It will make a Http Async call the Google Forms REST API endpoint with the given data and await for the response code, upon receiving the response it will return true or false based on it.

https://gist.github.com/UdaraAlwis/8d547f68ae3f629d40b2184f51acd43e

Please feel free to take a look at the full implementation on Gist link.

This methods is actually an improvement of one of my previous posts, You may RESTfully submit to your Google Forms… where I explain in depth how I hacked around to figure this out and what each line of code is meant to handle. Please look into that article if you’re keep for more details.

We’re basically looking for the 200 StatusCode value that determines successful submission to the Google Forms REST API endpoint.

Now our Core Library is ready to go!

Unit Test it yo!

Yep but its not done until we implement a proper set of test cases isn’t it! So let’s add some Unit Tests that will test the functionalities that we built into our Library.

Like I mentioned in the Project set up, we have added a xUnit Unit Test project into the solution, and make sure to add a reference of our Library project GoogleFormsToolkitLibrary into it.

Let’s start by adding a Tests.cs class to it, which we will use to implement the tests. For now, I will add two test cases,

RetrieveGoogleFormStructure_Success

This will test for LoadGoogleFormStructureAsync() feature and make sure we get a valid response and the expected data of the given Google Form page.

SubmitDataToGoogleForm_Success

Now this case will test for the SubmitToGoogleFormAsync() feature and make sure we get a successful boolean response, with the given Google Form link, and the form data dictionary object.

Oh let’s not forget to make sure they’re passing in the Test Runner! 😀 lol

All good to go it seems! for now at least lol 😛

This is really a great practice because if I ever break anything in the code during any new implementation or existing modifications, I would be able to notice it here in the tests.

Nugetting it!

Let’s not forget to publish the beauty to Nuget eh! Let’s make sure all the nuget properties are added to the package before uploading…

And you may find it on nuget right now!

Nuget: nuget.org/packages/GoogleFormsToolkitLibrary/

This whole project is open sourced and published to Github as well if you’re interested in looking at the code, or track the improvements I’m adding in the future!

Github: github.com/GoogleFormsToolkitLibrary

So that’s done!

Let’s try it out!

Now the beauty of this is that you can easily add this to any of your .NET projects as the whole library is based on .NET Standard.

Go ahead and add it to your .NET project from Nuget using either the Package Manager Console or the Nuget Package Manager.

Install-Package GoogleFormsToolkitLibrary

Use of GoogleFormsToolkitLibrary is pretty simple, just instantiate it and call upon the method you wanna use.

// Retrieve the structure of my sample Google Forms page
// https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

var googleFormLink =
"https://docs.google.com/forms/d/e/" +
"1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA" +
"/viewform";

var googleFormsToolkitLibrary = new GoogleFormsToolkitLibrary();
var result = await googleFormsToolkitLibrary.
                    LoadGoogleFormStructureAsync(googleFormLink);

 

That should work like a charm! Make sure to pass your Google Form’s link properly and make sure it is openly accessible, you should be able to see the magic pretty easily!

// Submit data to my sample Google Forms page
// https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

var googleFormLink =
	"https://docs.google.com/forms/d/e/" +
	"1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA" +
	"/formResponse";

Dictionary<string,string> formData = new Dictionary<string, string>
{
	// Question Field 1
	{"entry.1277095329", "Moon Rockets Launching"}, 

	// Question Field 2
	{"entry.995005981","Banana Plums"},

	// Question Field 3
	{"entry.1155533672","Monkeys with hoodies"},

	// Question Field 4
	{"entry.1579749043","Jumping Apples"},

	// Question Field 5
	{"entry.815399500_year","2019"},
	{"entry.815399500_month","11"},
	{"entry.815399500_day","11"},

	// Question Field 6
	{"entry.940653577_hour","04"},
	{"entry.940653577_minute","12"},
};

var googleFormsToolkitLibrary = new GoogleFormsToolkitLibrary();
var result = await googleFormsToolkitLibrary
                .SubmitToGoogleFormAsync(googleFormLink, formData);

 

The above should nicely respond with a successful true value! 😉

Something to keep in mind here is that you need to make sure you’re setting the correct Field Answer Submission ID properly, and when it comes to Multiple Answer selection fields, make sure the provided answers matches the available list of answers. Date Time fields should be carefully treated with their additional year,month,day suffixed fields and hour,minute respectively. Just go through my past few blog posts on Google Forms hacking, you’ll see for yourself!

What’s next?

Well building this and publishing it openly is just the first step, I’m going to continue building this further adding more features and performance improvements.

Then I will be building some demo Client apps which will use this library to implement some cool features… 😀

So keep an eye out!

There you have it, how I built a Google Forms Toolkit Library for .NET!

Share the love! 😀 Cheers!

Programmatically access your complete Google Forms skeleton!

So you wanna load the content of your Google Forms programmatically? access all the questions, answer options, field types, submit ids and so on as you wish? 😉 Stick around!

Hope yol remember in my last blog post, SCRIPTfully scrape off your Google Forms Field Ids… I was sharing a neat little script I built to extract the Field identifier IDs from your Google Forms page, so that you can use them with ease for auto filling question answer fields or submitting data using REST API into your Google Forms! So this is the nest step in this series of Google Forms Hacks!

In this post we’re gonna load your Google Forms skeleton programmatically!

What does it mean?

Yes, instead of loading your Google Forms page on the typical web browser, why not load the bare bone content of it as you wish? By this I mean,

  • List of Questions (your question content…)
  • The types of Questions (Short Answer, Paragraph, Checkbox, etc)
  • Available Answer options (Multiple choice answer questions…)
  • Title and Description of your Google Form, etc…
  • And many more details you have added to your Google Form…

Once you get access to those bare bone content or the skeleton structure of your Google Form, then you can do all kinds of stuff with it…

Then  you can render it as you wish and present to your users, re-render it into a Web App or a Desktop app or even a Mobile App with your own custom layouts, filtering, and validations! 😀

So how we gonna do it?

Simply put, we gonna extract the skeleton or the bare bone structure from our Google Forms page. Now as you may have figured out there’s no official API or SDK to access Google Forms services programmatically, therefore we obviously have to hack our way around this!

We’re going to build a little a script which will load the HTML content of our Google Form and perform a magical algorithm to extract our Google Forms structure! All the questions, answer options, validation, and etc the whole deal… 😉

Backstory…

In my last post hope you remember how I shared about scraping through the HTML content of our Google Form page to extract Field identifiers which is used to submit answers or otherwise known as Field Answer identifiers.

Then along that same time I was experimenting with trying to extract the whole structure of the design the same manner. But filtering through the HTML tags to retrieve the complex structure of a given Google Forms Question-Answer field structure seemed quite hectic.

Jackpot!

So while I was  going up and down the HTML of the page, trying to find a better way to extract our Google Forms Question-Answer structure, I came across this interesting piece of script at the end of the page.

As you can see at the bottom there’s java script starting with “FB_PUBLIC_LOAD_DATA_ = [nulll…” with a strange pattern of the content, which I’m not sure what the purpose of it in this page, but you can surely find this in any Google Forms page. This specific script snippet seem to be holding the whole skeleton of our Google Forms page as you can see, it containing the Question Content, Answer Fields and so on!

So there’s our little treasure… 😉

Now finding this bit wasn’t enough at all, I had to figure out how  to parse this properly to extract the data that we’re looking for, as you can see it’s got a little unorthodox structure in its content. Now that’s the next challenge!

Let the hacking begin…

Now for this post also let’s use the same sample questionnaire Google Form that I created for last post’s demo.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

So we’re going to load the HTML content of our Google Forms page, and then we’ll extract that mystery javascript code snippet, and parse the content of it to access the skeleton structure of our Google Forms!

Now first let me walk you through how to parse that mystery FB_PUBLIC_LOAD_DATA_  content! 😉 Let me warn you though, figure this out was no walk in the park but let me share the secret source with yol straight away.

Let me copy and paste it here from my sample Google Form and get started! Below you can clearly see how all the Questions along with Answers and Field identifiers in my sample Google Form are contained in this mystery code snippet.

var FB_PUBLIC_LOAD_DATA_ = [null,["Please fill up the following questions. You need to answer all the fields please! ;) ",[[122249536,"Hello there, this is question 1, could you answer?",null,0,[[1277095329,null,0]]],[1170747525,"Which one would you prefer as the answer from below?",null,2,[[995005981,[["Mango Peach",null,null,null,0],["Banana Plums",null,null,null,0],["Strawberry Pears",null,null,null,0]],1,null,null,null,null,null,0]]],[2147453523,"Well another question here wouldn't hurt now eh?",null,4,[[1155533672,[["Monkeys with hoodies",null,null,null,0],["Dogs with hats",null,null,null,0],["Cats with crowns",null,null,null,0]],1,null,null,null,null,null,0]]],[172187917,"How about this for a change?",null,3,[[1579749043,[["Running Banana",null,null,null,0],["Jumping Apples",null,null,null,0],["Rolling Pears",null,null,null,0]],1,null,null,null,null,null,0]]],[676251522,"What's the date would you like to be today?",null,9,[[815399500,null,1,null,null,null,null,[0,1]]]],[1280585510,"What time would it be right now? ",null,10,[[940653577,null,1,null,null,null,[0]]]]],null,null,null,[0,0],null,null,"Sample questionnaire!",48,[null,null,null,null,0],null,null,null,null,[2]],"/forms","Random Sample Questionnaire",null,null,null,"0",null,0,0,"","",0,"e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA",1];

In order to parse this into a known data structure you need to first remove the “var FB_PUBLIC_LOAD_DATA_ = ” and the “;” at the end if you notice carefully.

However by looking at the content you can make a guess that it should be some JSON based content structure.

So let’s use any online JSON parser tool and you should be able to see the formatted content as follows. ex: jsoneditoronline, codebeautify or jsonformatter

Now you get some readable structure yeah, where you are able to traverse through it’s content by expandable nodes array.

Pattern Recognition and Analysis!

Since we parsed the mystery content into a JSON Array tree, we can now traverse through the data easily and extract the specific data that we’re looking for.

Now since there’s no official documentation regarding the parsing of this data from structure, I guess we’re going to have to figure out ourselves, where to pick what data in this data tree and recognize the pattern.

So basically what we are going to be focusing on retrieving the following data,

  • Google Form Title, Description, Form ID
  • List of Question Fields

And in each question,

  • Question Field Text
  • Question Type
  • If submitting Answer is mandatory or not
  • Available answer list (Multiple answer selection)
  • Question Field Identifier (Field Id) or Answer Submission ID

Now that I believe comprises the complete structural skeleton of any given Google Form! 🙂

– General Google Form data

In the parent root you can see the Google Form Doc’s Name in the [3] index.

Then the Form Id in the index [14] of the array tree. And rest of the important bits seem to be in the [1] index node as you can see it has a lot of child nodes in it. Let’s look into that..

Now this index item seem to be containing a lot of information in child nodes. In its child node index [0] you can see the Description of our Google Form, and index [8] holds the Title.

Next the most important node, that is node index [1] which contains all the Question Fields data, and it looks like this.

Once you expand it you can view the List of child elements that represents the Question Fields in your Google Form!

Now you know you can easily access the Question fields by traversing [1][index of the question field]

Let’s try opening up a child node then! 😉

– Question Field data

Woot! Here you have it, the whole Question Field as expected, and specifically in this sample questionnaire, you can see how it shows all the details regarding the 1st Question Field.

Also notice the value I’m pointing to below, node [1][0][3] to be exact, that index holds the Question Type value “0”, which determines whether it’s a Short Answer, Paragraph, Dropdown, Checkbox field and so on. Then the second arrow, node [1][0][4][0][0], that right there is the unique Field identifier that we need to use when submitting answer to this question field. 😀 Thereby we could consider the value as Answer Submit Id as well.

Now that’s a single answer field, such as Short Answer and Paragraph Question field types in Google Form.

In the node [1][0][4][0][2] holds the value to determine whether Answer Required or not.

So how about we open up a child node of a Multiple Answer field?

Now below I’ve open up the node of a Multiple Answer selection Question Field.

– Multiple Answer Question Field data

Right here you can see in the Question Type value node it has number “2” as the value, which is what Multiple Answer Question Fields are denoted with.

Over here you get an extra child node underneath the Field identifier node as you recognized above.

This has a list of child nodes which as you can see holds the list of Answers Available for the Question Field.

So now we know how to load the list of Answers available for a given question, that we can traverse through.

Next let me dive a bit deep into Question Type identifier values..

– Hunt for Question Field types…

You already know Google Forms provide multiple types of Question Fields that can be added to your Google Form, and you saw above where to grab the Field type value. But how do you know which value maps to what type?

That’s why I had to run a trial and error recognition of trying to match those numeric values to the actual Field types, and I’ve finalized the list as follows…

  • Short Answer Field = 0
  • Paragraph Field = 1,
  • Multiple Choice Field = 2
  • Check Boxes Field = 4
  • Drop Down Field = 3
  • File Upload Field = 13

// File Upload – we’re not going to implement for this right now, because it needs user log in session implementation, a bit complicated. So let’s look into it later!

  • Linear Scale Field = 5
  • Grid Choice Field = 7

//Grid Choice – represents both: Multiple Choice Grid & Checkbox Grid

  • Date Field = 9
  • Time Field = 10

As you can see the assigning of the numeric values for the available types of Fields are not so straight forward and Google Forms tends to mix up or skips some mapping of the values without a proper order.  That’s why I mentioned it was a bit painful trial and error process.

Now we’ve walked through the recognition of the data pathways that we’re hoping to extract, let’s summarize our analysis as follows..

  • The most crucial node is index [1] which holds all the data we need
  • Although Google Form’s Id is in the root node [14]
  • Description is in the node [1][0]
  • Title is in the node [1][8]
  • All the Question Field data is in node [1][1]
  • We need to traverse through this child node list and fetch each item
    • Question Field Text is in  [1]
    • Question Field Type is in [3]
    • Question Field Id is in [4][0][0]
    • Question Field Answer required or not is in [4][0][2]
    • Multiple Answer Options are in [4][0][1]
    • Multiple Answer Options needs to be traversed and loaded
  • We need to create a mapping for numeric values of Question Field Types to readable values, Short Answer Field, Paragraph Field, Multiple Choice Field, etc.

So that’s the complete list of analysis that we have derived from this step, which we need to carry forward to our next level, that is implementing all this logic and retrieving the complete skeleton structure of your Google Forms page!

Let the coding begin!

So just like in the previous article, I’m going to use  dotnet and C# as the language for our little code snippet. And to parse HTML content, I choose HTMLAgilityPack. Then we need Newtonsoft.Json to perform our JSON data structure execution. Also I would be using a Console Project type in dotnet, pretty simple to begin with.

Given you have created the project and added the HTMLAgilityPack and Newtonsoft.Json to your dotnet project, let’s start by creating the model class

Enum mapping class for the Question Field Types that we identified before, so that we can easily cast our scraped out values in code.

Now you might say shouldn’t we create Model classes to represent Google Forms Fields and Google Form parent objects themselves, but I would rather keep that to a future post! 😉 let’s just try to keep things simple in this one!

Then let’s code the method that we’ll be executing the load our Google Form structure! I’m gonna be calling it ScrapeOffFormSkeletonFromGoogleFormsAsync() with a parameter passing in which will carry the URL link to a given Google Form! 😀

Let’s begin by adding the simple LoadFromWebAsync() using the HTMLAgilityPack, which will load the HTML content first.

Next let’s access the FB_PUBLIC_LOAD_DATA_  script content in our HTML doc.

As you can see we are filtering out the html nodes with the “//script” definition which contains “FB_PUBLIC_LOAD_DATA_” value in it. Then we load it into a variable fbPublicLoadDataJsScriptContent which of type string.

Next on, we gotta clean it up by removing the “var FB_PUBLIC_LOAD_DATA_ = ” and the “;” at the end if you notice carefully. So that we can parse the data to a JSON Array structure.

Now we’re ready to parse the content into a JSON Array using Newtonsoft.Json as below.

And also let’s access some basic data of our Google Form, such as Title, Description and the Form ID just like how we discussed in our pattern analysis of this array object structure. Then we load the most important index of the array [1][1] into arrayOfFields variable at the bottom.

Next on we are going to traverse through the list of field data indexes, but here I have added a special filter to identify if the given Field object is an actual Question or a Field placed as a Description Panel or an Image banner, which I have noticed people do to customize their Google Forms. In that case we ignore that object and move on to the next iteration.

There we are looping through each item in arrayOfFields and skipping off the filtered objects. As you can see above, we’re first loading the Field Question text value, then extracting the Question Type value, while using the Enum parser that we build before with mapping the readable Field Type values.

Speaking of accessing Answer Options List for Multiple Choice Questions, we’re handling that next in this code bit.

And we load it up to answerOptionsList object.

Then we load our next Values, Field Answer Submit ID and the value representing if the answer is required to submit or not, with a conversion to boolean which is true or false.

For the IsAnswerRequired value Google gives us “1” or “0” as the representation of true or false, so we need to do that mapping ourselves as you see above.

Then as the last stretch of our loop, let’s print it all out to the Console.

There now the data related to each field is now printed out to the Console nicely.

Let me share the full code snippet that puts it all together below. Strap your seat belts fellas its a long code snippet, therefore I’ll only put a link to it here! 😉

https://gist.github.com/UdaraAlwis/c338a9de4af4509ba0ff67e2c4f37f5c

Yeah click on that Gist link and view the full code snippet over at my Github!

You can use the above method in any of your dot net projects, as long as you have HtmlAgilityPack and Newtonsoft.Json nugets installed and imported in the code. Application is yours to imagine yo, just pass in your Google Forms link text to the method and you’re good to go!

Hit F5 and Run!

Now if you’re on Visual Studio, let’s just run this little snippet of magic eh! 😉

TADAAA! 😀

Here I’m using my simple demo Google Form link, passing it into the method in this little Console dotnet app, and you can see how it nicely loads all the question field data and all the information about my Google Form page.

Here’s a fun side-by-side comparison of programmatically accessing your complete Google Forms skeleton!

Pretty cool eh!

As far as my testing this little script works perfectly for any Google Form that contains the basic main types of Question Fields that are available in Google Forms as of this day!

Imagination is the limit yol! 😉

Well… That’s it!

Who would have thought the FB_PUBLIC_LOAD_DATA_ is such a mysterious yet awesome data snippet hiding in the rendered HTML content of a given Google Forms page! lol 😀

During my experimental research of cracking this mystery, I got some hints from the following python hacks that I derived the same logic into dotnet C# code.

https://gist.github.com/davidbau/8c168b2720eacbf4e68e9e0a9f437838

https://gist.github.com/gcampfield/cb56f05e71c60977ed9917de677f919c

Now keep in mind we do not have precise control whether Google will change these format and data structure patterns in future, so you gotta keep an eye out if you’re planning to use these hacks for a long term solid implementation. My suggestion would be to write up a series of Test cases (TDD yo!)

There you have it, the little magic script to programmatically accessing your complete Google Forms skeleton!

Share the love! 😀 Cheers!

SCRIPTfully scrape off your Google Forms Field Ids…

Y’all wanna scrape off the list of Field IDs of your Google Forms easily? yes programatically through a neat little script! Stick around y’all! 😉

So you remember my last blog posts about my journey of hacking around Google Forms, You may RESTfully submit to your Google Forms… and even in the one before, Let’s auto fill Google Forms with URL parameters… you might remember how crucial it was to hook up to the list of Field Identifiers in your Google Forms which you can use to Submit data to your Form using the REST API or even to populate the Auto filled data in the Form. Now that goes without saying this blog post is a follow up of the above two articles, therefore I would prefer if you took a sneak peak into those before continuing in this article.

So the way we retrieved the list of Field identifiers were by manually reading through the rendered HTML code and looking at the network traffic data. Specifically I introduced three methods as follows…

  • Method 1: Looking up page source HTML content.
  • Method 2: Inspecting HTML code of each field.
  • Method 3: Monitor the network traffic data.

So all those methods are completely manual, shouldn’t there be an easy way? Yes, I’ve been wondering that myself and I continued on experimenting…

Yes.. Scrape em off programmatically!

Wouldn’t it be easier if we could write a little script to retrieve the list of Field IDs in any given Google Form? Just scrape through the rendered HTML content and pick up the Field identifiers automatically! Oh think about how much time you could save! 😀

Well yes, that’s exactly what we’re going to do. Let’s build a simple script that could scrape off the list of Field IDs of your Google Forms without you breaking a sweat!

But to build the script you might have to break some sweat… 😉 Alright, so what this script is going to do is, given the link of a Google Form, it will load the HTML content of it, and scrape through it to find the elements that holds the Field Ids, filter them out and return the results! Quite straight forward eh! 😀

Let the hack begin…

Let’s use a dot net C# to write out script, and yes an absolutely biased choice given my favorite platform! lol But if you can grasp the process and idea behind each step you could easily reproduce the same script from any other language or framework!

It’s all about programming the our method of manually reading through the HTML code and figuring out the IDs, into a self executing code. This requires us understanding the pattern of which the HTML is rendered for each Field element in the Google Form, and I figured this out by repeatedly looking at the rendered HTML content. Once we understand the pattern and where to filter out the data that we need to scrape out, we can easily code it into our script.

Now for this post also let’s use the same sample questionnaire Google Form that I created for last post’s demo.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

So now we got a Google Form, let’s start the little hack by finding the field IDs in the form…

Identify the Pattern..

Now this is the most crucial step, let me walk you through by simply looking into the rendered HTML content of our Google Form.

Now for me though it took a lot of trial and error steps to recognize how all different types of question fields are rendered in Google Forms, where by created a bunch of sample Forms and analyzed their HTML structure continuously, until I was sure. 😉

I’ve explained regarding this pattern in my previous blog post: Let’s auto fill Google Forms with URL parameters… in which I dive into detailed steps of finding the IDs of each question field in your Google Form. If you go through it and focus on the section “Method 2: Inspecting each field” you could easily understand what I’m about to dive into. Therefore let me keep things simple in this article just to avoid repetition. 😀

Assuming you’re using Google Chrome let’s begin, by right clicking on any of the question fields in your Google Form and go to -> Inspect Element menu option.

If you carefully take a look at the rendered HTML node of our Short Answer question Field, it’s actually “input” type element and you can see how the “name” property holds the ID for the Field “entry.1277095329”. Now let’s take another type of an Field, how about Multiple choice selection question Field? 😮

When you’re reading through the parent node of the radio button elements, you can see at the bottom there’s an “input” type element, that’s set to hidden, with the Field ID that we’re looking for. Then how about a Checkbox selection Field? Let’s try the same inspection and see for ourselves… 😉

Now the parent node of it is a bit differently structured, but you can see how it follows the same pattern, of having an “input” type element which holds the ID of the question Field.

Also something to notice is that the same exact child node is repeated in each parent div element containing the ID value. And then just above the child elements, you got another field which carries the same values, but with a “_sentinel” suffix in the “name” property, which sorta creates a repetition of data that needs to be filtered out. So this is something you need to keep in mind, that we will need to filter out in our script. 😀

Next let’s try out a Paragraph question Field of Google Forms, and try to analyze it.

Now this fella got a “textbox” type element rendered, which also includes the “name” property that holds the Field ID, and now we know another element that needs special filtering in our script! 😉

Finally all the other types of Questions Fields in Google Forms follows almost the same pattern of rendered HTML, therefore without further adieu let’s try to analyze the patterns which we seen behind those elements.

Analyze the Pattern…

Now in this step you need to be able to see the full picture of the whole pattern of which those Google Forms question Field elements are rendered in their HTML environment. So after analyzing all those different fields we could draw a few major analysis that we need to keep in mind when we’re thinking of scraping out the Field IDs from the HTML…

  • “input” elements holds the Field IDs in their “name” attribute
  • “textarea” is an exceptional element which is used by Paragraph question type
  • the Field ID value begins with “entry.” prefix in the “name” attribute
  • Checkbox Field elements renderes a repetition of its “input” nodes
  • Also Checkbox Filed generates an extra “input” element with “_sentinel” suffix in ID
  • repeated nodes with same values should be filtered out

Now keeping all those in mind we need to implement the logic into our script, or in other words we need to code the above logic and filtering as rules into our little script that’ll scrap out the HTML of our Google Form to retrieve the Field IDs automatically.

Let the coding begin!

Assuming that you’re already experienced in dotnet, I’m not going to be diving in to spoon feeding details here, and rather focus on the important bits of the code. We’re going be using dotnet and C# as the language for our crawler script. And we need a library that could parse HTML content, and traverse through those content programmatically. Therefore I choose HTMLAgilityPack which is a well known and stable HTML parser for dotnet projects.

So the project type that you’re going to implement is totally up to you, but for this demo I would be using a Console Project in dotnet, pretty simple to begin with.

Given you have added the HTMLAgilityPack to your dotnet project, let’s create the method definition with a string parameter that will represent the URL link of our Google Form that we need to scrape off and it will be returning the list of Field IDs of the given Google Form!

Oh and make sure to make it an async method, hence we will need that for our web call that’ll load the HTML content.

Let’s use the HtmlWeb class which allows us to load an HTML content from a given url string asynchronously.

There we’re executing the LoadFromWebAsync() upon the given Google Forms Link and load it into memory. Next we need to implement out first line of filtering on top of the HTML content that we loaded into memory.

There we’re scooping off the “input” and “textarea” elements from the HTML content,  into a List object of type IEnumerable<HtmlNode>, given the DocumentNode which contains all the HTML elements of the Google Form that we just parsed into memory.

Like I said before we’re basically implementing the logic that we learned in the Pattern Analysis step, therefore next we go on to the next layer of filtering.

There we’re filtering the list of data based on the predicament, that we retrieve all the HTML elements which contains the “entry.” prefix in their “name” attribute, thus securing out Field ID values. Then we exclude all the elements that contains “_sentinel” suffix in their “name” attributes which governs the cleaning up of Checkbox field element repetition.

As you see above we’ve singled out the HTML elements that we’re targeting. Next we gotta do some final clean up of our scraped nodes.

We need to clean up any existing duplicate elements in the list, therefore we’re gonna group similar items, and pick the first element into a list of type List<HtmlNode>, which will eliminate the repetition nodes probably caused by Checkbox fields.

And finally we’re going to access the each Node’s “name” attribute, load it into a List.

And return the results. Oh just an add-on I’m printing out the scraped off Field ID elements into the Console.

Let me share the whole script down here…

Let’s try it out shall we! 😉

Hit F5!

Now I’m gonna use the sample Google Form that I created for this demo, pass its URL link into this little script, and hit F5 in Visual Studio!

Look at that beauty! 😉 The complete list of Question Field identifiers in my sample Google Forms just like we expected.

As far as my testing this little script works perfectly for any Google Form that contains the basic main types of Question Fields that are available in Google Forms as of this day!

Well… That’s it!

Basically you can scrape off any data from a given HTML content as long as you understand the pattern of which the HTML rendered targeting the pieces of data that you’re looking for! Likewise there could be many different types of Google Forms that contains different types Question Fields even with custom content in them, but at the end they all follow a certain HTML rendering pattern, which is just a matter of figuring out.

I would like to remind you again, the reason I considered this as a “SCRIPT” is due to the possibility of converting the same HTML scraping steps into any other language or framework easily, as long as you understand the pattern of the rendered HTML of your Google Form!

Now keep in mind all these are simple hacks and tricks derived by careful observation of rendered HTML content of any given Google Forms page, and we do not have precise control whether Google will change these format and rendering patterns in future, so you gotta keep an eye out if you’re planning to use these hacks for a long term solid implementation.
My suggestion would be to write up a series of Test cases (TDD yo!) which would test for the above process flows to make sure they’re working as expected and notify you in case of any changes from Google. 😉

There you have it, the little magic script to scrape off the list of Field IDs from your Google Forms page!

Share the love! 😀 Cheers!

You may RESTfully submit to your Google Forms…

You wanna submit responses to your Google Forms in a REST-ful API call, or rather programmatically in code or easily from a Postman-like tool? Then you’re welcome to stick around here… 😉

So you remember my last post on my journey of hacking around Google Forms, trying to be a smart-ass eh! Let’s auto fill Google Forms with URL parameters… Oh yeah that one right there, well that was just the tip of the ice berg of me playing around with Google Forms! Let me share the next set of cool tricks I figured out here! 😀

This little trick of submitting data RESTfully to your Google form, could become very handy if you wanted to build your own improved custom UI for submitting data to your Google Form, along with your own validations for the fields or even to quickly populate a bunch of sample data from your Form for experimental reason. Them awesome possibilities are endless! 😉

Well.. Google Forms with RESTful ?!?

So during my adventures into messing around with Google Forms, I figured out that we can submit data into our Google Forms using their REST API endpoint! So how cool is that eh, we can directly post data into our form RESTfully, from whatever the medium you prefer, programmatically, or Postman like tool! 😉

So in this post lemme share that cool trickery bits with you…

Let the hack begin…

Now keep in mind unlike the last post, this is a bit advanced trick which requires some experience on HTML and web development, well it could easily help at least.

We’re gonna get the REST endpoint source of our Google Form, package our question-field-answer data into a request object and submit it to the REST endpoint directly, using Postman or Programmatically in code.

Now for this post also let’s use the same sample questionnaire Google Form that I created for last post’s demo.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

Little sneak peak:

So now we got a Google Form, let’s start the little hack by finding the field IDs in the form…

Yes, you still gotta hook up the fields!

Remember in my last post I explained about the Fields in our Google Form having unique identifiers (or IDs) that we can use to attach data into for submission? well you still need em! 😛

Now for this you could still use the methods we discussed in the previous post to get the list of ID of the fields in your Google Form, but this time I’ll introduce some easier ways, since we’re moving to a bit advance trick…

Hooking up to the fields and Endpoint…

Keep in mind this requires a little bit experience in web dev! 😉 Basically we’re going to get the list of Field IDs by tracing the submission request call in our Google Form, which will also help us figure out the REST endpoint link.

So open up your Google Form in Chrome browser and open up developer tools by using the browser menu or on Windows click “Ctrl+Shift+I keys” in the keyboard.

Now to make the magic work, go to “Network” tab in the menu list which will allow us monitor the network trace that’s going to be sent from browser to Google Form submission REST endpoint.

Next, you need to fill up all the question fields in your Google Form and hit submit button. Carefully watch what happens in the developer console!

Yep a whole bunch of logs pops up, which shows you the traces of all the network calls that occurred in the last “Submit” button click. And in there the most important request log is the “formResponse” log as you seen above.

Click on formResponse log which will bring up all the details on it.

Now this is quite awesome, it will show you in 4 separate sections all the details about the Google Form submission data endpoint that just occurred.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/formResponse

The Request URL is the endpoint we’re going to be using to submit our form data and the Form Data section is where you’ll find the list of field identifiers of your Google Form.

Now that your holy grail of list of field identifiers in bulk. So go ahead, highlight that chunk of text and copy it up to some notepad to be used later.

Now if you noticed the ID with the “entry.1155533672_sentinel” is something that you can ignore, since its a repeated field coming from the Check box question field in your Google Form!

Just like that you can easily extract the list of IDs of the fields in your Google Form! 😀

entry.1277095329: Bibimbap Turtles
entry.995005981: Banana Plums
entry.1155533672: Dogs with hats
entry.1579749043: Jumping Apples
entry.815399500_year: 2019
entry.815399500_month: 11
entry.815399500_day: 11
entry.940653577_hour: 00
entry.940653577_minute: 30

Now that’s the form data sample from my Google Form! 😉

Shove it up into a Postman!

Or any of the web dev tools you prefer to make a simple REST api call. Up to you! lol. Let’s create a POST call with our Google Forms submission API endpoint which we retrieved in the above step.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/formResponse

Actually this URL you could easily make up using your Google Form publish url, just by replacing viewform with formResponse suffix.

So make sure to add the Body parameters of type x-www-form-urlencoded, and list out all the question field IDs and their values you’d like to inject in to the submission. Since then you need to apply header Content-Type as application/x-www-form-urlencoded which will enable our body parameters object.

Assuming you have set up all the body form fields properly, let’s fire it up! 😀

Fire it up!

Let’s go ahead and execute the REST posting! Hit “Send“, and bam!

You should get a successful 200 Status Code response with the above success message “Your Response has been recorded.” inside a whole bunch of HTML content and if you move to the “Preview” tab, you can see how the rendered UI being returned as well.

Now let’s say if you missed adding any specific field in the request body, that was already marked as “Required” in your Google Forms template, and you had hit “Send”. In that case it would return a bad request 400 Status Code with the error in HTML content, “This is a required question”, or with whatever the custom error message you configured your Google Form with.

Yeah you can even view in the Preview tab with the rendered HTML content.

Pretty neat eh! the same way it would behave in a browser environment you can duplicate in a RESTful environment such as Postman! 😀

Now let’s see how easy it is to push that into some code and execute this programatically!

Shove it up into a Code snippet!

Alright let’s shove that into a simple C# snippet where we could POST a simple HTTP request with the body parameters of our Google Form! Basically replicating the same execution as Postman you saw above! 😀

Above we’re using a simple dotnet HttpClient to execute our Google Form submission REST post call, by adding the body values dictionary into the request.

And then we’re printing the Status Code and the HTTP content response we get.

Hit F5!

If you hit F5 on the above code in Visual Studio, you should get the following.

We are successfully submitting our Google Form data to the REST endpoint and getting the success code, along with the usual HTML content, same as how got in Postman. 😉

Now let’s say if something went wrong, or you missed a required field in your request body,

It will show up the same error handling we got in Postman, as 400 Bad Request! And if you dive into the HTML content, you can see how the error message was also delivered back.

So now you know how to programmatically submit data to your Google Forms in code! 😉

Imagination is the limit yol! 😉

Well… That’s it!

It’s quite straightforward how Google has built these forms in such a simple manner for you to handle them easily as you wish. Kudos Google!

Now keep in mind all these are simple hacks and tricks derived by careful observation of html code and network traffic, and we do not have precise control whether Google will change these format and rendering patterns in future, so you gotta keep an eye out if you’re planning to use these hacks for a long term solid implementation.
My suggestion would be to write up a series of Test cases (TDD yo!) which would test for the above process flows to make sure they’re working as expected and notify you in case of any changes from Google. 😉

You can do all kinds of cool stuff with it! So there you have it, how you may RESTfully submit to your Google Forms!

Share the love! 😀 Cheers!

XAMVVM-01 A real walk-through of Xamarin.UITest with Xamarin.Forms!

Let’s take a chilled out walk through adding Xamarin.UITests to your Xamarin.Forms project! 😉

When I said chilled out, I meant literally a chill, no-fuss walk through adding UITests for your Xamarin.Forms project solution. There’s many articles and tutorials out there regarding this, but when I first started I couldn’t find a clear enough guide to begin with for myself, so I was stumbling around in confusion here and there until I self-learned a proper foundation. That’s why I thought of writing this post.

This is an attempt of sharing my experience and intuition of how to architect a better UITest structure for your Xamarin.Forms Solution helping you to get a clear and easy head start!

So I’m not gonna get into extreme baby steps, or details, but a clear fuss-free hands-on guide for starting off with your UITests, which I hope would give you a clear picture and understand of the whole shabang! After all I’m all about that solid project architecture!

Why UITests?

Hey, if you’re looking for a serious intro, please Google! I don’t like repetition of content lol.

Xamarin.UITests comes in handy when you want to have consistent assurance of the actual functionality of the app with the UI behaviour included. And between you and me, I actually love watching UITests being executed on Devices and Simulators, seeing you app actually being used like a human, giving you a whole feedback loop of the UI behaviour , is just an incredible experience! 😉

 

Let’s get started..

Just for the showcase of this awesomeness, I created a little App, which is called Textpad, where you simple take notes or texts of whatever you feel like. 😉 A very simple out of the box Xamarin.Forms app, and fully MVVM architectured code base with Prism. I named the solution as “XFWithUITest” just for this demo.

Whatever the default template of the Xamarin.UITest has provided, I have done several changes to it here and there for the clarity and of the code base as you will see in this article.

So I’m gonna walk you through a clean and well-structured manner of adding Xamarin.UITests to your project solution.

You can take a little sneak peak at it over here in my github repo:
XAMVVM-Playground/XFWithUITest

Structure is important!

There’s many ways to structure a UITest, but I like a clean separation of the elements in any solution architecture. Like here we’re going to separate our Tests from the actual Host projects.

So first, for the name of separation let’s add a Folder called “Tests” in your Xamarin.Forms solution. Yes, that’s the way to start!

Then let’s create our Xamarin.UITest project, right-click on the “Tests” folder in the VS Solution Explorer and go to Test tab and select Xamarin.UITest Cross-Platform Test Project!

Also pay extra attention to the Name and Location value for our UITest project. Append “.UITest” at the end of your project name. As of the location, make sure to add the path along with the “Tests” folder that we created above.

Next create a new Folder inside of that project called “Tests”, yes another one, which is where we’re actually placing our tests! Also create a new class called SetupHooks, which is where we’ll maintain all the hooks that are needed for our tests. (I’ll get into details for this in a later step)

Now it should look something like this!

Nothing more.

Delete anything else that’s unnecessary or not seen above! 😉

Off to next step!

Don’t forget the nugets!

Make sure all the necessary nuget packages are in place, which is just basically the following 3 nugets! yep that’s it!

Pay very careful attention here to the version of NUnit version 2.6.4, which is the minimum NUnit version supported by Xamarin.UITest as of today. (01/11/2018)

The deal with AppInitializer!

Now this right here is where your Tests will be firing up the app’s execution. There are many ways to structure this class and its functionality, but here’s my way…

This class comes pre-populated when you first create the UITest project, but I have made some changes of my own for the clarity of the code.

As you can see I’m passing in an extra boolean parameter “clearData”, which is to execute a clean instance of my App for testing.

I’m using the InstalledApp() call to load the Android and the iOS apps from the simulators, also I’m enabling the EnableLocalScreenshots() to get actual screenshots of my test instances as I wish. Yeah the fact that you can automatically capture screenshots during testing even when you run locally is really cool feature of Xamarin.UITests! 😉

Now instead of getting a hook on the InstalledApp(), you could use the path to the APK or IPA file using the ApkPath() or AppBundle() respective for Android and iOS, which is totally up to your choice.

Then I’m passing in the AppDataMode parameter according to my choosing of the “clearData” value.

SetupHooks holds the instances!

Remember earlier I created a class called SetupHooks? let’s set it up now!

public class SetupHooks
{
      public static IApp App { get; set; }

      public static Platform Platform { get; set; }
}

 

During UITests execution we’re holding a singular instance of the app in memory, which we’re calling through UITest’s functions to perform many operations, so to simplify that, here we’re holding a public static instance of the IApp and Platform object to be used in our Test cases.

Pretty neat eh! 😀

Let’s write the Tests!

Create a class called AppTests, which is where we’re going to place the Test fire up code and the rest of the tests for now!

namespace XFWithUITest.UITest.Tests
{
    [TestFixture(Platform.Android)]
    //[TestFixture(Platform.iOS)]
    public class AppTests
    { 
        public AppTests(Platform platform)
        {
            SetupHooks.Platform = platform;
        }

        [SetUp]
        public void BeforeEachTest()
        {
            SetupHooks.App =  
            AppInitializer.StartApp(SetupHooks.Platform, true);
        }

        [Test]
        ...
	// test cases begin here...
		
	}
}

 

There I have added the TestFixture attributes as required by NUnit to identify our tests, and notice how I have commented out the iOS platform, to show you that you could stick to one platform at a time for your ease of testing, instead of seeing failed tests in the Test Runner window! 😉

[SetUp] is where your Tests will initialize the actual App instance, thus retrieving a hook to the app’s instance for our Test cases to use.

You can see how I’m tying up the SetupHooks – Platform and App instances, through the initiation of the AppTests.

AppInitializer.StartApp(SetupHooks.Platform, true);

This gives a clean instance of the app for our tests cases to use, and up on your wish you could pass in “false” to the same method and get a data persisted instance of the app at anytime, anywhere in your tests! 😉

Now you’re all set to start writing your UITests, but before we begin I need you to check up on something else!

AutomationId for everything!

Whatever the UI element you need to get a hook on to or get a reference of, be it a Page, Button, Layout even a Label, you need to add a value to its AutomationId.

And make sure every AutomationId in a given Page context is unique for every element, otherwise the look up function will return all the elements that matches the given Id, which could lead to confusion in your tests 😉

IApp interface functions!

The Xamarin.UITest.IApp interface provides a whole bunch of functionalities for the app for us to play around with in order to execute our test scenarios.

Take a look here, Xamarin.UITest.IApp to see the list of powerful functions we can use. To name a few are Tap, Swipe, Scroll, WaitForElement and etc, to be performed on any given UI Element on the screen.

So now all you need to do is get a hook on any given element..

Getting a hook on an Element…

There’s several ways of doing this, most common is by the AutomationId of the Element

SetupHooks.App.Tap(c => c.Marked("Button1"))

Another is by the value of an Element’s property,

SetupHooks.App.Tap(c => c.Text("Click this Button!"))

Or you could do by even the Class name of the element. Choice is completely yours, pick the one best suited for your test case.

How to write a Test?

Now this is the coolest part, Xamarin.UITest allows us to get hooks on to UI Elements of the running App, then we perform actions on those elements and wait for the results and check if it resulted as expected through assertion using NUnit.

So its basically a little dance between Xamarin.UITest and NUnit Assertion! 😉

As a standard keep in mind to append “Test” at the end of each of your Test cases.

As you can see above I’m first waiting for the HomePage to appear, then I’m asserting it through NUnit. Then I look for the Label with “Hey there, Welcome!” text!

Action and Result, simple as that! 😀

Some advanced bits…

Here’s some advanced bits that could come in handy!

Getting the number of elements in a ListView
SetupHooks.App.Query(c => c.Marked("TextListView").Child()).Length
Getting an element in a ListView
Func<AppQuery, AppQuery> itemInListView = null;

if (SetupHooks.Platform == Platform.Android)
     itemInListView = 
     x => x.Class("ViewCellRenderer_ViewCellContainer").Index(0);
else if (SetupHooks.Platform == Platform.iOS)
     itemInListView = 
     x => x.Marked("<your listview automationId>").Index(0);

// change the index parameter to get the item you wish
Opening Context Menu in a ListView item
// pop up the Context menu in ListView item
if (SetupHooks.Platform == Platform.Android)
      SetupHooks.App.TouchAndHold(firstCellInListView);
else if (SetupHooks.Platform == Platform.iOS)
      SetupHooks.App.SwipeRightToLeft(firstCellInListView);
Enter Text into an Entry or Editor
SetupHooks.App.EnterText(
c => c.Marked("TextTitleEditor"), whateverYourText);
Wait for an element to disappear
SetupHooks.App.WaitForNoElement(c => c.Text("This label text"));

// either by Text or Marked as should work
Restarting the app anywhere…
// restarting app, persisting state

SetupHooks.App = AppInitializer.StartApp(SetupHooks.Platform, false);

Check out more here in this awesome git page: XamarinTestCloudReference

REPL is your tool!

Yes start using the REPL command line to see how your App’s UI is actually rendered by the native platform at any given execution time. Simply call this anywhere you wish in the UITests steps,

App.REPL();

And you’ll be presented with a CLI which will help you see the whole UI tree of the screen. Simply type “tree” in the CLI and you’re good!

Structuring the tests..

Now there’s many ways to structure all the test cases and scenarios, and there’s no strict standard way that should be followed, but whatever you’re comfortable or fits your project is totally fine and the choice is yours!

You could include all your Test cases in the AppTest class itself, or you can break them into separate classes regarding the Page, or the functionality type.

So for this demo I’m keeping all my UITest cases in the AppTest class itself.

Running the UITests locally!

Well now that we have structured the architecture, here’s the time for actual firing things up and you’ve got couple of things to remember!

You can run your Android Tests on Simulator and Device directly without any modification as long as you provide the right APK path or the App Id.

You can run your iOS Tests only on Visual Studio for Mac, and for the device you need to pass the provisioning details, and as of simulator, you need to pass the Simulator Id.

If you’re using InstalledApp() or ConnectToApp() in your AppInitializer, then make sure the app is already deployed or running in the devices or simulator.

Also make sure to keep your Devices or Simulators or Emulators screens switched on at all times, otherwise tests will break giving a waiting exception.

That’s it!

But I’m not completely satisfied with the architecture, so let’s kick it up a notch! 😀

Little cherry on top Architecture!

Like I said before there’s many ways to construct the architecture for your Test project, one of my favourite ways is by separating the test cases by Page Scenario, which I think is a much cleaner structure.

We’re going to create a base class, “TestBase” which has the constructor initiation and BeforeEachTest setup, then create a sub classes that inherits from it representing whatever the pages we have in the App.

It should look something like this!

And don’t forget you need to add TestFixture attribute for every single sub-class!

So what you’re gonna do is take apart all the Test cases you had in one class and move them into the related pages, simply cut and paste of the methods should do! Also on top of that you could abstract another layer of shared steps that we could reuse across these Page tests. 😀

Then it should give you a clean Test output as below.

There you go, all the Tests are now nicely aligned and structured under the given Page which it associates with!

Pretty neat eh!

So this above structure of mine is somewhat more of a simplification of the Page Object Architecture which is well explained here for Xamarin.UITests: https://www.codetraveler.io/

And even in this official github sample from Xamarin uses the same similar pattern: SmartHotel.Clients.UITests

Done!

As you can see its not that hard to set up your Xamarin.Forms project with UITest once you get the basic understanding of the moving parts and keep a clear structure in your head.

Now for some of you might be experiencing some issues with Xamarin.UITest, in which case I had too when I was first starting off. Therefore I ended up writing this post sharing my experience of solving them: Getting your Xamarin UITests to actually work! So if you’re having any issues getting your Xamarin.UITests to work in Visual Studio, that post might be able to help you. 🙂

Do check out my Github repo of this post:
XAMVVM-Playground/XFWithUITest

Thus concludes my real walk-through of Xamarin.UITests with Xamarin.Forms, in which I hope you got a clear understanding of how to properly structure your project and all the moving bits and pieces that gets the job done! 😀

Share the love! 😀

Cheers!

Using iOS 12 OTP Security Code AutoFill in Xamarin.Forms!

Let’s try new iOS 12 Security Code AutoFill feature in Xamarin.Forms!

The new iOS 12 update has brought a whole bunch of awesome updates, although which were mostly already was available on Android years back.

One of those features is getting rid of the old annoying OTP code filling process in iOS apps, where as every time you have to insert an OTP code into the app you have to quit the app and go to the messages and copy and past the OTP manually into the app.

But with iOS 12 update they have made it easier by allowing Apps to auto read OTP messages without going back and forth into the Message app or copy and pasting. iOS now automatically suggest the top most available OTP message in the inbox for your OTP Text fields in the Keyboard suggestion bar.

(Source: beebom.com)

Well its not practically auto-read but still much convenient than before 😉 lol

What happens is that when an OTP message receives into the Message Inbox, iOS runs a simple text matching algorithm that determines if that message is a valid OTP message or not and based on that keep a track of it in the memory, then when the user clicks on the OTP AutoFill enabled text field in an app, iOS keyboard popup that OTP as a suggestion in the keyboard. So that your users can fill up the OTP into the app without leaving the app or going back into the Messaging app. Pretty convenient!

iOS 12 update in Xamarin!

Following the new iOS 12 update, Xamarin has immediately released the support for it within weeks, so make sure to update your XCode and Xamarin nuget packs, to get your hands dirty with it!

Let’s try it in Xamarin.Forms!

Since iOS 12 is now fully supported on Xamarin, we can access those features in our Xamarin.Forms projects as well, by accessing the Xamarin native project levels. So before we get started please make sure you have updated your Xamarin iOS packages.

iOS provides a new property called UITextContentType.OneTimeCode for the TextContentType property of UITextField in Xamarin.

We’re gonna do this using a custom renderer allowing us to access the UITextField’s native properties, which is the native counterpart of Xamarin.Forms.Entry in iOS.

So let’s start by creating the Custom Control: OTPAutoFillControl

/// <summary>
/// OTP AutoFill Control for Xamarin.Forms
/// </summary>
public class OTPAutoFillControl : Entry
{

}

 

Then’ let’s createthe Custom Renderer!

[assembly: ExportRenderer(typeof(OTPAutoFillControl), typeof(OTPAutoFillControlRenderer))]
namespace XFOTPAutoFillControl.iOS
{
  public class OTPAutoFillControlRenderer: EntryRenderer
  {
    protected override void OnElementChanged(ElementChangedEventArgs<Entry> e)
    {
      base.OnElementChanged(e);

      if (e.NewElement != null)
      {
        Control.TextContentType = UITextContentType.OneTimeCode;
      }
    }
  }
}

 

There we go, setting up our Xamarin.Forms.Entry’s native counterpart UITextField for the OTP Auto-Fill feature in iOS 12.

Alright that’s it you’re all good to go.

<local:OTPAutoFillControl
	Grid.Row="2"
	FontSize="Large"
	HorizontalOptions="Center"
	Keyboard="Numeric"
	Placeholder="XXXXXX" />

 

Now use our OTPAutoFillControl anywhere in your Xamarin.Forms app, deploy to an iOS 12 Device, then retrieve an OTP message, and see the magic happens! 😉

How simple eh! 😀 yep even I was surprised at first.

Check out my github repo sample:
github.com/Xamarin-Playground/XFOTPAutoFillControl

iOS still doesn’t give out much awesome features or god-mode control of the device, but still this by far a really nice and easy to use feature from a developer’s perspective!

Few things to keep in mind!

First thing, iOS 12 – OTP Auto-Fill feature works perfectly in Xamarin Native and Xamarin.Forms because of the native magic of Xamarin. You have nothing to worry!

Then keep in mind, there’s a certain pattern of OTP messages iOS 12 supports for now, not to worry, its mostly standard ones such as below,

(Source: Apple Dev Conference WWDC 2018)

So please do keep an eye out for the OTP message structure if you run into any issues using this feature, otherwise it should work right out of the box like a charm! 😀

Now if you like to learn more about the iOS 12 developer’s update, check out their WWDC 2018 conference:

or wanna learn more of the new OTP AutoFill or Password generate in native iOS, check out these gems:

There you go!

Spread the love! 😀