Monthly Archives: December 2019

Stunning app Themes in Xamarin.Forms Shell projects!

Wanna build some awesome themes into your Xamarin.Forms Shell project? Change the App themes during the run time dynamically? Save the theme properties in User Settings? Then you’re at the right place! 😉

Themes in Xamarin.Forms Apps…

Given Xamarin.Forms is such an incredible cross platform mobile framework, it provides a lot of awesome features out of the box, but unfortunately it doesn’t straight away provide App Theme feature in your face. 😛  but…

It’s got all the bells and whistles you need to implement an awesome App Theme set up with ease, in combination of awesome Dynamic Binding, Static Binding and Style Properties giving you full control of how to handle implementing Themed Styles into your app. You just gotta put together all those bits and you can easily build App Themes functionality into the app.

This is actually quite powerful given the cross platform nature of Xamarin.Forms! Now there are plenty of documentation, articles and sample demos regarding this, so I’m not going to be repeating on that topic, but if you’re interested try out this office doc article from Microsoft: Theming a Xamarin.Forms Application Now that gives a very good step by step explanation on the topic for anyone to easily get started, but anyhow focusing back on this article…

Enters, Xamarin.Forms Shell Apps…

Then enters the recently introduced Xamarin.Forms Shell, a new paradigm of Xamarin.Forms App development. Now when I got started with this new awesomeness, I couldn’t find any direct articles or samples on how to implement App Themes in a Xamarin.Forms Shell projects.

So I had to figure it out myself, and that’s why I thought of sharing my experience with you guys. Now now, don’t get me wrong, since we’re still in the same context of Xamarin.Forms, the App Theming strategy is still the same as the official doc I shared about, but I may have found a better way and a bit of improvements on top of that. 😉

So let’s get started… Stunning app Themes in Xamarin.Forms Shell projects!

How to? in a nutshell!

Basically in a nutshell we’re going to define a set of Themes with Color Properties inside them that holds Color values unique to each Theme. Then we will build all a complete collection of Styles targeting all the components in our app such as Button, Labels and so on. Now in this case we need to stick to Dynamic Binding since we’re going to be switching between Themes during run time, so use of Xamarin.Forms DynamicResource binding all the way ah! 😉

Next we make sure all our UI elements are bounded to those Styles that we created before and still making sure to stick to Dynamic Binding. Now that we have established the binding chain from Theme Properties to Styles to UI Elements, we need to set up the default Theme to load in the ApplicationResources XAML node.

Once all that’s set up we shall be implementing a simple mechanism  to save the User’s App Theme preferences, with features such as Saving and Reloading those preferences on demand.

And that’s it! 😀

Sneak Peak!

Here’s a little sneaky peaky demo magic on Android and iOS.. 😉

If you’d like a sneak peak into the code before I get into the code details, you can find the whole demo project here in my github repo: https://github.com/UdaraAlwis/XFShellAdvThemeing

So in my demo code, I have used the default out of the box Xamarin.Forms Shell project template you get in Visual Studio 2019 latest update, so that you can easily familiarize yourself with the code and adopt the same implementation in your own code.

Set up of the project!

Let’s get started with the set up of the Project, but first should take a look into the structure we’re going to be implementing, where we need to first focus on the following aspects.

  • List of Themes for the App
  • List of properties in a given Theme
  • Styles built using the properties of Themes
  • Helper extension to change App Theme
  • Saving the User’s App Theme preferences

Now keeping those main aspects in mine, I’m going to assume that you have already set up your basic Xamarin.Forms Shell app project. I’m going to name my little demo as XFShellAdvThemeing, denoting Xamarin.Forms Shell Advanced Theme-ing! 😉

Alright there goes our Xamarin.Forms Shell App project, with strong base of MVVM baked in! You may have noticed I have added “Themes” folder, which will hold the App Themes that we’re going to create. And then a “Helpers” which will hold a simple extension method that we’re going to built for switching the App Theme selection by the user during run time.

Building the App Themes…

Time to define the App Themes, so first of all we need to be specific about what properties we are going to be using to each Theme, be it colors, fonts, images and so on, you need to be specific, and make sure all the Theme definitions follow the exact same format.

on github: /XFShellAdvThemeing/Models/Theme.cs

Here I have defined the list of Themes that I’m going to include in my App, and as for having the type of Enum, is for to be used later in the code to populate the data to the User. So make sure to add a record here every time you add a new Theme to your app.

This step is very crucial because you shouldn’t neglect this and try to change the app theme properties half way into the app development. If you’re working with a team,

…you need to define with the team designers together how the Themes of the app should behave, and what elements should be controlled over them.

For this demo I’m going to define the following list of properties inside each Theme.

  • Main Colors:
    • Primary Color
    • Accent Color
    • Secondary Color
  • Page Colors:
    • Page Background Color
    • Navigation Bar Color
  • Text Colors
    • Primary Text Color
    • Secondary Text Color
    • Tertiary Text Color

As you can see I have taken into consideration of having 3 set of main Colors, and as a back up since I’m going to be using native Navigation bar, a property to control its colors and the Page background as they render next to each other. Then finally the Colors for the Text inside the app.

Its very crucial you develop this kind of design centric thinking when you’re developing any mobile app…

So let’s create our beautiful little XAML snippets inside the Themes folder that are going to hold all the properties of each App Theme.

Just simply add a XAML page into the Themes folder and rename the parent node to ResourceDictionary of type and update the same in the code behind as well.

on github: /XFShellAdvThemeing/Themes/DarkTheme.xaml

There my first theme, LightTheme which holds the color values I need to customize in my app for Theme-ing! 😉 Feel free to add as many themes as you like following the same structure.

Defining the Styles…

Now these bits are the middle man between your UI elements and the Theme definitions. As I explained before we are going to create Styles targeting all the types of UI Elements that we’re using in the app, so that we can “Style” them with those! 😉 get it!? lol

on github: /XFShellAdvThemeing/App.xaml

You can define these Styles as a Global App resources or in Page levels as you wish, but I have added them into my App.xaml for this demo, such as Styles targeting Buttons, Labels, etc.

As you can see we are using DynamicResource binding to hold on to the Theme property values, so that we can update our Style property values dynamically in run time.

Now you might wonder why I have referenced Themes/LightTheme.xaml in the  global Resource Dictionary, well… that is to set the default theme as the Light App Theme that I just defined above. You can keep it as it is or watch me switch the App Theme dynamically during run time below… 😉

You also need to directly reference your Shell bound UI Elements using the Theme properties we defined earlier. Here I have added to to the same App.xaml global scope instead of keeping it in the AppShell.xaml scope, just for organizing the styles in one place.

Switching App Theme Dynamically…

Now of course we need to allow our App to be able to switch the Theme dynamically during run time according to User’s choice or some configuration built in. We can easily do this in Xamarin.Forms, using MergedDictionaries property by removing the existing Theme in memory Resources and switching to our choice of Theme Resources.

I have created a simple Helper extension, with the method SetAppTheme() which accepts the type of Them Enum value you need to use and returns the boolean result.

on github: /XFShellAdvThemeing/Helpers/ThemeHelper.cs

Like I said before this is where out little Helper extension comes into play, so just add this little snippet into your Helpers folder.

Based on the Theme Enum identifier value, we will be instantiating the Theme object, assigning it to the Resources in memory as you see above.

Now that’s all cool and stuff, but how about persisting this preferred theme selection?

Saving to User Preferences!

This can easily be done with the help of Xamarin.Essentials, which allows us to save Application context key values pairs using the Preferences API. Now I believe in Visual Studio when you create a new Xamarin.Forms Shell project by default it comes pre-installed with Xamarin.Essentials, otherwise make sure to add it to your project from nuget.

We are going to save the Selected App Theme settings with the key name “CurrentAppTheme” as below.

on github: /XFShellAdvThemeing/Views/ThemeSelectionPage.xaml.cs

And make sure to load it back to the app during the App’s launch event and call up on our little magic extension ThemeHelper.SetAppTheme() as shown here..

on github: /XFShellAdvThemeing/App.xaml.cs

You need to call that in the App() constructor invoke, so that we can load the saved App Theme settings instead of loading the default one that we set up in App.xaml resources.

 

Bingo! nice little App Theme Selection Page to our Xamarin.Forms Shell App! 🙂

Time for some action!

Here it is side by side iOS and Android,

 

Themes with more than Colors?

Now our App Themes aren’t always going to be as simple as a bunch of Color properties right?! It could even contain Fonts, Text Sizes, Images, Icons and so on. But if you’re wondering if that’s even possible in Xamarin.Forms, yes absolutely you can!

It’s basically no different than defining a Color property in your Theme.xaml file, just add the XAML node to the file and you’re good to go!

Make sure to give it a Key name value though, and reference it in your Styles as usual where applicable.

You can follow the same pattern for any kind of Theme property you want to add and basically you’re good to go! 😉

Github Repo: github.com/UdaraAlwis/XFShellAdvThemeing

Any property that you can reference usually from your XAML, you can easily include them in your App Theme and link the binding through Styles to your UI elements straight away.

Some Tips!

Here are some tips and tricks that might come in handy for you, during the whole shabang of “Theme-ing” your Xamarin.Forms Shell app projects.

More Unification! Less Repetition!

Make sure to avoid adding repetitive theme properties into the Theme XAML definitions, by unifying the Colors, Fonts, Icons you use in your app. As an example if you define a Text Color property in your Theme, make sure to use that only for Text Coloring Elements and Styles, try not to use them for other aspects.

So you can easily manage those properties in future and they’d be easy to understand for anyone to extend the properties. This is quite crucial when you maintain massive App Projects, and it wouldn’t affect the capability to grow the App code altogether.

Stubborn Native Elements?! Yikes!

Now as you probably know or don’t Xamarin.Forms doesn’t let you change the color values of your App’s few very native elements during run time. Such as,

– iOS/Android System Status Bar Colors

– Android UI Elements that contains horizontal bar such as Entry, Picker, etc. (Unless you’re using Xamarin.Forms Material Visual)

They require native Android/iOS level access to change during run time. So you need to have Custom Renderers or Native Bound Services that can be communicated through Xamarin.Forms layer during run time. Well that’s a blog post for another time! 😉

So here’s how it could be easily solved as you can see below,

My suggestion would be to maintain values of those UI elements as compatible as possible that could be matched with the Theme Colors you currently use. Such as light Gray, White or Black mostly.

You can easily set them up from Resources/values/styles.xml in your Android project.

And on iOS project’s Info.plist configuration.

And that’s pretty much it!

Share the love! 😀 Cheers!

I built a Google Forms Toolkit Library for .NET!

Watch me building the soon-to-be-famous (lol) my GoogleFormsToolkitLibrary for .NET C#, a nuget library that will help you easily access your Google Forms programmatically in dotnet, letting you load Field Question data, Submit Form data and so on! 😉

In my previous blog article, Programmatically access your complete Google Forms skeleton! where I shared about my adventure into working out a little hack to access your Google Forms page content programatically, I realized it was a solid implementation, and could be used with almost all kinds of openly accessible Google Forms.

So following up on that, this time I will be building a complete .NET library that will consist of all the awesome hacks and tricks that I built around playing with Google Forms myself, which I have also continuously written blog articles as well!

Backstory…

So far we have come across…

And then in my previous blog article, Programmatically access your complete Google Forms skeleton! I shared how we retrieving the following data on a given Google Form extensively,

  • Google Form Title, Description, Form ID
  • List of Question Fields

And in each Question Field,

  • Question Field Text
  • Question Type
  • If submitting Answer is mandatory or not
  • Available answer list (Multiple answer selection)
  • Question Field Identifier (Field Id) or Answer Submission ID

Then we analyzed the data structure of the FB_PUBLIC_LOAD_DATA_ script that we scraped out of the rendered HTML content, and even managed to parse it into a meaningful data structure which could be mapped to the data we are looking for.

So using that knowledge and expertise, let’s step up the game and build a solid library tool, which we can reuse easily in anywhere to access our Google Forms programmatically to perform all kinds of awesomeness as we wish! 😉

However if you haven’t gone through my previous blog articles in this Google Forms Hacks series, please do so before continuing to avoid any confusion, since I might not be diving into all the details.

Behold GoogleFormsToolkitLibrary v1!

So this library that I’m building will be on .NET Standards, which will allow you to use it in almost any kind of a .NET project. From Console apps, Desktop, Web and even to Mobile Apps in Xamarin!

It will provide you with the following awesome features!

  • Load information on your Google Form
  • Load Question Field data on your Google Form
  • Submit Form data to your Google Form

At least for now, I will be adding more features as I go along my journey of hacking around Google Forms! 😉

I will be publishing this library to Nuget, so that you can easily grab it into your .NET projects, also let’s add some fancy Test Driven goodness to it, using xUnit Tests! 😀

Project Solution Set up…

So I created a .NET Stanard version 2.0 library in Visual Studio, naming it GoogleFormsToolkitLibrary of course! 😉 The I added a Test folder into the Solution which will hold the xUnit Unit Test project, which I named .Tests!

Then don’t forget to add the HTMLAgilityPack and Newtonsoft.Json nugets to the Library solution which will allow us to scrape out the HTML content in a Google Form and use the Json content parsing logic!

Now the basic set up is done, let’s move on to the code!

Let the coding begin…

We need to model the data objects that we’re going to handle when it comes to the perspective of a Google Form. Basically following are the main entities of a Google Form as far as we have recognized in our previous blog articles!

  • A Google Form
  • A Google Form Question Field
  • The Google Form Field Type

Then as of the functionality I would be adding two public methods that we can call upon to execute.

  • Load Google Form Structure, data, and form fields
  • Submit form data to a given Google Form

Crafting the Models…

So we let’s start by building Model classes that will represent the entities we need.

GoogleFormsFieldTypeEnum

Let’s start with the simplest Model we could build that is the Google Form Field Type object, which will represent the type of a given Question Field. I basically described in detail about this entity and how I build the entity model myself through trial and error in my previous blog article:

Let’s create an enum model class giving it the name  GoogleFormsFieldTypeEnum.

We’re adding the Description tags to hold the human readable value of each type.

GoogleFormField

Now this fella right here is a big deal, that is the Google Form Question Field object, which will represent a single Question Field in a given Google Form. A whole list of these fields comprises in every Google Form.

Let’s create a model class giving it the name  GoogleFormField.

As you can see we have created properties inside this model that will hold all the values that we discussed previously, that could consist in a given Google Form Question Field.

GoogleForm

Now here’s the top entity, representing a whole Google Form page content. Let’s create a model class giving it the name  GoogleForm.

Pretty straight forward set of properties that would usually contain in any Google Form as you can see above. Now make sure to add those Model classes to the Models directory in your project.

Alrighto, we got the model classes done, then let’s move on to the actual functional coding!

Implementing the core features…

We’re going add two main functionality to this library, for now as following! 😉

LoadGoogleFormStructureAsync

Now this method will intake a string parameter that will carry your Google Forms link, and it will return an object of type GoogleForm that we define above. Which obviously contains the given Google Form’s generic information and Question Field list data including Question Type, Answer Options, Submission Id, etc. Now that’s quite a handful eh! 😉

Task<GoogleForm> 
LoadGoogleFormStructureAsync(string yourGoogleFormsUrl)

So this fella will make Http Async call to load the rendered HTML of the Google Form into memory, run through my magical data scraping algorithm, build the GoogleForm object and return it! Sounds pretty simple but don’t let the implementation confuse you, take a look here! 😉

https://gist.github.com/UdaraAlwis/dcb473f9f07d5024376f1289c1b08ace

Please feel free to take a look at the full implementation on Gist link.

This method is more of an extension of the ScrapeOffFormSkeletonFromGoogleFormsAsync() method that I shared with you all in my last blog post, which I gotten into in depth details about explaining each step of the code.

So to reduce repetition I’m not going to repeat the same here, rather I would share the important bits to explain that are improvements on top of the previous implementation.

One of the differences I would say is that now we’re injecting the data that we scraped out into our model objects, as you can see we’re initializing out GoogleForm and GoogleFormField data objects here, and then loading the GoogleForm data object values.

Then inside the loop we’re loading the GoogleFormField object values as well, and then each object will then be added to the QuestionFormFieldList property of the GoogleForm. That’s pretty much it! 😉

SubmitToGoogleFormAsync

This method will intake a string parameter that will carry your Google Forms link, and  a Dictionary object containing the Form field answer submission id and answer values mapping. As a result it will return a boolean value denoting the success or failure of the submission task.

Task<bool> SubmitToGoogleFormAsync
(string yourGoogleFormsUrl, Dictionary<string, string> formData)

It will make a Http Async call the Google Forms REST API endpoint with the given data and await for the response code, upon receiving the response it will return true or false based on it.

https://gist.github.com/UdaraAlwis/8d547f68ae3f629d40b2184f51acd43e

Please feel free to take a look at the full implementation on Gist link.

This methods is actually an improvement of one of my previous posts, You may RESTfully submit to your Google Forms… where I explain in depth how I hacked around to figure this out and what each line of code is meant to handle. Please look into that article if you’re keep for more details.

We’re basically looking for the 200 StatusCode value that determines successful submission to the Google Forms REST API endpoint.

Now our Core Library is ready to go!

Unit Test it yo!

Yep but its not done until we implement a proper set of test cases isn’t it! So let’s add some Unit Tests that will test the functionalities that we built into our Library.

Like I mentioned in the Project set up, we have added a xUnit Unit Test project into the solution, and make sure to add a reference of our Library project GoogleFormsToolkitLibrary into it.

Let’s start by adding a Tests.cs class to it, which we will use to implement the tests. For now, I will add two test cases,

RetrieveGoogleFormStructure_Success

This will test for LoadGoogleFormStructureAsync() feature and make sure we get a valid response and the expected data of the given Google Form page.

SubmitDataToGoogleForm_Success

Now this case will test for the SubmitToGoogleFormAsync() feature and make sure we get a successful boolean response, with the given Google Form link, and the form data dictionary object.

Oh let’s not forget to make sure they’re passing in the Test Runner! 😀 lol

All good to go it seems! for now at least lol 😛

This is really a great practice because if I ever break anything in the code during any new implementation or existing modifications, I would be able to notice it here in the tests.

Nugetting it!

Let’s not forget to publish the beauty to Nuget eh! Let’s make sure all the nuget properties are added to the package before uploading…

And you may find it on nuget right now!

Nuget: nuget.org/packages/GoogleFormsToolkitLibrary/

This whole project is open sourced and published to Github as well if you’re interested in looking at the code, or track the improvements I’m adding in the future!

Github: github.com/GoogleFormsToolkitLibrary

So that’s done!

Let’s try it out!

Now the beauty of this is that you can easily add this to any of your .NET projects as the whole library is based on .NET Standard.

Go ahead and add it to your .NET project from Nuget using either the Package Manager Console or the Nuget Package Manager.

Install-Package GoogleFormsToolkitLibrary

Use of GoogleFormsToolkitLibrary is pretty simple, just instantiate it and call upon the method you wanna use.

// Retrieve the structure of my sample Google Forms page
// https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

var googleFormLink =
"https://docs.google.com/forms/d/e/" +
"1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA" +
"/viewform";

var googleFormsToolkitLibrary = new GoogleFormsToolkitLibrary();
var result = await googleFormsToolkitLibrary.
                    LoadGoogleFormStructureAsync(googleFormLink);

 

That should work like a charm! Make sure to pass your Google Form’s link properly and make sure it is openly accessible, you should be able to see the magic pretty easily!

// Submit data to my sample Google Forms page
// https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

var googleFormLink =
	"https://docs.google.com/forms/d/e/" +
	"1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA" +
	"/formResponse";

Dictionary<string,string> formData = new Dictionary<string, string>
{
	// Question Field 1
	{"entry.1277095329", "Moon Rockets Launching"}, 

	// Question Field 2
	{"entry.995005981","Banana Plums"},

	// Question Field 3
	{"entry.1155533672","Monkeys with hoodies"},

	// Question Field 4
	{"entry.1579749043","Jumping Apples"},

	// Question Field 5
	{"entry.815399500_year","2019"},
	{"entry.815399500_month","11"},
	{"entry.815399500_day","11"},

	// Question Field 6
	{"entry.940653577_hour","04"},
	{"entry.940653577_minute","12"},
};

var googleFormsToolkitLibrary = new GoogleFormsToolkitLibrary();
var result = await googleFormsToolkitLibrary
                .SubmitToGoogleFormAsync(googleFormLink, formData);

 

The above should nicely respond with a successful true value! 😉

Something to keep in mind here is that you need to make sure you’re setting the correct Field Answer Submission ID properly, and when it comes to Multiple Answer selection fields, make sure the provided answers matches the available list of answers. Date Time fields should be carefully treated with their additional year,month,day suffixed fields and hour,minute respectively. Just go through my past few blog posts on Google Forms hacking, you’ll see for yourself!

What’s next?

Well building this and publishing it openly is just the first step, I’m going to continue building this further adding more features and performance improvements.

Then I will be building some demo Client apps which will use this library to implement some cool features… 😀

So keep an eye out!

There you have it, how I built a Google Forms Toolkit Library for .NET!

Share the love! 😀 Cheers!

Programmatically access your complete Google Forms skeleton!

So you wanna load the content of your Google Forms programmatically? access all the questions, answer options, field types, submit ids and so on as you wish? 😉 Stick around!

Hope yol remember in my last blog post, SCRIPTfully scrape off your Google Forms Field Ids… I was sharing a neat little script I built to extract the Field identifier IDs from your Google Forms page, so that you can use them with ease for auto filling question answer fields or submitting data using REST API into your Google Forms! So this is the nest step in this series of Google Forms Hacks!

In this post we’re gonna load your Google Forms skeleton programmatically!

What does it mean?

Yes, instead of loading your Google Forms page on the typical web browser, why not load the bare bone content of it as you wish? By this I mean,

  • List of Questions (your question content…)
  • The types of Questions (Short Answer, Paragraph, Checkbox, etc)
  • Available Answer options (Multiple choice answer questions…)
  • Title and Description of your Google Form, etc…
  • And many more details you have added to your Google Form…

Once you get access to those bare bone content or the skeleton structure of your Google Form, then you can do all kinds of stuff with it…

Then  you can render it as you wish and present to your users, re-render it into a Web App or a Desktop app or even a Mobile App with your own custom layouts, filtering, and validations! 😀

So how we gonna do it?

Simply put, we gonna extract the skeleton or the bare bone structure from our Google Forms page. Now as you may have figured out there’s no official API or SDK to access Google Forms services programmatically, therefore we obviously have to hack our way around this!

We’re going to build a little a script which will load the HTML content of our Google Form and perform a magical algorithm to extract our Google Forms structure! All the questions, answer options, validation, and etc the whole deal… 😉

Backstory…

In my last post hope you remember how I shared about scraping through the HTML content of our Google Form page to extract Field identifiers which is used to submit answers or otherwise known as Field Answer identifiers.

Then along that same time I was experimenting with trying to extract the whole structure of the design the same manner. But filtering through the HTML tags to retrieve the complex structure of a given Google Forms Question-Answer field structure seemed quite hectic.

Jackpot!

So while I was  going up and down the HTML of the page, trying to find a better way to extract our Google Forms Question-Answer structure, I came across this interesting piece of script at the end of the page.

As you can see at the bottom there’s java script starting with “FB_PUBLIC_LOAD_DATA_ = [nulll…” with a strange pattern of the content, which I’m not sure what the purpose of it in this page, but you can surely find this in any Google Forms page. This specific script snippet seem to be holding the whole skeleton of our Google Forms page as you can see, it containing the Question Content, Answer Fields and so on!

So there’s our little treasure… 😉

Now finding this bit wasn’t enough at all, I had to figure out how  to parse this properly to extract the data that we’re looking for, as you can see it’s got a little unorthodox structure in its content. Now that’s the next challenge!

Let the hacking begin…

Now for this post also let’s use the same sample questionnaire Google Form that I created for last post’s demo.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

So we’re going to load the HTML content of our Google Forms page, and then we’ll extract that mystery javascript code snippet, and parse the content of it to access the skeleton structure of our Google Forms!

Now first let me walk you through how to parse that mystery FB_PUBLIC_LOAD_DATA_  content! 😉 Let me warn you though, figure this out was no walk in the park but let me share the secret source with yol straight away.

Let me copy and paste it here from my sample Google Form and get started! Below you can clearly see how all the Questions along with Answers and Field identifiers in my sample Google Form are contained in this mystery code snippet.

var FB_PUBLIC_LOAD_DATA_ = [null,["Please fill up the following questions. You need to answer all the fields please! ;) ",[[122249536,"Hello there, this is question 1, could you answer?",null,0,[[1277095329,null,0]]],[1170747525,"Which one would you prefer as the answer from below?",null,2,[[995005981,[["Mango Peach",null,null,null,0],["Banana Plums",null,null,null,0],["Strawberry Pears",null,null,null,0]],1,null,null,null,null,null,0]]],[2147453523,"Well another question here wouldn't hurt now eh?",null,4,[[1155533672,[["Monkeys with hoodies",null,null,null,0],["Dogs with hats",null,null,null,0],["Cats with crowns",null,null,null,0]],1,null,null,null,null,null,0]]],[172187917,"How about this for a change?",null,3,[[1579749043,[["Running Banana",null,null,null,0],["Jumping Apples",null,null,null,0],["Rolling Pears",null,null,null,0]],1,null,null,null,null,null,0]]],[676251522,"What's the date would you like to be today?",null,9,[[815399500,null,1,null,null,null,null,[0,1]]]],[1280585510,"What time would it be right now? ",null,10,[[940653577,null,1,null,null,null,[0]]]]],null,null,null,[0,0],null,null,"Sample questionnaire!",48,[null,null,null,null,0],null,null,null,null,[2]],"/forms","Random Sample Questionnaire",null,null,null,"0",null,0,0,"","",0,"e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA",1];

In order to parse this into a known data structure you need to first remove the “var FB_PUBLIC_LOAD_DATA_ = ” and the “;” at the end if you notice carefully.

However by looking at the content you can make a guess that it should be some JSON based content structure.

So let’s use any online JSON parser tool and you should be able to see the formatted content as follows. ex: jsoneditoronline, codebeautify or jsonformatter

Now you get some readable structure yeah, where you are able to traverse through it’s content by expandable nodes array.

Pattern Recognition and Analysis!

Since we parsed the mystery content into a JSON Array tree, we can now traverse through the data easily and extract the specific data that we’re looking for.

Now since there’s no official documentation regarding the parsing of this data from structure, I guess we’re going to have to figure out ourselves, where to pick what data in this data tree and recognize the pattern.

So basically what we are going to be focusing on retrieving the following data,

  • Google Form Title, Description, Form ID
  • List of Question Fields

And in each question,

  • Question Field Text
  • Question Type
  • If submitting Answer is mandatory or not
  • Available answer list (Multiple answer selection)
  • Question Field Identifier (Field Id) or Answer Submission ID

Now that I believe comprises the complete structural skeleton of any given Google Form! 🙂

– General Google Form data

In the parent root you can see the Google Form Doc’s Name in the [3] index.

Then the Form Id in the index [14] of the array tree. And rest of the important bits seem to be in the [1] index node as you can see it has a lot of child nodes in it. Let’s look into that..

Now this index item seem to be containing a lot of information in child nodes. In its child node index [0] you can see the Description of our Google Form, and index [8] holds the Title.

Next the most important node, that is node index [1] which contains all the Question Fields data, and it looks like this.

Once you expand it you can view the List of child elements that represents the Question Fields in your Google Form!

Now you know you can easily access the Question fields by traversing [1][index of the question field]

Let’s try opening up a child node then! 😉

– Question Field data

Woot! Here you have it, the whole Question Field as expected, and specifically in this sample questionnaire, you can see how it shows all the details regarding the 1st Question Field.

Also notice the value I’m pointing to below, node [1][0][3] to be exact, that index holds the Question Type value “0”, which determines whether it’s a Short Answer, Paragraph, Dropdown, Checkbox field and so on. Then the second arrow, node [1][0][4][0][0], that right there is the unique Field identifier that we need to use when submitting answer to this question field. 😀 Thereby we could consider the value as Answer Submit Id as well.

Now that’s a single answer field, such as Short Answer and Paragraph Question field types in Google Form.

In the node [1][0][4][0][2] holds the value to determine whether Answer Required or not.

So how about we open up a child node of a Multiple Answer field?

Now below I’ve open up the node of a Multiple Answer selection Question Field.

– Multiple Answer Question Field data

Right here you can see in the Question Type value node it has number “2” as the value, which is what Multiple Answer Question Fields are denoted with.

Over here you get an extra child node underneath the Field identifier node as you recognized above.

This has a list of child nodes which as you can see holds the list of Answers Available for the Question Field.

So now we know how to load the list of Answers available for a given question, that we can traverse through.

Next let me dive a bit deep into Question Type identifier values..

– Hunt for Question Field types…

You already know Google Forms provide multiple types of Question Fields that can be added to your Google Form, and you saw above where to grab the Field type value. But how do you know which value maps to what type?

That’s why I had to run a trial and error recognition of trying to match those numeric values to the actual Field types, and I’ve finalized the list as follows…

  • Short Answer Field = 0
  • Paragraph Field = 1,
  • Multiple Choice Field = 2
  • Check Boxes Field = 4
  • Drop Down Field = 3
  • File Upload Field = 13

// File Upload – we’re not going to implement for this right now, because it needs user log in session implementation, a bit complicated. So let’s look into it later!

  • Linear Scale Field = 5
  • Grid Choice Field = 7

//Grid Choice – represents both: Multiple Choice Grid & Checkbox Grid

  • Date Field = 9
  • Time Field = 10

As you can see the assigning of the numeric values for the available types of Fields are not so straight forward and Google Forms tends to mix up or skips some mapping of the values without a proper order.  That’s why I mentioned it was a bit painful trial and error process.

Now we’ve walked through the recognition of the data pathways that we’re hoping to extract, let’s summarize our analysis as follows..

  • The most crucial node is index [1] which holds all the data we need
  • Although Google Form’s Id is in the root node [14]
  • Description is in the node [1][0]
  • Title is in the node [1][8]
  • All the Question Field data is in node [1][1]
  • We need to traverse through this child node list and fetch each item
    • Question Field Text is in  [1]
    • Question Field Type is in [3]
    • Question Field Id is in [4][0][0]
    • Question Field Answer required or not is in [4][0][2]
    • Multiple Answer Options are in [4][0][1]
    • Multiple Answer Options needs to be traversed and loaded
  • We need to create a mapping for numeric values of Question Field Types to readable values, Short Answer Field, Paragraph Field, Multiple Choice Field, etc.

So that’s the complete list of analysis that we have derived from this step, which we need to carry forward to our next level, that is implementing all this logic and retrieving the complete skeleton structure of your Google Forms page!

Let the coding begin!

So just like in the previous article, I’m going to use  dotnet and C# as the language for our little code snippet. And to parse HTML content, I choose HTMLAgilityPack. Then we need Newtonsoft.Json to perform our JSON data structure execution. Also I would be using a Console Project type in dotnet, pretty simple to begin with.

Given you have created the project and added the HTMLAgilityPack and Newtonsoft.Json to your dotnet project, let’s start by creating the model class

Enum mapping class for the Question Field Types that we identified before, so that we can easily cast our scraped out values in code.

Now you might say shouldn’t we create Model classes to represent Google Forms Fields and Google Form parent objects themselves, but I would rather keep that to a future post! 😉 let’s just try to keep things simple in this one!

Then let’s code the method that we’ll be executing the load our Google Form structure! I’m gonna be calling it ScrapeOffFormSkeletonFromGoogleFormsAsync() with a parameter passing in which will carry the URL link to a given Google Form! 😀

Let’s begin by adding the simple LoadFromWebAsync() using the HTMLAgilityPack, which will load the HTML content first.

Next let’s access the FB_PUBLIC_LOAD_DATA_  script content in our HTML doc.

As you can see we are filtering out the html nodes with the “//script” definition which contains “FB_PUBLIC_LOAD_DATA_” value in it. Then we load it into a variable fbPublicLoadDataJsScriptContent which of type string.

Next on, we gotta clean it up by removing the “var FB_PUBLIC_LOAD_DATA_ = ” and the “;” at the end if you notice carefully. So that we can parse the data to a JSON Array structure.

Now we’re ready to parse the content into a JSON Array using Newtonsoft.Json as below.

And also let’s access some basic data of our Google Form, such as Title, Description and the Form ID just like how we discussed in our pattern analysis of this array object structure. Then we load the most important index of the array [1][1] into arrayOfFields variable at the bottom.

Next on we are going to traverse through the list of field data indexes, but here I have added a special filter to identify if the given Field object is an actual Question or a Field placed as a Description Panel or an Image banner, which I have noticed people do to customize their Google Forms. In that case we ignore that object and move on to the next iteration.

There we are looping through each item in arrayOfFields and skipping off the filtered objects. As you can see above, we’re first loading the Field Question text value, then extracting the Question Type value, while using the Enum parser that we build before with mapping the readable Field Type values.

Speaking of accessing Answer Options List for Multiple Choice Questions, we’re handling that next in this code bit.

And we load it up to answerOptionsList object.

Then we load our next Values, Field Answer Submit ID and the value representing if the answer is required to submit or not, with a conversion to boolean which is true or false.

For the IsAnswerRequired value Google gives us “1” or “0” as the representation of true or false, so we need to do that mapping ourselves as you see above.

Then as the last stretch of our loop, let’s print it all out to the Console.

There now the data related to each field is now printed out to the Console nicely.

Let me share the full code snippet that puts it all together below. Strap your seat belts fellas its a long code snippet, therefore I’ll only put a link to it here! 😉

https://gist.github.com/UdaraAlwis/c338a9de4af4509ba0ff67e2c4f37f5c

Yeah click on that Gist link and view the full code snippet over at my Github!

You can use the above method in any of your dot net projects, as long as you have HtmlAgilityPack and Newtonsoft.Json nugets installed and imported in the code. Application is yours to imagine yo, just pass in your Google Forms link text to the method and you’re good to go!

Hit F5 and Run!

Now if you’re on Visual Studio, let’s just run this little snippet of magic eh! 😉

TADAAA! 😀

Here I’m using my simple demo Google Form link, passing it into the method in this little Console dotnet app, and you can see how it nicely loads all the question field data and all the information about my Google Form page.

Here’s a fun side-by-side comparison of programmatically accessing your complete Google Forms skeleton!

Pretty cool eh!

As far as my testing this little script works perfectly for any Google Form that contains the basic main types of Question Fields that are available in Google Forms as of this day!

Imagination is the limit yol! 😉

Well… That’s it!

Who would have thought the FB_PUBLIC_LOAD_DATA_ is such a mysterious yet awesome data snippet hiding in the rendered HTML content of a given Google Forms page! lol 😀

During my experimental research of cracking this mystery, I got some hints from the following python hacks that I derived the same logic into dotnet C# code.

https://gist.github.com/davidbau/8c168b2720eacbf4e68e9e0a9f437838

https://gist.github.com/gcampfield/cb56f05e71c60977ed9917de677f919c

Now keep in mind we do not have precise control whether Google will change these format and data structure patterns in future, so you gotta keep an eye out if you’re planning to use these hacks for a long term solid implementation. My suggestion would be to write up a series of Test cases (TDD yo!)

There you have it, the little magic script to programmatically accessing your complete Google Forms skeleton!

Share the love! 😀 Cheers!