Category Archives: Web Crawling

I built a Google Forms Toolkit Library for .NET!

Watch me building the soon-to-be-famous (lol) my GoogleFormsToolkitLibrary for .NET C#, a nuget library that will help you easily access your Google Forms programmatically in dotnet, letting you load Field Question data, Submit Form data and so on! 😉

In my previous blog article, Programmatically access your complete Google Forms skeleton! where I shared about my adventure into working out a little hack to access your Google Forms page content programatically, I realized it was a solid implementation, and could be used with almost all kinds of openly accessible Google Forms.

So following up on that, this time I will be building a complete .NET library that will consist of all the awesome hacks and tricks that I built around playing with Google Forms myself, which I have also continuously written blog articles as well!

Backstory…

So far we have come across…

And then in my previous blog article, Programmatically access your complete Google Forms skeleton! I shared how we retrieving the following data on a given Google Form extensively,

  • Google Form Title, Description, Form ID
  • List of Question Fields

And in each Question Field,

  • Question Field Text
  • Question Type
  • If submitting Answer is mandatory or not
  • Available answer list (Multiple answer selection)
  • Question Field Identifier (Field Id) or Answer Submission ID

Then we analyzed the data structure of the FB_PUBLIC_LOAD_DATA_ script that we scraped out of the rendered HTML content, and even managed to parse it into a meaningful data structure which could be mapped to the data we are looking for.

So using that knowledge and expertise, let’s step up the game and build a solid library tool, which we can reuse easily in anywhere to access our Google Forms programmatically to perform all kinds of awesomeness as we wish! 😉

However if you haven’t gone through my previous blog articles in this Google Forms Hacks series, please do so before continuing to avoid any confusion, since I might not be diving into all the details.

Behold GoogleFormsToolkitLibrary v1!

So this library that I’m building will be on .NET Standards, which will allow you to use it in almost any kind of a .NET project. From Console apps, Desktop, Web and even to Mobile Apps in Xamarin!

It will provide you with the following awesome features!

  • Load information on your Google Form
  • Load Question Field data on your Google Form
  • Submit Form data to your Google Form

At least for now, I will be adding more features as I go along my journey of hacking around Google Forms! 😉

I will be publishing this library to Nuget, so that you can easily grab it into your .NET projects, also let’s add some fancy Test Driven goodness to it, using xUnit Tests! 😀

Project Solution Set up…

So I created a .NET Stanard version 2.0 library in Visual Studio, naming it GoogleFormsToolkitLibrary of course! 😉 The I added a Test folder into the Solution which will hold the xUnit Unit Test project, which I named .Tests!

Then don’t forget to add the HTMLAgilityPack and Newtonsoft.Json nugets to the Library solution which will allow us to scrape out the HTML content in a Google Form and use the Json content parsing logic!

Now the basic set up is done, let’s move on to the code!

Let the coding begin…

We need to model the data objects that we’re going to handle when it comes to the perspective of a Google Form. Basically following are the main entities of a Google Form as far as we have recognized in our previous blog articles!

  • A Google Form
  • A Google Form Question Field
  • The Google Form Field Type

Then as of the functionality I would be adding two public methods that we can call upon to execute.

  • Load Google Form Structure, data, and form fields
  • Submit form data to a given Google Form

Crafting the Models…

So we let’s start by building Model classes that will represent the entities we need.

GoogleFormsFieldTypeEnum

Let’s start with the simplest Model we could build that is the Google Form Field Type object, which will represent the type of a given Question Field. I basically described in detail about this entity and how I build the entity model myself through trial and error in my previous blog article:

Let’s create an enum model class giving it the name  GoogleFormsFieldTypeEnum.


// using System.ComponentModel;
/// <summary>
/// Found the Field type representation values with trial
/// and error try out of blood sweat and tears lol! 😉
/// </summary>
public enum GoogleFormsFieldTypeEnum
{
[Description("Short Answer Field")]
ShortAnswerField = 0,
[Description("Paragraph Field")]
ParagraphField = 1,
[Description("Multiple Choice Field")]
MultipleChoiceField = 2,
[Description("Check Boxes Field")]
CheckBoxesField = 4,
[Description("Drop Down Field")]
DropDownField = 3,
// FileUpload – Not supported (needs user log in session)
[Description("File Upload Field")]
FileUploadField = 13,
[Description("Linear Scale Field")]
LinearScaleField = 5,
// represents both: Multiple Choice Grid | Checkbox Grid
[Description("Grid Choice Field")]
GridChoiceField = 7,
[Description("Date Field")]
DateField = 9,
[Description("Time Field")]
TimeField = 10,
}

We’re adding the Description tags to hold the human readable value of each type.

GoogleFormField

Now this fella right here is a big deal, that is the Google Form Question Field object, which will represent a single Question Field in a given Google Form. A whole list of these fields comprises in every Google Form.

Let’s create a model class giving it the name  GoogleFormField.


// using System.Collections.Generic;
/// <summary>
/// A Question Field in a Google Form
/// </summary>
public class GoogleFormField
{
/// <summary>
/// Type of the Question Field
/// </summary>
public GoogleFormsFieldTypeEnum QuestionType { get; set; }
/// <summary>
/// Question text of the Field
/// </summary>
public string QuestionText { get; set; }
/// <summary>
/// The unique Id need to be used
/// when submitting the answer
/// I also refer to this as: Field Id
/// </summary>
public string AnswerSubmissionId { get; set; }
/// <summary>
/// Available Answer List for any kind of
/// multiple answer selection field
/// </summary>
public List<string> AnswerOptionList { get; set; } = new List<string>();
/// <summary>
/// If the answer is required to Submit
/// </summary>
public bool IsAnswerRequired { get; set; }
}

As you can see we have created properties inside this model that will hold all the values that we discussed previously, that could consist in a given Google Form Question Field.

GoogleForm

Now here’s the top entity, representing a whole Google Form page content. Let’s create a model class giving it the name  GoogleForm.


/// <summary>
/// A model representing a Google Form structure
/// consist of main the properties of a Google Form
/// </summary>
public class GoogleForm
{
/// <summary>
/// Document Name of your Google Form
/// </summary>
public string FormDocName { get; set; }
/// <summary>
/// Form ID of your Google Form
/// </summary>
public string FormId { get; set; }
/// <summary>
/// Title of your Google Form
/// </summary>
public string Title { get; set; }
/// <summary>
/// Description of your Google Form
/// </summary>
public string Description { get; set; }
/// <summary>
/// List of Question Fields in your Google Form
/// </summary>
public List<GoogleFormField> QuestionFieldList { get; set; }
}

view raw

GoogleForm.cs

hosted with ❤ by GitHub

Pretty straight forward set of properties that would usually contain in any Google Form as you can see above. Now make sure to add those Model classes to the Models directory in your project.

Alrighto, we got the model classes done, then let’s move on to the actual functional coding!

Implementing the core features…

We’re going add two main functionality to this library, for now as following! 😉

LoadGoogleFormStructureAsync

Now this method will intake a string parameter that will carry your Google Forms link, and it will return an object of type GoogleForm that we define above. Which obviously contains the given Google Form’s generic information and Question Field list data including Question Type, Answer Options, Submission Id, etc. Now that’s quite a handful eh! 😉

Task<GoogleForm> 
LoadGoogleFormStructureAsync(string yourGoogleFormsUrl)

So this fella will make Http Async call to load the rendered HTML of the Google Form into memory, run through my magical data scraping algorithm, build the GoogleForm object and return it! Sounds pretty simple but don’t let the implementation confuse you, take a look here! 😉

https://gist.github.com/UdaraAlwis/dcb473f9f07d5024376f1289c1b08ace

Please feel free to take a look at the full implementation on Gist link.

This method is more of an extension of the ScrapeOffFormSkeletonFromGoogleFormsAsync() method that I shared with you all in my last blog post, which I gotten into in depth details about explaining each step of the code.

So to reduce repetition I’m not going to repeat the same here, rather I would share the important bits to explain that are improvements on top of the previous implementation.

One of the differences I would say is that now we’re injecting the data that we scraped out into our model objects, as you can see we’re initializing out GoogleForm and GoogleFormField data objects here, and then loading the GoogleForm data object values.

Then inside the loop we’re loading the GoogleFormField object values as well, and then each object will then be added to the QuestionFormFieldList property of the GoogleForm. That’s pretty much it! 😉

SubmitToGoogleFormAsync

This method will intake a string parameter that will carry your Google Forms link, and  a Dictionary object containing the Form field answer submission id and answer values mapping. As a result it will return a boolean value denoting the success or failure of the submission task.

Task<bool> SubmitToGoogleFormAsync
(string yourGoogleFormsUrl, Dictionary<string, string> formData)

It will make a Http Async call the Google Forms REST API endpoint with the given data and await for the response code, upon receiving the response it will return true or false based on it.

https://gist.github.com/UdaraAlwis/8d547f68ae3f629d40b2184f51acd43e

Please feel free to take a look at the full implementation on Gist link.

This methods is actually an improvement of one of my previous posts, You may RESTfully submit to your Google Forms… where I explain in depth how I hacked around to figure this out and what each line of code is meant to handle. Please look into that article if you’re keep for more details.

We’re basically looking for the 200 StatusCode value that determines successful submission to the Google Forms REST API endpoint.

Now our Core Library is ready to go!

Unit Test it yo!

Yep but its not done until we implement a proper set of test cases isn’t it! So let’s add some Unit Tests that will test the functionalities that we built into our Library.

Like I mentioned in the Project set up, we have added a xUnit Unit Test project into the solution, and make sure to add a reference of our Library project GoogleFormsToolkitLibrary into it.

Let’s start by adding a Tests.cs class to it, which we will use to implement the tests. For now, I will add two test cases,

RetrieveGoogleFormStructure_Success

This will test for LoadGoogleFormStructureAsync() feature and make sure we get a valid response and the expected data of the given Google Form page.

SubmitDataToGoogleForm_Success

Now this case will test for the SubmitToGoogleFormAsync() feature and make sure we get a successful boolean response, with the given Google Form link, and the form data dictionary object.

Oh let’s not forget to make sure they’re passing in the Test Runner! 😀 lol

All good to go it seems! for now at least lol 😛

This is really a great practice because if I ever break anything in the code during any new implementation or existing modifications, I would be able to notice it here in the tests.

Nugetting it!

Let’s not forget to publish the beauty to Nuget eh! Let’s make sure all the nuget properties are added to the package before uploading…

And you may find it on nuget right now!

Nuget: nuget.org/packages/GoogleFormsToolkitLibrary/

This whole project is open sourced and published to Github as well if you’re interested in looking at the code, or track the improvements I’m adding in the future!

Github: github.com/GoogleFormsToolkitLibrary

So that’s done!

Let’s try it out!

Now the beauty of this is that you can easily add this to any of your .NET projects as the whole library is based on .NET Standard.

Go ahead and add it to your .NET project from Nuget using either the Package Manager Console or the Nuget Package Manager.

Install-Package GoogleFormsToolkitLibrary

Use of GoogleFormsToolkitLibrary is pretty simple, just instantiate it and call upon the method you wanna use.

// Retrieve the structure of my sample Google Forms page
// https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

var googleFormLink =
"https://docs.google.com/forms/d/e/" +
"1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA" +
"/viewform";

var googleFormsToolkitLibrary = new GoogleFormsToolkitLibrary();
var result = await googleFormsToolkitLibrary.
                    LoadGoogleFormStructureAsync(googleFormLink);

 

That should work like a charm! Make sure to pass your Google Form’s link properly and make sure it is openly accessible, you should be able to see the magic pretty easily!

// Submit data to my sample Google Forms page
// https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

var googleFormLink =
	"https://docs.google.com/forms/d/e/" +
	"1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA" +
	"/formResponse";

Dictionary<string,string> formData = new Dictionary<string, string>
{
	// Question Field 1
	{"entry.1277095329", "Moon Rockets Launching"}, 

	// Question Field 2
	{"entry.995005981","Banana Plums"},

	// Question Field 3
	{"entry.1155533672","Monkeys with hoodies"},

	// Question Field 4
	{"entry.1579749043","Jumping Apples"},

	// Question Field 5
	{"entry.815399500_year","2019"},
	{"entry.815399500_month","11"},
	{"entry.815399500_day","11"},

	// Question Field 6
	{"entry.940653577_hour","04"},
	{"entry.940653577_minute","12"},
};

var googleFormsToolkitLibrary = new GoogleFormsToolkitLibrary();
var result = await googleFormsToolkitLibrary
                .SubmitToGoogleFormAsync(googleFormLink, formData);

 

The above should nicely respond with a successful true value! 😉

Something to keep in mind here is that you need to make sure you’re setting the correct Field Answer Submission ID properly, and when it comes to Multiple Answer selection fields, make sure the provided answers matches the available list of answers. Date Time fields should be carefully treated with their additional year,month,day suffixed fields and hour,minute respectively. Just go through my past few blog posts on Google Forms hacking, you’ll see for yourself!

What’s next?

Well building this and publishing it openly is just the first step, I’m going to continue building this further adding more features and performance improvements.

Then I will be building some demo Client apps which will use this library to implement some cool features… 😀

So keep an eye out!

There you have it, how I built a Google Forms Toolkit Library for .NET!

Share the love! 😀 Cheers!

Programmatically access your complete Google Forms skeleton!

So you wanna load the content of your Google Forms programmatically? access all the questions, answer options, field types, submit ids and so on as you wish? 😉 Stick around!

Hope yol remember in my last blog post, SCRIPTfully scrape off your Google Forms Field Ids… I was sharing a neat little script I built to extract the Field identifier IDs from your Google Forms page, so that you can use them with ease for auto filling question answer fields or submitting data using REST API into your Google Forms! So this is the nest step in this series of Google Forms Hacks!

In this post we’re gonna load your Google Forms skeleton programmatically!

What does it mean?

Yes, instead of loading your Google Forms page on the typical web browser, why not load the bare bone content of it as you wish? By this I mean,

  • List of Questions (your question content…)
  • The types of Questions (Short Answer, Paragraph, Checkbox, etc)
  • Available Answer options (Multiple choice answer questions…)
  • Title and Description of your Google Form, etc…
  • And many more details you have added to your Google Form…

Once you get access to those bare bone content or the skeleton structure of your Google Form, then you can do all kinds of stuff with it…

Then  you can render it as you wish and present to your users, re-render it into a Web App or a Desktop app or even a Mobile App with your own custom layouts, filtering, and validations! 😀

So how we gonna do it?

Simply put, we gonna extract the skeleton or the bare bone structure from our Google Forms page. Now as you may have figured out there’s no official API or SDK to access Google Forms services programmatically, therefore we obviously have to hack our way around this!

We’re going to build a little a script which will load the HTML content of our Google Form and perform a magical algorithm to extract our Google Forms structure! All the questions, answer options, validation, and etc the whole deal… 😉

Backstory…

In my last post hope you remember how I shared about scraping through the HTML content of our Google Form page to extract Field identifiers which is used to submit answers or otherwise known as Field Answer identifiers.

Then along that same time I was experimenting with trying to extract the whole structure of the design the same manner. But filtering through the HTML tags to retrieve the complex structure of a given Google Forms Question-Answer field structure seemed quite hectic.

Jackpot!

So while I was  going up and down the HTML of the page, trying to find a better way to extract our Google Forms Question-Answer structure, I came across this interesting piece of script at the end of the page.

As you can see at the bottom there’s java script starting with “FB_PUBLIC_LOAD_DATA_ = [nulll…” with a strange pattern of the content, which I’m not sure what the purpose of it in this page, but you can surely find this in any Google Forms page. This specific script snippet seem to be holding the whole skeleton of our Google Forms page as you can see, it containing the Question Content, Answer Fields and so on!

So there’s our little treasure… 😉

Now finding this bit wasn’t enough at all, I had to figure out how  to parse this properly to extract the data that we’re looking for, as you can see it’s got a little unorthodox structure in its content. Now that’s the next challenge!

Let the hacking begin…

Now for this post also let’s use the same sample questionnaire Google Form that I created for last post’s demo.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

So we’re going to load the HTML content of our Google Forms page, and then we’ll extract that mystery javascript code snippet, and parse the content of it to access the skeleton structure of our Google Forms!

Now first let me walk you through how to parse that mystery FB_PUBLIC_LOAD_DATA_  content! 😉 Let me warn you though, figure this out was no walk in the park but let me share the secret source with yol straight away.

Let me copy and paste it here from my sample Google Form and get started! Below you can clearly see how all the Questions along with Answers and Field identifiers in my sample Google Form are contained in this mystery code snippet.

var FB_PUBLIC_LOAD_DATA_ = [null,["Please fill up the following questions. You need to answer all the fields please! ;) ",[[122249536,"Hello there, this is question 1, could you answer?",null,0,[[1277095329,null,0]]],[1170747525,"Which one would you prefer as the answer from below?",null,2,[[995005981,[["Mango Peach",null,null,null,0],["Banana Plums",null,null,null,0],["Strawberry Pears",null,null,null,0]],1,null,null,null,null,null,0]]],[2147453523,"Well another question here wouldn't hurt now eh?",null,4,[[1155533672,[["Monkeys with hoodies",null,null,null,0],["Dogs with hats",null,null,null,0],["Cats with crowns",null,null,null,0]],1,null,null,null,null,null,0]]],[172187917,"How about this for a change?",null,3,[[1579749043,[["Running Banana",null,null,null,0],["Jumping Apples",null,null,null,0],["Rolling Pears",null,null,null,0]],1,null,null,null,null,null,0]]],[676251522,"What's the date would you like to be today?",null,9,[[815399500,null,1,null,null,null,null,[0,1]]]],[1280585510,"What time would it be right now? ",null,10,[[940653577,null,1,null,null,null,[0]]]]],null,null,null,[0,0],null,null,"Sample questionnaire!",48,[null,null,null,null,0],null,null,null,null,[2]],"/forms","Random Sample Questionnaire",null,null,null,"0",null,0,0,"","",0,"e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA",1];

In order to parse this into a known data structure you need to first remove the “var FB_PUBLIC_LOAD_DATA_ = ” and the “;” at the end if you notice carefully.

However by looking at the content you can make a guess that it should be some JSON based content structure.

So let’s use any online JSON parser tool and you should be able to see the formatted content as follows. ex: jsoneditoronline, codebeautify or jsonformatter

Now you get some readable structure yeah, where you are able to traverse through it’s content by expandable nodes array.

Pattern Recognition and Analysis!

Since we parsed the mystery content into a JSON Array tree, we can now traverse through the data easily and extract the specific data that we’re looking for.

Now since there’s no official documentation regarding the parsing of this data from structure, I guess we’re going to have to figure out ourselves, where to pick what data in this data tree and recognize the pattern.

So basically what we are going to be focusing on retrieving the following data,

  • Google Form Title, Description, Form ID
  • List of Question Fields

And in each question,

  • Question Field Text
  • Question Type
  • If submitting Answer is mandatory or not
  • Available answer list (Multiple answer selection)
  • Question Field Identifier (Field Id) or Answer Submission ID

Now that I believe comprises the complete structural skeleton of any given Google Form! 🙂

– General Google Form data

In the parent root you can see the Google Form Doc’s Name in the [3] index.

Then the Form Id in the index [14] of the array tree. And rest of the important bits seem to be in the [1] index node as you can see it has a lot of child nodes in it. Let’s look into that..

Now this index item seem to be containing a lot of information in child nodes. In its child node index [0] you can see the Description of our Google Form, and index [8] holds the Title.

Next the most important node, that is node index [1] which contains all the Question Fields data, and it looks like this.

Once you expand it you can view the List of child elements that represents the Question Fields in your Google Form!

Now you know you can easily access the Question fields by traversing [1][index of the question field]

Let’s try opening up a child node then! 😉

– Question Field data

Woot! Here you have it, the whole Question Field as expected, and specifically in this sample questionnaire, you can see how it shows all the details regarding the 1st Question Field.

Also notice the value I’m pointing to below, node [1][0][3] to be exact, that index holds the Question Type value “0”, which determines whether it’s a Short Answer, Paragraph, Dropdown, Checkbox field and so on. Then the second arrow, node [1][0][4][0][0], that right there is the unique Field identifier that we need to use when submitting answer to this question field. 😀 Thereby we could consider the value as Answer Submit Id as well.

Now that’s a single answer field, such as Short Answer and Paragraph Question field types in Google Form.

In the node [1][0][4][0][2] holds the value to determine whether Answer Required or not.

So how about we open up a child node of a Multiple Answer field?

Now below I’ve open up the node of a Multiple Answer selection Question Field.

– Multiple Answer Question Field data

Right here you can see in the Question Type value node it has number “2” as the value, which is what Multiple Answer Question Fields are denoted with.

Over here you get an extra child node underneath the Field identifier node as you recognized above.

This has a list of child nodes which as you can see holds the list of Answers Available for the Question Field.

So now we know how to load the list of Answers available for a given question, that we can traverse through.

Next let me dive a bit deep into Question Type identifier values..

– Hunt for Question Field types…

You already know Google Forms provide multiple types of Question Fields that can be added to your Google Form, and you saw above where to grab the Field type value. But how do you know which value maps to what type?

That’s why I had to run a trial and error recognition of trying to match those numeric values to the actual Field types, and I’ve finalized the list as follows…

  • Short Answer Field = 0
  • Paragraph Field = 1,
  • Multiple Choice Field = 2
  • Check Boxes Field = 4
  • Drop Down Field = 3
  • File Upload Field = 13

// File Upload – we’re not going to implement for this right now, because it needs user log in session implementation, a bit complicated. So let’s look into it later!

  • Linear Scale Field = 5
  • Grid Choice Field = 7

//Grid Choice – represents both: Multiple Choice Grid & Checkbox Grid

  • Date Field = 9
  • Time Field = 10

As you can see the assigning of the numeric values for the available types of Fields are not so straight forward and Google Forms tends to mix up or skips some mapping of the values without a proper order.  That’s why I mentioned it was a bit painful trial and error process.

Now we’ve walked through the recognition of the data pathways that we’re hoping to extract, let’s summarize our analysis as follows..

  • The most crucial node is index [1] which holds all the data we need
  • Although Google Form’s Id is in the root node [14]
  • Description is in the node [1][0]
  • Title is in the node [1][8]
  • All the Question Field data is in node [1][1]
  • We need to traverse through this child node list and fetch each item
    • Question Field Text is in  [1]
    • Question Field Type is in [3]
    • Question Field Id is in [4][0][0]
    • Question Field Answer required or not is in [4][0][2]
    • Multiple Answer Options are in [4][0][1]
    • Multiple Answer Options needs to be traversed and loaded
  • We need to create a mapping for numeric values of Question Field Types to readable values, Short Answer Field, Paragraph Field, Multiple Choice Field, etc.

So that’s the complete list of analysis that we have derived from this step, which we need to carry forward to our next level, that is implementing all this logic and retrieving the complete skeleton structure of your Google Forms page!

Let the coding begin!

So just like in the previous article, I’m going to use  dotnet and C# as the language for our little code snippet. And to parse HTML content, I choose HTMLAgilityPack. Then we need Newtonsoft.Json to perform our JSON data structure execution. Also I would be using a Console Project type in dotnet, pretty simple to begin with.

Given you have created the project and added the HTMLAgilityPack and Newtonsoft.Json to your dotnet project, let’s start by creating the model class

Enum mapping class for the Question Field Types that we identified before, so that we can easily cast our scraped out values in code.


// using System.ComponentModel;
/// <summary>
/// Found the Field type representation values with trial
/// and error try out of blood sweat and tears lol! 😉
/// </summary>
public enum GoogleFormsFieldTypeEnum
{
[Description("Short Answer Field")]
ShortAnswerField = 0,
[Description("Paragraph Field")]
ParagraphField = 1,
[Description("Multiple Choice Field")]
MultipleChoiceField = 2,
[Description("Check Boxes Field")]
CheckBoxesField = 4,
[Description("Drop Down Field")]
DropDownField = 3,
// FileUpload – Not supported (needs user log in session)
[Description("File Upload Field")]
FileUploadField = 13,
[Description("Linear Scale Field")]
LinearScaleField = 5,
// represents both: Multiple Choice Grid | Checkbox Grid
[Description("Grid Choice Field")]
GridChoiceField = 7,
[Description("Date Field")]
DateField = 9,
[Description("Time Field")]
TimeField = 10,
}

Now you might say shouldn’t we create Model classes to represent Google Forms Fields and Google Form parent objects themselves, but I would rather keep that to a future post! 😉 let’s just try to keep things simple in this one!

Then let’s code the method that we’ll be executing the load our Google Form structure! I’m gonna be calling it ScrapeOffFormSkeletonFromGoogleFormsAsync() with a parameter passing in which will carry the URL link to a given Google Form! 😀

Let’s begin by adding the simple LoadFromWebAsync() using the HTMLAgilityPack, which will load the HTML content first.

Next let’s access the FB_PUBLIC_LOAD_DATA_  script content in our HTML doc.

As you can see we are filtering out the html nodes with the “//script” definition which contains “FB_PUBLIC_LOAD_DATA_” value in it. Then we load it into a variable fbPublicLoadDataJsScriptContent which of type string.

Next on, we gotta clean it up by removing the “var FB_PUBLIC_LOAD_DATA_ = ” and the “;” at the end if you notice carefully. So that we can parse the data to a JSON Array structure.

Now we’re ready to parse the content into a JSON Array using Newtonsoft.Json as below.

And also let’s access some basic data of our Google Form, such as Title, Description and the Form ID just like how we discussed in our pattern analysis of this array object structure. Then we load the most important index of the array [1][1] into arrayOfFields variable at the bottom.

Next on we are going to traverse through the list of field data indexes, but here I have added a special filter to identify if the given Field object is an actual Question or a Field placed as a Description Panel or an Image banner, which I have noticed people do to customize their Google Forms. In that case we ignore that object and move on to the next iteration.

There we are looping through each item in arrayOfFields and skipping off the filtered objects. As you can see above, we’re first loading the Field Question text value, then extracting the Question Type value, while using the Enum parser that we build before with mapping the readable Field Type values.

Speaking of accessing Answer Options List for Multiple Choice Questions, we’re handling that next in this code bit.

And we load it up to answerOptionsList object.

Then we load our next Values, Field Answer Submit ID and the value representing if the answer is required to submit or not, with a conversion to boolean which is true or false.

For the IsAnswerRequired value Google gives us “1” or “0” as the representation of true or false, so we need to do that mapping ourselves as you see above.

Then as the last stretch of our loop, let’s print it all out to the Console.

There now the data related to each field is now printed out to the Console nicely.

Let me share the full code snippet that puts it all together below. Strap your seat belts fellas its a long code snippet, therefore I’ll only put a link to it here! 😉

https://gist.github.com/UdaraAlwis/c338a9de4af4509ba0ff67e2c4f37f5c

Yeah click on that Gist link and view the full code snippet over at my Github!

You can use the above method in any of your dot net projects, as long as you have HtmlAgilityPack and Newtonsoft.Json nugets installed and imported in the code. Application is yours to imagine yo, just pass in your Google Forms link text to the method and you’re good to go!

Hit F5 and Run!

Now if you’re on Visual Studio, let’s just run this little snippet of magic eh! 😉

TADAAA! 😀

Here I’m using my simple demo Google Form link, passing it into the method in this little Console dotnet app, and you can see how it nicely loads all the question field data and all the information about my Google Form page.

Here’s a fun side-by-side comparison of programmatically accessing your complete Google Forms skeleton!

Pretty cool eh!

As far as my testing this little script works perfectly for any Google Form that contains the basic main types of Question Fields that are available in Google Forms as of this day!

Imagination is the limit yol! 😉

Well… That’s it!

Who would have thought the FB_PUBLIC_LOAD_DATA_ is such a mysterious yet awesome data snippet hiding in the rendered HTML content of a given Google Forms page! lol 😀

During my experimental research of cracking this mystery, I got some hints from the following python hacks that I derived the same logic into dotnet C# code.

https://gist.github.com/davidbau/8c168b2720eacbf4e68e9e0a9f437838

https://gist.github.com/gcampfield/cb56f05e71c60977ed9917de677f919c

Now keep in mind we do not have precise control whether Google will change these format and data structure patterns in future, so you gotta keep an eye out if you’re planning to use these hacks for a long term solid implementation. My suggestion would be to write up a series of Test cases (TDD yo!)

There you have it, the little magic script to programmatically accessing your complete Google Forms skeleton!

Share the love! 😀 Cheers!

SCRIPTfully scrape off your Google Forms Field Ids…

Y’all wanna scrape off the list of Field IDs of your Google Forms easily? yes programatically through a neat little script! Stick around y’all! 😉

So you remember my last blog posts about my journey of hacking around Google Forms, You may RESTfully submit to your Google Forms… and even in the one before, Let’s auto fill Google Forms with URL parameters… you might remember how crucial it was to hook up to the list of Field Identifiers in your Google Forms which you can use to Submit data to your Form using the REST API or even to populate the Auto filled data in the Form. Now that goes without saying this blog post is a follow up of the above two articles, therefore I would prefer if you took a sneak peak into those before continuing in this article.

So the way we retrieved the list of Field identifiers were by manually reading through the rendered HTML code and looking at the network traffic data. Specifically I introduced three methods as follows…

  • Method 1: Looking up page source HTML content.
  • Method 2: Inspecting HTML code of each field.
  • Method 3: Monitor the network traffic data.

So all those methods are completely manual, shouldn’t there be an easy way? Yes, I’ve been wondering that myself and I continued on experimenting…

Yes.. Scrape em off programmatically!

Wouldn’t it be easier if we could write a little script to retrieve the list of Field IDs in any given Google Form? Just scrape through the rendered HTML content and pick up the Field identifiers automatically! Oh think about how much time you could save! 😀

Well yes, that’s exactly what we’re going to do. Let’s build a simple script that could scrape off the list of Field IDs of your Google Forms without you breaking a sweat!

But to build the script you might have to break some sweat… 😉 Alright, so what this script is going to do is, given the link of a Google Form, it will load the HTML content of it, and scrape through it to find the elements that holds the Field Ids, filter them out and return the results! Quite straight forward eh! 😀

Let the hack begin…

Let’s use a dot net C# to write out script, and yes an absolutely biased choice given my favorite platform! lol But if you can grasp the process and idea behind each step you could easily reproduce the same script from any other language or framework!

It’s all about programming the our method of manually reading through the HTML code and figuring out the IDs, into a self executing code. This requires us understanding the pattern of which the HTML is rendered for each Field element in the Google Form, and I figured this out by repeatedly looking at the rendered HTML content. Once we understand the pattern and where to filter out the data that we need to scrape out, we can easily code it into our script.

Now for this post also let’s use the same sample questionnaire Google Form that I created for last post’s demo.

https://docs.google.com/forms/d/e/1FAIpQLSeuZiyN-uQBbmmSLxT81xGUfgjMQpUFyJ4D7r-0zjegTy_0HA/viewform

So now we got a Google Form, let’s start the little hack by finding the field IDs in the form…

Identify the Pattern..

Now this is the most crucial step, let me walk you through by simply looking into the rendered HTML content of our Google Form.

Now for me though it took a lot of trial and error steps to recognize how all different types of question fields are rendered in Google Forms, where by created a bunch of sample Forms and analyzed their HTML structure continuously, until I was sure. 😉

I’ve explained regarding this pattern in my previous blog post: Let’s auto fill Google Forms with URL parameters… in which I dive into detailed steps of finding the IDs of each question field in your Google Form. If you go through it and focus on the section “Method 2: Inspecting each field” you could easily understand what I’m about to dive into. Therefore let me keep things simple in this article just to avoid repetition. 😀

Assuming you’re using Google Chrome let’s begin, by right clicking on any of the question fields in your Google Form and go to -> Inspect Element menu option.

If you carefully take a look at the rendered HTML node of our Short Answer question Field, it’s actually “input” type element and you can see how the “name” property holds the ID for the Field “entry.1277095329”. Now let’s take another type of an Field, how about Multiple choice selection question Field? 😮

When you’re reading through the parent node of the radio button elements, you can see at the bottom there’s an “input” type element, that’s set to hidden, with the Field ID that we’re looking for. Then how about a Checkbox selection Field? Let’s try the same inspection and see for ourselves… 😉

Now the parent node of it is a bit differently structured, but you can see how it follows the same pattern, of having an “input” type element which holds the ID of the question Field.

Also something to notice is that the same exact child node is repeated in each parent div element containing the ID value. And then just above the child elements, you got another field which carries the same values, but with a “_sentinel” suffix in the “name” property, which sorta creates a repetition of data that needs to be filtered out. So this is something you need to keep in mind, that we will need to filter out in our script. 😀

Next let’s try out a Paragraph question Field of Google Forms, and try to analyze it.

Now this fella got a “textbox” type element rendered, which also includes the “name” property that holds the Field ID, and now we know another element that needs special filtering in our script! 😉

Finally all the other types of Questions Fields in Google Forms follows almost the same pattern of rendered HTML, therefore without further adieu let’s try to analyze the patterns which we seen behind those elements.

Analyze the Pattern…

Now in this step you need to be able to see the full picture of the whole pattern of which those Google Forms question Field elements are rendered in their HTML environment. So after analyzing all those different fields we could draw a few major analysis that we need to keep in mind when we’re thinking of scraping out the Field IDs from the HTML…

  • “input” elements holds the Field IDs in their “name” attribute
  • “textarea” is an exceptional element which is used by Paragraph question type
  • the Field ID value begins with “entry.” prefix in the “name” attribute
  • Checkbox Field elements renderes a repetition of its “input” nodes
  • Also Checkbox Filed generates an extra “input” element with “_sentinel” suffix in ID
  • repeated nodes with same values should be filtered out

Now keeping all those in mind we need to implement the logic into our script, or in other words we need to code the above logic and filtering as rules into our little script that’ll scrap out the HTML of our Google Form to retrieve the Field IDs automatically.

Let the coding begin!

Assuming that you’re already experienced in dotnet, I’m not going to be diving in to spoon feeding details here, and rather focus on the important bits of the code. We’re going be using dotnet and C# as the language for our crawler script. And we need a library that could parse HTML content, and traverse through those content programmatically. Therefore I choose HTMLAgilityPack which is a well known and stable HTML parser for dotnet projects.

So the project type that you’re going to implement is totally up to you, but for this demo I would be using a Console Project in dotnet, pretty simple to begin with.

Given you have added the HTMLAgilityPack to your dotnet project, let’s create the method definition with a string parameter that will represent the URL link of our Google Form that we need to scrape off and it will be returning the list of Field IDs of the given Google Form!

Oh and make sure to make it an async method, hence we will need that for our web call that’ll load the HTML content.

Let’s use the HtmlWeb class which allows us to load an HTML content from a given url string asynchronously.

There we’re executing the LoadFromWebAsync() upon the given Google Forms Link and load it into memory. Next we need to implement out first line of filtering on top of the HTML content that we loaded into memory.

There we’re scooping off the “input” and “textarea” elements from the HTML content,  into a List object of type IEnumerable<HtmlNode>, given the DocumentNode which contains all the HTML elements of the Google Form that we just parsed into memory.

Like I said before we’re basically implementing the logic that we learned in the Pattern Analysis step, therefore next we go on to the next layer of filtering.

There we’re filtering the list of data based on the predicament, that we retrieve all the HTML elements which contains the “entry.” prefix in their “name” attribute, thus securing out Field ID values. Then we exclude all the elements that contains “_sentinel” suffix in their “name” attributes which governs the cleaning up of Checkbox field element repetition.

As you see above we’ve singled out the HTML elements that we’re targeting. Next we gotta do some final clean up of our scraped nodes.

We need to clean up any existing duplicate elements in the list, therefore we’re gonna group similar items, and pick the first element into a list of type List<HtmlNode>, which will eliminate the repetition nodes probably caused by Checkbox fields.

And finally we’re going to access the each Node’s “name” attribute, load it into a List.

And return the results. Oh just an add-on I’m printing out the scraped off Field ID elements into the Console.

Let me share the whole script down here…


// Imports you might need! 😉
//using HtmlAgilityPack;
//using System;
//using System.Collections.Generic;
//using System.Linq;
//using System.Threading.Tasks;
private static async Task<List<string>> ScrapeOffListOfFieldIdsFromGoogleFormsAsync(string yourGoogleFormsUrl)
{
HtmlWeb web = new HtmlWeb();
var htmlDoc = await web.LoadFromWebAsync(yourGoogleFormsUrl);
// Select the "input", "textarea" elements from the html content
var fields = new[] { "input", "textarea" }; // two types of fields
var htmlNodes = htmlDoc.DocumentNode.Descendants().
Where(x => fields.Contains(x.Name));
// Filter out the elements we need
htmlNodes = htmlNodes.Where(
x =>
// Get all that elements contains "entry." prefix in the name
x.GetAttributeValue("name", "").Contains("entry.") &&
// Ignored the "_sentinel" elements rendered for checkboxes fields
!x.GetAttributeValue("name", "").Contains("_sentinel"));
// remove any duplicates (possibly caused by Checkboxes Fields)
var groupedList = htmlNodes.GroupBy(x => x.OuterHtml);
var cleanedNodeList = new List<HtmlNode>();
foreach (var groupedItem in groupedList)
{
cleanedNodeList.Add(groupedItem.First());
}
// retrieve the Fields list
var fieldIdList = new List<string>();
foreach (var node in cleanedNodeList)
{
// grab the Field Id
var fieldId = node.GetAttributeValue("name", "");
fieldIdList.Add(fieldId);
Console.WriteLine(fieldId);
}
return fieldIdList;
}

Let’s try it out shall we! 😉

Hit F5!

Now I’m gonna use the sample Google Form that I created for this demo, pass its URL link into this little script, and hit F5 in Visual Studio!

Look at that beauty! 😉 The complete list of Question Field identifiers in my sample Google Forms just like we expected.

As far as my testing this little script works perfectly for any Google Form that contains the basic main types of Question Fields that are available in Google Forms as of this day!

Well… That’s it!

Basically you can scrape off any data from a given HTML content as long as you understand the pattern of which the HTML rendered targeting the pieces of data that you’re looking for! Likewise there could be many different types of Google Forms that contains different types Question Fields even with custom content in them, but at the end they all follow a certain HTML rendering pattern, which is just a matter of figuring out.

I would like to remind you again, the reason I considered this as a “SCRIPT” is due to the possibility of converting the same HTML scraping steps into any other language or framework easily, as long as you understand the pattern of the rendered HTML of your Google Form!

Now keep in mind all these are simple hacks and tricks derived by careful observation of rendered HTML content of any given Google Forms page, and we do not have precise control whether Google will change these format and rendering patterns in future, so you gotta keep an eye out if you’re planning to use these hacks for a long term solid implementation.
My suggestion would be to write up a series of Test cases (TDD yo!) which would test for the above process flows to make sure they’re working as expected and notify you in case of any changes from Google. 😉

There you have it, the little magic script to scrape off the list of Field IDs from your Google Forms page!

Share the love! 😀 Cheers!

Welcome to ÇøŋfuzëÐ SøurcëÇødë Tech Innovations !

Welcome to the ignition of Tech Innovations in Sri Lanka. We are a Sri Lankan based company that focuses on Technological Innovations and pushing the the boundaries beyond the Imagination.

CoverPic1

 

I have always been extremely passionate about Innovation and Creativity, and as a result of that I used to invent new things using whatever I learned. Even back in School Days I used to develop new inventions and win awards at competitions. That drove me to University, while studying Software Engineering, I always used to build new software tools y my self in whatever the theory that we learned during lectures, whereas most of the time I got too obsessed and I went ahead learning everything by myself about any specific subject. 
I have always dreamed of solving real life problems through new inventions and ideas, which is still the force that drives me forward. So as a result of my passion towards innovations and experimentation, I happened be developing so many softwares and apps which I had published to the public market. This sparked me the idea of initiating a startup of my own which I could drive through my passion. So here it is, Ladies and Gentlemen, Welcome to  ÇøŋfuzëÐ SøurcëÇødë Tech Innovations !
– Udara Alwis a.k.a. [CODENAME : ÇøŋfuzëÐ SøurcëÇødë]

Connect with us on our official Facebook page from the below link,

ProfilePic1 qrcode (2)

We already have a series of ongoing Innovative Development Projects and below are some of them.

Windows Phone App Development

We have already developed and published a series of Innovative and Creative Windows Phone Applications and published on Microsoft Windows Store which has gotten over 160,000 downloads worldwide along with a rapidly growing user base.

Over 30 Windows Phone Apps published…

Over 160,000 Users Worldwide…

CoverPic2

As we go on forward we are hoping to keep on developing more and more Innovative Windows Phone mobile apps. Also we are offering services to the public and business, so if you are in need of developing a Windows Phone app for your Company or Business, contact us immediately.

You can view all our Windows Phone apps by from below link,

462x120_WP_Store_blk

 

Sri Lankan Newspaper Cartoons Project

Welcome to “Sri Lanka Newspaper Cartoons” project, A centralized portal for showcasing and viewing all Sri Lankan Newspaper Cartoons published across the web. How we do this ? We have hosted cloud servers that crawls throughout the web searching for Newspaper Cartoons using our unique Web Crawling technologies and algorithms, where we will capture them and present to you by streaming from those servers.

Slide1

http://srilankannewspapercartoons.tk/

We simply bring all the Sri Lankan Newspaper Cartoons that are published across the web to one place and let you access them in a very easy to use, friendly, interactive way. We bring an ultimate experience of viewing Sri Lankan Newspaper Cartoons. Stunning visuals, interactive user experience blended together along with a faster and easier access and look up. “Sri Lanka Newspaper Cartoons”, we are empowering talented Sri Lankan Cartoon Artists to showcase and reach more audience easily and quickly and for the User, we are bringing an ultimate experience to view all Sri Lankan Newspaper Cartoons published across web, right from your fingertips….

Our fully automated system that is running behind this project was able to fetch over 11,000 Sri Lankan Newspaper Cartoons, while making this project the Sri Lanka’s first ever Largest Sri Lankan Newspaper Cartoons collection ever created.

Over 11,000 Newspaper Cartoons…

We are also opened up for Advertisers to publish ads on this project’s website and mobile app which are being used by over 1000s of users worldwide…

2 6 5 7

Download the Windows Phone app of our Sri Lankan Newspaper Cartoons project –

 http://www.windowsphone.com/en-us/store/app/sri-lankan-cartoons/83bfab81-883d-421e-8aba-c621bf67aee8

Here are some of the special features that we are bring you…

  • View all the Sri Lankan Newspaper Cartoons that has ever been published in Newspaper from one single place as you wish…
  • Stay up to date with the latest Sri Lankan Newspaper Cartoons and you can even view the oldest Cartoons ever…
  • Like or Dislike and Rate every single Cartoon…
  • Add Keywords and Tags to Cartoons as you like..
  • Easily share Cartoons with your Social Networks…

Sri Lanka Newspaper Cartoons Gallery

 

So join with us for the ignition of the next step of Sri Lankan Newspaper Cartoons entertainment…

Sri Lankan Memes Book

Welcome to the first ever largest Sri Lankan Memes collection ! The most awesome place for Sri Lankan Memes ! (^_-) Join with us today…

http://srilankanmemes.com/

CoverPhoto3

So are you Bored at Home ? or Work ? 9gag much ? a Memes fan ?
Welcome to the first ever largest Sri Lankan Memes collection !
Join Sri Lankan Memes Book today for the most awesome Sri Lankan Memes entertainment…

Welcome to the epic Book of all the Sri Lankan Memes Published on Facebook with over 15,000 Memes encountering Thanks to our unique intelligent Web Crawlers executing on Cloud… We are still on Beta level.. Stay tuned for our Official Release ! (^_-)

Sri Lankan Memes Book brings you the most amazing experience you ever had with Sri Lankan Memes entertainment. Such as,

  • View all the best Memes on Facebook
  • Even the Latest Memes, Recent Memes, Older Memes, most Liked or Disliked Memes, most Viewed Memes and so on..
  • View all the Memes under your favorite Meme pages
  • Vote Your favorite Memes
  • Daily updates of all the best Memes on Facebook
  • Like or Dislike Memes as you wish
  • Easily Share on Facebook or Twitter
  • View the best, Sri Lankan funny videos on Youtube…

You can also Watch the most awesome funny Sri Lankan Videos on Youtube via Sri Lankan Memes Book TV !
http://srilankanmemes.com/View_MemeStream_TV.aspx

News1

 
Visit our official Website –
http://srilankanmemes.com/

Facebook –
https://www.facebook.com/SriLankanMemesBook

Google –
https://plus.google.com/115627309497218505361
You can Subscribe to our Mailing list to receive daily awesome Sri Lankan Memes right into your inbox – http://goo.gl/oF6rTA

logo-base-test

Please drop a Like to the page if you think its awesome 😀 ! This is the first time such an app has been developed, so your support is highly appreciated. Please share with your friends as much as possible. 🙂

We have also opened up for Advertisers to publish Ads and Banners on our website and the mobile application. Please contact us if you are interested…

I Was There Map

I Was There Map, allows you to view all the places you have visited and your Facebook Check-ins. You can view all the amazing information statistics you have never known about those places and your check-ins. You will be shown all the interesting information about You and those check-ins like no other.
This project was developed based on ASP .net and C#. We have developed by our own custom API to grab data from Facebook for each and every user upon their permissions. We have invented a couple of innovative Algorithms in order to generate the relational data and information regarding the user and their check-ins information.

I Was There Map

Please check out more information from the below link,

https://theconfuzedsourcecode.wordpress.com/2014/04/03/i-was-there-map-is-ready-to-blow-your-mind/

Your Friends Map

Using ‘Your Friends Map’ you can view all your Facebook friends locations and hometowns in the world map. And it shows you a full list of all the locations and hometowns of your friends separately along with the number of friends in each location. This project was developed based on ASP .net and C#, where as I have used my own Custom Facebook API for pulling and streaming data from Facebook by the user’s permissions when the log in to the App.

We have invented a series of innovative algorithms to process the data and produce information regarding the user and their Friends, to populate statistical relationships.

Capture3

Please check out more information from the below link,

https://theconfuzedsourcecode.wordpress.com/2013/12/02/your-friends-map-is-ready-for-you/

Sri Lankan Photography Project

This project brings all Sri Lankan Photography that are published on Facebook to a one place, giving an ultimate experience to the user. With “Sri Lankan Photography”, we are empowering talented Sri Lankan Photographers to reach more audience easily and quickly and for the User, we are bringing a stunning ultimate experience for you to view all Sri Lankan Photography published on Facebook, right from your fingertips. This app provides so many special features, such as facilities keep in touch with the user’s favorite photographers as they wish, instantly viewing all the photo albums published on their public pages. Sharing those photography with user’s friends immediately on demand.

FlipCycleTileLarge

Please go to this link to view the full Article of our Sri Lankan Photography Project,

https://theconfuzedsourcecode.wordpress.com/2014/09/27/project-sri-lankan-photography/

Mobile In-App Advertising Project (Phase 1)

We have launched a project where we are giving In-App Advertising facilities to Advertisers and Companies in a series of popular Mobile apps developed by ÇøŋfuzëÐ SøurcëÇødë Tech Innovations.

Untitled

 

Please go to this link to view the Full Article of this project –

https://theconfuzedsourcecode.wordpress.com/2014/10/20/ultimately-boost-your-marketing-campaign-with-mobile-in-app-advertising/

Undisclosed Projects (under development) –

Intelligent Data Mining App (based on Social Media) Framework

A framework to fetch public Social Media data and coming up with analytical information through data mining…

Lanka Tuk Tuk Services Project

A new way of thinking about three wheeler services in Sri Lanka… (Coming Soon)

Mobile In-App Advertising Platform

A mobile in-app advertising platform for Sri Lankans…

 

So thats a little heads up about ÇøŋfuzëРSøurcëÇødë Tech Innovations initiation. So if any of you are interested of being a partner or an investor we would be very grateful to have you.

Till next time everyone.. 🙂 Cheers !

https://www.facebook.com/296474577210450

CoverPic1

Project: Sri Lankan Memes Book!

Welcome to the first ever largest Sri Lankan Memes collection ! The most awesome place for Sri Lankan Memes ! (^_-) Join with us today…

So are you Bored at Home ? or Work ? 9gag much ? a Memes fan ?
Welcome to the first ever largest Sri Lankan Memes collection !
Join Sri Lankan Memes Book today for the most awesome Sri Lankan Memes entertainment…

Visit our official Website –
http://srilankanmemes.com/

Facebook –
https://www.facebook.com/SriLankanMemesBook

Mobile App –
Available on Windows Phone and Windows 8 Store

Welcome to the epic Book of all the Sri Lankan Memes Published on Facebook with over 15,000 Memes encountering Thanks to our unique intelligent Web Crawlers executing on Cloud!

An entertainment gallery of social memes in Sri Lanka, aggregating across hundreds of Meme entertainment pages on social media using intelligent social media crawling bots, that I developed. This project included a Website, Mobile app and a Facebook page. 

Sri Lankan Memes are one of the fastest growing trends and entertainment in the country and among Sri Lankans all over the world.

This whole project, had two segments that I have developed,

  • The Mobile App, Sri Lankan Meme Reader
  • The Website, Sri Lankan Memes Book

Let’s dig in one by one,

Mobile App: Sri Lankan Meme Reader

“Sri Lankan Meme Reader”, a simple attempt to bring all the favorite and popular Sri Lankan Memes to one place. This app allows you to view all the Sri Lankan Memes which were posted on Facebook daily through some of the top Sri Lankan Meme pages. This is the first initiative of such attempt, bringing all the Sri Lankan Memes published across Facebook pages to a one single portal and giving an amazing experience of Sri Lankan Memes for people.

Sri Lankan Meme Reader has the capability of fetching all the latest Memes available instantly even when new Memes are posted on Facebook. Whenever a new meme has been posted to Facebook, you can instantly view them through this app as our Cloud servers stay awake to fetch every single one of them and bring to the user in real time. 😉

Launched on 21/06/2013, initially on Windows Phone App store and then to Windows 8 App Store.

Windows Phone App: http://www.windowsphone.com/en-us/store/app/sl-meme-reader/211662d2-c538-4c55-aef7-78d5c854dbef

Windows 8 Store App: http://apps.microsoft.com/windows/en-us/app/sri-lankan-meme-reader/deac8bd9-3547-4016-8d5d-a4934fbe5d76

Once the app is launched user can click on “Start” button where as it will take the user to the next screen where the user will be given to choose their favorite Meme Page or to View all of them at once.

After selecting that the user will be asked What do they want to view from the selected category, whether they want to view the Memes as a Photo Stream or as one by on.

Based on the choice Users will be able to stream through the available Meme Images, while sharing them as they wish instantly on Social Networks, and they can even view the original source of the Meme Image and find out more information about them.

Here’s a bit more on the Windows 8 Store App…

View thousands of Sri Lanka Meme Images right from your mobile phone with a very easy to use, attractive user interface, or on your Windows 8 Laptop at your convenience. You know, you can even share your favorite Sri Lankan Memes right from your phone with Facebook or any other Social media network. The best way to easily stay up to date with your favorite Sri Lankan Meme pages, is “Sri Lankan Meme Reader” App! 😉

Now on to the Website that I built using the same back-end servers…

The Website, Sri Lankan Memes Book: http://srilankanmemes.com/

“Sri Lankan Memes Book”, is the Web interface of the Sri Lankan Memes app whereas this website will be having a lot more extended features of the app. Streaming over 10,000 Sri Lankan Memes from our cloud servers, this web app will be giving an immersive experience of Sri Lankan Memes Entertainment like never before for the community.

The users will be able to view Sri Lankan Memes published across various Facebook pages right from this website just like the Mobile app as well as users will be able to Like or Dislike the memes that they view and rate them accordingly according to an automated ranking system.

Launched on 24/08/2014, we have developed the website in a way where, once the user navigates to the website home page, it will load random Meme Images as blocks on top and then at the bottom of it will be shown a button where users would be able to view more on the Meme Gallery section. By clicking that Users can view more Memes on Gallery as Image blocks, by clicking on them, user will be navigated to individual views of each of them.

The top menu bar of the site has 6 main sections –

  • Meme Stream – Allows user to navigate to Meme Stream page of the website, whereas user can seamlessly go through a long list of Memes while scrolling down the page. This feature offers 5 features, View Recent Memes, View Random Memes, Most Liked, Most Disliked, and Most Viewed and so on. According to the users choice the Meme Stream will be loaded.
  • Meme Stream TV – Watch the most awesome funny Sri Lankan Youtube Videos, bringing to your on Sri Lankan Memes Book!
  • Meme Gallery – Allows user to navigate to Meme Gallery, whereas user can seamlessly go through a long gallery of Meme Images.
  • Vote A Meme – Allows users to Vote for Memes Images and rank them, and they can also inform the Site admins to remove a certain meme if they are inappropriate.
  • Memes by Page – Allows users to view Sri Lankan Memes under each Meme Page available.
  • About Us – Contains information about the Sri Lankan Memes Book Project.

Then in the home page when the user scrolls down further the Meme Stream will start to appear, showing the memes in a Blog stream manner, whereas Users can instantly like them or dislike them and even Share them on Social Networking sites. In the right side bar all the latest memes will be displayed and quick links to other sections of the site. User can keep on scrolling down and more Meme images will automatically load as user keeps on going.

Sri Lankan Memes Book brings you the most amazing experience you ever had with Sri Lankan Memes entertainment. Such as,
– View all the best Memes on Facebook
– Even the Latest Memes, Recent Memes, Older Memes, most Liked or Disliked Memes, most Viewed Memes and so on..
– View all the Memes under your favorite Meme pages
– Vote Your favorite Memes
– Daily updates of all the best Memes on Facebook
– Like or Dislike Memes as you wish
– Easily Share on Facebook or Twitter
– View the best, Sri Lankan funny videos on Youtube…

A community base driven project where the users are given features to Like or Dislike the Memes they view and rank them accordingly according to an automated ranking system. View any Sri Lankan Memes based on your choice, whether it’s Recent New ones, most liked ones or most viewed ones and so on…

Share any of the Sri Lankan Memes right from the website itself with Facebook or Twitter…

Oh Yeah, it’s awesome!

“Sri Lankan Meme Reader” and “Sri Lankan Memes Book” is a very unique and innovative projects because this is the first time such initiative has been launched in order to provide a single portal for viewing and showcasing creative and entertaining Sri Lankan Memes while forming the largest Sri Lankan Memes collection database.

  • View thousands of Sri Lanka Memes right from your mobile phone with a very easy to use, attractive user interface
  • Easily stay up to date with your favorite Sri Lankan Meme community pages and experience the best of Sri Lankan Memes Entertainment…
  • A community base driven initiative where the users are given features to Like or Dislike the Sri Lankan Memes they view and rank them accordingly according to an automated ranking system.
  • View any type of Memes based on your choice, whether it’s your favorite Meme Page, Most Like or Disliked Memes, or even most viewed Memes and so on…
  • Implementation of State of the art Cloud Technology and a unique innovative offline data caching technology. So no more heating up your phone, as all the heavy weighted process executions are being leveraged to the Cloud!
  • Offline Image Caching facility saves you Data Connection usage and you can view pre-loaded Meme images even if your data connection is offline…
  • Share your favorite Memes right from your phone with Facebook or any other Social media network…

Encouraging our enthusiastic and creative Sri Lankan Memes community while showcasing their great talents and providing Sri Lankans an amazing experience of Sri Lankan Memes Entertainment is the key objective of this project.

“Sri Lankan Meme Reader”, a simple attempt to bring all your favorite and popular Sri Lankan Memes to a one place. This app allows you to view all the Sri Lankan Memes which were posted on Facebook daily, through some of the top Sri Lankan Meme fan pages.

Alongside with the mobile app a Web app has also being launched as “Sri Lankan Memes Book” with the same set of capabilities and a lot more features for the end user. This web app allows users to gain an extended experience of Sri Lankan Meme Reader app.

This is the first time such an app has been developed, so your support is highly appreciated. Please share with your friends as much as possible. 🙂 Thanks.

– Udara Alwis.
[ÇøŋfuzëÐ SøurcëÇødë]

Sri Lanka Newspaper Cartoons Gallery Web Launch…

So I hope you guys might have heard or remember “Sri Lankan Newspaper Cartoons” App which i released sometime back for Windows Phone ? which was the first and only Sri Lankan App for Viewing and Showcasing Newspaper Cartoons published across web. Well if you have no idea what the heck it is, you can check it out here 🙂

http://www.windowsphone.com/en-us/store/app/sri-lankan-cartoons/83bfab81-883d-421e-8aba-c621bf67aee8

So recently, I have taken it to a next level by releasing a Web Interface, using the same back end and porting it to the web as an Experiment.. 🙂
So here it is,

Sri Lanka Newspaper Cartoons – The Ultimate Centralized portal for Sri Lanka Newspaper Cartoons Entertainment

http://lkpapercartoons.confuzed-sourcecode.com/