Get Started with ElasticSearch

ElasticSearch is a highly scalable search engine with a REST API. It is open source and focuses on distributed real time search & analytics.

In this post, we will see how we can quickly get started with ElasticSearch on a Mac. ElasticSearch also works nicely on Windows machines.

Installation:

On a Mac machine, we can install ElasticSearch using Homebrew. Simply run the following command from Terminal:
   brew install elasticsearch

Verify installation:

ElasticSearch comes with REST API. By default it uses port 9200. So we can test the installation by visiting http://localhost:9200 from any browser. If everything is installed properly, the output should look similar to the following screen shot.

elasticsearch-1

Communicating with ElasticSearch:

We can talk to ElasticSearch using it’s JSON based REST API. We could use any HTTP client like curl in command line, Fiddler, etc. However, for this tutorial we are using Sense – a Chrome plugin. Sense is a developer console for ElasticSearch.

Basic Database operations:

We will use the following dataset of a few book titles and try out some basic database operations. First, we will create an index and insert the data into database. Then we will run a simple search, update and delete operation.

Title Author PublishDate Link
ElasticSearch Server Rafal Kuc; Marek Rogozin’ski Feb 21, 2013 http://www.amazon.com/ElasticSearch…
Mastering ElasticSearch Rafal Kuc; Marek Rogozin’ski Oct 25, 2013 http://www.amazon.com/Mastering…
ElasticSearch Cookbook Alberto Paro Dec 24, 2013 http://www.amazon.com/ElasticSearch…
The LogStash Book James Turnbull Dec 22, 2013 http://www.amazon.com/The-LogStash…

Creating an index and inserting data:

An index in ElasticSearch is similar to a database in SQL Server. An index contains documents. A document can be compared to a row in a traditional SQL database.

We can insert a document using the HTTP PUT method. Since it is the first entry and we didn’t explicitly create the index, ElasticSearch will automatically create the index for us. You can download the create script from here and paste into Sense console. Now if we hover the mouse over the command, it will show a green play button. Click the green play button to execute each insert command.

es - sense

Alternately, we could first create an index by sending a PUT request to the server with the index name. Then we could also create mapping (somewhat similar to a schema in SQL) to define fields. In such case, if the data contains additional fields, ElasticSearch will automatically update the mapping for us.

Search:

ElasticSearch provides a wide range of search functionalities. It has its won DSL (Domain Specific Language) to define queries. Right now, we are going to try out some basic search options.

At minimun, we can specify the search text as a query parameter. Lets run a search on books index for any match on “ElasticSearch”. As you see in the following screen shot, we can run the query from a browser. The response provides a number of additional information in JSON format including a “_score”. The value in score represents how closely it matches with the query.

es-search-1

In the same way, we could also get documents by ID. For example, a GET request to this URL “http://localhost:9200/books/book/1” would return JSON object representing the book with ID=1.

While we can run search queries from the browser, we should use the query DSL to utilize the full potential of ElasticSearch. This webcast in the product website nicely explains the query DSL. Lets run a match query using the DSL on the Sense plugin in Chrome:

es dsl hint

As we type for a search query, the plugin shows useful hints to complete the query. The following query will run a full text search on the title field in books index to find any match for the word “ElasticSearch”.

es fulltext

Update & Delete:

We use the same PUT command to update a document as we do for insert. ElasticSearch inserts a document if it doesn’t exist already; otherwise it updates the properties and increments the version no#.

update

To delete a document, we need to send a HTTP DELETE request. For example, the following request will delete the book with ID = 4

delete

What Next:

Now that we covered the steps on how to get up and running with ElasticSearch, I would like to focus on few specific features in the upcoming post. Some of my favourite features are options in FullText search, spelling suggestion and structure of the query DSL.

TouchToolkit October CTP now available

We just released the October community technical preview (CTP) of TouchToolkit at the codeplex project website. A new feature we introduced in this version is the option to add new primitive conditions & return types in the gesture language. While its still a CTP, it now has all the features targeted for the 1.0 version.

The key features of TouchToolkit are:

  • A domain-specific language to define custom gestures
  • Add new primitive conditions and return types in the gesture definition language
  • Supports Windows 7/WPF 4.0 Touch API, Microsoft Surface, Silverlight and TUIO
  • A record/replay module to simplify multi-touch testing and simulate multi-user scenarios
  • A test framework to write unit test for testing gestures (integrated with Visual Studio Test)
  • A visual effects framework to provide consistent visual feedback for touch interactions & gestures

In next few days, I will try to explain each of these features in detail.

Link a Silverlight project with an existing website/web application in Visual Studio 2010

Generally we create Silverlight projects using the templates available in Visual Studio. When we select a Silverlight application template, the template also creates a website project to run the Silverlight application. But what if I already have a Silverlight application and now want to link it to a website so that it will automatically drop the latest binary files in ClientBin folder whenever I build the application.

The current TouchToolkit project template only creates the Silverlight project and developers need to create the website and link the Silverlight project manually. I will fix it in the next version but for the time being, here are the steps you can follow to do the task.

Step 1: Go to the website/web project properties –> Silverlight applications tab

image

Step 2: Click the add button. It will popup the following window. Select the application project from the drop down menu.

image

Step 3: Click add and save the change (Ctrl+S). Now it will automatically create and fillup the ClientBin folder and also create a test page.

I hope this will help. I will try to do this as part of the template in next release of TouchToolkit.

Introducing TouchToolkit: a tool to simplify the multi-touch application development and testing complexities

I have been working on this for quite sometime now and finally (I think) its time to make the first CTP release. I am currently doing the last minute checks and hope to publish it within a day or two. The source code and some documentation is available in the project site at CodePlex.

TouchToolkit’s objective is to reduce the complexity of multi-touch application development & testing. It currently supports Silverlight 4 and WPF 4.0. Some of the key features are:

Gesture Definition Language (GDL): TouchToolkit provides a domain-specific language (DSL) to define custom gestures. While it comes with most of the common gestures out of the box, you can easily create your own custom gesture using few lines of code. Here is a code block that defines the “Zoom” gesture:

image

Device Independent: The internal architecture of TouchToolkit is not dependent on any particular device. So, unlike Microsoft Surface SDK you can use it for almost any touch enabled devices. The framework exposes a provider model for touch inputs. You only need to write a provider for the new device and your application will start working without any change. As proof of concept, we build providers for Silverlight, Windows 7, Microsoft Surface, SMART Tabletop*, AnotoPen and TUIO* protocol.

archi-overview.png

Simulator: Multi-touch devices are generally expensive. As a result developers often need to share one device and do majority of their development work from regular desktop computers. It becomes inconvenient when they frequently need to move back and forth. Another common challenge is to test multi-user scenarios. Every time you need to debug/test a multi-user multi-touch interaction, you need people to help you out. To overcome these challenges, TouchToolkit provides a recorder tool that you can use to record data from actual device and later playback in your regular desktop computer through a virtual device. It can also run multiple recorded touch interactions in parallel to simulate multi-user scenarios.

Write Unit Tests to Validate Touch Interactions: Finally, we need automated tests to maintain quality of the software as its not practical to test everything manually every time we make a change. However, gestures often include complex set of touch interactions that needs to execute in an asynchronous fashion. Writing unit tests using existing frameworks (i.e. NUnit, Visual Studio Unit Test) that involves asynchronous code execution and device virtualization is a bit complicated. TouchToolkit simplifies this to a great extent by encapsulating the device virtualization and asynchronous gesture validation logics. Here is a sample Unit Test code to validate the “Zoom” gesture:

image

The above code is actually a Visual Studio Unit Test and you can run it as part of continuous integration system or manually just like any other VS unit test. The GestureTestFramework.Validate(…) method handles the complexities of simulating a virtual device, getting the recorded data etc.

I hope you get some idea about TouchToolkit. Please let me know you thoughts. I will soon start writing about each of the sections in detail.

Writing Unit-Tests for Asynchronous Methods

In ASE Lab, we always try to follow Test Driven Development (TDD) approach. As part of my research, I have been developing TouchToolkit for sometime now. Its aim is to simplify the multi-touch application developer process from a number of key areas including cross-device support, DSL for defining new gestures, automated acceptance test support for touch interactions, tools for visual feedbacks, and so on.

Recently, I started developing the storage module for TouchToolkit. It will allow developers to record user interactions from live applications that later can be used for debugging, automated tests, live demo, tutorials, etc. Since the toolkit can be used in applications built on Silverlight, WPF or XNA, I needed to consider different storage medium (i.e. isolated storage for Silverlight, local file system for WPF) for local cache. But the challenge I am facing here is to write UnitTests for the remote-storage module that uses asynchronous web service calls.

Now, lets think about a scenario where I need to test an asynchronous get() method. First I need to save an unique value via another asynchronous call and wait until it completes the process. Then invoke the actual get() method and wait until the data is received from server to validate with the expected values. While discussing this issue in lunch break, the first suggestion I got was to wait for callback for x amount of time using Thread.Sleep(). Now, obviously this is can give false alerts and can significantly increase the time to run the tests.

The interesting thing is, it is not necessary to have the validation codes (i.e. Assert) inside the test method. It can be inside the callback method too. And using the Lamda expression we can have these callbacks as inline functions. The Visual Studio Unit Test runner is actually smart enough to figure that out. So, here is my final code for testing one of the get() methods:

[TestMethod()]
public void GetGestureTest()
{
// Data set
string userName = GetUniqueGestureName(),
gestureName = GetUniqueGestureName(),
projectName = GetUniqueProjectName(),
gestureData = "GestureData-" + Guid.NewGuid();

StorageManager target = new StorageManager(userName);

// 1. Save the test data in storage
target.SaveGesture(projectName, gestureName, gestureData, (errorOnSave) =>
{
if (errorOnSave == null)
{
// 2. Get the data from storage and validate
target.GetGesture(projectName, gestureName, (projName, gesName, gesData, errorOnGet) =>
{
if (errorOnGet == null)
{
// 4. Check the validation result
Assert.AreSame(projectName, projName, "Actual project name doesn't match with the expected value");
Assert.AreSame(gestureName, gesName, "Actual gesture name doesn't match with the expected value");
Assert.AreSame(gestureData, gesData, "Actual gesture data doesn't match with the expected value");
}
else
{
Assert.Fail(errorOnGet.Message);
}
});
}
else
{
Assert.Fail(errorOnSave.Message);
}
});
}

How to import your blog content from WordPress to Blogger

WordPress is great for blogging and provides many useful features. But Blogger is also getting better and provides additional features like free domain mapping that costs additional fees in WordPress and also limits your control over the domain.

So, if you ever want to move your blog from WordPress to Blogger, you might find that quite challenging. Although you can easily import from most of the blogging sites into WordPress using the built-in feature (Tools –> Import), the other way is not so smooth. Because Blogger can’t process the WordPress eXtended RSS or WXR file.

Here is a nice website called wordpress2blogger that converts the WXR file into Blogger friendly format. Now, you can simply download the WXR file from WordPress (Tools –> Export), convert it using this site and then, import using Blogger’s import tool (Settings –> Basic, Blog Tools –> Import blog).

Implementing your own base class for user controls in Silverlight 2

The objective is to create your own base class for user controls to implement application related features and at the same time also use the features provided by the Visual Studio (i.e. auto generate a partial class that initialize all UI elements). The process can be described best with three projects:

For the sake of simplicity I created a simple UserControlBase class extending from the UserControl class. This class can contain the common methods and properties as needed for your application. Here I have added some dummy methods and properties.

namespace BaseLibrary
{
public class UserControlBase : UserControl
{
public int Id { get; set; }
public void DoSomeThing()
{
//…
}
}
}

Then, lets create a TestControl class and a TestControl.xaml in the class library where we like the have the custom controls:

namespace CustomControls
{
public partial class TestControl : UserControlBase
{
public TestControl()
{
InitializeComponent();
}

}
}

Now, here is the trick. Look closely to the xaml. Instead of regular UserControl, we used our own base class. To do so, we also have to include the namespace.

UserControlBase x:Class=”MyControls.TestControl”
xmlns=”http://schemas.microsoft.com/client/2007″
xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
xmlns:bl=”clr-namespace:BaseLibrary;assembly=BaseLibrary”
Width=”150″ Height=”50″>

In this way, Visual Studio also generates the partial class properly. But there is one side effect: the Visual Studio will not be able to show you the UI preview in designer. I haven’t found any work around yet.

Update:

I forgot to add the reference in AsseblyInfo.cs file of BaseLibrary project. Once you add the following reference the Visual Studio will render the UI properly. Thanks to Michael for pointout the issue.

[assembly: XmlnsDefinition("http://schemas.microsoft.com/client/2007", "BaseLibrary")]

I have also updated the sample code.
Download Sample Code

Hosting Silverlight 2.0 in IIS

If you host Silverlight 2.0 application in IIS you might find that the silverlight object is not loading. If that’s the case, first thing you should check is the MIME type list. By default, the Silverlight package extension *.xap is not included.

To resolve that, go to IIS Manager, properties of your web site and check the “Http Headers” tab.

1

Then click the MIME Types button. You should see a list as bellow.

2

If the .xap extension is not in the list then click New… and add the following entry:

Extension: .xap
MIME type: application/x-silverlight-app

Now, check your site. You should get the silverlight objects properly.

kick it on DotNetKicks.com

Converting Silverlight 1.1 Alpha projects to Silverlight 2 Beta 1

Just installed the runtime and Silverlight Tools for Visual Studio 2008. You can find the necessary tools here. The beta 1 now has a good set of controls including Data Grid, cool!

To run the Silverlight 1.1 Alpha projects in Beta 1, you need to manually modify few things.

First, if you try to open the old projects in VS 2008 after installing the latest bits, you will get the following error message:

“Unable to read the project file ‘… .csproj’. … The imported project “C:Program FilesMSBuildMicrosoftVisualStudiov9.0SilverlightMicrosoft.Silverlight.Csharp.targets” was not found.”

Just open the project file with notepad and replace the node value.

Old:

New:

Save the project file and now you can load it in Visual Studio. Now that you have the projects opened, if you expand the Reference node you will see this:

image

You need to delete agclr, system.silverlight and system.Xml.core from the reference list. Then add the new assemblies: System.Windows, System.Windows.Browser, System.Windows.Controls and System.Windows.Controls.Extended.

image

Ok, lets do a rebuild now. Don’t worry if you get a long list or errors. Here is a quick list the things you may just do find and replace:

  • There is a PointCollection class now. So if you have List then you can change it to PointCollection
  • For Keyboard events (i.e. KeyUp, KeyDown…) replace KeyboardEventArgs with KeyEventArgs
  • System.Windows.Browser.Serialization is moved, so you need to block //using System.Windows.Browser.Serialization
  • Previously, in xaml we used to write x:Class=”SilverlightApplication1.Page;assembly=ClientBin/… .dll”. Now we don’t need to mention the assembly part. Just write x:Class=”SilverlightApplication1.Page”
  • For mouse events (i.e. MouseDown, MouseUp…) replace the EventHandler with MouseEventHandler

Now, look into the web service. Lets assume you have a service named MyService and you have added the necessary web reference. The Silverlight 1.1 Alpha would generate a MyService class and to call any method asynchronously you would probably write something like this:

void Test()
{
MyService service = new MyService();
AsyncCallback callback = new AsyncCallback(FooCallback);
Core.BeginFoo(parameter, service);
}

FooCallback(IAsyncResult result)
{
MyService service = result.AsyncState as MyService;
service.EndSyncCommand(result);
}

In silverlight 2 beta 1, you need to do this slightly differently. It will generate a MySerciceSoapClient class. So, you may write something like this:

void Test()
{
MyServiceSoapClient service = new MyServiceSoapClient();

service.FooAsync();

service.FooCompleted += new EventHandler(service_FooCompleted);
}

void service_FooCompleted(object sender, FooCompletedEventArgs e)
{


}

Ok, that’s all I had to do to make my project compile properly in beta 1. Please note that there may be better ways to do this.

Calculate you project health from 7 questions

David posted this great list of 7 questions. Each of them are most essential for any project. I learned some of them from my experience in the hard way. Here is the checklist that every project manager should check:

1. Are you using a version control tool?

2. Do the people on your team know how to use version control to ‘time travel‘ through project versions?

3. Do you have good test coverage?

4. Can you build the project, run the tests, and update project documentation with one command?

5. Does the team get an automated email within 15 minutes of an accidental broken build?

6. Does that email contain the changeset from version control that broke the build?

7. Does your team have to work late nights whenever you have to release your project into production?

If you answered ‘no‘ to any of the first 6 questions, changes are you answered ‘yes‘ to the seventh one.

Read his post here.