Friday, August 15, 2008

Javascript Search / Sift Tool v1.0

Javascript Search / Sift Tool v1.0


How to use the demo: You will see various blocks with text. Inside each of these are sets of attributes that you may search on. The datasets behind every div is just below. Have a look and that will give you a sense of how the sift tool works.

So, allow me to give you a couple of starting searches. After looking at the datasets below, type the following. At each step, take note of what the sift tool is doing:

  1. mammal
  2. hit backspace until just "m" of "mammal" remains.
  3. hit the clear button.
  4. now, type "wings+ape" (nothing, but it does find items as you type).
  5. now, change the + to a | (a pipe). Interesting results! Here is where things get fun. You got any field that matched wings and any field that matched ape. So any bird or insect with wings was returned.
  6. now, leave whats in step 5, but anywhere in the search box, type "!" the exclamation point. Everything else is returned.
  7. play around with it.

Water Bear||Insect||Touch||Feet

Download: Version 1

What's coming in version 2:
  1. Support for buckets of objects
  2. Support for keeping track of counts of what remains
  3. Support for search convenience buttons

About the sift tool and logic:

Sifting is a little different than searching. Just a little. I called this "search / sift" so web searches would find it. Nobody is looking for "sift". With sifting, all of the data is already available in memory, be it a whole bunch of P tags, divs or whatnot. In this blog, I use the words "sift" and "search" interchangeably even though this really does "sift". I have 10 years of saying "search" so its hard not to say it.

I was recently challenged with developing a UI that returned a sizable amount of searchable data, on the order of say, 600 items, each with 10 searchable parameters, giving us 6,000 pieces of data.

While this is not, on the whole, a lot of information... there are plenty of uses for such an application when the lists are finite and not generated by users (but even then this can be modified to suit such a purpose).

I'll assume you know how to develop interfaces, and will also assume that you have a text box somewhere. My examples will be written using mootools v1.11, but really there are only a couple of moo-related features I'm using. It won't take much to move this to version 1.2.

Overview of interface:
  • Supports OR with "|" pipe.
  • Supports AND with "+" plus.
  • Supports NOT with "!" exclamation point.
  • Searches are always wildcards
  • Allows several ways to search: by enter key, by a search button and by searching while the user types. Although the code supports all three... My example shows the "search while you type method".
  • Searches are easy to customize, and have the option of returning a value or not.
  • Note: You cannot do And with Or in the same sift.
Overview of code:
  • For our purposes, I am coding using JSON, with mootools framework, and have segmented my methods inside of an "app" variable, which is the only global variable I need. That is why you will see references to "app".
  • Also, I am not using any enclosures, since I did not find it useful for this task. (beware "overuse of cool techniques").
  • Testing only on IE for the moment (but Firefox for me is always faster here), I've got this optimized to do 6.09 milliseconds per object searched.
  • This was tested on 2.79 Ghz CPU with 2 G ram. The faster the machine that the user sifts with, the faster the sift.

A couple rules beforehand:

  • Don't send an object in the dom to your function or assign it to a variable, since this can cause memory leaks in IE. Instead, just call it by reference.
  • Don't use mootools elements when you are creating them by the hundreds, as the garbage collection utility in mootools simply cannot handle that amount of information and will cause your browser to think its running slow scripts when onUnload() is called.
  • Don't forget commas at the end of every function in JSON, but not the last function. leave that comma-less. Since my code is broken up, I've taken them out. See the js file for the entire structure.

I apologize for the formatting issues below. This html editor keeps eating up my pre tags. For a good look at the code, just download it here: Download: Version 1

Step 1, setting up the tools :

  • setupSiftTool. I call this when the view comes into focus the first time. This could be your enclosure if you wanted.
  • I set up my box called "siftBox" by finding it in the DOM by reference. In this case, I did not inject it into the DOM but i could have. You can easily just add the events to your text box old-school style if you want.
  • "clearButton" is a button right next to the search box. This does exactly what you think it does.
  • cArray is a shadow array. It is there because of the fact that it is always faster to iterate an array in memory then it is to comb the DOM. We will use this to keep track of what has already been found so that the search gets faster the more you search.

initialize : function(){
this.arrayOfObjects = $$(".siftItem");
$each(this.arrayOfObjects, function(item, i){
item["data"] = $A(item.getAttribute("data").split("||"));

setupSiftTool : function(){
Here you can decide, based on the number of expected items how you want the
search to be triggered. I like to use a search box when the data set gets to large,
which means the user has to hit enter or click the box.
Until that time, I let the search work continuously as they fire the keyup event.
You can use if(event.keyCode === 13) to ensure it happens with enter key.
$("siftBox").addEvent('keyup', function(event){
$('siftBox').value = "";
this.cArray = [];

Step 2, the iterator :

  • You'll notice here how I partition my logic. All methods should be tight, small, and serve a particular purpose. Like this blog, I keep things short and sweet if I can.
  • captureKeys: responsible for iterating over a set of objects, and this is where I assume you are in-memory. I call it simply, arrayOfObjects
  • forEach is a mootools paradigm equal to a loop over an array. It is bound to "this", or app.
  • testFailAttribute is coming in step 3. This is the core logic.
  • If you want to clean up this code so that it doesn't call the same logic in both the if and the else statement, then go ahead. I have it this way for clarity.
  • Note: arrayOfObjects I assume you will build up yourself. In my case, I do it by iterating over the dom to find Divs. These are all Divs.
  • this.Tm, a test-match boolean. Explanation in step 3.

captureKeys : function(){
app.arrayOfObjects.forEach(function(obj, index){
return false;
app.cArray[app.cArray.length] =;
app.cArray.remove(; = (window.ie6) ? "inline" : "block"; // ensures a fix to ie6 margin impl.
this.tM = null;
this.isAnd = -1;
this.isOr = -1;
this.isNot = -1
this.firstStyle = "";
this.secondStyle = "";
this.returnNeg = false;
this.returnPos = true;

Step 3, the core logic:

  • testFailAttribute. Here you are trying to see if an object does *not* match.
  • this.Tm, a test-match boolean. We use this to ensure that we're not calling the first section of testFailAttribute every time the user searches.
  • isAnd and isOr tell us the jist of what the user is trying to do. At this time, I'm not letting them mix the two.
  • doShowObj is an associative attribute, one that tracks the lifecycle of the object. The reason this is here is so that the UI can search as the user types. In this case, I want to use the enter key, but it supports searching while typing as well.
  • testFailMatch is a list of items that the conditions must meet or the test fails.
  • escapeReqExp comes from mootools framework. You'll need to be careful not to process certain characters.

testFailAttribute : function(obj,box){
this.firstStyle = "inline";
this.secondStyle = "none";
this.returnNeg = false;
this.returnPos = true;
this.isNot = $(box).value.indexOf("!");
this.isAnd = $(box).value.indexOf("+");
this.isOr = $(box).value.indexOf("|"); //must be one or the other.
if(this.isOr === -1) this.tM = $(box).value.toLowerCase().split(/\+/g);
if(this.isAnd === -1) this.tM = $(box).value.toLowerCase().split(/\|/g);
if(this.isOr !== -1 && this.isAnd !== -1) return;
obj["doShowObj"] = 0;
this.tM.forEach(function(m, index){
index = index+1;
m = m.escapeRegExp();
if(this.isOr !== -1){
if(index === 1 && obj["doShowObj"] > 0) obj["doShowObj"]--;
if(obj["doShowObj"] > 0) obj["doShowObj"]--;
if(this.isNot !== -1){
this.firstStyle = "none";
this.secondStyle = "inline";
this.returnPos = false;
this.returnNeg = true;
if(this.isOr === -1){
if(obj["doShowObj"] === this.tM.length){ = this.firstStyle;
return this.returnNeg;
}else{ = this.secondStyle;
return this.returnPos;
if(this.isOr !== -1){
if(obj["doShowObj"] > 0){ = this.firstStyle;
return this.returnNeg;
}else{ = this.secondStyle;
return this.returnPos;

Step 4, the attributes to sift on:

  • testFailMatch. This is a list of attributes on the object itself (in my case, dom attribtues added to the div.) I cut it down to one, but just add as many as you want with "&&".
  • Here you can be creative. If you want the sift to eliminate items based on lets say, a displayName, then add that. But you can be clever about it and add convenience for the user, by also adding other attributes. Lets say you have a dataType attribute. If the user searches for "date" you can surprise them by returning anything with "date" in the name, or any item that is a date dataType, even if "date" is not in the name.

testFailMatch : function(obj,testMatch){
testMatch = testMatch.replace("!","");
if(!(obj["data"][0].toLowerCase().test(testMatch)) && !(obj["data"][1].toLowerCase().test(testMatch)) && !(obj["data"][2].toLowerCase().test(testMatch)) && !(obj["data"][3].toLowerCase().test(testMatch))){
return true; // meaning, if the item passes none of these tests, hide it an add it to the array.
return false;

No comments: