A look at INDEX & MATCH functions

In my earlier post, I looked into the VLOOKUP function and its uses. You may also find more examples this post by Colin Legg. Also, Charles Williams had a nice entry on how VLOOKUP performs, and the difference between exact search and approximate search in terms of algorithm.

In this post, I will focus on the INDEX and MATCH functions and how, used together, they can do the same things as the VLOOKUP functions and sometimes more.

Basics

Introduction

In the following we will use the following reference table, from which we want to extract price information, given some product number.

image

Suppose we want to get the price corresponding to product number XT987. We will first look for the value XT987 in the Product Number column and will get its relative position within the Product Number range. This is where the MATCH function comes into play.

In our case the formula will look like this:
=MATCH(“XT987”, product_number, 0)
We ask Excel to look for the value XT987 in the Product Number column, performing an exact search (that is what the 0 value stands for.) Excel will return the value 2

.image

Once done we will retrieve the price from the same row in the Price column. This is where INDEX comes into play.

In our case the formula will look like this:
=INDEX(price,2)
We ask Excel to return the value found at the 2nd place in the Price column.

image

This is of course possible to do this in one single formula:
=INDEX(price, MATCH(“XT987”, product_number, 0))

Why use INDEX+MATCH instead of VLOOKUP?

Nesting a MATCH function in an INDEX function does not look as nice or readable as a VLOOKUP formula. So what is it good for?

1. Since we can provide a lookup column (product_number) and a result column (price)  independently, we are not limited to using data with the lookup column on the left-hand side of the result column.
This is similar to what the LOOKUP function (not VLOOKUP, not HLOOKUP) can do, except LOOKUP requires your data to be sorted.

2. Actually, we are not even limited to using columns, or ranges. The formula would not change if the lookup and result column were both rows, or one in row one in column. We can also use a formula with hard-coded arrays, if we want:
=INDEX({12,569,321},EQUIV(“XT987”,{“XR134″,”XT987″,”ZL988”},0))

3. The match function supports one additional option for approximate searches, which allows, once again, for more flexibility.

4. Finally, and this is probably the most powerful feature, INDEX returns a reference and not a value. That means, if you used a range for your result column, then you will retrieve a cell reference, which means you can use the result of this formula with functions that take ranges as input values, or with range operators.

The MATCH function

The Syntax

The syntax of the match function is very simple, and will probably remind you of the VLOOKUP syntax.

=MATCH(key_value, lookup_range, lookup_type)

The lookup_value is the value you are looking for. In our example, this will be “XT987”.
The lookup_range will be the range where you are looking for the value. In our case, this will be the column Product Number.
The lookup_type parameter can take the values –1, 0, or 1. When this parameter has the value 0, Excel will perform an exact search. When this parameter is either –1 or 1, Excel will perform an approximate search (more on that later.)

The return value is the position within the lookup range (or array, since MATCH also accepts arrays), where the key value was found.

Lookup types

The look up type parameter accepts 3 values: –1, 0, or 1.

0 means the search will be exact. This corresponds to the search behavior of the VLOOKUP function, when we choose the value FALSE for its last parameter.
1 means the search will be an approximate search, and requires the data to be sorted in ascending order. This corresponds to the default search behavior of the VLOOKUP function, or when we choose the value TRUE for its last parameter.
-1 means the search will be an approximate search, and requires the data to be sorted in descending order. This has no equivalent in the VLOOKUP function.

Suppose your data looks like this and you want to search through the Part Number column.

The MATCH function will return the following values, depending on the value provided for the type parameter.

image

For the type –1, Excel expects data sorted in descending order and, to keep it simple, will return errors most of the time. For the type 0, the value will return an error if the searched value could not be found. For the type 1, Excel will return the last position of the value in the column, or if not found the position of the last available value that is “smaller” than the searched value.

So a value of 0 for the parameter type will return #N/A for a non existing value.

Now, if the data were sorted in descending order

image

the option –1 would return the following results.

image

It will return the first value in the list, that is “larger” than the searched value.

The INDEX function

The Syntax

The INDEX function is actually really complex, and would deserve its own post. Basically, it will take up to 4 parameters, and has two syntax forms
=INDEX(matrix, r)
=INDEX(matrix, r, c)
=INDEX(matrix, r, c, range_index)

This function allows you to fetch a reference to the cell on the r-th row of the c-th column of the input matrix.  To keep things simple, I will not talk about the last parameter in this post.

One important thing to note, is that the function also allows you to fetch a whole row, a whole column, or  even a whole matrix.

For example, the following formula
=INDEX(matrix, 0, 1)
will retrieve the first column of the input matrix.

Optional parameters

Officially, the index function must have at least 2 arguments. However, we may skip any argument past the matrix  and the function will still return valid results. For example,
=INDEX(matrix,,1)
(note the two commas) will return the same result as
=INDEX(matrix,0,1)

The following function
=INDEX(column,)
does the same as
=INDEX(column,0)
and will return our input column.

Multiple lookup with INDEX & MATCH

Suppose we have the following table that tracks the price of different products depending on the year.

image

Now we want to re-use this information in another table. That is, we want to fetch the relevant price, based on some reference to a year value (key_year), and a reference to a product number (key_product).

We will first use the MATCH function to find the first and the last row number for the relevant year, in the Year column.

Let us define a named calculation for the index of the first row where key_year can be found
first_cell_index =MATCH(key_year,year,0)

Let us define a named calculation for the index of the last row where key_year can be found
last_cell_index =MATCH(key_year,year,1)

Given these formulas, and using the fact that INDEX returns cell references, we are now able to fetch the relevant Product Number and Price ranges.

Let us start by using our named calculations in the index functions, to define two new named calculations.
For the first relevant cell in the Product Number range
first_cell_product = INDEX(product_number, first_cell_index )
and for the last cell:
last_cell_product = INDEX(product_number, last_cell_index)

The relevant range can then be constructed by using the range operator “:. This gives the following formula:
range_product  =first_cell_product : last_cell_product

Similarly, we can define the relevant range for the price
first_cell_price = INDEX(price, first_cell_index )
last_cell_price = INDEX(price, last_cell_index )
range_price = first_cell_price : last_cell_price

Using the above named calculations and applying an INDEX + MATCH formula again, we can now match the relevant information for the given key_product.
=INDEX(range_price,
MATCH(key_product, range_product,0)
)

Advertisements

A look at the VLOOKUP function

Since Bill Jelen (MrExcel) launched a VLOOKUP week, I figured it was a good occasion to take a look at this function and compare it with the INDEX/MATCH alternative. In order to prevent my post from being too long, I decided to dedicate this entry to the VLOOKUP function only.

Basics

What the function does

What is the VLOOKUP function about? It allows you to search for a key in a table and find the corresponding value in another column on the corresponding row.

Suppose you have the following reference table for product prices and you want to look for the price of product XT987.

image

First, you would look for the value XT987 in the first column and find the row where this value is.

image

Then, you would move to the second column (the price column) to retrieve the desired value.

image

The syntax

This is what the VLOOKUP function does for you. The function has two syntax forms and comes in two flavors.

=VLOOKUP(key_value, lookup_table ,result_column_index, approximate_search)
=VLOOKUP(key_value, lookup_table ,result_column_index)

key_value is actually the value we are looking for. In my example, this would be XT987.

lookup_table would be our reference table.

result_column_index would be the index our price column (that is, 2).

approximate_search is an option specifying how Excel will look for the value. This parameter tells Excel to perform an approximate search or not. It takes the values FALSE or TRUE and is optional (see the second syntax form), the default value being TRUE. Note that the option TRUE also requires you to sort your table in ascending order (of the key value) for the function to work properly.

Usually FALSE is the option you want to use. Let us see what the difference is.

Approximate or exact search

The case for an exact search

Suppose you have a product number Y5678 and want to find the corresponding price in your table. Now, as you may have noticed, this product is not in our table. Depending on the value you chose for the last parameter, you will get different results.

For an exact search, the formula will look like this:

=VLOOKUP(“Y5678”, lookup_table ,2, FALSE)

and Excel will return #NA since it could not find any corresponding value.

For the  approximate search, the formula will look like this:

=VLOOKUP(“Y5678”, lookup_table ,2, TRUE)

or like this:

=VLOOKUP(“Y5678”, lookup_table ,2)

In that case, Excel will go through the product numbers and will look for Y5678 or the last value that is “smaller” than Y5678. In our case, XT987 is the last value in our Product Number column that comes before Y5678 in alphanumeric order.

Excel will thus return 569, which may not be the result you expected.

The case for an approximate search

Interval searches

Approximate search is well suited for interval searches.

Suppose you have the following table.

image

You want to associate the corresponding shipment fee with each product.

Suppose the shipping price of a product is based on its weight.

If a product weights less than 5 kg shipment will cost 5€.
If a product weights no less than 5 kg but less than 20 kg shipment will cost 20€.
If a product weights no less than 20 kg but less than 90 kg shipment will cost 45€.
If a product weights more than 90 kg shipment is not available.

Then you could use the following lookup table to map the min weight to the shipment fee:

image

Then the following equivalent formulas will give you expected results

=VLOOKUP(weight, lookup_table_shipment, 2)
=VLOOKUP(weight, lookup_table_shipment, 2, TRUE)

When applied to the table, this will give the following results.

image

Let us take the product ZLM988 as an example. The weight of this product is 86 kg. Excel will look for the last key (the minimum weight) in the lookup table, that is equal to or smaller than

86. The corresponding entry is 20. Excel will then return the shipment on the corresponding row and will thus correctly return 45.

Partial searches

This problem comes from a discussion group on a well-known professional networking platform. The original poster was looking for a way to categorize products based on their product number. Now this series number would start with letters just as in my previous examples. Now the part number could have between 1 and 3 characters (I guess, maybe more in the future) and the list of part numbers would be updated regularly. People provided some smart sophisticated solutions, but none had come up with most simple one. Which is to use the VLOOKUP function.

Now, let us take our last table, and categorize these products based on their part number.

image

For this, we will use the following lookup table.

image

Now, one of the following formulas will do the job:

=VLOOKUP(product_number, lookup_table_categories, 2)
=VLOOKUP(product_number, lookup_table_categories, 2, TRUE)

and will return the expected results:

image

Be cautious though. Since we are using an approximate search, we may still have the issue that some part number is missing in the lookup table, thus leading to an incorrect result. Suppose you add another product XSM654 to the data table, whose part number XSM corresponds to the Electronic Devices category. If this part number is not available in our lookup table, the last entry whose value precedes XSM654 in alphanumerical order, is XT. Excel will legitimately return an incorrect category (Books).

You will thus have to implement some formula, to catch these sort of errors.

Note though that you can also use VLOOKUP to retrieve the part number from the lookup table. In our last example, the formula
=VLOOKUP(product_number, lookup_table_categories, 1)
will return XT, when the product_number equals XSM654.

This allows to test whether our lookup table has been correctly maintained or not. For example, you could use the following formula to test whether or not product_number actually starts with the part number returned by the VLOOKUP function:
=SEARCH(VLOOKUP(product_number,lookup_table_categories,1),product_number) = 1

Streaming Office 2010 and Click-to-run

I realize my comment is about 2 years late, but I just had the visit of salesmen trying to sell me a better internet connection. Since I have had no broadband limitation whatsoever in the recent months given the actual content offer, I declined. Furthermore these guys could have been a pair of WLAN free-riders attempting some social engineering scheme.

This, however, led me to the thought I may have to upgrade my connection, should the streaming of applications become prevalent.

It then stroke me that Microsoft Office 2010 already offers streaming capabilities. Microsoft called their technology Click-To-Run, and is already available for Office 2010.

I recently had to re-install Office 2010 and first chose the option. The installation process is shortened to nothing and you can start using your applications straight away. That is absolutely fantastic in principle, and also for practical purposes if you do not use any out of process add-ins.

Unfortunately, I had to re-install the Click-To-Run version with an old-school full install, because it would not work with Powerpivot, for example. I hope Microsoft will find a way to make this sort of installation work with all these add-ins that are out there, including their own. Otherwise, the technology may well fall into oblivion.

Common queries : Grouping / aggregating over several columns.

I was looking for common SQL queries which could be used as a battery test for business intelligence tools.

I found quite a bunch of examples on the following website http://www.artfulsoftware.com/infotree/queries.php and decided to take a look at them.

Two query examples had the same concern.

1). Suppose you have the following table that summarizes squash court bookings.

image

You want each member of the club to pay half the fee, if he only appears in one column (member1 or member2), or to pay the full price, if he appears in both columns.

2). Suppose you have the following teams and games tables.

image

image

You want to get this output:

image

For each team, you want to count the number of games played (if the team appears in the column team1 or team2 from the games table) and number of games won (if the team appears in the team1 column and score1 > score2 or it appears in the team2 column and score1 < score2 ), lost, drawn and the number of goals scored, …

Writing a SQL query to do this is not hard. On may go through a UNION operation and then summarize the data accordingly, as proposed the site mentioned above, but I find the following query to be closer to the natural formulation of the problem:

SELECT
T.name
, SUM(CASE
WHEN T.id = team1 AND score1 > score2 THEN 1
WHEN T.id = team2 AND score1 < score2 THEN 1
ELSE 0
END) AS WON
, SUM(CASE
WHEN T.id = team1 AND score1 = score2 THEN 1
WHEN T.id = team2 AND score1 = score2 THEN 1
ELSE 0
END) AS DRAW
    … and so on
FROM teams AS T
LEFT JOIN games AS G
ON T.id = G.team1
OR T.id = G.team2
GROUP BY T.name

If you had no separate table for the team, you would have to replace the team table in the previous query with a sub-query. Something along the line :  (SELECT teamname1 FROM games UNION SELECT teamname2 FROM games).

Using Excel, building such a summary table is easy, provided you have the list of team names you are interested in. Unfortunately, there is no easy way to retrieve a list of unique values from both columns out-of-the box. Of course, this can still be done with some VBA to automate the process, but this is not native. Please note the spreadsheet from Google Doc offers the nice UNIQUE function, which enables to get a list of distinct values from a range, but the function will not work for values spread across 2 columns. Pivot tables will not work in that case either, and they do not support many-to-many relationships anyway. This is true for the standard pivot table, this is also true for Powerpivot V1. (V2 will offer some support for overriding a relationship with an expression.)

Now, what I would like to see in a BI system is the ability to support such queries, and as well support for building such queries with some simple drag and drop operation. If you know of a tool with these features, please let me know.

%d bloggers like this: