Equivalent of VLOOKUP in DAX – Part II – using TOPN

In my previous entry, I presented the LOOKUPVALUE function. In this article, I will explain how you can use the TOPN function to mimic the behavior of VLOOKUP when doing “approximate” searches.

Ken Puls’s article presents a typical use case for this pattern. Let me start with briefly sketching his problem and describe how he solves it.

The effective tax rate problem

Ken uses data representing sales at a daily level. His problem is to determine the effective tax rate for a given row, when tax rates change over time.

In his article, he uses meaningful sales data. Since the core of the problem is to find the relevant tax rate for a specific date, and I want to focus on that, my ‘Data’ table looks like this:

Date
06/01/2010
07/01/2010
08/01/2010
03/01/2013
04/01/2013
05/01/2013

Here is the ‘Tax Rates’ table. [Start Date] represents the day when a tax rate became effective.

Start Date Tax Rate
07/01/2010 0,12
04/01/2013 0,05

Note that Gerhard Brueckl’s problem is similar: he presents a way to assign the relevant fiscal period to a date in a dimension table. In his model, fiscal periods are defined by their start date as well.

A solution

Ken Puls’ idea is to add a calculated column to his data. This column will contain the value of [Start Date] for the relevant tax rate. He will then create a relationship between the ‘Data’ table and ‘Tax Rates’ table based on this new column.

His calculated column contains the MAX value of [Start Date] for all rows of ‘Tax Rate’ where ‘Tax Rate’[Start Date] is not greater than the current date in the data table. In other words, he replicates the logic of the following Excel formula:

= VLOOKUP( [Date] , 'Tax Rates', 1, TRUE)

The DAX expression for a calculated column looks like this:

[Tax Rate Start Date] := 
 CALCULATE(  MAX( 'Tax Rates'[Start Date]) 
   , FILTER('Tax Rates', 'Tax Rates'[Start Date] <= 'Data'[Date] )  
 )

It filters the ‘Tax Rates’ table, and calculates the max value of ‘Tax Rates'[Start Date] for the filtered table.

Extending the solution (TOPN)

We can use TOPN to mimic the behaviour of VLOOKUP.

For an approximate search, we will follow this logic:

  • use a FILTER expression to eliminate irrelevant rows (as in the previous formula)
  • use the TOPN function to filter the result and only return the last row in ‘Tax Rates’
  • return the value in the required column for the row we just found.

Here, “the last row”  means the row with the latest [Start Date].

Let us rewrite Ken’s expression using the TOPN function:

[Effective Tax Rate - Start Date] :=
 CALCULATE (  VALUES('Tax Rates'[Start Date])
  , TOPN(1
  , FILTER('Tax Rates', 'Tax Rates'[Start Date] <= 'Data'[Date]  )
   , 'Tax Rates'[Start Date]
    )
 )

The advantage of this pattern is that we can use it to query any column in our lookup table:

[Effective Tax Rate - Rate] :=
 CALCULATE ( VALUES('Tax Rates'[Tax Rate])
  , TOPN(1
   , FILTER('Tax Rates', 'Tax Rates'[Start Date] <= 'Data'[Date]  )
   , 'Tax Rates'[Start Date]
    )
 )

Dealing with ties

There might be a problem with the previous formula: TOPN may return more than 1 row. This will be the case, if there are ties in your data.

In our example, if the ‘Tax Rates’ table has several entries for the same [Start Date], then the VALUES expression might return more than one value. In that case, the expression will fail.

Examples:

Start Date
Tax Rate
Start Date
Tax Rate
07/01/2010 0,12 07/01/2010 0,12
04/01/2013 0,05

04/01/2013 0,05

07/01/2010 0,12 07/01/2010 0,13
The formula still works. The formula returns an error.

This may be totally fine. Most of the time, this is what we want: if a reference table contains contradictory information, we want to notice it.

However, in some cases, you may want your formula to ignore that, and take one of these values anyway.

In that case, you can add a sort expression to the TOPN expression, so as to guarantee that only one row (or value) is returned.

Edit: Although this is what I first wrote below, SAMPLE does not behave like a TOPN with ties. How I came to this conclusion is unclear, as I can no longer find the test data I used. Although, this might seem to be true in this specific case, you should not assume this will work every time.

However, you can wrap the TOPN expression within a SAMPLE expression to only get one row (or more if required).

You may also use the SAMPLE function instead of the TOPN function. What does it do? Basically the same as TOPN except in case of a tie. In that case, SAMPLE will exactly return the required number of rows.

  • Which rows? The engine will decide by itself.
  • Will it always return the same rows? This is not guaranteed.
  • Will the rows returned change on each execution? This is not guaranteed.

As a side-note, if you want to write a query that returns exactly 3 rows from your ‘Data’ table, but want these rows to be randomly drawn between each execution, then you can do the following.

EVALUATE SAMPLE( 3,'Data', RAND() )
Advertisements

Equivalent of VLOOKUP in DAX–Part I

One frequent DAX requirement is to write a formula that behaves like the VLOOKUP function in Excel.

Gehrard Brueckl recently blogged about how to map a date to a fiscal period, when fiscal periods are defined in a separate table, and each period is defined by its start date. Ken Puls recently wrote an article about calculating an effective tax rate with DAX.

In this first post, I will present the LOOKUPVALUE function.

Quick notes about VLOOKUP

VLOOKUP, a classic Excel function, accepts two values (TRUE or FALSE) for its last argument.

Assuming we defined names for Value, Range, i, the formula will look like this:
=VLOOKUP(Value, Range, i, FALSE)

VLOOKUP will look for the first occurrence of Value in the first column of Range. If the value cannot be found, the formula will return #NA, else it will return the value in the i-th column on the same row.

Note that VLOOKUP also works with arrays.

When the last parameter is TRUE (or omitted), then VLOOKUP requires your data to be sorted in ascending order. It will look for the last value in the first column of Range that is not greater than Value, and will return the value in the i-th column on the same row, or #NA if none exists.

More details on VLOOKUP here, there, or everywhere on the internet.

LOOKUPVALUE

Syntax

DAX V2 introduced a LOOKUPVALUE, that has a related behavior.

A notable difference is that LOOKUPVALUE allows you to provide criteria on several columns. In Excel formulas, this would require you to use concatenate columns, or even replace VLOOKUP with some INDEX/MATCH formula.

The syntax looks like this:
= LOOKUPVALUE( Table[OutputColumn], Table[LookupColumn1], “SearchValue1”,  Table[LookupColumn1],  “SearchValue2”)

The syntax looks like the vector form of the LOOKUP function, if you put the order of arguments aside. It actually looks like the syntax of the LOOKUPS function, which does not actually exist. Smile

Basically, you first have to define which column contains your output, then define your first lookup column, then provide the value you are looking for, then define your second lookup column, and so on …

Behavior

LOOKUPVALUE behaves differently from VLOOKUP:

  1. whereas VLOOKUP works with arrays, LOOKUPVALUE does not work with column expressions. In other words, the columns arguments must reference columns that physically exist in your model. Also, remember that each column must belong to the same table.
  2. if several rows in your table match your criteria, then VLOOKUP in Excel will only return the first (or last) match. On the contrary, LOOKUPVALUE in DAX may return an error if several rows match your criteria. This will occur if [OutputColumn] does not contain the same value for all matching rows.
  3. a BLANK value is returned if there is no match.

Also, not surprisingly, you must address columns by names and cannot use indexes to do this.

When to use LOOKUP value?

When your lookup table is a parameter of your model. By that, I mean a static table that you can import in your model.

Obviously, when the behavior described above suits your requirements.

When you want to calculate search values on the fly, or use different search values that come from different tables. In other words, when you cannot use PowerPivot relationships.

Attribute relationships and cube space in SSAS

Defining attributes on your dimension will improve processing and querying performance but might also affect the results of your queries, sometimes leading to unexpected results.

This has already been discussed by Jeffrey Wang and other bloggers but I figured a new treatment of this topic might be helpful (at least to me).

Without attribute relationships.

Let us take the Date dimension as an example. It has [Year],[Month], and [Date] attributes and contains data for the years 2011-2013. No attribute relationship has been defined yet.

The cube has a measure called [# Days] which represents the number of days for a given period.

In the following, I will not explicitly mention the dimension when writing tuples. This is for the sake of readability. This is of course not recommended in production.

The following table shows what happens, when  you overwrite the context in derived measures.

 
Year Month [# Days] ( [# Days], [Month].[All]) ( [# Days], [Year].[All]) ( [# Days], [Month].&[2013/01]) ( [# Days], [Year].&[2013])
2013 2013/01 31 365 31 31 31
2013 All 365 365 1096 31 365
All All 1096 1096 1096 31 365
All 2013/01 31 1096 31 31 31

The first two columns describe what members are being queried on rows, the other columns show the result of the corresponding measure definition.

The rule in the case where no relationship has been defined is quite simple:  overwriting an attribute has no impact on unrelated attributes.

Attribute relationships modify cube space

Now, suppose we define a relationship [Month]->[Year] in SSAS. [Month] is the source attribute, and [Year] the related attribute.

This relationship tells SSAS there is a n:1 relationship between [Month] and [Year]. In other words, one month can only belong to one year.

Setting [Month] to a specific member, say [2013/01] in your query, will effectively set [Year] to the matching member (in this case, [2013]).

The table shows the effective context that results from what you specified in your query.

Query Effective
Year Month Year Month
All All All All
All 2013/01 2013 2013/01
2013 All 2013 All
2013 2013/01 2013 2013/01
2012 All 2012 All
2012 2012/02 2012 2012/02

When you specify the tuple ( [Year].[All], [Month].&[2013/01] )  in your query, SSAS will actually evaluate ( [Year].[2013], [Month].&[2013/01] )  even though you explicitly requested [Date].[Year].[All].

This might not seem worth mentioning, but it is. It becomes quite important, when you add further attributes and relationships to the [Month] attribute.

Partial tuples in measures.

Overwriting context in a measure would be simple, if you only ever provided complete tuples. However, providing complete tuples would make measures hard to maintain and read. When you specify a partial tuple in a measure, you only specify a few attributes. SSAS implicitly provides members for the attributes you did not include in your tuple.

Overwriting source attributes.

When  you set an attribute to a specific member (e.g. [2013/01]) , its related attributes are implicitly set to their corresponding member (e.g. [2013]).

The rule when setting an attribute to its [All]member,  is less straightforward, and can be expressed this way:

  1. if the current member is [All], current members of related attributes will be kept
  2. otherwise related attributes will be set to [All]

In the table below, I stressed the case where case #1 applies.

Effective ( [Month].[All]) ( [Month].[2013/01])
Year Month Year Month Year Month
All All All All 2013 2013/01
2013 All 2013 All 2013 2013/01
2013 2013/01 All All 2013 2013/01
2012 All 2012 All 2013 2013/01
2012 2012/02 All All 2013 2013/01

Overwriting related attributes.

Setting a related attribute (e.g.  [Year]) in a measure, while omitting the source attribute, will modify the context by implicitly setting its source attribute  (e.g.  [Month]) to its[All] members.

This simple rule can be visualized in the table below.

Effective ( [Year].[All] ) ( [Year].[2013] )
Year Month Year Month Year Month
All All All All 2013 All
2013 All All All 2013 All
2013 2013/01 All All 2013 All
2012 All All All 2013 All
2012 2012/02 All All 2013 All

Context propagation.

Although it might look so at first sight, writing ( [Year].[All] ) in a measure is not the same as writing ( [Year].[All], [Month].[All] ).

Suppose we add new attributes to our date dimension ([Month In Year] and [Day In Month]), and add the appropriate relationships.

Our attribute relationships now look like this:

  • [Date] –> [Day In Month]
  • [Date] –> [Month] –> [Year]
  • [Date] –> [Month] –> [Month In Year]

Explicitly overwriting an attribute in a measure implicitly overwrite its source and related attributes. Implicitly overwriting an attribute, however, only implicitly overwrites its source attributes.

So, if we write ( [Year].[All] ) in a measure, this will not overwrite [Month In Year], or [Day In Month].

The table below summarizes what we obtain for different tuples:

Tuple Year Month Month In Year Date Day In Month
( [Year].[All] ) All All Current member All Current member
( [Year].[All],  [Month].[All] ) All All All All Current member
( [Year].[All], [Month].[All], [Date].[All] ) All All All All All

Also, I mentioned earlier relationship modify cube space.  Suppose you have the following query.

WITH MEMBER [Measures].[My Measure] AS
    (
        [Measures].[# Days], 
            [Date].[Year].[2012]
    )
SELECT  
  [My Measure] ON COLUMNS
 , ([Date].[Month].[2013/01]) ON ROWS 
FROM [My Cube]

The result will be 31.

The reason behind this, is that writing ([Date].[Month].[2013/01]) on rows, is the same as writing:

( [Year].[2013], [Month].[2013/01], [Month In Year].[1], [Date].[All], [Day In Month].[All]  )

Since overwriting [Year] will not overwrite [Month In Year], the measure [# Days] will be calculated for:

( [Year].[2012], [Month].[All],     [Month In Year].[1], [Date].[All], [Day In Month].[All]  )

Conclusion

When overwriting attributes in a measure, you had better not rely too much on implicit members. Rather use the ROOT function wherever appropriate. Check your relationship tree. Check your measures against every sibling attribute in the relationship tree (or other descendants of ascendant nodes).

If you want to override any filter on the [Date] level, including any filter through [Day In Week] or other related attributes then include the [Date].[Date].[All] member in your measure.

If you want to override the filter on the [Month] attribute, as any filter on the [Date] attribute, but keep current filters on [Year] and [Month In Year], then use:

( [Year].CURRENTMEMBER, [Date].[All] )

Notes on “Usage of dates argument in a row context” by Marco Russo

Marco Russo recently blogged about the usage of dates arguments in a row context.  The measure he presents as producing “wrong results”, actually produces the results I expected. For this reason, I had a hard time understanding the motivation behind his article and decided to write a follow-up in case anyone else missed his very subtle explanations about context in DAX expression. If you have not done it yet, I recommend you read his entry, before and/or after reading mine.

So, suppose you have a table called OrderDate with a Date column and one row for each day, and define the following measures:

LastDay : = LASTDATE(OrderDate[Date])

NumLastDay := COUNTROWS(
FILTER(OrderDate,
OrderDate[Date] = [LastDay]
)
)

If you evaluate the measure NumLastDay for the month of January, for example, you will get 31 as a result. As already explained by Marco, FILTER iterates the OrderDate table: It creates a row context for every row of the OrderDate table present in the current filter context. The measure LastDay will be evaluated for every single row(day). Hence, the expression OrderDate[Date] = [LastDay] will always return TRUE.

Contrary to Marco, I was expecting this. Also, you can get the same result if you define LastDay as
LastDay := MAX(OrderDate[Date])
That is why I did not immediately get his point, and what was so special about the LASTDATE function.

However, LASTDATE and MAX behave differently when used directly within the FILTER formula:

DirectCall – MAX := COUNTROWS(
FILTER(OrderDate,
OrderDate[Date] = MAX(OrderDate[Date])
)
)

will return 1 when evaluated for the month of January. The evaluation of OrderDate[Date] within the MAX expression is done for the current filter context, disregarding the row context created by the FILTER expression.

On the contrary,

DirectCall – LASTDATE := COUNTROWS(
FILTER(OrderDate,
OrderDate[Date] = LASTDATE(OrderDate[Date])
)
)

will return 31 when evaluated for the month of January.

The filter expression
OrderDate[Date] = LASTDATE(OrderDate[Date])
actually behaves like the following expressions
OrderDate[Date] = CALCULATE(MAX(OrderDate[Date]))
OrderDate[Date] = CALCULATE(MAXX(OrderDate,[Date]))
OrderDate[Date] = MAXX(CALCULATETABLE(OrderDate),[Date]))
OrderDate[Date] = CALCULATE(LASTDATE(OrderDate[Date]))
OrderDate[Date] = LASTDATE(CALCULATETABLE(OrderDate[Date]))

where the CALCULATE and CALCULATETABLE parts create a new filter context from the current row context, for the evaluation of OrderDate[Date].

That is the point Marco Russo makes in his article about the LASTDATE function. He presents two solutions to calculate a measure for the last day within the current filter context (see his article).

Here is mine, by the way

=CALCULATE(
COUNTROWS(OrderDate),
LASTDATE(OrderDate[Date])
)

Some interesting facts about CUBE functions in Excel

Did you know that … ?
– references to a cell containing a CUBE function return an OLAP object, just as the source cell;
– range operations on cells containing CUBE formulas return ranges of OLAP objects, that you can use in other CUBE functions;
– a copy & paste as value operation will remove the formula, but the destination cell will contain a new OLAP object containing the original definition.

It means you can:
– use your favorite functions (INDEX, VLOOKUP, …) on ranges of cells containing cube functions and use the result in a CUBEVALUE function;
– let end users choose the sets and measures they want to see with simple data validation lists that use range of cells containing CUBE formulas. You might not even need to use any lookup functions or IF statements nested within a CUBE formula to build your dashboard. Just keep it simple.

The fact that copy & paste as value operations on CUBE formula create OLAP objects may hurt you:
1. The objects remain bound to the source, as can be seen when trying such an operation in another workbook;
2. The objects will be updated with every connection refresh. The fact the target no longer contains any formula, may lead end-users to think they made a backup of earlier values, whereas the result will be updated on every connection;
3. The result of a copy & paste as value operation will retain the original definition. For example, copying a cell containing a CUBESET function will keep its references to the same members even after the original set or cells referenced have been modified. However, if the definition referred to a named set, then the result object will also change when this named set changes.

%d bloggers like this: