Skip to main content

Dealing with High Cardinality columns

Are you still struggling with slow reports and a complex data model and you have tried every approach to optimise it? One of the major reason behind your slow reports are high cardinality fields. What is this cardinality now? The simplest definition for it would be the unique values present in a particular column. 

Do note we are talking about the unique values not distinct values there is a slight difference between these two.

Want to learn more about cardinality here is a great article from Arno. If you have a background in data analysis then the term cardinality may not be alien for you. When we talk about optimising the data model then it is always advised to reduce the cardinality as more the unique values available in a particular column  more it will take the memory and impacts the performance.

In this article, we will focus on a few easy steps to reduce the cardinality of the data model. The very fundamental and the simplest step will be to only include the fields that are necessary and identify the ones which are not used at all in the report. Discard them it will give your report a slight boost.

How to identify unused column? You can do it via the help of external tools such as DAX Studio. If you are beginner to DAX Studio then do visit our most visited article on it. You can connect to DAX Studio and view the metrics that will give you this information.


In this case, we have identified we are not using Product Key. When we get rid off the Product Key the size of your report reduced to half from 12mb to 5mb.


Second, try to reap the benefits of splitting columns such Date/Time. Fundamental step will be to check do you need the time at all? In most cases you do not need it and you can change the data type to only date but if you need it then make sure date and time are split in 2 columns. 

Isn't this easy? This article mainly details with the fundamentals to optimize your reports. Another fundamental step is to check if you are dealing with the correct data type or not. How to check this? Longer route will be to go over each and every field and see the data type but as a reader of analyst in action you are PRO. 

How PROs do it? Once you are connected to DAX Studio you can view at a glance in view metrics. In the contoso dataset we have changed the data type for Quantity to decimal number. Let's see how the metrics look with the decimal number data type.


Do notice the total size column for Quantity. Now, we are reverting back to whole number and you can see the drop in the total size.


Lastly, try to avoid the use of calculated columns in your reports. Let's take an example from Contoso data, I have created a calculated column to get the cost. Now let's take a look what is exactly is happening behind the scenes. 


Whhaattt!! suddenly you can see the Cost is on the top in your metrics which is actually not good for your data model. If you take a close look at the cardinality column for the Cost it is the highest and hence slowing down your data model. Now, imagine if you have 10-20 of those calculated columns. Avoiding calculated columns is always considered as a best practice in Power BI.


"All our articles are intended to address all the frequently asked questions related to a topic. Do leave a comment if it answers any of your question. "


Thanks for Reading Let's connect on LinkedIn. For more such blogs and pro tips do follow us





Comments

Popular posts from this blog

Ultimate Beginners Guide to DAX Studio

There are zillions of external tools available with Power BI but DAX Studio is one of the most commonly used tools to work with DAX queries. It is a perfect tool to optimize the DAX and the data model. In this blog let's shed some light on the basic functionalities that can take your report to the next level. ARE YOU READY?  To start you will need the latest version of the DAX Studio. You can download it from their website . Don't worry you don't have to pay for the license. Fortunately, DAX Studio is a free tool As a BI Developer, I am using DAX Studio regularly. Based on my experience I use it for several purposes but in this blog, I will highlight the most common ones. Extracting a dump of all the measures used in your PBIX. Why do we need to do this? It can be used for documentation purposes also sometimes we try to reuse the DAX and such a dump comes in handy in this scenario. How to achieve it? Open the DAX Studio it is located under the external tools once you open t

Append v/s Merge in Power BI

Let's discuss another problem of the week. As a Power BI user, there are times when you want to combine queries. What are the ways to do so? In most cases, you can attain it by using either append or merge and both serve different purposes. Let's understand what do these terms mean in Power BI and how they are functionally different from each other.  It is quite common to get data from various sources and you need to combine those data depending on a particular column which is common in both tables so that you can add extra information or column to your big table. In such cases, we use merge queries. How to perform merge queries? For instance, I am considering Sample Superstore data and we will merge the returns table to the order table. You will find both merge and append in the home tab in extreme right in the power query editor. ProTip - You will find two options when you click on the drop-down in merge which are merge queries and merge queries as new. When you use merge que

Use Relationship in DAX

Data modeling is an essential part of creating perfect visuals. While creating complex data models there can be a case where you can find an inactive relationship represented by dotted lines and it occurs because you already have an active relationship between the two tables. But as a developer, you need to use both the relationship. How can it be done? You can use "Use Relationship" in such cases. Use relationship can be added to your DAX and act as a modifier or enhancer for calculation. It activates the inactive relation. But make sure you have an inactive relationship in place before using the use relationship function. Let's see how it works on Sample Superstore data. In my fact table I have two dates- Order date and Ship date. I am making the two relations between my date table and fact table. The relation between the sample superstore (date) to date table (date) is active while the relation between the sample superstore (ship date) to date table (date) is inactive