Miguel Escobar Published April 9, 2019

Combine or Append Data in Power BI / Power Query: Main Concepts

Power BIPower Query

IMPORTANT: You can read the official documentation that Microsoft has on this topic from the following link (url).

I’ve previously done a series on Merge / JOIN operations (First Part here) and it’s now time to do one on Combine / Append operations.

so…How do you combine / append / stack tables with Power BI / Power Query?

There are multiple ways to accomplish this, but we’re going to start with the basics.

Scenario: Combine Data from 2 CSV files

image

Imagine that we have 2 CSV files.

  • Sales data for the year 2017
  • Sales data for the year 2018

In reality, our data sources can be any. It can be 1 CSV and 1 Excel file, maybe 1 table from a database and another one from an API.

To Power BI / Power Query, it’s a bit indifferent as they’ll go through the same process regardless of their data source, so what I’m about to show you applies to every single data source possible within Power BI / Power Query.

We want to append both of these into just 1 table. How do we do that?

Connecting to the Data

This is a pretty straightforward process where we can just select our data source from the “Get Data” window and go through the import process which will display a simple browse window so you can select your file and then give you a preview of your data like this:

image

To finish things off, you can just click on Edit so we can go straight to the Power Query Editor.

This is pretty simple and in our case our csv files have the correct format thus not requiring any transformation steps to be done to it other than the ones that were automatically done by Power BI / Power Query.

We repeat this process for the 2018 file and then we’ll end up with 2 queries:

image

Those are the queries or tables that we want to stack on top of each other. How do we do that?

Combining / Stacking / Appending Tables

image

This is truly the easiest part, now all you need to do is find the button that reads Append Queries and then a new window will appear where you can combine all the queries that you want.

image

It doesn’t need to be just 2 tables, it can be as many tables as you want, which means that next year when I get the data for 2019, I can just create a new query and add this to this list of tables that need to be combined and I’ll be set and done.

The output of this will be a table with all of the columns from all the tables that we combined and all of the data from both.

image

Considerations

Data Load

When you first create the queries, they’ll be, by default, set to load themselves to your Data Model or created as new tables inside of Excel if you’re using Power Query for Excel.

This might create some data duplication which is completely unnecessary. To stop this from happening, you can change the behavior of the Sales 2017 and Sales 2018 queries to be Connection Only.

You can simply right click the query and disable the Load by unchecking the Enable Load option.

image

Data Privacy and Query Folding

When combining your data it follows the sample guidelines for Data Privacy (here) and Query Folding (here) as showcased in the previously written articles. This means that you need to be aware of how these concepts play out in your specific scenario, so I highly recommend that you read the articles in this topic in case you’re hitting any errors or performance issues.

The Table.Combine Function

image

When you go through the process described in this post, Power Query will end up using a function called the Table.Combine function.

When Power Query uses this function, it resembles the same experience that happens when you do a SQL UNION ALL. This means that Power Query will try to find matching columns from all of the tables and stack data accordingly, but if one table has columns that the other doesn’t, then it’ll fill any empty spaces with null values.

This function is BY FAR the best method when it comes to appending data as it’s a function that was designed to make this happen.

Power BIPower Query
0 0 votes
Article Rating
Subscribe
Notify of
guest
15 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Daniel Lamarche
1 year ago

Hello Miguel,
Thanks for this tutorial. I’ll take a chance sharing with you a problem with a later version of Power BI (Version: 2.75.5649.861 64-bit (November 2019)).
I follow you tutorial religiously but I do not get an Append1 query in the query pane. It is simply not there. I’m a bit new to all this and I’m learning for a pastime. I tried with other online Tuts to no avail.
If you’ve heard of this oddity maybe you will want to quickly share with me. I fact I’m a member of Power Query Academy and I love it.
Thanks
Daniel

Daniel L.
Reply to  Miguel Escobar
1 year ago

Thanks for your reply. On the Microsoft Power BI Community some friends explained that I should have used “Append Query as New”.
I will use your suggestion since I subscribed to to the Power Query complete course.
Thanks for getting back to me…
Daniel

Wilner
Wilner
1 year ago

Thanks for the article Miguel. Very, very, helpful. I do have a question. When I use append with two excel spreadsheets for example, I see that it doubles the data for both spreadsheet. removing duplicates doesn’t work. Any idea how to append two different spreadsheets with the unique values of each spreadsheet?

Thanks!

Lorena
Lorena
1 year ago

I have two data environments that have the same data layouts however, the columns start with either EDI or MI then the rest is the same. Is there a way to do this to allow these columns to match?

Williams
Williams
1 year ago

Hello Miguel, great tutorial, i just have a question, when i try to append more than 2 tables the option “three or more tables” is not showed, i have excel 2016, do i have to abilitate it or something

Olga
Olga
1 year ago

Thank you!Great article. I still have the problem of having `null` values after appending the queries. How can I change that function?

payam
payam
1 year ago

Hi Dears,

I want to append 2 or more excel sheets in Power BI that 5 columns of them are similar but the other ones are not the same.
would you please help me how can I do it?
after appending, the result is not proper and correct.

thanks in advanced.

Sandra
Sandra
9 months ago

Hi Miguel,
great tutorial! I have a monthly htm file, which always has at least three tables in it, but the number is different for every month. I always need to append the tables starting from table3 to n, and it works when I do it manually, but is there a way to tell the query “take the tables 3 to n” so that it works automatically each month?
I appreciate your help, thank you!

Hani
Hani
5 months ago

Hi, i have monthly sales data. Let’s say i start loading Jan data & do some modification to it. Add column, calculated field etc. Then, i want to auto load Feb data using Get Data from Folder. And append it with Jan data. I hit error. Is it not possible to append new data into old data that has been modified in query earlier? So i have to redo all the modification to my new data then only i can append it to my old data? Is this the case? Thanks..

ashkan ahmadi
ashkan ahmadi
3 months ago

hi
how could we append to queries without same columns .
we should create sample columns in each queries to unified them ?