One of my most popular posts is one about recurring dates with offset in Power Query (url). It has thousands of hits, but one of the key follow up questions that people have after reading that is “How do I fill dates between dates?”
This is the post where I’ll cover that very same topic to show you exactly how you can use Power Query / Power BI to fill dates in the easiest fashion possible.
You can follow along by downloading the sample file from the button below.
Case 1: Fill continuous Dates between dates
Imagine that we have a table like the one below:
Person | Start Date | End Date | Hours per day |
---|---|---|---|
Miguel | 26-Jun-19 | 09-Jul-19 | 7 |
Bill | 22-Jun-19 | 08-Jul-19 | 8 |
Ken | 20-Jun-19 | 05-Jul-19 | 10 |
Rob | 21-Jun-19 | 30-Jun-19 | 8.5 |
[/table]
What we want to create is a simple set of dates for that timeframe from the start date to the end date.
How do we fill the dates between those two dates with Power BI / Power Query?
It’s actually easier than you think. This technique is one that I commonly see my friend Ken Puls using when working with dates. See, you can transform those dates into a number and the create a list or sequence of numbers based on that start date and end date like this:
The key here is that list or sequence of numbers using this formula:
{[Start Date]..[End Date]}
I wish that it worked with dates, but it only works with numeric values, so that’s why we need to convert the dates into a number before creating our new custom column.
The result of that is going to be a new column with all of our dates inside a list.
You can click on the icon that looks like two arrows going in opposite directions and do a “Expand to New Rows” operation:
and then you can convert the new column into a date data type, remove the start and end date and that will give us the result that we’re looking for which is a simple way to fill in the dates between two dates.
Again, this is the easiest possible scenario and probably the most common one that you might find in the real world. If you need to fill in consecutive dates between two dates then this is the best way to make that happen.
Case 2: Fill only x amount of days
in the previous case we figured out how fill dates between two specific dates. We had a start and an end date, but what if you only have the start date and you want to figure out a specific set of dates from that start date?
In this case we have a table that looks like this:
Person | Start Date | Valid for |
---|---|---|
Miguel | 9-Jul-19 | 7 |
Bill | 2-Jul-19 | 1 |
Ken | 4-Jul-19 | 3 |
Rob | 8-Jul-19 | 90 |
[/table]
and the situation is quite simple. To the Start Date, we want to add the amount of days from the “Valid for” column.
To give you more context, imagine that this is a table that shows us the access that each person will have during a timeframe and we want to see how many of us will have a certain access during a specific date or set of dates. The overall scenario is more complex than that as it also has it could have to do with other fields, but I’ve simplified the scenario just for demonstration purposes.
The next step is just to add a custom column. No need to convert the columns into numbers this time – we actually need this column as date this time and add the following formula:
List.Dates([Start Date], [#”Valid for “], Duration.From(1))
I’m using a function called List.Dates which takes an initial date and creates a list of dates from it.
- The first parameter of this function is the starting date
- The second parameter of this function is the amount of dates that this function should return from the starting date
- The third parameter is the amount of days between dates and it has to be passed as a Duration data type. In my case I’m saying that I want a new daily date. That’s why I use Duration.From(1) but you could also use #duration(1,0,0,0)
After expanding the new column this is how it looks like:
One crucial and REALLY important thing to mention is that the “counting” of the days starts from the 0 hours of the start date. This means that it is including the start date as the first date and that’s why the initial list of dates for Miguel in that table only goes until the 15 of July.
Case 3: Fill specific day of the week between dates
In the post that I previously mentioned on “Recurring dates”, I basically play with the last parameter of the List.Dates function to get only dates that will have a “gap” of x amount of days between them which I define using the last parameter of this List.Dates function with the duration.
This is a revisit to that specific case and here’s the initial table:
Initial Appointment | Patient | Follow ups needed | Frequency (every x days) |
---|---|---|---|
11-Jan-18 | Audie Livengood | 3 | 14 |
28-Oct-17 | Curt Gatz | 2 | 7 |
The objective with this one is to come up with the Follow up dates for an appointment. Imagine that we went to the hospital and the Doctor says that she wants to see us in 2 weeks from that appointment or that she wants to see us every 2 weeks for the next two months or so.
We can use the same formula that I have in my previous post. It’s the best way to work:
and after you expand the new column and set the correct data type for this new column you get this:
In this case we basically played around with the List.Dates function to fit it to our needs. I highly recommend that you read the original article if you want a more in-depth look at that specific scenario.
Other cases not covered
You might find other cases where you need to do something quite specific like find the 5th or last Friday within a given set of dates or find the second to last working weekend of a season.
These are usually highly specific scenarios that I’d personally tackle by creating a custom function, so if you’re ever in a situation different to the ones listed here, please let me know in the comments. I’d love to know about these other scenarios and see if we could update the case list from this post.
Dealing with Date and Time
So far we’ve been dealing with nothing but dates, but what about datetime values?
For that we can’t create a sequence of numbers since the sequence of numbers only works with integers and the List.Dates function only works with Dates, so what can we use?
For datetime fields we can use a function called List.DateTimes
Let’s look at this example:
DateTime | Total Alarms | Alarm every (minutes) |
---|---|---|
7/9/19 8:00 AM | 20 | 30 |
7/20/19 10:30 AM | 10 | 5 |
The goal here is simple: we need to come up with the exact date and time when these alarms should go off. We have a datetime that gives us when the alarm should go off the first time, then for how many times it has to go off (Total Alarms column) and then how often should it sound in minutes in that Alarm every (minutes) column.
Load that data to the Power Query window and make sure that your column is of the datetime data type.
Then we can create a custom column using this formula:
List.DateTimes([DateTime],[Total Alarms], #duration(0,0,[#”Alarm every (minutes)”],0) )
Note how the function looks similar too the List.Date function and it is exactly the same, except that the first parameter needs to be a datetime value instead of a date value.
The really important takeaway here is how the #duration part works and this is what you need to understand:
- #duration has 4 arguments and each of them are integer values
- #duration(days, hours, minutes, seconds)
After we expand the column, this is how our table looks like:
Other considerations for Datetime
In some cases, you don’t need to use a datetime but rather just use the time portion and work that way as a duration straight from the Power Query interface, but if you happen to need to use a datetime value, then List.DateTimes would be your best way and don’t forget about the power of the #duration keyword.
Miguel:
I did not see how you excluded weekends. Your data sample by definition would not include weekends but what if your interval was every 3 days excluding weekends?
Hey Brian!
Are you referring to case #3? I think that I misused the word “weekday”. What I actually meant was “día de la semana” or “day of the week”. I’ll change that asap!
In regards to something like “exclude weekends”, creating a custom function based on the List.Dates would be the best thing to do.
thx!
Hi Miguel,
Thanks very much for the post. Regarding excluding weekends, I’d like to know how a custom function would be beneficial, as I am not able to picture how it would work, if only the parameter start date of List.dates was a table of holidays and the list is generated based on the holiday table…
After giving it some thought, I was able to achieve the outcome by duplicating the query and adding extra days for new end date based on how many holiday days are present, however I’m interested to know your approach utilising custom function. Thank you
For the scenario that Brian mentioned, there’s really no native function in Power Query that can handle that directly. That’s why creating your own custom function to handle such scenario would be the best way to go, since there’s really nothing out of the box that can do this for you.
I’d probably have a list of dates that we couldn’t have an “appointment” and then try to go from there. I’d most likely end up using some sort of iteration with List.Generate, and just create my own “List.DatesNonHoliday” version. This was the first thing that came into my mind, but it woudln’t be anywhere near as fast as the List.Dates when there are no special cases
Hi Miguel
Thanks for the post, it’s a great help – I often work with time periods and events of a duration.
I have an unsolved problem of filling in dates for discontinuous consecutive periods that I hope you can help with. The problem is described here in the Power BI Community:
https://community.powerbi.com/t5/Desktop/Appending-rows-with-adjacent-time-periodes/m-p/605952#M288509
Hey Soren! I see that the question was marked as solved, but it wasn’t really solved whatsoever – you still have a situation with it. Do you think that you could post your updated question on the official Power Query forum?
https://social.technet.microsoft.com/Forums/en-US/home?forum=powerquery
One thing, if your operations rely on sorting, then it would be best if you buffer your table to keep that order of the sorting , but either way, there’s probably a better approach and the best way would be to figure that out on that forum
Hi Miguel
thanks for your reply. I have, as you suggested, set up my question in the Officeille Power Query forum.
https://social.technet.microsoft.com/Forums/en-US/4a0bbf13-6a63-4539-8dc2-4837281d8f4e/appending-rows-with-adjacent-time-periodes?forum=powerquery
Nice!
I see that you’re already getting some replies and possible solutions.
I do categorize this pattern as more on the “custom” approach where you’d need to create your own function or own heavy M-based approach to reach your desired solution.
Thanks Miguel
Yes I have received some good solutions that seem to do the job.
Is that applicable for Direct Query or Import Mode? I have similar requirement to generate list of date between two interval every 15 minutes interval. Is that possible to do in that way in Direct Query mode?
Best regards,
Veasna
The blog post was written using import mode, but I haven’t tested it on direct query. I’m not sure that it’ll work as i don’t think that list.dates is a function that is foldable. Probably with a database it would be better to create the view and just connect to that view
Thanks for your reply. As mostly said, including you recommend to create view for that kind of generating. Would you mind to help to generate that kind of view? I have try sometime before but no luck to get it works.
Regards,
Veasna
Hey, I’d recommend that you post your full scenario on the official Power BI forum (https://community.powerbi.com/t5/Desktop/bd-p/power-bi-designer ) or perhaps the official power query forum (https://social.technet.microsoft.com/Forums/en-US/home?forum=powerquery)
This comment system is really not designed to be a forum, so it’s always a better idea to use the public community forums
I rebuild the query to simplify it
let
Source = Excel.CurrentWorkbook(){[Name=”myquery”]}[Content],
#”Changed Type” = Table.TransformColumnTypes(Source,{{“DateTime”, type datetime}, {“Total Alarms”, Int64.Type}, {“Alarm every (minutes)”, Int64.Type}}),
#”MaxDate” = List.Max(#”Changed Type”[DateTime]),
#”MinDate” = List.Min(#”Changed Type”[DateTime]),
#”Duration” = Number.From(#”MaxDate” – #”MinDate”),
#”ListDate” = List.DateTimes(#”MinDate”,#”Duration”,#duration(1,0,0,0))
in
#”ListDate”
hey!
The code that I use for the last DateTime scenarios works on a row by a row basis.
The code that you have basically creates a completely new list from the Min and Max dates of your whole column, which would fall more into the case 1, but not really as you’d be creating a Calendar table and not filling dates between dates on a row by row basis.
Just something to take in consideration and be careful about! These are different concepts
Hi.
Looking to create a Table showing the Time range, not include date and hour number. This table will outer join to the Point of Sales transaction table. This will help to show a chart with each hour on the X Axis and number of sales per hour. Table should have Start Hour and Hour Number. In my transaction model I have Start Hour and Hour number already.
Sounds good!
Dear all,
Thank you for these examples, they have helped me a lot!
I am trying to find a solution to a very similar problem:
I have a data source in which the start time (Date time) and end time (date time) of machine failures is registerd. (So a row consist of columns with machine specifics and then 2 columns with a start and end time of the failure.
I managed to expand the rows in such a way that i now have a row for each day that a machine is ofline (great!). However, I would like to take it one step further and take the start time and end time (hours) into account.
So the result I am looking for is a row for each day (with date) a machine is off-line, followed by a column with the Hours the machine was offline that day. E.G. if it was off-line the whole day it should say 24h, and when it starts to be off-line at 02:00 PM it should say 10 hours (same for the day it gets fixed).
I think this problem is very similar to your ”Dealing with Date and Time” example. However, I am not intrested in a duration and have not succeeded to tune this example to a solution that works for me.
Any ideas on how i could approach this problem?
Hope to hear from you,
Kind regards
Stijn HIllenius
Dear Miguel,
It’s a very good explanation. There’s only one thing I couldn’t find yet. Do you know if it is possible include just month and year in the date range?
Best regards
You could change the pattern and use a function such as Date.AddMonths which should give you a way to move through months without any issues
Hi! This is great! Can you please provide guidance on how to tackle case where end date is null in case #1 ?
It really depends on what a null means to you and how you can interpret that null.
From a purely technical point, the function needs an end date of sorts, so it can’t really be null.
Thank you so much for this. I’ve been looking all over the internet for a way to covert a date range into rows for each day, and the solution is so simple.
Really made my day.
Hi Miguel!
Great post, I love it!
I have a little bit different scenario on which I’m currently working and I am stuck.
What if I want to fill two dates but only retrieve end of year and start of year between 2 dates?
For example I have pay info like this:
ID Start End Event Wage
1111 2000-05-26 2004-09-17 Initial pay 48,5
1111 2004-09-18 2011-11-30 Promotion 55,4
Unfortunately, there’s really no native function for that scenario. You’ll need to come up with your own custom function and define your logic. I’d probably have another table where I define what my “start of Year” and “end of year” represent, as a year could be fiscal, calendar, retail, chinese, etc etc etc. Then using that Table I’d try to use it as a helper table.
Hope this puts you in the right direction!
hello Sir
Good Day
i have a table with the next structure
hostname date status
——————————–
machine 1 3/4/2021 10:00pm down
machine 1 3/4/2021 11:00pm up
machine 1 3/4/2021 12:00am up
machine 1 3/5/2021 01:00am up
.
.
.
machine 2 3/4/2021 10:00pm up
machine 2 3/4/2021 11:00pm up
machine 2 3/4/2021 12:00am up
machine 2 3/5/2021 01:00am up
.
.
machine 3 3/4/2021 10:00pm down
machine 3 3/4/2021 11:00pm down
machine 3 3/4/2021 12:00am down
machine 3 3/5/2021 01:00am down
.
.
machine n 3/4/2021 10:00pm n
machine n 3/4/2021 11:00pm n
machine n 3/4/2021 12:00am n
machine n 3/5/2021 01:00am n
could you guide me how I can get the the machines that have been during 5 days complete with status down ( check that every day have some rows with differents status)
expected results:
machine 1
machine3
machine x
machine x
machine x
thanks in advance
Hey!
Could you please post your scenario on the official Power BI forum and share the link here?
Unfortunately, the comments section of this blog are not well suited for these type of engagements as things don’t look great and are a bit hard to read if they’re code based and we can’t share images either.
If you’d like to see through our remote consultancy services, you can reach out to me via email to miguel@poweredsolutions.co
Best!
Thank you for this article!
Thank you for the lesson. I have used the fill dates between start and end dates from this blog to help build a model for our business.
I have another expand date problem I cannot solve. I have a date and associated data column with periodic dates in it: in this case the quarterly expected growth in employee headcount for a client company. I need to expand the table to have every day between these quarterly dates (we could use first or last day of each quarter as the day for the value) placing in these expanded rows either the LastValue or modeling a liner growth as difference between the last and next values dividing by the number of days between. Your help would be very much appreciated.
Hey Mitchell!
I’m usually terrible at visualizing the scenarios with just text 😅
One thing that I do think is that there are a multitude of ways to solve problems in Power Query and usually the more optimal ones are the ones that try to leverage core functionality of the tool like Joins.
I’d recommend that you post your full-blown scenario with as many details and images as you can on the official Power Query forum:
https://community.powerbi.com/t5/Power-Query/bd-p/power-bi-services
Post the link to your thread on here so I can take a look at it.
Best!
Hi, Thank you for your post. Is it possible to add a column for each month or each day for a specific period? For example, I have a contract for 2 years with a daily quantity of 5 units. Is it possible to generate a column with the value of 5 for each day between contract start and end date?
Thanks
Hey Emir!
Similarly to what happened with Mitchell’s comment. I’m usually terrible at visualizing the scenarios with just text 😅
One thing is that it’s often easier to create rows and then pivoting them into columns rather than trying to dynamically create N number of columns. Nevertheless, it all relies on the logic that you want to follow and exactly what you’re trying to achieve.
I’d recommend that you post your full-blown scenario with as many details and images as you can on the official Power Query forum:
https://community.powerbi.com/t5/Power-Query/bd-p/power-bi-services
Post the link to your thread on here so I can take a look at it.
Best!
Hi. Is it possible to fill months between start date and end date, not days? I have contracts that have start date and end date with a contractual monthly volume listed. When I fill between the dates the monthly contractual volume is populated on each day. Appreciate the help!
there might be a way! There are no native functions that do this in Power Query, but you could create your own custom function to have this specific logic by leveraging a native function like https://docs.microsoft.com/en-us/powerquery-m/date-addmonths and perhaps a list like {0..N} where N is the number of months and then you simply do a List.Transform over it with the Date.AddMonths
Hi, quick question I have created the custom column to list all the days between 2 dates, how can I delete the weekends? Is it even worth the effort from an efficiency perspective? I am currently filtering them out in the viz.
if it’s for a data model, usually a calendar table is the best approach to filter dates, but if you don’t need those rows then it really depends on your definition of what weekends are. One way would be to simply add a column that denotes what Weekday that value is and just filter out what weekdays are weekend for you.
Can you believe that in some industries Friday, Saturday and Sunday are considered weekend? in others just Saturday and Sunday. In other specific cases I’ve seen Thursday through Sunday :O
How to generate dates between dates in same column, say I have only start date and the end date is next new date. How to fill the dates until new dates found?
Hey!
Not completely sure I follow, but i’d recommend posting your full scenario with sample input and expected output on the official Power Query forum:
https://community.powerbi.com/t5/Power-Query/bd-p/power-bi-services
it’s good but i want only the dates between start and end dates, exclude the start and end dates ,which means no no need to get start and end dates in the list output.
for example if start date is 1/2/2019 and end date is 1/5/2019 .so i want the output :
:1/3/2019
1/4/2019
You could modify the function to pass those values and filter them out of the result. There’s native function to make it happen, but you could create your own custom function or your own custom logic.
Thank you! This is great. On the first option, what if we only want the list to be a month instead of days? In other words, it just counts the first (or last) day of the month.
You’d need to come up with your own custom function for it, but there’s quite a number of ways to tackle that scenario. If you only need a count of months, you’ll have to answer questions such as, “what’s a month?” and then implement a logic that uses that definition