Tag Archives: data warehouse

Tenets of a Real-time Data Warehouse

I’ve written a couple of blog posts this week on Tabular Models, and for a good reason.  I’m presenting a session called Data Modeling Best Practices for Enterprise Tabular Models at the PASS Summit next week, so naturally it’s a topic that’s on my mind.  But I’m also presenting another session on how to develop a Real-time Data Warehouse, so I decided that I should write a post on that topic too, since it’s also been on my mind.

Real-time DW Tenets

Instead of discussing the technology implementation and associated difficulties of loading a data warehouse in real-time,  I thought I would start with the three tenets that I think everyone needs to consider before embarking on a project like this.  Everyone develops a real-time solution differently based on their needs, technology, hardware, and other factors, but there are a few basic ideas to keep in mind no matter how you go about it.

Process only the data you must and nothing more.  To meet the challenge of real-time, don’t do anything extra, and don’t touch any data you don’t have to touch.  You always want to handle as little data as possible while still getting the job done.

Don’t impact the Source Systems. Contention with the source databases will almost certainly cause the real-time process to fail.  Make sure  your real-time processes don’t use resources that are needed by the applications that generate the source data.  Try to be as invisible to them as possible.

Take advantage of what SQL Server does for you.  SQL Server is a rich product with a great number of features and tools that can help you with this endeavor, so take advantage of them.  In my session I discuss how Replication, CDC, SSIS and other tools can be used as part of the solution.  Don’t write code you don’t have to write.

PASS Summit 2012

To see me build a functioning real-time data warehouse in real-time, come to my presentation at the PASS Summit on Friday, November 9.  The session is at 9:45am and is title Real-Time Data Warehouse and Reporting Solutions