Wednesday, 25 November 2009

Inspiring...

I just finished reading William Kamkwamba's book - The boy who harnessed the wind. It is truely an inspiration - I'd recommend this to everyone, especially if you tend to make excuses for why you haven't achieved anything.

I hope a bit of it rubs off on me !

Friday, 25 September 2009

Peak oil & end of the information age

Up until fairly recently I would say I've been very ignorant around oil. I had heard of peak oil, but had no understanding of what it really meant beyond the price of petrol and the environmental impact.

But I have recently done a lot of reading of the subject of peak oil and now understand that that is just the tip of the iceberg. Our society is completely dependent of cheap power. Due to the continuing growth in both the developed and developing world, demand increases every year, but supply does not. On day soon - maybe tomorrow, maybe in 10 years, the amount of oil coming out of the ground will reduce. This will permanently change the dynamics of the world - we will find alternative fuels, but they won't be anywhere near as cheap, at least in the short term. Think of all the manufacturing that has been moved out of western countries - it only makes sense because it is cheap to transport to where the customers are. This applies to food also - it won't be economical to transfer fruit from South America to Europe for example. So now we come to this thing - the internet. The biggest cost (when you add the manufacturing, transport, and running costs) of this is power related. Will is continue to be feasible to ran suites like face-book of twitter when costs double of triple ?

I think there is a really danger that a lot of what we do will not last in the post oil world.. The millions of people sitting at PC's in office buildings may need to head out into the real world again and start growing and making things.. which might not be so bad after all !

Thursday, 17 September 2009

SQL Server Integration Services 2008 64 Bit frustrations

We recently needed to purchase new infrastructure for our data warehousing environment. As we aren't likely to do this for a while, we decided to bite the bullet and go 64 bit everything; OS & SQL Server 2008. This has been a much harder than were envisaged and given how straight forward 64 bit UNIX is, somewhat disappointing.

The first issue we became aware of is there are no 64 bit excel drivers available yet. Apparently they will arrive in Office 2010, which given we are using Office 2003 at the moment, won't arrive here for another 6 years. This has meant for about 1/2 our packages they will only work in 32 bit mode.

Next we found that BIDS only runs in 32 bit anyway, so if we addressed the above issue, we would have problems maintaining these packages anyway. This pretty much says to me that at least from an SSIS point of view, 64 bit SQL Server is not really for prime time.

Finally we had a lot of problem installing 32 bit ODBC drivers. They installed fine, but we then couldn't find them in the odbc administrator. It turns out their are 3 (maybe more ) obdc administrators in Win Svr 2K8 64bit. Solution: Using the odbcad32.exe located in %windir%\syswow64 .

I have seen the Microsoft adds that state the WOW starts now. Maybe they were referring to that directory...

NB: This process is ongoing, so I will update this post if we discover any more gotchas...

Monday, 6 July 2009

Best practice for slowly changing Fact tables.

I've been trying to find some evidence of best practice for facts which are updated and have been unable to find much written on this subject. While this is quite common for case management type systems (e.g. HP Service Centre, Salesforce.com), it is not typical for data warehousing in general, as most literature focuses on more traditional transaction based systems (e.g. bank accounts, PABX data, order/billing etc.).

The challenge we have then is to determine when the dimensions of a fact record should be changed, and when they should be left unchanged if the fact itself is altered. Obviously for SCD Type 1, this is a non-issue; we aren't keeping a record of the previous Dimensions value, so the fact must be linked to the newest version anyway. But for SCD Type 2, there are three possible options:
  1. Determine the Dimension at the first instance of the fact being recorded in the warehouse.
  2. Update the fact's dimension's to the version of the dimension that existed at the time of the fact being modified.
  3. Update to the Current version of the dimension at the time the fact is updated in the warehouse.
While 1 is the simplest to implement, it doesn't reflect the real world usage of many source systems. For example an address Dimension may start of with very minimal details initially, but will gain more details over time. It is unlikely that we would want to stay with the minimally populated dimension once we have richer data available.

Option 2 to me is probably the best 'default' option in this case. It makes sense to me that if the fact is updated, it should reflect the dimensions at that point in time. For example if sale was made to the sales rep for region 1 (e.g. 'Bob') and Region 1 is later re-assigned to 'Fred', we don't necessarily want that sale updated to 'Fred' as he never made that sale. I'm sure there are cases though where this may not be the right approach.

Option 3 to me in most cases would give an incorrect view of the data, so therefore is not what we want either, however it may be that in the source system certain data is only updated periodically and that for that system the most recent version is deemed to be the 'correct' version.

I'm include to use Option 2 in most cases, but I would appreciate any feedback on this, as there may be other scenarios I haven't considered. Also, if you have come across any articles linked to this, please pass these on to me.

What are your thoughts ?

Monday, 27 April 2009

10 skills developers will need in the next five years

Click on the title for the original article. Here is how I score on the list:

1: One of the “Big Three” (.NET, Java, PHP) -CHECK
2: Rich Internet Applications (RIAs) - FAIL
3: Web development -A BIT
4: Web services -A BIT
5: Soft skills -CHECK
6: One dynamic and/or functional programming language -CHECK
7: Agile methodologies -A BIT
8: Domain knowledge -CHECK
9: Development “hygiene” -CHECK
10: Mobile development -A BIT

So I give myself 7/10. Need to geek up a bit !

Thursday, 26 March 2009

iMovie 08

I'm currently learing how to use iMovie 08. Despite all the criticism it gets, I am starting to like it. I was wondering though, is there any way to add functionality ?

Also, are there any public domain clip libraries I can use to enhance my projects ?

Thursday, 12 March 2009

Off topic...

Just found a great website for money management. Click on the litle to have a look...

Monday, 19 January 2009

Gartner's 5 predictions for BI in 2009 and beyond

I'm not a huge fan of Gartner, due to the cash for comments nature of their business, but I thought the following was an interesting piece. Of particular relevance, I found the last point one that I identify with:

"And Gartner's fifth prediction is that by 2012 one-third of analytic applications applied to business processes will be delivered via coarse-grained mashups. IT shops are no longer focusing on grand visions such as SOA, which involve building composite applications out of fine-grained services or portals that merely display operational and analytical information next to each other, the analyst firm explained. Coarse-grained mashups, on the other hand, can be used to overlay analytical insights."

I interpret this, as tactical solutions will be successful, where as big bang solutions will fail. In my experience, that is almost always the case!

a Big gap...

I haven't posted anything for a few months. Sorry for the gap. I have started a new job (based around SQL Server and SSIS) and have a new daughter, so I have been a bit busy !

I'll try to make more regular posts this year - promise !