Tuesday, 14 January 2014

Happy new year - A present from SAP

I got back to work this year to find that over the break SAP had release Data Services 4.1 Service Pack 3.  This contains a lot of fixes, especially if you are a Salesforce.com user.

I think you will need access to the sap portal to view it, but the release notes are here:

DS 4.1 SP3

Now to shake of the Holiday cobwebs and get cracking with 2014 !

Tuesday, 24 December 2013

Business Intelligence and Santa

As its Christmas Eve,  I thought I would post a link to a Microsoft Power Bi dashboard which shows NORAD's Santa tracking data.



Tracking Santa


Merry Christmas !

Tuesday, 17 December 2013

Understanding SAP tables

I haven't been working with SAP for that long and I often find it a challenge to understand if I have the right table.  The names are criptic and the same data is often repreated in many different tables, so what seems right, sometimes isn't.

I came across a good website yesterday for understanding this.  I thought i'd share it here for others:

http://www.se80.co.uk/


I hope this helps you !

Wednesday, 4 December 2013

Agile BI road blocks

We have been trying to run out BI team in a more agile fashion over the last year with limited success.  There are a few main areas we have hit problems in:

Source System knowledge:  Due to the fact that we use big ERP/SAAS solutions like SAP and SFDC, out business don't have the technical knowledge of where their data comes from in the underlying systems.  This means we have needed to build this knowledge up in our team, which takes a lot of time (We don't have the budget to bring in experts).

Getting to the data - 2 main areas this hits us - Cloud solutions that don't provide adequate interfaces for getting to their data for BI & SAP ERP, which prohibits getting data more than once a day for some key areas (unless you buy Hana for $$$$).  Its almost as if these vendors want you to be locked-in to their solution !

Bad data - a classic, but still an issue. 

Design - I'm starting to think Star schema's and particularly slowly changing dimensions are agile handbrakes.. The amount of work that goes into adding fields to SCD2 tables and not having it generate a new generation requires a lot of extra coding.

These are only a few, but hopefully someone has a solution - and I don't mean a new tool - I am really trying to find a better process, if there is one !

Look forward to hearing your thoughts

Tuesday, 3 December 2013

Prioritising the Data warehouse

One area which is always contentious is determining which problems to solve first.  I found some great insides on this in David Stein's post titles "If We Build It, Will They Come?".  The full link is below, but I particularly like the simple Gartner-like Matrix below:


Thanks David !

If We Build It, Will They Come?

Tuesday, 8 October 2013

Work life balance

As I write this it is Tuesday.  Yesterday was a public holiday here so we got a log weekend.  I have a great time with my family and feel refreshed for the first time in months.  So why can't I do that every week if I want to ?

To me, I don't do 40+ hours a week 48 weeks a year very well.  It burns me out.  For some people that may be perfect, but not me.  One day I hope to find an employer that lets me work 4 days a week or 46 weeks a year (or both).  I think that would be about perfect. 

Has anyone found the perfect balance ?  Are there any employers out their who let their employees pick the conditions that suit them ?

Let me know if you've found the perfect balance !

Thursday, 15 August 2013

Moving data services repositories



We recently moved out repositories from one SQL server to another.  We have no end of problems and are still finding little issues.  So I thought it would help others if I posted a few tips:

  1. Before starting, make sure everything is checked into the central repository.
  2. For local repositories, you are better off to create new one's on the destination server, and remove the old ones.
  3. Ensure that you have everything correct in IPS/BO before worrying about the job server designer.
  4. If you get errors trying to add repositories into the job server, you will need to change the job server name.  One the repos are added, just change it straight back.  This seems to be a bug in the job server manager.

I hope these help !   If I knew this before we stared, I would have had a much easier few weeks !

Tuesday, 12 February 2013

Another Data Services link..


http://dwhnotes.com/data-integrator


A few notes to supplement the wiki..

Friday, 11 January 2013

SQL Server 2012: Tabular & Multidimensional models on the same machine

Wow, 3 posts in a day - must be a record for me !

I have been trying to compare the pros and cons of moving to tabular for my company.  I wanted to be able to look at them side my side, but it wasn't obvious how to.  Luckly someone else worked it out for me:

http://www.jamesserra.com/archive/2011/07/sql-server-denali-installing-ssas-multidimensional-vs-tabular-mode/

Big thanks to James Serra for working this out !


Space issues when installing Data Services patches and upgrades

I thought I'd pass on an issue I had installing the latest Data Services patch.

The patches are compressed with WinRAR.  When you double click on the patch, it decompresses it to your temporary directory.  In our case that was on the C: drive, but the server in question did not have enough room on the C: drive.  This resulted in a corrupted install of data services - not good !

To fix this, I did the following:

Open a command prompt as administrator.

Manually set the TMP and TEMP variables to point to another location (in out case f::\temp).  I then ran the patch executable from this command prompt session.  This successfully uncompressed.  I then did a 'repair' of my data services installation which fixed the problem.

I hope this saves you from having the same issue !




Happy New Year !

I'm back at work and getting into a few new things.  SQL Server 2012, specifically Tabular models.  Also upgrading SAP Data Services to the latest patch set.

What will 2013 bring ?  Will the big data hype settle down.  Will the NO-SQL crowd be subsumed by the NEW-SQL crowd ?  Will the industry come up with some more buz-words (That one's a definite !)

Anyway, happy new year to my reader(s) !




Thursday, 6 September 2012

Top links for SAP BO Data Services

Just wanted to post the main SAP BO Data Services links I use.  If you are learning data Services, these should be part of your regular reading:


Hope these are of help to you !


Friday, 30 March 2012

Data Services: RTFM saves a stressful morning !

Given I just spent a stressful morning reinstalling IPS bause I didn't read the fine print (Short version - don't run setup.exe), I thought I'd share this helpful series I found:


Installing SAP Data Services 4.0 : Part 1 - Information Platform Services

Installing SAP Data Services : Part 2 - Data Services

Wednesday, 21 March 2012

Hello Blog..

Where does the time go ? I thought it must have been about 6 weeks since I last posted - turns out it is closer to 6 months !  Appalling !

Anyway, I've been deep into SAP BO Data Services 4 and am really enjoying working with the product.  I hope to make some more focused posted around this soon.

Hopefully not in another 6 months...



The 5 Qualities of Remarkable Bosses

Interesting article...

Friday, 11 November 2011

Implementing an Agile ETL process in SQL Server


I was fortunate enough recently to attend the DW 2.0 Conference in Sydney.  One of the presenters, Therese Ahlstam (from the Genesee Academy) advocated a technique for building data warehouse's she referred to as Hyper-agility.  The idea was to avoid the need to constant schema changes by storing Staging & ODS tables as Key/Value pairs.  To me this seems to fit the Serialisation/de-serialisation design pattern so common in many software frameworks these days.

This idea seemed quite radical to me, but I've been thinking on this ever since and I decided to have a look at how feisible this really is.  As it turns out, while the syntax is a bit tricky, this is not only possible but relatively easy to do in MS SQL Server thanks to the PIVOT/UNPIVOT commands.

Here is an example:

CREATE TABLE SourceTable 
(pkID int, 
Name VARCHAR(50), 
Address VARCHAR(250), 
Phone VARCHAR(30) 
)
GO


INSERT INTO SourceTable VALUES (1,'Bob Smith', '1 Miller St Sydney', '555-1234');
INSERT INTO SourceTable VALUES (2, 'John wayne', '10 california Dr Hollywood', '+1-555-4567');
INSERT INTO SourceTable VALUES (3,' Billy Bowden', '4 Wainui road Wellington', '+64-21-3456789');
GO


--Unpivot the table.
SELECT pkID, KeyCol, ValCol INTO unpivotSourceTable
FROM 
   (SELECT pkID, CAST(Name as VARCHAR(250)) as Name,Address,cast(Phone as VARCHAR(250)) as Phone from SourceTable) ST
UNPIVOT
   (ValCol FOR KeyCol IN       (Name,Address,Phone)
)AS uST
GO


select * from  unpivotSourceTable
ORDER BY pkID, KeyCol

pkID KeyCol ValCol
1 Address 1 Miller St Sydney
1 Name Bob Smith
1 Phone 555-1234
2 Address 10 california Dr Hollywood
2 Name John wayne
2 Phone +1-555-4567
3 Address 4 Wainui road Wellington
3 Name Billy Bowden
3 Phone +64-21-3456789

Note we need to convert all the columns to the same data type to get this to work (which makes sense) .  This is how we would store the data in our Operational data store and/or staging database.

But how do we get back to a regular row when we need it ?  We pivot it !

SELECT pkID, [Name],[Address],[Phone]
FROM 
(select pkID, KeyCol, ValCol FROM unpivotSourceTable) uST
PIVOT
(MIN(ValCol) for KeyCol in ([Name],[Address],[Phone]) )as pST




pkID Name Address Phone
1 Bob Smith 1 Miller St Sydney 555-1234
2 John wayne 10 california Dr Hollywood +1-555-4567
3 Billy Bowden 4 Wainui road Wellington +64-21-3456789


Do you think there is value in this approach ?  I've yet to put it into practice, but if you have, let me know.  I'd be a bit concerned about the performance for very large tables, but I can see how this would speed up development.


Thursday, 13 October 2011

SAP (Business Objects) Data Services

Hi All,

I am about to embark on a project with the Data services tool set.  It seems like a great tool, but there are two things I am struggling to find at present:

  • Any published books
  • Good websites, blogs, forums etc.  


If you have any good resources for me please leave a comment here for me.

Thanks !

Sunday, 27 February 2011

Saturday, 26 February 2011

Great quote !

The beginning of wisdom is a definition of terms. Socrates

Saturday, 19 February 2011

SQL Server 2011 download page

FYI - I've found CTP releases not worth the trouble in the past, but if you are keen to download SQL Server 2011, here is the link.

Friday, 18 February 2011

MDM in SQL Server 2011

Similar to my earlier post on SSIS, here is the page for Master data management.

Whats new in SSIS 2011

I've been looking for some information about what is coming in SQL Server 2011 for SSIS.  There isn't a lot out there, but what Microsoft has released is available here.  If you know of a better source of info, please let me know !

Saturday, 3 April 2010

Best etl for complex files

Further to my last post, if anyone knows of an etl tool that eats complicated proprietary files (multiple lines per record, arbitrary format) please drop me a line !

Thanks

Wednesday, 31 March 2010

Re-learning perl

I haven't used perl much in the last 5 or so years.  I prefer 'modern' languages like ruby but some ETL tasks have come up on our project that are way beyond what can nicely be handled with SSIS.  Rather than having to use SQL as a text parsing language, we have decided to use perl.

An example of the type of problem we are facing is this:  We get a file where the first character determines the format of the rest of the file.  To get a complete transaction, you need to process 3 lines.  For example we have (this is for a telco system):

1[TAB]123345[TAB]Acme Co
2[TAB]+61290099009
3[TAB]+61380088008[TAB]OUTBOUND[TAB]0.56[TAB]0.09

Where [TAB] means a tab character.

So we get the account number and customer name on line 1, calling number on line 2, the called number, call type, duration and cost on line 3.  There is a 1-N relationship from 1 to 2 and 2 to 3.

so rather than write the most complicated cursor ever we are using perl.  The thing is i'm a bit rusty.

I'm interested to know:
a)  What are some good books for re-learning perl (as opposed from starting from scratch)
b)  Do you agree with the approach ?  how would you tackle this problem ?

Leigh.

Sunday, 14 March 2010

Mobile blogging

I just got a new mobile blogging app. Doesn't really suit the more detailed posts I prefer to make but I'll give it a try !

My boss resigned last week largely due to the lack of focus management have been giving BI. Looks like a turbulent period ahead. Hopefully it gives management the wake up call (I believe) they need.

Thursday, 11 March 2010

Can your Business Intelligence team be too good for your organisation ?

Firstly, I don't think for a second I am too good, but I suspect my team is 'too good' for our organisation.

What do I mean by this ? We have been quite a successful business intelligence team delivering to various areas of the business for the last couple of years so when management decided to put in a new multi-million dollar billing system we were brought in from the start to integrate this into out BI world.

And here is where things started to go wrong.  Because we have more experience and skill in extracting data from our various systems, we were quickly roped into the data extraction, cleansing and migration tasks.  Because the application vendor was too expensive in developing interfaces to our other systems and to external organisations, these became deliverables of the data warehouse.  It has ended up that we are now responsible for 75% of technical tasks on the implementation of this new billing system.

Now we are receiving a lot of praise and thanks from management, but we no longer have capacity to cover the BI work we had planned for this financial year.  Our people are getting frustrated and we are starting to lose staff.

The billing system still has another 3-6 months of work to complete (not counting any BAU tasks we may be left with at the end) and most of our team are out of patience.

I have tried to think of ways we can get out of this and back to our core area, but short of us screwing op royally (which we would never do intentionally) I can't see how we can convince management to let us return to our core area.

Any ideas ?  Have you been in this situation and what did you do about it ?

Wednesday, 25 November 2009

Inspiring...

I just finished reading William Kamkwamba's book - The boy who harnessed the wind. It is truely an inspiration - I'd recommend this to everyone, especially if you tend to make excuses for why you haven't achieved anything.

I hope a bit of it rubs off on me !

Friday, 25 September 2009

Peak oil & end of the information age

Up until fairly recently I would say I've been very ignorant around oil. I had heard of peak oil, but had no understanding of what it really meant beyond the price of petrol and the environmental impact.

But I have recently done a lot of reading of the subject of peak oil and now understand that that is just the tip of the iceberg. Our society is completely dependent of cheap power. Due to the continuing growth in both the developed and developing world, demand increases every year, but supply does not. On day soon - maybe tomorrow, maybe in 10 years, the amount of oil coming out of the ground will reduce. This will permanently change the dynamics of the world - we will find alternative fuels, but they won't be anywhere near as cheap, at least in the short term. Think of all the manufacturing that has been moved out of western countries - it only makes sense because it is cheap to transport to where the customers are. This applies to food also - it won't be economical to transfer fruit from South America to Europe for example. So now we come to this thing - the internet. The biggest cost (when you add the manufacturing, transport, and running costs) of this is power related. Will is continue to be feasible to ran suites like face-book of twitter when costs double of triple ?

I think there is a really danger that a lot of what we do will not last in the post oil world.. The millions of people sitting at PC's in office buildings may need to head out into the real world again and start growing and making things.. which might not be so bad after all !

Thursday, 17 September 2009

SQL Server Integration Services 2008 64 Bit frustrations

We recently needed to purchase new infrastructure for our data warehousing environment. As we aren't likely to do this for a while, we decided to bite the bullet and go 64 bit everything; OS & SQL Server 2008. This has been a much harder than were envisaged and given how straight forward 64 bit UNIX is, somewhat disappointing.

The first issue we became aware of is there are no 64 bit excel drivers available yet. Apparently they will arrive in Office 2010, which given we are using Office 2003 at the moment, won't arrive here for another 6 years. This has meant for about 1/2 our packages they will only work in 32 bit mode.

Next we found that BIDS only runs in 32 bit anyway, so if we addressed the above issue, we would have problems maintaining these packages anyway. This pretty much says to me that at least from an SSIS point of view, 64 bit SQL Server is not really for prime time.

Finally we had a lot of problem installing 32 bit ODBC drivers. They installed fine, but we then couldn't find them in the odbc administrator. It turns out their are 3 (maybe more ) obdc administrators in Win Svr 2K8 64bit. Solution: Using the odbcad32.exe located in %windir%\syswow64 .

I have seen the Microsoft adds that state the WOW starts now. Maybe they were referring to that directory...

NB: This process is ongoing, so I will update this post if we discover any more gotchas...

Monday, 6 July 2009

Best practice for slowly changing Fact tables.

I've been trying to find some evidence of best practice for facts which are updated and have been unable to find much written on this subject. While this is quite common for case management type systems (e.g. HP Service Centre, Salesforce.com), it is not typical for data warehousing in general, as most literature focuses on more traditional transaction based systems (e.g. bank accounts, PABX data, order/billing etc.).

The challenge we have then is to determine when the dimensions of a fact record should be changed, and when they should be left unchanged if the fact itself is altered. Obviously for SCD Type 1, this is a non-issue; we aren't keeping a record of the previous Dimensions value, so the fact must be linked to the newest version anyway. But for SCD Type 2, there are three possible options:
  1. Determine the Dimension at the first instance of the fact being recorded in the warehouse.
  2. Update the fact's dimension's to the version of the dimension that existed at the time of the fact being modified.
  3. Update to the Current version of the dimension at the time the fact is updated in the warehouse.
While 1 is the simplest to implement, it doesn't reflect the real world usage of many source systems. For example an address Dimension may start of with very minimal details initially, but will gain more details over time. It is unlikely that we would want to stay with the minimally populated dimension once we have richer data available.

Option 2 to me is probably the best 'default' option in this case. It makes sense to me that if the fact is updated, it should reflect the dimensions at that point in time. For example if sale was made to the sales rep for region 1 (e.g. 'Bob') and Region 1 is later re-assigned to 'Fred', we don't necessarily want that sale updated to 'Fred' as he never made that sale. I'm sure there are cases though where this may not be the right approach.

Option 3 to me in most cases would give an incorrect view of the data, so therefore is not what we want either, however it may be that in the source system certain data is only updated periodically and that for that system the most recent version is deemed to be the 'correct' version.

I'm include to use Option 2 in most cases, but I would appreciate any feedback on this, as there may be other scenarios I haven't considered. Also, if you have come across any articles linked to this, please pass these on to me.

What are your thoughts ?

Monday, 27 April 2009

10 skills developers will need in the next five years

Click on the title for the original article. Here is how I score on the list:

1: One of the “Big Three” (.NET, Java, PHP) -CHECK
2: Rich Internet Applications (RIAs) - FAIL
3: Web development -A BIT
4: Web services -A BIT
5: Soft skills -CHECK
6: One dynamic and/or functional programming language -CHECK
7: Agile methodologies -A BIT
8: Domain knowledge -CHECK
9: Development “hygiene” -CHECK
10: Mobile development -A BIT

So I give myself 7/10. Need to geek up a bit !

Thursday, 26 March 2009

iMovie 08

I'm currently learing how to use iMovie 08. Despite all the criticism it gets, I am starting to like it. I was wondering though, is there any way to add functionality ?

Also, are there any public domain clip libraries I can use to enhance my projects ?

Thursday, 12 March 2009

Off topic...

Just found a great website for money management. Click on the litle to have a look...

Monday, 19 January 2009

Gartner's 5 predictions for BI in 2009 and beyond

I'm not a huge fan of Gartner, due to the cash for comments nature of their business, but I thought the following was an interesting piece. Of particular relevance, I found the last point one that I identify with:

"And Gartner's fifth prediction is that by 2012 one-third of analytic applications applied to business processes will be delivered via coarse-grained mashups. IT shops are no longer focusing on grand visions such as SOA, which involve building composite applications out of fine-grained services or portals that merely display operational and analytical information next to each other, the analyst firm explained. Coarse-grained mashups, on the other hand, can be used to overlay analytical insights."

I interpret this, as tactical solutions will be successful, where as big bang solutions will fail. In my experience, that is almost always the case!

a Big gap...

I haven't posted anything for a few months. Sorry for the gap. I have started a new job (based around SQL Server and SSIS) and have a new daughter, so I have been a bit busy !

I'll try to make more regular posts this year - promise !

Friday, 17 October 2008

Recession proof IT jobs

If like me, you've been on the job market recently, you may be a little concerned about taking a job and then being made redundant down the line. Its for this reason I think that people tend to stay in the one place a lot more whne things are tough.

I found this link, to supposedly recession proof IT jobs. Thankfully, database administration is on the list. Lets hope its right !

Thursday, 9 October 2008

If only it was real !


I know this is obviously, fake, but I saw it today and thought it was pretty funny. Bill has left Microsoft, so who knows what he might do in his spare time !

Thursday, 26 June 2008

Sybase on jruby on rails

It goes without saying that most of what I do is on Sybase. I was looking to build a POC for a data maintenance app a few weeks ago and I decided to try and do it with rails. I quietly realised the only option for using rails in my company would be jruby on rails deployed to tomcat; that is from their point of view it would be a java app.

So I updated my jruby environment to the latest (1.12 at the time of writing) and activerecord-jdbc-adapter to 0.81. As luck would have it some preliminary Sybase code had just been included in this release.

It didn't seem to work, however so I did some investigation. It was clashing with the MS-SQL driver (both can use the jtds driver, and it seems no one had considered the idea of one driver talking to different databases before). This started my interaction with the jruby community. People were extremely helpful and immediately made a change to how the database config worked to add a 'dialect' keyword so we could now use both sybase and MS-SQL with the jtds driver !

I was pretty pumped by now, so I set up the required files for running the simple unit tests, create a test sybase database (15.02 on XP) and started testing. Here is what I went though:

First run - 100% failure
Solution - the primary key definition was gibberish from sybase's point of view. By looking at what the MS-SQL driver did I was able to work out what was needed, added an overriding method and re-tested.

Second run - 70% Success !
This motivated me a lot so I addressed the next issues (drop index syntax) by adding another method and retested.

3rd Run - 99% success !
I was basically there. The last issue required no ruby code changes - just setting 'select into/bulk copy' on for the test database.

4th run - 100% success !
Ok, well there were lots more than 4 runs due to typos, database issues,etc. but only 4 that were relevant to getting the driver passing all the tests.

I have submitted a patch (article title links to the Jira ticket) and hopefully this will make it into an upcoming release. Is you don't want to wait, you should be able to apply the patch to activerecord-jdbc-adapter 0.82.

Not that I never have actually gotten to the rails app yet.. Hopefully soon. Now back to my day job :-(

Thursday, 22 May 2008

Sybase gets a world record for the largest datawarehouse

Well done Sybase ! Hopefully it gets to print before the Microsoft & Oracle Benchmark factories churn out their own world records.

Wednesday, 12 March 2008

ASE 15 vs MySQL 5

I'm back at work after a 3 month break (not long enough !). Here is a comparison I've found of ASE to MySQL (click the article title). I can't say whether or not its accurate, but Its worth a read.

Hopefully I will be making more regular posts from now on...

Thursday, 15 November 2007

Erlang

Ok,

I decided, I'm learning Erlang. I've installed it (port install erlang - too easy) and I've order the pragmatic programmer's book. If you have any tips, pitfalls or general suggestions, drop me a note.

One downer is a quick search this morning came up with no jobs advertised for Erlang in Australia. I hope that changes soon. If you know of any Australian companies that use Erlang, please let me know !

Wednesday, 7 November 2007

Seven habits of highly effective developers

While I do some of these, I am going to try apply these next time I start something and see if it makes a difference...

Friday, 19 October 2007

What is the easiest/best first functional language to learn ?

I'm planning to teach myself a functional language, but I'm not sure where to start. Thinks that have been catching my eye are Erlang, Haskell or Scala.

If you have any advice, please let me know !

Wednesday, 26 September 2007

(Spenser) for Hire !

Ok. Obscure reference.

What I'm trying to say is I've decided that in a couple of months I will move on from my current position and after a couple of months break I will be looking for a new position. If you think you might have anything for me starting Feb/March next year, let me know !

Wednesday, 15 August 2007

Mobits Adam A221Tablet PC - Help !

This is a bit off topic, but...

I bought a Mobits Adam A221 mini-tablet PC on Ebay. My plan was to get Linux working on it, but so far I have been unable to get either the WiFi or the Tablet/Pen interface working.

There seems to be next to no info on these on the web - if you have one and you have had any success could you let me know ?

Under windows the drivers it uses are:
Tablet - Fujitsu touchscreen driver. It is running on COM1
WiFi - Uses PRISM USB drivers.

Any ideas ? If I spend anymore time mucking around with this my wife will kill me !

Tuesday, 7 August 2007

The cost of hiring poor programmers

In Sydney, where I live, there is a real skill shortage at the moment. Even highly profitable companies (who one assumes can afford it) are having trouble getting good people and filling roles.

One thing I have seen several times is that when companies can't find the person they want, they lower their expectations. They keep doing this until they can hire 'someone'.

This has several negative effects on the company, and not many positive ones:
  • The person can't possibly fill the requirements the company had in the first place.
  • Existing Employees will resent the fact that inferior people are being bought in, pushing more work to them without them receiving the benefits (which in a buyers market the new, inferior guy gets).
  • 2 leads to more turnover, which can lead to more of these people being hired, which can cause things to snowball.

While I never appreciate being overworked, I really don't appreciate making up for people who aren't up to the job.

My view is if you can't find the right people you need to look at why you aren't appealing to those people. And if you are budget constrained don't throw your hands up in the air. Money is rarely the biggest factor in these things. You need to ask yourself what sort of people you need and what those people want. Things to think about include:
  • Leave/working hours - not everyone is a 9-5*5*48 weeks a year kind of guy. The best person working 3 days a week or 9 months a year, is a far better asset than an inferior person doing 9-5 all year.
  • work environment - If you insist your storage engineer wears a suit you will miss out on all the good storage engineers. If you insist your LAMP installation is deployed on a Windows, you will miss out on all the good Apache/Perl/etc. people.
  • Telecommuting. If you trust people enough to let them work where they are most comfortable, they will do more work. If you don't trust your employees to work from home they won't trust you to reward them for the work they do.

So don't settle. Become the employer people want to work for !

Monday, 30 July 2007

%$&*@!

Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaargh !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Tuesday, 17 July 2007

101 Ways To Know Your Software Project Is Doomed

See link - This would be funny if it wasn't largely true

Friday, 13 July 2007

World's youngest criminal ?


I had to get my baby daughter a passport so taking a suitable photo was a challenge. I think it looks like a mug shot. What do you think ?

Why do Management get to make the technical decisions ?

I overheard a discussion recently about the merits of various technologies and which direction our organisation should be going in. The thing is, the people discussion it were management and not people who actually have to deliver using those technologies.

How much time and money is wasted I wonder, because the decisions get made by the people higher up the chain as opposed to the people closest to the issues ?

Things never go well when people people aren't expected to 'eat their own lunch'.

Wednesday, 4 July 2007

Jobs Vacancies and me

I get a lot of emails and phone calls from people wanting to talk to me about jobs. I appreciate the interest, but most of the opportunities I am contacted about are not of interest to me. So to clarify:
  • Please don't contact me about jobs in the finance industry. I already work in this industry and have lots of contacts here (unless you are offering 7 figure salaries ! )
  • Please do contact me about jobs in other industries
  • Please do contact me about jobs with flexible hours/days.
  • Please do contact me if your project involves new and interesting technologies.
Thanks !

Monday, 2 July 2007

Signs you (or someone else) are no longer technically relevant

I have come across a lot of people in my career that are well past the point where they can continue to evolve with technology. These people tend to hold an organisation back, and need to be removed from areas of influence if not encouraged to move on completely. I hope I'm not one yet.

Here are what I believe some of the signs of this are:

  • They consider Java a new and inovative technology
  • They don't bother following industry news and developments
  • They are resistant to the introduction of new technologies
  • They worry about whether the open source software you propose will scale while they continue writing excel macros in vb script
  • They think Linux is radical
  • They are concerned about certifications from old school vendors
  • They have been in the same job for more than 5 years

what do you think ? Am I being a bit harsh ?

Friday, 27 April 2007

Keira

I haven't been making a lot of posts lately. My first child, Keira was born 2 weeks ago and nothing else is getting much attention at the moment. I couple of photos should be available here
( http://www.flickr.com/photos/7992966@N04/ ) if i've worked that flickr thing out properly.

Otherwise, I'll get back to you shortly.

Thursday, 29 March 2007

Workers walk away from bad bosses

A survey has found 82 per cent of office workers have quit a job rather than put up with an outrageous boss.

Personally, I have done this twice in my career and should have done it a 3rd time.

One thing that has always disappointed me is that companies seem to be willing to put up with bad managers and see their other staff leave. When the market is as good as it is now, you can easily see teams completely turnover in a year. You would think companies would learn !

So if your a manager complaining about the cost of new staff, ask yourself why the old staff left in the first place...

Friday, 23 March 2007

The Active Record pattern (or why I love Ruby on Rails)

If you don't know what active record is, I am not going to tell you. Google it, and/or read about it on the Ruby on Rails website.

If you are one of those people that think struts and Hibernate are the best combination since the Captain and Tennille (cause I know you Struts people are into that shit :-) ), then you won't like it at all, because it takes you away from messing about with configurations files and makes you work on the problem at hand.

I really buy into the convention over configuration mantra for a couple of reasons:
  • Opinions are like arseholes
  • Most object-only guys can't (or don't bother to) get the database right, and activerecord encourages sorting out the design first
Unfortunately, I don't get to work with Ruby at work, I have to use Java. I've yet to find a fully functional Active Record Implementation for Java yet. If you find one,. please let me know !

Wednesday, 21 March 2007

TPC-C to die

One of the least reliable and easily distorted benchmarks is nearing the end of its life. Having seen this from the vendor's side, the TPC-C benchmarks of so little relevance to real world database performance as to be virtually useless to companies looking to compare the performance of different vendors. What the benchmark does tell you, is who is investing money in their databases, as running a TPC benchmark costs a lot of money. The reason you hardly ever see Sybase or Postgres or MySQL in the top ten is they can't afford / aren't investing to run the benchmarks. Looking at the top ten this morning you see Oracle, IBM and Microsoft filling up the whole list. No surprises there.

Whether the new benchmark (TPC-E) is any better, is the question. It is based on a financial scenario, which may work against Oracle (which has never been great at high volumes of small transactions which are typical of financial systems) a bit, but its unlikely to be much more real-world than the old one. Good companies will have to run their own benchmarks, like they always have.

Wednesday, 14 March 2007

Putt's Law

I have managed to avoid coming across this until today, when I heard and interview with the Author. I couldn't agree more.

"Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand."
—Putt's Law

Monday, 12 March 2007

Cursors (I RANT)

Cursors. Satan's operand.
I hate them. There is nothing more annoying that working through someone's crap code which is failing, to find you need to decrypt their perverted thinking just to work out what is going on.
Apart from the fact that cursors are slow, I have never (I said 'NEVER') seen a case where cursors are necessary. In every case I've ever come across, there has been a better way of doing things. If you need to process data row by row, you can do this better and faster using a while loop, a select statement and a couple of marker variables.
If your reading this and you think I'm wrong, then you need to do a bit more research on set theory and then you may (I hope) come to understand why cursors are such terrible operations for a relational database.
I suspect cursors were invented to allow old COBOL programmers to be able to work with databases.
If I was offered a job where I never had to deal with cursors, and badly written code in general, I would gladly take a pay cut (well, just a small one) !

Wednesday, 21 February 2007

On CIO's (part 2)

Its nice to see other people think the same as I. Slashdot references a CIO article which supports my theory that these CIO's won't last long:

http://it.slashdot.org/article.pl?sid=07/02/20/1924218&from=rss

Tuesday, 13 February 2007

On CIO's

I was reading an article on itnews.com.au about how CIO's are moving ahead of CFO's in some companies these days. I've never seen it myself , but I can't help but thinking this isn't a particularly good thing. In my career I have worked under probably a dozen CIO's (I won't count those when the organisation was a software house or an IT consultancy). Only one of those CIO's I would define as good at what they did (an ironically he was only an 'acting' CIO). If I was to characterise the CIO's I've worked under, the following are true of almost all of them:


  • They started off their careers in technical positions, but moved into the management side of IT as soon as they could.


  • They have never done any study in IT beyond undergraduate level (if your lucky), but always seem to have an MBA or other degree that looks good to non-IT people.


  • They don't understand current technology at all, but do understand vendor management.

  • They give the impression that they see being CIO as a stepping stone into becoming CEO (although i've never seen that happen).

  • Their direct reports are wannabe CIO's themselves, so all the above points are true of them also.


All together this paints a pretty ugly picture. These people are so out of touch, and it shows. The end result is you only ever get one of two messages from a CIO, ether:

  • "We need to cut costs and place better controls on out environment"

or

  • "We need to invest more and allow our people the freedom to innovate"


Thus it ends up being left to lower level people to run the organisation as best they can in between doing their real jobs.

Personally, I think the days of these CIO dinosaurs is slowly coming to an end. But it isn't us I.T. people that will bring them down.

Its the Marketing manager who asks why he can't install FireFox, only to realise the CIO has never heard of FireFox. Or the research analyst who it prohibited from using Python to develop their financial simulations because the CIO thinks Excel is what he should be using. These CIO's now know less about IT than your average person under 40, and once tech savvy non-IT people becomes CEO's these technically incompetent CIO's will disappear.

Good riddance to bad rubbish !

Monday, 12 February 2007

MySQL vs Sybase / MS SQL Server – a comparison

(Migrated from old website)

As I database consultant that has been working with Sybase and MS SQL Server for a long time I have been interested to see the rapid growth of MySQL. I decided to evaluate MySQL from the point of view of porting a database to it from Sybase/Ms SQL Server.

For those of you who don’t know, the two (Syb & MS) are still about 90% the same, at least from a Transact SQL point of view. They also both still come with that old friend – the pubs database (pubs2 & 3 on Sybase, pubs on MS SQL Server). Therefore what better to use for my sample database as pubs ! Note that I will not provide my converted script due to copyright issues, but if you are a Sybase of SQL Server user feel free to do the same. This is a very good test as pubs has been purpose build to try and use many of the different features available. Here is the results of my test starting from the point of my download of MySQL 5.0 alpha (the newest version available when I started this) :

Steps

1. Install MySQL, client tools & documentation.

2. RTFM for about 10 minutes.

3. set up my server using the winmysqladmin program. It gives me a working server ! Connect to the server: from a command prompt, type ‘mysql’ (no options).

I'm in a 'test' database. Follow a basic tutorial from the manual.

Learn a few basic commands like "show databases;" "select now();", & "select user();" - This seems pretty easy !

4. Decided to create a database. I'm going to create pubs2. Following the tutorial I find the syntax:

"create database pubs2;" Fails. Go back to RTFM.

5. Find out that there is a "test" user that I am connected as with no privileges. Find out I should be using the 'root' login. connect as root with no password:

mysql -uroot mysql. Follow recommendations to set a root password:

SET PASSWORD FOR root@localhost=PASSWORD('new_password');

Log out and login with the password:

mysql -uroot mysql -pnew_password

I'm in. Now try again:

create database pubs2; - Success !

Run "show databases;" - its there.

6. type "use pubs2"; - It works !

7. Cut and paste create table authors... from instpubs2. Get an error:

ERROR 1064 (42000): You have an error in your SQL syntax. Check the manual that

corresponds to your MySQL server version for the right syntax to use near 'id not null,

au_lname varchar(40) not null,

au_fname varchar(20) not null,

ph' at line 2

Sounds like a simple error, as id is a user define data type. Don't know how

to create those yet, so I'll replace it with the underlying type - varchar(11).

8. Try again - Success ! Now for Rest:

Publishers - ok

royshed - replaces tid with varchar(6), then ok.

sales - ok

-- Decided to work ou how to create udfs so I don't have to keep editing the scripts.

Spent about 1 hour earching the web and manuals. It doesn’t support them. Ill continue the way I was.

salesdetail - ok

titlesauthor - ok

titles - Error:

ERROR 1064 (42000): You have an error in your SQL syntax. Check the manual that

corresponds to your MySQL server version for the right syntax to use near 'money null,

advance money null,

total_sales int null,

notes varchar(200) null' at line 6

Thats right - no money datatype ! That wouldn't be a big deal if we had UDFs, but we don't. More re-writing ! (I’ll use decimal) and I am going to try replace image with the blob type. I don't know what the difference really is, but this is just a basic test. It works.

9.

After getting through the tables. I get to constraints. Primary key constraints seem ok, but foreign keys can’t be created. After looking on the manual I find I need to create indexes before foreign keys. I do this, and can then progress. At this point however alarm bells are ringing about how it handles constraints. The manual is warning me that when using not null and default constraints that behaviour will not be as I would expect. For example if you try to insert a null into a not null column, MySQL will replace the null with a '0' or a '' (empty string) rather than rejecting the insert. Is sort of defeats the purpose and I would avoid using these features until they 'fix' them. The manual says you can recompile MySQL to change this, but doesn't make

it clear on what the new behaviour will be.

10.

We need to skip some default clauses here. This requires editing the table creation scripts and manually placing the defaults on the relevant columns, or editing the inserts to include this. I have chosen to edit the data. Note I also had to replace

some occurrences of "" with a single ' as I am not sure how MySQL handles double quotes inside double quotes.

11. Indexes. MySQL doesn't seem to support the clustered/non clusters index types. I created the indexes without specifying the type and it works fine. The manual says that all MySQL indexes are b-trees with some compression.

12. Rules. Rules are not supported. I am going to ignore them as I think the only way to work around them would be to put the functionality in the app (not desirable) or in triggers.

13. Triggers. The manual says triggers are scheduled for 5.1. I can confirm they don't work in 5.0. I am having to push a lot of functionality back to the app (it theory – I’m not actually building an app).

14. Views. Not yet supported. Should be in 5.1 Derived tables (unnamed views) are supported, but this doesn’t help here.

15. Stored procedures. Apparently they are supported in 5.0 but I need to recompile the server. My C skills are probably not us to this, and it obviously isn't ready for prime time is it isn't in the server, so I'll forget them.

16. Grants. Grant all succeeded. Grant create table etc... didn’t. Give the number of things I have had to leave out, I don’t think I’ll worry about fixing this, but the manual suggest you can grant these things at the user level.

Other notes:

  • Temporary tables have limited functionality and cannot be used in the same way as a real table. For example the following doesn’t work:
SELECT * FROM temporary_table, temporary_table AS t2;
  • BCP like functionality is provided by the load data command. I only did some basic testing, but is seemed very fast (faster than bcp).
  • The security model seems quite basic; groups and roles seem absent, so granting seems to need to be done on each user.

So what do I think of MySQL ?

This product has great potential. By the time version 5.1 becomes a stable production release it will be usable for a lot of applications. By the time 6.x comes out I would expect they have caught up in the important areas of data and domain integrity to the point where is will become a feasible option to companies out their. It wasn't that long ago that people tried to write off Apache & Linux. I can see MySQL being the next big thing in Open source, if it isn't already.

Having said that, as it stands now (5.0) I would not be willing to use MySQL for anything more critical than a reporting database. The lack of system wide rules, defaults and the problems with constraints mean that you would have to build all

of your referential and domain integrity into the app. That’s a DBA's worst nightmare!

It is worth noting here that there is an offshoot of MySQL called MaxDB (formally SAPDB). This has been developed by MySQL and SAP and has been developed to try and address these enterprise limitations. I have yet to look at it, but if I have time I will also try and conduct the same exercise against this.

The future of Sybase

(Copied from my old website).

People often ask me if Sybase will still be around in X years. The short answer is absolutely.

Firstly, this is my own personal opinion. As I used to work for Sybase I am somewhat biased, but I am presenting my reasons here and you can judge for yourself. Note also my figures are just to illustrate my point. If I did know the real figures (which I dont) I would not be allowed to use them anyway.

Since I first started working for Sybase around 7 years ago Sybase has grown in terms of RDBMS sales every year. "What ?" you say, I thought Sybase had lost market share. That really depends on what you are asking. The database industry (as has the whole software industry) has been growing every year for decades. There has been an explosion in the number of RDBMS systems deployed over the past 10 years and Sybase has grown with it. What you have probably heard (which is true) is that Sybase has lost market share as a percentage of market size. So when the world wide market for RDBMS systems was say, 500,000 (for arguments sake - I dont know the figures) Sybase had 20% of the market, or 100,000 customers. Due to Market growth the world wide market is now at least tens of millions (a conservative guess I would think). So if the worldwide market is now 20,000,000 (which does not seem as unreasonable guess) and Sybase has 3% of that market then Sybase has 600,000 customers - growth of 600% !

This of course refers to Enterprise RDBMS sales alone. Sybase sells over 100 other products (including several other databases) and some of those products are the most popular in their industry segment, for Example Adaptive Server Anywhere, which has around 70% of the mobile and embedded database market. Sybase is doing quite well really.

Update: Since I wrote this a few years ago, Sybase has acquired several companies, ASE 15 has been released and Sybase has continued to do fine.

My First Post

Ok. If you've ever looked at my website before, you'll know I'm not very good at keeping it up to date. The solution I have come up with is to not even try.

From now on I'll replace it with this blog. Hopefully I actually keep this up to date !

Cheers,

Leigh Kennedy.