I’m recruiting a SQL Server DBA

I’m looking to recruit a SQL Server production DBA for my current employer, dotDigitalGroup. If you fancy coming and working on a big sector leading SaaS project with some very intelligent developers and technicians, check out the advert here

http://careers.stackoverflow.com/jobs/22183/sql-server-dba-dotdigital-group-plc?a=qcXiuBO

and our dedicated job site with details about the team here

http://www.dotdigitaljobs.com/

SQL Server 2008 R2 cluster setup fails on mount points

(Update from Monday morning, the workaround shown below works for me)

Another day, another set up problem (sigh) I was installing a second SQL 2008 R2 instance on a new cluster today. The first one had gone well without incident but the second one had a crucial change in that it used mount points. It was a fairly complex install with 8 mount points under the root volume and it took a while to complete all the setup configurations. When it got started it failed with the following selection of errors:

1. A dialog box with the error

Wait on database engine recovery handle failed. Check the SQL Server error log for potential causes.

2. The setup finished and pointed me to the usual error log files in the setup directories. In the summary log file I saw an equivalent error

Detailed results:
Feature: Database Engine Services
Status: Failed: see logs for details
MSI status: Passed
Configuration status: Failed: see details below
Configuration error code: 0x4BDAF9BA@1306@24
Configuration error description: Wait on the Database Engine recovery handle failed. Check the SQL Server error log for potential causes.
Configuration log: C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\20120420_114647\Detail.txt

3. When opening the referred to detail.txt file I found pretty much the same again

2012-04-20 12:17:28 Slp: Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.
2012-04-20 12:17:28 Slp: Wait on the Database Engine recovery handle failed. Check the SQL Server error log for potential causes.
2012-04-20 12:17:28 Slp: The configuration failure category of current exception is ConfigurationFailure
2012-04-20 12:17:28 Slp: Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.

4. The text file pointed me to the SQL Server error log of the partially installed instance. When I opened this, I found the “real” cause

2012-04-20 16:00:51.51 spid8s Clearing tempdb database.
2012-04-20 16:00:51.52 spid8s Error: 5123, Severity: 16, State: 1.
2012-04-20 16:00:51.52 spid8s CREATE FILE encountered operating system error 5(failed to retrieve text for this error. Reason: 15100) while attempting to open or create the physical file 'O:\data\tempdb.mdf'.
2012-04-20 16:00:51.52 spid8s Error: 5123, Severity: 16, State: 1.
2012-04-20 16:00:51.52 spid8s CREATE FILE encountered operating system error 5(failed to retrieve text for this error. Reason: 15105) while attempting to open or create the physical file 'O:\data\tempdb.mdf'.

The SQL Server error code 5123 is why the instance couldn’t start, as it’s unable to create a tempdb database data file, and no SQL Server can start without tempdb being present. The “real” error though is the reason for this occurring which is error code 5 from the OS when SQL runs the createfile API. This is one that everyone probably recognises which is “access denied”. So the bottom line here was that I was getting access denied on the root of my mount point for tempdb (which is where I had placed the data files as part of setup configuration).

I checked through the other parts of setup and the rest of the system databases had managed to write to the root of other mount point drives (which seemed strange), and of more relevance, I noted that the SQL Server service accounts had been granted full access to the mount point root directories as part of the setup, so theoretically there ought not to be a permission error!

I spent a few hours digging around, tried the install again with new mount points and a clean position, but encountered the exact same problem. Eventually I figured that it was due to the fact that there are “issues” with the permissions on root mount points. This is distinctly not well documented within official MS sources (or not that I could find) and certainly not within the official pre-reqs.

http://msdn.microsoft.com/en-us/library/ms189910(v=sql.105).aspx

The best I could find was a connect item here

http://connect.microsoft.com/SQLServer/feedback/details/569895/installation-of-sql-server-2008-r2-on-mount-points-fails

a blog post here

http://getyouriton.blogspot.co.uk/2009/08/serious-gotchas-with-mounted-drives-or.html

and a forum post here

http://social.msdn.microsoft.com/Forums/en/sqlsetupandupgrade/thread/da05e467-6852-4e2d-8b8f-d253b479acfc

However the accepted workaround is that you should create a sub directory under the root of the mount point and then you’ll be fine. I’ll be trying this method next week as its 8pm on a Friday now and time for the weekend. If it works I’ll probably post a bug requesting that the documentation gets changed to point this out, as it would have me a whole lot of time.

As always, good luck with your setup……

Example of SQL Server collation usage and problems

I was discussing with some developers this week why SQL Server collation mattered and what are typical things that can go wrong. This is a huge topic and is an often over looked feature of database and server design, especially when you want to build applications that run in multiple geographic locations and that should accept multiple language inputs. Being as I’ve worked extensively with the Finnish Swedish collation from my time working in Sweden, I have a reasonable level of understanding of some of the issues that can come up.

I wrote the following script to demonstrate an example of what can happen if you don’t get your collations correct. It’s probably the most common example I’ve seen over the years.

1. You have a SQL Server that is running one of the main Latin collations for English, be it UK, US, current windows collation or legacy SQL Server ones.

2. You have data from multiple languages stored in unicode columns such as nvarchar

3. You sort or search on the data and those people who have used data from non English languages that have special or extra characters in the alphabet, do not get the results that they expect.

This script only shows a fraction of the things that can actually go wrong, but it provides a simple demonstration of sorts and searches producing unexpected results. It creates its own database and tables on any test server you may choose to run it on (and then deletes them afterwards). It will work as a demo on any server as long as it’s not already running a Finnish Swedish collation.


use master
go

--i'm creating a non-default latin collation as per the legacy product I was looking at
--in this case however the results are the same as if you're running the more common default windows server collation 'Latin1_General_CI_AS'
create database collation_test collate SQL_Latin1_General_CP1_CI_AS
go

use collation_test
go

select SERVERPROPERTY('collation') as 'server collation'

select collation_name as database_collation from sys.databases where name = 'collation_test'

if exists
(
select name from sys.objects where name = 'collation_test' and type = 'u'
)

begin
 drop table collation_test
end

create table collation_test
(
test nvarchar(10)
)

set nocount on
go

insert collation_test (test) values ('aaaa')
insert collation_test (test) values ('ääää')
insert collation_test (test) values ('åååå')
insert collation_test (test) values ('öööö')
insert collation_test (test) values ('bbbb')
insert collation_test (test) values ('zzzz')

set nocount off
go

print 'select the results from the tables in differing collations'
print 'in Swedish alphabet the characters åäö follow after Z as characters 27 to 29'
print''
select test as "order in latin collation"
from collation_test
order by test

select test as "order in Finnish_Swedish_CI_AS collation"
from collation_test
order by test collate Finnish_Swedish_CI_AS

print 'do searches on the table'
print ''

select test as "search in latin collation"
from collation_test
where test > 'b'

select test as "search in Finnish_Swedish_CI_AS collation"
from collation_test
where test > 'b'
collate Finnish_Swedish_CI_AS

--clean up
use master
go
drop database collation_test
go

The bottom line is that if you want to do collations and mixed languages properly, you need to think about this stuff carefully at the design stage.

Some thoughts and follow up to KAM careers fair at KTH Stockholm

The last 3 weeks of my UK day job have been a tad boring, but this is the case sometimes. I’ve just been trying to get a task out the door which involved lots of large TSQL code for reporting and data analysis. This type of thing leaves me cold and is not something I get involved with very often (well not in this century at least, I used to do this stuff in ’97/98 but hey….it was a favour for someone and it’s nearly done now!) anyway fortunately I was in Stockholm last week to talk at the Royal Institute of Technology careers days (Kungliga Tekniska högskolan Arbetsmarknadsdag for those Swedish speakers amongst you) which was a fantastic time.

I met some seriously intelligent people and enjoyed chewing the fat with many of them about starting out a career in the IT sector. I hope that I provided a rather different outlook to some of the speakers and exhibitors at the conference as I left school at 17 with no higher education whatsoever (let alone a bachelors or masters degree) and it took me 6 years or so to even get someone to give me a job remotely close to IT and another 2 to actually get myself placed in a real IT job in a software house. Add to this my rather varied career paths since which have veered between investment banks to year one start-ups and Microsoft, and I hope that I gave a different perspective to some of the students. it certainly felt like a success and the company I was representing (Basefarm – who I still consult for regularly) received a large number of applications for internships and employment.

As well as being on the Basefarm stand all day just chatting to whoever came along, I did a talk in the afternoon entitled “This much I know……” where I expanded on some of the above thoughts and compared and contrasted a number of different companies, company types, job types and locations and what they were like to work for, and how they had helped or hindered the development of my career. Hopefully those present learnt a few tips to help them mould their own career paths. I try to not make the talk into a lecture, because I don’t want to give the impression I know too much, or that I’m teaching people how they should act, it’s far more an approach of hoping that people will pick up tips and trends based on some of the things I’ve experienced along the way (both good and bad).

I’ll probably be doing similar stuff at other universities in Sweden over the next year so, but I’d be happy to be involved in any UK based ones as well, so if you’re organising a careers fair in the UK, please drop me a line if you’d be interested in having me along. My current employer in the UK, dotDigitalGroup, has taken on a large number of IT graduates over the years and I think it’s a testament to the company the number of technical employees that they still retain where it’s their first job after university. It’s quite uncommon in my experience for an organisation to be able to maintain such loyalty and I think it speaks volumes for how much people like working there. I think we’d be able to share some interesting career thoughts and possibilities with UK based students, so feel free to drop me a line if this sounds interesting to you.

Business card printing by moo.com

I printed some new business cards for GlobalGoat Consultants this week, using my shiny new logo as designed by http://www.digitalgreen.co.uk/ . I used http://uk.moo.com/ for this service and can say that I was really happy with the results.

Their interface was both simple to use and suitably variable enough to allow me to get exactly what i wanted and their prices and quality of cards were excellent. I’ve used them before for other vanity projects in the past because I like the fact that as well as using your own images (or their stock ones if you like) you can use anything from your flickr library. I have thousands of flickr images built up over the past years and I used a selection of 8 of my favourites (mixed landscapes of the UK and Sweden) for the back of the business cards.

If you come and chat to me next week at Kista arbetsmarknadsdag then you might even get one 🙂

I’ll be speaking at Kista Arbetsmarknadsdag (KAM) next week – 28th March

I’ll be speaking at Kista Arbetsmarknadsdag (KAM) next week on 28th March. It’s not a technical talk since it’s a career day for students, it’s a “this much I’ve learnt” type of talk about career building in the IT sector. I’ll be there representing Basefarm AB as I help then with their windows recruitment, as I know the recruitment market and the company from  my time in Stockholm.

I’m lucky enough to have had a fairly varied career in the IT sector, ranging from start-ups to Microsoft and in the talk I just compare and contrast some of the different things that come along in the course of an IT career.

I’ll also be on the Basefarm stand from time to time during the day, so please come along and say hello if you’re attending.

LAMP for a beginner, lessons from my WordPress build

When I finally got round to rebuilding this site into its current format, I searched around for a while before choosing WordPress as the platform to put it together with. As a complete newbie to this platform, and also as a person who has mostly used Windows software for the past 15 years, there were a few things to learn along the way, and here’s some information about some of those points.

Firstly this breaks into 2 areas, client and server. When I say client in this case I refer to a laptop I use as a general machine for mail and stuff, but which is also the development environment. This machine is a very old Toshiba M70 from 2005 running a vanilla install Ubuntu 11.10. Being as a was a complete novice in terms of putting LAMP together I looked for suitable instructions online and found this

http://jaredheinrichs.com/the-ultimate-ubuntu-wordpress-development-machine.html

Which I can’t recommend enough as a great step by step guide. I had one problem on this which I documented in the comments of that article (which was not the fault of the article but a subsequent problem / config issue in webmin). The problem was that one couldn’t login to webmin at all after install even with root. When this was occurring I must say that I felt completely useless as I’m such a linux noob that I didn’t really know where to start troubleshooting. If this type of thing occurs in windows I just work it out using various tools, but in linux I’m quite stuck where to start. Anyway the solution was in the ubuntu forums here:

http://ubuntuforums.org/showthread.php?t=1745926

Beyond that the client build was very smooth and is happily up and running on my crappy old laptop.

On the server side I’m hosting with one.com who have been very good to me and were very efficient when I did my domain transfer last year from another provider who I won’t mention here 😉 The only thing you could say against one.com is that you only get one MYSQL database for your entire domain, so this could cause an issue to bigger more complex installs and sites maybe, but works just fine for me. The install was super simple exactly as WordPress documentation says it should be. I FTP’d the files over to the site and extracted them and I was pretty much up and running.

In terms of custom configuration I’m using the following:

Cruz theme – purchased through themeforest.net – not the most complex  theme compared to some, but very worth the small fee to purchase it. Very well documented as well.

BackupBuddy – this is a great plug in, it’s not cheap, but it does exactly what it says, it backs everything up with a click once its setup, plugins and all. Being a SQL guy you can be sure that I tested the restore process as well, and can also confirm that it was very simple and seems very stable. This also comes highly recommended. You can move the backup files here there and everywhere, either automated or manual and I push mine to Amazon S3 automatically from the plugin.

This is what I would consider the bare minimum, in that its deployed, it runs, I can develop on it in a separate environment and more importantly I can back it up.

I like to be plugin light currently, as it keeps it lower maintenance for me, and is less risk I feel, as I hate risk and don’t want to spend lots of time troubleshooting compatibility issues. So the only other things I run is

Google XML Sitemaps – which simply generates a sitemap – no more no less

and

SyntaxHighlighter Evolved – which deals with the rather nice code syntax for many different languages.

I did consider an SEO plugin, but decided for the moment that my needs were not worthly of such granular control, and also the more competent ones seemed to require a certain amount of config times (unsurprisingly considering the subject) which I wasn’t prepared to devote just now!

Overall I’m fairly happy with the experience. I did pay for a few things, but the prices were very reasonable and were well worth it in my opinion (especially the backup).

SQL Server script to populate a table of sequential dates in multiple formats

I was writing lots of TSQL today for something I was working on, which in itself is a rather rare occasion since I do so much more infrastructure and architecture stuff nowadays. I needed to write a script to populate a table with a series of rows containing sequential dates. I had to create a table which had 2 columns in it, one was [date] datatype and the other needed to be a 6 character string representing the year and the month as numbers concatenated together, for example 15th March 2012 would become 201215.

Then this table needed several rows covering every day over a number of years. Here’s the code just in case anyone fancies reusing it.

create table #dates
(
CustomMonth char(6),
lookupdate date
)

declare @date_increment date
--change your start date here if you fancy
set @date_increment = '2005-01-01'

--change your end date here if you fancy
while @date_increment < '2012-12-31'
begin

set @date_increment = DATEADD(day,1,@date_increment)

insert #dates (CustomMonth, lookupdate)
select convert(char(4),(datepart(year,@date_increment)))
	+ RIGHT('0' + CONVERT(VARCHAR(2), DATEPART(MM, @date_increment)), 2),
@date_increment

end

--check it looks ok
select * from #dates

Create automated PGP task in SSIS using GnuPG to decrypt files

Previously I wrote about my efforts to automate the decryption of files with SSIS using the gpg2.exe which comes as part of the GnuPG package. The original article is here

SSIS Task to decrypt PGP files automatically

However after deploying the working solution into production, to be run as a scheduled task, I found out that this package and solution still had some issues. I found that it was behaving rather differently when it was deployed into the production environment as opposed to running in the BIDS environment. When executing the exact same code in production which worked processing the exact same files in development (and I mean the exact same, same account, same files, same everything) I got error which looked like this (sanitised for security)

Error: 2012-03-13 11:16:07.10 Code: 0xC0029151 Source: run decrypt Execute Process Task Description: In Executing "C:\Program Files (x86)\GNU\GnuPG\gpg2.exe" "--batch --passphrase-fd 0 --decrypt-files [myfilename]" at "", The process exit code was "2" while the expected was "0". End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 11:16:02 Finished: 11:16:07 Elapsed: 4.609 seconds. The package execution failed. The step failed.

So I was again getting error code 2 which I had previously, for which the –batch switch had previously resolved the issue in development. So the error code was the same, but the reason obviously had to be different now. This required a little more investigation to get to the bottom of. Firstly I ran Process Monitor, which is often my first port of call in such scenarios, to check whether I was hitting some obscure permissions errors when running in live with the SQL Agent. It turned out totally clean (As an aisde I had done the same when initially installing GnuPG to resolve and issue that it couldn’t access a temp directory it required to do decryption).

A bit of research through the web and the full documentation of GnuPG left me using a further switch:

--status

which allowed me to look at some of the status messages from the output which were previously being swallowed by the SSIS task when run in production. There was SSIS logging enabled but it wasn’t getting anything back from the gpg2 executable beyond the status code.

I used a couple of different versions of this switch which looked like this

gpg2 --batch --status-fd 2 c:\gk\test\output.txt --decrypt-files test.gpg

which outputs the status messages to c:\gk\test\output.txt, or you can do this

gpg2 --batch --status-fd 2 --decrypt-files test.gpg

which outputs the messages to the console

Either way you end up with the following output (again slightly sanitised)

[GNUPG:] FILE_START 3 test.gpg
[GNUPG:] ENC_TO [hexstring] 1 0
[GNUPG:] USERID_HINT [hexstring] [mykeyname] (mykeyname) &lt;myemail@email.com&gt;
[GNUPG:] NEED_PASSPHRASE [hexstring] [hexstring] 1 0
[GNUPG:] GOOD_PASSPHRASE
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] PLAINTEXT 62 [integer] test_.txt
[GNUPG:] PLAINTEXT_LENGTH 901226
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] GOODMDC
[GNUPG:] END_DECRYPTION
[GNUPG:] FILE_DONE

but unfortunately this still didn’t give me anything to go on, as it still worked in the test environment, but not in the production one. Eventually by playing with the logging levels and these switches in production I got the details out in the SSIS log which contained this ket string

gpg: decryption failed: No secret key

I then realised that I was being an idiot and that the service account that I was running the SQL Agent under did not have the certificate registered under that userid. I had only imported the certificate into Kleopatra for the development userid I logged in with and not for the service account. I simply imported the certificate to the service account profile and then everything worked. This meant that the original instructions and code were valid, but I thought I’d put this post up in case anyone did the same stupid thing as me. It’s worth remembering that the certificates are by default imported at a user level into Kleopatra.

SSIS Task to decrypt PGP files automatically

Update 2012-03-13 – If you still get error code 2 after running this code in production (but it continues to work in your development environment) you might like to look at the subsequent post I did about further troubleshooting of this issue

This is something that I spent a few hours on recently which I wanted to share. The requirement here is to create a SSIS task to automatically decrypt a variable number of files that have been encrypted with PGP. This task will live within a larger SSIS package which does other typical SSIS tasks; fetching files from FTP, moving them around a file system, streaming them into a database and so forth.

The key here is that the task needs to be completely automated so that no user interaction is required , i.e. typing in the passphrase or other such matters. Whilst working this out I was browsing around the web and found various solutions but none was 100% perfect for my particular requirements. Initially all the options I tried either required me to enter the passphrase or returned error codes even on success. This post assumes a certain familiarity with SSIS development and PGP.

The PGP tool I used was the latest GPG4WIN installed to the default location (this means that the actual executable is:

C:\Program Files (x86)\GNU\GnuPG\gpg2.exe

The PGP files I was receiving were encrypted with the public key I had passed to the external source, and were simply decrypted using the GUI or the command line if I was prepared to type in the passphrase.

The way I automated this in SSIS was as follows:

Create a Foreach Loop to allow the processing of multiple files. The collection properties looked like this:

Foreach loop collection

The variable mapping look like this

foreach loop variable mappings

Inside this Foreach Loop I create an Execute Process Task.The process properties look like this:

Execute process task

The Expressions properties look like this.

Execute process expressions

It’s important to note that the arguments property on the process page are set by the expression, not hard coded, although they subsequently appear here. It’s critical to form the arguments in the expression builder to get them to work properly. The expression in text format is:

“–batch –passphrase-fd 0 –decrypt-files ” + @[User::receive_file_name]

Part of this syntax is undocumented in the GPG help files and had to be picked from the web. The confusion that I had was that I found an article which used gpg.exe and not gpg2.exe and mine version seemed to behave differently. The passphrase here is held in a variable in the package and then passed to the command line as the StandardInputVariable. This is what the [-fd 0] string achieves in the syntax. However, this still doesn’t work properly unless you pass the –batch parameter. If you don’t pass –batch then you still get challenged for the passphrase. If you run the package in debug mode you get the dialog box challenge, which you can then type into, but if you run in production mode, the task just fails with error code 2.

Whilst looking at this problem I also experimented with storing the passphrase in a file and various other options. Although the above option worked for me, I also noted a custom component is available for purchase at http://www.cozyroc.com/ssis/pgp-task which might be wort investigating if you have a budget to acquire the tools and an enterprise grade ETL requirement.

Graham