SQL

Response: Give Me A Coconut and Six Months

I was tagged by SQL Server Expert Brent Ozar in his response to a great, thought provoking blog post called Give Me a Coconut and Six Months by Tim Ford (SQLAgentMan on Twitter).

The short summary of the post is if you had six months free of distraction, what would you turn your attention to.  Tim’s choices included backups, security, and monitoring, which I think is a great “solid foundation” to work from.

Brent posits that if he became more effective at data mining, he would be able to provide a business with critical insight with which to improve sales, product focus, and develop key personnel.

If I had more time (and skills), I could tell executives things like:

  • These are the top five customers who are about to leave us.
  • These are the top five products that are about to go viral, and we need to stock more ASAP.
  • These are the top five salespeople who need coaching to produce more revenue.

Walk into an executive’s office with this kind of information, and you’re a hero.

Here’s the shocker Brent… I agree. 

I agree that delivering that kind of data to management is the Holy Grail of IT projects.  Before coming into the IT realm, I ran a small business and I would have killed for information along those lines. 

One thing I think is missing from Brent’s discussion is that there are three parts to this type of data mining:

  • The first part is the technical aspect on how to retrieve the data from databases and other sources of information, which Brent probably has handily covered. 
  • The second part is not a technical question at all.  The second part of the data mining is actually coming up with questions you have for your data.  The quote above from Brent’s post highlights those, and point to an area of expertise outside the technical.  Brent is demonstrating (and probably should have said explicitly) that a domain/business knowledge is very important in determining these requests.  This is often difficult for the stereotypical IT person, but is essential if you are interested in moving your career past a technical implementer. 
  • So, it appears Brent’s real goal is the the third part of this process, which is translating these questions (taking the business knowledge) and retrieving and correlating the data (using his technical skills) that he will need to make these determinations.

Brent continues on to cover a common area of disagreement between us.

I kick the PowerShell horse a lot, and here it comes again. If you’re in IT, listen up: you’re either cutting costs, or making money.  Guess which one has more upside.  If you truly bust your hump, become an amazing scripting deity, and save 99% of your time, you just saved 99% of your salary.  If you’re really good, you might save 10 people 99% of their time.

I work as the sole admin/accidental DBA/desktop support/multimedia support/”if it has a blinking LED light” support for an agency that collects lots and lots of data.  As a scripting practitioner, if I save myself 25%, 40%, or even 60% of my time not having to solve the same problems over and over, I’m free to plume the mysteries of my database and convert the bits stored their into meaningful data and even information that is usable and actionable. 

Not every environment has the luxury of being able to afford someone of Brent’s caliber to come in and learn their business and help them develop methods of turning their data into knowledge.  In my situation and that of many small to medium businesses, scripting is the tool that enables the IT generalist to explore and branch out into these other areas.

Brent may feel ” you can go into data mining and make 100 salespeople twice as effective” and that IT people who keep things running are replaceable and he may be right, but I believe, especially in tighter economic times, specialists become a luxury that only a few can afford where efficient generalists that know the business become more in demand.

Don’t be mad Brent… it’s okay to be wrong every once in a while! :)

So if I had six months of no interruptions and could focus on certain specific projects, I would:

  • Finish automating (via PowerShell, SQL, or other means) common tasks
  • Develop the questions my administrators (not technical admins, but departmental administrators) would like to ask our data sources
  • Build tools to answer those questions via reports, graphs, network maps, or other data visualization techniques (like the cool stuff Doug Finke and Chad Miller have been doing)

I personally think that EVERY technical person should have a grasp on the basic business environment, be aware of what is happening in their company’s industry (or at least their department’s industry for larger organizations), and begin to develop more in depth domain knowledge as to the business processes.

Due to the odd path I took to becoming an IT person, I’ve actually accumulated quite a bit of domain knowledge about the law enforcement, the various jobs, information requirements, and the ins and outs of our workflows and data.  This background gives me a great starting place when going to look at my data, since scripting has given me the time to do so. 

Judging from the pattern in Tim’s and Brent’s posts, I’m supposed to tag a few people..

How about:

  • Doug Finke (developer, data visualization explorer,  and PowerShell MVP),
  • Hal Rottenberg (administrator, VMWare vExpert, podcaster, and PowerShell MVP),
  • and Wes Stahler (administrator, blogger, and PowerShell enthusiast)

Turn Your Stored Procedures Into PowerShell Functions – MetaProgramming With PowerShell

UPDATE: The script was moved to Google Code.  The links in the post have been updated to reflect that.  Or you can just go here… http://code.google.com/p/poshcodegen/

I’ve been working on some data conversion at work, converting records from one system to a new system.  I’ve built quite a library of SQL queries with PowerShell wrappers for dealing with data in the first system, but I don’t have the same luxury with the new system.

The new system does, however, have a nice set of stored procedures that make moving data into their application much easier.

I started writing my conversion scripts in PowerShell, since I do have to do some processing on the records to accommodate the new workflow and data layout.  I was looking at having to call almost 100 stored procedures through various parts of this process.  That is a lot of boiler plate code or referring back to the database often to check parameter names and types.  So, I’ve written a little PowerShell script that will take stored procedures (either as a parameter or from the pipeline) and create a function that wraps that stored procedure.

The benefits of this are great with V2 CTP3 or with a more advanced editor (one that will provide tab completion on parameters).

One a wider scope, I think that this type of utility is one of PowerShell’s great strengths.  Using PowerShell for metaprogramming (another example here on the Telling Machine blog) can be a great time saver.  I spent a couple of hours working on this script, but it would have cost me much more time to handle each case individually.

Now, when I hear “metaprogramming”, my head starts to hurt a bit as I start to think about programs about programs about programs, but this isn’t that bad.  PowerShell makes this pretty easy to understand though.  To create a function dynamically, all that is needed is a string that contains text that the PowerShell runtime can evaluate (PowerShell will check for syntax errors, but not logic errors – as is the case with any script or function).

Example:

$text = ‘Get-ChildItem *.ps1 | Measure-Object’

Set-Item –Path function:global:Get-PowerShellScriptCount –Value $Text

This takes advantage of the Function provider and creates a function object in the global scope with the specified name and $Text is turned into a scriptblock.  I can then call that function as needed.

Since I know PowerShell, to build dynamic functions I just have to create text that can be evaluated to do the function I need.

Here’s what my script does after it runs a query against the database to get the stored procedure’s text:

  1. parses the text to get the parameter names and types
  2. using the names and types, it sets the parameters for the function (if someone wants to add some logic to make it type safe, that would rock!)
  3. builds the text to create a SqlConnection object to handle the database connection
  4. builds the text to create a SqlCommand object and sets the type of command to be a stored procedure
  5. builds the text to populate the parameters, including setting up the output parameters (if any)
  6. builds the text to run the stored procedure
  7. builds the text to put any output parameters into a PSObject as properties.
  8. create a new function with the name of the stored procedure and uses the text built in the previous steps as the scriptblock for the function.

What are you automating with PowerShell?

How about trying to automate some of your automation code?

New-StoredProcFunction.ps1 here.

PSMDTAG:metaprogramming sql stored procedure

UPDATED SCRIPT: Thanks to Chad Miller for the idea.. instead of parsing the text of the stored procedure, the parameter information is available in the Information_Schema.Parameters

“Diff”ing Database Table Columns

Previously, I published a script on comparing what tables two databases contained.  Going a bit further, I put together a script that compares the columns and what type of data they store.

Compare-DatabaseColumns has similar parameters to the Compare-DatabaseSchema script.

  • Table – One or more tables to compare columns from
  • SqlServerOne – SQL Server for the first database
  • FirstDatabase – Name of the first database for the comparison
  • SqlUsernameOne – SQL user name for the first database
  • SqlPasswordOne – SQL password for the first database
  • SqlServerTwo – SQL Server for the second database
  • SecondDatabase – Name of the second database for comparison
  • SqlUsernameTwo – SQL user name for the second database
  • SqlPasswordTwo – SQL password for the second database
  • FilePrefix – Prefix for the log file name
  • Log – Switch parameter that saves one CSV file with the difference in the tables.  If the Column switch parameter is chosen also, it will save one CSV file per table with differences in the columns

This script can also take pipeline input, either strings or a property of “Name” or “TableName”.

You can find this script on PoshCode.org.

Comparing Database Schemas

I regularly am working with several versions of a database for an application that I manage (a live database, training database, test database, and previous version database).  Occasionally, I need to know what the differences between the databases are, especially after our vendor updates my test environment or right after an update in my training or live environment. 

Since I spend a good portion of my day in PowerShell, I wrapped some system table queries in a PowerShell script and use Compare-Object to find any differences in the tables and compare the column definitions as well.  The queries targets only user tables.

Compare-DatabaseSchema.ps1 takes several parameters.

  • SqlServerOne – SQL Server for the first database
  • FirstDatabase – Name of the first database for the comparison
  • SqlUsernameOne – SQL user name for the first database
  • SqlPasswordOne – SQL password for the first database
  • SqlServerTwo – SQL Server for the second database
  • SecondDatabase – Name of the second database for comparison
  • SqlUsernameTwo – SQL user name for the second database
  • SqlPasswordTwo – SQL password for the second database
  • FilePrefix – Prefix for the log file name
  • Log – Switch parameter that saves one CSV file with the difference in the tables.  If the Column switch parameter is chosen also, it will save one CSV file per table with differences in the columns
  • Column – Switch parametr that enables a comparison of the columns in the tables that match.  Columns are compared by name and datatype.

I still have to add some checks for the various constraints, but that will come later.

You can find the script on PoShCode.org.