Parsing a URL with SQL Server Functions

Problem

In my environment I have a few tables with thousands of rows that have URLs in particular columns.  We are going through some changes in the environment and need to be able to parse out the data (base URL, file name, query parameters) from the specific columns then move the parsed data to another database design.  Moving the data is no problem.  I plan on using SQL Server 2005 Integration Services and I have a good sense of how to use that tool.  By the way, all of the URLs are in varchar fields if that makes any difference.  Can you provide any recommendations on how to parse that data?

Solution

Although there are probably a few different ways to approach this task, let's break it apart and see how we can code each one of these scenarios with T-SQL commands in SQL Server 2005.  I assume you will be parsing the data from a single column to three new columns in a new database\table design.  Since you are familiar with SQL Server 2005 Integration Services, I will not dig into those details and just focus on how to capture the data that you need.  So let's take a look at each scenario and give an explanation on the T-SQL coding in SQL Server 2005.

Example Table and Data

Here an example table and example data for this tip:

USE zCustomerDB
GO
CREATE TABLE ExampleURLs (URL_ID INT IDENTITY(1,1) PRIMARY KEY NOT NULL,CompleteURL VARCHAR(500))GOINSERT INTO [zCustomerDB].[dbo].[ExampleURLs] ([CompleteURL])VALUES ('http://www.test.com/page.asp?tip=1156')GOINSERT INTO [zCustomerDB].[dbo].[ExampleURLs] ([CompleteURL])VALUES ('http://www.test.com/page.asp?tip=1299')GOINSERT INTO [zCustomerDB].[dbo].[ExampleURLs] ([CompleteURL])VALUES ('http://www.test.com/page.asp?tip=1262')GOINSERT INTO [zCustomerDB].[dbo].[ExampleURLs] ([CompleteURL])VALUES ('http://www.test.com/page.asp?tip=1240')GOINSERT INTO [zCustomerDB].[dbo].[ExampleURLs] ([CompleteURL])VALUES ('http://www.test.com/page.asp?tip=1226')GO

cenario 1 - Base URL

As an example, let's say the data you are working with looks like 'http://www.mssqltips.com/sqlservertip/1156/opportunities-with-sql-server-2005-express-edition/' and you need to capture the 'www.mssqltips.com' portion of the column.  Below is an example of what that code could look like.  This code is fairly straight forward with a single SUBSTRING command performing all of the heavy lifting.  The REVERSEcommand is used to find the last slash so the entire directory structure is returned.  The CHARINDEX command is used to determine the position of characters such as '//' in 'http://' and '/' in the URL directory path.

SELECT URL_ID,CompleteURL, -- URL(CHARINDEX('//', CompleteURL, 1) + 1), -- Position of the double slashesCHARINDEX('/', REVERSE (CompleteURL), 1), -- Position of the last single slashSUBSTRING(CompleteURL,
(CHARINDEX('//', CompleteURL, 1) + 2), CHARINDEX('/', REVERSE (CompleteURL), 1) ) -- Final stringFROM zCustomerDB.dbo.ExampleURLs
GO

Scenario 2 - File Name

In this scenario let's capture file name which is the 'tip.asp' portion of the 'http://www.mssqltips.com/sqlservertip/1156/opportunities-with-sql-server-2005-express-edition/' URL.  This code makes additional usage of the REVERSE and SUBSTRING commands, but the subtle portion of this code is positioning each portion of the SUBSTRING correctly.  This is accomplished by using the '+1' in the starting position (second position) of the SUBSTRING command and using a '-1' in the length position (third position) of the SUBSTRING command.  In this circumstance the CHARINDEX command is used to determine the position of the '?' and '/' characters.  Here is an example of what that code could look like:

SELECT URL_ID,REVERSE (CompleteURL), -- Backwards version of the URLCHARINDEX('/', REVERSE (CompleteURL), 1), -- Position of the slashCHARINDEX('?', REVERSE (CompleteURL), 1), -- Position of the question mark((CHARINDEX('/', REVERSE (CompleteURL), 1)) - (CHARINDEX('?', REVERSE (CompleteURL), 1))-1), -- LengthREVERSE(RTRIM(SUBSTRING(REVERSE (CompleteURL),
(CHARINDEX('?', REVERSE (CompleteURL), 1)+1),
((CHARINDEX('/', REVERSE (CompleteURL), 1)) - (CHARINDEX('?', REVERSE (CompleteURL), 1))- 1)))) -- Final parsed valueFROM zCustomerDB.dbo.ExampleURLs
GO

Scenario 3 - Query Parameters

In the final example capturing the query parameters is based on finding the correct position for the question mark '?' in the URL.  With the 'http://www.mssqltips.com/sqlservertip/1156/opportunities-with-sql-server-2005-express-edition/' sample data, you would need to capture the 'tip=1156' portion of the column.  Here is an example of what that code could look like by leveraging the REVERSE, CHARINDEX and SUBSTRING commands once again in a little different context:

SELECT URL_ID,REVERSE (CompleteURL), -- Backwards version of the URLCHARINDEX('?', REVERSE (CompleteURL), 1), -- Position of the question markREVERSE(RTRIM(SUBSTRING(REVERSE (CompleteURL), 1, CHARINDEX('?', REVERSE (CompleteURL), 1) - 1))) -- Final parsed valueFROM zCustomerDB.dbo.ExampleURLs
GO