SQL Server Delete Duplicate Rows

时间：2016-07-30 10:35:00 阅读：226 评论：0 收藏：0 [点我收藏+]

标签：

There can be two types of duplication of rows in a table

1. Entire row getting duplicated because there is no primary key or unique key.

2. Only primary key or unique key value is different, but remaining all values are same.

Scenario 1: Delete duplicate rows without primary key or unique key.

Let us create the following example.

create table customers1 (CustId Int, CustName Varchar(20), CustCity Varchar(20), Passport_Number Varchar(20)) go

Insert into customers1 Values(1, ‘John‘, ‘Paris‘, ‘P123X78‘)

Insert into customers1 Values(2, ‘Martin‘, ‘London‘, ‘L873X92‘)

Insert into customers1 Values(3, ‘Smith‘, ‘New York‘, ‘N293Y99‘)

Insert into customers1 Values(1, ‘John‘, ‘Paris‘, ‘P123X78‘) go

select * from customers1 go

技术分享

We want remove one of the duplicate records of John.

By issuing the following summary query, we can see which see which records are duplicate.

select * from customers1 Group by Custid,CustName, CustCity, Passport_Number Having count(*) > 1

技术分享

Now we will add this row to a local temporary table.

Select * into #Temp_customers1 from customers1 where 1 = 2 Insert into #Temp_customers1 select * from customers1 Group by Custid,CustName, CustCity, Passport_Number Having count(*) > 1

Now the situation is that the duplicate row is in the local temporary table. All we need to now is to delete records from main table customers1 as per matching custid of the local temporary table.

Delete from customers1 where custid in (select Custid from #Temp_customers1)

Will the above query work? Not entirely, as by using the above query, we lost all the duplicate records!! Let us see the table again.

select * from customers1 go

Now to keep one record of John, we will take help of the local temporary table again. Let us add the same record from temporary table into customers1 table.

Insert into Customers1 select * from #Temp_customers1 go

Finally we got a single record of John at the end. Let us confirm by seeing the Customers1 table.

select * from customers1 go

技术分享

Once done, we can drop the local temporary table.

Scenario 2: Delete duplicate rows where primary key or unique key value is different but remaining values are same.

Let us create the following example.

create table customers2 (CustId Int Primary Key, CustName Varchar(20), CustCity Varchar(20), Passport_Number Varchar(20)) go

Insert into customers2 Values(1, ‘John‘, ‘Paris‘, ‘P123X78‘)

Insert into customers2 Values(2, ‘Martin‘, ‘London‘, ‘L873X92‘)

Insert into customers2 Values(3, ‘Smith‘, ‘New York‘, ‘N293Y99‘)

Insert into customers2 Values(4, ‘John‘, ‘Paris‘, ‘P123X78‘)

Insert into customers2 Values(5, ‘John‘, ‘Paris‘, ‘P123X78‘)

select * from customers2 go

Here is the same customer’s record, but this time John’s record has been added thrice with different customer ids but same Passport number!

技术分享

Scenario 2.a: Delete Duplicate rows but keep one using CTE

We need to use the technique of Self Join initially to check for duplicate records containing different custid but same passport number.

select distinct a.* from customers2 a join customers2 b on a.custid <> b.custid and a.CustName = b.CustName and a.CustCity = b.CustCity and a.Passport_Number = b.Passport_Number

技术分享

Now we have realized that custid 1, 4 & 5 are duplicate. The self-join statement accompanied by delete statement will give us the desired output of keeping the last duplicate record by eliminating all the previous duplicate records. We will use the Common Table Expression (CTE) and put the Self Join query in it.

With Duplicates as (select distinct a.custid as Customer_ID from customers2 a join customers2 b on a.custid <> b.custid and a.CustName = b.CustName and a.CustCity = b.CustCity and a.Passport_Number = b.Passport_Number ) Delete from Customers2 where custid in (select Customer_ID from Duplicates) and custid <> (select max(Customer_ID) from Duplicates)

Let’s check which rows got deleted.

select * from customers2 go

技术分享