Troubleshooting replication issues when the subscriber is initialized from the backup

When the subscriber is initialized from backup, there can be so many things that can go wrong. Below are some of the error messages that might encounter during replication setup.

Msg 21408

Cannot create the subscription. You must specify a value of "Active" or "Subscribed" for the @status parameter. This is because the value specified for the @sync_type parameter is "initialize with backup" or "replication support only".

Msg 18786

The specified publication does not allow subscriptions to be initialized from a backup. To allow initialization from a backup, use sp_changepublication: set 'allow_initialize_from_backup' to 'true'.

Msg 21407

Cannot create the subscription. If you specify a value of "initialize with backup" for the @sync_type parameter, you must subscribe to all articles in the publication by specifying a value of "all" for the @article parameter.

Msg 21399

The transactions required for synchronizing the subscription with the specified log sequence number (LSN) are unavailable at the Distributor. Specify a higher LSN.

Msg 21397

The transactions required for synchronizing the nosync subscription created from the specified backup are unavailable at the Distributor. Retry the operation again with a more up-to-date log, differential, or full database backup.

Quick Checks:

-- on the publisher database

select allow_initialize_from_backup, min_autonosync_lsn,* from syspublications

allow_initialize_from_backup = 1 min_autonosync_lsn = the lsn from which the commands should be applied at the subscriber

--on distribution database

select min_autonosync_lsn,* from distribution.dbo.mspublications

min_autonosync_lsn will be the same as the syspublications.min_autonosync_lsn

During the execution of sp_addsubscription, we perform validation in sp_MSaddautonosyncsubscription procedure before we create the subscription.

--on distribution database

select subscription_seqno,publisher_seqno,ss_cplt_seqno,* from distribution.dbo.mssubscriptions

min_autonosync_lsn should be always less than the subscription_seqno

When we execute sp_addsubscrption with @backupdevicename='backup device', we are executing the “RESTORE HEADERONLY” on the backup and getting the LASTLSN of the latest backup if the backup set contains multiple backups. All this logic is handled in sp_MSextractlastlsnfrombackup proc. This proc is executed inside sp_MSaddautonosyncsubscription procedure.

RESTORE HEADERONLY FROM DISK on the backup set and get the lastLSN from the latest backup. Use the following SQL to get the binary form of the lsn.

declare @numericlsn numeric(25,0)

declare @high4bytelsncomponent bigint,

@mid4bytelsncomponent bigint,

@low2bytelsncomponent int

--set the lsn here

set @numericlsn = 93000000070800001

select @high4bytelsncomponent = convert(bigint, floor(@numericlsn / 1000000000000000))

select @numericlsn = @numericlsn - convert(numeric(25,0), @high4bytelsncomponent) * 1000000000000000

select @mid4bytelsncomponent = convert(bigint,floor(@numericlsn / 100000))

select @numericlsn = @numericlsn - convert(numeric(25,0), @mid4bytelsncomponent) * 100000

select @low2bytelsncomponent = convert(int, @numericlsn)

SELECT convert(binary(4), @high4bytelsncomponent) + convert(binary(4), @mid4bytelsncomponent) + convert(binary(2), @low2bytelsncomponent)

This is the lsn from which the Distribution agent will pick up the commands from msrepl_transactions and apply at the subscriber.

Potential Problems

If the backup set contains multiple backup sets, the sp_addsubscrption logic select the first backup set and not the most recent backup set. As result, the Subscriber may not have the most recent copy of the Publication database. After all, reason we we’re doing this method is to reduce the data transfer time to the Subscriber. We didn’t count on the backup set having multiple copies of the Published database. Kinds of defeats trying to “save time” using Backups.

Solution: When using “allow initialization from backup files” always create the Publisher backup to a NEW backup (*.bak) file.

How to initialize the subscriber from backup

Here is the step by step procedure on how to initalize subscriber from backup

1. Create the publication using user interface Replication Wizard.
2. Under subscription options for the publication, set “allow initialization from backup files” to true
3. Create a new full backup of the publisher database. If you have existing full backup of the publisher database, you can still use that backup set but we have take a new log backup or differential backup of the publisher database and restore at the subscriber.
4. Replication UI does not allow the option to create the subscription to allow initialization from back. We have to TSQL when creating the subscription.
   
   exec sp_addsubscription @publication = N'Repl2000', …..
   @sync_type = N'initialize with backup',
   @backupdevicetype='Disk',
   @backupdevicename='C:\Repl2000_RestoreThis.bak'--this is the last backup used to restore on the subscriber that was taken after the publication was created
   go
   exec sp_addpushsubscription_agent …….
   go
   
   
5. This should create the subscription and the Distribution agent should be replicating the commands successfully.

How to identify Foreign Key dependencies in SQL Server database?

To understand the database schema design especially to determine the referential integrity constraints and dependencies, here I’m sharing a sample script which will help you in following:

1)      To identify which table(s) are dependent on a given object

2)      Which columns are being referenced (have foreign keys defined on them)

3)      What action to perform in response to update/delete operations

Note: If you would like to run the below script to identify the forign key dependencies w.r.t to only one table e.g. ‘My Table’, uncomment the filter in the WHERE clause

SELECT ROW_NUMBER() OVER (ORDER BY OBJECT_NAME(fk.referenced_object_id), clm1.name) as 'S.No',fk.referenced_object_id as ReferencedObjID,

       constraint_column_id as ColumnID,
       OBJECT_NAME(fk.referenced_object_id) as [ReferencedTable(Parent)],
       SCHEMA_NAME (CAST(OBJECTPROPERTYEX(fk.referenced_object_id,N'SchemaId') AS bit)) as [ParentSchema],
       clm2.name as ReferencedColumnName,
       OBJECT_NAME(constraint_object_id) as ConstraintName,
       fk.parent_object_id as ReferencingObjID,
       OBJECT_NAME(fk.parent_object_id) as [ReferencingTable (Foreign)],
       clm1.name as ForeignKeyColumn,
       SCHEMA_NAME (CAST(OBJECTPROPERTYEX(fk.parent_object_id,N'SchemaId') AS bit)) as [ForeignSchema],
      [Action on Update] = CONVERT(varchar,CASE OBJECTPROPERTY(fk.referenced_object_id,'CnstIsUpdateCascade')
                           WHEN 1 THEN 'CASCADE'
                           ELSE 'NO_ACTION'
                           END),
       [Action on Delete] = CONVERT(varchar,CASE OBJECTPROPERTY(fk.referenced_object_id,'CnstIsDeleteCascade')
                           WHEN 1 THEN 'CASCADE'
                           ELSE 'NO_ACTION'
                           END)

FROM sys.foreign_key_columns fk
       JOIN sys.columns clm1
          ON fk.parent_column_id = clm1.column_id
          AND fk.parent_object_id = clm1.object_id
       JOIN sys.columns clm2
          ON fk.referenced_column_id = clm2.column_id
          AND fk.referenced_object_id= clm2.object_id
--WHERE OBJECT_NAME(fk.referenced_object_id) = 'My Table'  --- table name which is being referenced by other tables via Foreign Keys

ORDER BY OBJECT_NAME(fk.referenced_object_id)

Below is the output from AdventureWorks database

I hope this would help you at some point in carrying out your DBA or Developer task in understanding the database Schema design.

Do you know the Maximum number of databases that can be mirrored?

Recently I came across an interesting scenario where the SQL Server was frequently generating Deadlock Scheduler mini dumps on the mirrored SQL Server instance. In this post I put together various pieces of information, so thought of sharing with you.

Scenario: Suppose you are using SQL Server 2008 R2 Enterprise x64 edition installed on windows Server 2008 SP1. You are using SQL Server database mirroring feature and has setup database mirroring on 104 databases. You are frequently getting Deadlock Scheduler dump (Error 17884) on the mirror site, even though you restart SQL Server instance on the mirror site, still the Error 17884 deadlock scheduler appears within few minutes on the mirror site.

After analyzing the Deadlock Scheduler mini dump, generated on the very first occurrence of error 17884.

Important to note, SQL Server generate mini memory dump only on the first occurrence of Error 17884 and for the subsequent occurrences, it simply job an entry in the SQL Server error log something like

"New queries assigned to process on Node 0 have not been picked up by a worker thread in the last 360 seconds. Blocking or long-running queries can contribute to this condition, and may degrade client response time. Use the ""max worker threads"" configuration option to increase number of allowable threads, or optimize current running queries. SQL Process Utilization: 10%%. System Idle: 88%%."

I found that the following:

Ø  There were 464 workers performing Parallel operation (i.e. Query Parallelism).

Ø  No lock blocking seen at the Server.

Ø  There were 372 workers performing DBM redo operation.

Total workers: 710

Workers in RUNNABLE state: 0

As per SQL Server configuration settings

name	minimum	maximum	config_value	run_value
max worker threads	128	32767	704	704

Before I proceed further Lets check the Mirror Server configuration

Intel(R) Xeon(R) CPU           L5630  @ 2.13GHz. Bios Version is HP  - 2

16 x64 level 8664, 2 Mhz processor (s).

Total Physical Memory = 49141 MB           
Available Physical Memory = 45098 MB       

For 64 bit operating system:

If total # of available logical CPU’s > 4 :   

Max worker threads = 512 + ((logical CPUS’s - 4) * 16

In our case max worker threads = 512 + ((16 - 4) * 16=704  which is default no. of worker thread SQL Server will spawn.

You must be wondering Max Worker Threads setting is correctly set as per above calculation

Before I comment on this, let’s talk about Database mirroring requirements for worker threads

·        Principal server:

1 global thread and 2 threads per each of the mirrored databases.

·        Mirror server: 64 bit architectures:

1 global thread, 2 threads per mirrored databases and one additional thread for each mirrored database for every 4 processor cores.

In short, the number of thread needed on mirror server are greater than those on the primary, so you need to use the mirror server system configuration settings to calculate the no. of databases that can be mirrored.

Number of threads required per mirrored database = (2+16/4) = 6 threads per database. Where 16 is the number of logical CPU cores on the mirror server

Maximum number of databases that can be mirrored = 704/6 = 117

Now you must be wondering, in our scenario, we are just mirroring 104 databases (which is less than 117 i.e. Max databases that can be mirrored), then what could be the cause error 17884 (Deadlock Scheduler) occurring too frequently on the mirror server.

In Summary, not all the 704 workers would be available to perform mirroring activities as some threads are assigned to core database activity on the server. Hence you have to set the value of max server thread to 1024 and you will be able to bring your mirror server back online and operational.

Additionally, you can run the below query to determine the number of worker threads available on your SQL Server.

select count(*) from sys.dm_os_threads

In my case it returned 759

Does SQL Server replication support secure FTP (FTPS) protocol ?

Few days back I came across an intertesting scenario and was asked the above question, so I thought of sharing with you guys...

Scenario: Let’s say, you are using SQL Server replication to replicate data between two sites which are connected over the public network via VPN. You are able to successfully initialize the subscription using the initial snapshot delivered over the FTP. Due to security reasons, you would like to replace FTP by an encrypted solution (FTPS, SFTP, FTP over SSH). Now the question is which of the these encrypted solutions (FTPS,SFTP,FTP over SSH) is compliant with Microsoft SQL Server replication?

To answer the above question:  In windows, the Secured version of FTP is “FTPS”. Whereas in UNIX System, it was called “FTP over SSH” or “FTPS”. The "FTPS" abbreviation is often mistakenly used to specify some kind of Secure FTP, by which people most often mean FTPS. Another (similar) mistake is that FTPS is thought to be some kind of FTP over SSL. In fact, FTPS is an abbreviation of "SSH File Transfer Protocol." This is not FTP over SSL and not FTP over SSH (which is also technically possible, but very rare).

Now I’ve successfully configured my FTPS site for snapshot folder and ensured that the following conditions are met:

1.     The Publisher must be running SQL Server.

2.     An FTP/FTPS server must be enabled and configured to point to the C:\Program Files\Microsoft SQL Server\MSSQL\Repldata\Ftp directory (default) or network share which you specified while configuring your distributor or in the specific publication properties SNAPHOT page.

For more information about configuring an FTP/FTPS server, see the Internet Information Services (IIS) documentation.

3.     The publication is configured to use an FTP snapshot: FTP Snapshot page in publication properties.

Note: This option is for delivery of Snapshot to subscriber using the FTP site/address you specify, not for generation of snapshot on FTP/FTPS site.

But I’m unable to successfully configure SQL Server replication. Though I’m able to create either PUSH/PULL subscription but unable to initialize the subscription by delivering an initial snapshot over the FTPS.

The replication (Distribution/Merge) agent fails with the below error:

The process could not connect to FTP site '<ftp_site_address>' using port n (Source: MSSQL_REPL20032)

The operation times out.

You are able to access the FTPS site from both the distributor and subscriber machine but the initial snapshot delivery is still failing with above error.

 

You must be wondering what could be the reason? Am I missing sometime? Did I misconfiguration something? and so on….

 

The cause behind this error is, SQL Server replication, till the latest release of SQL Server 2008 R2 doesn’t have any infrastructure to support FTPS. So what are my options, I’m concerned about data security while replicating the data over the wire on public network.

 

You can only use FTP for an initial snapshot delivery on to the subscriber in SQL Server replication. So far the security of data is concerned, the solution is twofold:

 

1)      If you are concerned about the security of initial snapshot delivery via FTP using VPN on public network, You use the following workaround which will require some custom code to be implemented by you:

a.      Generate an initial snapshot.

b.      Compress the snapshot folder, and encrypt the zip file. Or you can write your own custom code to encrypt the snapshot folder or compressed snapshot folder.

c.      Push the encrypted zip file to the FTP and transfer it across to the subscriber.

d.      De-crypt the zip file and uncompress the snapshot folder locally on the subscriber.

e.      Use –AltSnapshotFolder distribution/Merge agent command line parameter by specifying the location of the snapshot folder copied locally on the subscriber (step d above)

f.       Run the Distribution/Merge agent to deliver an initial snapshot to initialize the subscription.

2)      To ensure secure replication of on-going changes (TDS Streams) after the initial snapshot delivery, Use SSL Encryption on the distribution server.

I hope this would help you and/or could have clarified your doubts or have answered questions that you may have related to use of secured version of FTP (FTPS) in SQL Server replication. I look forward to your comments.

Happy Learning..!

When you rename a domain group in Active Directory, you are going to have some issues related to the login in SQL Server

Few days back I came across an interesting scenario related to a login in the SQL Server replication.

The Active Directory group used to grant access to a snapshot publication does not show up in the Publication access list (PAL) anymore when viewing the properties of the publication. However, the AD group does show up in the MSPublication_Access table for the snapshot publication in the Distribution database.

There is a disparity between what the Replication property settings and what is actually in the MSPublication_Access table. When I try to add the group to the PAL in the GUI, it doesn't give any error message and completes. But when I again check the PAL, it doesn't show up. Does it sound weird?

Now the question is what could have caused such a weird scenario ?

On investigating in a real life customer scanrio, what I found something very interesting.

1) The login was changed in the AD sometime back. The change was that the login was renamed but not deleted/recreated. So the SID remained same for the changed login in the AD. But because of this change in the AD, SQL Server was never aware of the change of the name. The login with the old name still exist in the server and mapped to the same SID.

2) Another interesting finding, somehow the login was deleted and recreated at the SQL Server level. So when the login was created again at the SQL Server, the new name was created on the SQL Server but the same Old SID was associated with it.

But the entry for the login remained there in the mspublication_access table. This login was never removed during the deletion of the login from the SQL Server. Afterwards whenever they used GUI to check the PAL, it was unable to find the login name from the SQL Server logins and hence didn't show up in the PAL list. But whenever you try to add the login in the PAL, it was able to find an existing record with the same SID and hence failed to add the login. But it never gave any error message.

To workaround the above weird scenario, go ahead and Delete the record manually from the MSPublication_Access table in distribution database. Then again add the login to the SQL Server and Publication Access List (PAL).