Heres a nasty little problem, we have a SharePoint farm that consists of 4 servers, two Web Front Ends, and application server and a SQL server.
This is Microsoft SharePoint 2007 Enterprise SP2 running on Microsoft Windows 2003 R2 SP2 (32-bit), all was well until a recent automatic update was installed which broke the application server.
The application server hosted the “SharePoint Central Administration Website” and also SQL Reporting Services 2008 that was linked into the SharePoint farm.
PROBLEM(S)
The following morning after the update the Central Administration Website started to show the following error: “Cannot connect to the configuration database.”
A reboot of all the farm servers and tried again, still no joy, I could not access the “Central Administration Website” running on the application server.
It’s worth noting however the web front ends were unaffected and happily connected to the databases, and our main SharePoint site collection was accessible and worked fine. To make things more confusing the backup “Central Administration Website” that ran on one of the web front end servers also worked fine, but this showed that the configuration database was not corrupted which was something.
Another of the problems caused was that the SQL Reporting services server was dead too, it complained it could not access the database with the following errors (in the event log) when starting up:
Event Type: Error
Event Source: Report Server (MSSQLSERVER)
Event Category: Management
Event ID: 107
Date: 04/03/2012
Time: 09:08:32
User: N/A
Computer: OAKSMFSPAPP01
Description:
Report Server (MSSQLSERVER) cannot connect to the report server database.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.However also in the SQL Reporting server logs that in my case were located here:
D:\Program Files\Microsoft SQL Server\MSRS10.MSSQLSERVER\Reporting Services\LogFiles\ReportServerService__03_04_2012_08_38_37.log
03/04/2012-08:42:43:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the
error log for more details., ;
Info: Microsoft.ReportingServices.Diagnostics.Utilities.InternalCatalogException: An internal error occurred on the report server. See the error
log for more details. ---> System.TypeInitializationException: The type initializer for 'Microsoft.ReportingServices.Diagnostics.CatalogItemNameUtility' threw an exception. --->
System.Runtime.InteropServices.SEHException: External component has thrown an exception.Something that many posts on the Internet advised was to re-run the “SharePoint products and Technologies Configuration Wizard” to try to fix SharePoints issues. There was also talk of running the psconfig command to reinstall SharePoint on the server. However both methods resulted in the same error message as shown below:
“An exception of type system.typeinitializationException was thrown. Additional exception information: The type initializer for ‘System.Data.Common.DbConnectionOptions’ threw an exception.”
But the above string: “System.Data.Common.DbConnectionOptions” turned out to be the key to it, this is a component of .net framework and is used by both SQL Reporting Services and SharePoint, this was failing therefore neither application could contact the SQL server databases.
CAUSE and SOLUTION
It turns out that when we looked through the list of all the updates installed and checked the event logs there were a list of about 10 updates that got applied, however, the two specific ones were those related to Microsoft .NET Framework 2.0 Service Pack 2.
KB2656352
KB2633880
Removing these two updates and rebooting of all the farm servers resolved the issues we were seeing and the “SharePoint Central Administration WebSite” and SQL Reporting services started to work again as expected.
Other bits I also tried:
Reinstall of .net 2 and 3 using the following commands:
.NET Framework 2.0 SP2 – silent repair
msiexec /fpecmsu {C09FB3CD-3D0C-3F2D-899A-6A1D67F2073F} REINSTALL=ALL /l*v %temp%\netfx20sp2_repair_log.txt /qb
.NET Framework 3.0 SP2 – silent repair
msiexec /fpecmsu {A3051CD0-2F64-3813-A88D-B8DCCDE8F8C7} REINSTALL=ALL /l*v %temp%\netfx30sp2_repair_log.txt /qb
I also tried to reinstall the SQL Client using the sqlncli.msi installer pack, as it appeared to be missing, in my case it was located here, I just ran this to ensure SQL client was installed correctly.
C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Update Cache\KB2546951\ServicePack\x86\setup\x86\sqlncli.msi