Sunday, 27 September 2015

SQL Timeout Exception Prevention and Tuning Stored Procedures


Prevent Stored Procedure Timeout

Write efficient Stored Procedures

Prevent More Disk Reads

Prevent Dead Locks


System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.


The following are the collection of my learning as a SQL developer.

Because of large data on the tables, the join operations and other operations will take up lot of time hence causing the application to time out, also reducing the performance.


Parameter Sniffing
In short, parameter sniffing is one exec plan for all parameter combinations. 

- This option causes the compilation process to ignore the value for the specified variable and use the specified value instead.
- Unlike the other workaround (Option(Recompile)), this option avoid the overhead of recompiling the query every time.


This is the most common problem, The workaround is to declare the input params  again with different variable name and use the
 new variable through out the SP.

In the Example @local_SetupId

CREATE PROCEDURE [dbo].[pr_get_user_ids]
(
@in_SetupId  int
)
As
begin
Declare @local_SetupId int =@in_SetupId
Select UserId from UserDetail where SetupId=@local_SetupId

end




Use Local Temp Tables




Store the data from large tables such as Product / Orders/ Hierarchy into Local temp Tables and Use them when joining – This is will Prevent the joining of heavy tables.

This is Subject to temp Db Size. Consider the temp Db size.

Select only the needed columns for your Transaction operation and put it into a local temp  table. Hence on performing joins the tables will be lighter  and preventing the use of excess Resource.

In the Example We have filtered out few columns  from productHierarchy table using setupid in Where  clause and used it to join with Product table.

SELECT * into #TempProductHierarchy FROM
(SELECT ProductHierarchy,ProductHierarchyId,ProductId from ProductHierarchyDetails where SetupId=@local_SetupId ) AS TPH          

SELECT ProductCodeInterface from masters.Product P
inner join #tempProductHierarchy on P.ProductId=#TempProductHierarchy.ProductId

Instead of
SELECT ProductCodeInterface from masters.Product P
inner join masters.ProductHierarchyDetail PHD on P.ProductId=PHD.ProductId
where P.SetupId=@local_SetupId

Use NON Clustered Index
Use Non Clustered Index on temp tables for quicker Execution

Once you have created a temp table, set index on the primary key of the table – on which you will be performing the join operation.
In the example we have read the Product Hierarchy  table into a temp table and we have created a non-clustered  index onto the primary key i.e ProductId

 SELECT * into #TempProductHierarchy FROM
(SELECT ProductHierarchy,ProductHierarchyId,ProductId from ProductHierarchyDetails
where SetupId=@local_SetupId ) AS TPH                     

Create NonClustered Index IDX_TP4 On #TempProductHierarchy(ProductId)



Use With(NOLOCK)
NOLOCK typically (depending on your DB engine) means give me your data, and I don't care what state it is in, and don't bother holding it still while you read from it. It is all at once faster, less resource-intensive

Use NOLOCK on master tables and not everywhere.
Using NoLock may become dangerous sometimes, so check the query exec plan when in doubt.

Select * from  ProductHierarchy  WITH(NOLOCK)











Join On the InputParameter
This will reduce the operation size and the disk read

Join on input parameter instead of filtering in where condition.
As you can see here we have joined on  setupid i.e the input parameter instead of  joining on the setupid of the other table  and then filtering in where condition. 

SELECT * from trans.Promotion P
inner join masters.Product PP with(NOLOCK)   on PP.SetupId=@local_SetupId
inner join masters.Customer CC with(NOLOCK) on CC.SetupId=@local_SetupId

Instead Of
SELECT * from trans.Promotion P
inner join masters.Product PP with(NOLOCK)   on PP.SetupId=P.SetupId
inner join masters.Customer CC with(NOLOCK) on CC.SetupId=P.SetupId
Where PP.SetupId=@local_SetupId


Prevent the use of “Select *”

Causes Indexing issues and Binding issues

Dont use "SELECT *" in a SQL query

Insead use Select ProductId, ProductName from Products



Use EXISTS

To check if any data exists in a particular table, use EXISTS  instead of relying on Count its more effective.


SELECT OrderId,AmendVersion FROM trans.OrdersWHERE EXISTS (SELECT top 1 OrderId FROM trans.OrderExtract WHERE SetupId=1099)



Use Local Temp Tables (#TempTableName)







Prevent using Global hash tables (##)


SELECT * into #TempProduct FROM(SELECT ProductId,LevelId,ProductCodeInterface,Name from Product where SetupId=@local_SetupId and IsActive=1) AS TP                                          








Use Local Variables to Store the FunctionCall return

Prevent calling the string functions or date functions Over and over again, instead store ‘em in local variables  if you are going to reuse the value.

Declare @sampleString varchar(max) = 'nevermind the promotions'
Declare  @sizeOfString int
set @sizeOfString= len(@sampleString)
SELECT @sizeOfString




Use Try - Catch

BEGIN TRY
-- Logic / Query here
END TRY           
            
BEGIN CATCH           
 -----------------------------------------------------------------------           
           
  DECLARE           
      @ErrMsg  VARCHAR(255) -- Error Message           
     ,@ErrNo   INT    -- Error Number           
     ,@ErrSeverity INT    -- Error Severity           
     ,@ErrProc  VARCHAR(255) -- Error Procedure           
     ,@ErrLine  INT    -- Error Line           
               
  SELECT           
      @ErrMsg  = ERROR_MESSAGE()           
     ,@ErrNo   = ERROR_NUMBER()           
     ,@ErrSeverity = 17           
     ,@ErrProc  = ERROR_PROCEDURE()           
     ,@ErrLine  = ERROR_LINE()           
                           
  RAISERROR  (           
      @ErrMsg           
     ,@ErrSeverity           
     ,1           
     ,@ErrNo           
     ,@ErrLine           
     ,@ErrProc           
     )           
 -----------------------------------------------------------------------           
 END CATCH           
END           
           
           
           
     
 Use  SET NOCOUNT ON

Whenever we write any procedure and execute it a message appears in message window that shows number of rows affected with the statement written in the procedure.

When SET NOCOUNT is ON, the count is not returned.

SET NOCOUNT ON
Select PromotionId from Promotion 


Prevent Usage of DDL Statements

Do not try to use DDL statements inside a stored procedure that will reduces the chance to reuse the execution plan.
DDL statements like CREATE,ALTER,DROP,TRUNCATE etc.


Use Alias



If an alias is not present, the engine must resolve which tables own the specified columns. A short alias is parsed more quickly than a long table name or alias. If possible, reduce the alias to a single letter

--Wrong Statement
SELECT PromotionId , P.VersionedPromotionId, Name, PIE.InvestmentTypeId from Promotion P
Inner join PromotionInvestment PIE on PIE.VersionedPromotionId=P.VersionedPromotionId
where P.Name='Blah'

--Correct Statement
SELECT P.PromotionId , P.VersionedPromotionId, P.Name, PIE.InvestmentTypeId from Promotion P
Inner join PromotionInvestment PIE on PIE.VersionedPromotionId=P.VersionedPromotionId
where P.Name='Blah'



Don't use UPDATE instead of CASE

Take this scenario, for instance: You're inserting data into a temp table and need it to display a certain value if another value exists. Maybe you're pulling from the Customer table and you want anyone with more than $100,000 in orders to be labeled as "Preferred." Thus, you insert the data into the table and run an UPDATE statement to set the CustomerRank column to "Preferred" for anyone who has more than $100,000 in orders. The problem is that the UPDATE statement is logged, which means it has to write twice for every single write to the table. The way around this, of course, is to use an inline CASE statement in the SQL query itself. This tests every row for the order amount condition and sets the "Preferred" label before it's written to the table.






Avoid Functions on RHS

Dont use this

select *
from Promotion
where YEAR(StartDate) = 2015
and  MONTH(StartDate) =

Use this

Select *
From Promotion
Where StartDate between '6/1/2015'
    and '6/30/2015'



 Specify optimizer hints in SELECT

most cases the query optimizer will pick the appropriate index for a particular table based on statistics, sometimes it is better to specify the index name in your SELECT query.
Do not use this unless you know what you are doing.

SELECT *
FROM Promotion
WITH ( Index(IdxPromotionId))
WHERE Name = 'blah'
and Setupid=1099 




Hope these tips will help you prevent and solve the timeout exception  you face. If you want to add any please mention in the comments.

Monday, 21 April 2014

Consuming the Twitter API v1.1

A short tutorial on Consuming the Twitter API 1.1
Consuming Twitter API in Dot Net c#, using this you can collect tweets aggregate it to do some data-mining or for a simple widget on your website to display your recent tweets or tweets about any particular topic or entity. Here I shall show a simple way to search for tweets of a particular entity and display it on a GridView of a ASP.Net Application.



Twitter is an online social networking and microblogging service  that enables users to send and read short 140-character text messages, called "tweets"

Companies have already started using twitter as it is the new way to promote, connect and brand a company.
Why?? – Connecting with customers, InstantFeedBack, Latest News, Marketing etc

Why Twitter Analytics are Imp?
Analyzing your followers, your own activity, and what other people are talking about online are all smart ways to make sure you’re getting the most benefit out of your social media presence.
see what users are saying about your business or products, and monitor Twitter feedback as best as possible.




Twitter API v1.1 

Twitter bases its application programming interface (API) off the Representational StateTransfer (REST) architecture. 
By allowing third-party developers partial access to its API, Twitter allows them to create programs that incorporate Twitter's services.


Twitter supports a few authentication methods and with a range of OAuth authentication styles you may be wondering which method you should be using. When choosing which authentication method to use you should understand the way that method will affect your users experience and the way you write your application.




Initially Twitter had API v1 which was pretty much straight forward 
Where you constructed the url and pinged it to return ur result in JSON or XML

Then when the API was updated to v1.1 many third party apps were affected coz of it.
There are many reasons for this, like request limitations to prevent the abuse of the service.


In this article I shall focus on getting the data of a twitter home-timeline or a twitter search.

If you use the...
Send...
REST API
Streaming API

What can it do?


Steps
Create an twitter account
Head to dev.twitter.com/apps/ and log in using your Twitter ID and password. 

Click the Create a new application button and enter the name and description of your application. The website should be a page where you can download your code but, since you’re still writing it, enter your home page URL and change it later. Leave the callback URL blank.



Next
•Create an Access Token
Click the Create my access token button at the bottom of the Details tab on your application’s page. You’ll now see various strings against:

•OAuth: Consumer key
•OAuth: Consumer secret
•Token: Access token
•Token: Access token secret


Now coming to Visual Studio

Hope you got nugget Manager if not install it from extension manager

Create a ASP Application Project.

Go to Nuget Package Manager-> Package Manager Console

In the console that will appear at the bottom of VS type in
Install-Package TweetSharp

TweetSharp is a Twitter API library that greatly simplifies the task of adding Twitter to your desktop, web, and mobile applications. You can build simple widgets, or complex application suites using TweetSharp.


using TweetSharp; // In v1.1, all API calls require authentication

var service = new TwitterService(_consumerKey, _consumerSecret);
service.AuthenticateWith(_accessToken, _accessTokenSecret); 

Here we see we need to replace the keys and tokens which we had generated earlier from the twitter website.

var tweets = service.ListTweetsOnHomeTimeline(new ListTweetsOnHomeTimelineOptions());
foreach (var tweet in tweets)
{
    Console.WriteLine("{0} says '{1}'", tweet.User.ScreenName, tweet.Text);
}
You can try out this simple app on console, that will get all the statuses from your current timeLine


Here is my app , 
A simple web app with a gridView to hold the tweets. All tweets that contains "Bruce Wayne" are searched from twitter and displayed here.



The code:
Default.aspx.cs

using TweetSharp;
 
 
namespace Tweetz
{
    public class TweetAttribs {
 
        public string imageUrl { getset; }
        public string userName { getset; }
        public string tweetText { getset; }
 
 
}
 
    public partial class _Default : System.Web.UI.Page
    {
 
        protected void Page_Load(object sender, EventArgs e)
        {
            List<TweetAttribs> tweetsList = new List<TweetAttribs>();
            var service = new TwitterService("oHLA""OgKbtiOg");
            service.AuthenticateWith("22fwghS7T""HMEB5jvSVve");
            var options = new SearchOptions { Q = "Bruce Wayne"  };
 
            var tweets = service.Search(options);
            foreach (var tweet in tweets.Statuses)
            {
                TweetAttribs taObj = new TweetAttribs();
                taObj.userName = tweet.User.ScreenName;
                taObj.imageUrl = tweet.User.ProfileImageUrl;
                taObj.tweetText = tweet.Text;
                tweetsList.Add(taObj);
            }
 
            GridView1.DataSource= tweetsList.ToList();
           // GridView1.BackImageUrl = "http://9to5mac.files.wordpress.com/2012/07/screen-shot-2012-07-08-at-8-45-25-pm.png";
            GridView1.DataBind();
 
        }
    }
}

note:  The keys used are samples, please generate your own and then run the code.

Default.aspx
        <asp:GridView ID="GridView1" runat="server" AutoGenerateColumns="False">
            <Columns>
                <asp:ImageField DataImageUrlField="imageUrl" HeaderText="Pic">
                </asp:ImageField>
                <asp:BoundField DataField="userName" HeaderText="ScreenName" />
                <asp:BoundField DataField="tweetText" HeaderText="Tweet" />
            </Columns>
 
        </asp:GridView>

References:
So just with a few lines of code and with some major help from tweet sharp we were able to fetch the info we wanted.
TweetSharp simplifies things a lot.