High Speed Efficient Data Upload (ASP.NET/SQL DATA)

Hi, This can be achieved by using OPENXML in SQL Server to import data. Application will pass all data in form of XML to SQL Stored Procedure. Stored Procedure will parse that data in populates in a temporary table. After populating INSERT/UPDATE will be performed using joins with permanent tables. I have 8.5 years experience in SQL Server 2005|2008|2008R2|2012 and involve in designing databases, Indexing Strategies, table partitioning, writing Stored Procedures, Triggers, functions and views. I have expertise in writing complex queries with proper joins and index selection. In past, i involved in balancing between concurrency and locking and lock escalation. Thanks

By bulk copy, do you mean the BCP utility? I can code both methods, BCP and BULK INSERT so we can compare which is faster. There is however another method. Using the SQL Server Integration Utility.

Hello, I am very much interested to do this work as i am experienced a lot with asp.net. I also did similar work before. Your requirements could be done in several ways as you also mentioned. I will prefer BulkInsert as it is most faster, but in that case field mapping is very important. I will consider another thing, if you have a primary key column, to check each data existence in the database, i will load the DB's existing primary key value in memory, then to check each record, i will do it in memory rather than going to database for each record, this will effect less time consuming. Anyway, looking forward to have a positive response from you. Thanks

Dear Sir, If you are trying to batch update / insert bulk of data to remote SQL Server; it is not an efficient solution to update records either in batch or record by record. I suggest if you do not have SQL Server on web server, then install IIS on SQL Server if not installed already, then create an FTP site, now upload file using your UI directly to FTP site at specific location and execute a stored procedure to load bulk data. As you require if record exists then update else insert; for this purpose we normally uses change tracking feature or sync framwork; but as we are trying to compare existing records with file, so I need to determine best and efficient solution for you after reviewing your data and technical scenario. Looking forward to hearing from you! Regards, Habib ur Rehman PMP, MCSE - BI MCSA - SQL Server MCSD - Web Applications

hi i have a sound knowledge of development and have 7 years of experience tell me when we can discuss about the same

Hi, I have gone through your description.I have understood your requirement and I can do this job confidently. Please let me know if you have any concern.

Good morning, I'm experienced SQL Server developer and I have a huge background working with integration solutions, ETL and BI systems. You can check my linkedin profile to review my competency. (in/viniciusdavila) We can do that data import with a few techniques, and it's necessary to understand your environment before. 1* We can build a integration package (not so good and complicated to maintain) 2* We can use bulk insert (better performance but limited) 3* We can create C# SQL CLR Procedures and deploy at your database (amazing performance) With SQL CLR we can do many workarounds to increase your performance, like: 1* Upload a zip package to your database 2* Unpack the zip before the imports 3* Insert in your tables with parallel processing over threads. I will study your needs and provide the best solution. I'm eager to get a answer from you, if you have any questions don't hesitate to ask.

i can do you it. i have a experience in visual basic. you can assign this to me. i am doing in visual basic since more than a year and have a good command on it.

We can use EF 6.1.0 to make it fast and also we can use TPL , if in case it can be applied, so that can have faster thread performance. Thanks.

Dear Sir/Madam, My name is Andriy Bilan and I’m a software developer with more than 10 years of experience in IT. Last 6 years I work with C#/.NET (+ tools/technologies around it) and Databases (mainly SQL Server) I’m very interested in project you published and I’d like to offer my services to you. You asked about technical description of my approach for your project. I would like to see your DB structure and clarify couple things, but here is my initial plan: 1. Use bulk insert to transfer all data from machine with webserver to the machine with SQL Server. All data will be inserted into TEMP TABLES(S); 2. For update: update PersistentTableWithData set ... from PersistentTableWithData inner join TempTableWithData on ... 3. Similar approach for insert. As you see, data will be transferred only once - for bulk insert, all other operations (including search & check for an existing record) will be executed on the SQL Server machine side by SQL Server engine itself. I expect this approach should be fast enough to meet your needs. Timeframe for this project is 7 days. And I have one questions: will you be able to send me backup of your DB (possibly empty or with demo data) so that I will be able to work on this project on my computer? Please, contact me if you think I'm the right person for this project or in case any questions. Thank You, Andriy Bilan.

Uploading Data from Excel Methodology: -Use ADO framework and reference library -Take connection string of server, password and username -Establish ADO connection with server -Execute SQL Queries (like INSERT) through VBA connection query execute command -Optimize insertion algorithm (this depends on data structure. I upload rows one by one which gets updated as records in SQL database. Tried uploading Data blocks of excel but found very tedious on SQL query side. So will stick to one by one row upload) -If any, conditions need to be created, example if data already exists and needs to be checked for duplicacy or any such criteria, then will have to use Recordset function of ADO to fetch SQL query results into VBA This is what I can do, if this fits your need. Please get in touch. Thanks.

Purpose:- Load data from a delimited text file into SQL Server as FAST AS POSSIBLE. Present Scenario:- Code already exists, and works that does the following: a) Upload from client to webserver b) Read the text file into dataset(s) on the webserver. Problem Statement:- The current code for this is too slow. Proposed Solution:- The project will be carried out mostly on the DataBase server side using a better option/feature in SQL Server called “SQL Merge” where the data manipulation is faster than bulk activities as it allows us to perform all the update activities in one pass of the DataBase table(s). The data will be passed to the DB side using a Stored proc parameter of type - table. However knowledge of the table relations is of utmost importance to keep the referential integrity. I plan to put constraints on the child tables with foreign key relationships with cascading update and delete statements so that the related data also gets deleted when the related row is deleted from the parent table. Assumptions:- 1) The rest of the application is built in .NET 2) DataBase table knowledge / references and relationship will be provided so that the same can be maintained while the update is being done. Timeline (Days) :- 1-2: Understanding the environment and work already done. 3: Design, prototyping. 4-5: Development of the system based on the prototype 6: Overall integration 7-8: Testing, revision and integration testing 9-10: Documentation