Scale out and Scale up is the requirements coming after you have run your system for a while in customer’s environment. The imagination of these concepts coming into the system architecting with the experience you got as a Programmer, Designer, and Architect. But having enough consideration up front will save your customer impression on you in a better place.
Recently we found same scenario to run through that one of our major product was struggling to deal with large scale data files, from both incoming and outgoing. In early days in customer environment, system was running in acceptable time duration in import and exports the data files with number of clients. By the time new clients were added to the system and the amount of the data to be deal with went high. System was struggling to response to this mass amount of data to be worked with.
Finally discussions ended up with to have a restructuring of the architecture of the system to deal this sort of bulky data files. Means we needed to have a door in the system API to deal with these large scale data files and directly push them into the system without transferring data through rushed network data channels.
And lots of Database retrieval and insertion scenarios were restructured to minimize unnecessary resource consuming and used some database data processing optimization techniques like indexing, remove few computed columns unnecessary.
Here are the things we came up with as point wise
1 1. Changes Made in API to accept list of entity data to accept large number of business entity data at once.
2 2. Minimized database connection open and close times. Always tried keep one connection opened to communicate with the database as possible. For an example huge number of data records are inserted to the data with one connection opened. There were some situations it was not possible to use one connection opened throughout all the process.
3 3. Data inserting and Data retrieval Optimization – introduce indexing for some Database Table fields. And removed few computing columns which are taking mass of times unnecessarily in data inserting and the retrievals.
Had lots of discussion in the mean time of process to restructure the system and everything has been ended up with unexpected performance gains. Now it’s looking great when its compared to the earlier version of the system.
Team effort brought system into a great level now.
