Software System Scalability...

Scale out and Scale up is the requirements coming after you have run your system for a while in customer’s environment. The imagination of these concepts coming into the system architecting with the experience you got as a Programmer, Designer, and Architect. But having enough consideration up front will save your customer impression on you in a better place.

Recently we found same scenario to run through that one of our major product was struggling to deal with large scale data files, from both incoming and outgoing. In early days in customer environment, system was running in acceptable time duration in import and exports the data files with number of clients. By the time new clients were added to the system and the amount of the data to be deal with went high. System was struggling to response to this mass amount of data to be worked with.

Finally discussions ended up with to have a restructuring of the architecture of the system to deal this sort of bulky data files. Means we needed to have a door in the system API to deal with these large scale data files and directly push them into the system without transferring data through rushed network data channels.

And lots of Database retrieval and insertion scenarios were restructured to minimize unnecessary resource consuming and used some database data processing optimization techniques like indexing, remove few computed columns unnecessary.

Here are the things we came up with as point wise

1 1. Changes Made in API to accept list of entity data to accept large number of business entity data at once.

2 2. Minimized database connection open and close times. Always tried keep one connection opened to communicate with the database as possible. For an example huge number of data records are inserted to the data with one connection opened. There were some situations it was not possible to use one connection opened throughout all the process.

3 3. Data inserting and Data retrieval Optimization – introduce indexing for some Database Table fields. And removed few computing columns which are taking mass of times unnecessarily in data inserting and the retrievals.

Had lots of discussion in the mean time of process to restructure the system and everything has been ended up with unexpected performance gains. Now it’s looking great when its compared to the earlier version of the system.

Team effort brought system into a great level now.

97 Tips for Software Architects

I got this List of Tips for the Software Architects from the Kalahara's Blog and Credit Should go to him (One of the admirable Architect I worked with). I published it here for my references. Here is the list… 1. Don't put your resume ahead of the requirements 2. Simplify essential complexity; diminish accidental complexity 3. Chances are your biggest problem isn't technical 4. Communication is King; Clarity and Leadership its humble servants 5. Architecting is about balancing 6. Seek the value in requested capabilities 7. Stand Up! 8. Skyscrapers aren't scalable 9. You're negotiating more often than you think 10. Quantify 11. One line of working code is worth 500 of specification 12. There is no one-size-fits-all solution 13. It's never too early to think about performance 14. Application architecture determines application performance 15. Commit-and-run is a serious crime. Respect your Colleagues 16. There Can be More than One 17. Business Drives 18. Simplicity before generality, use before reuse 19. Architects must be hands on 20. Continuously Integrate 21. Avoid Scheduling Failures 22. Architectural Tradeoffs 23. Database as a Fortress 24. Use uncertainty as a driver 25. Scope is the enemy of success 26. Reuse is about people and education, not just architecture 27. There is no 'I' in architecture 28. Get the 1000ft view 29. Try before choosing 30. Understand The Business Domain 31. Programming is an act of design 32. Time changes everything 33. Give developers autonomy 34. Value stewardship over showmanship 35. Warning, problems in mirror may be larger than they appear 36. The title of software architect has only lower-case 'a's; deal with it 37. Software architecture has ethical consequences 38. Everything will ultimately fail 39. Context is King 40.It's all about performance 41. Engineer in the white spaces 42. Talk the Talk 43. Heterogeneity Wins 44. Dwarves, Elves, Wizards, and Kings 45. Learn from Architects of Buildings 46. Fight repetition 47. Welcome to the real world 48. Don't Control, but Observe 49. Janus the Architect 50.Architects focus is on the boundaries and interfaces 51.Challenge assumptions - especially your own 52.Record your rationale 53.Empower developers 54.It is all about the data 55.Control the data, not just the code 56.Don't Stretch The Architecture Metaphors 57. Focus on Application Support and Maintenance 58. Prepare to pick two 59. Prefer principles, axioms and analogies to opinion and taste 60. Start with a Walking Skeleton 61. Share your knowledge and experiences 62. Make sure the simple stuff is simple 63. If you design it, you should be able to code it 64. The ROI variable 65. Your system is legacy, design for it 66. If there is only one solution, get a second opinion 67. Understand the impact of change 68. You have to understand Hardware too 69. Shortcuts now are paid back with interest later 70. "Perfect" is the Enemy of "Good Enough" 71. Avoid "Good Ideas" 72. Great content creates great systems 73. The Business Vs. The Angry Architect 74. Stretch key dimensions to see what breaks 75. Before anything, an architect is a developer 76. A rose by any other name will end up as a cabbage 77. Stable problems get high quality solutions 78. It Takes Diligence 79. Take responsibility for your decisions 80. Dont Be a Problem Solver 81. Choose your weapons carefully, relinquish them reluctantly 82. Your Customer is Not Your Customer 83. It will never look like that 84. Choose Frameworks that play well with others 85. Making a strong business case 86. Pattern Pathology 87. Learn a new language 88. Dont Be Clever 89. Build Systems to be Zuhanden 90. Find and retain passionate problem solvers 91. Software doesnt really exist 92. Pay down your technical debt 93. You can't future-proof solutions 94. The User Acceptance Problem 95. The Importance of Consommé 96. For the end-user, the interface is the system 97. Great software is not built, it is grown Regards Lakmal

Introduction to Software Architecture (From the Books)

The goal of software design is to build a model that meets all customer requirements and leads to successful implementation. What is the architecture design? The architecture design defines relationship between structural element of the software, Styles and design patterns that can be used to achieve requirements defined for the system, and the constraints that can affect the in which architecture is implemented.(Galan and Shaw, 1996). IEEE Std 1471 defines system architecture as "Fundamental organization of a system embodied in its elements, their relationship to each other, and to the environment, and the principle guiding its design and evolutions". Who is responsible for the Architecture Design? Software Architect and the designers, Translate system requirements into Software architecture design. Why software Architecture is so important? Poor design may leads the system to not to meet the customer requirements Frailer to adapt to the future changes. May not be re-usable Exhibits unexpected behaviors Perform badly Etc. When the Software Architecture Design is conducted? After Software requirement Specification is finalized, Software Architect and the designers can start working software Architecture Design. What is the outcome of a Software Architecture design? Software Design Description (SDD) Design Overview, Purpose, And the Scope Decomposition of description (Modules, Data and Processes) Dependency and Connection description (Between the modules, data,, and processes) Attributes User Interface Description Detailed Design Architecture design is front strategy design for the detailed design.. During Architecture Design Stage, User Accessible Elements Interconnection that are visible to the stakeholders. Detailed Design, (Also called tactical design) Local Design Constraints Internal details of each element A functional requirement specifies the functionality of the system whereas a nonfunctional requirement specifies the system qualities, Constraints, and Behaviors. List of tasks that has to be done by the software architect.. Perform static partition and decomposition of a system into sub systems and communication among sub systems. Software element can be configured, delivered, developed, deployed, and is replaceable in the future. Each element's interface encapsulates details and provides loose coupling with other elements. Establish dynamic control relationships among different sub systems in terms of data flow, flow control orchestration, or message dispatching. Consider and evaluates alternative architecture styles that can suit the problem domain at hand. Perform tradeoff analysis on quality attributes and other non functional requirements during the selection of architecture style. Architecture Styles Architecture style abstracts common properties of family of similar designs. What's includes in Architecture style? Rules Constraints Set of Patterns of how to structure set of elements and connectors. What are the key components of an Architecture Style? Elements that perform functions required by the system. Connectors that enable communication, coordination and cooperation among the elements of the system. Constraints that defines how elements can be integrated to form the system Attributes that describes the advantages and disadvantages of the chosen structure. Software Quality Attributes There are three types of software quality attributes are discussed in Software Architecture Design. Software Implementation Attributes Interoperability Universal Accessibility Ability Exchange data among internal components and the outside world Loose dependency of infrastructure Maintainability and Extendibility Testability Portability Systems level independence of Software and Hardware platforms Scalability Ability to adapt into new customer requirement changes Flexibility Ease of system modification to cater to different environments or problems for which the system was not originally designed. Runtime Attributes Availability System's capability to be available 24/7. Security Ability to cope with malicious attacks from outside or inside the system Security can be achieved by installing firewalls, establish authentication and authorization process, and using encryption. Performance Response Time Throughput Resource utilization Usability The Level of human satisfaction of the using system. Completeness Correctness Compatibility Friendly User Interfaces Complete Documentation Technical Support Reliability Failure frequency, Accuracy of output results The Mean-Time –to Failure Ability to recover from Failure The Failure Predictability Maintainability Extensibility Adaptability Serviceability Testability Compatibility Configurability Ease of software system to change Business Attributes Time to market Time it's takes from requirements analysis to product release Cost Expense of building, Maintaining and Operating Lifetime Period of the product alive before retirement. Typical Quality Attributes of Tradeoff pairs Tradeoff between space and time Example, Increase the time efficiency of a hash table means a decrease of in its space efficiency. Tradeoff between Reliability and Performance Tradeoff Between Scalability and Performance Software Architecture Design Guidelines Thinks of what to before thinking how to do it. Think of abstract design before thinking concrete design. Think of non-functional requirements early in the design process This of software reusability and extendibility as much as possible Try to perform high cohesion within each element and loose coupling Tolerate refinement of design Avoid ambiguous design and over detailed design. Summery Software Architecture Specification consists of software elements, Connectors and collaboration among the elements and desired quality attributes. An Architecture style is set of rules, constraints, or patterns guide how to structure a system into set of elements and connectors, and how to govern overall structure design patterns of constituent element types and their run time interaction.

25 Famous Thinkers and Their Inspiring Daily Rituals

Daily Rituals, that’s depends person to person, but it’s always nice to have a read on world’s famous people’s daily rituals.

Found a brief reveal of 25 of world famous thinkers, writers and politician’s daily rituals. I always find interesting to read about the people who contributed more to change the world.

Here is the article

Lakmal

Top 100 Blogs in the World..

I Found this list of World's Top 100 Blogs from the www.noop.nl and all credit should go to them. For my references I publish the same list here

TT	LT	Blog / Site	Author / Twitter
1	1	Joel on Software	Joel Spolsky
2	3	Coding Horror	Jeff Atwood
3	6	Martin Fowler's Bliki	Martin Fowler
4	11	The Daily WTF	(various)
5	67	Signal vs. Noise	(various)
6	5	Scott Hanselman's Computer Zen	Scott Hanselman
7	10	Rands in Repose	Michael Lopp
8	7	Bokardo: Social Design	Joshua Porter
9	8	Stevey's Blog Rants	Steve Yegge
10	19	Stack Overflow	Jeff Atwood
11	12	Eric.Weblog()	Eric Sink
12	9	Lambda the Ultimate	(various)
13	13	Raible Designs	Matt Raible
14	14	Dare Obasanjo aka Carnage4Life	Dare Obasanjo
15	17	Jon Udell	Jon Udell
16	15	UIE Brain Sparks	Jared Spool
17	23	ThoughtBlogs	(various)
18	27	CodeBetter.Com	(various)
19	25	High Scalability	(various)
20	28	Object Mentor Blog	(various)
21	41	Regular Geek	Rob Diana
22	21	Otaku, Cedric's Weblog	Cedric
23	47	Mike Cohn's Blog: Succeeding with Agile	Mike Cohn
24	37	NOOP.NL	Jurgen Appelo
25	18	The Berkun Blog	Scott Berkun
26	38	Good coders code, great reuse	Peteris Krumins
27	61	James Bach’s Blog	James Bach
28	20	Artima Weblogs	(various)
29	33	Google Testing Blog	(various)
30	22	J.D. Meier's Blog	J.D. Meier
31	--	Alex Payne	Alex Payne
32	26	10x Software Development	Steve McConnell
33	29	secretGeek	Leon Bambrick
34	--	Business of Software Blog	Neil Davidson
35	--	Contrast | The Blog	(various)
36	34	Enterprise Architecture: From Incite comes Insight...	James McGovern
37	43	Project Shrink	Bas de Baar
38	45	StevenHarman.net	Steven Harman
39	24	Shanine.com / omar /	Omar Shahine
40	66	Software by Rob	Rob Walling
41	57	Agile Testing	Grig Gheorghiu
42	36	Alistair Cockburn	Alistair Cockburn
43	77	Agile Software Development	(various)
44	56	Tyner Blain	Scott Sehlhorst
45	44	Interoperability Happens	Ted Neward
46	48	Pure Danger Tech	Alex Miller
47	39	Managing Product Development	Johanna Rothman
48	64	Elegant Code	(various)
49	58	Coding the Architecture	(various)
50	--	Ted Leung on the Air	Ted Leung
51	--	Relevance Blog	(various)
52	49	Reforming Project Management	Hal Macomber
53	46	James Shore: The Art of Agile	James Shore
54	59	Knowing.NET	Larry O'Brien
55	--	Blankenthoughts	Jeff Blankenburg
56	79	David Chelimsky	David Chelimsky
57	82	Aligning Technology, Strategy, People & Projects	Eric Brown
58	40	It's Just a Bunch of Stuff That Happens	Eric Burke
59	30	Dr. Dobb's CodeTalk	(various)
60	50	Better Projects	Craig Brown
61	75	Lean Software Engineering	Corey Ladas
62	51	{Codesqueeze}	Max Pool
63	68	/\ndy	Andy Hunt
64	69	All About Agile	Kelly Waters
65	72	Herding Cats	Glen Alleman
66	97	Exploration Through Example	Brian Marick
67	42	Object Technology	Jeff Sutherland
68	55	Implementing Scrum	Mike Vizdos
69	52	Agile Management Blog	David Anderson
70	96	Bit-Player	Brian Hayes
71	--	The Mendicant Bug	Jason Adams
72	--	Software Project Management	Pawel Brodzinski
73	--	A Software Insiders Point of View	R "Ray" Wang
74	71	Meme Agora	Neal Ford
75	85	Word Aligned	Thomas Guest
76	84	Chris Spagnuolo's EdgeHopper	Chris Spagnuolo
77	65	Evolving Web	Jim Benson
78	53	The Braidy Tester	Micahel
79	--	Tester Tested!	Pradeep Soundararajan
80	86	The Cutter Blog	(various)
81	74	Jeffrey Palermo (.com)	Jeffrey Palermo
82	93	Test Obsessed	Elisabeth Hendrickson
83	--	averyBlog	James Avery
84	--	GrokCode	Jess
85	--	Wide Awake Developers	Michael Nygard
86	--	Notes from a Tool User	Mark Levison
87	60	Petzold Book Blog	Charles Petzold
88	87	Agile Developer Venkat's Blog	Venkat Subramaniam
89	95	Creative Chaos	Matthew Heusser
90	--	Moserware	Jeff Moser
91	--	Codemonkeyism	Stephan Schmidt
92	88	Collaborative Software Testing	Jonathan Kohl
93	90	Jeff Patton's Holistic Product Design & Development	Jeff Patton
94	32	Curious Cat	John Hunter
95	--	Yourdon Report	Ed Yourdon
96	81	LeadingAnswers	Mike Griffiths
97	99	Silk and Spinach	Kevin Rutherford
98	--	Lazycoder	Scott Koon
99	92	Clarke Ching - More Chilli Please	Clarke Ching
100	--	Raven's Brain	Raven Young

Lakmal

Agile

Agile is so close to the process what we learned in university that Spiral Process Of software development. But Agile is not only about Spiral model, it has more meaning than that.

First, Agile is a philosophy (Conceptual Framework); it's not a Process Framework. There are process frameworks, developed by using agile philosophy, like, XP, SCRUM, Lean, DFDM, FDD Crystal and etc.

Agile - Group of Philosophies that provides the abilities to handle changing requirements

• Lean - Move closer to customer, shorter cycles, eliminate waste, decide as late as possible, empower the team, build in integrity
  
• DSDM - Empower the team to make decisions, emphasize frequent product delivery, integrate testing throughout, promote collaboration and cooperation between all stakeholders
  
• FDD - Center development around the feature, create a domain model with domain experts
  
• Crystal - Emphasize people, gather techniques from other methods, improve communications, adapt the process itself (shrink or grow to fit)
  
• Scrum - Manage a prioritized list of requires on a product backlog, collaborate through daily standup meetings, exhibit the product upon iteration completion, use retrospectives to correct the process
  
• XP - Emphasize the values of communication, simplicity, feedback, and courage; use specific technical and collaborative practices, including TDD, re-factoring, pair programming, continuous integration, open workspace, and automated acceptance tests
  

Agile Manifesto

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Twelve principles underlie the Agile Manifesto, including:

• Customer satisfaction by rapid, continuous delivery of useful software
• Working software is delivered frequently (weeks rather than months)
• Working software is the principal measure of progress
• Even late changes in requirements are welcome
• Close, daily cooperation between business people and developers
• Face-to-face conversation is the best form of communication (co-location)
• Projects are built around motivated individuals, who should be trusted
• Continuous attention to technical excellence and good design
• Simplicity
• Self-organizing teams
• Regular adaptation to changing circumstance

Regards

Lakmal