Our tools are only as good as our pattern language.
Diagnostics is the mother of problem solving.
All areas of human activity involve the use of diagnostics. Proper diagnostics identifies the right problems to solve. We are now a part of a non-profit organization dedicated to the developing and promoting the application of such diagnostics: systemic and pattern-oriented (pattern-driven and pattern-based).
Previously we introduced Pattern-Oriented Debugging Process where software diagnostics played the major role as a part of debugging. However, in the case of a separate software diagnostic process, we introduce Pattern-Oriented Diagnostic Analysis Process, that incorporates diagnostic analysis requirements elicitation from problem description analysis and diagnostic report construction. Both new additions require separate pattern catalogs. Problem description analysis pattern catalog is already being extended, and the new catalog for diagnostic report construction is under development and will be published soon. The central process part, diagnostic artifact analysis, already has two extensive analysis pattern catalogs for software log analysis and memory analysis. The process is illustrated in the following diagram:
General trace and log analysis patterns allow application of uniform problem detection and solving approach across diverse software environments. This pattern language covers any execution artifact from a small debugging trace to a distributed log with billions of messages from hundreds of computers, thousands of software components, threads, and processes. Pattern-oriented trace and log analysis is applicable to troubleshooting and debugging Windows, Mac OS X, Linux, FreeBSD, Android, iOS, z/OS, and any other possible computer platform including networking and IoT. Its pattern catalog is a part of pattern-oriented software diagnostics, forensics, prognostics, root cause analysis, and debugging developed by Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org). This reference reprints with corrections 133 patterns originally published in Memory Dump Analysis Anthology volumes 3 - 9 and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog). Full-color diagrams accompany most pattern descriptions. The second edition includes 33 more patterns and improved pattern index.
By connecting various memory spaces (user, kernel, physical, virtual, orbifold, manifold, fiber bundle, adjoint), trace and log spaces, and problem description narratives we introduce Software Diagnostics Space as a search space for finding problem patterns using general and concrete analysis patterns. Using mathematical metaphors we view it as a general graph of statements from Software Problem Narrative (graph vertices) and various software narratives such as logs, traces and memory spaces (edges). Software problem narratives may be different from software problem descriptions (which we get from software users and which have their own analysis patterns) because they are controlled narratives of actor interactions while working with software (top right corner of software narratology square). For completeness, every software narrative edge has vertices by default as start and stop vertices.
Let’s look at one example depicted in the following diagram:
We have the problem description from a user who couldn’t exercise some software functionality unless some service was restarted. This is a problem description narrative (no. 1). A software support person constructed the problem reproduction setup narrative (no. 2) and recorded problem narrative no. 3 – 5 with tracing the client and server software and taking memory snapshots (Adjoint Space trace and log analysis pattern) of the corresponding service and another Coupled Process (memory analysis pattern).
This can all be depicted in the following general graph (multigraph) diagram where loops show adjoint spaces (“instantaneous” artifact snapshots like memory, data):
Such graphs may not be connected, and loops may be replaced by additional multiple edges with extra vertices.
The practical usage of such graphs can be demonstrated by their construction during problem analysis. Suppose that we have a problem description:
After its analysis we construct a problem narrative:
During its trace analysis we identify needed software trace edges:
We add multiple edges if tracing involves several software systems or different trace varieties:
Contains reprinted articles in full color from Memory Dump Analysis Anthology related to pattern-oriented software diagnostics with additional comments showing the historical development of this autonomous and distinctive discipline over the last 10 years.
Previously we based software trace and log analysis on software narratology. While continuing further research and development in that direction we are now constructing a new software trace and log analysis system called TopoLog based on ideas and techniques from topology (originally called analysis situs in Latin: analysis of the situation) as a part of theoretical software diagnostics. Over the years we described a few trace and log analysis patterns based on topological metaphors: Quotient Trace, Message Cover, Fiber Bundle, Sheaf of Activities, and Adjoint Space. Before starting our pattern work on software trace analysis we considered threads as braids in abstract space, and, after the first analysis patterns, we considered multithreading as multibraiding. For general software traces and logs including memory snapshots we propose topological state analysis, for example, analysis of the covering space via open and closed Memory Regions and Region Strata.
The following direct links can be used to order the book now:
Contains revised, edited, cross-referenced, and thematically organized selected articles from Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org) and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog) about software diagnostics, root cause analysis, debugging, crash and hang dump analysis, software trace and log analysis written in March - September 2016 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software, technical support and escalation engineers dealing with complex software issues, security researchers, reverse engineers, malware and memory forensics analysts. This volume is fully cross-referenced with volumes 1 – 9a and features:
- 11 new crash dump analysis patterns
- 11 new software log and trace analysis patterns
- New structural memory pattern
- Introduction to Riemann root cause analysis language
- Introduction to problem solving as code
- Introduction to Dia|gram graphical diagnostic analysis language
- Introduction to iterative pattern-oriented root cause analysis
- Definition of theoretical software diagnostics
After writing so much about software diagnostics, we introduce its abstract generalising principles of pattern orientation and systems thinking as Theory of Software Diagnostics. We were thinking about the importance of theory for quite some time until we got acquainted with the work of Leo Klejn who coined a term “theoretical archaeology.” Then we also decided to coin the similar term for software meta-diagnostics since we compiled two books as guides to software diagnostics principles irrespective of software platforms, vendors, and their software products: Software Diagnostics and Principles of Memory Dump Analysis and plan to publish a compilation of related theoretical articles (Theoretical Software Diagnostics, ISBN-13: 978-1-908043-98-6, forthcoming September 2016). Looking for the development of theoretical archaeology as guidance makes sense because it emerged recently in contemporary times and also deals with artefacts, historical reconstruction, and time- and memory-related issues, albeit of a different nature. While working on theoretical foundations and principles for many years, we had to learn theories, ideas, and metaphors of other disciplines used in software diagnostics that we call software para-diagnostic theories by analogy with para-archaeological (coined by Klejn) theories such as history, sociology, linguistics. In his book Introduction to Theoretical Archaeology: Meta-archaeology, Klejn made a few remarks on the required theoretical education. We would like to reformulate them in relation to theoretical software diagnostics:
When we introduced A.P.M. patterns-based root cause analysis methodology (Artefacts. Patterns. Mechanisms.), it may have made an impression of a waterfall-type process with some iterations between artefact collection and diagnostic analysis when collected artefacts are not good. However, software post-construction problem solving is usually iterative, with memory dumps and software logs collected again and again after the preliminary root cause analysis.
To illustrate the iterative nature of the process we first name its stages as Artefact Acquisition for Artefacts, Artefact Analysis for Patterns (diagnostics), and Analysis of Analysis for Mechanisms (root cause analysis):
Now we rearrange these AA stages:
After the preliminary root cause analysis (Analysis of Analysis) we may need to gather more artefacts for further diagnostics and more precise RCA, and this is reflected in more focused stages:
We introduce Problem Solving as Code as a process of developing, managing, and provisioning problem-solving methods and tools. Some problem-solving methodologies such pattern-oriented problem solving developed by Software Diagnostics Institute as a part of Diagnostics Science require constantly evolving pattern catalogues which can be stored in version control systems. For example, pattern-oriented software problem solving involves pattern-oriented problem description analysis and software execution memory and trace artefact acquisition, pattern-driven and pattern-based software diagnostics (including
One of the current Software Diagnostics Institute projects is the development of Dia|gram graphical language for pattern-oriented software diagnostics, forensics, prognostics, root cause analysis and debugging. It combines the best features from:
The purpose of Dia|gram language is twofold:
Additionally, Dia|gram may be used for presentation and analysis of higher-order pattern narratives.
Software Diagnostics Institute also proposes the UML profile for Software Diagnostics with additional diagram types: artefact acquisition map, activity backtrace, and implementation internals. This work is only started, and more will be presented in subsequent articles.
Image generated by 3D-XplorMath
Incepted and named in February 2009 shortly before the first software trace and log analysis pattern was published in April the same year, Riemann Programming Language was thought of as a software problem description language capable of generating software problem-solving tools (including TaaS version). A book was planned for publication in 2010: The Riemann Programming Language (ISBN: 978-1906717605). The main motivation at that time for the name was the metaphorical correspondence between multi-valued functions represented by Riemann surfaces and software defects as alternative branches of computation. Since the significant development of pattern-oriented software diagnostics, introduction of network and performance analysis pattern languages and patterns-based root cause analysis methodology we now make Riemann Programming Language an optional coding complement to Riemann Root Cause Analysis Pattern Language. The latter includes diagnostic analysis pattern languages for trace analysis and memory analysis developed by Software Diagnostics Institute including structural memory patterns in the context of general log analysis. We can now consider another analogy with multi-valued functions where the same general diagnostic patterns in a memory dump or log can be generated by different source code. Riemann RCA Pattern Language facilitates the transformation of software narrative artefacts into much shorter analysis narratives through the process of articoding. The resulting analysis artefacts can be programmatically processed to generate diagnostic, troubleshooting and debugging configurations, classes and functions, frameworks and plugins, components and nodes. The following diagram describes this process:
The Riemann programming language should not be confused with Riemann monitoring system which was named and developed later elsewhere by a different group of people and which is about collecting events and not about their collective analysis using pattern-oriented analysis methodology developed by Software Diagnostics Institute. Regarding event monitoring, Software Diagnostics Institute also develops platform-independent software trace and log acquisition patterns for better use of various monitoring systems.
The following direct links can be used to order the book now:
The full transcript of Software Diagnostics Services Training with 28 step-by-step exercises, notes, source code of specially created modeling applications and more than 100 questions and answers. Covers more than 60 crash dump analysis patterns from x86 and x64 process, kernel, complete (physical), and active memory dumps. The 4th edition was fully reworked to use WinDbg 10 and now covers memory dumps from Windows 10 x64. It also includes optional legacy exercises from the previous editions covering Windows Vista and Windows 7.
We introduced Narratology of Things as a combination of Software Narratology of Things and Hardware Narratology. Since memory dump analysis may be considered as a part of general trace and log analysis we open a new research direction for Diagnostics of Things (DoT) based on Narratology of Things and Pattern-Oriented Trace and Log Analysis, which also includes software execution artifacts of things and pattern-oriented network trace analysis from IoT.
On the 26th of March 2006, 10 years ago, dumpanalysis.org was registered! It was still a long way towards pattern-oriented software diagnostics. The main product of our activity, Memory Dump Analysis Anthology, is now in 10 books.
The following direct links can be used to order the book now:
Contains revised, edited, cross-referenced, and thematically organized selected articles from Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org) and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog) about software diagnostics, root cause analysis, debugging, crash and hang dump analysis, software trace and log analysis written in August 2015 - February 2016 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software, technical support and escalation engineers dealing with complex software issues, security researchers, reverse engineers, malware and memory forensics analysts. This volume is fully cross-referenced with volumes 1 – 8 and features:
- 9 new crash dump analysis patterns
- 9 new software log and trace analysis patterns
- 15 Linux core dump analysis pattern variants
- New workaround pattern
- New memory dump analysis case study
- Introduction to pattern-oriented software internals, pattern paradigms, pattern stacks, pattern repertoire
- Introduction to software diagnostics canvas
- Introduction to patterns-based root cause analysis methodology
- Introduction to a protein metaphor for software traces and logs
- Definition of software diagnostics scope
- Introduction to artificial debugger and pseudo-memory dumps
- Definition of tool-centric and pattern-centric software diagnostics, forensics, prognostics
Should software diagnostics stand alone as a separate, distinct diagnostics or be a part of some other diagnostics? We considered initially 3 types of diagnostics: medical, technical, and software:
The objects of medical diagnostics are obviously humans and mostly their biological artefacts (we say mostly because linguistic and textual artifacts can also be used for diagnosis). The objects of technical diagnostics are structures and systems made from natural and artificial organic and inorganic engineering materials (Handbook of Technical Diagnostics: Fundamentals and Application to Structures and Systems, ISBN: 978-3642258497, pp. 11 – 16). The objects of software diagnostics are obviously software systems.
We define software diagnostics as “a discipline studying abnormal software structure and behavior in software execution artefacts (such as memory dumps, software and network traces and logs)” (Introduction to Philosophy of Software Diagnostics, Part 1, page 7) or, more generally, to include the context of forensics as “a discipline studying signs of software structure and behavior in software execution artefacts (such as memory dumps, software and network traces and logs)” (Pattern-Oriented Software Forensics: A Foundation of Memory Forensics and Forensics of Things, page 18).
Although, there are many conceptual similarities in general between these diagnostics, there are three features of software that set software diagnostics apart:
There are also humanistic artefacts (A New History of the Humanities: The Search for Principles and Patterns from Antiquity to the Present, ISBN: 978-0199665211) such text (including historical documents), language, literature, and music which are symbolic and digital or can be digitised. Diagnostics takes the form of literary criticism and text reconstruction (philology). Many similarities there gave rise to Software Narratology and Systemic Software Diagnostics:
One of the problems in teaching software diagnostics and postmortem debugging is a simulation of complex software problem scenarios. There are plenty of real life memory dumps available but due to security considerations they cannot be shared outside of the organisation. Here we introduce Artificial Debugger project (ArtDbg) that simulates the I/O of the real debugger, for example, WinDbg, GDB, or LLDB.
The real memory dump is analyzed in the real debugger, and all debugger input and output is saved in a log file. This log file is then scanned for any potentially sensitive information, and all such information is eliminated (Security Problem, Memory Dump Analysis Anthology, Volume 1, page 224). It is then converted into a binary pseudo-memory dump format.
ArtDbg is used to open and analyze such pseudo-memory dumps. It allows using the same real debugger commands that were used to generate the log file. Such commands will output the same information that was available from the real debugger. Some real debugger commands that were not used to generate the log file may also be used if possible.
Software Diagnostics Services plans to use the first version of ArtDbg in their new version of Accelerated Windows Memory Dump Analysis training scheduled in March 2016 to teach some very complex enterprise diagnostic scenarios in addition to WinDbg exercises with real memory dumps.
The following direct links can be used to order the book now:
The full transcript of Software Diagnostics Services training with 13 step-by-step exercises.
In 2011, we introduced iterative and incremental A.C.P. root cause analysis methodology. Its name is an abbreviation from the three main constituents: artifacts, checklists, and patterns. We get software execution artifacts such as memory dumps and software logs, use checklists to guide us in our problem analysis efforts, recognize patterns of abnormal software structure and behavior, and ultimately find out root cause(s). Recognized patterns may prompt us to revisit checklists for further guidance and request more software execution artifacts. The process is illustrated in the following diagram:
At that time, pattern-oriented software diagnostics was not yet fully developed so the proposed root cause analysis methodology was primarily debugger commands-based for memory analysis (in the form of checklists) and analysis patterns-based for software trace and log analysis where checklists were patterns-based. Pattern sequences (former pattern succession) help in finding root causes.
With the development of pattern-oriented software diagnostics, we realized the centrality of patterns and the division of patterns into general and concrete problem patterns and problem analysis patterns.
This brought the revision of A.C.P methodology where checklists become attributes of artifact collection and pattern catalogues:
However, the causal nation of root cause analysis is not explicitly mentioned in the process. Many problem patterns can be caused differently, for example, Dynamic Memory Corruption patterns can be caused by buffer overwrites and underwrites, invalid API parameters, double memory releases, and even memory manager defects resulting in similar diagnostic indicators seen in memory snapshots and traces.
Therefore, we introduce the notion of Mechanism to describe the possible cause of a diagnostic pattern. Such mechanisms replace pattern sequences as causal analysis tools. Mechanisms can also be organized into catalogues and have checklists. Mechanisms provide software internal links between software construction, post-construction, and deconstruction pattern paradigms.
The resulting iterative and incremental A.P.M. methodology (Artifacts. Patterns. Mechanisms.) is illustrated in the following diagram:
The forthcoming volumes of Memory Dump Analysis Anthology (from volume 9) will now include Software Problem Mechanisms and Pattern-Oriented Root Cause Analysis Case Studies chapters in addition to usual pattern chapters.
This is a forthcoming book about architecture, design, and implementation of troubleshooting and debugging tools for software technical support. Preliminary information is:
This training course is a combined and reformatted version of the two previous books Windows Debugging: Practical Foundations and x64 Windows Debugging: Practical Foundations. The new format makes it easy to switch between and compare x86 and x64 versions. The book also has a larger format similar to other training courses from Software Diagnostics Services, punctuation and code highlighting improvements, the output and screenshots from the latest WinDbg 10, and consistently uses WinDbg (X86) for 32-bit examples and WinDbg (X64) for 64-bit examples.
The book contains two separate sets of chapters and corresponding illustrations. They are named Chapter x86.NN and Chapter x64.NN respectively. There is some repetition of content due to the shared nature of x64 and x86 platforms. Both sets of chapters can be read independently. We included x86 chapters because many Windows applications are still 32-bit and executed in 32-bit compatibility mode on x64 Windows systems.
This introductory training course can complement the more advanced course Accelerated Disassembly, Reconstruction and Reversing.
Now, we introduce another metaphor: software traces and logs are “proteins” generated by software code. Such “proteins” are mapped to software functionality:
There are many similarities between protein structural analysis and that of traces and logs. For example, sequence motifs are analogous to Motif trace analysis pattern (that originally came from motives in mathematics), and structural motifs are possible via Characteristic Message Blocks and Activity Regions or any other arrangement of structural patterns:
We introduce canvas for pattern-oriented software diagnostics process to facilitate better diagnostic analysis reports. A piece of paper is divided into 3 columns: Input, Analysis, and Output.
The Input column is divided into two sections: Problem Description Patterns and DA+TA Collection Patterns. Problem Description pattern catalog was introduced earlier in Volume 7 of Memory Dump Analysis Anthology to help with accurate software problem identification. DA+TA Collection pattern catalogs for dump artifacts (DA) and trace artifacts (TA) were also introduced in Volume 7 as Memory Acquisition and Trace Acquisition pattern catalogs.
The Analysis column is divided into two sections: Analysis Patterns and Problem Patterns. The former are about diagnostic analysis techniques, and the latter are about diagnosed software problems. The distinction between them was introduced in Volume 8b. All such patterns can be found in Memory Analysis pattern catalog and Trace and Log Analysis pattern catalog .
The Output column is also divided into two sections: Analysis Report Patterns and Recommended Action Patterns. The corresponding pattern catalogs are under development. The former is about patterns for useful and meaningful diagnostic analysis reports, and the latter is about the good workaround, troubleshooting, and debugging recommendations.
Each cell is subdivided into General and Concrete patterns where the latter are specific product patterns such as a memory access violation in a specific module.
The first version of the canvas template can be downloaded from here.
By software internals, we mean how software actually works instead of how it was intended to work. Intention and actuality may also coincide, but, sometimes, they don’t, especially in the cases of victimware and malware, and their discrepancy, therefore, cause various problems. Learning software internals, especially operating system internals, is a necessary step towards better software construction, effective and efficient troubleshooting and debugging, successful forensics, malware and vulnerability research.
There are different approaches to teaching software internals of operating systems and products. Some authors prefer computer science approach from general software design and architecture principles; some prefer software hands-on troubleshooting approach such as troubleshooting and debugging; some prefer reversing and deconstructive approach. Learners of different backgrounds may find particular approaches difficult to internalize. Here we refer to our own experience learning operating system internals when our main background was in software design, programming, and debugging (before we moved to software support to study software diagnostics).
We think that the ideal way would be to cover all aspects of software construction and post-construction using patterns. However, different phases of software evolution use different pattern paradigms. Software construction phase mainly uses software construction pattern stack of requirements analysis, architecture, design, and implementation patterns which are patterns of software construction problem solving. Software construction pattern paradigm considers patterns as solutions to recurrent problems of building software. Software post-construction phase mainly uses diagnostic patterns and debugging pattern stack [Analysis, Architectural, Design, Implementation and Usage Debugging Patterns, Memory Dump Analysis Anthology, Volumes 6 - 7]. Diagnostic pattern paradigm considers patterns as indicators of software behavior such as signals, symptoms, and signs describing software execution problems. Debugging patterns are similar to software construction patterns in documenting common recurrent problems of fixing software defects. They provide solutions to recurrent debugging problems. Similar can be said about problem workaround patterns and patterns of building software troubleshooting and debugging tools (Debugware patterns [Memory Dump Analysis Anthology, Volumes 2, 4, 6]). There is also a third pattern paradigm: software deconstruction patterns including reversing (ADDR patterns) and structural memory patterns [Memory Dump Analysis Anthology, Volume 5]. This paradigm just shows how things are really are organized before and during software execution including software / hardware interface.
In summary, we have three pattern paradigms corresponding to the domain of construction, post-construction, and deconstruction.
All three pattern paradigms are needed to describe software internals, its architecture, implementation, inner workings, and problem-solving. Instead of devising a new set of patterns to describe software internals we propose a pattern stack of existent patterns. In its rough and simple form it consists of 3 stack slots corresponding to 3 pattern paradigms:
Each pattern paradigm can be expanded into its own substack or a collection (when order is not important) of pattern categories:
Individual software internal features can be described by a concrete pattern stack of patterns from pattern category subcatalogs, for example, software fault processing pattern stack (only a few important conceptual patterns and pattern subcatalogs are listed):
We believe that teaching software internals, for example, operating system internals, using a pattern stack approach would appeal to all types of learners: software architects, software developers, software support engineers, and software security researchers.