Diagnostics Science

All areas of human activity involve the use of diagnostics. Proper diagnostics identifies the right problems to solve. We are now a part of a non-profit organization dedicated to the developing and promoting the application of such diagnostics: systemic and pattern-oriented (pattern-driven and pattern-based).

Book: Accelerated Windows Memory Dump Analysis, Fourth Edition

The fourth edition is available for sale in PDF format from Software Diagnostics Services.

The third edition is available for Safari Books Online subscribers

The second edition is available for SkillSoft Books24x7 subscribers

The full transcript of Software Diagnostics Services Training with 28 step-by-step exercises, notes, source code of specially created modeling applications and more than 100 questions and answers. Covers more than 60 crash dump analysis patterns from x86 and x64 process, kernel, complete (physical), and active memory dumps. The 4th edition was fully reworked to use WinDbg 10 and now covers memory dumps from Windows 10 x64. It also includes optional legacy exercises from the previous editions covering Windows Vista and Windows 7.

  • Title: Accelerated Windows Memory Dump Analysis: Training Course Transcript and WinDbg Practice Exercises with Notes, Fourth Edition
  • Authors: Dmitry Vostokov, Software Diagnostics Services
  • Publisher: OpenTask (May 2016)
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 872 pages
  • ISBN-13: 978-1908043467

Table of Contents

Diagnostics of Things (DoT)

We introduced Narratology of Things as a combination of Software Narratology of Things and Hardware Narratology. Since memory dump analysis may be considered as a part of general trace and log analysis we open a new research direction for Diagnostics of Things (DoT) based on Narratology of Things and Pattern-Oriented Trace and Log Analysis, which also includes software execution artifacts of things and pattern-oriented network trace analysis from IoT.

10 years!

On the 26th of March 2006, 10 years ago, dumpanalysis.org was registered! It was still a long way towards pattern-oriented software diagnostics. The main product of our activity, Memory Dump Analysis Anthology, is now in 10 books.

Software Diagnostics Institute Timeline

Memory Dump Analysis Anthology, Volume 9a

The following direct links can be used to order the book now:

Buy Kindle version

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

Buy Paperback from Book Depository

Also available in PDF format from Software Diagnostics Services

Contains revised, edited, cross-referenced, and thematically organized selected articles from Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org) and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog) about software diagnostics, root cause analysis, debugging, crash and hang dump analysis, software trace and log analysis written in August 2015 - February 2016 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software, technical support and escalation engineers dealing with complex software issues, security researchers, reverse engineers, malware and memory forensics analysts. This volume is fully cross-referenced with volumes 1 – 8 and features:

- 9 new crash dump analysis patterns
- 9 new software log and trace analysis patterns
- 15 Linux core dump analysis pattern variants
- New workaround pattern
- New memory dump analysis case study
- Introduction to pattern-oriented software internals, pattern paradigms, pattern stacks, pattern repertoire
- Introduction to software diagnostics canvas
- Introduction to patterns-based root cause analysis methodology
- Introduction to a protein metaphor for software traces and logs
- Definition of software diagnostics scope
- Introduction to artificial debugger and pseudo-memory dumps
- Definition of tool-centric and pattern-centric software diagnostics, forensics, prognostics

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 9a
  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 179 pages
  • Publisher: OpenTask (February 2016)
  • ISBN-13: 978-1-908043-35-1

Table of Contents

The Scope of Software Diagnostics

Should software diagnostics stand alone as a separate, distinct diagnostics or be a part of some other diagnostics? We considered initially 3 types of diagnostics: medical, technical, and software:

The objects of medical diagnostics are obviously humans and mostly their biological artefacts (we say mostly because linguistic and textual artifacts can also be used for diagnosis). The objects of technical diagnostics are structures and systems made from natural and artificial organic and inorganic engineering materials (Handbook of Technical Diagnostics: Fundamentals and Application to Structures and Systems, ISBN: 978-3642258497, pp. 11 – 16). The objects of software diagnostics are obviously software systems.

We define software diagnostics as “a discipline studying abnormal software structure and behavior in software execution artefacts (such as memory dumps, software and network traces and logs)” (Introduction to Philosophy of Software Diagnostics, Part 1, page 7) or, more generally, to include the context of forensics as “a discipline studying signs of software structure and behavior in software execution artefacts (such as memory dumps, software and network traces and logs)” (Pattern-Oriented Software Forensics: A Foundation of Memory Forensics and Forensics of Things, page 18).

Although, there are many conceptual similarities in general between these diagnostics, there are three features of software that set software diagnostics apart:

  • A software system or its artefact can be copied.
  • The software can be its own model. This follows from the previous feature since we can copy the software execution state and then study the effects of its execution independently. However, it is possible to have different software models for diagnostics as, for example, in Projective Debugging.
  • Software execution artefacts are both symbolic and digital.

There are also humanistic artefacts (A New History of the Humanities: The Search for Principles and Patterns from Antiquity to the Present, ISBN: 978-0199665211) such text (including historical documents), language, literature, and music which are symbolic and digital or can be digitised. Diagnostics takes the form of literary criticism and text reconstruction (philology). Many similarities there gave rise to Software Narratology and Systemic Software Diagnostics:

Teaching Complex Diagnostic Scenarios with Artificial Debugger (ArtDbg) and Pseudo-Memory Dumps

One of the problems in teaching software diagnostics and postmortem debugging is a simulation of complex software problem scenarios. There are plenty of real life memory dumps available but due to security considerations they cannot be shared outside of the organisation. Here we introduce Artificial Debugger project (ArtDbg) that simulates the I/O of the real debugger, for example, WinDbg, GDB, or LLDB.

The real memory dump is analyzed in the real debugger, and all debugger input and output is saved in a log file. This log file is then scanned for any potentially sensitive information, and all such information is eliminated (Security Problem, Memory Dump Analysis Anthology, Volume 1, page 224). It is then converted into a binary pseudo-memory dump format.

ArtDbg is used to open and analyze such pseudo-memory dumps. It allows using the same real debugger commands that were used to generate the log file. Such commands will output the same information that was available from the real debugger. Some real debugger commands that were not used to generate the log file may also be used if possible.

Software Diagnostics Services plans to use the first version of ArtDbg in their new version of Accelerated Windows Memory Dump Analysis training scheduled in March 2016 to teach some very complex enterprise diagnostic scenarios in addition to WinDbg exercises with real memory dumps.

Book: Accelerated Linux Core Dump Analysis

The following direct links can be used to order the book now:

Buy Kindle version

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

Buy Paperback from Book Depository

Also available in PDF format from Software Diagnostics Services.

The full transcript of Software Diagnostics Services training with 13 step-by-step exercises.

  • Title: Accelerated Linux Core Dump Analysis: Training Course Transcript with GDB Practice Exercises
  • Authors: Dmitry Vostokov, Software Diagnostics Services
  • Publisher: OpenTask (December 2015)
  • Language: English
  • PDF: 228 pages
  • ISBN-13: 978-1908043979

Table of Contents

Patterns-Based Root Cause Analysis Methodology

In 2011, we introduced iterative and incremental A.C.P. root cause analysis methodology. Its name is an abbreviation from the three main constituents: artifacts, checklists, and patterns. We get software execution artifacts such as memory dumps and software logs, use checklists to guide us in our problem analysis efforts, recognize patterns of abnormal software structure and behavior, and ultimately find out root cause(s). Recognized patterns may prompt us to revisit checklists for further guidance and request more software execution artifacts. The process is illustrated in the following diagram:

At that time, pattern-oriented software diagnostics was not yet fully developed so the proposed root cause analysis methodology was primarily debugger commands-based for memory analysis (in the form of checklists) and analysis patterns-based for software trace and log analysis where checklists were patterns-based. Pattern sequences (former pattern succession) help in finding root causes.

Pattern catalogues as checklists were later introduced for First Time Software Diagnosis methodology. Patterns were also introduced for memory dump and trace collection.

With the development of pattern-oriented software diagnostics, we realized the centrality of patterns and the division of patterns into general and concrete problem patterns and problem analysis patterns.

This brought the revision of A.C.P methodology where checklists become attributes of artifact collection and pattern catalogues:

However, the causal nation of root cause analysis is not explicitly mentioned in the process. Many problem patterns can be caused differently, for example, Dynamic Memory Corruption patterns can be caused by buffer overwrites and underwrites, invalid API parameters, double memory releases, and even memory manager defects resulting in similar diagnostic indicators seen in memory snapshots and traces.

Therefore, we introduce the notion of Mechanism to describe the possible cause of a diagnostic pattern. Such mechanisms replace pattern sequences as causal analysis tools. Mechanisms can also be organized into catalogues and have checklists. Mechanisms provide software internal links between software construction, post-construction, and deconstruction pattern paradigms.

The resulting iterative and incremental A.P.M. methodology (Artifacts. Patterns. Mechanisms.) is illustrated in the following diagram:

The forthcoming volumes of Memory Dump Analysis Anthology (from volume 9) will now include Software Problem Mechanisms and Pattern-Oriented Root Cause Analysis Case Studies chapters in addition to usual pattern chapters.

DebugWare: The Art and Craft of Writing Troubleshooting and Debugging Tools, Second Edition

This is a forthcoming book about architecture, design, and implementation of troubleshooting and debugging tools for software technical support. Preliminary information is:

  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Paperback: 256 pages
  • ISBN-13: 978-1-908043-95-5
  • Publisher: Opentask (January 2016)
  • Language: English
  • Product Dimensions: 22.86 x 15.24

Front cover:

Draft Table of Contents

Book: Practical Foundations of Windows Debugging, Disassembling, Reversing

Available in PDF format from Software Diagnostics Services

This training course is a combined and reformatted version of the two previous books Windows Debugging: Practical Foundations and x64 Windows Debugging: Practical Foundations. The new format makes it easy to switch between and compare x86 and x64 versions. The book also has a larger format similar to other training courses from Software Diagnostics Services, punctuation and code highlighting improvements, the output and screenshots from the latest WinDbg 10, and consistently uses WinDbg (X86) for 32-bit examples and WinDbg (X64) for 64-bit examples.

The book contains two separate sets of chapters and corresponding illustrations. They are named Chapter x86.NN and Chapter x64.NN respectively. There is some repetition of content due to the shared nature of x64 and x86 platforms. Both sets of chapters can be read independently. We included x86 chapters because many Windows applications are still 32-bit and executed in 32-bit compatibility mode on x64 Windows systems.

This introductory training course can complement the more advanced course Accelerated Disassembly, Reconstruction and Reversing.

Product information:

  • Title: Practical Foundations of Windows Debugging, Disassembling, Reversing: Training Course
  • Authors: Dmitry Vostokov, Software Diagnostics Services
  • Language: English
  • Product Dimensions: 28.0 x 21.6
  • Paperback: 350 pages
  • Publisher: OpenTask (October 2015)
  • ISBN-13: 978-1-908043-94-8

Table of Contents

Software Traces and Logs as Proteins

In the past, we introduced structural and behavioral memory and software trace analysis patterns as DNA of software behavior. This metaphor can be illustrated in the following diagram:

Now, we introduce another metaphor: software traces and logs are “proteins” generated by software code. Such “proteins” are mapped to software functionality:

There are many similarities between protein structural analysis and that of traces and logs. For example, sequence motifs are analogous to Motif trace analysis pattern (that originally came from motives in mathematics), and structural motifs are possible via Characteristic Message Blocks and Activity Regions or any other arrangement of structural patterns:

Software Diagnostics Canvas

We introduce canvas for pattern-oriented software diagnostics process to facilitate better diagnostic analysis reports. A piece of paper is divided into 3 columns: Input, Analysis, and Output.

The Input column is divided into two sections: Problem Description Patterns and DA+TA Collection Patterns. Problem Description pattern catalog was introduced earlier in Volume 7 of Memory Dump Analysis Anthology to help with accurate software problem identification. DA+TA Collection pattern catalogs for dump artifacts (DA) and trace artifacts (TA) were also introduced in Volume 7 as Memory Acquisition and Trace Acquisition pattern catalogs.

The Analysis column is divided into two sections: Analysis Patterns and Problem Patterns. The former are about diagnostic analysis techniques, and the latter are about diagnosed software problems. The distinction between them was introduced in Volume 8b. All such patterns can be found in Memory Analysis pattern catalog and Trace and Log Analysis pattern catalog .

The Output column is also divided into two sections: Analysis Report Patterns and Recommended Action Patterns. The corresponding pattern catalogs are under development. The former is about patterns for useful and meaningful diagnostic analysis reports, and the latter is about the good workaround, troubleshooting, and debugging recommendations.

Each cell is subdivided into General and Concrete patterns where the latter are specific product patterns such as a memory access violation in a specific module.

The first version of the canvas template can be downloaded from here.

Pattern-Oriented Software Internals: Pattern Paradigms and Software Internals Pattern Stack

By software internals, we mean how software actually works instead of how it was intended to work. Intention and actuality may also coincide, but, sometimes, they don’t, especially in the cases of victimware and malware, and their discrepancy, therefore, cause various problems. Learning software internals, especially operating system internals, is a necessary step towards better software construction, effective and efficient troubleshooting and debugging, successful forensics, malware and vulnerability research.

There are different approaches to teaching software internals of operating systems and products. Some authors prefer computer science approach from general software design and architecture principles; some prefer software hands-on troubleshooting approach such as troubleshooting and debugging; some prefer reversing and deconstructive approach. Learners of different backgrounds may find particular approaches difficult to internalize. Here we refer to our own experience learning operating system internals when our main background was in software design, programming, and debugging (before we moved to software support to study software diagnostics).

We think that the ideal way would be to cover all aspects of software construction and post-construction using patterns. However, different phases of software evolution use different pattern paradigms. Software construction phase mainly uses software construction pattern stack of requirements analysis, architecture, design, and implementation patterns which are patterns of software construction problem solving. Software construction pattern paradigm considers patterns as solutions to recurrent problems of building software. Software post-construction phase mainly uses diagnostic patterns and debugging pattern stack [Analysis, Architectural, Design, Implementation and Usage Debugging Patterns, Memory Dump Analysis Anthology, Volumes 6 - 7]. Diagnostic pattern paradigm considers patterns as indicators of software behavior such as signals, symptoms, and signs describing software execution problems. Debugging patterns are similar to software construction patterns in documenting common recurrent problems of fixing software defects. They provide solutions to recurrent debugging problems. Similar can be said about problem workaround patterns and patterns of building software troubleshooting and debugging tools (Debugware patterns [Memory Dump Analysis Anthology, Volumes 2, 4, 6]). There is also a third pattern paradigm: software deconstruction patterns including reversing (ADDR patterns) and structural memory patterns [Memory Dump Analysis Anthology, Volume 5]. This paradigm just shows how things are really are organized before and during software execution including software / hardware interface.

In summary, we have three pattern paradigms corresponding to the domain of construction, post-construction, and deconstruction.

All three pattern paradigms are needed to describe software internals, its architecture, implementation, inner workings, and problem-solving. Instead of devising a new set of patterns to describe software internals we propose a pattern stack of existent patterns. In its rough and simple form it consists of 3 stack slots corresponding to 3 pattern paradigms:

Each pattern paradigm can be expanded into its own substack or a collection (when order is not important) of pattern categories:

Individual software internal features can be described by a concrete pattern stack of patterns from pattern category subcatalogs, for example, software fault processing pattern stack (only a few important conceptual patterns and pattern subcatalogs are listed):

We believe that teaching software internals, for example, operating system internals, using a pattern stack approach would appeal to all types of learners: software architects, software developers, software support engineers, and software security researchers.

World Software Diagnostics Day

We propose August 14 as the World Software Diagnostics Day. The rationale behind this chosen date:

  • 0n14 is 0xE(rror)
  • August is the 7th month based on 0-numbering scheme, and 7 is the number usually identified with user space
  • August is the 8th month based on 1-numbering scheme, and 8 is the number usually identified with kernel space
  • The first entry in Software Diagnostics Library was written on August 14, 2006 (Crash Dump Analysis blog at that time)
  • In 2016 we celebrate 10 years of DumpAnalysis.org which in 2006 laid the foundations of pattern-oriented software diagnostics (now, together with TraceAnalysis.org, Software Diagnostics Institute), and, therefore, the first World Software Diagnostics Day will be celebrated in 2016

Authorized Training Providers

Software Diagnostics Institute, the founder of Pattern-Oriented Software Diagnostics, Forensics, Prognostics, appoints Software Diagnostics Services as the authorized training provider to teach pattern-oriented software diagnostics and the following associated subfields based on pattern-oriented software diagnostics methodology:

  • Pattern-oriented memory dump analysis
  • Pattern-oriented memory forensics
  • Pattern-oriented network forensics
  • Pattern-oriented software trace and log analysis
  • Pattern-oriented network trace analysis
  • Pattern-oriented malware analysis
  • Pattern-oriented vulnerability analysis
  • Pattern-oriented reverse engineering
  • Pattern-oriented software forensics
  • Pattern-oriented software prognostics
  • Pattern-oriented software performance analysis
  • Pattern-oriented debugging process
  • Pattern-oriented software execution artifact collection
  • Pattern-oriented static code analysis
  • Pattern-oriented software internals
  • Pattern-oriented root cause analysis

The list of seminars introducing and explaining pattern-oriented software diagnostics: http://www.patterndiagnostics.com/training-materials

The list of training courses and reference materials is available on Software Diagnostics Services website.

Pattern Repertoire

When developing, debugging, maintaining, and supporting software it is important to have knowledge of general problem analysis patterns of abnormal software structure and behavior. As with any language, such a pattern language comes in passive and active usage variants. A passive pattern user knows about a particular analysis pattern but may have difficulty recognizing it in a specific software execution artifact such as a memory dump or log. A passive user may become an active user after training and experience. “Pattern Repertoire” is such user knowledge (or awareness) of a specific set of patterns from pattern catalogues (see the definition of repertoire). An active pattern repertoire of a Windows software engineer first becomes passive when the engineer starts working with Linux, Mac OS X, Android, iOS, z/OS, or any other platform but then becomes active through experience. In any case, the possession of a pattern repertoire and its size (the pattern language vocabulary) is an important asset. What distinguishes the pattern repertoire from a mere set of pattern names is the knowledge of its syntax (analysis pattern combinations), semantics (the meaning of analysis patterns), and pragmatics (active usage in concrete situations and passive knowledge of usage in general situations). Similar to software construction pattern repertoire which helps with software construction problem analysis, architecture, design, and implementation, software post-construction pattern repertoire helps to perform software post-construction problem diagnostic analysis and debugging more efficiently and effectively. A few words must be said about implicit pattern knowledge. Some patterns may be obvious and used unconsciously. However, their explicit naming helps with communicating analysis decisions, troubleshooting and debugging recommendations, and creating problem-solving case studies.

Here, we would like to show the building of our diagnostic analysis pattern repertoire for memory dumps, software traces, and logs. The following picture illustrates the growth of the number of software diagnostic analysis patterns with each successive volume of Memory Dump Analysis Anthology (excluding malware analysis patterns published in Volume 7, and structural memory patterns published in Volume 5):

The picture shows, for example, that volumes 6 – 8 introduced more analysis patterns than volumes 1 – 5. Volumes 7 and 8 alone introduced 63% of the current trace and log analysis patterns, and 32% of the current crash and hang memory dump analysis patterns.

Advanced Software Diagnostics and Debugging Reference

Tables of Contents and Indexes of WinDbg Commands from all volumes

Click on an individual volume to see its description and table of contents:

You can buy the 9-volume set from Software Diagnostics Services with a discount and also get free access to Software Diagnostics Library.

Praise for the series:

I have been working with reversing, dumps, IAT, unpacking, etc. and I am one of the few at my workplace that like analyzing hangs and crashes. I always knew that I had more to learn. So I continuously look for more info. Many links directed me to dumpanalysis.org. Frankly speaking, its spartan/simple design made me question its seriousness. But after reading some articles, I immediately decided to order "Memory Dump Analysis Anthology". I have only read 100 pages so far. But I am stunned. It is such an amazing book. How the author refines / reconstructs the call stack, and finds useful information in the stack is incredible. I am enormously thankful for the effort that the author has put in making these books. They are very didactic even though the topic is a bit hard. It is a real treasure.

Mattias Hogstrom

Memory Dump Analysis Anthology, Volume 8b

The following direct links can be used to order the book now:

Buy Kindle version

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

Buy Paperback from Book Depository

Also available in PDF format from Software Diagnostics Services

Contains revised, edited, cross-referenced, and thematically organized selected articles from Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org) and Software Diagnostics Library (former Crash Dump Analysis blog, DumpAnalysis.org/blog) about software diagnostics, debugging, crash dump analysis, memory forensics, software trace and log analysis written in December 2014 - July 2015 for software engineers developing and maintaining products on Windows platforms, quality assurance engineers testing software, technical support and escalation engineers dealing with complex software issues, security researchers, reverse engineers, malware and memory forensics analysts. This volume is fully cross-referenced with volumes 1 – 7, 8a, and features:

- 12 new crash dump analysis patterns
- 15 new software log and trace analysis patterns
- New memory dump analysis case study
- Introduction to articoding
- Introduction to special and general trace and log analysis
- Introduction to projective debugging
- Introduction to artifact-malware
- Introduction to concrete and general problem analysis patterns

Product information:

  • Title: Memory Dump Analysis Anthology, Volume 8b
  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Language: English
  • Product Dimensions: 22.86 x 15.24
  • Paperback: 169 pages
  • Publisher: OpenTask (July 2015)
  • ISBN-13: 978-1-908043-54-2

Table of Contents

Pattern! What Pattern?

There is confusion about patterns of diagnostics such as related to crash dump analysis and software trace and log analysis. We are often asked about pattern percentage detection rate or whether it is possible to automate pattern diagnostics. Before asking and answering such questions, it is important to understand what kinds of patterns are meant. Patterns of diagnostics can be subdivided into concrete and general problem patterns, and, also, into concrete and general analysis patterns.

Problem patterns are simply diagnostic patterns, and they can be defined as (fusing Diagnostic Pattern1 and Diagnostic Problem2 definitions):

A common recurrent identifiable set of indicators (signs) together with a set of recommendations and possible solutions to apply in a specific context.

Concrete Problem Patterns are particular sets of indicators, for example, an exception stack trace showing an invalid memory access in the particular function from the particular component/module code loaded and executed on Windows platform.

But such indicators can be generalized from different products and OS platforms giving rise to General Problem Patterns forming a pattern language. Our previous example can be generalized as Exception Stack Trace showing Invalid Pointer and Exception Module. Concrete Problem Patterns are the implementation of the corresponding General Problem Patterns.

Now, it becomes clear why Memory Analysis Pattern Catalog doesn’t have any concrete BSOD bugcheck numbers. Most of such numbers are concrete implementations of Self-Diagnosis pattern.

Then we have Concrete Analysis Patterns as particular techniques to uncover Concrete Problem Patterns. For example, thread raw stack analysis for historical information to reconstruct a stack trace. Again, such techniques vary between OS platform and even between debuggers.

Generalizing again, we have General Analysis Patterns, for example, analyzing Historical Information in Execution Residue to construct Glued Stack Trace.

General Problem Pattern descriptions may already reference General Analysis Patterns, and in some cases both may coincide. For example, Hidden Exception pattern uses Execution Residue pattern as a technique to uncover such exceptions.

Most of Software Trace and Log Analysis Patterns are General Analysis Patterns that were devised and cataloged to structure the analysis of the diverse logs from different products and OS platforms3. For example, a specific data value common to both working and problem logs that helps to find out the missing information from the problem description can be generalized to Inter-Correlation analysis between the problem trace and Master Trace using Shared Point.

This partitioning is depicted in the following diagram:

Software Diagnostics Institute innovation is in devising and cataloging general problem and analysis patterns and providing some concrete analysis implementations on specific OS platforms such as Windows and Mac OS X.

1 Pattern-Oriented Software Forensics: A Foundation of Memory Forensics and Forensics of Things, page 13
2 Ibid., page 14
3 Malware Narratives: An Introduction, page 14

Artifact-Malware and its Primary and Secondary Effects

A few days ago we saw an article in Facebook stream about trolling airline passengers. When they descend to an airport, they read a different city name written in large letters on the roof of some house.

An idea came to us to model this behavior for memory dump analysis: when we analyze crash dumps we usually rely on the output of some commands that redirect or reformat the contents of memory. For example, lmv WinDbg command shows module resource information such as its product name, copyright information, etc. What if that information were deliberately crafted to deceive and disturb software diagnostics and debugging process, and ultimately to explore possible vulnerabilities there? Popular debuggers have their own vulnerabilities1 which may be used not only for anti-debugging purposes. When we say “deliberately crafted” we don’t mean Fake Module2 malware analysis pattern that is about a module that tries to present itself as another legitimate, well-known module. Also, we are not concerned with false positive decoy artifacts3. In our case Artifact-Malware, as we call it (or Arti-Malware for short, not to confuse with anti-malware), intentionally leaves malicious legitimate artifacts in software execution artifacts (such as memory dumps, traces, and logs) deliberately structured to alter execution of static analysis tools such as debuggers, disassemblers, reversing tools, etc. Such artifacts in artifacts may suggest exploring them further as possible culprits of abnormal software behavior thus triggering certain software and human vulnerabilities, and even social engineering attacks (when they suggest calling a phone number).

To model this, we quickly created a small Visual C++ project called TrollingApp and inserted version info resource. Normally WinDbg lmv command would show something like this:

0:000> lmv m TrollingModule
start             end                 module name
00000001`3ff50000 00000001`3ff58000   TrollingModule C 
(private pdb symbols)
    Loaded symbol image file: TrollingModule.exe
    Image path: ...\TrollingApp\x64\Release\TrollingModule.exe
    Image name: TrollingModule.exe
    Timestamp:        Sat Jun 27 10:28:47 2015 (558E6CCF)
    CheckSum:         00000000
    ImageSize:        00008000
    File version:
    Product version:
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     1809.04b0
    CompanyName:      TODO: 
    ProductName:      TODO: 
    InternalName:     TrollingModule.exe
    OriginalFilename: TrollingModule.exe
    FileDescription:  TODO: 
    LegalCopyright:   Copyright © 2015 
        by Software Diagnostics Institute

Since LegalCopyright is the last field shown in the formatted output, we changed it to contain the long string of “\r\n” characters intended to scroll away module information. The string was long as it was allowed by the resource compiler.

 FILEOS 0x40004L
    BLOCK "StringFileInfo"
        BLOCK "180904b0"
            VALUE "CompanyName", "TODO: "
            VALUE "FileDescription", "TODO: "
            VALUE "FileVersion", ""
            VALUE "InternalName", "TrollingModule.exe"
            VALUE "LegalCopyright", "\r\n\r\n\r\n ... "
            VALUE "OriginalFilename", "TrollingModule.exe"
            VALUE "ProductName", "TODO: "
            VALUE "ProductVersion", ""
    BLOCK "VarFileInfo"
        VALUE "Translation", 0x1809, 1200

The program itself is very simple triggering a NULL pointer exception to generate a crash dump (we configured LocalDumps registry key on Windows 7).

int _tmain(int argc, _TCHAR* argv[])
	int *p = 0;

	*p = 0;
	return 0;

So we opened a crash dump and checked the stack trace which suggested checking information about TrollingModule (as Exception Module4 memory analysis pattern):

Loading Dump File [C:\MemoryDumps\TrollingModule.exe.2076.dmp]
User Mini Dump File with Full Memory: Only application data is available

Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x64
Product: WinNt, suite: SingleUserTS Personal
Machine Name:
Debug session time: Sat Jun 27 10:28:58.000 2015 (UTC + 1:00)
System Uptime: 3 days 21:28:51.750
Process Uptime: 0 days 0:00:01.000
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(81c.1604): Access violation - code c0000005 (first/second chance not available)
00000000`7769186a c3 ret

0:000> .symfix c:\mss

0:000> .reload

0:000> kL
Child-SP RetAddr Call Site
00000000`001fe6d8 000007fe`fd741430 ntdll!NtWaitForMultipleObjects+0xa
00000000`001fe6e0 00000000`77541723 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`001fe7e0 00000000`775bb5e5 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`001fe870 00000000`775bb767 kernel32!WerpReportFaultInternal+0x215
00000000`001fe910 00000000`775bb7bf kernel32!WerpReportFault+0x77
00000000`001fe940 00000000`775bb9dc kernel32!BasepReportFault+0x1f
00000000`001fe970 00000000`776d3398 kernel32!UnhandledExceptionFilter+0x1fc
00000000`001fea50 00000000`776585c8 ntdll! ?? ::FNODOBFM::`string'+0x2365
00000000`001fea80 00000000`77669d2d ntdll!_C_specific_handler+0x8c
00000000`001feaf0 00000000`776591cf ntdll!RtlpExecuteHandlerForException+0xd
00000000`001feb20 00000000`77691248 ntdll!RtlDispatchException+0x45a
00000000`001ff200 00000001`3ff51002 ntdll!KiUserExceptionDispatch+0x2e
00000000`001ff908 00000001`3ff51283 TrollingModule!wmain+0x2
00000000`001ff910 00000000`775359ed TrollingModule!__tmainCRTStartup+0x10f
00000000`001ff940 00000000`7766c541 kernel32!BaseThreadInitThunk+0xd
00000000`001ff970 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

But when we executed lmv command we saw the blank screen with some UNICODE symbols at the end:

Not only we triggered the scroll but the artifact buffer somehow caused additional unintended consequences.
We were also surprised by the second order effects. We were curious about what that Unicode string was meant and copied it to Google translate page in IE. It was hanging afterward spiking CPU when we were switching to that tab. We tried to save a crash dump using Task Manager, but it failed with a message about an error in ReadProcessMemory API and, although, the crash dump was saved, it was corrupt. The tab was recovered, and we were not able to reproduce it again. Perhaps, the browser was already in an abnormal state because on the second attempt it behaved better:

Simple Google search shows that such output also appeared in different problems such as related to PDF printing:

In conclusion, we say that the primary effect of arti-malware is abnormal software behavior in static analysis tools. We have the secondary effect when information produced by a static analysis tool triggers abnormal software behavior in another analysis tool.

1 M. Sikorski, A. Honig, Practical Malware Analysis, Debugger Vulnerabilities, page 363
2 D. Vostokov, Memory Dump Analysis Anthology, Volume 7, page 240
3 A. Walters, N. Petroni, Jr., Volatools: Integrating Volatile Memory Forensics into the Digital Investigation Process
4 D. Vostokov, Memory Dump Analysis Anthology, Volume 8a, page 80

Coding and Articoding

The analysis of software traces and logs is largely a qualitative activity. We look for specific problem domain patterns using general analysis patterns (problem domain pattern hierarchy). Some methodological aspects of this software defect research are similar to qualitative research method in social sciences. The latter method uses the so-called coding techniques for data analysis. Software traces and debugger logs from memory dumps are software execution artifacts we previously called DA+TA (Dump Artifact + Trace Artifact, Memory Dump Analysis Anthology, Volume 3, page 330). We propose to use similar “coding” techniques to annotate them with diagnostic indicators, signal and sign mnemonics, and patterns (such as software diagnosis codes). We, therefore, call this software post-construction “coding” as articoding (artecoding), from artifact (artefact) + coding, to distinguish it from software construction coding. Such articoding forms a part of software post-construction problem solving (Introduction to Pattern-Driven Software Problem Solving, page 8). Articodes form a second order software narrative and can be articoded too.

Many software tools were developed for assisting qualitative research coding, and these can be reused for “coding” debugger logs, for example. In addition to those tools, general word and table processing programs can be used as well for some types of artifacts. Here we show MS Word for a WinDbg log example. The debugger log with stack traces from all processes and threads was loaded into MS Word template table with 3 columns. The first column is the log itself, the second column is for diagnostic indicators (such as critical section, CPU consumption, ALPC wait, etc.), and the third column is for pattern language articodes (here we use pattern names from Memory Analysis Pattern Catalog, for traces we can use MS Excel and Trace and Log Analysis Pattern Catalog):

(Click to expand this screenshot)

Formatting and highlighting creativity here is unlimited. Irrelevant parts from the log can be deleted, and the final analysis log can have only relevant annotated tracing information.

Projective Debugging

Modern software systems and products are hard to debug despite their elaborate tracing and logging facilities. Typical logs may include millions of trace messages from hundreds and thousands of components, processes, and threads. The postmortem diagnostic analysis became more structural after the introduction of Trace and Log Analysis Patterns but live debugging requires a lot more efforts. Here we introduce Projective Debugging as a tool for trace-level debugging. Its main idea is to analyze, diagnose and debug the so-called “projected” execution of software as seen from the original software execution traces and logs:

Picture 1. Original software execution is mapped into projected software as seen from traces and logs.

Please notice, that Projective Debugging is different from the so-called Prototype Debugging by creating models after the software product is built (some engineering methodologies prescribe that prototypes should be discarded before building the product):

Picture 2. Prototype software is mapped into the product.

The problems diagnosed and solved in the projected system are fed back into the original system:

Picture 3. Debugged projected software is mapped into the original software.

The implementation of the main idea of Projective Debugging is that: we can take a trace or log, interpret every trace message according to some rules, and translate it into executable code mirroring components and execution entities such as user sessions, processes, and threads. This is a task for the Projective Debugger, and it is illustrated in the following diagram where we borrowed notation from UML:

Picture 4. The logs and traces from the original product execution are translated by Projective Debugger.

For example, the Projective Debugger (ProjectiveDebugger.exe) interprets these very simple messages below, and creates a process PID220.exe (we have only one thread), and then opens a file “data.txt”. After 10 seconds, it closes the file.

Time     PID  TID  Message 
11:00:00 220  330  Open “data.txt”
11:00:10 220  330  Close “data.txt”

In addition to executing code corresponding to messages using the same time deltas as read from the trace, it may scale execution time proportionally, for example, executing 2-day log in a matter of minutes. Such scaling may also uncover additional synchronization bugs.

The trace may be pre-processed, and all necessary objects created before execution or it may be interpreted line by line. For complex traces, the projected source code may be generated for later compilation, linking, and execution. Once the projected code is executed, breakpoints may be set on existing traces, and other types of Debugging Implementation Patterns may be applied. Moreover, we may re-execute the trace several times and even have some limited form of backward execution.

The resulting code model can be inspected by native debuggers for the presence of Memory Analysis Patterns and can even have its own logging instrumentation with traces and logs analyzed by another instance of the Projective Debugger:

Picture 5. Projected Product Execution is inspected by a native debugger and also generates its own set of traces and logs to be projected to another model by another instance of the Projected Debugger.

We created the first version of the Projective Debugger and successfully applied to a small trace involving synchronization primitives across several threads. The Projective Debugger was able to translate it into an executable model with the same number of threads and the same synchronization primitives and logic. The resulting process was hung as expected and we attached a native debugger (WinDbg for Windows) and found a deadlock.

Since traces are analyzed from platform-independent Software Narratology perspective it is possible to get a trace from one operating system, and then, after applying a set of rules, re-interpret it into an executable model on another operating system. We created the similar multithreading test program on Mac OS X that was hung and reinterpreted its trace into an executable model under Windows:

Picture 6. The traces and logs from the original product execution on Mac OS X are projected by a Windows version of Projective Debugger into an executable model for a Windows platform.

Since resultant executable models can also have corresponding logging instrumentation mirroring original tracing, any problems found in executable models can be fixed iteratively and, once the problem disappears, the resulting fix or configuration may be incorporated into the original product or system.

If tracing involves kernel space and mode, a specialized projected executable can be created to model the operating system and driver level.

The more tracing data you have, more real your projected execution becomes. However, we want to have enough tracing statements but not to complicate the projected model. Then, ideally, we should trace only relevant behavior, for example, corresponding to use cases and architecture.

Projective Debugging may also improve your system or product maintainability by highlighting whether you need more tracing statements and whether you need more accurate and complete tracing statements.

Encyclopedia of Crash Dump Analysis Patterns

The following direct links can be used to order the book now:

Buy Paperback from Amazon

Buy Paperback from Barnes & Noble

Buy Paperback from Book Depository

Available for SkillSoft Books24x7 subscribers

Also available in PDF format from Software Diagnostics Services

This reference reprints with corrections, additional comments, and classification 326 alphabetically arranged and cross-referenced memory analysis patterns originally published in Memory Dump Analysis Anthology volumes 1 – 8. This pattern catalog is a part of pattern-oriented software diagnostics, forensics, and prognostics developed by Software Diagnostics Institute (DumpAnalysis.org + TraceAnalysis.org). Most of the patterns are illustrated with examples for WinDbg from Debugging Tools for Windows with a few examples from Mac OS X for GDB.

Product information:

  • Title: Encyclopedia of Crash Dump Analysis Patterns: Detecting Abnormal Software Structure and Behavior in Computer Memory
  • Authors: Dmitry Vostokov, Software Diagnostics Institute
  • Language: English
  • Product Dimensions: 24.6 x 18.9
  • Paperback: 1,058 pages
  • Publisher: OpenTask (March 2015)
  • ISBN-13: 978-1-906717-21-6

Table of Contents

Memory Dump and Live Memory Visualization and Picture Extraction

This is a reprint of several articles with permission from Memory Dump Analysis Anthology, Volume 1.

As the first step towards Memory Dump Tomography (Memory Dump Analysis Anthology, Volume 1, page 522) we created a small program that interprets a memory dump as a picture. We can visualize crash dumps with it. The tool is available for free download.

Simply run it from the command prompt and specify full paths to a dump file and an output BMP file. The memory dump file will be converted by default into true color, 32 bits-per-pixel bitmap. We can specify other values: 8, 16 and 24.

Dump2Picture version 1.0
Written by Dmitry Vostokov, 2007
Usage: Dump2Picture dumpfile bmpfile [8|16|24|32]

For example:

C:\Dump2Picture>Dump2Picture.exe MEMORY.DMP MEMORY.BMP 8
Dump2Picture version 1.0
Written by Dmitry Vostokov, 2007
1 file(s) copied.

Below are some screenshots of bitmap files created by the tool. We can think about them as visualized kernel or user address spaces.

Vista kernel memory dump (8 bits-per-pixel):

Vista kernel memory dump (16 bits-per-pixel):

Vista kernel memory dump (24 bits-per-pixel):

Vista kernel memory dump (32 bits-per-pixel):

Notepad process user memory dump (8 bits-per-pixel):

Notepad process user memory dump (16 bits-per-pixel):

Notepad process user memory dump (24 bits-per-pixel):

Notepad process user memory dump (32 bits-per-pixel):

Mspaint process user memory dump (32 bits-per-pixel):

Mspaint process user memory dump after loading "Toco Toucan.jpg" from Vista Sample Pictures folder (32 bits-per-pixel):

Dump2Picture can be used to explore memory leaks visually. We created the following small program in Visual C++ that leaks 64Kb every second:

#include "stdafx.h"
#include <windows.h>
int _tmain(int argc, _TCHAR* argv[])
  while (true)
    printf("%x\n", (UINT_PTR)malloc(0xFFFF));
  return 0;

Then we sampled 3 dumps at 7Mb, 17Mb and 32Mb process virtual memory size and converted them as 16 bits-per-pixel bitmaps. On the pictures below we can see that the middle black memory area grows significantly. Obviously malloc function allocates zeroed memory and therefore we see black color.

7Mb process memory dump:

17Mb process memory dump:

32Mb process memory dump:

If we zoom in the black area we would see the following pattern:

Colored lines inside are heap control structures that are created for every allocated block of memory. If this is correct then allocating smaller memory blocks would create a hatched pattern. And this is true indeed. The following program leaks 256 byte memory blocks:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
  while (true)
    printf("%x\n", (UINT_PTR)malloc(0xFF));
  return 0;

The corresponding process memory picture and zoomed heap area are the following:

Making allocations 4 times smaller makes heap area to look hatched, and zoomed picture is more densely packed by heap control structures:

#include "stdafx.h"
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
  while (true)
    printf("%x\n", (UINT_PTR)malloc(0xFF/4));
  return 0;

Here is another example. One service was increasing its memory constantly. The crash dump picture shows huge hatched dark region in the middle:

and if we zoom in this region:

Because the pattern and allocation size look uniform it could be the true heap memory leak for some operation that allocates constant size buffers. After opening the dump and looking at heap segments that had grown the most we see the same allocation size indeed:

0:000> !.\w2kfre\ntsdexts.heap -h 5
HEAPEXT: Unable to get address of NTDLL!NtGlobalFlag.
Index Address Name Debugging options enabled
1: 00140000
2: 00240000
3: 00310000
4: 00330000
5: 00370000
Segment at 00370000 to 00380000 (00010000 bytes committed)
Segment at 01680000 to 01780000 (00100000 bytes committed)
Segment at 019C0000 to 01BC0000 (00200000 bytes committed)
Segment at 01BC0000 to 01FC0000 (00400000 bytes committed)
Segment at 01FC0000 to 027C0000 (00800000 bytes committed)
Segment at 027C0000 to 037C0000 (01000000 bytes committed)
Segment at 037C0000 to 057C0000 (02000000 bytes committed)

Segment at 057C0000 to 097C0000 (00155000 bytes committed)
057B96E0: 01048 . 01048 [07] - busy (1030), tail fill
057BA728: 01048 . 01048 [07] - busy (1030), tail fill
057BB770: 01048 . 01048 [07] - busy (1030), tail fill
057BC7B8: 01048 . 01048 [07] - busy (1030), tail fill
057BD800: 01048 . 01048 [07] - busy (1030), tail fill
057BE848: 01048 . 01048 [07] - busy (1030), tail fill
057BF890: 01048 . 00770 [14] free fill
Heap entries for Segment07 in Heap 370000
057C0040: 00040 . 01048 [07] - busy (1030), tail fill
057C1088: 01048 . 01048 [07] - busy (1030), tail fill
057C20D0: 01048 . 01048 [07] - busy (1030), tail fill
057C3118: 01048 . 01048 [07] - busy (1030), tail fill
057C4160: 01048 . 01048 [07] - busy (1030), tail fill
057C51A8: 01048 . 01048 [07] - busy (1030), tail fill

An alternative to converting memory dumps to picture files is to save a memory range to a binary file and then convert it to a BMP file. Thus we can view the particular DLL or driver mapped into address space, heap or pool region, including live memory:

To save a memory range to a file use WinDbg .writemem command:

.writemem d2p-range.bin 00800000 0085e000


.writemem d2p-range.bin 00400000 L20000

We wrote a WinDbg script that saves a specified memory range and then calls a shell script that automatically converts the saved binary file to a BMP file and then runs a picture viewer registered for .bmp extension.

The WinDbg script code (mempicture.txt):

.writemem d2p-range.bin ${$arg1} ${$arg2}
.if (${/d:$arg3})
  .shell -i- mempicture.cmd d2p-range ${$arg3}
  .shell -i- mempicture.cmd d2p-range

The shell script (mempicture.cmd):

dump2picture %1.bin %1.bmp %2

Because WinDbg installation folder is assumed to be the default directory for both scripts and Dump2Picture.exe they should be copied to the same folder where windbg.exe is located. On our system it is

C:\Program Files\Debugging Tools for Windows

Both scripts are now included in Dump2Picture package available for free download.

To call the script from WinDbg use the following command:

$$>a< mempicture.txt Range [bits-per-pixel]

where Range can be in Address1 Address2 or Address Lxxx format, bits-per-pixel can be 8, 16, 24 or 32. By default it is 32.

For example, we loaded a complete Windows x64 memory dump and visualized HAL (hardware abstraction layer) module:

kd> lm
start end module name
fffff800`00800000 fffff800`0085e000 hal
fffff800`01000000 fffff800`0147b000 nt
fffff97f`ff000000 fffff97f`ff45d000 win32k

kd> $$>a< mempicture.txt fffff800`00800000 fffff800`0085e000
Writing 5e001 bytes...

C:\Program Files\Debugging Tools for Windows>dump2picture d2p-range.bin d2p-range.bmp

Dump2Picture version 1.1
Written by Dmitry Vostokov, 2007

1 file(s) copied.

C:\Program Files\Debugging Tools for Windows>d2p-range.bmp
<.shell waiting 10 second(s) for process>
.shell: Process exited

and Windows Picture and Fax Viewer application was launched and displayed the following picture:

Syndicate content