Tracing

While an application generated by Code RealTime is running, you can turn on tracing of messages that are exchanged between capsule instances at run-time. Such traces can be visualized graphically as sequence diagrams and can be very useful when troubleshooting certain kinds of bugs in an application such as race conditions, dead-locks, performance problems, unexpected error situations etc. Here are some examples of questions that can be answered by analyzing captured traces:

Which messages were received by a capsule that caused it to enter a certain state (e.g. an unexpected error state)?
Why did a capsule instance receive a certain message (e.g. an unexpected error message)?
Which capsule instances in the application exchange messages most frequently? (For performance reasons you typically want such capsule instances to run in the same thread since intra-thread communication is much faster than inter-thread communication.)
Are there capsule instances in the application which seem to be unused (and hence could be candidates for removal)?
Is the handling of messages fast enough, or are there some transitions that are too slow?

To learn how to turn on tracing in your application, see Controlling Tracing.

A trace captured by Code RealTime is saved in a file with the default name .trace.art-trace, placed in the same folder where the running executable is located. You can use this file in at least three different ways:

Open the file in Code RealTime as a sequence diagram for analysis.
Open the file in Code RealTime as a text file for analysis.
Process the file programmatically, for example to analyze it algorithmically or to translate it into another trace file format.

Trace File Contents

The main focus for tracing is the message communication between capsule instances, so a captured trace file mostly consists of such messages, and the capsule instances that send and receive those message. But by using an API provided by the TargetRTS your application can write additional information in the form of custom notes into the trace.

Both messages and notes may contain timestamps that tell when they occurred. Which timestamps that are captured, if any, is controlled by a trace configuration file. Note that not all configurations of the TargetRTS allow timestamps to be captured. The configuration setting RTS_TRACE must be enabled and a C++ language standard of at least C++ 11 must be supported by your compiler.

When interpreting a trace it's important to consider that the exchange of a message between two capsule instances involves three specific actions that take place in this precise order:

The message is sent. This happens when the sending capsule instance places the message into the message queue of the receiver's controller. The point in time when this happens is referred to below as time1_send. Note that when synchronous communication is used this action does not take place since the message then is not placed in a message queue but directly delivered to the receiver.
The message is received. This happens when the message is delivered to the receiving capsule instance by its controller. The point in time when this happens is referred to below as time2_receive.
The message has been handled. This happens when the receiver capsule instance has handled the message, i.e. after the transition in the receiver's state machine that was triggered by the message (if any) has run to completion. The point in time when this happens is referred to below as time3_handle.

A message only appears in the trace when all these actions have taken place. This means that messages are ordered in the trace according to when they have been handled, not according to when they were sent or received.

View Traces as Sequence Diagrams

To view an .art-trace file as a sequence diagram right-click on it in the Explorer view, or in the text editor, and perform the command Open Sequence Diagram. The trace will then be visualized graphically as a sequence diagram. On the top there are lifelines which represent the capsule instances in the application. Each lifeline has a vertical line, and between these lines there are arrows that represent messages that were exhanged between capsule instances.

The order of the message lines tell you in which order messages were handled or received by capsule instances in the application. If the trace configuration has enabled capturing of the time2_receive timestamp (it's enabled by default) then messages are ordered chronologically, from top to bottom, according to that timestamp. This means that the diagram then shows the order in which messages were received by capsule instances in the application, which often is the most natural way of interpreting a sequence diagram. However, if the time2_receive timestamp is not available, message lines appear instead in the same order as in the .art-trace file. This is the order in which messages were handled by capsule instances in the application. To know exactly when messages were handled the trace configuration should enable capturing of the time3_handle timestamp.

Note

When messages are sorted according to when they were handled, rather than received, the order of nested synchronous messages in the sequence diagram appear in a "bottom-up" order which can feel unintuitive at first glance. That is, if a capsule invokes a message m1 on a capsule which in turn invokes a message m2 on another capsule, the message m2 will come before m1 in the sequence diagram. While m1 is received before m2, the opposite is true for handling of the message; m2 is handled before m1. If your application uses synchronous communication it's therefore recommended to use a trace configuration which enables capturing of the time2_receive timestamp. Remember that it's not only messages that you invoke which are handled synchronously - also the initialize message for a nested part is handled synchronously. That is, initialization of parts will appear "bottom-up" if time2_receive timestamp is not available, and "top-down" otherwise.

Note

Even if a sequence diagram also shows a strict ordering of the sending of messages (i.e. the start of a message line), it's important to remember that the messages only are ordered according to the times when they were received or handled. In the diagram message lines are horizontal, but that should not be interpreted as that nothing happens between a message is sent, received and eventually handled. It's fully possible that a received message in fact was sent before the message that precedes it in the trace was both received and sent.

The image below shows a sequence diagram for an .art-trace file captured from the TcpRangeCounter sample application.

We can see lifelines that represent the counter and server capsule instances and we can see that the server sent a message for the setMax event with 12 as data to the counter. It then sent also the resumeCounting event (without any data).

The lifeline for a capsule instance has the name of the part that contains the instance. If the part has non-single multiplicity, the lifeline shows the index of the capsule instance within square brackets ([0] for the first instance). If you hover over a lifeline a tooltip appears with more information, such as the dynamic type of the capsule instance and its structured path in the application's composite structure. For more information about the capsule instance, double click on the lifeline header to navigate to the .art-trace file.

Colors are used to distinguish different types of lifelines:

Green for the lifeline that represents the top capsule instance which is created at application start-up. It has the name "application".
Dark blue lifelines represent capsule instances that are created after the top capsule instance has been created, when the application runs.
Light blue lifelines represent certain TargetRTS components that may interact with capsule instances.
Purple for the lifeline that represents an unknown sender of messages (see Messages without Sender).

There are two TargetRTS components that can interact with capsule instances. <timer> is the component of the TargetRTS that implements timers and you will see when a capsule instance receives a timeout message for a timer that has timed out. Inside the TargetRTS this capsule instance has the name specials and is typed by RTTimerActor. The <system> lifeline is used for showing when capsule instances are created and initialized (by sending the initialize message).

The sequence diagram may contain note boxes. They are for example used for showing if tracing was paused for some time, which then means the sequence diagram doesn't show messages that may have been exchanged during that time.

If you hover over the message line label you can see the port of the receiver capsule instance on which the message was received. For more information about the message, double-click on the message line to navigate to the message in the .art-trace file. If the trace contains timestamps (time2_receive and time3_handle) the message line tooltip also shows the difference between these two timestamps. This is the time it took for the receiver to handle the received message.

Synchronous Communication

Messages for synchronous communication have a special visualization in the sequence diagram. The invoke message connects to a rectangle on the receiver lifeline which shows that the sender (i.e. caller) is blocked while the receiver (i.e. callee) handles the message. The reply message is shown by a dashed line. If the reply is explicit, the reply message and its data is shown.

In the example below the first invoke message has an implicit reply, while the second has an explicit reply:

Messages without Sender

The sender of a message is computed at the time of receiving the message based on checking which port the receiver port is connected to. This means that sometimes the sender is not known, or that only limited information about the sender is available:

Messages sent by the special TargetRTS instances <system> or <timer> have no sender port and sometimes also no receiver port.
Messages that are injected into the application by using the Art Debugger have no sender.
Messages that are programmatically injected into the application (for example by the TCP server library) have no sender.
If the sender of a message disconnects the sender port from the receiver port immediately after a message has been sent on the port (and before the controller dispatches the message), it will not be possible for the receiver to know about the sender. This is, however, a quite unusual situation.

A special external lifeline is used for representing the unknown sender of a message.

View Traces as Text

While a sequence diagram can be useful for showing an overview of the run-time interaction between capsule instances, it does not show all information that was captured in the trace. To see all information you can open an .art-trace file in the text editor.

The textual representation of a trace uses a domain specific tracing language, where each line tell about something that happens in the application at run-time:

Tracing starts. Comments are then printed at the beginning of the .art-trace file to show the trace configuration that will be used for the trace. Other information is also printed, for example the current time (according to the chosen timestamp precision and the name of the running application).

// "trace": {
//     "application": "Top.EXE"
//     , "start_time": "2025-11-13 16:32:40.435320100"
// }

A capsule instance receives a message. The first time a capsule instance receives a message it is declared in the .art-trace file using the instance keyword. The sending capsule instance is also declared at this time, if it was not already declared previously. The message itself appears on a line further below these instance declarations.
Something else of interest happens. A note with some informative text is then printed to the .art-trace file. For example, when tracing is paused a note "Tracing is paused..." is printed.

In addition to this domain specific tracing language, extra information may also appear at the end of lines as JSON. What information that is included is affected by the trace configuration that is used.

Instance

A capsule instance appears in an .art-trace file the first time it sends or receives a message. If tracing is turned on when a capsule instance is created, the first message it receives is the initialize message from <system>. However, if tracing is turned on at a later point in time, after the capsule instance was created, it will not appear in the trace until it either receives or sends a message. Here is an example of how a capsule instance appears in a trace:

Memory address The memory address uniquely identifies the capsule instance throughout its lifetime.
Structure path This shows the location of the capsule instance in the application's composite structure. The first name specifies a part in the top capsule, and subsequent names specify nested parts. For parts with non-single multiplicity the index within that part shows where the capsule instance is located. The structure path of a capsule instance is often the same throughout its lifetime, but with the use of plugin parts it's possible to move a capsule instance from one part to another at run-time, and in that case the structure path will change.
Dynamic type This is the actual run-time type of the capsule instance. It's either the same as the capsule that types the part where it is contained, or a capsule that inherits (directly or indirectly) from it. The dynamic type of a capsule instance remains the same throughout its lifetime.

In the example above we see a capsule instance at memory address 0x26a59291f60 with the dynamic type Pinger. The instance is located at index 1 of the part ping (which hence has non-single multiplicity). Its container capsule instance is located in the part ps of the top capsule.

You can Ctrl-click on underlined part names in the structure path to navigate to a part in the Art file where it's defined. In the same way you can navigate to the capsule that is the dynamic type of the capsule instance.

At the end of an instance line there can be a JSON object with additional information, for example the name of the thread that runs the capsule instance:

{ "thread_name": "MyCustomThread" }

Message

A message appears in an .art-trace file when a message has been handled by a capsule instance. This means that a message first has to be sent by the sender, received by the receiver, and finally also handled by the receiver before it appears in the trace. Here is an example of how messages appear in a trace:

Sender The sender is identified by its memory address, but the name of the part that contains it is also printed to make the trace more readable.
Sender port This is the port of the sending capsule on which the message was sent. If that port has non-single multiplicity the port index is specified within square brackets.
Receiver The receiver is identified by its memory address, but the name of the part that contains it is also printed to make the trace more readable.
Receiver port This is the port of the receiver capsule on which the message was received. If that port has non-single multiplicity the port index is specified within square brackets.
Event This is the protocol event which the message is an instance of. The receiver port is typed by the protocol that defines this event (or a protocol from which it inherits).
Data This is the data that is carried by the message. By default the message data is represented by its ASCII encoding which is the data type followed by the value. If the message carries no data an empty string is printed (i.e. event()). If the event data type cannot be encoded (for example because it has no encode function or the configuration macro OBJECT_ENCODE is unset) then ? is printed.

In the example above we see two messages. The first one was sent from a capsule instance in the ping part and received by a capsule instance in the pong part, and the other one was sent in the opposite direction. We can see that the first message was sent on the port pingPort at index 0, while the second message was sent on the port pongPort (which has single multiplicity). The first message was received on the port pongPort while the second message was received on the port pingPort at index 1. The first message was of the event ping and carried an int value 0, while the second message was of the event pong and carried no data.

You can Ctrl-click on underlined addresses to navigate within the trace file to the instance declaration of the sender or receiver capsule instances. There you can see more details about the instances than what is shown in the message itself. You can also Ctrl-click on underlined port names to navigate to where the sender or receiver port is defined in an Art file. In the same way you can navigate to the definition of the protocol event.

If the sender of a message is not known (see Message without Sender) it's represented in the trace with a fake instance external at address 0x0. This makes it easy to distinguish these "external" messages from others.

At the end of a message line there can be a JSON object with additional information, for example timestamps telling when the message was received and handled, and if the message was synchronously invoked or replied rather than sent:

{ "invoke": "0xa7dcdff610", "reply": "0xa7dcdff610", "time2_receive": 1020128300, "time3_handle": 1020132300 }

The address specified for "invoke" and "reply" is the reply buffer address which pairs an invoke message with its corresponding reply message. For invokes with an implicit reply, both these appear in the same message (as in the example above).

Note

A note shows some text at a particular location in a trace, to tell that something interesting has happened. A timestamp is also included at the end of a note in a JSON object. For example:

note "An internal error occurred!" { "time": 3067253000 }

Your application can programmatically write custom notes into a trace.

When tracing is turned off, but the application keeps running, a note with the text "Tracing is paused..." is printed to the trace. It serves as a delimiter in the trace file in case tracing later is turned on again. Messages that were received while the tracing was paused will obviously not be included in the trace, and the note shows the place in the trace where this could have happened.

Controlling Tracing

By default the tracing capability is enabled in the TargetRTS. You can disable it through the RTS_TRACE configuration setting if you don't plan to capture any traces. This will make your executable slightly smaller.

When your application starts to run, tracing is initially turned off. To start capturing a trace you need to turn it on. There are several ways to do this as described in the sections below.

While tracing is turned on, trace events will be captured and written to the trace file. However, for performance reasons the file is not immediately flushed. The trace file will be flushed when the application terminates, but quite often you may want to do it before that. Refer to the sections below for how to force the trace file to be flushed.

Captured trace files will be placed in the same folder as where the running executable is located. The default name of a trace file is .trace.art-trace but you can change this using a trace configuration. Note that by default a newly generated trace file will overwrite an existing trace file in the same folder. This too can be configured in the trace configuration.

Capturing all messages that are exchanged in an application for the whole time it runs will usually lead to very large traces. Such big traces are cumbersome to work with, and sequence diagrams may not be possible to render. And even if they could be rendered, they may not be easy to use due to the size. It's better to only turn on tracing when the application reaches a point when something interesting is about to happen. For example, if you are troubleshooting a bug that occurs when a certain transition runs in a capsule's state machine, you can debug the application with the Art Debugger and turn on tracing when the source state of that transition is reached. Then you can manually send the event which will trigger the interesting transition, and then turn tracing off. Thereby the trace file will only show what happened when that transition executed, which makes it small and manageable.

Or you could debug your application with a C++ debugger and put a breakpoint that gets hit before something interesting happens. Then you can turn on tracing and let the application run until it hits another breakpoint, and then turn off tracing again. The trace then shows you which messages were exchanged in the application between those two breakpoints were hit.

From the Command Line

If you want to trace what happens in the application from the very beginning you can turn tracing on from the command-line when you launch the application. Launch your executable with the -URTS_DEBUG argument as the very first argument. For example:

./Top.EXE -URTS_DEBUG="trace on;quit"

This command will turn on tracing at application start-up and then quit the RTS Debugger to let the application run without debugger at full speed. In this case the only way to turn off tracing before the application terminates is to do it programmatically using TargetRTS APIs.

You can also launch the application like this:

./Top.EXE -URTS_DEBUG="trace on;continue" -obslisten=12345

The main difference here is that you then later can attach the Art Debugger on the specified port, and turn off tracing from there.

Other RTS Debugger commands can of course also be used. For example, to trace the first 5 messages that are sent in the application:

./Top.EXE -URTS_DEBUG="trace on;go 5"

Read more about the -URTS_DEBUG argument here.

From the command-line you can also specify a trace configuration by means of the -traceConfig command-line argument when you start your application.

From the RTS Debugger

The RTS Debugger provides a command trace for turning tracing on or off:

RTS debug: ->trace on
  Tracing is on

If you are unsure whether tracing is currently on or off, you can run the command without argument.

A useful side-effect of the trace command is that it will flush the current trace file, if any.

From the Art Debugger

To turn tracing on or off from the Art Debugger invoke the Send to App command from the command palette. The trace command from the RTS Debugger can be accessed from the popup that appears.

From the C++ Debugger

Turning tracing on or off when you debug your application in a C++ debugger is often very useful. You do it by using the API provided by the TargetRTS class RTTracer:

RTTracer::setEnabled(bool) Turns tracing on or off.
RTTracer::isEnabled() Returns if tracing is currently turned on or off.
RTTracer::flushTrace() Forces the current trace file, if any, to be flushed, so it contains all trace events received so far.

Exactly how to call these functions depends on which C++ debugger you are using. For example, in the Visual Studio C++ Debugger you do it from the Immediate Window:

A typical workflow may look like this:

Build the application and the TargetRTS with debug symbols as described here.
Place a breakpoint at a place where you want to start tracing.
Start the application from the C++ debugger and let it run until the breakpoint is hit.
Call RTTracer::setEnabled(true) from the C++ debugger to turn tracing on.
Place a breakpoint at a place where you want to stop tracing.
Resume execution of the application and let it run until the breakpoint is hit.
Call RTTracer::setEnabled(false) from the C++ debugger to turn tracing off. Or call RTTracer::flushTrace() if you want to keep tracing but want to check what the trace contains so far.

The RTTracer class also provides a few other functions for controlling tracing. While these also can be used from a C++ debugger, it's more common to use them programmatically from your application code.

Programmatically

You can turn tracing on or off from your application code by using the API provided by the TargetRTS class RTTracer. See above for which functions that are available to use.

One scenario where it can be useful to enable tracing programmatically is if your C++ debugger does not support calling functions, or if you need to trace a certain scenario in an application that doesn't have debug symbols. It can also be useful to build tracing functionality into your application and for example use it to trace unexpected error scenarios that may occur. The collected trace files can then help you troubleshoot problems that have occurred when the application ran in the production environment.

In addition to the RTTracer functions mentioned above, the following are also available and can sometimes be useful to call by your application:

RTTracer::configure(const char*) Configure tracing by means of a trace configuration.
RTTracer::isConfigured() Returns true if a custom trace configuration has been used for configuring tracing. Otherwise false is returned (meaning that all trace configuration properties will have their default values).
RTTracer::note(const char*) Write a custom note to the trace.

Custom Notes

A note is automatically written to the trace whenever tracing is turned off, as explained here. However, it can also be very useful to write custom notes to the trace using the RTTracer::note(const char*) function. For example, you can print a note

when a run-time error occurs (e.g. "Out of memory" or "Internal error")
when an assertion fails (e.g. "assert(is_ok) failed")
to print information that helps in troubleshooting a problem (e.g. "Value of my_var was 5")
when the application has reached a certain interesting point (e.g. "All initializations were successfully completed")
...

In addition to making traces more readable, notes also get a timestamp which makes it possible to know exactly when they were written. Thereby you can relate them to messages in a trace, which would be harder if you instead would have written the note to a separate log file.

Example

You can find a sample application that writes custom notes to a trace here.

Trace Configuration

You can configure what information to capture in a trace by providing a trace configuration. This is a JSON file with the name trace.config, or file extension .trace.config. You can provide the trace configuration when starting the application by means of the command-line argument -traceConfig=<trace config JSON file>. The path to the trace configuration file can either be absolute or relative to the location of the executable. In some command shells you may need to enclose the path in double quotes.

You can also provide a trace configuration programmatically as explained here.

To create a trace configuration file use the command File - New File - New Trace Configuration. In the popup that appears specify the name of the trace configuration or keep the suggested default name.

The table below lists all trace configuration properties that can be set in a trace configuration file, and their default values in case they are not set.

Property	Type	Default Value
timestamp.mode	String	"relative"
timestamp.precision	String	"nano"
timestamp.time1_send	Boolean	false
timestamp.time2_receive	Boolean	true
timestamp.time3_handle	Boolean	false
trace_file.name	String	".trace"
trace_file.overwrite	Boolean	true

Timestamp Mode

Property: mode
Type: String
Valid values: "absolute", "relative" (default)

Timestamps can either be absolute or relative. An absolute timestamp specifies the time that has passed since January 1st 1970 (the Unix epoch), while a relative timestamp specifies the time that has passed since the application was started.

Here is an example that specifies to capture absolute timestamps:

"timestamps": {
    "mode": "absolute"
}

Timestamp Precision

Property: precision
Type: String
Valid values: "milli", "micro", "nano" (default)

Timestamps are by default given with nano second precision. This applies to both absolute and relative timestamps. For some applications nano second precision may be too high and lead to unnecessary big timestamp numbers, and then you can choose microseconds, or even milliseconds.

Send Time

Property: time1_send
Type: Boolean
Valid values: true, false (default)

The first timestamp that can be captured for a message exchange, is when the sender puts the message in the message queue of the receiver's controller. This timestamp is referred to as time1_send. Note that when synchronous communication is used the message is not placed in a message queue but is directly delivered to the receiver. This timestamp is therefore only relevant for asynchronous communication.

Note

This timestamp is currently not supported and the configuration property time1_send is ignored.

Receive Time

Property: time2_send
Type: Boolean
Valid values: true (default), false

The second timestamp that can be captured for a message exchange, is when the message is delivered to the receiver by its controller. Unless the message will be discarded by the receiver, this timestamp tells when the transition that is triggered by the received message is just about to start executing. This timestamp is referred to as time2_receive and appears in JSON at the end of a message in the trace file.

Handle Time

Property: time3_handle
Type: Boolean
Valid values: true, false (default)

The third timestamp that can be captured for a message exchange, is when the message has been handled by the receiver, that is when the triggered transition has ran to completion and control is returned back to the TargetRTS. This timestamp is referred to as time3_handle and appears in JSON at the end of a message in the trace file.

The difference between time3_handle and time2_receive tell how much time it took for the receiver to process a received message. For performance reasons this time should not be too big, because while a capsule instance is processing a dispatched message, no other capsule instances run by the same controller can execute.

Here is an example of a message for which both the timestamps time2_receive and time3_handle have been captured (with nano second precision). The difference between the timestamps tell that it took 3900 ns for the receiver capsule instance to process this message.

0x0 external -> 0x16e4693b390 guesser.numberP : sendAnswer(bool false) { "time2_receive": 1033355200, "time3_handle": 1033359100 }

Trace File Name

Property: name
Type: String
Valid values: Any string that specifies a valid file name. The default is ".trace".

This property specifies the base name of the trace file (the file extension is always .art-trace). You can use the following variables in the string to obtain a file name that is unique:

${datetime} This variable expands to a human-readable string that contains the current date and time, on the format YYYY-MM-DD_HH-MM-SS. It's intended to be used when you invoke the application manually, and therefore isn't more granular than seconds.
${microseconds} This variable expands to the current time, expressed as the number of microseconds since the Unix epoch (January 1st 1970). It's intended to be used when the application is invoked many times programmatically, and where a precision of seconds is not enough.

Here is an example that specifies a trace file name using both these variables (although normally only one of them is used):

"trace_file": {
    "name": "${datetime}_tracefile_${microseconds}"
}

Overwrite

Property: overwrite
Type: Boolean
Valid values: true (default), false

By default a new trace file that is generated will overwrite an existing trace file with the same name in a folder. To prevent this you can set this property to false. If an attempt is made to generate a new trace with a name that already exists in the folder, and if this property is set to false, then the application will terminate with an error message.

Process a Trace Programmatically

The parser for .art-trace files is open source and available on GitHub as well as an npm package. This makes it easy to write your own scripts (JavaScript or TypeScript) which can read an .art-trace file and process it in a custom way.

A few samples are available in the GitHub repository that you can use as a starting point.

jsSequenceDiagram

This sample script translates an .art-trace file into the textual format that the js-sequence-diagrams web service uses for rendering sequence diagrams. These sequence diagrams contain less details than .art-trace sequence diagrams and are mainly intended to serve as documentation that can explain how an application behaves internally in certain situations.

plantUML

This sample script translates an .art-trace file into the textual format that the plantUML web service uses for rendering sequence diagrams. These sequence diagrams contain less details than .art-trace sequence diagrams and are mainly intended to serve as documentation that can explain how an application behaves internally in certain situations. PlantUML supports more styling options than jsSequenceDiagram.

googleTraceEventFormat

This sample script translates an .art-trace file into the Google Trace Event Format, which is a JSON-based trace format. You can then open such a JSON file in a trace viewer that supports this format, for example Perfetto, for visualization and analysis.