MESSAGE
| DATE | 2015-02-17 |
| FROM | Ruben Safir
|
| SUBJECT | Subject: [LIU Comp Sci] OS Class Ctp 2 Dtrace
|
From owner-learn-outgoing-at-mrbrklyn.com Tue Feb 17 02:46:38 2015
Return-Path:
X-Original-To: archive-at-mrbrklyn.com
Delivered-To: archive-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix)
id EC938161162; Tue, 17 Feb 2015 02:46:37 -0500 (EST)
Delivered-To: learn-outgoing-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix, from userid 28)
id D85391612E7; Tue, 17 Feb 2015 02:46:37 -0500 (EST)
Delivered-To: learn-at-nylxs.com
Received: from mailbackend.panix.com (mailbackend.panix.com [166.84.1.89])
by mrbrklyn.com (Postfix) with ESMTP id 66C71161162
for ; Tue, 17 Feb 2015 02:46:37 -0500 (EST)
Received: from [10.0.0.19] (unknown [96.57.23.82])
by mailbackend.panix.com (Postfix) with ESMTPSA id EAD8419F13
for ; Tue, 17 Feb 2015 02:46:36 -0500 (EST)
Message-ID: <54E2F1DB.5020202-at-panix.com>
Date: Tue, 17 Feb 2015 02:46:35 -0500
From: Ruben Safir
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0
MIME-Version: 1.0
To: learn-at-nylxs.com
Subject: [LIU Comp Sci] OS Class Ctp 2 Dtrace
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Sender: owner-learn-at-mrbrklyn.com
Precedence: bulk
Reply-To: learn-at-mrbrklyn.com
Does anyone understand this? It is from Chapter to of the OS class
text. I'm very confused by the text here because it seems to be saying
that when probes are turned on they are rewritten and I don't know what
is rewritten. Can you rewrite a binary while its in memory? That would
be news to me.
Ruben
2.8.3
DTrace
DTrace is a facility that dynamically adds probes to a running system, both
in user processes and in the kernel. These probes can be queried via the D
programming language to determine an astonishing amount about the kernel,
the system state, and process activities. For example, Figure 2.20
follows an
application as it executes a system call (ioctl()) and shows the functional
calls within the kernel as they execute to perform the system call.
Lines ending
with “U” are executed in user mode, and lines ending in “K” in kernel mode.
88
Chapter 2 Operating-System Structures
Figure 2.19 The Windows task manager.
Debugging the interactions between user-level and kernel code is nearly
impossible without a toolset that understands both sets of code and can
instrument the interactions. For that toolset to be truly useful, it
must be able
to debug any area of a system, including areas that were not written with
debugging in mind, and do so without affecting system reliability. This tool
must also have a minimum performance impact—ideally it should have no
impact when not in use and a proportional impact during use. The DTrace tool
meets these requirements and provides a dynamic, safe, low-impact debugging
environment.
Until the DTrace framework and tools became available with Solaris 10,
kernel debugging was usually shrouded in mystery and accomplished via
happenstance and archaic code and tools. For example, CPUs have a breakpoint
feature that will halt execution and allow a debugger to examine the
state of the
system. Then execution can continue until the next breakpoint or
termination.
This method cannot be used in a multiuser operating-system kernel without
negatively affecting all of the users on the system. Profiling, which
periodically
samples the instruction pointer to determine which code is being
executed, can
show statistical trends but not individual activities. Code can be
included in
the kernel to emit specific data under specific circumstances, but that code
slows down the kernel and tends not to be included in the part of the kernel
where the specific problem being debugged is occurring.
2.8 Operating-System Debugging
89
# ./all.d ‘pgrep xclock‘ XEventsQueued
dtrace: script ’./all.d’ matched 52377 probes
CPU FUNCTION
0 –> XEventsQueued
U
0
–> _XEventsQueued
U
0
–> _X11TransBytesReadable
U
0
<– _X11TransBytesReadable
U
0
–> _X11TransSocketBytesReadable U
0
<– _X11TransSocketBytesreadable U
0
–> ioctl
U
0
–> ioctl
K
0
–> getf
K
0
–> set_active_fd
K
0
<– set_active_fd
K
0
<– getf
K
0
–> get_udatamodel
K
0
<– get_udatamodel
K
...
0
–> releasef
K
0
–> clear_active_fd
K
0
<– clear_active_fd
K
0
–> cv_broadcast
K
0
<– cv_broadcast
K
0
<– releasef
K
0
<– ioctl
K
0
<– ioctl
U
0
<– _XEventsQueued
U
0 <– XEventsQueued
U
Figure 2.20 Solaris 10 dtrace follows a system call within the kernel.
In contrast, DTrace runs on production systems—systems that are running
important or critical applications—and causes no harm to the system. It
slows activities while enabled, but after execution it resets the system
to its
pre-debugging state. It is also a broad and deep tool. It can broadly debug
everything happening in the system (both at the user and kernel levels and
between the user and kernel layers). It can also delve deep into code,
showing
individual CPU instructions or kernel subroutine activities.
DTrace is composed of a compiler, a framework, providers of probes
written within that framework, and consumers of those probes. DTrace
providers create probes. Kernel structures exist to keep track of all
probes that
the providers have created. The probes are stored in a hash-table data
structure
that is hashed by name and indexed according to unique probe identifiers.
When a probe is enabled, a bit of code in the area to be probed is rewritten
to call dtrace probe(probe identifier) and then continue with the code’s
original operation. Different providers create different kinds of
probes. For
example, a kernel system-call probe works differently from a user-process
probe, and that is different from an I/O probe.
DTrace features a compiler that generates a byte code that is run in the
kernel. This code is assured to be “safe” by the compiler. For example,
no loops
are allowed, and only specific kernel state modifications are allowed when
specifically requested. Only users with DTrace “privileges” (or “root”
users)
90
Chapter 2 Operating-System Structures
are allowed to use DTrace, as it can retrieve private kernel data (and
modify
data if requested). The generated code runs in the kernel and enables
probes.
It also enables consumers in user mode and enables communications between
the two.
A DTrace consumer is code that is interested in a probe and its results.
A consumer requests that the provider create one or more probes. When a
probe fires, it emits data that are managed by the kernel. Within the
kernel,
actions called enabling control blocks, or ECBs, are performed when probes
fire. One probe can cause multiple ECBs to execute if more than one consumer
is interested in that probe. Each ECB contains a predicate (“if
statement”) that
can filter out that ECB. Otherwise, the list of actions in the ECB is
executed. The
most common action is to capture some bit of data, such as a variable’s
value at
that point of the probe execution. By gathering such data, a complete
picture of
a user or kernel action can be built. Further, probes firing from both
user space
and the kernel can show how a user-level action caused kernel-level
reactions.
Such data are invaluable for performance monitoring and code optimization.
Once the probe consumer terminates, its ECBs are removed. If there are no
ECBs consuming a probe, the probe is removed. That involves rewriting the
code to remove the dtrace probe() call and put back the original code. Thus,
before a probe is created and after it is destroyed, the system is
exactly the
same, as if no probing occurred.
DTrace takes care to assure that probes do not use too much memory or
CPU capacity, which could harm the running system. The buffers used to hold
the probe results are monitored for exceeding default and maximum limits.
CPU time for probe execution is monitored as well. If limits are
exceeded, the
consumer is terminated, along with the offending probes. Buffers are
allocated
per CPU to avoid contention and data loss.
An example of D code and its output shows some of its utility. The following
program shows the DTrace code to enable scheduler probes and record the
amount of CPU time of each process running with user ID 101 while those
probes are enabled (that is, while the program runs):
sched:::on-cpu
uid == 101
{
self->ts = timestamp;
}
sched:::off-cpu
self->ts
{
-at-time[execname] = sum(timestamp - self->ts);
self->ts = 0;
}
The output of the program, showing the processes and how much time (in
nanoseconds) they spend running on the CPUs, is shown in Figure 2.21.
Because DTrace is part of the open-source OpenSolaris version of the Solaris
10 operating system, it has been added to other operating systems when those
2.9 Operating-System Generation
# dtrace -s sched.d
dtrace: script ’sched.d’ matched 6 probes
ˆC
gnome-settings-d
142354
gnome-vfs-daemon
158243
dsdm
189804
wnck-applet
200030
gnome-panel
277864
clock-applet
374916
mapping-daemon
385475
xscreensaver
514177
metacity
539281
Xorg
2579646
gnome-terminal
5007269
mixer applet2
7388447
java
10769137
Figure 2.21 Output of the D code.
91
systems do not have conflicting license agreements. For example, DTrace has
been added to Mac OS X and FreeBSD and will likely spread further due to its
unique capabilities. Other operating systems, especially the Linux
derivatives,
are adding kernel-tracing functionality as well. Still other operating
systems
are beginning to include performance and tracing tools fostered by
research at
various institutions, including the Paradyn project.
|
|
