MESSAGE
| DATE | 2015-04-17 |
| FROM | Ruben
|
| SUBJECT | Subject: [LIU Comp Sci] more advanced coding musings ...
|
From owner-learn-outgoing-at-mrbrklyn.com Fri Apr 17 20:31:01 2015
Return-Path:
X-Original-To: archive-at-mrbrklyn.com
Delivered-To: archive-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix)
id E92A1161190; Fri, 17 Apr 2015 20:31:00 -0400 (EDT)
Delivered-To: learn-outgoing-at-mrbrklyn.com
Received: by mrbrklyn.com (Postfix, from userid 28)
id D73371612DF; Fri, 17 Apr 2015 20:31:00 -0400 (EDT)
Delivered-To: learn-at-nylxs.com
Received: from mail-qg0-f46.google.com (mail-qg0-f46.google.com [209.85.192.46])
by mrbrklyn.com (Postfix) with ESMTP id 05033161190
for ; Fri, 17 Apr 2015 20:30:34 -0400 (EDT)
Received: by qgeb100 with SMTP id b100so29507706qge.3
for ; Fri, 17 Apr 2015 17:30:33 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20130820;
h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to
:subject:content-type:content-transfer-encoding;
bh=nEmnwpFBaw1Ftnw/UHEee2OYkICsiXDQD3INokov0LM=;
b=Q9caSEPGz+tXgQA7CinHz1lIanwPozbXuXMLEK9sFJcjjjF/WXiSbdNQ3/zlLj5/g1
Itij64ahUggIO94wnQmhiXRKmNml3W4RjOboItLAi+59i1Lfbh3tCkw2BG4VZn+xY4PC
x1uj/hTURvLO7VC2L8Ltko4607SUrRXZKI4kD37S/dwSFoFNpom3HHjseMmnnUBhw6hg
N1557S/yijYZSjPcEPx+veqnCML5E219BQAratWR+VhTv2dgNbiFqdz2Bk1gZGOPO27P
12AgXn4W/dFtBuvN3MaNM2ZNgSDukdzWCIjM4eQkmIRVQFFwGsDxg79AAK27VuheWKme
k60g==
X-Gm-Message-State: ALoCoQnBAmYacmMZHx2CTqixOxRx8paOyXgd3/UQDV425Mq99R/3yWdZZCAFonb2xKIJ49m/n+7/
X-Received: by 10.140.151.75 with SMTP id 72mr6810450qhx.89.1429317033012;
Fri, 17 Apr 2015 17:30:33 -0700 (PDT)
Received: from [10.0.0.19] (www.mrbrklyn.com. [96.57.23.82])
by mx.google.com with ESMTPSA id q34sm9273470qkq.4.2015.04.17.17.30.32
for
(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Fri, 17 Apr 2015 17:30:32 -0700 (PDT)
Message-ID: <5531A5A7.8030501-at-my.liu.edu>
Date: Fri, 17 Apr 2015 20:30:31 -0400
From: Ruben
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0
MIME-Version: 1.0
To: learn-at-nylxs.com
Subject: [LIU Comp Sci] more advanced coding musings ...
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Sender: owner-learn-at-mrbrklyn.com
Precedence: bulk
Reply-To: learn-at-mrbrklyn.com
Binder vs. kdbus
A lot of people have asked about replacing Android’s binder
code with kdbus. I originally thought
this could be done, but as time has gone by, I’ve come to the conclusion
that this will not happen with the first version of kdbus, and possibly
can never happen.
First off, go read that link describing binder
that I pointed to above, especially
all of the links to different resources from that page. That should give
you more than you ever wanted to know about binder.
Short answer
Binder is bound to the CPU, D-Bus (and hence kdbus), is bound to RAM.
Long answer
Binder
Binder is an interface that Android uses to provide synchronous calling
(CPU) from one task to a thread of another task. There is no queueing
involved in these calls, other than the caller process is suspended
until the answering process returns. RAM is not interesting besides the
fact that it is used to share the data between the different callers.
The fact that the caller process gives up its CPU slice to the answering
process is key for how Android works with the binder library.
This is just like a syscall, and it behaves a lot like a mutex. The
communicating processes are directly connected to each other. There is
an upper limit of how many different processes can be using binder at
once, and I think it’s around 16 for most systems.
D-Bus
D-Bus is asynchronous, it queues (RAM) messages, keeps the messages in
order, and the receiver dequeues the messages. The CPU does not matter
at all other than it is used to do the asynchronous work of passing the
RAM around between the different processes.
This is a lot like network communication protocols. It is a very
“disconnected” communication method between processes. The upper limit
of message sizes and numbers is usually around 8Mb per connection and a
normal message is around 200-800 bytes.
Binder
The model of Binder was created for a microkernel-like device (side
note, go read this wonderful article
about the history of Danger
written by one of the engineers at that company for a glimpse into where
the Android internals came from, binder included.) The model of binder
is very limited, inflexible in its use-cases, but very powerful and
/extremely/ low-overhead and fast. Binder ensures that the same CPU
timeslice will go from the calling process into the called process’s
thread, and then come back into the caller when finished. There is
almost no scheduling involved, and is much like a syscall into the
kernel that does work for the calling process. This interface is very
well suited for cheap devices with almost no RAM and very low CPU resources.
So, for systems like Android, binder makes total sense, especially given
the history of it and where it was designed to be used.
D-Bus
D-Bus is a create-store-forward, compose reply and then
create-store-forward messaging model which is more complex than binder,
but because of that, it is extremely flexible, versatile, network
transparent, much easier to manage, and very easy to let fully untrusted
peers take part of the communication model (hint, never let this happen
with binder, or bad things will happen…) D-Bus can scale up to huge
amounts of data, and with the implementation of kdbus it is possible to
pass gigabytes of buffers to every connection on the bus if you really
wanted to. CPU-wise, it is not as efficient as binder, but is a much
better general-purpose solution for general-purpose machines and workloads.
CPU vs. RAM
Yes, it’s an over simplification of a different set of complex IPC
methods, but these 3 words should help you explain the differences
between binder and D-Bus and why kdbus isn’t going to be able to easily
replace binder anytime soon.
Never say never
Ok, before you start to object to the above statements, yes, we could
add functionality to kdbus to have some blocking ioctl calls that
implement something like: |write question -> block for reply and read
reply one answer| for the request side, and then on the server side do:
|write answer -> block in read| That would get kdbus a tiny bit closer
to the binder model, by queueing stuff in RAM instead of relying on a
thread pool.
That might work, but would require a lot of work on the binder library
side in Android, and as a very limited number of people have write
access to that code (they all can be counted on one hand), and it’s a
non-trivial amount of work for a core function of Android that is
working very well today, I don’t know if it will ever happen.
But anything is possible, it’s just software you know…
Thanks
Many thanks to Kay Sievers who came up with the CPU vs. RAM description
of binder and D-Bus and whose email I pretty much just copied into this
post. Also thanks to Kay and Lennart for taking the time and energy to
put up with my silly statements about how kdbus could replace binder,
and totally proving me wrong, sorry for having you spend so much time on
this, but I now know you are right.
Also thanks to Daniel Mack and Kay for doing so much work on the kdbus
kernel code, that I don’t think any of my original implementation is
even present anymore, which is probably a good thing. Also thanks to
Tejun Heo for help with the memfd implementation and cgroups help in kdbus.
Binary Blobs to C Structures
Sep 11th, 2013
Sometimes you don’t have access to vim’s wonderful xxd
tool, and you need to use
it to generate some .c code based on a binary file. This happened to me
recently when packaging up the EFI signing tools for Gentoo.
Adding a build requirement of vim for a single autogenerated file was
not an option for some users, so I created a perl version of the |xxd
-i| command line tool.
This works because everyone has perl in their build systems, whether
they like it or not. Instead of burying it in the efitools package,
here’s a copy of it for others to use if they want/need it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
|#!/usr/bin/env perl
#
# xxdi.pl - perl implementation of 'xxd -i' mode
#
# Copyright 2013 Greg Kroah-Hartman
# Copyright 2013 Linux Foundation
#
# Released under the GPLv2.
#
# Implements the "basic" functionality of 'xxd -i' in perl to keep build
# systems from having to build/install/rely on vim-core, which not all
# distros want to do. But everyone has perl, so use it instead.
use strict;
use warnings;
use File::Slurp qw(slurp);
my $indata = slurp(-at-ARGV ? $ARGV[0] : \*STDIN);
my $len_data = length($indata);
my $num_digits_per_line = 12;
my $var_name;
my $outdata;
# Use the variable name of the file we read from, converting '/' and '.
# to '_', or, if this is stdin, just use "stdin" as the name.
if (-at-ARGV) {
$var_name = $ARGV[0];
$var_name =~ s/\//_/g;
$var_name =~ s/\./_/g;
} else {
$var_name = "stdin";
}
$outdata .= "unsigned char $var_name\[] = {";
# trailing ',' is acceptable, so instead of duplicating the logic for
# just the last character, live with the extra ','.
for (my $key= 0; $key < $len_data; $key++) {
if ($key % $num_digits_per_line == 0) {
$outdata .= "\n\t";
}
$outdata .= sprintf("0x%.2x, ", ord(substr($indata, $key, 1)));
}
$outdata .= "\n};\nunsigned int $var_name\_len = $len_data;\n";
binmode STDOUT;
print {*STDOUT} $outdata;
|
Yes, I know I write perl code like a C programmer, that’s not an insult
to me.
|
|
