MESSAGE
| DATE | 2021-01-26 |
| FROM | Richard Purdie
|
| SUBJECT | Re: [Hangout - NYLXS] Future plans for Autotools
|
Firstly, thanks for the 2.70 release! Yocto Project hasn't switched yet
as whilst we tested it and reported issues, there was a last minute
change which broke things for us. We'll get that sorted and upgrade.
I am one of the lead developers of the Yocto Project and at its heart
is OpenEmbedded which is basically a cross compiling operating system
build tool. We primarily target Linux and specialise in allowing
complete customisation of the end resulting images.
We try and solve a complex problem, pulling together all the pieces of
open source needed for an operating system including the toolchain,
cross compiling them and allowing the user to customise them in
infinitely different ways. Where the software uses autotools, we
actually regenerate the configure scripts and makefiles from the source
files before configuring and building the software. We do this since we
may need to include arch support that isn't part of the original
configure script, or to include fixes we carry to components like
libtool, which I'm say to say doesn't work properly cross compiling out
the box.
One scary statistic is that if we look at where our builds spend time,
40% of our wall clock build time is spent in "do_configure" tasks,
which is mostly running autotools and is single threaded. The compile
takes much less time as we can do things in parallel. Of that 40%,
about half of it is in autoreconf and half is running configure itself.
I did profile configure and the problem is that its basically a fork
bomb. Tons of forks to do trivial shell operations. I have less insight
how to speed up the autoreconf side of things as that mostly seems to
be in m4, although an m4 deletion party of obsolete things we don't
care about has come to mind before.
I like autotools, I've spent a lot of time debugging them but I also
know they solve some hard problems. I do wish there were more active
releases and an upstream we could engage with and get issues discussed
and resolved somehow.
I will observe that I once worked on a Makefile based build system
(buildroot) and I then somehow ultimately became the bitbake maintainer
which can be thought of as a custom python tool for parsing and dealing
with metadata. Over the years I have come to love python for handling
our recipes and configuration data. I saw suggestion of using some
other language for configure than shell and the idea sounds intriguing,
particularly after my positive experiences with python.
In terms of CI, Yocto Project had a challenge there as we have huge
possible numbers of configurations. Its taken us a few years but we do
now have a CI system I'm quite proud of. In around 6 hours we can take
a configuration and build it out from scratch for 4 principle
architectures in 32, x32/n32 and 64 bit variants for multiple init
systems, kernel versions, libc implementations and much much more. As
an idea, here is a link to one of our test reports:
https://autobuilder.yocto.io/pub/non-release/20210120-9/testresults/testresult-report.txt
That does correctly say it attempted about 2.6 million tests. Included
there are booting images under qemu for all our principle target
architectures and running the test suites included with many pieces of
software, including the testsuites with gcc, glibc, binutils, perl and
python and the ltp suite. We also cover things like reproducibility.
When we heard about the new release we did run 2.70 under this test
matrix and reported issues back so hopefully the release benefited from
this indirectly.
Doing something simpler with build testing for autoconf and other GNU
projects would be entirely possible, I mention this mainly to show what
can be done!
I do worry about the future for autotools, I see projects moving over
to alternatives and I am happy to see the performance gains we get from
it.
I'd mention that the copyright assignment situation needed for
contributions is prohibitive to people being able to participate
(including me in the past). GNU's thoughts on version control and
project interconnection is also somewhat discouraging to people in the
wider open source ecosystem although I don't follow it closely.
I've also not seen mention of it but the fact that GCC and the
toolchain use an ancient version of autoconf has always been rather sad
to me.
I do think autotools has a proud heritage but to survive into the next
decade, I do think it needs to change direction, stop supporting old
obsolete systems and embrace more new technology, much like it would
have done originally. Reducing to the lowest common denominator is ok
for a while but ultimately signs a death warrant as I've seen observed
on the libtool list by devlopers in recent years too.
Everything here is meant as constructive support of trying to find a
way to allow autotools to succeed into the future.
Cheers,
Richard
_______________________________________________
Hangout mailing list
Hangout-at-nylxs.com
http://lists.mrbrklyn.com/mailman/listinfo/hangout
|
|
