MESSAGE
| DATE | 2020-12-26 |
| FROM | Matthias Peng
|
| SUBJECT | Re: [Hangout - NYLXS] Confused about two development utils [EXT]
|
From hangout-bounces-at-nylxs.com Sat Dec 26 03:23:13 2020
Return-Path:
X-Original-To: archive-at-mrbrklyn.com
Delivered-To: archive-at-mrbrklyn.com
Received: from www2.mrbrklyn.com (www2.mrbrklyn.com [96.57.23.82])
by mrbrklyn.com (Postfix) with ESMTP id A1B0E164054;
Sat, 26 Dec 2020 03:23:12 -0500 (EST)
X-Original-To: hangout-at-www2.mrbrklyn.com
Delivered-To: hangout-at-www2.mrbrklyn.com
Received: by mrbrklyn.com (Postfix, from userid 1000)
id 4333116403F; Sat, 26 Dec 2020 03:23:01 -0500 (EST)
Resent-From: Ruben Safir
Resent-Date: Sat, 26 Dec 2020 03:23:01 -0500
Resent-Message-ID: <20201226082301.GA4921-at-www2.mrbrklyn.com>
Resent-To: hangout-at-mrbrklyn.com
X-Original-To: ruben-at-mrbrklyn.com
Delivered-To: ruben-at-mrbrklyn.com
Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org
[3.227.148.255]) by mrbrklyn.com (Postfix) with ESMTP id 35291164017
for ; Sat, 26 Dec 2020 03:01:30 -0500 (EST)
Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153])
by mxout1-ec2-va.apache.org (ASF Mail Server at
mxout1-ec2-va.apache.org) with SMTP id B0AEF44D96
for ; Sat, 26 Dec 2020 08:01:30 +0000 (UTC)
Received: (qmail 8131 invoked by uid 500); 26 Dec 2020 08:01:30 -0000
Mailing-List: contact modperl-help-at-perl.apache.org; run by ezmlm
Precedence: bulk
Delivered-To: mailing list modperl-at-perl.apache.org
Received: (qmail 8120 invoked by uid 99); 26 Dec 2020 08:01:29 -0000
Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org)
(116.203.196.100)
by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Dec 2020 08:01:29 +0000
Received: from localhost (localhost [127.0.0.1])
by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org)
with ESMTP id 26C8C1FF3A1
for ; Sat, 26 Dec 2020 08:01:29 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org
X-Spam-Flag: NO
X-Spam-Score: -0.001
X-Spam-Level:
X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31
tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_MSPIKE_H2=-0.001,
SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled
Authentication-Results: spamproc1-he-de.apache.org (amavisd-new);
dkim=pass (2048-bit key) header.d=gmail.com
Received: from mx1-ec2-va.apache.org ([116.203.227.195])
by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new,
port 10024) with ESMTP id RgafuvRUOczX for ;
Sat, 26 Dec 2020 08:01:26 +0000 (UTC)
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.196;
helo=mail-oi1-f196.google.com; envelope-from=pengmatthias-at-gmail.com;
receiver=
Received: from mail-oi1-f196.google.com (mail-oi1-f196.google.com
[209.85.167.196])
by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with
ESMTPS id B7890BC95A
for ; Sat, 26 Dec 2020 08:01:25 +0000 (UTC)
Received: by mail-oi1-f196.google.com with SMTP id x13so6566202oic.5
for ; Sat, 26 Dec 2020 00:01:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc; bh=U7SWrUJuzeU4r3T5btalDT4HmdiMxH5+7/ASjn33wgk=;
b=XoPwBRYaQQRSxtVh922IhdrD4WlfOXjYczpX9XwBn9aLpriZFpNnIi/mI9ggd1/fT0
3Yr/yknHV6g3XLBv48a9kT5fOnMgqLhOYKC8w7Mz8n2fI717zQfmu4LhxURGdl2GT8WL
g2iXKqPGe+Ey5Hl+blCag4g26JNVqBr76XR8zo6GF8QMjAyY6+rIVwv/xdmjpli5jRq+
MMfyvZUMPHOqRG+ALGeldLBOzPaxlZoJ4wvE0YipKYyM99acyEx7eDePo2rM56Ru/jgc
DRvea4xXhkoMTm5nni3oYYSnH3ZwNMnJspOEgHxcLmnqBGQ0gwjP9zwLrcHzg0VqUB+E
2Jfg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=U7SWrUJuzeU4r3T5btalDT4HmdiMxH5+7/ASjn33wgk=;
b=ntJTH6QfQR+ZvFQDZLVCBNSx9iu2PRYebpt42F/1ikgeHE0iV+fPjIF9fA/9/J2+10
gq1q+JrxVAwDeJwXjXL1EmV27MXsgkNSicsX/sTJ3N0NJkngcvad7MCnLj8n3/w3ctWV
zzvYr9XDJ37NW5vKk4kKGBQY9DMvyeViip2B5j61G/NdVpF9gZMv2lel/w3yo8EEcelB
x2f7JhwKZprJ2hp4GrMFYdtwIc9gdZKWB3PbcxVobFmMJm8ZqSK7NdKmV+IpQpOC5cD1
OAXw7M1l7zWHyRftpcoFKbYeUhi8RNivNxRVe71h1t/yiR5wpV0U1EqUBDq3FucnDsKh
lzzQ==
X-Gm-Message-State: AOAM5335lloQyweVIjvmVSJ8zFS1w/g1lhj00aG3YzDG9heCqFFWuLGk
tjaiGxwaN7KW7AZryZ8ygnQBoivTDm524RjAyFY=
X-Google-Smtp-Source: ABdhPJzKulpclxTkJnuZ1mL7bC2vvIHx69zRe+i255UKqlQn5O5lfXHcx+2Uhfhlo7NuWX1PNrieo26iSJg8v16upTE=
X-Received: by 2002:a54:4815:: with SMTP id j21mr7227541oij.154.1608969685206;
Sat, 26 Dec 2020 00:01:25 -0800 (PST)
MIME-Version: 1.0
References:
<971cc41d-b30e-7fc1-25a2-4a63f028321d-at-ice-sa.com>
<90ae0836-d487-926c-89e4-696a46fae57d-at-ice-sa.com>
<335e0e3cca2e4dd3aeb5f91d83ea08c0-at-sanger.ac.uk>
<11d9dcd77b2a4de7a98592c31664eb0c-at-sanger.ac.uk>
<5638d10591eb43c58c225deff698f14c-at-sanger.ac.uk>
<477c1bbb-4647-81fe-595b-d54d0d9e00f4-at-ice-sa.com>
In-Reply-To:
From: Matthias Peng
Date: Sat, 26 Dec 2020 16:01:14 +0800
Message-ID:
To: Sandhya
Cc: modperl-at-perl.apache.org
Subject: Re: [Hangout - NYLXS] Confused about two development utils [EXT]
X-BeenThere: hangout-at-nylxs.com
X-Mailman-Version: 2.1.30rc1
List-Id: NYLXS Tech Talk and Politics
List-Unsubscribe: ,
List-Archive:
List-Post:
List-Help:
List-Subscribe: ,
Content-Type: multipart/mixed; boundary="===============0059712626=="
Errors-To: hangout-bounces-at-nylxs.com
Sender: "Hangout"
--===============0059712626==
Content-Type: multipart/alternative; boundary="0000000000008a8b4d05b75972a1"
--0000000000008a8b4d05b75972a1
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
If I have been using modperl for development, does it influence I drive
Maserati? :)
> unsubscribe.
>
> On Fri, Dec 25, 2020 at 10:30 PM Andr=C3=A9 Warnier (tomcat/perl) <
> aw-at-ice-sa.com> wrote:
>
>> Hello James.
>> Bravo and many thanks for this excellent overview of your activities. Of
>> course the setup
>> (in your previous message) and the activities are very impressive by
>> themselves.
>> But in addition, even though your message is not in itself a perl
>> advocacy message, I feel
>> that it would have its right place in some perl/mod_perl advocacy forum,
>> because it
>> touches on some general idea which are valid /also/ for perl and mod_per=
l.
>> It was very refreshing to read for once a clear expos=C3=A9 of why it is=
still
>> important
>> nowadays to think before programming, to program efficiently, and to
>> choose the right tool
>> for the job at hand (be it perl, mod_perl, or any other) without the kin=
d
>> of off-the-cuff
>> general a-priori which tend to plague these discussions.
>>
>> And even though our own (commercial) activities and setups do not have
>> anything even close
>> to the scope which you describe, I would like to say that the same basic
>> principles which
>> you mention in your expos=C3=A9 are just as valid when you scale-down as=
when
>> you scale-up.
>> ("--you can=E2=80=99t just throw memory, CPUs, power at a problem =E2=80=
=93 you have to
>> think =E2=80=93 how can I do what I need to do with the least resources.=
.")
>> Even when you think of a single server, or a single server rack, at any
>> one period in time
>> there is always a practical limit as to how much memory or CPUs you can
>> fit in a given
>> server, or how many servers you can fit in a rack, or how many additiona=
l
>> Gb of bandwidth
>> you can allocate per server, beyond which there is a sudden "quantum
>> jump" as to how
>> practical and cost-effective a whole project becomes.
>> In that sense, I particulary enjoyed your examples of the database and o=
f
>> the additional
>> power line.
>>
>>
>> On 24.12.2020 02:38, James Smith wrote:
>> > We don=E2=80=99t use perl for everything, yes we use it for web data, =
yes we
>> still use it as the
>> > glue language in a lot of cases, the most complex stuff is done with C
>> (not even C++ as
>> > that is too slow). Others on site use Python, Java, Rust, Go, PHP,
>> along with looking at
>> > using GPUs in cases where code can be highly parallelised
>> >
>> > It is not just one application =E2=80=93 but many, many applications=
=E2=80=A6 All with
>> a common goal of
>> > understanding the human genome, and using it to assist in developing
>> new understanding and
>> > techniques which can advance health care.
>> >
>> > We are a very large sequencing centre (one of the largest in the world=
)
>> =E2=80=93 what I was
>> > pointing out is that you can=E2=80=99t just throw memory, CPUs, power =
at a
>> problem =E2=80=93 you have to
>> > think =E2=80=93 how can I do what I need to do with the least resource=
s. Rather
>> than what
>> > resources can I throw at the problem.
>> >
>> > Currently we are acting as the central repository for all COVID-19
>> sequencing in the UK,
>> > along with one of the largest =E2=80=9Cwet=E2=80=9D labs sequencing da=
ta for it =E2=80=93 and
>> that is half the
>> > sequenced samples in the whole world. The UK is sequencing more
>> COVID-19 genomes a day
>> > than most other countries have sequenced since the start of the
>> pandemic in Feb/Mar. This
>> > has lead to us discovering a new more transmissible version of the
>> virus, and it what part
>> > of the country the different strains are present =E2=80=93 no other co=
untry in
>> the world has the
>> > information, technology or infrastructure in place to achieve this.
>> >
>> > But this is just a small part of the genomic sequencing we are looking
>> at =E2=80=93 we work on:
>> > * other pathogens =E2=80=93 e.g. Plasmodium (Malaria);
>> > * cancer genomes (and how effective drugs are);
>> > * are a major part of the Human Cell Atlas which is looking at how the
>> expression of genes
>> > (in the simplest terms which ones are switched on and switched off) ar=
e
>> different in
>> > different tissues;
>> > * sequencing the genomes of other animals to understand their evolutio=
n;
>> > * and looking at some other species in detail, to see what we can lear=
n
>> from them when
>> > they have defective genes;
>> >
>> > Although all these are currently scaled back so that we can work
>> relentlessly to support
>> > the medical teams and other researchers get on top of COVID-19.
>> >
>> > What is interesting is that many of the developers we have on campus
>> (well all wfh at the
>> > moment) are all (relatively) old as we learnt to develop code on
>> machines with limited CPU
>> > and limited memory =E2=80=93 so that things had to be efficient, had t=
o be
>> compact=E2=80=A6. And that is
>> > as important now as it was 20 or 30 years ago =E2=80=93 the data we ha=
ndle is
>> going up faster than
>> > Moore=E2=80=99s Law! Many of us have pride in doing things as efficien=
tly as
>> possible.
>> >
>> > It took around 10 years to sequence and assemble the first human genom=
e
>> {well we are still
>> > tinkering with it and filling in the gaps} =E2=80=93 now at the instit=
ute we
>> can sequence and
>> > assemble around 400 human genomes in a day =E2=80=93 to the same quali=
ty!
>> >
>> > So most of our issues are due to the scale of the problems we face =E2=
=80=93
>> e.g. the human genome
>> > has 3 billion base-pairs (A, C, G, Ts) , so normal solutions don=E2=80=
=99t
>> scale to that (once
>> > many years ago we looked at setting up an Oracle database where there
>> was at least 1 row
>> > for every base pair =E2=80=93 recording all variants (think of them as=
spelling
>> mistakes, for
>> > example a T rather than an A, or an extra letter inserted or deleted)
>> for that base pair=E2=80=A6
>> > The schema was set up =E2=80=93 and then they realised it would take 1=
2 months
>> to load the data
>> > which we had then (which is probably less than a millionth of what we
>> have now)!
>> >
>> > Moving compute off site is a problem as the transfer of the level of
>> data we have would
>> > cause a problem =E2=80=93 you can=E2=80=99t easily move all the data t=
o the compute =E2=80=93
>> so you have to bring
>> > the compute to the data.
>> >
>> > The site I worked on before I became a more general developer was doin=
g
>> that =E2=80=93 and the
>> > code that was written 12-15 years ago is actually still going strong =
=E2=80=93
>> it has seen a few
>> > changes over the year =E2=80=93 many displays have had to be redevelop=
ed as the
>> scale of the data
>> > has got so big that even the summary pages we produced 10 years ago
>> have to be summarised
>> > because they are so large.
>> >
>> > *From:*Mithun Bhattacharya
>> > *Sent:* 24 December 2020 00:06
>> > *To:* mod_perl list
>> > *Subject:* Re: Confused about two development utils [EXT]
>> >
>> > James would you be able to share more info about your setup ?
>> >
>> > 1. What exactly is your application doing which requires so much memor=
y
>> and CPU - is it
>> > something like gene splicing (no i don't know much about it beyond
>> Jurassic Park :D )
>> >
>> > 2. Do you feel Perl was the best choice for whatever you are doing and
>> if yes then why ?
>> > How much of your stuff is using mod_perl considering you mentioned not
>> much is web related ?
>> >
>> > 3. What are the challenges you are currently facing with your
>> implementation ?
>> >
>> > On Wed, Dec 23, 2020 at 6:58 AM James Smith >> js5-at-sanger.ac.uk>>
>> > wrote:
>> >
>> > Oh but memory is a problem =E2=80=93 but not if you have just a sm=
all
>> cluster of machines!
>> >
>> > Our boxes are larger than that =E2=80=93 but they all run virtual =
machine
>> {only a small
>> > proportion web related} =E2=80=93 machines/memory would rapidly be=
come in
>> our data centre - we
>> > run VMWARE [995 hosts] and openstack [10,000s of hosts] + a
>> selection of large memory
>> > machines {measured in TBs of memory per machine }.
>> >
>> > We would be looking at somewhere between 0.5 PB and 1 PB of memory
>> =E2=80=93 not just the
>> > price of buying that amount of memory - for many machines we need
>> the fastest memory
>> > money can buy for the workload, but we would need a lot more CPUs
>> then we currently
>> > have as we would need a larger amount of machines to have 64GB
>> virtual machines {we
>> > would get 2 VMs per host. We currently have approx. 1-2000 CPUs
>> running our hardware
>> > (last time I had a figure) =E2=80=93 it would probably need to go =
to
>> approximately 5-10,000!
>> > It is not just the initial outlay but the environmental and
>> financial cost of running
>> > that number of machines, and finding space to run them without
>> putting the cooling
>> > costs through the roof!! That is without considering what
>> additional constraints on
>> > storage having the extra machines may have (at the last count a
>> year ago we had over
>> > 30 PBytes of storage on side =E2=80=93 and a large amount of offsi=
te backup.
>> >
>> > We would also stretch the amount of power we can get from the
>> national grid to power
>> > it all - we currently have 3 feeds from different part of the
>> national grid (we are
>> > fortunately in position where this is possible) and the dedicated
>> link we would need
>> > to add more power would be at least 50 miles long!
>> >
>> > So - managing cores/memory is vitally important to us =E2=80=93 mo=
ving to
>> the cloud is an
>> > option we are looking at =E2=80=93 but that is more than 4 times t=
he price
>> of our onsite
>> > set-up (with substantial discounts from AWS) and would require an
>> upgrade of our
>> > existing link to the internet =E2=80=93 which is currently 40Gbit =
of data
>> (I think).
>> >
>> > Currently we are analysing a very large amounts of data directly
>> linked to the current
>> > major world problem =E2=80=93 this is why the UK is currently bein=
g
>> isolated as we have
>> > discovered and can track a new strain, in near real time =E2=80=93=
other
>> countries have no
>> > ability to do this =E2=80=93 we in a day can and do handle, sequen=
ce and
>> analyse more samples
>> > than the whole of France has sequenced since February. We probably
>> don=E2=80=99t have more of
>> > the new variant strain than in other areas of the world =E2=80=93 =
it is
>> just that we know we
>> > have because of the amount of sequencing and analysis that we in
>> the UK have done.
>> >
>> > *From:*Matthias Peng >> pengmatthias-at-gmail.com>>
>> > *Sent:* 23 December 2020 12:02
>> > *To:* mod_perl list >> modperl-at-perl.apache.org>>
>> > *Subject:* Re: Confused about two development utils [EXT]
>> >
>> > Today memory is not serious problem, each of our server has 64GB
>> memory.
>> >
>> >
>> > Forgot to add - so our FCGI servers need a lot (and I mean a
>> lot) more memory than
>> > the mod_perl servers to serve the same level of content (just
>> in case memory blows
>> > up with FCGI backends)
>> >
>> > -----Original Message-----
>> > From: James Smith >
>> > Sent: 23 December 2020 11:34
>> > To: Andr=C3=A9 Warnier (tomcat/perl) >> aw-at-ice-sa.com>>;
>> > modperl-at-perl.apache.org
>> > Subject: RE: Confused about two development utils [EXT]
>> >
>> >
>> > > This costs memory, and all the more since many perl modules
>> are not
>> > thread-safe, so if you use them in your code, at this moment
>> the only safe way to
>> > do it is to use the Apache httpd prefork model. This means tha=
t
>> each Apache httpd
>> > child process has its own copy of the perl interpreter, which
>> means that the
>> > memory used by this embedded perl interpreter has to be counte=
d
>> n times (as many
>> > times as there are Apache httpd child processes running at any
>> one time).
>> >
>> > This isn=E2=80=99t quite true - if you load modules before the=
process
>> forks then they can
>> > cleverly share the same parts of memory. It is useful to be
>> able to "pre-load"
>> > core functionality which is used across all functions {this is
>> the case in Linux
>> > anyway}. It also speeds up child process generation as the
>> modules are already in
>> > memory and converted to byte code.
>> >
>> > One of the great advantages of mod_perl is Apache2::SizeLimit
>> which can blow away
>> > large child process - and then if needed create new ones. This
>> is not the case
>> > with some of the FCGI solutions as the individual processes ca=
n
>> grow if there is a
>> > memory leak or a request that retrieves a large amount of
>> content (even if not
>> > served), but perl can't give the memory back. So FCGI processe=
s
>> only get bigger
>> > and bigger and eventually blow up memory (or hit swap first)
>> >
>> >
>> >
>> >
>> >
>> > --
>> > The Wellcome Sanger Institute is operated by Genome Research
>> Limited, a charity
>> > registered in England with number 1021457 and a company
>> registered in England
>> > with number 2742969, whose registered office is 215 Euston
>> Road, London, NW1
>> =3Dgmail&source=3Dg>
>> 2
>> > [google.com]
>> > <
>> https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_ma=
ps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-=
26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=
=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=
=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D
>> >BE.
>> >
>> >
>> >
>> > --
>> > The Wellcome Sanger Institute is operated by Genome Research
>> > Limited, a charity registered in England with number 1021457
>> and a
>> > company registered in England with number 2742969, whose
>> registered
>> > office is 215 Euston Road, London, NW1
>> =3Dgmail&source=3Dg>
>> 2 [google.com]
>> > <
>> https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_ma=
ps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-=
26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=
=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=
=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D
>> >BE.
>> >
>> > -- The Wellcome Sanger Institute is operated by Genome Research
>> Limited, a charity
>> > registered in England with number 1021457 and a company registered
>> in England with
>> > number 2742969, whose registered office is 215 Euston Road,
>> London, NW1 2BE
>> ry=3Dgmail&source=3Dg>
>> .
>> >
>> > -- The Wellcome Sanger Institute is operated by Genome Research
>> Limited, a charity
>> > registered in England with number 1021457 and a company registered in
>> England with number
>> > 2742969, whose registered office is 215 Euston Road, London, NW1 2BE
>> ry=3Dgmail&source=3Dg>
>> .
>>
>>
--0000000000008a8b4d05b75972a1
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
If I have been using modperl for development, does it inf=
luence I drive Maserati? :) quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
solid;padding-left:1ex">
unsubscribe.
ass=3D"gmail_quote"> ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px=
solid rgb(204,204,204);padding-left:1ex">Hello James.
Bravo and many thanks for this excellent overview of your activities. Of co=
urse the setup
(in your previous message) and the activities are very impressive by themse=
lves.
But in addition, even though your message is not in itself a perl advocacy =
message, I feel
that it would have its right place in some perl/mod_perl advocacy forum, be=
cause it
touches on some general idea which are valid /also/ for perl and mod_perl.<=
br>
It was very refreshing to read for once a clear expos=C3=A9 of why it is st=
ill important
nowadays to think before programming, to program efficiently, and to choose=
the right tool
for the job at hand (be it perl, mod_perl, or any other) without the kind o=
f off-the-cuff
general a-priori which tend to plague these discussions.
And even though our own (commercial) activities and setups do not have anyt=
hing even close
to the scope which you describe, I would like to say that the same basic pr=
inciples which
you mention in your expos=C3=A9 are just as valid when you scale-down as wh=
en you scale-up.
("--you can=E2=80=99t just throw memory, CPUs, power at a problem =E2=
=80=93 you have to
think =E2=80=93 how can I do what I need to do with the least resources..&q=
uot;)
Even when you think of a single server, or a single server rack, at any one=
period in time
there is always a practical limit as to how much memory or CPUs you can fit=
in a given
server, or how many servers you can fit in a rack, or how many additional G=
b of bandwidth
you can allocate per server, beyond which there is a sudden "quantum j=
ump" as to how
practical and cost-effective a whole project becomes.
In that sense, I particulary enjoyed your examples of the database and of t=
he additional
power line.
On 24.12.2020 02:38, James Smith wrote:
> We don=E2=80=99t use perl for everything, yes we use it for web data, =
yes we still use it as the
> glue language in a lot of cases, the most complex stuff is done with C=
(not even C++ as
> that is too slow). Others on site use Python, Java, Rust, Go, PHP, alo=
ng with looking at
> using GPUs in cases where code can be highly parallelised
>
> It is not just one application =E2=80=93 but many, many applications=
=E2=80=A6 All with a common goal of
> understanding the human genome, and using it to assist in developing n=
ew understanding and
> techniques which can advance health care.
>
> We are a very large sequencing centre (one of the largest in the world=
) =E2=80=93 what I was
> pointing out is that you can=E2=80=99t just throw memory, CPUs, power =
at a problem =E2=80=93 you have to
> think =E2=80=93 how can I do what I need to do with the least resource=
s. Rather than what
> resources can I throw at the problem.
>
> Currently we are acting as the central repository for all COVID-19 seq=
uencing in the UK,
> along with one of the largest =E2=80=9Cwet=E2=80=9D labs sequencing da=
ta for it =E2=80=93 and that is half the
> sequenced samples in the whole world. The UK is sequencing more COVID-=
19 genomes a day
> than most other countries have sequenced since the start of the pandem=
ic in Feb/Mar. This
> has lead to us discovering a new more transmissible version of the vir=
us, and it what part
> of the country the different strains are present =E2=80=93 no other co=
untry in the world has the
> information, technology or infrastructure in place to achieve this.>
>
> But this is just a small part of the genomic sequencing we are looking=
at =E2=80=93 we work on:
> * other pathogens =E2=80=93 e.g. Plasmodium (Malaria);
> * cancer genomes (and how effective drugs are);
> * are a major part of the Human Cell Atlas which is looking at how the=
expression of genes
> (in the simplest terms which ones are switched on and switched off) ar=
e different in
> different tissues;
> * sequencing the genomes of other animals to understand their evolutio=
n;
> * and looking at some other species in detail, to see what we can lear=
n from them when
> they have defective genes;
>
> Although all these are currently scaled back so that we can work relen=
tlessly to support
> the medical teams and other researchers get on top of COVID-19.
>
> What is interesting is that many of the developers we have on campus (=
well all wfh at the
> moment) are all (relatively) old as we learnt to develop code on machi=
nes with limited CPU
> and limited memory =E2=80=93 so that things had to be efficient, had t=
o be compact=E2=80=A6. And that is
> as important now as it was 20 or 30 years ago =E2=80=93 the data we ha=
ndle is going up faster than
> Moore=E2=80=99s Law! Many of us have pride in doing things as efficien=
tly as possible.
>
> It took around 10 years to sequence and assemble the first human genom=
e {well we are still
> tinkering with it and filling in the gaps} =E2=80=93 now at the instit=
ute we can sequence and
> assemble around 400 human genomes in a day =E2=80=93 to the same quali=
ty!
>
> So most of our issues are due to the scale of the problems we face =E2=
=80=93 e.g. the human genome
> has 3 billion base-pairs (A, C, G, Ts) , so normal solutions don=E2=80=
=99t scale to that (once
> many years ago we looked at setting up an Oracle database where there =
was at least 1 row
> for every base pair =E2=80=93 recording all variants (think of them as=
spelling mistakes, for
> example a T rather than an A, or an extra letter inserted or deleted) =
for that base pair=E2=80=A6
> The schema was set up =E2=80=93 and then they realised it would take 1=
2 months to load the data
> which we had then (which is probably less than a millionth of what we =
have now)!
>
> Moving compute off site is a problem as the transfer of the level of d=
ata we have would
> cause a problem =E2=80=93 you can=E2=80=99t easily move all the data t=
o the compute =E2=80=93 so you have to bring
> the compute to the data.
>
> The site I worked on before I became a more general developer was doin=
g that =E2=80=93 and the
> code that was written 12-15 years ago is actually still going strong =
=E2=80=93 it has seen a few
> changes over the year =E2=80=93 many displays have had to be redevelop=
ed as the scale of the data
> has got so big that even the summary pages we produced 10 years ago ha=
ve to be summarised
> because they are so large.
>
> *From:*Mithun Bhattacharya <get=3D"_blank">mithnb-at-gmail.com>
> *Sent:* 24 December 2020 00:06
> *To:* mod_perl list <get=3D"_blank">modperl-at-perl.apache.org>
> *Subject:* Re: Confused about two development utils [EXT]
>
> James would you be able to share more info about your setup ?
>
> 1. What exactly is your application doing which requires so much memor=
y and CPU - is it
> something like gene splicing (no i don't know much about it beyond=
Jurassic Park :D )
>
> 2. Do you feel Perl was the best choice for whatever you are doing and=
if yes then why ?
> How much of your stuff is using mod_perl considering you mentioned not=
much is web related ?
>
> 3. What are the challenges you are currently facing with your implemen=
tation ?
>
> On Wed, Dec 23, 2020 at 6:58 AM James Smith <sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk <mailto:mailto:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk>> >
> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Oh but memory is a problem =E2=80=93 but not if you=
have just a small cluster of machines!
>
>=C2=A0 =C2=A0 =C2=A0Our boxes are larger than that =E2=80=93 but they a=
ll run virtual machine {only a small
>=C2=A0 =C2=A0 =C2=A0proportion web related} =E2=80=93 machines/memory w=
ould rapidly become in our data centre - we
>=C2=A0 =C2=A0 =C2=A0run VMWARE [995 hosts] and openstack [10,000s of ho=
sts] + a selection of large memory
>=C2=A0 =C2=A0 =C2=A0machines {measured in TBs of memory per machine }.<=
br>
>
>=C2=A0 =C2=A0 =C2=A0We would be looking at somewhere between 0.5 PB and=
1 PB of memory =E2=80=93 not just the
>=C2=A0 =C2=A0 =C2=A0price of buying that amount of memory - for many ma=
chines we need the fastest memory
>=C2=A0 =C2=A0 =C2=A0money can buy for the workload, but we would need a=
lot more CPUs then we currently
>=C2=A0 =C2=A0 =C2=A0have as we would need a larger amount of machines t=
o have 64GB virtual machines {we
>=C2=A0 =C2=A0 =C2=A0would get 2 VMs per host. We currently have approx.=
1-2000 CPUs running our hardware
>=C2=A0 =C2=A0 =C2=A0(last time I had a figure) =E2=80=93 it would proba=
bly need to go to approximately 5-10,000!
>=C2=A0 =C2=A0 =C2=A0It is not just the initial outlay but the environme=
ntal and financial cost of running
>=C2=A0 =C2=A0 =C2=A0that number of machines, and finding space to run t=
hem without putting the cooling
>=C2=A0 =C2=A0 =C2=A0costs through the roof!! That is without considerin=
g what additional constraints on
>=C2=A0 =C2=A0 =C2=A0storage having the extra machines may have (at the =
last count a year ago we had over
>=C2=A0 =C2=A0 =C2=A030 PBytes of storage on side =E2=80=93 and a large =
amount of offsite backup.
>
>=C2=A0 =C2=A0 =C2=A0We would also stretch the amount of power we can ge=
t from the national grid to power
>=C2=A0 =C2=A0 =C2=A0it all - we currently have 3 feeds from different p=
art of the national grid (we are
>=C2=A0 =C2=A0 =C2=A0fortunately in position where this is possible) and=
the dedicated link we would need
>=C2=A0 =C2=A0 =C2=A0to add more power would be at least 50 miles long!<=
br>
>
>=C2=A0 =C2=A0 =C2=A0So - managing cores/memory is vitally important to =
us =E2=80=93 moving to the cloud is an
>=C2=A0 =C2=A0 =C2=A0option we are looking at =E2=80=93 but that is more=
than 4 times the price of our onsite
>=C2=A0 =C2=A0 =C2=A0set-up (with substantial discounts from AWS) and wo=
uld require an upgrade of our
>=C2=A0 =C2=A0 =C2=A0existing link to the internet =E2=80=93 which is cu=
rrently 40Gbit of data (I think).
>
>=C2=A0 =C2=A0 =C2=A0Currently we are analysing a very large amounts of =
data directly linked to the current
>=C2=A0 =C2=A0 =C2=A0major world problem =E2=80=93 this is why the UK is=
currently being isolated as we have
>=C2=A0 =C2=A0 =C2=A0discovered and can track a new strain, in near real=
time =E2=80=93 other countries have no
>=C2=A0 =C2=A0 =C2=A0ability to do this =E2=80=93 we in a day can and do=
handle, sequence and analyse more samples
>=C2=A0 =C2=A0 =C2=A0than the whole of France has sequenced since Februa=
ry. We probably don=E2=80=99t have more of
>=C2=A0 =C2=A0 =C2=A0the new variant strain than in other areas of the w=
orld =E2=80=93 it is just that we know we
>=C2=A0 =C2=A0 =C2=A0have because of the amount of sequencing and analys=
is that we in the UK have done.
>
>=C2=A0 =C2=A0 =C2=A0*From:*Matthias Peng <hias-at-gmail.com" target=3D"_blank">pengmatthias-at-gmail.com <mailto:href=3D"mailto:pengmatthias-at-gmail.com" target=3D"_blank">pengmatthias-at-gmail=
.com>>
>=C2=A0 =C2=A0 =C2=A0*Sent:* 23 December 2020 12:02
>=C2=A0 =C2=A0 =C2=A0*To:* mod_perl list <erl.apache.org" target=3D"_blank">modperl-at-perl.apache.org <mailto: href=3D"mailto:modperl-at-perl.apache.org" target=3D"_blank">modperl-at-perl.apa=
che.org>>
>=C2=A0 =C2=A0 =C2=A0*Subject:* Re: Confused about two development utils=
[EXT]
>
>=C2=A0 =C2=A0 =C2=A0Today memory is not serious problem, each of our se=
rver has 64GB memory.
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Forgot to add - so our FCGI servers n=
eed a lot (and I mean a lot) more memory than
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0the mod_perl servers to serve the sam=
e level of content (just in case memory blows
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0up with FCGI backends)
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-----Original Message-----
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0From: James Smith <to:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk <mailto:ref=3D"mailto:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk>&=
gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sent: 23 December 2020 11:34
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0To: Andr=C3=A9 Warnier (tomcat/perl) =
<aw-at-ice-sa.com &l=
t;mailto:aw-at-ice-sa.coma>>>;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.org" target=3D"_blank">modperl-at-perl.apache.org <mailto:ailto:modperl-at-perl.apache.org" target=3D"_blank">modperl-at-perl.apache.org>>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Subject: RE: Confused about two devel=
opment utils [EXT]
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > This costs memory, and all the =
more since many perl modules are not
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0thread-safe, so if you use them in yo=
ur code, at this moment the only safe way to
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0do it is to use the Apache httpd pref=
ork model. This means that each Apache httpd
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0child process has its own copy of the=
perl interpreter, which means that the
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory used by this embedded perl int=
erpreter has to be counted n times (as many
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0times as there are Apache httpd child=
processes running at any one time).
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0This isn=E2=80=99t quite true - if yo=
u load modules before the process forks then they can
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cleverly share the same parts of memo=
ry. It is useful to be able to "pre-load"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0core functionality which is used acro=
ss all functions {this is the case in Linux
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0anyway}. It also speeds up child proc=
ess generation as the modules are already in
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory and converted to byte code.>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0One of the great advantages of mod_pe=
rl is Apache2::SizeLimit which can blow away
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0large child process - and then if nee=
ded create new ones. This is not the case
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with some of the FCGI solutions as th=
e individual processes can grow if there is a
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory leak or a request that retriev=
es a large amount of content (even if not
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0served), but perl can't give the =
memory back. So FCGI processes only get bigger
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0and bigger and eventually blow up mem=
ory (or hit swap first)
>
>
>
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The Wellcome Sanger Institute =
is operated by Genome Research=C2=A0 Limited, a charity
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0registered in England with number 102=
1457 and a=C2=A0 company registered in England
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with number 2742969, whose registered=
=C2=A0 office is oad,+London,+NW1?entry=3Dgmail&source=3Dg">215 Euston Road, London, NW1=
2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0["noreferrer" target=3D"_blank">google.com]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<ofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_maps_search_s-2B215-2BEusto=
n-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=3DDwMF=
aQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=3DoH2yp0ge1ecj=
4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=3DxU3F=
4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D" rel=3D"noreferrer" target=
=3D"_blank">https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.goog=
le.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry=
-3Dgmail-26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8u=
clZFI0SqQnqBo&r=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_o=
gNXEVR-4ixdkrhy5khQjA&s=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&a=
mp;e=3D>BE.
>
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The Wellcome Sanger Institute =
is operated by Genome Research
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Limited, a charity registered =
in England with number 1021457 and a
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0company registered in England =
with number 2742969, whose registered
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0office is ww.google.com/maps/search/215+Euston+Road,+London,+NW1?entry=3Dgmail&so=
urce=3Dg">215 Euston Road, London, NW1 2 [ rel=3D"noreferrer" target=3D"_blank">google.com]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<ofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_maps_search_s-2B215-2BEusto=
n-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=3DDwMF=
aQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=3DoH2yp0ge1ecj=
4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=3DxU3F=
4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D" rel=3D"noreferrer" target=
=3D"_blank">https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.goog=
le.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry=
-3Dgmail-26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8u=
clZFI0SqQnqBo&r=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_o=
gNXEVR-4ixdkrhy5khQjA&s=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&a=
mp;e=3D>BE.
>
>=C2=A0 =C2=A0 =C2=A0-- The Wellcome Sanger Institute is operated by Gen=
ome Research Limited, a charity
>=C2=A0 =C2=A0 =C2=A0registered in England with number 1021457 and a com=
pany registered in England with
>=C2=A0 =C2=A0 =C2=A0number 2742969, whose registered office is =3D"https://www.google.com/maps/search/215+Euston+Road,+London,+NW1+2BE?ent=
ry=3Dgmail&source=3Dg">215 Euston Road, London, NW1 2BE.
>
> -- The Wellcome Sanger Institute is operated by Genome Research Limite=
d, a charity
> registered in England with number 1021457 and a company registered in =
England with number
> 2742969, whose registered office is maps/search/215+Euston+Road,+London,+NW1+2BE?entry=3Dgmail&source=3Dg">=
215 Euston Road, London, NW1 2BE.
--0000000000008a8b4d05b75972a1--
--===============0059712626==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Hangout mailing list
Hangout-at-nylxs.com
http://lists.mrbrklyn.com/mailman/listinfo/hangout
--===============0059712626==--
--===============0059712626==
Content-Type: multipart/alternative; boundary="0000000000008a8b4d05b75972a1"
--0000000000008a8b4d05b75972a1
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
If I have been using modperl for development, does it influence I drive
Maserati? :)
> unsubscribe.
>
> On Fri, Dec 25, 2020 at 10:30 PM Andr=C3=A9 Warnier (tomcat/perl) <
> aw-at-ice-sa.com> wrote:
>
>> Hello James.
>> Bravo and many thanks for this excellent overview of your activities. Of
>> course the setup
>> (in your previous message) and the activities are very impressive by
>> themselves.
>> But in addition, even though your message is not in itself a perl
>> advocacy message, I feel
>> that it would have its right place in some perl/mod_perl advocacy forum,
>> because it
>> touches on some general idea which are valid /also/ for perl and mod_per=
l.
>> It was very refreshing to read for once a clear expos=C3=A9 of why it is=
still
>> important
>> nowadays to think before programming, to program efficiently, and to
>> choose the right tool
>> for the job at hand (be it perl, mod_perl, or any other) without the kin=
d
>> of off-the-cuff
>> general a-priori which tend to plague these discussions.
>>
>> And even though our own (commercial) activities and setups do not have
>> anything even close
>> to the scope which you describe, I would like to say that the same basic
>> principles which
>> you mention in your expos=C3=A9 are just as valid when you scale-down as=
when
>> you scale-up.
>> ("--you can=E2=80=99t just throw memory, CPUs, power at a problem =E2=80=
=93 you have to
>> think =E2=80=93 how can I do what I need to do with the least resources.=
.")
>> Even when you think of a single server, or a single server rack, at any
>> one period in time
>> there is always a practical limit as to how much memory or CPUs you can
>> fit in a given
>> server, or how many servers you can fit in a rack, or how many additiona=
l
>> Gb of bandwidth
>> you can allocate per server, beyond which there is a sudden "quantum
>> jump" as to how
>> practical and cost-effective a whole project becomes.
>> In that sense, I particulary enjoyed your examples of the database and o=
f
>> the additional
>> power line.
>>
>>
>> On 24.12.2020 02:38, James Smith wrote:
>> > We don=E2=80=99t use perl for everything, yes we use it for web data, =
yes we
>> still use it as the
>> > glue language in a lot of cases, the most complex stuff is done with C
>> (not even C++ as
>> > that is too slow). Others on site use Python, Java, Rust, Go, PHP,
>> along with looking at
>> > using GPUs in cases where code can be highly parallelised
>> >
>> > It is not just one application =E2=80=93 but many, many applications=
=E2=80=A6 All with
>> a common goal of
>> > understanding the human genome, and using it to assist in developing
>> new understanding and
>> > techniques which can advance health care.
>> >
>> > We are a very large sequencing centre (one of the largest in the world=
)
>> =E2=80=93 what I was
>> > pointing out is that you can=E2=80=99t just throw memory, CPUs, power =
at a
>> problem =E2=80=93 you have to
>> > think =E2=80=93 how can I do what I need to do with the least resource=
s. Rather
>> than what
>> > resources can I throw at the problem.
>> >
>> > Currently we are acting as the central repository for all COVID-19
>> sequencing in the UK,
>> > along with one of the largest =E2=80=9Cwet=E2=80=9D labs sequencing da=
ta for it =E2=80=93 and
>> that is half the
>> > sequenced samples in the whole world. The UK is sequencing more
>> COVID-19 genomes a day
>> > than most other countries have sequenced since the start of the
>> pandemic in Feb/Mar. This
>> > has lead to us discovering a new more transmissible version of the
>> virus, and it what part
>> > of the country the different strains are present =E2=80=93 no other co=
untry in
>> the world has the
>> > information, technology or infrastructure in place to achieve this.
>> >
>> > But this is just a small part of the genomic sequencing we are looking
>> at =E2=80=93 we work on:
>> > * other pathogens =E2=80=93 e.g. Plasmodium (Malaria);
>> > * cancer genomes (and how effective drugs are);
>> > * are a major part of the Human Cell Atlas which is looking at how the
>> expression of genes
>> > (in the simplest terms which ones are switched on and switched off) ar=
e
>> different in
>> > different tissues;
>> > * sequencing the genomes of other animals to understand their evolutio=
n;
>> > * and looking at some other species in detail, to see what we can lear=
n
>> from them when
>> > they have defective genes;
>> >
>> > Although all these are currently scaled back so that we can work
>> relentlessly to support
>> > the medical teams and other researchers get on top of COVID-19.
>> >
>> > What is interesting is that many of the developers we have on campus
>> (well all wfh at the
>> > moment) are all (relatively) old as we learnt to develop code on
>> machines with limited CPU
>> > and limited memory =E2=80=93 so that things had to be efficient, had t=
o be
>> compact=E2=80=A6. And that is
>> > as important now as it was 20 or 30 years ago =E2=80=93 the data we ha=
ndle is
>> going up faster than
>> > Moore=E2=80=99s Law! Many of us have pride in doing things as efficien=
tly as
>> possible.
>> >
>> > It took around 10 years to sequence and assemble the first human genom=
e
>> {well we are still
>> > tinkering with it and filling in the gaps} =E2=80=93 now at the instit=
ute we
>> can sequence and
>> > assemble around 400 human genomes in a day =E2=80=93 to the same quali=
ty!
>> >
>> > So most of our issues are due to the scale of the problems we face =E2=
=80=93
>> e.g. the human genome
>> > has 3 billion base-pairs (A, C, G, Ts) , so normal solutions don=E2=80=
=99t
>> scale to that (once
>> > many years ago we looked at setting up an Oracle database where there
>> was at least 1 row
>> > for every base pair =E2=80=93 recording all variants (think of them as=
spelling
>> mistakes, for
>> > example a T rather than an A, or an extra letter inserted or deleted)
>> for that base pair=E2=80=A6
>> > The schema was set up =E2=80=93 and then they realised it would take 1=
2 months
>> to load the data
>> > which we had then (which is probably less than a millionth of what we
>> have now)!
>> >
>> > Moving compute off site is a problem as the transfer of the level of
>> data we have would
>> > cause a problem =E2=80=93 you can=E2=80=99t easily move all the data t=
o the compute =E2=80=93
>> so you have to bring
>> > the compute to the data.
>> >
>> > The site I worked on before I became a more general developer was doin=
g
>> that =E2=80=93 and the
>> > code that was written 12-15 years ago is actually still going strong =
=E2=80=93
>> it has seen a few
>> > changes over the year =E2=80=93 many displays have had to be redevelop=
ed as the
>> scale of the data
>> > has got so big that even the summary pages we produced 10 years ago
>> have to be summarised
>> > because they are so large.
>> >
>> > *From:*Mithun Bhattacharya
>> > *Sent:* 24 December 2020 00:06
>> > *To:* mod_perl list
>> > *Subject:* Re: Confused about two development utils [EXT]
>> >
>> > James would you be able to share more info about your setup ?
>> >
>> > 1. What exactly is your application doing which requires so much memor=
y
>> and CPU - is it
>> > something like gene splicing (no i don't know much about it beyond
>> Jurassic Park :D )
>> >
>> > 2. Do you feel Perl was the best choice for whatever you are doing and
>> if yes then why ?
>> > How much of your stuff is using mod_perl considering you mentioned not
>> much is web related ?
>> >
>> > 3. What are the challenges you are currently facing with your
>> implementation ?
>> >
>> > On Wed, Dec 23, 2020 at 6:58 AM James Smith >> js5-at-sanger.ac.uk>>
>> > wrote:
>> >
>> > Oh but memory is a problem =E2=80=93 but not if you have just a sm=
all
>> cluster of machines!
>> >
>> > Our boxes are larger than that =E2=80=93 but they all run virtual =
machine
>> {only a small
>> > proportion web related} =E2=80=93 machines/memory would rapidly be=
come in
>> our data centre - we
>> > run VMWARE [995 hosts] and openstack [10,000s of hosts] + a
>> selection of large memory
>> > machines {measured in TBs of memory per machine }.
>> >
>> > We would be looking at somewhere between 0.5 PB and 1 PB of memory
>> =E2=80=93 not just the
>> > price of buying that amount of memory - for many machines we need
>> the fastest memory
>> > money can buy for the workload, but we would need a lot more CPUs
>> then we currently
>> > have as we would need a larger amount of machines to have 64GB
>> virtual machines {we
>> > would get 2 VMs per host. We currently have approx. 1-2000 CPUs
>> running our hardware
>> > (last time I had a figure) =E2=80=93 it would probably need to go =
to
>> approximately 5-10,000!
>> > It is not just the initial outlay but the environmental and
>> financial cost of running
>> > that number of machines, and finding space to run them without
>> putting the cooling
>> > costs through the roof!! That is without considering what
>> additional constraints on
>> > storage having the extra machines may have (at the last count a
>> year ago we had over
>> > 30 PBytes of storage on side =E2=80=93 and a large amount of offsi=
te backup.
>> >
>> > We would also stretch the amount of power we can get from the
>> national grid to power
>> > it all - we currently have 3 feeds from different part of the
>> national grid (we are
>> > fortunately in position where this is possible) and the dedicated
>> link we would need
>> > to add more power would be at least 50 miles long!
>> >
>> > So - managing cores/memory is vitally important to us =E2=80=93 mo=
ving to
>> the cloud is an
>> > option we are looking at =E2=80=93 but that is more than 4 times t=
he price
>> of our onsite
>> > set-up (with substantial discounts from AWS) and would require an
>> upgrade of our
>> > existing link to the internet =E2=80=93 which is currently 40Gbit =
of data
>> (I think).
>> >
>> > Currently we are analysing a very large amounts of data directly
>> linked to the current
>> > major world problem =E2=80=93 this is why the UK is currently bein=
g
>> isolated as we have
>> > discovered and can track a new strain, in near real time =E2=80=93=
other
>> countries have no
>> > ability to do this =E2=80=93 we in a day can and do handle, sequen=
ce and
>> analyse more samples
>> > than the whole of France has sequenced since February. We probably
>> don=E2=80=99t have more of
>> > the new variant strain than in other areas of the world =E2=80=93 =
it is
>> just that we know we
>> > have because of the amount of sequencing and analysis that we in
>> the UK have done.
>> >
>> > *From:*Matthias Peng >> pengmatthias-at-gmail.com>>
>> > *Sent:* 23 December 2020 12:02
>> > *To:* mod_perl list >> modperl-at-perl.apache.org>>
>> > *Subject:* Re: Confused about two development utils [EXT]
>> >
>> > Today memory is not serious problem, each of our server has 64GB
>> memory.
>> >
>> >
>> > Forgot to add - so our FCGI servers need a lot (and I mean a
>> lot) more memory than
>> > the mod_perl servers to serve the same level of content (just
>> in case memory blows
>> > up with FCGI backends)
>> >
>> > -----Original Message-----
>> > From: James Smith >
>> > Sent: 23 December 2020 11:34
>> > To: Andr=C3=A9 Warnier (tomcat/perl) >> aw-at-ice-sa.com>>;
>> > modperl-at-perl.apache.org
>> > Subject: RE: Confused about two development utils [EXT]
>> >
>> >
>> > > This costs memory, and all the more since many perl modules
>> are not
>> > thread-safe, so if you use them in your code, at this moment
>> the only safe way to
>> > do it is to use the Apache httpd prefork model. This means tha=
t
>> each Apache httpd
>> > child process has its own copy of the perl interpreter, which
>> means that the
>> > memory used by this embedded perl interpreter has to be counte=
d
>> n times (as many
>> > times as there are Apache httpd child processes running at any
>> one time).
>> >
>> > This isn=E2=80=99t quite true - if you load modules before the=
process
>> forks then they can
>> > cleverly share the same parts of memory. It is useful to be
>> able to "pre-load"
>> > core functionality which is used across all functions {this is
>> the case in Linux
>> > anyway}. It also speeds up child process generation as the
>> modules are already in
>> > memory and converted to byte code.
>> >
>> > One of the great advantages of mod_perl is Apache2::SizeLimit
>> which can blow away
>> > large child process - and then if needed create new ones. This
>> is not the case
>> > with some of the FCGI solutions as the individual processes ca=
n
>> grow if there is a
>> > memory leak or a request that retrieves a large amount of
>> content (even if not
>> > served), but perl can't give the memory back. So FCGI processe=
s
>> only get bigger
>> > and bigger and eventually blow up memory (or hit swap first)
>> >
>> >
>> >
>> >
>> >
>> > --
>> > The Wellcome Sanger Institute is operated by Genome Research
>> Limited, a charity
>> > registered in England with number 1021457 and a company
>> registered in England
>> > with number 2742969, whose registered office is 215 Euston
>> Road, London, NW1
>> =3Dgmail&source=3Dg>
>> 2
>> > [google.com]
>> > <
>> https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_ma=
ps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-=
26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=
=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=
=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D
>> >BE.
>> >
>> >
>> >
>> > --
>> > The Wellcome Sanger Institute is operated by Genome Research
>> > Limited, a charity registered in England with number 1021457
>> and a
>> > company registered in England with number 2742969, whose
>> registered
>> > office is 215 Euston Road, London, NW1
>> =3Dgmail&source=3Dg>
>> 2 [google.com]
>> > <
>> https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_ma=
ps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-=
26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=
=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=
=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D
>> >BE.
>> >
>> > -- The Wellcome Sanger Institute is operated by Genome Research
>> Limited, a charity
>> > registered in England with number 1021457 and a company registered
>> in England with
>> > number 2742969, whose registered office is 215 Euston Road,
>> London, NW1 2BE
>> ry=3Dgmail&source=3Dg>
>> .
>> >
>> > -- The Wellcome Sanger Institute is operated by Genome Research
>> Limited, a charity
>> > registered in England with number 1021457 and a company registered in
>> England with number
>> > 2742969, whose registered office is 215 Euston Road, London, NW1 2BE
>> ry=3Dgmail&source=3Dg>
>> .
>>
>>
--0000000000008a8b4d05b75972a1
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
If I have been using modperl for development, does it inf=
luence I drive Maserati? :) quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
solid;padding-left:1ex">
unsubscribe.
ass=3D"gmail_quote"> ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px=
solid rgb(204,204,204);padding-left:1ex">Hello James.
Bravo and many thanks for this excellent overview of your activities. Of co=
urse the setup
(in your previous message) and the activities are very impressive by themse=
lves.
But in addition, even though your message is not in itself a perl advocacy =
message, I feel
that it would have its right place in some perl/mod_perl advocacy forum, be=
cause it
touches on some general idea which are valid /also/ for perl and mod_perl.<=
br>
It was very refreshing to read for once a clear expos=C3=A9 of why it is st=
ill important
nowadays to think before programming, to program efficiently, and to choose=
the right tool
for the job at hand (be it perl, mod_perl, or any other) without the kind o=
f off-the-cuff
general a-priori which tend to plague these discussions.
And even though our own (commercial) activities and setups do not have anyt=
hing even close
to the scope which you describe, I would like to say that the same basic pr=
inciples which
you mention in your expos=C3=A9 are just as valid when you scale-down as wh=
en you scale-up.
("--you can=E2=80=99t just throw memory, CPUs, power at a problem =E2=
=80=93 you have to
think =E2=80=93 how can I do what I need to do with the least resources..&q=
uot;)
Even when you think of a single server, or a single server rack, at any one=
period in time
there is always a practical limit as to how much memory or CPUs you can fit=
in a given
server, or how many servers you can fit in a rack, or how many additional G=
b of bandwidth
you can allocate per server, beyond which there is a sudden "quantum j=
ump" as to how
practical and cost-effective a whole project becomes.
In that sense, I particulary enjoyed your examples of the database and of t=
he additional
power line.
On 24.12.2020 02:38, James Smith wrote:
> We don=E2=80=99t use perl for everything, yes we use it for web data, =
yes we still use it as the
> glue language in a lot of cases, the most complex stuff is done with C=
(not even C++ as
> that is too slow). Others on site use Python, Java, Rust, Go, PHP, alo=
ng with looking at
> using GPUs in cases where code can be highly parallelised
>
> It is not just one application =E2=80=93 but many, many applications=
=E2=80=A6 All with a common goal of
> understanding the human genome, and using it to assist in developing n=
ew understanding and
> techniques which can advance health care.
>
> We are a very large sequencing centre (one of the largest in the world=
) =E2=80=93 what I was
> pointing out is that you can=E2=80=99t just throw memory, CPUs, power =
at a problem =E2=80=93 you have to
> think =E2=80=93 how can I do what I need to do with the least resource=
s. Rather than what
> resources can I throw at the problem.
>
> Currently we are acting as the central repository for all COVID-19 seq=
uencing in the UK,
> along with one of the largest =E2=80=9Cwet=E2=80=9D labs sequencing da=
ta for it =E2=80=93 and that is half the
> sequenced samples in the whole world. The UK is sequencing more COVID-=
19 genomes a day
> than most other countries have sequenced since the start of the pandem=
ic in Feb/Mar. This
> has lead to us discovering a new more transmissible version of the vir=
us, and it what part
> of the country the different strains are present =E2=80=93 no other co=
untry in the world has the
> information, technology or infrastructure in place to achieve this.>
>
> But this is just a small part of the genomic sequencing we are looking=
at =E2=80=93 we work on:
> * other pathogens =E2=80=93 e.g. Plasmodium (Malaria);
> * cancer genomes (and how effective drugs are);
> * are a major part of the Human Cell Atlas which is looking at how the=
expression of genes
> (in the simplest terms which ones are switched on and switched off) ar=
e different in
> different tissues;
> * sequencing the genomes of other animals to understand their evolutio=
n;
> * and looking at some other species in detail, to see what we can lear=
n from them when
> they have defective genes;
>
> Although all these are currently scaled back so that we can work relen=
tlessly to support
> the medical teams and other researchers get on top of COVID-19.
>
> What is interesting is that many of the developers we have on campus (=
well all wfh at the
> moment) are all (relatively) old as we learnt to develop code on machi=
nes with limited CPU
> and limited memory =E2=80=93 so that things had to be efficient, had t=
o be compact=E2=80=A6. And that is
> as important now as it was 20 or 30 years ago =E2=80=93 the data we ha=
ndle is going up faster than
> Moore=E2=80=99s Law! Many of us have pride in doing things as efficien=
tly as possible.
>
> It took around 10 years to sequence and assemble the first human genom=
e {well we are still
> tinkering with it and filling in the gaps} =E2=80=93 now at the instit=
ute we can sequence and
> assemble around 400 human genomes in a day =E2=80=93 to the same quali=
ty!
>
> So most of our issues are due to the scale of the problems we face =E2=
=80=93 e.g. the human genome
> has 3 billion base-pairs (A, C, G, Ts) , so normal solutions don=E2=80=
=99t scale to that (once
> many years ago we looked at setting up an Oracle database where there =
was at least 1 row
> for every base pair =E2=80=93 recording all variants (think of them as=
spelling mistakes, for
> example a T rather than an A, or an extra letter inserted or deleted) =
for that base pair=E2=80=A6
> The schema was set up =E2=80=93 and then they realised it would take 1=
2 months to load the data
> which we had then (which is probably less than a millionth of what we =
have now)!
>
> Moving compute off site is a problem as the transfer of the level of d=
ata we have would
> cause a problem =E2=80=93 you can=E2=80=99t easily move all the data t=
o the compute =E2=80=93 so you have to bring
> the compute to the data.
>
> The site I worked on before I became a more general developer was doin=
g that =E2=80=93 and the
> code that was written 12-15 years ago is actually still going strong =
=E2=80=93 it has seen a few
> changes over the year =E2=80=93 many displays have had to be redevelop=
ed as the scale of the data
> has got so big that even the summary pages we produced 10 years ago ha=
ve to be summarised
> because they are so large.
>
> *From:*Mithun Bhattacharya <get=3D"_blank">mithnb-at-gmail.com>
> *Sent:* 24 December 2020 00:06
> *To:* mod_perl list <get=3D"_blank">modperl-at-perl.apache.org>
> *Subject:* Re: Confused about two development utils [EXT]
>
> James would you be able to share more info about your setup ?
>
> 1. What exactly is your application doing which requires so much memor=
y and CPU - is it
> something like gene splicing (no i don't know much about it beyond=
Jurassic Park :D )
>
> 2. Do you feel Perl was the best choice for whatever you are doing and=
if yes then why ?
> How much of your stuff is using mod_perl considering you mentioned not=
much is web related ?
>
> 3. What are the challenges you are currently facing with your implemen=
tation ?
>
> On Wed, Dec 23, 2020 at 6:58 AM James Smith <sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk <mailto:mailto:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk>> >
> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Oh but memory is a problem =E2=80=93 but not if you=
have just a small cluster of machines!
>
>=C2=A0 =C2=A0 =C2=A0Our boxes are larger than that =E2=80=93 but they a=
ll run virtual machine {only a small
>=C2=A0 =C2=A0 =C2=A0proportion web related} =E2=80=93 machines/memory w=
ould rapidly become in our data centre - we
>=C2=A0 =C2=A0 =C2=A0run VMWARE [995 hosts] and openstack [10,000s of ho=
sts] + a selection of large memory
>=C2=A0 =C2=A0 =C2=A0machines {measured in TBs of memory per machine }.<=
br>
>
>=C2=A0 =C2=A0 =C2=A0We would be looking at somewhere between 0.5 PB and=
1 PB of memory =E2=80=93 not just the
>=C2=A0 =C2=A0 =C2=A0price of buying that amount of memory - for many ma=
chines we need the fastest memory
>=C2=A0 =C2=A0 =C2=A0money can buy for the workload, but we would need a=
lot more CPUs then we currently
>=C2=A0 =C2=A0 =C2=A0have as we would need a larger amount of machines t=
o have 64GB virtual machines {we
>=C2=A0 =C2=A0 =C2=A0would get 2 VMs per host. We currently have approx.=
1-2000 CPUs running our hardware
>=C2=A0 =C2=A0 =C2=A0(last time I had a figure) =E2=80=93 it would proba=
bly need to go to approximately 5-10,000!
>=C2=A0 =C2=A0 =C2=A0It is not just the initial outlay but the environme=
ntal and financial cost of running
>=C2=A0 =C2=A0 =C2=A0that number of machines, and finding space to run t=
hem without putting the cooling
>=C2=A0 =C2=A0 =C2=A0costs through the roof!! That is without considerin=
g what additional constraints on
>=C2=A0 =C2=A0 =C2=A0storage having the extra machines may have (at the =
last count a year ago we had over
>=C2=A0 =C2=A0 =C2=A030 PBytes of storage on side =E2=80=93 and a large =
amount of offsite backup.
>
>=C2=A0 =C2=A0 =C2=A0We would also stretch the amount of power we can ge=
t from the national grid to power
>=C2=A0 =C2=A0 =C2=A0it all - we currently have 3 feeds from different p=
art of the national grid (we are
>=C2=A0 =C2=A0 =C2=A0fortunately in position where this is possible) and=
the dedicated link we would need
>=C2=A0 =C2=A0 =C2=A0to add more power would be at least 50 miles long!<=
br>
>
>=C2=A0 =C2=A0 =C2=A0So - managing cores/memory is vitally important to =
us =E2=80=93 moving to the cloud is an
>=C2=A0 =C2=A0 =C2=A0option we are looking at =E2=80=93 but that is more=
than 4 times the price of our onsite
>=C2=A0 =C2=A0 =C2=A0set-up (with substantial discounts from AWS) and wo=
uld require an upgrade of our
>=C2=A0 =C2=A0 =C2=A0existing link to the internet =E2=80=93 which is cu=
rrently 40Gbit of data (I think).
>
>=C2=A0 =C2=A0 =C2=A0Currently we are analysing a very large amounts of =
data directly linked to the current
>=C2=A0 =C2=A0 =C2=A0major world problem =E2=80=93 this is why the UK is=
currently being isolated as we have
>=C2=A0 =C2=A0 =C2=A0discovered and can track a new strain, in near real=
time =E2=80=93 other countries have no
>=C2=A0 =C2=A0 =C2=A0ability to do this =E2=80=93 we in a day can and do=
handle, sequence and analyse more samples
>=C2=A0 =C2=A0 =C2=A0than the whole of France has sequenced since Februa=
ry. We probably don=E2=80=99t have more of
>=C2=A0 =C2=A0 =C2=A0the new variant strain than in other areas of the w=
orld =E2=80=93 it is just that we know we
>=C2=A0 =C2=A0 =C2=A0have because of the amount of sequencing and analys=
is that we in the UK have done.
>
>=C2=A0 =C2=A0 =C2=A0*From:*Matthias Peng <hias-at-gmail.com" target=3D"_blank">pengmatthias-at-gmail.com <mailto:href=3D"mailto:pengmatthias-at-gmail.com" target=3D"_blank">pengmatthias-at-gmail=
.com>>
>=C2=A0 =C2=A0 =C2=A0*Sent:* 23 December 2020 12:02
>=C2=A0 =C2=A0 =C2=A0*To:* mod_perl list <erl.apache.org" target=3D"_blank">modperl-at-perl.apache.org <mailto: href=3D"mailto:modperl-at-perl.apache.org" target=3D"_blank">modperl-at-perl.apa=
che.org>>
>=C2=A0 =C2=A0 =C2=A0*Subject:* Re: Confused about two development utils=
[EXT]
>
>=C2=A0 =C2=A0 =C2=A0Today memory is not serious problem, each of our se=
rver has 64GB memory.
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Forgot to add - so our FCGI servers n=
eed a lot (and I mean a lot) more memory than
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0the mod_perl servers to serve the sam=
e level of content (just in case memory blows
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0up with FCGI backends)
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-----Original Message-----
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0From: James Smith <to:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk <mailto:ref=3D"mailto:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk>&=
gt;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sent: 23 December 2020 11:34
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0To: Andr=C3=A9 Warnier (tomcat/perl) =
<aw-at-ice-sa.com &l=
t;mailto:aw-at-ice-sa.coma>>>;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.org" target=3D"_blank">modperl-at-perl.apache.org <mailto:ailto:modperl-at-perl.apache.org" target=3D"_blank">modperl-at-perl.apache.org>>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Subject: RE: Confused about two devel=
opment utils [EXT]
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > This costs memory, and all the =
more since many perl modules are not
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0thread-safe, so if you use them in yo=
ur code, at this moment the only safe way to
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0do it is to use the Apache httpd pref=
ork model. This means that each Apache httpd
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0child process has its own copy of the=
perl interpreter, which means that the
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory used by this embedded perl int=
erpreter has to be counted n times (as many
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0times as there are Apache httpd child=
processes running at any one time).
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0This isn=E2=80=99t quite true - if yo=
u load modules before the process forks then they can
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cleverly share the same parts of memo=
ry. It is useful to be able to "pre-load"
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0core functionality which is used acro=
ss all functions {this is the case in Linux
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0anyway}. It also speeds up child proc=
ess generation as the modules are already in
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory and converted to byte code.>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0One of the great advantages of mod_pe=
rl is Apache2::SizeLimit which can blow away
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0large child process - and then if nee=
ded create new ones. This is not the case
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with some of the FCGI solutions as th=
e individual processes can grow if there is a
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory leak or a request that retriev=
es a large amount of content (even if not
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0served), but perl can't give the =
memory back. So FCGI processes only get bigger
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0and bigger and eventually blow up mem=
ory (or hit swap first)
>
>
>
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The Wellcome Sanger Institute =
is operated by Genome Research=C2=A0 Limited, a charity
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0registered in England with number 102=
1457 and a=C2=A0 company registered in England
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with number 2742969, whose registered=
=C2=A0 office is oad,+London,+NW1?entry=3Dgmail&source=3Dg">215 Euston Road, London, NW1=
2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0["noreferrer" target=3D"_blank">google.com]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<ofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_maps_search_s-2B215-2BEusto=
n-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=3DDwMF=
aQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=3DoH2yp0ge1ecj=
4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=3DxU3F=
4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D" rel=3D"noreferrer" target=
=3D"_blank">https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.goog=
le.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry=
-3Dgmail-26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8u=
clZFI0SqQnqBo&r=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_o=
gNXEVR-4ixdkrhy5khQjA&s=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&a=
mp;e=3D>BE.
>
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The Wellcome Sanger Institute =
is operated by Genome Research
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Limited, a charity registered =
in England with number 1021457 and a
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0company registered in England =
with number 2742969, whose registered
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0office is ww.google.com/maps/search/215+Euston+Road,+London,+NW1?entry=3Dgmail&so=
urce=3Dg">215 Euston Road, London, NW1 2 [ rel=3D"noreferrer" target=3D"_blank">google.com]
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<ofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_maps_search_s-2B215-2BEusto=
n-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=3DDwMF=
aQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=3DoH2yp0ge1ecj=
4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=3DxU3F=
4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D" rel=3D"noreferrer" target=
=3D"_blank">https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.goog=
le.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry=
-3Dgmail-26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8u=
clZFI0SqQnqBo&r=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_o=
gNXEVR-4ixdkrhy5khQjA&s=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&a=
mp;e=3D>BE.
>
>=C2=A0 =C2=A0 =C2=A0-- The Wellcome Sanger Institute is operated by Gen=
ome Research Limited, a charity
>=C2=A0 =C2=A0 =C2=A0registered in England with number 1021457 and a com=
pany registered in England with
>=C2=A0 =C2=A0 =C2=A0number 2742969, whose registered office is =3D"https://www.google.com/maps/search/215+Euston+Road,+London,+NW1+2BE?ent=
ry=3Dgmail&source=3Dg">215 Euston Road, London, NW1 2BE.
>
> -- The Wellcome Sanger Institute is operated by Genome Research Limite=
d, a charity
> registered in England with number 1021457 and a company registered in =
England with number
> 2742969, whose registered office is maps/search/215+Euston+Road,+London,+NW1+2BE?entry=3Dgmail&source=3Dg">=
215 Euston Road, London, NW1 2BE.
--0000000000008a8b4d05b75972a1--
--===============0059712626==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Hangout mailing list
Hangout-at-nylxs.com
http://lists.mrbrklyn.com/mailman/listinfo/hangout
--===============0059712626==--
|
|
