0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hell=
o James.

Bravo and many thanks for this excellent overview of your activities. Of co=
urse the setup

(in your previous message) and the activities are very impressive by themse=
lves.

But in addition, even though your message is not in itself a perl advocacy =
message, I feel

that it would have its right place in some perl/mod_perl advocacy forum, be=
cause it

touches on some general idea which are valid /also/ for perl and mod_perl.<=
br>
It was very refreshing to read for once a clear expos=C3=A9 of why it is st=
ill important

nowadays to think before programming, to program efficiently, and to choose=
the right tool

for the job at hand (be it perl, mod_perl, or any other) without the kind o=
f off-the-cuff

general a-priori which tend to plague these discussions.



And even though our own (commercial) activities and setups do not have anyt=
hing even close

to the scope which you describe, I would like to say that the same basic pr=
inciples which

you mention in your expos=C3=A9 are just as valid when you scale-down as wh=
en you scale-up.

("--you can=E2=80=99t just throw memory, CPUs, power at a problem =E2=
=80=93 you have to

think =E2=80=93 how can I do what I need to do with the least resources..&q=
uot;)

Even when you think of a single server, or a single server rack, at any one=
period in time

there is always a practical limit as to how much memory or CPUs you can fit=
in a given

server, or how many servers you can fit in a rack, or how many additional G=
b of bandwidth

you can allocate per server, beyond which there is a sudden "quantum j=
ump" as to how

practical and cost-effective a whole project becomes.

In that sense, I particulary enjoyed your examples of the database and of t=
he additional

power line.





On 24.12.2020 02:38, James Smith wrote:

> We don=E2=80=99t use perl for everything, yes we use it for web data, =
yes we still use it as the

> glue language in a lot of cases, the most complex stuff is done with C=
(not even C++ as

> that is too slow). Others on site use Python, Java, Rust, Go, PHP, alo=
ng with looking at

> using GPUs in cases where code can be highly parallelised

>

> It is not just one application =E2=80=93 but many, many applications=
=E2=80=A6 All with a common goal of

> understanding the human genome, and using it to assist in developing n=
ew understanding and

> techniques which can advance health care.

>

> We are a very large sequencing centre (one of the largest in the world=
) =E2=80=93 what I was

> pointing out is that you can=E2=80=99t just throw memory, CPUs, power =
at a problem =E2=80=93 you have to

> think =E2=80=93 how can I do what I need to do with the least resource=
s. Rather than what

> resources can I throw at the problem.

>

> Currently we are acting as the central repository for all COVID-19 seq=
uencing in the UK,

> along with one of the largest =E2=80=9Cwet=E2=80=9D labs sequencing da=
ta for it =E2=80=93 and that is half the

> sequenced samples in the whole world. The UK is sequencing more COVID-=
19 genomes a day

> than most other countries have sequenced since the start of the pandem=
ic in Feb/Mar. This

> has lead to us discovering a new more transmissible version of the vir=
us, and it what part

> of the country the different strains are present =E2=80=93 no other co=
untry in the world has the

> information, technology or infrastructure in place to achieve this.>
>

> But this is just a small part of the genomic sequencing we are looking=
at =E2=80=93 we work on:

> * other pathogens =E2=80=93 e.g. Plasmodium (Malaria);

> * cancer genomes (and how effective drugs are);

> * are a major part of the Human Cell Atlas which is looking at how the=
expression of genes

> (in the simplest terms which ones are switched on and switched off) ar=
e different in

> different tissues;

> * sequencing the genomes of other animals to understand their evolutio=
n;

> * and looking at some other species in detail, to see what we can lear=
n from them when

> they have defective genes;

>

> Although all these are currently scaled back so that we can work relen=
tlessly to support

> the medical teams and other researchers get on top of COVID-19.

>

> What is interesting is that many of the developers we have on campus (=
well all wfh at the

> moment) are all (relatively) old as we learnt to develop code on machi=
nes with limited CPU

> and limited memory =E2=80=93 so that things had to be efficient, had t=
o be compact=E2=80=A6. And that is

> as important now as it was 20 or 30 years ago =E2=80=93 the data we ha=
ndle is going up faster than

> Moore=E2=80=99s Law! Many of us have pride in doing things as efficien=
tly as possible.

>

> It took around 10 years to sequence and assemble the first human genom=
e {well we are still

> tinkering with it and filling in the gaps} =E2=80=93 now at the instit=
ute we can sequence and

> assemble around 400 human genomes in a day =E2=80=93 to the same quali=
ty!

>

> So most of our issues are due to the scale of the problems we face =E2=
=80=93 e.g. the human genome

> has 3 billion base-pairs (A, C, G, Ts) , so normal solutions don=E2=80=
=99t scale to that (once

> many years ago we looked at setting up an Oracle database where there =
was at least 1 row

> for every base pair =E2=80=93 recording all variants (think of them as=
spelling mistakes, for

> example a T rather than an A, or an extra letter inserted or deleted) =
for that base pair=E2=80=A6

> The schema was set up =E2=80=93 and then they realised it would take 1=
2 months to load the data

> which we had then (which is probably less than a millionth of what we =
have now)!

>

> Moving compute off site is a problem as the transfer of the level of d=
ata we have would

> cause a problem =E2=80=93 you can=E2=80=99t easily move all the data t=
o the compute =E2=80=93 so you have to bring

> the compute to the data.

>

> The site I worked on before I became a more general developer was doin=
g that =E2=80=93 and the

> code that was written 12-15 years ago is actually still going strong =
=E2=80=93 it has seen a few

> changes over the year =E2=80=93 many displays have had to be redevelop=
ed as the scale of the data

> has got so big that even the summary pages we produced 10 years ago ha=
ve to be summarised

> because they are so large.

>

> *From:*Mithun Bhattacharya <get=3D"_blank">mithnb-at-gmail.com>

> *Sent:* 24 December 2020 00:06

> *To:* mod_perl list <get=3D"_blank">modperl-at-perl.apache.org>

> *Subject:* Re: Confused about two development utils [EXT]

>

> James would you be able to share more info about your setup ?

>

> 1. What exactly is your application doing which requires so much memor=
y and CPU - is it

> something like gene splicing (no i don't know much about it beyond=
Jurassic Park :D )

>

> 2. Do you feel Perl was the best choice for whatever you are doing and=
if yes then why ?

> How much of your stuff is using mod_perl considering you mentioned not=
much is web related ?

>

> 3. What are the challenges you are currently facing with your implemen=
tation ?

>

> On Wed, Dec 23, 2020 at 6:58 AM James Smith <sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk <mailto:mailto:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk>> >
> wrote:

>

>=C2=A0 =C2=A0 =C2=A0Oh but memory is a problem =E2=80=93 but not if you=
have just a small cluster of machines!

>

>=C2=A0 =C2=A0 =C2=A0Our boxes are larger than that =E2=80=93 but they a=
ll run virtual machine {only a small

>=C2=A0 =C2=A0 =C2=A0proportion web related} =E2=80=93 machines/memory w=
ould rapidly become in our data centre - we

>=C2=A0 =C2=A0 =C2=A0run VMWARE [995 hosts] and openstack [10,000s of ho=
sts] + a selection of large memory

>=C2=A0 =C2=A0 =C2=A0machines {measured in TBs of memory per machine }.<=
br>
>

>=C2=A0 =C2=A0 =C2=A0We would be looking at somewhere between 0.5 PB and=
1 PB of memory =E2=80=93 not just the

>=C2=A0 =C2=A0 =C2=A0price of buying that amount of memory - for many ma=
chines we need the fastest memory

>=C2=A0 =C2=A0 =C2=A0money can buy for the workload, but we would need a=
lot more CPUs then we currently

>=C2=A0 =C2=A0 =C2=A0have as we would need a larger amount of machines t=
o have 64GB virtual machines {we

>=C2=A0 =C2=A0 =C2=A0would get 2 VMs per host. We currently have approx.=
1-2000 CPUs running our hardware

>=C2=A0 =C2=A0 =C2=A0(last time I had a figure) =E2=80=93 it would proba=
bly need to go to approximately 5-10,000!

>=C2=A0 =C2=A0 =C2=A0It is not just the initial outlay but the environme=
ntal and financial cost of running

>=C2=A0 =C2=A0 =C2=A0that number of machines, and finding space to run t=
hem without putting the cooling

>=C2=A0 =C2=A0 =C2=A0costs through the roof!! That is without considerin=
g what additional constraints on

>=C2=A0 =C2=A0 =C2=A0storage having the extra machines may have (at the =
last count a year ago we had over

>=C2=A0 =C2=A0 =C2=A030 PBytes of storage on side =E2=80=93 and a large =
amount of offsite backup.

>

>=C2=A0 =C2=A0 =C2=A0We would also stretch the amount of power we can ge=
t from the national grid to power

>=C2=A0 =C2=A0 =C2=A0it all - we currently have 3 feeds from different p=
art of the national grid (we are

>=C2=A0 =C2=A0 =C2=A0fortunately in position where this is possible) and=
the dedicated link we would need

>=C2=A0 =C2=A0 =C2=A0to add more power would be at least 50 miles long!<=
br>
>

>=C2=A0 =C2=A0 =C2=A0So - managing cores/memory is vitally important to =
us =E2=80=93 moving to the cloud is an

>=C2=A0 =C2=A0 =C2=A0option we are looking at =E2=80=93 but that is more=
than 4 times the price of our onsite

>=C2=A0 =C2=A0 =C2=A0set-up (with substantial discounts from AWS) and wo=
uld require an upgrade of our

>=C2=A0 =C2=A0 =C2=A0existing link to the internet =E2=80=93 which is cu=
rrently 40Gbit of data (I think).

>

>=C2=A0 =C2=A0 =C2=A0Currently we are analysing a very large amounts of =
data directly linked to the current

>=C2=A0 =C2=A0 =C2=A0major world problem =E2=80=93 this is why the UK is=
currently being isolated as we have

>=C2=A0 =C2=A0 =C2=A0discovered and can track a new strain, in near real=
time =E2=80=93 other countries have no

>=C2=A0 =C2=A0 =C2=A0ability to do this =E2=80=93 we in a day can and do=
handle, sequence and analyse more samples

>=C2=A0 =C2=A0 =C2=A0than the whole of France has sequenced since Februa=
ry. We probably don=E2=80=99t have more of

>=C2=A0 =C2=A0 =C2=A0the new variant strain than in other areas of the w=
orld =E2=80=93 it is just that we know we

>=C2=A0 =C2=A0 =C2=A0have because of the amount of sequencing and analys=
is that we in the UK have done.

>

>=C2=A0 =C2=A0 =C2=A0*From:*Matthias Peng <hias-at-gmail.com" target=3D"_blank">pengmatthias-at-gmail.com <mailto:href=3D"mailto:pengmatthias-at-gmail.com" target=3D"_blank">pengmatthias-at-gmail=
.com>>

>=C2=A0 =C2=A0 =C2=A0*Sent:* 23 December 2020 12:02

>=C2=A0 =C2=A0 =C2=A0*To:* mod_perl list <erl.apache.org" target=3D"_blank">modperl-at-perl.apache.org <mailto: href=3D"mailto:modperl-at-perl.apache.org" target=3D"_blank">modperl-at-perl.apa=
che.org>>

>=C2=A0 =C2=A0 =C2=A0*Subject:* Re: Confused about two development utils=
[EXT]

>

>=C2=A0 =C2=A0 =C2=A0Today memory is not serious problem, each of our se=
rver has 64GB memory.

>

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Forgot to add - so our FCGI servers n=
eed a lot (and I mean a lot) more memory than

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0the mod_perl servers to serve the sam=
e level of content (just in case memory blows

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0up with FCGI backends)

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-----Original Message-----

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0From: James Smith <to:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk <mailto:ref=3D"mailto:js5-at-sanger.ac.uk" target=3D"_blank">js5-at-sanger.ac.uk>&=
gt;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Sent: 23 December 2020 11:34

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0To: Andr=C3=A9 Warnier (tomcat/perl) =
<aw-at-ice-sa.com &l=
t;mailto:aw-at-ice-sa.coma>>>;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.org" target=3D"_blank">modperl-at-perl.apache.org <mailto:ailto:modperl-at-perl.apache.org" target=3D"_blank">modperl-at-perl.apache.org>>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Subject: RE: Confused about two devel=
opment utils [EXT]

>

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > This costs memory, and all the =
more since many perl modules are not

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0thread-safe, so if you use them in yo=
ur code, at this moment the only safe way to

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0do it is to use the Apache httpd pref=
ork model. This means that each Apache httpd

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0child process has its own copy of the=
perl interpreter, which means that the

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory used by this embedded perl int=
erpreter has to be counted n times (as many

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0times as there are Apache httpd child=
processes running at any one time).

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0This isn=E2=80=99t quite true - if yo=
u load modules before the process forks then they can

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0cleverly share the same parts of memo=
ry. It is useful to be able to "pre-load"

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0core functionality which is used acro=
ss all functions {this is the case in Linux

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0anyway}. It also speeds up child proc=
ess generation as the modules are already in

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory and converted to byte code.>
>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0One of the great advantages of mod_pe=
rl is Apache2::SizeLimit which can blow away

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0large child process - and then if nee=
ded create new ones. This is not the case

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with some of the FCGI solutions as th=
e individual processes can grow if there is a

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0memory leak or a request that retriev=
es a large amount of content (even if not

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0served), but perl can't give the =
memory back. So FCGI processes only get bigger

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0and bigger and eventually blow up mem=
ory (or hit swap first)

>

>

>

>

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The Wellcome Sanger Institute =
is operated by Genome Research=C2=A0 Limited, a charity

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0registered in England with number 102=
1457 and a=C2=A0 company registered in England

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0with number 2742969, whose registered=
=C2=A0 office is 215 Euston Road, London, NW1 2

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0["noreferrer" target=3D"_blank">google.com]

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<ofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_maps_search_s-2B215-2BEusto=
n-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=3DDwMF=
aQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=3DoH2yp0ge1ecj=
4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=3DxU3F=
4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D" rel=3D"noreferrer" target=
=3D"_blank">https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.goog=
le.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry=
-3Dgmail-26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8u=
clZFI0SqQnqBo&r=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_o=
gNXEVR-4ixdkrhy5khQjA&s=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&a=
mp;e=3D>BE.

>

>

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0--

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The Wellcome Sanger Institute =
is operated by Genome Research

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Limited, a charity registered =
in England with number 1021457 and a

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0company registered in England =
with number 2742969, whose registered

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0office is 215 Euston Road, Lon=
don, NW1 2 [nk">google.com]

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<ofpoint.com/v2/url?u=3Dhttps-3A__www.google.com_maps_search_s-2B215-2BEusto=
n-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry-3Dgmail-26source-3Dg&d=3DDwMF=
aQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=3DoH2yp0ge1ecj=
4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_ogNXEVR-4ixdkrhy5khQjA&s=3DxU3F=
4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&e=3D" rel=3D"noreferrer" target=
=3D"_blank">https://urldefense.proofpoint.com/v2/url?u=3Dhttps-3A__www.goog=
le.com_maps_search_s-2B215-2BEuston-2BRoad-2C-2BLondon-2C-2BNW1-2B2-3Fentry=
-3Dgmail-26source-3Dg&d=3DDwMFaQ&c=3DD7ByGjS34AllFgecYw0iC6Zq7qlm8u=
clZFI0SqQnqBo&r=3DoH2yp0ge1ecj4oDX0XM7vQ&m=3DfriR8ykiZ-NWYdX6SrbT_o=
gNXEVR-4ixdkrhy5khQjA&s=3DxU3F4xE2ugQuDWHZ4GtDn9mPBCKcJJOI0PYScsSNjSg&a=
mp;e=3D>BE.

>

>=C2=A0 =C2=A0 =C2=A0-- The Wellcome Sanger Institute is operated by Gen=
ome Research Limited, a charity

>=C2=A0 =C2=A0 =C2=A0registered in England with number 1021457 and a com=
pany registered in England with

>=C2=A0 =C2=A0 =C2=A0number 2742969, whose registered office is 215 Eust=
on Road, London, NW1 2BE.

>

> -- The Wellcome Sanger Institute is operated by Genome Research Limite=
d, a charity

> registered in England with number 1021457 and a company registered in =
England with number

> 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.<=
br>