foreign
like to welcome you to the third session
in this uh this block of talk so Nick
Moore will be talking about
bioinformatics and his counters project
and Nick Moore has been a contributor to
micro Python and has spoken at several
conferences before including a pycon and
then it's called then it's conf Au so
please uh give a warm welcome to Nick
hi everyone
um yeah good day my name's Nick Moore
I've previously talked at conferences
like this about things like micropython
um and about Django and things like that
and now I'm currently working in
bioinformatics
um if you're considering changing around
fields in this way I recommend it's a
great way to constantly feel imposter
syndrome for your entire life it's great
but it's a really interesting project
and I hope you'll find a lot of fun like
I'm finding it a lot of fun
um
I'm going to start the talk by
introducing some Concepts about
bioinformatics just so you have some
idea of what the hell I'm talking about
um this work is funded by Walter and
Eliza Hall Institute
um
and also I'd like to thank uh some of my
collaborators from Brooklyn Betty
um Institute and the University of
Washington genome Sciences uh who are
helping out because I'm a software
developer not a scientist certainly not
a bioinformatician so the only way to
make a project like this work is to be
constantly working hand in hand with
people who actually do the science
um content warnings on this presentation
we briefly mentioned cancer we briefly
mentioned covid-19 we do not go into any
kind of gross medical details this is
about numbers
um not about the actual diseases
themselves
um
so bioinformatics in nine and a half
minutes this will be cool
um we'll move through it fairly quickly
uh
there'll be a chance to chat about this
stuff over lunch or whatever I've got
the green say hi sticker and I really
mean it if you want to talk about this
stuff all lunchtime we can talk about
this stuff also my colleague Ellen's
sitting right there so you can bug him
about bioinformatics things too
so approximately nine and a half minutes
or we're going to go through fairly
approximately about bioinformatics
starting with what the hell is
bioinformatics it's the analysis of
biological data so we get lots and lots
of data from biological processes how do
we analyze them arguably it got started
in about 1952
when we first realized you could
sequence proteins you could break
proteins down to their building blocks
and sequence them and so Albert Sanger
sequenced
insulin so we actually properly
understood how insulin worked in the
body etc etc and Alan Turing of all
people decided to take some time off for
inventing Computing to also invent
bioinformatics just in his spare time
in a paper that that kind of derived the
chemical basis of morphogenesis
before anyone you have any of the actual
Parts worked he'd already written the
paper that turned out to be largely
correct once we found out that we could
read the parts it's a pretty amazing
work it really got Computing once
computers got invented to do a lot of
the hard work for us Margaret day Huff
was a very early Pioneer of this stuff
writing software in Fortran to actually
align proteins or subsequences and work
out how they would all fit together and
really interestingly this protein
information resource thing is a very
early free online database 1984 and
people are already going hey information
wants to be free especially if it's a
health information it's really useful
information for researchers I think
that's an amazing thing perhaps
unsurprisingly given how close they are
in our time of science there's a lot of
parallels between biological sciences
and computer science and I'll talk a lot
about that here because a lot of your
software people
cell biology
there's a lot we still don't know about
cell biology even though we're made of
cells
um that's mostly because it happens
inside the cells and what happens inside
the cells stays inside the cells they
Once you pull them apart to find out how
they work they're generally dead and not
so interesting anymore
um but experiments let us make theories
about how they must work internally so
you can change the inputs you can see
what the cell does
generally dies but you know you can
measure that and make some conclusions
about how the cell must be working it's
a lot like black boss box testing of
software where you don't need to look
inside the box you can still test how
it's behaving just by changing the
inputs and monitoring the outputs
the human genome which I'm sure is a
term you've heard it's the program that
all of your cells run all of them run
the same program it's a bit like having
lots of things running the same
container image but different parts of
the code get run depending on what that
particular cell is doing
there's 23 chromosomes in there give or
take each of them is a very long
molecule like a long long long long tape
of 100 million or so building blocks
called nucleotides sat there's four of
those so they make up an enormous long
tape
eerily eerily like a turing machine when
it comes down to it
um all of you so I said that already all
of your cells have the same genome how
it's expressed depends on what the
cell's job is
um
these days we can actually read the DNA
there's a technique called nanopore
electrophoresis which basically sucks
the DNA molecule through a teeny tiny
hole a lot like a small child eating
spaghetti in a disgusting way
and it reads the molecule at one base at
a time and so we can actually read the
binary this is pretty exciting and using
some other techniques that you might
have heard of things like crispr and
things like that we can actually write
the binary we can hack little changes
into the binaries which is way cool so
we can read and write the binaries so of
course we understand everything
and not so much a bit like in software
reading and writing the binaries is
great but it's not the only thing we do
know a bit about the syntax of the
genome
genes code for proteins they're wrapped
in kind of a header and a footer it's
getting Eerie isn't it that sort of
regulate how much they get run and
things like that some of them are tiny
they have one little protein hundreds of
base pairs in the binary
other one of them's a huge they have
dozens of proteins millions of bases
they're like those enormous classes you
see in some bits of software
um
and we know that they're sort of wrapped
up in this way within the chromosomes
there are no debug symbols and no
comments it's very inconvenient
um what we can do is we can look at the
genes and we can go we think we know
what this does we'll give it a name we
can just assign them a name it's very
much like what you do if you're ever
reverse engineering a binary you you go
through with a pencil if you're very old
or with guidra if you're rather younger
and you assign a thing and you go this
function I think does this so I'll give
it a name
um
we know of about 20 000 genes that
encode proteins we think we know what
they do
that totals about three percent of the
entire genome
we know from other stuff there's a
reference in my things which I'll send
you later that 80 of it at least does
something
so there's a fair way to go I think it's
fair to say if to to say we completely
understand the genome well no
people have said that we can download
the whole thing we don't necessarily
understand the whole thing
okay so I'm about to talk a lot about
Gene variants so let's just talk about
what that is
um
those little
symbols on the tape are called
nucleotides or bases
and they vary everyone's genome is
different there's this concept of a
reference genome which is like what is
normal
no one has that actual genome that's
just like a a reference it's a typical
genome
um at one point one person did have that
genome I think which was the guy who
started the Human Genome Project wasn't
it his personal sequence no anyway
um
so everyone's different you know in some
way or another the most common kind of
variant is just like a single base
change somewhere there flip a bit on and
off if it's a really common thing that
happens in people we call it a
polymorphism
um because it's you know many shapes if
it's rare we call it a variant because
it's a little bit more scary
some of these variants are really easily
observable
um infant lactose intolerance and adult
lactase Persistence of both variants in
LCT and mcm6 genes
um infant lactose intolerance we call a
variant because it's rare and it's scary
adult lactase persistence we call a
polymorphism because a lot of people
have that that's where you can still
drink milk as an adult
or hair color is another example of a
gene variant that we kind of understand
really well well enough that we can look
at the human genome recovered from a
5700 year old bit of chewing gum and
tell what color hair the woman who
chewed that gum had as well as their
skin color eye color and also she had
duck and I think it was chestnuts for
lunch because we can read all that DNA
out of a piece of 5700 year old chewing
gum
we've also found correlations between
people with severe covert symptoms and
long covert and a particular variant on
Fox P4 which is another Gene that seems
to be involved in lung stuff so that's
really interesting and might lead to
some understanding of how covert works
and how to help people with long covert
that's a correlation they found we don't
actually know necessarily why yet
there are more subtle problems as well
in cancers
um
cells are damaged by radiation or
chemicals or just bad luck
cell repair mechanisms then can either
fix that damage or kill the cell off if
it's unfixable
those cell repair mechanisms have genes
they're created by genes those genes
can have variants and some of those
variants can reduce the effectiveness of
those repair mechanisms
it's much harder to find these variants
or how important they are because these
are rare things in people's lives are
complicated and it's a little bit like
looking for a bug in an exception
Handler
in Live code that's already exposed to
the Internet it's very hard to simulate
this sort of stuff in in the real world
um
so which brings me to Multiplex assays
of variant effect and that's my rodent
of unusual size which is what the thing
always reminds me of
um
there's a few thousand known variants of
these sorts of cellular repair genes and
there's a lot more we don't know about
because we've never seen them clinically
they've never happened in a patient who
happened
to have a particular cancer of a type
that saw them go to the hospital and
have their genes sequenced and so on and
so forth we haven't seen them clinically
some of them might be serious a lot of
them are going to be just background
noise so they're not that important but
we want to know which ones are which
so Multiplex essays take a different
approach they say well let's make all
the possible variants let's get those
machines that can read and write the
binary like I was mentioning before and
we'll write all the possible variants
and we'll make an experiment on all the
possible variants and we'll measure
their their we'll sort of score them how
dangerous is this basically the outcome
is the same we want to know which
variants you should get worried about
and which variants you shouldn't because
that way you can tailor treatments to
people's individual
genetics
um
there's a resource that Alan talked
about on Friday called mavdb that exists
to store and and kind of retrieve these
sort of results sets so that people can
compare them and use them and so on and
so forth so look that one up on video if
you want that
I'll talk briefly about a couple of
techniques that are used for these
because they're important to the
software we're developing one of them is
a thing called saturation genome editing
this particular paper studies about 4
000 single base variants of brca1 looks
at the cell growth assessed after 11
days I think it is and has
a it basically tries all these different
possible variants and assesses how much
they have affected the cells it has good
correlation with the variants we know of
clinically so it looks like it's really
valuable work and it identifies some new
pathogenic variants some new dangerous
variants so that could be really really
important for for future medicine for
treatment if you happen to have one of
those variants that might be very
important to your life
this shows a little chart of the the
outcome of that study where you can see
that a huge number of these variants on
brca1 really this they have a score
around zero they didn't make much of a
difference
um there's a bunch more that have a very
bad score they've made a big difference
brca1 no longer worked sorry it says up
there LOF which means loss of function
um the brca1 cellular repair mechanism
doesn't work anymore and there's a bunch
of intermediate ones which is really
interesting those are things that that
um it works still but maybe not as well
as it should similar work is underway on
similar cellular repair mechanisms such
as brca1 and pal B2 which are a similar
kind of mechanisms and also very
important for cancer research
another technique I just want to talk
about briefly is a thing called vampseq
which is really cool I mostly want to
talk about it because it takes jellyfish
proteins and attaches them to a genome
basically you make cells of every
possible variant of a particular Gene
you attach a fluorescent marker protein
to that that um
Gene as well so it's expressed along
with the thing that you're doing and
then you see how those cells reproduce
the more the gene is expressed the more
the cell glows and then you actually
physically sort the cells into bins
based on how much they glow and then you
sequence each bin
um sorry then you sequence each bin and
you count the population of each
sequence and from that you can derive a
score
the important bit about this is you can
derive then a heat map of which which
variants wear on the gene matter the
most
um it's a very cost effective technique
because instead of running thousands of
separate
um
sort of little tests you're putting
putting things in big bins at a time
which makes it a little bit more
effective currently being used to infect
investigate some Factor 9 hemophilia
which is really interesting under
various conditions
um so that's a completely different
thing now you've probably noticed that
the technique here is completely
different to the technique I talked
about before SGA it's it's we have
different inputs we have different
outputs we have different maths scoring
is different Etc they're using some of
the same Tech
but there's some big differences
so the software I'm writing isn't about
that stuff at all my software I'm
running is about counting things now
computers are really good at counting
things
they've been really good at counting
things ever since they've got fed Punch
Cards and
um we had things called tabulating
machines and so on
they're really great at counting things
so this is really quite a simple problem
to solve obviously
um from the the paper I've just
mentioned on sge we have data for 13
exons over 163 files we have 350 million
sequences we have 66 billion bases in
those sequences which isn't a ridiculous
amount of data I mean it's it's a lot
but it's not ridiculous right but those
data sizes are only you know they're
increasing but it's not ridiculous as
such so obviously there's an easy
solution to this we just put them all in
a text oh sorry we put them all in a
text file and we run sort on them and we
run that through unique and that spits
out the counts it's done from soft
it's already optimized it runs on your
computer it's multi-core it's fine
you've already got it if you've got any
kind of Unix machine it's it'll work out
fine
um unfortunately the problem is humans
um the the actual problem isn't sorting
and Counting the actual problem is file
formats and metadata and filtering and
grouping and transformation and
statistics you have to do all of these
things to the numbers before and after
you count them statistics is
particularly important because you don't
want to do this where you find a
correlation because you're looking for a
correlation for anyone who can't see it
this is a correlation of the Internet
Explorer market share versus the murder
rate in the United States of America and
you can see they're very nicely
correlated if you craft them in the
right way
you don't want that to happen you need
to run tests as you look at this data to
make sure that you're seeing a real
effect and not some kind of statistical
correlation
previously a lot of this work was done
in spreadsheets there's some pretty
major problems with that for a start
we're heading up into the billions of
rows which doesn't play very well in
Excel also Excel has this real tendency
to want everything to be a date and
genes have names like I mentioned before
and some of those names are things like
sept2 and March 3 and
Excel thinks they're dates this paper
estimates that roughly 20 percent of
Cell Biology papers have errors
introduced by these kinds of things this
is it's a better update now but it's
it's a ridiculously large number also
just
any series of digits with one e in them
is turned into a floating Point number
thanks real helpful Excel
so spreadsheets are not a great thing
for this so a lot of people write
programs which is great python are very
popular to coordinate all of that stuff
you probably need some bash script as
well and maybe a little bit of a make
file yeah it's fine we can learn how to
write some make maybe snake make as well
we can coordinate all of these pieces
together and all you have to do is take
someone who's a newly minted cancer
research of keen to do cancer research
and send them off to learn Unix and
Python and R and all of this stuff for
like a year or so because they have got
nothing better to be doing with their
lives actually it will be a lot better
if we could just get them working on
cancer research straight away so
um
previously we've had a tool called
enrich 2 which is a GUI based tool that
people can feed these data files to have
them processed they get spat back out
again it's very nice it's GUI based it's
easy to use it's written in Python 2
um it's specific to some specific
experimental setups though and it's
difficult to kind of grow from where
enrich is to cover all of these
different kinds of experiment that
people were coming up with because
files are always increasing scientists
are always coming up with new kinds of
experiment to run that is part of the
point of doing science is you should be
able to say
I've got a new way to do this I want to
try my new way of running this
experiment
um we want to enable that
so we've started on this new project
called Countess count based experiment
scoring and statistics so as the name
suggests it's all about counting things
but it's also about the scoring and the
statistics and and how to support
multiple kinds of experiment
it's got a graphical user interface
because our users like to use programs
that look like programs not like
script
um
but it's plugin based so you can add and
remove modules from it really easily it
uses flexible data pipelines so the data
kind of flows through it neatly
and rather than having a whole kind of
bunch of here's my program and my other
program and my make file and calls this
and calls that it just has a single text
configuration file that binds all that
stuff together
um its architecture looks something like
this to try and keep all the modules a
bit separate there is a GUI
it's written in TK enter
I suspect it will not always be written
in TK enter because that has proved not
necessarily a great idea and as per
Russell's talked just before this there
are alternatives that are definitely
worth considering but it's very separate
from the rest of the code so I'm not
actually all that worried about that we
can come back and replace that later
as an overview of what it currently
looks like here is a screenshot of what
it currently looks like the nice thing
about tikanta it does actually work
nicely over X11 forwarding if you've got
to do that
um
the window is split into three parts
basically there's this tree of plugins
and how they modularly connect together
to flow data in and out of the system
on the left there things are loading
things up from files on the right
they're writing things out to files and
the data kind of flows through all those
modules on its way through it's fairly
drag and drop state is saved to
configuration files Etc
uh each plugin has a configuration
screen so you can just fill in the
blanks in the form all that's saved to
the config file and most essentially as
you're doing this stuff there's live
data previews of stuff not the whole
file
100 million rows or so at the moment
maybe we'll make that a little more
flexible as we go along but the idea is
that it it processes a bunch of rows so
that you can see what you're doing if
you're writing a regular expression you
get to see that live that you're making
the changes and that it's working which
is great there's also CLI which you can
run that just basically runs the same
config file but run it without having to
look at it which is useful if you want
to use some bigger computer somewhere
further away some of these runs to run
the whole file can take hours and hours
to run so it's kind of nice if you can
just SSH somewhere and do that
um
all right the config files are really
simple there that any file format that
you see everywhere because it's easy to
diff it works nicely in revision control
it's relatively simple you can hand it
if you have to it's loaded and saved by
the GUI it's loaded by the CLR module
all the fun stuff happens in here this
is the the pipeline code is basically
all about
coordinating multiple processes and the
flow of data through the system and all
this and all the tricky stuff happens in
there it's all hidden away nicely in
there
um it's a work in progress it can be
improved there's lots of work to do in
there
um but that's what it's aiming to do it
calls out to the plugins and makes the
plugins do things for it and the plugins
can be written in such a way that they
don't need to think about
multi-processing you do you write the
plugin in a way that does like a map and
a reduce for example and let the
pipeline worry about how to make that
actually happen
um
so there are basically three kinds
there's reading files there's altering
your data and writing files back they
all subclass one of the many subclasses
that are available that do all of that
kind of structural
stuff about this is a map reducy kind of
plug-in versus a very large number of
the plugins literally just take a row at
a time transform it in some way and spit
out another column into that table so if
you want to calculate a score for
example you might take account and
another count and you might divide one
by the other and take a log of that and
that will give you a score
you don't need to run that all in one
place all on one table or anything it
can run on many rows separately in
different places on many CPUs
concurrently so we allow that to happen
by not making the plugin writer
understand that they can just implement
the function that works on a row and not
worry about anything else so that's nice
anyone can write a plugin this is
actually one of the coolest things in
Python recently is this whole entry
points mechanism if you write a a little
Clause into your pup your plugins Pi
Project file
all it needs to do is say hey this has
an entry point it's a countess plugins
entry point
um
and you declare that basically here
here's my plugin when someone installs
that with Pip or whatever
Countess can actually see that that
plugin exists and it can can write it
the coding contest looks a little bit
like that it basically just goes through
all the entry points from that import
lid entry points function and it says
um
are these valid plugins do they actually
derive from that base plugin if they do
then we'll count them we'll make them
available to the user
and lastly the the library is underlying
all this stuff
um
so a lot of the stuff runs on pandas
because pandas is really handy for this
and numpy there are other libraries too
for example there's one called minimap 2
that I've wrapped up to be a contest
plugin
mini map 2 is an example of a thing that
runs per per row it takes a variant and
it turns it into a little kind of a diff
almost so you have a sequence and you
say I want to know where this sequence
comes from and and how it's different
from the reference genome so it could
run once per row and wrapping that up
takes a a dozen lines of code to just
basically say all right for each line
this function gets cold so this function
just needs to look up this particular
variant and compare it and stuff like
that
that's really good just as an example
I've got five minutes so this is perfect
as an example of Countess in in used in
Anger this is kind of loosely based on
that SGA work before
so the first
as you select the little plugins on the
left pane there it shows you their
config and their own individual output
on the right there so the first thing is
they are loading up all the fastq files
163 files get loaded and what it's
actually doing is grabbing the first
little bit out of each file to see have
a representative sample of that data and
you can see that data down in the bottom
sort of third of the screen over the
right there
so that shows you the actual contents of
those fast queue files and fast use a
particular file format used a lot in
this this thing
there's another plug in there that
literally is just extracting some data
from those columns that are in the fastq
file and altering it a little bit so
that we can use it later
along with those big fast queue files
comes a little metadata file called SRA
run table and there's some metadata in
there we need so we also load that up
it's an it's called SRA runtable.txt
it's a CSV file who's surprised so it
basically reads us there's a CSV reader
that reads the CSV file
and then we have to transform that a bit
because it's also not quite what we
wanted so we pulled some different
columns out of it that aren't actually
CSV columns they're just like embedded
within the text strings in the CSV and
now we can do like a database join of
these two data sources using a join
plugin which is one of the built-in
plugins
um
we have variants we want to compare them
to the reference genome so we have a
separate file again with the reference
genome for each of the exons in it
we then join that with the data I think
you're seeing where this going the data
is all kind of flowing together
we split out data one way for the the
things that are unchanged oh no I
pressed a button
we split out the data one way for the
um
the stuff that's unchanged and one way
for the stuff that is changed oh I
pressed the wrong button
sorry about this
and then we finally we call the variance
we say okay they're changed these are
all changed in what way are they changed
um so we go through and we we make like
a little diff for each one
and then we take counts of each diff How
likely are these or how common are these
compared to how common are the ones that
are unchanged we calculate a score from
that and finally because this is science
we write it all out to a CSV file
because CSV files Make the World Go
Round
um and all of that's in a configuration
file so you can change any one of those
steps and it's it's relatively easy to
take someone else's work replicated
alter it a little bit maybe just change
the way it scored maybe just
um alter the way you've filtered the
data early on maybe you add in a step
you just go back there somewhere and say
I want to throw out all of these ones
because these ones look like garbage to
me this is a these low quality reads I'm
going to get rid of and I'm going to see
whether my my statistics come out nicer
at the at the other end
the idea is that they should be very
flexible very easy for people to change
if you change your approach you can
still like your sequencing approach you
could change the first few steps while
leaving the rest of the steps alone and
check that the answer still come out
about the same if you wanted to change
the scoring you can leave the first
steps alone and change the last steps
you could change the middle if you
happen to change the ways you're
thinking about things
so that's contests hopefully we've seen
a little bit about bioinformatics we've
seen some interesting problems that we
want to solve and why they're important
um
one of the really lovely things about
this job is at any time you feel like
you know your work is pointless and
you're staring into a video monitor all
day and night for no reason you see
brca1 here and ovarian cancer there and
and so on and so forth and you realize
oh hell yeah no this actually is this is
quite meaningful at the end of the day
and hopefully we can really help
move people faster into those research
activities hopefully you've seen some
interesting stuff about python
techniques a little bit about the entry
points and stuff and maybe a Nifty
approach to data pressing I look I think
that this thing is actually useful
Beyond bioinformatics potentially I
think that there's potential for this
kind of approach to work in other
Sciences as well I hope I'd love to talk
to people about that if you if you're
working in some other science as well
um
and I think it's been a really
interesting project and I'm very glad to
be working on it at the moment
um so thank you very much I think we
should have time for a couple of
questions
thank you very much Nick okay we have
time for one question uh and if you have
any additional questions please ask Nick
afterwards in the lunch lunch hallway
track all right do we have any questions
yeah I should also mention actually just
while I've got the mic here um that URL
will give you all the slides and all the
notes from all the slides so if you if
you wanted to see anything closer up
it's there yes we have one question
so you're putting together a workflow
for crunching lots of stuff you
mentioned briefly that you're spreading
out the work so do you have like a
particular technique you're using or
pickling and sending the jobs around
uh yeah starting with the work
um at the moment it's all running on one
computer rather than distributed
um it could also be the the back end of
this thing could also run on a desk for
example it does have distributed sort of
pandas bit distributed
um
and we've we've worked with that
previously
um I think that's probably a good way to
go
at the moment I've been concentrating on
getting it working on sort of multiple
CPUs and that abstraction layer of
um
how do you write a plug-in that knows
enough about what is going on that it
can say hey I want to do a map reduce
or hey I don't care I just want to work
on a row at a time just give me rows I
just want raw rows and I'll give you
back a column
um what I want to do is get that
abstraction right and then in some ways
that will take care of itself because
uh whatever back end you choose to run
it on
um it becomes its problem right uh it's
how does it assign CPUs does it
distribute stuff uh in some
circumstances doesn't need to the other
thing that I ran into very quickly and I
if anyone follows me on um uh thingo
fetty verse
um you might have seen me whinging about
this at one point I finally got to the
point where I was using all 16 CPU cores
on my computer and it was great and it
made a lot of loud noise with the fans
all turning on and stuff and I went this
is brilliant and then I ran out of Fram
and it was killed by the ocean killer
um and so that really you know one of
the really important bits is actually
not just how to use all the CPUs but how
do we not read all the data how do we
wait until some data comes out the other
end before we read more data and I I
think that's a
a worthy problem of evaluation but also
it's something that the plugins kind of
if we get the API for the plugins right
we can come back to it in some ways
because
um
because the plugins can then exploit
that that new understanding
without changing the plugins or changing
people's experiments like conceivably we
can go we can already run a lot of these
experiments on one computer in one place
um I mentioned before that the research
is underway on pal B2 and I spoke to the
guy who's researching that and I said oh
so have you got a similar amount of data
to this brca11 he says oh yeah it's
similar it's three times as much
um so yeah literally in the space of a
couple of years data is tripled in size
that we're dealing with so yeah I think
it's it's quite likely that these
problems will get bigger and bigger and
bigger and we'll just be chasing Moore's
Law the whole time so um but that's it's
a really interesting thing to consider
is do we use Dash do we use spark do we
use a million different possible
clustering things
thank you Nick we'd like to offer you
this gift as Thanksgiving of our
appreciation for your interesting talk
thank you very much no worries thank you
all right
foreign