foreign
ER Williams who's going to introduce
fast API so Atlanta's been a fixture of
their python community and in Canberra
organizing their Canberra python meetup
group for a long time then she was in
Perth and Sydney and now back back in
Canberra and uh the Canberra python you
know group's going very strong with over
a thousand members now so uh
um yeah please uh give a round of
applause to Elena
yeah indeed we're quite proud of our
Canberra python news group but also
Perth user group as well with Russ try
that okay so
um this is a fast API talk uh thank you
so much for coming it's a very tough
session to choose so I'm grateful for
everybody who's here
um
as Ed said my name is Elena Williams uh
this talk is entirely in GitHub exactly
as it is
um anyone who's seen my talks before
will know that I have too much content
so let's go
um what we're doing here today is we're
just going to touch upon a small number
of Technologies and describe where they
the fundamentals of where they sort of
come from so I'm kind of assuming you
you can Python program already and you
have an interest in this stuff but you
don't do it already
uh what we're not going to be talking
about is we're not going to be talking
about full web applications because apis
are intended to sort of float above your
data layer before your front end
um and you know data layers in the old
days were idbs full stop now they could
be anything they could be a python list
so they could be a file sitting on a
computer they could be something a
cluster of something extremely exotic uh
so I'm not going to talk about data lays
in any kind of detail but I will be
talking about gnaf we're not also not
going to be talking about deployment
because fast API basically deploys on
serverless and you can pick your own
vendor
uh very good another thing important to
note I am not from geoscience Australia
they are a different organization they
are just down the road some of the
people I care about the very most in the
world work there or have worked there
and they do like rocks rocks and stuff
um whereas geoscape work we do a dresses
where addresses are a name for a fizzy
is a name for a physical location like a
house or an office or a um a mailbox or
a shop or a store or an antenna or an
ATM or a oh what was the next one it was
a barbecue block boat shed um this here
is actually a file out of Gina this is
called an authority file
um actually I thought this was going to
be slightly Square so I'm going to pull
this in a little bit
um and uh this is a list of every single
type of address that has been supplied
to Australia for legit has been used or
we'll be using really our favorites are
Marine birth and mazonet because
everyone's like what's a mazonet nobody
knows
um so where do these addresses get
minted they are produced by local
government agencies or councils or land
planning authorities
um that's the simple answer there are
more than 500 of those uh but they also
get fed up to the states who do their
own processes on them and then there are
a couple of public sector departments
who maintain their own independent
address lists for their own special
purposes and to clarify there are no
rules about what an address should be
there's no regulatory body to make sure
addresses comply with some standard
there are guidelines
um so
there is no single a single address list
until the 19 now preparing for the 1992
census and um I was not going to include
the next slide but
um I on the flight over the
acknowledgment of country was so
lackluster that I was like you know what
I'm going to put that slide in
um because this is a picture of the 1921
census from the National Library of
Australia this is all complete aside by
the way
um firstly apologies to Tasmania uh
it's on a serious note I think all of us
here uh know what it feels like or most
of us majorities to be treated as if you
don't exist this is an artifact from
history of government level treatment
like you do not exist
um so on that I would like to give a
shout out and respect to all of our
first Nations brothers and sisters and
Elders who are here with us now respect
um but anyway enough that garbage
politics behind us we're organizing a
census for 1996.
um and nobody had a single list of
addresses because we want uh uh single
list of addresses so the states believe
it or not came together and formed a
company one share each to each estate in
the Commonwealth and it's a not a
multinational entity a private company
not the government not the federal go
owned by the government but not the
federal government the states and I I
actually think that's pretty cool
and we make a single jury-coded National
address file um because when you get all
of the sources that create addresses
they only agree on about two-thirds of
them and then you know there's like some
three to ten percent of them that are
just like could be anything
um but in the decades since I've been
putting this together there's um they've
developed a lot of sophisticated
strategies for building up data layers
and you know providing services to you
know like
heaps of stuff like the emergency
services and um you know lateral
committee Council whatever commission
that's the word
um and sourcing National critical
infrastructure and it's apparently worth
like millions of dollars at the economy
because it's free from data.gov
um and it's apparently the most
downloaded data source from data.gov.gov
dot a u and in my uh abstract I refer to
GF as a monster not because it's
particularly big it's only 15 million
records or so but because it is 203
files and they all have to fit together
just so
um but don't ask me there's a talk by
Alex Lee at this very conference was 2
p.m now it's 11 A.M tomorrow where he
apparently took apart Jane f
um so he probably knows better than I do
um but there is a flattened version that
came out last year and uh but it's a
trade-off because you missed out on that
lovely Authority file like I showed at
the beginning enough of that we're here
to talk about code no matter a matter of
fact we're here to talk about fast API
I mean sorry first API
I mean
fast API
um and you know with all of the respect
in the world to Tom and Marcello and
Armin and David for all of the work that
they've done to put this Library
together as well as Sebastian Ramirez of
course she's done the launcher of the
work here
um so this is a
how quickly fast API has become a
competitor against the major python
Frameworks and we are going to look at
some fast API example code the 101 code
saw it here in this very place yesterday
so fast API sorry
fast API I I have one joke
um
uh but I'm going to drill down into more
into the differences here so I don't
know if you can see the flask influence
uh but there are differences first
you've got this get here rather than
route
um and that's a HTTP verb you're allowed
all of the ones that are available
um makes sense for apis secondly that
word async
we're going to come back to that
um finally we're returning a
serializable rather than a string of
marker
because apis produced serializables
which went Json or XML or whatever it
might be but fast API in truth rests
upon pedantic who's kind of main core
mission is to produce validated
serializables so really these things
we're going to talk about async pedantic
very good we're going to do a very quick
tight lightweight example of a fast API
app what I think is a pretty sweet
example oh no so before that we're going
to install it you only need fast API and
uvicon which Noah was here saying is a
pretty Rock and good web server you just
run this and you can run it like that I
have it running here this whole talk is
running out of fast API using flask
static Server Like This is actually
JavaScript but it's it's running from
Fast API to why that's there
um this is running live too your nice
lovely Swagger docs from flask
now we're going to do the example
um so I think this is a really great
example of a really tidy little API
delivery so in this box here whatever I
type into here is going into my search
string here and it's going to filter
this list of wordy words here so if I
type a big seed and you can see it's
running live here and that's the point
I've got here so if I can type whatever
I want and it will filter out that list
or not whatever okay you get it that's a
really quick fast API
very case scenario for use case for
Australian
um this is like an address flavored one
and this is a shout out to my friend
Rachel who gave who gave it to look um
the other a few years ago about
addresses which I strongly recommend
seeing this is the real life version
that we use that's plugged into gnaf and
the internet seems to be oh it works but
we'll give an example here though sadly
the challenge with delivering gnaf in
this manner is not the type head it's
parsing the address string very
interesting problem then connecting that
past address string to searching on our
data source and then we also keep this
data fresh every day uh which these are
all things that we
a extremely interesting problems that
I'd love to talk to anyone about later
if you're interested Okay so
that is a really quick example of fast
API let's talk about
um async but
um truth is I'm not going to erasing
heaps of people are um this is Zach
Hatfield Dodd's talking about async at
this very Year's pycon us
um former Canberra python organizer and
we miss him dearly
um shout out to everybody else involved
too but there's one quick tldr which I
will give you to async if you like me
you walk up to AC and go I have a degree
on how async Works um if you want to run
async and python um The Interpreter
needs to know at runtime that you're
running an awaitable you do this by
using this async await syntax if I want
this get data function to be
asynchronous it's going to go off and go
fetch my data from wherever it needs to
be called using this await you have to
have the async and a weight pair and
that await has to be called by an async
function that's been called by an await
all the way up and down the stack and
if you forget to put in async or await
you'll start to get really wacky
exception exceptions like so if the word
co-routine is popping up in your
exceptions just double check you've
gotten all your racing Center weights
it's a really easy thing to forget
that's all I have to say about async so
the other thing I was going to talk
about is pedantic so hands up if you
have heard of pedantic please because I
want to know oh wow I know keep your
hand up if you've heard of pedantic V2
hands up if you've installed pedantic V2
oh my heart goes down to people okay so
I'll say um when this talk is proposed
Pi negative 2 wasn't released I propose
this talk based on the experiences that
we'd had with pedantic V1 because it was
a lot of fun
um they have fixed everything or pretty
much everything that I had to say about
pedantic V1 so that's not a relic of
history and so I got back five minutes
of my talk that I'm going to fill with
pictures instead
um
because I'm not going to talk about
pedantic right now in fact I am going to
talk about typing in Python in general
because as we all know python is weekly
type what was the one yesterday it was
gradually typed
um and people who feel strongly like
that about that are like a little bit
extreme either for or against it and I
would argue that our beloved babyfl this
is my personal unsubstantiated Theory
um he stepped back from that role
because of the heat that he got about
his very gentle attempts to introduce
typing
um because as we know everything in
Python is a object all the way up and
down you can do whatever you like to it
you try and do something to it and it
will just give it a red hot go there is
a time and a place when you don't want
to do that and there are a lot of
different ways to shout out to Alec
Thomas if you're in the house like he's
a local actress team are too
um but the current hotness in 2023 is
pyrantic um so thank you to Samuel
Colvin and the whole team there uh they
got several million dollars in VC
funding at the release of python
pedantic V2 which was a complete rewrite
up from the ground up in Rust though the
API is still exactly the same so let's
look at some pedantic code so
pedantic
jokes pedantic um nobody thinks that's
funny for me but everyone
um so this is your standard pedantic
base model I think it's pretty obvious
what's going on here we have annotating
some types
we can stack our annotations we can have
custom annotations
uh and I think as would be expected if
you have a dictionary here full of stuff
that matches annotations and in the old
days we used to just have an orm and we
would never see anything like this but
nowadays because you're kind of getting
your data back from whatever generally
you're going to get it back as a key
pair sort of deal which fits well into a
dictionary which we then use the keyword
argument syntax to put into our pedantic
object and thus a pedantic instance
comes out as would be expected
if we do something like this in pedantic
if you do that in a normal python object
it would just make your instance and
everything would be fine as your tuition
probably suggests it does not like it
um it will say like you try to put in a
string I wanted an integer why would you
do that uh also there's this concept
called literal in Django we'd call these
choices
um I think this is like a perfect
example like in no case when we're
putting together a dress do we want
something in Australia that is not on
that list of nine things ever
further that literal is not a list it is
a literal and everything inside that
literal they are not strings they are
enums so beware of that don't treat them
like they're it's a list because you
can't
um so as intuition would suggest if you
put something in there that's not good
it will tell you it doesn't like it it
should be one of those great
uh other things you can do is there's
this concept of field which should be
very familiar to everyone who's smooth
orms it's an analogous sort of concept
you can do validation uh you can do set
defaults you can set fancy defaults
um you can do all kinds of stuff also
worth noting is pedantic inheritance can
be a bit strange like async it's got to
be pedantic all the way up and down the
stack so it's got something to validate
against
um
and it also does not oh you can use
abstract base classes but your attribute
uh value inheritance can be a little bit
weird so just don't take for granted
that that's just going to work as you
would expect also my I'm a Django guy
and maybe I just use that property a lot
but I was really mucked around by the
fact that
um it doesn't handle our property like
easily off the bat um but you can use
this thing called computed field and
that figures it out for you so first API
and pedantic just play really well
together and they're implement it in
that way you want to return a
serializable you can Chuck it out as a
dictionary or as a pedantic object and
that will just return your API request
okay
now uh let's change gears entirely and I
want to talk about Geo Json
um so check out that standard oh yeah
and I understand it's an RTF and an ISO
don't at me uh so this is when this is
no longer python this is a Json object
to notation this is a standard geojson
response um it has three things in it by
the standard it has a type which has to
be a string that for string has to be
kind of feature there's another thing
but we'll get to that in a minute but
basically it's feature
um there's a geometry this is a very
tightly defined object it's got two
things in it a type with a string of the
type of geometry you have there are a
bunch of them they're also tightly
defined and you must have a coordinate
that's it geometry type coordinates and
then they're this property attribute
this can contain whatever you like you
can leave it empty you have to provide
it you leave it empty or you can put
anything in it we overload ours very
hard
um trust me you can do a lot with them
um so type geometry properties easy
um if we wanted to model that in
pedantic it might look something like
this so you get your geometry object
with time coordinates you've got your
feature type geometry properties sweet
um the other thing you can provide is a
feature collection
um which as the name suggests is a
collection of features uh where it's a
type geometry properties and that's
basically it's not the whole gear Json
standard but it's like most of it in
about a minute and a half
um so the other thing I want to dive
into is the types of geometries I just
want to spend a minute talking about
these because I think there are some
points of note if you're new to the Geo
field um you do folk please come at me
because
okay first thing Point
um we say in English vernacular lat long
if you think about it light is with
y-axis
um and long is the x-axis
um I thought this would have been more
figured out it's not as figured out as
you would think it should be there's
also the whole degree versus float thing
just we use Cartesian coordinates every
time we're doing something technical but
don't bet your life on it um so Point
array two things light long
um
this is for things like where we are
right now
um and some other nice pointies in
Planet earthy stuff a line string is an
array of points in order to make a
string
we use this to construct roads oh
internet internet no internet
um
my friend Alex Ware who we spoke
yesterday has spent a lot of time doing
Rhodes data her and John Martin who's
also here that like intersecting Road
strings across borders and making sure
they all line up they've been doing a
lot of that lately got no idea how
they're doing it but if you're
interested they could talk your ear off
about it
um uh the other another type is a
polygon which has a an array of points
where the first and last point is the
same like a building footprint uh then
the last one I'm going to share is multi
polygon which is uh everybody's favorite
and why is it everybody's favorite
because it's quadruply nested uh but if
you think about it if you're going to
have an array multiple Donuts it has to
be quadrupally nested uh and we use
these like day in day out to do like
yeah we use multi polygons for basically
everything
um cool so if we wanted to model these
in pedantic
um honestly this is on here because it's
like list list list
someone's interview
um so I really want to say thank you to
Osteo because they produce this Library
called shapely and if you have a valid
set of coordinates
um sorry I did forget to mention that um
there is handedness in Python with
polygons so uh strictly tight geojson
expected to be clockwise you don't
always have to do it but um strictly
validated will kick come come at you if
you don't
um here like shapely just you give it
your geometry and this like it will just
give you an outline if you want to just
calculate do all the math stuff for you
um this is an Eric dot area and it just
you can feed a geometry and it just does
that nasty stuff like makes it go away
which is great
um uh so in summary Geo Jason also
allows 3D
um also allowed to mix and match your
points and mix and match your geometries
in your feature collection so let's do a
geojson query for like here
um so we've got back a bunch of
buildings from gnaf sweet
um there are like there are four but
only two of them had names because of
reasons um so type property geometry
Json
um the query is for here
um I want to believe that at least
someone else in this room is thinking
about this XK CD comic
um where you know this is us here
um
so I just want you to know if you're in
debt you can probably use in Australia
GTA 2020
um though if you absolutely must convert
your datums to other datums which sounds
like a fiendish problem to do Pi Project
is just magic does it so there is a
library in the world imagine it's a bit
of a bit hard to install
um but you want to convert datum's Magic
okay so why do we care
um basically like we're doing a lot of
digital Twins and digital atlases at the
moment um but you can just pick a
library a mapping library off the shelf
just do it
um I picked leaflet I got their 101
example I just copied and pasted in that
gnaf response from before and butter
Bing butter boom and I checked in some C
for data because like like we're really
smart like it's not educational 95 so
it's just ABS since it's not data but I
thought that was funny
um so here's some more data this is like
Regional Buildings we've got going here
and land
Parcels I think um this is a mesh Block
in the suburb of Adelaide
um this is like the electorate of
Adelaide and the suburb and this is like
all of the state of South Australia
you'll notice that is 310
000 points uh we don't rather do some
other cool stuff like mix and match like
um uh pointing roads you can do some
polygons can do some shaded polygons can
do some 3D geometries
um can do some tree height and um those
tree Heights are actually for real
um uh major Australian Telco uses those
trees Heights to met to like calculate
shadows and clutter and stuff
um
oh gosh I do some pools and some solar
panels and I was just trying that off
because I think I think this is a really
great example of making sense of this
statement here
which is that threading is for working
in parallel async is for waiting in
parallel
so I got a bunch of async functions
pitching off to get my data of all kinds
of stuff so that started electrics one
that was 310 that's very big we're
coming up to the limits of all kinds of
stuff you are hard-pressed to turn that
response around and quicker than you
know like 600 to 8 000 milliseconds
right
um but and so if you've got two of them
and you're stacking them up like that
your response is gonna get very very
long so fast API is perfect here for
running them all at the same time and
then the hour response will only be as
big as our biggest response that we can
get as many different bits of data as we
want Chuck it back on the same Geo Json
response in our massively overloaded
properties and um
and it will be very quick it'll only
take as long as our biggest response
um and then that's a different problem
and I forgot my weight uh because it's a
really easy mistake uh and that is my
entire talk uh thank you very much
you've been a great audience
it's a
thank you very much Elena for a very
animated energetic talk and a very
professional author polished slide deck
so thank you so much and uh yeah um are
there any questions
yes Russell
Mike
thank you thank you very much
um
the you mentioned GTA 2020 at the end
there because there no reference data or
no is there any way to know that your
geojson data has been uh has been coded
against GTA 2020 versus WPS generally um
a header is a prior to CRS and
the pain that is petroleum
um is spatial I guess is the yes but
um yeah and these are all aliases in
fact and gdo 94 is used still commonly
but like I said um usually if it's going
to be specified the datum is going to be
specified it will be in the response
headers and altitude is relative to sea
level mean sea level if it's a three if
it's a 3D coordinate
um good question don't know check the
check your references
um I have people who could give you a
very very very long-winded answer about
that I am not that I don't think this is
the time of the place but
are there any other questions
yes
you could probably choose I'm gonna bet
hello you've got a lot of structured
data there so you the the Json input
which is quite well structured and so
you can expect to bring that into a
pedantic model fairly easily but earlier
you mentioned the addresses which are
very loosey-goosey did you find just did
you end up writing pedantic models to
deal with them and if so was it an
emotionally traumatic experience
unfortunately that's a yes or no
question the answer is yes
um but yeah we have pedantic pretty much
all the way up and down the stack and um
yeah yeah no you you've been there
you know the feelings um so pedantic V1
was less performant than Titanic V2
um and we did have to do some things to
work around that um yeah but overall it
is nice particularly if you've got
people who've come from statically typed
languages
um they find it a lot saner to just be
in a typed world so you can have some
engineer work with Engineers who come
from their background and they they're
cool
family one
we go oh no we do we've we've got ways
yeah and they I strongly recommend
identity too it's really it's it's it
works really well and there's a a tool
for switching from B1 to V2 which seems
to work fine
all right we have any more questions
hello
um
uh building up on the several different
libraries that there are for validation
I notice it looks very much like data
classes as well hello yeah
um what does pudanta give you over uh
like the standard Library data class
it just behaves differently the honest
answer is I don't really know I have
used data classes probably as much as
I've used pedantic but I can't they just
use them for different stuff um I've
never really thought about that framing
um okay so on the lid of it pedantic
says it's good for validation validation
and serialization whereas data classes I
don't I just use them for type
annotation and you know some minor sort
of validation great question let's find
the answer
later thanks Luke
we have time for one more quick question
yes
gifs it's a pretty cool standard what
influences it had on the international
landscape oh that so great so it is
currently the time in history where I
think a lot of these standards are you
know every are just coming to the point
where it's coming becoming a bit
competitive where you used to be just
governments and now there are other
players going hey we can do this um and
so there's been a bit of a
standardization
um and um uh discussion amongst the
different entities who produce this
level of data set
um about about that but it's the very
early days of that history I think so
um I think it's reasonable for me to say
that we do talk to other agencies around
the world who produce these similar
comparable artifacts but they are
different in each region like for
example Malaysia doesn't have one at all
um but
um you know the whole standardization of
addresses is it's I think we're at the
beginning of that conversation that's a
very excellent insight
thank you very much all right we have a
small gift for you as a token as well
appreciation
um so I'll present you with that and
that as well thank you very much
thank you very much
that's awesome
[Applause]