Hello, and thank you for listening to the MicroBinfeed podcast. Here, we will be
discussing topics in microbial bioinformatics. We hope that we can give you some
insights, tips, and tricks along the way. There is so much information we all
know from working in the field, but nobody writes it down. There is no manual,
and it's assumed you'll pick it up. We hope to fill in a few of these gaps. My
co-hosts are Dr. Nabil Ali Khan and Dr. Andrew Page. I am Dr. Lee Katz. Both
Andrew and Nabil work in the Quadram Institute in Norwich, UK, where they work
on microbes in food and the impact on human health. I work at Centers for
Disease Control and Prevention and am an adjunct member at the University of
Georgia in the U.S. Hello and welcome to the MicroBinfeed podcast. All three of
us are here today, Nabil, Andrew, and me, Lee. We're talking about working with
other people's bioinformatics software, installing and managing, and all of
that, which is often the most tedious and painful part of the job. So much so
that we often rewrite a whole tool just because we can't install an existing
one. Today, we're going to give some advice on how you can avoid all of this and
get up and running as painless as possible. So often you'll hear us talking
about containers and virtual machine, and you'll wonder why do we suggest
something that seems so incredibly complicated as the easiest way to do your
work? So if we talk about some of what we've had to go through in the past, you
might start to get an idea of why we recommend these approaches. And maybe at
the end of the episode, we'll talk about some of the best ways that you can
manage your software installation today if you're starting a new project. So
let's get started with the far gone past when we used to deal with compiling
source code. I think, Andrew, do you want to lead us off and take us through
what it used to be like in the old days? Yeah, this is because I'm the old man,
is it? Well, I didn't say that. I just found out recently that I'm the older
man. I had no idea. Yeah, I remember. We found out that my birthday is just like
a few months before yours. I'm still the young one. Okay. So yeah, back in the
older days, and now we're only talking about 10 years, it was a pain to do any
kind of bathmatic software installations. In fact, I was on a team and we had
one person basically spend all their time figuring out how to install all of
these random programs that are coming out, like go and edit line 237 of your
configuration, then go and compile it and then install 30 different
dependencies. But you have to have exact libraries and exact dependencies. And
it would only work when the moon was full in the sky and absolute nightmare.
We're quite lucky that we're getting away from that. A lot of competitions who
are coming along now don't know how easy they have it. They might complain about
how difficult some things are to install, but my God, they've got it so easy
compared to what it was like only a few years ago. And that's only improved
because of a huge amount of effort to try and make this stuff work a lot better.
And so we don't all have to spend hours and hours and hours just getting the
very basics work and getting a random dependency of a dependency of a dependency
to work. I remember a lot of software used to be, you know, you have to compile
it and they might just give you, here's the command you need to compile it with
no other information. You want no information about what libraries you need or
versions or what compiler you even need. It's just like, you know, GCC, blah,
blah, blah, and no make files to make your life easier. And of course you'll
probably fail because that's, that's how things go when you go and compile
random code that you just got probably emailed to you from someone random or
you, it's a tar download from a paper and then off you go. Things did improve
slightly when people started actually including things like make files to make
compilations a bit, a little bit easier if they actually, if you didn't have to
go and edit them. And of course, all of these things require is like a lot of
expertise just to get some very basic stuff working. If you've ever had to go
and do software installations or software development on OSX, you'll know that
there's a lot of stuff you need. It doesn't work out of the box. You got to go
and like install Xcode and command line tools and stuff like that just to get
some basic compilers. But no one tells you, you need to go and spend, you know,
a couple of hours downloading gigs of data to actually get your absolutely most
fundamental basic underlying system set up. And nor does anyone tell you that
you need, you know, all these random libraries like gzip files and things like
that. So it's a challenge to get up and running and things have improved
greatly. So that's kind of what we're going to talk about. But first, yeah,
dependency hell is something that where we've all been. Nabil, you have an
example there. I've put a classic, what's called a diamond dependency, which is
possibly the worst kind of dependency hell that you can run into. Now, this is
something which you as a user shouldn't have to deal with. You have a tool, you
install the tool, that's fine. Then the tool has a submodule at its own
dependency that it relies on. It has maybe a bunch of different ones. Let's say
it has one or two of them. And those submodules in turn have a shared
dependency. One of them requires version one, the dependency requires a
dependency of version one, and then the other requires version two. So you have
this conflict. And that is a hell of a lot of fun because there's actually no
reasonable way to resolve this. Something is running, looking for version one,
something's running for version two. The same thing is installed as a common
library. And you are sitting there at the top going like, looking at this stack
tracer, this doesn't work. At this point, this thing is looking for this thing
and it's not there. There's a function that's in version two that it's just not
able to find. And you're like, I have no idea what is going on. I just wanted to
align some amino acid sequences and go about my day. What is this nonsense? You
can spend an incredible amount of time trying to, to fix this. And it can be to
the stage where you have to hack the submodule to use the other version and sort
of like jig it a bit. I've had one where I had to, I won't mention the tool, but
it was using an older library that didn't have a reserve word. And then in the
newer version, the newest versions, it did have a reserved word. And so it had
this conflict. It was trying to do, trying to look for something that was
actually pointing to something else. And so I had to go in and actually hack it
to make it work. And then I could get my job done. So it's like half a day
you're spending trying to get this stuff to work. In some cases it is
unresolvable. There is no way to figure this out. They are too interconnected
and they're both conflicting to the point where it's just like, there's nothing
you can do. I remember having to go and edit other people's code because, you
know, they had assumed that some tools would be available and they'd assumed a
particular version. Like do you remember the old version, the 0.1.19 or
whatever? And then there was a big... Oh, that one. Yeah. That one. Yeah. And
then you'd have different tools. They'd want different versions of some tools,
but within the same kind of, you know, script that you're running. And all of
them would call some tools expecting either the old version or new version, but
no way to distinguish between the two. And so I think the only way I could
resolve that was actually going in and editing, well, the executables to
explicitly call them like some tools, 0.1.19 or whatever. And then the new
version, and then put those hard code into the scripts, like, but you don't want
to be doing that. You know, that, as you say, it wastes half a day, a day
easily, or even more sometimes just doing all of this, you know, little
engineering work to make stuff work. And it requires, like, often you have to
dig into vast quantities of code and actually really figure out where the actual
problem is. And you don't even know that. Yeah. So there's other dependency
issues. And so we've been taught... So that one is like a diamond dependency.
You can have things like a circular dependency, which is what exactly what it
sounds, A needs B and then B needs A. And you're just like, wait a minute, that
shouldn't have happened. Or you can even have something more subtle, which is
just the fact that there's a lot of bloat. I think Andrew kind of pointed this
out before, like A needs B, which needs C, which needs D, which needs E, which
needs F, which needs G. And then your thing is sitting there forever, compiling
and compiling and compiling. And at some point it'll break. The more in that
chain, the more fragile it is. We run into it with bioinformatics because the
code we're usually running is academic. So it's not strongly tested and it's not
really designed for production code that people can just pick up and use a lot
of the cases. We call it academic quality for a reason. Yeah. But this is more
of a generic problem with all software engineering. And there is a lot of effort
that's been put to try and avoid this stuff over the last decade or so. I
remember one of the simplest ones I used to run into was when you were on a
Linux system and you wanted Java, you had Java, open Java SDK installed. That
was kind of what you got from the package manager. But often some required
something specific, required the proprietary Java, which is the different type
of Java where you had to get from somewhere else. And so your thing would be
like, oh, I don't run because this isn't the correct version of Java. But it's
Java, but it's the latest one. You didn't realize that there was a difference
between the Sun Java development kit versus the open Java development kit.
Nobody told you this. That's not in the documentation. You just have to figure
this out somehow. And eventually you rack your brain and go, oh wait, yeah,
okay, fine. You find it on some random forum. Some guy posted it two years ago.
Oh yeah, just do this. Lee, do you have any horror dependency stories or
horrible running bloatware things? Xcode is definitely bloatware.  Two gigs just
to get GCC just does my head in. As the resident Python hater, I will, I'm going
to bring it out there. Bring it on. I was reviewer three one time and somebody
brought to my attention for review some software, which, you know, in the end
they made it really great and I'm not going to name names, but they made it
really great. But they had a dependency on BioPython, but not directly. They had
a dependency on a software that required BioPython version one point, maybe it's
like 1.7 point something. And then they required some other software. That other
software required BioPython 1.6 point something. They obviously were able to
install it on their own, on their own computer. And they did it one time
probably. But then as a fresh new reviewer on their software installing it and
not understanding Python very well, the libraries, frankly, like how can you
possibly install this thing to review it? I feel like they don't pay attention
to those rules and everything that semver.org puts forward, which is really
underrated. So Lee's talking about semantic versioning for those at home. So
with semantic versioning, those numbers actually mean something. The first digit
is your major version. And if you're versioning within that, nothing should
break that the next digit, you know, from 1.6 to 1.7, it shouldn't break
anything. And then if you patch something, you change some documentation, do
something very, very, very minor. That's the third digit. That's the patch. And
people need to pay attention to that. There are some software out there where
they just completely rename the executable and bless them. They've made our
lives easier. And Kraken comes to mind. They completely changed the executable
names to have a two after it when they increased the version number. And that
helps everybody. Well, I mean, if you're going to bash on Python, I've got one
word for you, mate. Perl5lib. Perl5lib. Can you explain the joy of Perl5lib? Oh
yeah. I'm not going to defend it too much. So Perl5lib. And I'll also, you know
what? I'm going to put R into that too. So CRAN and CPAN, they both put their
libraries into, you know, whatever the installation directory is for your
libraries. Usually it's in your home slash lib. And it doesn't matter what the
version is. You have like our biofilo or bioperl4perl. They all go in the same
directory. It doesn't matter what the version is. It's just in that directory.
And then your Perl or our program points to that thing. The different concurrent
versions are not available. It's going to get confused as soon as it starts
looking in those directories for its requirements. I'm not going to defend it.
As you keep installing things, one, you know, you install a new thing. And so
its version of whatever overwrites the previous one, which breaks the old
software. I mean, they could have, they could have fixed it by labeling the
directories after the right versions or something, but nobody bothered to do it.
Nobody could be, what do you say? They can't be arsed to do it. Yeah. A system
level PIP for Python has exactly the same problem. I think this is probably
where the notion of virtual environments came from. And why a lot of people
said, you know, you have to do it this way. Very much promoted as the way to get
around this problem. And if you don't know what, if you don't know what a
virtual environment is, the idea is just to have a localized version of the
environment for the interpreter language with all of those libraries separated.
So you basically have separate installs of Perl or Python or whatever, and they
initiate and they run your code so that you can separate all these different
scripts that have these different requirements. If you can install them at all.
If you can install them at all, sometimes, yeah. I was looking through the Rory
issues the other day. I haven't looked through much at all in a long time. And
Rory's written Perl, and my God, the poor people trying to install it. So you
got people, you know, going and they try CPAN, right? Because Rory, the Perl
libraries are actually in CPAN. And you know, it goes and downloads all these
different dependencies, like dependencies of dependencies of dependencies, and
then something fails, and then more fails. And it's just all of this big, you
know, verbose output where it doesn't actually tell you what's gone wrong. I
know from better experience, it probably means, you know, if you can't install
your Digest SHA-1 module, it probably means that you don't have a compiler
properly installed underneath, or possibly you don't have your profile lib set,
but usually not. People just can't work this stuff out, you know, from these,
you know, very verbose messages of obscure things that are buried deep within
the bowels of an installation script. So yeah, God help his soul. Yeah, I see on
your issues page that other people are helping other people. I guess you haven't
visited for quite some time. Well, I guess the problem is, you know, I've got a
job doing other stuff, you know, and you don't get paid to maintain stuff that
was written eight years ago. Yeah. Nine years ago. There's already been an
exciting switch because I saw one issue, and we might as well talk about it,
which is the M1 Apple architecture. So people might not realize that the
internals of your central processor in your computer differs from machine to
machine, such that software might not actually run. And this is every time you
come up with a new architecture, there's this teething period where they try and
have to patch things and get it cost compatible. And so a lot of people who have
bought shiny new Apple laptops with an M1 chip in it, can't install basic
software like Rory, as I'm seeing in the one of the issues because that hasn't
been fixed yet. And it is very fundamental. People don't realize that an Intel
CPU is very different from an ARM chip. They fundamentally have different
instructions and you can't emulate one on top of the other, but that obviously
slows things down or whatever. But that's a very, very fundamental thing that is
gonna break things. And it's gonna cause problems for a long time to come, I
think, particularly for stuff like mathematics. You know, I would love to fix
the M1 Apple stuff for Rory, but I don't have one of those laptops. So if
someone wants to buy me one, then off you go. I'll gladly accept donations of a
brand new high-performance laptop. It sounds like a good request. It reminds me
actually of a bug report someone sent in for a different program called Artemis.
And it's like a graphical visualization program for mathematics. And the guy was
complaining that the font size doesn't really work very well, 4K resolution
screens. And I'm like, okay, but I can't test it because I don't have, you know,
a massive high-resolution screen in front of me. This, you know, this is what,
five years ago. And, but you can buy me one if you want. It sounds very fair.
Would you do it for free after you got the equipment? Maybe, I wouldn't make any
promises, you know. On one of mine, I committed maybe a mortal sin for my high-
quality SynthPipeline live set. I actually put in the make file, all of the wget
steps, all of the, everything to install everything all at once into
subdirectories and all the dependencies are installed for you. And so it's like
a 20 minute installation with a bloated installation directory. I did it because
I wanted to have a tight control over the versioning and make sure it works. But
it feels like such, I don't know what the word is, but it feels like such a
mortal sin with Linux and Unix ideologies. I've done it. I've done it. There are
just points where it's like, you can't keep coming, having people coming back
and saying this version is the wrong version or whatever. Like you just go, oh,
I'm just going to embed the binaries or just have it pull it from somewhere.
Well, that's a Torsen as a procker. He embeds lots of the binaries. Seems to
work, except for, was it table two ASN? I know that that's two years out of
date, which probably means that it's broken because it's expired. I saw, I think
the latest hack was to emulate the date to be, to not to stay the same. The
system date to the program would see it as always like not expired. So for those
of you who don't know, this is a very famous program from NCBI, which
automatically expired after I think a year and it just would stop working. So
pipelines would magically stop working for no really good reason. It was just
this, this program didn't like to be old, wanted to be forever young. Phantom of
Youth. But you could check it by changing the system date, which is what people
had to resort to doing. And they containerized that hack too. Yeah, they
containerized it. So it's forever in this time, like locked in this time
capsule, thinking it's 2021 forever. I think the latest table to ASN is fixed
finally. It got to the attention of some good people at NCBI, but I mean, that
was, I think that program had that issue for years. That was a standard issue
apply. If anyone, yeah, Prokka is not working, table to ASN, table to ASN, that
was it. There was one more twisty problem, getting back to dependency hell. I
think that one of his programs, maybe it was Prokka, had a dependency on Hammer
3 and Hammer 2. And so somehow it was embedded and it pointed to the correct
executable somehow. Yeah, that often, that's always a thing for pipelines
because you use Hammer 3 as a new thing for something specific you're doing. And
then you use like an older program that does something else, but then it's using
that. Like that's kind of where you usually rub up against this sort of problem.
There's really not that much you can do about it. So another thing I've noticed
creeping in over time is Windows subsystem for Linux. And that causes no end of
problems because people assume, oh, it's Linux on Windows. But no, stuff doesn't
necessarily work. It's emulated on top.  of Windows and, you know, it's
translating stuff and not everything is going to work exactly as it should on
Linux. And so bioinformatics software doesn't necessarily work. I don't use
Windows, so I can't even test it. I don't have a Windows machine and nor would I
want to waste my time because, you know, it's Windows. If we get you a Windows
computer, will you take it? Not a chance. Such a tease because everything else,
it seems to work. It seems to install and compile. And then it runs into that
thing we were talking about where it just looks for some random dependency or
shared library or something like that and just goes, no, not there. Not
available. Sorry. And it just breaks. And it's really tempting to do it. But
honestly, if you, if you want to do that, I would probably suggest you just get
like an SSD and just chuck a Linux distro on that and boot from the SSD. If you
really don't want to dual boot or have a separate system that's running Linux
and you have to keep Windows. Some, some, some people are sort of stuck with
Windows because that's the, that's what they get for, for work. And then they're
trying to make, trying to explore the space and then they're not really able to,
to get started. So I kind of feel for them. On the, as, as much as I am a Python
and R hater, I do like, I do want to give a little bit of applause to Rust, but
they do it in the best and worst way possible for the dependency hell. What they
do is if there is some kind of diamond dependency or dependency hell, they
actually download all different versions that you need into different sub
directories into the project that you're working on and then compile the
project. So it's a bloated source code directory, but it gives you a very
reliable executable at the end. Is it a single executable? Like is it compiled
or is it a script? It's self-contained. It's, it doesn't depend on external
libraries when it's done, when the dust has settled. But, you know, of course,
depending on whichever project you're on it, it's one or more executables. Since
we've been talking about all the horrible things that has happened, let's talk
about how people can avoid all this nonsense and how they should approach
managing their software installs. What would be the easiest way? I think if I
had a small project where I need, where I have some data and I'm playing around
with something, you know, I've got some, some bam files and I'm just playing
around with them. I would probably reach for a separate condo environment for
that project and I would fill it with all of the things I need. So whatever
Python version I want or Perl version or R version, all the software, so SAM
tools or whatever from, or Torstyverse, whatever I need, build that into that
single condo environment, use the Python or use the Perl or R or whatever in
that condo environment and keep that as a single separate virtual environment
where for my installation and do that for every single project. I keep separate
environments for each project, unless it's something that's a little more
verbose and bloated. I might keep that as a, its own installed environment that
I sort of jumped to just for running a specific tool, a very complicated tool,
but that's how I found you can avoid a lot. Now there are reasons for this. The
reason is you don't want to dirty your system level environment, you, the only
thing that should be in your base, even in your base condo. So when you install
condo, you get, you get a free environment base. Generally I'd advise you don't
install anything in base other than Mamba, I suppose. And maybe you want to
upgrade your Pip, maybe you want to upgrade your Python, like whatever, but I
wouldn't install like actual software in your base environment. I'd always
create a separate environment for my project and just fill that with all the
nonsense. If I muck it up, I can just trash it and make another one and rebuild
it easily. What I also would recommend people don't do is inside a condo
environment, make another environment, like a Python environment or something
like that. And try to manage it that way because you just wind up getting
confused that there's an environment inside your environment, which is inside
your environment. And you don't know which point you need to initialize to run
whatever you're trying to run. So yeah, just keep it flat, one condo env for
your project. How would you approach it Lee? What would you add to that? I don't
know if I'm, if I'm alone in this. I kind of even took, I took what you said,
even less step further that you don't want to dirty your base environment. I
have it so that conda doesn't even load unless I asked for it. So tell me what
you think about this. I have alongside like my dot profile dot bash RC. I have a
different script dot bash conda and I don't even boot up the computer with
conda. If I want to have conda, then I'll have to run source on bash conda to
even load it up. And then I'll have to load my environments. So that way my base
environment is squeaky clean before I start off on things. I have learned to
adapt. So yeah, I'd agree with it because you do accidentally start installing
on base. So it's good to just be totally hands off and explicitly, yeah, invoke
it. I'd agree. Yeah. That's a good way to do it. If you find you're dirtying
your base a lot. And after I do that, I do have my, like, it's like really
stripped down just like a little bit of an upgraded Python condas in there and
mambas in there. And then it lets me, I can start creating new virtual M's or
whatever, but I try to keep each one minimal too. What do you think, Andrew? So
what's the difference between conda and mamba? So mamba is like a, is a in place
replacement for conda and it handles a lot of functions like installing and
things like that. And the main benefit of it is it avoids you looking at solving
environment for 20 minutes. Oh yeah. That's gotten worse over time, hasn't it?
Yeah. So mamba is a like for like sort of package. So you, you create your
virtual env, you create your conda environment, you install mamba, conda install
mamba or whatever, and then you can start doing mamba install SAM tools and
it'll do it lightning fast. The same way, it's the same exact same parameters as
you would for conda. So conda install, conda create, but you can just use a
mamba instead and that will do it much, much, much faster. I don't know what the
black magic is internally, but I've seen it take something that in conda took me
15 minutes. It did it in like one. Did you sell your soul for that? I've been,
I've been told that that mamba sometimes doesn't work because it takes
shortcuts, but I don't know what that means. Has it ever not worked for you?
It's never not worked for me, but then I think because I've started making my
environments specific for each project, they're fairly simple and it's really
half a dozen tools and they don't, it's unlikely that they're going to collide.
So I think I might be making it easier. I haven't really tried to use mamba in a
case like a single, like a single environment and really just installed
everything. It might wind up colliding or something like that. So as conda
mamba, what do you use for dependency management or is it, do you use pip or
homebrew or whatever? I mean, homebrew is a Mac thing anyway. Well, there's
Linux brew, you know? Yeah. I never got on that bandwagon. There was Perl brew
actually at one point I remember. Oh, that's still, that's still here. Oh, is
it? If you want. Yeah. Because it's not virtual. It's a way to install Perl
locally for yourself so you can have the latest thing for yourself and have the
latest libraries and stuff. Yeah. If you're ever using Rory and you need to fix
your Perl environment, then it's a good way to get it started. I don't know. I
use conda for everything these days. So if it's not in conda, then it doesn't
exist. Pretty much. We've had plenty of guests on the show who are very keen to,
to get everything onto conda. So I think most things are sort of up there and
it's not that difficult to get your own code into conda, by the way. Maybe we
can do an episode sometime that talks about how you actually do that sort of
packaging. Yeah, I've done a few times. It is quite straightforward. Once you do
it once, like there is a bit of kind of fluff around it. Once you do it once,
then it is actually very straightforward to do it. You can rinse and repeat over
and over and over again. And then updating stuff is basically automatic, you
know? Okay. My method so far is to make some software that everyone likes and
then somebody does it for me. The easy way. Yes. Actually, one of the hardest
ones I ever came across is apt-get, you know, the apt system for Ubuntu and
Debian. And they have a very, very high bar for quality and a lot of review. And
if you want to become a developer of modules for apt or whatever they're called,
like you need to be mentored, you know, for six months or a year, this kind of
thing. It's like really, really difficult. But once you get stuff in, then it's
really, you know, it stays there. It lives. It's not going to die tomorrow. It's
quite stable. Interesting. So like all the things that I apt-get, like SAM tools
or whatever, like people have been trained to put that in there? Yeah. Yeah.
They have a core team who, who do that kind of stuff. Debian med, I think
they're called, and they do a lot of work voluntarily just to get that stuff in
and fair play to them. It does work. And you know, you can do apt-get Rory and
it should install and it should install it in five years time with all the right
dependencies. Have you ever used like Docker or Singularity, you know, for, for
managing your Python? Not for Python. I've never had, I've never executed Python
scripts using Docker or Singularity. That's an interesting one. Usually they're
much more complicated things I'll try to run. One of the, one in saying Docker
and Singularity, we're talking about containers and these are different to
virtual M's because they're sort of immutable boxes that have a thing pre-built
inside it. And you're just dealing with this black box that has all of this pre-
configured software or whatever inside of it. And it's, it's more or less
detached from the operating system. One thing I find really useful with, with,
with Docker is standing up things like databases, like if you need Postgres or
something like that, or you need a web server or something.  Docker is way easy
just to say, you know, just to stand that up rather than having to install it
yourself onto an OS. And it's obviously once you've got the configuration for
Docker to do that, then you can port it to wherever you like. That's one really
useful thing about Docker. Generally more complicated software I'll use
Singularity. That's just so that because Singularity plays better with HPC and
I'm jumping between the two. So that's my preference. But Docker sometimes is
really easy. Like Docker Rory is really easy just to get that up and run as an
example. So maybe that means that if you did run a container and it's running
some complicated pipeline, like maybe they have their own version of Python or
something that you did run on there, just kind of behind the scenes. Yeah.
Behind the scenes, definitely. They would probably be relying on it, but just to
run Python, I don't think I've run it to Docker before. Well, you can get kind
of nice base systems, you know, with particular versions of Python or particular
versions of Debian installed. And from there, you know, it gives you a nice
solid place to work from. Or even what I've done before is create Docker
containers with just conda, you know, a few different scripts, which will just
basically go into the container, do conda install, blah. And so you've got one
tool in one container. I know you can get, people have done this already, you
know, but it's quite a nice way of keeping stuff separated. But then of course,
if you have the complication of how do you actually work with multiple different
containers? And I think that's something we covered in the other episode we had
the other day, the workflow description languages. Yeah. And we've also, if you,
if you teamed it back in with the Staff B episodes as well, we talked a lot
about containers and maybe merging them, having multiple tools. So it's sort of
like, rather than a IQ3 container, you might have a phylogenetics container that
has IQ3 and something else and a bunch of other stuff in there, one go. It
depends how much you want to keep separated. The nice thing is, is like the
Docker files are something you can just get and configure to your liking and
build. There's a lot of templates for people to use. So yeah, you can have a
good, particularly for things like R, it's great to get R templates and then
tell it install all these extra packages for me. And then I have an R
environment that has everything ready to go. And that, that that's really,
really nice. Wait, are you saying that you run R from a container then? I run R
from a container. Yeah. Always now. No, I did use Condor the other day. I won't
say always, but I have, it's very easy to get tidy. Like, I think it's Rocker to
get tidyverse, the dev tools and whatever packages you want in a Docker file.
And then you just build a container and then you just stand that up and you just
go, you know, it's like, hooray. It's even better when you hook that up to
something like RStudio or Jupyter notebooks or something so that it runs, it
produces that like hub session for you. So you go to it on the browser. You
don't even see the Docker part. You've stood the whole thing up in the container
and you just deal with it from, from a GUI from the outside and you're doing all
your interactive scripting from that. It's super, super nice. I mean, that's
what Docker is for. Docker is for sanding up microservices like that. That's the
point of it. And it's really good for that reason. Incredible. So I guess we
didn't put a note in the notes for this, but I mean, I feel like the next
natural step is do you use these containers or virtual LEMs in a team
environment? Does that help you out or is that nonsense? We have a shared
directory of singularity containers that we pass around much like in the old
days, you might have a shared directory of binary files that someone's carefully
handcrafted to work in your environment. We pass containers around, which just
to make life a bit easier. Are they in brand envelopes under a counter? Sort of,
sort of. Sorry. I'm not sure if you get that reference. In Ireland, a common
form of bribery was to put money into a brand envelope and, you know, put it
under the table. Very nice. That's how you're going to get your Mac computer,
your M1. I make no promises. So that's all the time we have for today. I should
point out that this episode was actually inspired by a tweet I saw from Aaron
Vincent, who is at Dark Cosmologist on Twitter. And he was talking about some of
the advice he's gotten for installing Python. And it's this long list that says
definitely use Anaconda, definitely don't use Anaconda, definitely use Homebrew,
definitely don't use Homebrew, definitely use pip to install packages,
definitely don't use pip to install packages. And that inspired us because it
reminded us that actually, if you're just getting started, it's incredibly
complicated to figure out what the real advice is. So that's what we've been
talking about today. What are some of the best approaches for managing your
software packages on your local machine so that you can get through all of the
hellish stuff we used to and get, get down to getting your work done. And
hopefully we've got some tips in there that have been, that have been helpful
for everyone out there. So it's been me, Nabil, Lee, and Andrew with you today
on the MicroBinfee podcast, and we will see you next time. The opinions
expressed here are our own and do not necessarily reflect the views of CDC or
the Quadram Institute.