Private Packagist's Nils Adermann

Private Packagist's Nils Adermann

Apr 09, 2024

In this video, Scott talks to Nils Adermann about composer, Private Packagist, and Nil's talk at php[tek] 2024. Links: Private Packagist - https://packagist.com/ Tickets to php[tek] 2024 - http://tinyurl.com/tek-2024-tickets 00:00 Intro 00:53 What is Composer? 04:19 What is Private Packagist 15:20 php[tek] 2024 16:13 What is your talk about? 18:24 Outro

Transcript

Scott Keck-Warren Scott Keck-Warren 0:04
Hello developers and welcome to the PHP Community Corner where we have conversations with members of the web development community. I'm your host Scott Keck Warren and on today's episode we're talking to Nils Adtermann, the creator of Composer, about Composer, Private Packagist, and his talk at PHP Tech 2024. Nils Adtermann created Composer, the dependency manager for PHP, founded Private Packagist, a package repository as a service, and is a founding board member of the PHP Foundation. Through working on open source and commercial PHP projects for over 2 decades, he learned firsthand what daily problems PHP developers face. His projects are now focused on improving the developer experience around PHP and providing businesses with tools to use PHP more effectively. Thank you for finding time to talk to us today.
B
Speaker B 0:51
Yeah, pleasure to be here.
Scott Keck-Warren Scott Keck-Warren 0:53
So can we start out for our listeners who may not know what it is that Composer does and how they can use it?
B
Speaker B 0:59
Sure. Happy to give a brief introduction to Composer. I mean, these days I kind of hope that most PHP developers start out using Composer, but on one side, there are still people starting fresh today and there are still people working on older projects that are not using Composer yet. So Composer has become the standard tool for PHP developers to install packages. And in most common cases, that means you're going to go and you're going to install maybe like a framework that you're working with. In some cases, it's going to mean you're going to use Composer to manage a few libraries that you're going to use. And most frequently, you'll install them from packages.org because they're open source tools that you're going to want to use. But they can also be either purchased third-party plugins to a system you're using, or they might be internal packages if you're working in a larger team and you have different people, different groups working on different aspects of an application, you can turn these individual parts into Composer packages and then use Composer, the command line utility, to actually manage which release versions get installed and which version. So it very much just is a classical package manager specifically built for PHP applications. And I think important parts that I always like to highlight about how Composer works is that instead of being this classical model of just being like, okay, install this thing, install that thing, the core idea to Composer is its lock file, Meaning it maintains a state of what the versions and the packages that you're using in an application are, so that you can easily replicate the same installation procedure on another system. Either that's a teammate, that's some production server that you're trying to ship an application to, but the core of it is always you first resolve all the things that you're trying to install and whatever dependencies they have into one state, and then you can take that lock file and based on that, you can replicate installing the exact same things in different systems.
Scott Keck-Warren Scott Keck-Warren 2:53
I unfortunately remember the time before Composer where we were dealing with things like Pear or installing packages through like apt-get or something like that. So I just cannot thank you enough for creating Composer. It's just like, I can't even imagine how many hundreds or billions of man-hours have been saved due to Composer over The stuff that we had before that worked, but it's just not as nice as what Composer provides.
B
Speaker B 3:18
Yeah, I think that's very much a part of how this is built, is just try to have as easy a developer experience as possible, make it very simple and straightforward to use third-party packages, but also to provide the necessary tooling to do so effectively. Meaning, I think that's maybe a bit more of the things that we've been thinking about in more recent years and developing Composer further. So nowadays Composer has utilities like warning you about known security vulnerabilities in packages that you're installing. And just overall trying to improve the error output and just making it as straightforward as possible for you to use all this third-party software. And I think it's very important for developer tooling to kind of always show you a path forward so you can help yourself. And that's very much what we try to do with Composer, so that even in those situations where something doesn't quite work, that at least you have enough information available to figure out how to continue on with your work.
Scott Keck-Warren Scott Keck-Warren 4:19
Yeah, and it really is just like a really beneficial piece of, of the tooling that we have as part of the PHP ecosystem. So this is a free product, right? Composer is a free product. Um, you, as you are now providing, or you have been, I guess, for a while now, you have this Private Packagist solution. Can you talk to us a little bit about what that does and what problem it solves for people?
B
Speaker B 4:42
Right, sure. So Private Packagist essentially is a package repository, either as a service or self-hosted for businesses to use internally. The point is to both either have a place where you can keep your internal packages that you don't want to publish on the internet for others to use, but also to have a place where you can store a mirrored copy of any third-party dependency that you rely on on, so that you're not as dependent on that being available. The background behind starting that company was actually that we got to find a way to permanently sustain and finance the Composer open source project. Because as you mentioned, this has become such an important part of developing PHP that it would be problematic if there weren't anybody maintaining this anymore. And so building Private Packages as a company was the solution simply because it provides extra utilities, services to businesses working with PHP who are able to pay for those. And on the flip side, that means we can continue to provide Composer and packages.org for open source packages for free. And beyond just being a package repository, Private Packages tries to supply like a few additional tools, utilities that help you work with dependencies. So that is things like feature we call update-review, which automatically comments on pull requests that you make that modify this composer.lock file so that you get an easy overview over what actually changes in PRs. And you can make it part of your code review process to actually check into which dependencies have been updated here intentionally or not. Because often enough, when you're trying to update one dependency, it kind of pulls in lots of other dependencies. And it's very difficult to review this JSON lock file with lots and lots of lines changed. So that's like one utility that I think provides a lot value to businesses to get a better grasp of what their dependencies are actually doing and updates. But also other tools like security monitoring is something that we offer, which is kind of based on what's built into Composer now. So we added— and I think that's a core thing with what we always try to do with private packages is we try to identify the things that we can easily ship public open source for everybody to use. And then maybe there is some extra value that is not useful to the individual developer, that we can actually make part of the Private Packages product and ask for money for. So security monitoring takes the composer audit command to the next level, where it's not just checking the database, but instead we regularly do this for you and then notify you. So like we'll send you an email or we send you a Slack message or some webhook so that you can react immediately to new vulnerabilities being discovered in dependencies that you actually use.
Scott Keck-Warren Scott Keck-Warren 7:29
That's fascinating. Cause it's, so the thing that I really like, there was a lot there and I'm gonna have to unpack some of that. But the thing that I, that you mentioned that I found just amazing is that whole concept of it hooks into the pull request process. I can tell you what's going on there. Cause, cause essentially like when on our team, somebody adds a package, right. And it updates the composer.json and then it also updates the composer.lock file. And we just don't even look at that because usually it'll be, if we're lucky, 1,000 lines have changed. Right. Cause it's just, there's like, you know, it has, we added this package and it has dependencies and it updated other dependencies. So, like, I'm really fascinated about that. I'm definitely gonna have to look into that, like, soon. Can I ask you about this whole concept of a private package? So, I'm used to this concept of, you know, we have Packagist and Packagist provides public packages. And so, what is the workflow that exists? So, like, if I have a package that I want for my team, but we only want it to be existing inside of our product. So, so would we still have the same, it's on GitHub and then it's ho— and then Packages provides like an, a repository for that?
B
Speaker B 8:35
Right. So Private Packages really tries to work as like, as similar to what people are used to with packages.org. So you can hook this up to any Git repository. It could be on GitHub, but it might be on Bitbucket, GitLab, or some other Git hosting service that you use. And then it automatically gets set up with webhooks so that anytime you push code there, the new versions immediately become available for installation with Composer, and we simply provide kind of the composer repository, which is basically the metadata index for the versions that are available in that Git repository. Because otherwise, if you add— I mean, I think this works fine if you add like one Git repository to a composer.json, like you can do that like with a private Git repository. But once you start adding like 5 or 10, things start getting slower. Because the problem is that Composer will actually have to go and basically look at each individual tag in each of these repositories, and check the composer.json file inside of that tag to find out what dependencies each version has, so that it can figure out which of these versions are actually compatible with what it's installing right now. And just to get the metadata, it then ends up having to really do a lot of operations on all these repositories. So the point of having a package repository in that regard is simply that there is a central index that already contains all of this metadata. It's a single HTTP request, and there you go with all the versions and all the metadata. And so Composer can very quickly install packages from these private repositories. On the flip side, there's also the actual installation part. So now we're talking about metadata, but then there's the actual files that you're installing. And one upside of this is that you get an additional copy. So I mean, it happens that GitHub is unavailable, unfortunately, like last year, that happened quite a few times. And so you can actually still deploy because you have a copy of that file stored on private packages. And the way that this works with private packages is actually it for the lock file stores both download locations. So if private packages ever was unavailable, it could download the file directly from GitHub. So you kind of just get an additional copy. And you're a bit safer so that your deployment process isn't blocked just because like the code hosting utility can't be used to download that package right now.
Scott Keck-Warren Scott Keck-Warren 10:47
That's genius. I mean, like, I'm just— you're telling me this, I'm just flabbergasted that this exists, right? Because like, I'm like, oh, this solves some problems. Like, because we've had— we had that problem last year where where GitHub was down and we just couldn't deploy our software. Right. We just like, we're at a standstill because like, it just, it just wouldn't, wouldn't pull the packages. That's, I, I'm just flabbergasted. I'm so sorry. I didn't even know that this existed until just now. Um, well, so I got to, I got to go talk to my boss about this now, I guess. Um, that is, I'm sorry. Just really completely flabbergasted by that. That, that is, I think like that whole, it's that caching ability is just.
B
Speaker B 11:22
Right. So I think that's the important part. It's not just for your internal packages, but also for all the external ones. So I think an important thing that people don't necessarily realize about packages.org is that it doesn't store any of the code. It's really just a metadata index. So if you go and just use regular Composer require for some package, it doesn't download the package from packages.org. It downloads it from wherever the open source maintainer hosts it. So in, I think, nearly 99% of cases, that's GitHub. But on one side, there's the other 1%, where this is, who knows, some random server they're hosting in their basement. And so your deployment process suddenly depends on this random server that this person has set up somewhere working, if your deployment process requires you actually downloading and installing that package. Or like you said, GitHub might not be available. I mean, that's a relatively low risk. What's more likely is actually that some open source maintainer decides to delete their library, for example. They don't want to maintain it anymore, and they decide, we're just going to delete this version, or this package. And you're like, well, okay, sure, I have to replace this, but I still want to be able to deploy tomorrow before I actually have time to rewrite this myself or take this over or something. And so again, making sure that you actually have a copy yourself of all your dependencies is also a very important thing, I think, from a business security continuity perspective.
Scott Keck-Warren Scott Keck-Warren 12:45
Yeah. And I'm lucky enough, I guess, to be in the perspective of I have the ability to make purchasing decisions. for lack of a better term. And so, I do think it's important that we have these open source products, we have Packagist, we have Composer, but those can't exist without some sort of a funding model, which you guys have found, right? And not only is it a— not just a, hey, we need money to fund this, but we're actually providing a valuable service that businesses can help to reduce their risk, right? Because we couldn't push for a couple hours A lot, you know, like last year and, and that resulted in some amount of downtime on our server that actually costs us money. Whereas if we had just, if I had known this existed, this would have been in place and we wouldn't have had that, that problem. Um, so, you know, if you were a team lead, you should definitely be looking into this kind of thing. As you know, it does, it wasn't even that expensive, right? It's like less than €20 a month or something like that.
B
Speaker B 13:42
Well, it's a user-based model. So there's a base price of €59 a month that includes the first 3 users. And then it's about €17 for each additional user. And then there's some volume discounts too, if you actually come with a larger team.
Scott Keck-Warren Scott Keck-Warren 13:54
But if we have one person maintaining the package, that's just one user?
B
Speaker B 13:59
Right. So I think the important part is to realize that, yes, I mean, this doesn't mean that everyone that you have on GitHub, like your, I don't know, random other members of your team have to have an account. Who has to have an account are developers that actually run a composer update command and trigger this mirroring of packages. And then your automated tools is that there are these free read-only authentication tokens that you can use for your CI process, for example. So you're really just paying for the developers who actually actively work with the packages.
Scott Keck-Warren Scott Keck-Warren 14:24
Yeah. So it seems really reasonable. So is there anything else we should talk about with this before we go to the commercial?
B
Speaker B 14:30
I very much hope I covered all the important points of packages. I think the one thing to bring up maybe is just that we are still working on additional features like these, and we'll include them in this standard package. So that's like upcoming things or automated updates, things that you're probably familiar with at this point. I always recommend like Dependabot or Renovatebot and like other utilities available. And we're trying to really build something very specific to PHP that just solves a few things a bit better than what's currently possible with them for PHP projects specifically. And yeah, so I think just even for future, there's a lot more interesting stuff yet to happen to make working with dependencies in PHP easier and a nicer experience.
Scott Keck-Warren Scott Keck-Warren 15:13
Awesome. So we'll be right back with more after this word from our sponsors. Attention all PHP developers. Did you know that one of the best ways to improve your skills is to attend a conference? It's true. And have I ever got news for you because PHP Tech is returning for another year of leadership, expertise, and networking all in one event. It's a relaxing atmosphere for tech leaders and developers to share, learn, grow, and grow professionally while also providing you the knowledge to solve your everyday problems. You'll learn things like building for the PHP command line interface from Steve Grunwell and the dark corners of the SPL from Ami Adams. Join us April 23rd to 25th, 2024, just outside of Chicago, Illinois. Or online as all talks will be streamed live. If you can't make it during the conference itself, all talks will be available later for you to watch at your convenience. You'll even get access to last year's talks. Tickets are on sale now at tech.phparch.com or use my link in the description for a discount. And now back to the video. And we're back with Nils. So you're going to be one of the, the, you're one of the conference sponsors and you're also going to be one of the speakers at PHP Tech 2024 this year. And your talk is about the Composer Guide to Supply Chain Security. Can you give us a little bit of an intro to what that talk is going to be about to kind of get people interested in it?
B
Speaker B 16:33
Yeah, sure. So I think supply chain security is a subject that most people have at least heard of at this point. But I think to a lot of people, it's still a bit unclear what exactly that means specifically for them. Maybe you can even make sense of this in a broader context. But for my PHP developer day-to-day job, what exactly is actually involved And a key part of a supply chain for any PHP project is Composer, because it's the thing that helps you assemble your application, that gets you all these different things from different places. And so specifically to Composer, I'm going to try to give you a few tips, some information. I mean, something like I mentioned earlier, like just explaining to people how packages.org actually works, like which information Composer gets from where. Understanding that metadata comes from packages.org, but the installation source is somewhere else. And what types of things should you be doing because of that? So for example, a composer.lock file contains the URLs that packages are being downloaded from. And so for example, like a malicious takeover of a package on packages.org could keep the package as is but modify which URL the package gets downloaded from. That's something that would actually be visible to you in the composer.lock file. So if you were to review the lock file, you should be noticing, oh, like suddenly this is no longer pointing to GitHub but to some random domain I've never heard of before, and I might want to take a look at that. And kind of explaining like what are the risks and again, how far should you maybe build your own tooling or like what tooling is available. So sure, I'll also point out a bit like which of these things private packages can help with. But there's also other things that either other tools are good for or that are just things that you should be aware of and kind of make decisions for yourself on like what risk is acceptable to my business or kind of what are things that I maybe want to build a few automated parts as part of my CI process for. What types of things do I maybe just need like a manual process for, like things we want to review on a monthly basis and go through and to just generally educate people a bit on what are the risks involved in using third-party packages and Composer in a PHP project, so that people know where to look and have an idea of what to think about, what to consider. And it's not just this black box that happens to kind of install things without really getting like, where does that come from? What can I do to make sure that I get what I expect to receive? So I think that's the goal of the talk.
Scott Keck-Warren Scott Keck-Warren 18:56
Okay. That sounds really fascinating, to be 100% honest with you. I'm like floored every time I've been talking to like speakers for like the last couple months about what they're going to talk about. And every time I like talk to them, I'm like, oh, I want to go talk to see that talk. You know what I mean? It's just like one like quality talk after another. And I really do think like supply chain security is a serious problem, right? Like it's something that we definitely have to pay attention to as developers because we have that whole thing about like there's that XKCD comic where there's like like, you know, your application and then all the way down at the bottom, there's like some package developed by somebody in their basement or whatever, how that works. And like, that's, that's what we have here too, right? It's like some package could disappear and somebody could just jump in its place and, and cause problems essentially. I know that like, that's less of a problem with the PHP ecosystem because we like packages has stuff in place to prevent that for more or less. But I do know that like Node has had problems in the past where they they were able to inject, if you have this package, then it gets this package instead of that package. So it's always a concern. So that's definitely something that people should be aware of. And I'm really curious now. Yeah.
B
Speaker B 20:04
And I think even though maybe there are some technical measures that we do take to prevent this from being a problem, there are always still other issues that are more difficult to control, which again, I don't have solutions for, but I'm more advising people on what to look into. The package may not just suddenly get deleted, but if whoever the maintainer— we have enough packages in the PHP world that a lot of people rely on that are maintained by a single person. And if that person just decides to not do that anymore, like, are you— like, would you find out that there are no updates to this dependency anymore? Like, do you have a process in place for what you would do about this? And so I think that's also part of it is just kind of, yeah, get an idea of like, what are the things you should consider and think about when using third-party software?
Scott Keck-Warren Scott Keck-Warren 20:45
Right. And like, do we, do we know that everybody, all the packages that we're installing that all of the maintainers have good security hygiene and they don't, like their password on packages isn't, you know, password123 and somebody guesses that and then is able to take that over. Yeah. And I, and I, and I'm not at all like, this is a problem that like is just industry-wide, right? Like, I know I'm not expecting packages to be perfect for it. I do know that like, it's like nobody's gonna be able to go, we can solve supply chain. It's, it's like a pipe dream, right? But it is nice to hear that you all are putting thoughts into that.
B
Speaker B 21:16
Yeah, I think that's the important part is that, yeah, like you say, this is an industry-wide problem. And it's definitely one that's becoming bigger, and it's more noticeable. And I mean, as the importance of software has been growing over the last decades, and so has the importance of all the parts putting it together. And I think specifically the realization that supply chains for software have become so complicated have led to this realization that supply chain security is becoming more and more of a concern and issue than it used to be. But yeah, we're doing our part. We're trying to constantly monitor this, try to add features or add extra layers of protection to try and help people manage this, the risk that is inherent in having any sort of supply chain. And like you said, it's certainly not something you can solve, but I think it's a mix of providing the necessary tools and educating people about the things that they might have to do.
Scott Keck-Warren Scott Keck-Warren 22:08
Yeah, absolutely. And, and again, like, I'm, I like, I hope I didn't come off as sounding like, uh, overly critical or anything like that.
B
Speaker B 22:15
So, oh, not at all. Okay, good. Good. No worries. I was just trying to manage expectations.
Scott Keck-Warren Scott Keck-Warren 22:20
Yeah, absolutely. Especially for stuff like this, you know, where like, we're, it's all like free, right? There's so much free software and we have to assume that there's gonna be, there's a downside to that, right? Like, you know, it's, it's free to get it off of GitHub or wherever and it's free to use Packagist. And there are risks associated with that. So yeah, absolutely. So is there anything else that you'd like to direct our listeners to before we say goodbye for today?
B
Speaker B 22:42
Yeah, aside from like checking out private packages that we talked about so much, I'd really like everybody to take a look at the PHP Foundation, which we started 2 and a half years ago now, to find a new way of financing development of the PHP core, the language. And I think that's also something that businesses relying on PHP should really get involved in, help sponsor, to make sure that all of us together as the companies using PHP, make sure that PHP remains actively developed and maintained into the future. So please check out the PHP Foundation.
Scott Keck-Warren Scott Keck-Warren 23:13
Absolutely. And I think I unfortunately will have to have you back on to talk about that because that's, that's also another important piece of this, this whole ecosystem. So, so thank you for your time today. It was, it was nice meeting you.
B
Speaker B 23:23
Yeah, you're very welcome. It was fun talking to you.
Scott Keck-Warren Scott Keck-Warren 23:26
I have to say another heartfelt thank you to Nils for his time today and for your time for listening to this episode. If you're not a subscriber, make sure you subscribe with whatever your preferred method is and maybe share this episode with one of your coworkers so they can learn a little bit about Composer and packages as well. This podcast is available both in an audio version on the PHP Architect website as well as a video version on our YouTube channel. This is Scott Cack Warren for the PHP Community Corner signing off and reminding you to keep listening, keep coding, and keep reading.