Scott Keck-Warren
0:04
Hello developers and welcome to the PHP Architect Community Corner where we have conversations with members of the web development community. I'm your host Scott Keck Warren and today we're talking with Mahmoud about Postgres and Neon. Mahmoud is a software engineer focused on developer tools. He works at the intersection of product, marketing, education, and community. He is currently a developer advocate at Neon. Thank you, Mahmoud, for finding some time to talk to us today.
Thank you for having me. Pleasure to be here.
Scott Keck-Warren
0:30
Absolutely. So I wanted to have you on today to at least to start with, to talk about what Postgres is and why people should be using Postgres over like some of the other options.
Sure. So Postgres is open source. Over the past couple of years, it's kind of become the default nowadays. It's like if you're building an app and that means a database, you most likely can't go wrong with Postgres unless you have like a very specific use case where you might need like a specific type of database. But honestly, Postgres is really good at a lot of things and can get you pretty, pretty far. So yeah, because it's open source, it's also extensible. So there's a big community of extensions which allow you to kind of customize the behavior of Postgres, making it very powerful and flexible. And yeah, there's a lot of knowledge that's shared about Postgres. So, you know, if you choose it as your database, well, you're going to have a good time. You're not going to feel like, oh, I'm this person who's trying out like a new tool and like, oh, you ask a question and then you don't see an answer or like an abandoned GitHub issue or anything like that. So yeah, that's kind of why it's pretty much the default nowadays. And I would say, unless you need something specific, you're fine with Postgres as well as like its relational model, which is pretty robust and has been around for pretty sure more than 30+ years, maybe even 40. So yeah, that's pretty much it.
Scott Keck-Warren
2:02
Okay. And so would you say one of the big— so I know that there's this big schism with the MySQL people and everything else, right? Because there's MariaDB and there's Postgres and those are very heavily— MySQL has a corporate overlord essentially, and people don't always like that. And I think Postgres is— I've seen a lot of people talk about how it has all these cool functionality to it, but, and it's not being held back by, um, by the, uh, by Oracle essentially. Is that, is that kind of like one of the, one of the main reasons that people also look at it or?
I would say so. I would say it's just because like you can just self-host Postgres, you're good to go. Like there is, it's really truly open source, uh, which I think is very compelling. So.
Scott Keck-Warren
2:44
Oh, cool. And, and so could we use it as like, just like, so I, I always look at it as like my default is MySQL when that's probably like the bad way to look at it because for years that was like my default. And now I'm looking at like other alternatives because I have time to do that. And so, is it truly like we could just like drop in Postgres in— for— to replace like MariaDB or MySQL? Or is there a little bit more involved in it?
I mean, I guess it depends on how you're using each database. I don't personally have a ton of expertise in like other flavors. I've been lucky enough for Postgres to be my default. So, I was like, oh, cool. It's what I chose and kind of like now what a lot of developers are choosing, which is awesome. But again, I think all of the other options are probably fine. They work, but it's like you can't go wrong with Postgres. That's, I think, my personal, at least, opinion. So yeah.
Scott Keck-Warren
3:40
So is there, is there a feature in Postgres that like you really particularly like?
I really like, so recently with all the AI hype and things, you know, everyone is building like OpenAI OpenAI wrappers, and you essentially want to store vector data in Postgres, and that's possible using the pgvector extension. So it's an extension which was like released by a community member, and it adds custom data types for vectors. So you can store vectors and query them and find like the most similar vectors or the not so similar, which just makes Postgres honestly a database you can use when building AI applications, which is very convenient. So if you want to add any fancy feature like semantic search where you kind of search by the meaning of the query rather than doing keyword matching. You can easily do that with Postgres and pgvector. So I would say if I had to choose something, that would probably be it, but there are some pretty other cool use cases as well.
Scott Keck-Warren
4:39
Okay. So that sounds good. So normally we would put stuff out into a full-text search. Does it have the ability to kind of do full-text search?
Yeah. It also has full-text search capabilities. Yeah, that is true. And you can do pretty cool stuff where you kind of do a hybrid search, where you do, uh, keyword-based matching, but also as well do like, uh, semantic search. That's, that's pretty cool. I haven't, uh, had the chance to deeply kind of explore this, but I, I know it's possible.
Scott Keck-Warren
5:06
Oh, that's really cool. I, I had to look into this more. I, it's one of these things where like, I was like, I need to play with this more and see like what the functionality is, cuz I might be able to use it for like my day job and for side projects. And, um, it is really like fascinating to hear. Like about those kind of things that, that people like about it. Um, we'll be right back after this word from our sponsors. Do you want higher clarity in production, but don't have the time to earn your degree in observability? Me too. Forget logs, metrics, and traces. Honeybadger Insights is built around structured events. When you send your application logs and other events to Honeybadger, you'll unlock the power of Honeybadger's powerful new querying language, BadgerQL. You can then use BadgerQL to ask any question about your data. Convert any event into a metric, and chart your metrics on a custom dashboard. You can do all of this on Honeybadger's free plan as part of their comprehensive monitoring suite, which includes error tracking, uptime monitoring, status pages, and more. Speaking of error tracking, did you know that an error is really just a first-class event in Honeybadger? In fact, you can use Insights and BadgerQL to explore all of your existing Honeybadger data in new ways. It's pretty cool. Give it a try today at honeybadger.io. That's honeybadger.io. And we're back. So you work for Neon, which is a cloud-based Postgres provider. Can you talk to us a little bit about the services that you all provide?
For sure. Yeah. So Neon is fully managed Postgres. You can have Postgres, it's open source, and you can self-host it on your own and it's, it's cool. But typically you might want to offload that type of work to a managed service, which gives you a bunch of features out of the box regarding like high availability or automated backups. And it's like, it just takes care of a lot of little things that you will most likely do anyway, regardless of the app that you're building. So yeah, I would say it's like the first pitch would be like, oh, it's managed Postgres, but it's a little bit more than that. And the fact that with Postgres, like just, we get like, I will refer to it as like vanilla Postgres, which you can kind of self-host. everything is just in one box. So it is a monolith, you have a server and the server, you have all the different components in the server, all in one box. So you have your compute, so RAM and CPU, and you also have your storage, so disk. And the thing is, whenever you want to provision a database, you'll typically have to pick a size for the database upfront. How many resources do you need? And After you're doing that, after you do that, and let's say you need more resources or you need less resources, well, the process is kind of like not very automatic. And at the same time, because of Postgres's architecture, it doesn't take advantage of like modern cloud infrastructure where you can actually pay for resources on demand and only like kind of pay for what you use. So what happened was a new architecture emerged, which is based on the idea of separating, separating storage And compute, and several database providers kind of came up with their own solution, but none of them were open source. So that's kind of like how Neon was born. It was the open source Postgres database, but with a different architecture where you have the storage and compute components separate. So this surprisingly unlocked a bunch of features that are pretty mind-blowing. So one. With PostgreSQL NEON, it automatically scales up or down. So you can kind of set a minimum or maximum size for the resources and resources will be dynamically allocated. So you don't now, you don't need to do this kind of capacity planning exercise where you have to try to guess how much storage you need, how much compute you need. So you don't end up in a scenario where you either over-provision, so you're paying for extra resources that are unused, Or you under-provision. So you're like, oh, your database can't handle the workload. So now you're kind of running into these bottlenecks. So I would say this is kind of one of the features. There's another feature where, well, this is kind of— dives a little deep into how you build a service. And one of the challenging things is the fact that Postgres is stateful. You have a server, you have state, you have the data. So if you ever want to manage that or allocate the resources, things are— they get a little bit challenging. So if you have a compute-heavy workload, let's say you're doing lots of queries, things that require a lot of memory or CPU, and you need more than that, then you will need to upgrade to a bigger box essentially. But because when you have the separation of storage and compute, what happens is you can be like, oh, we just need more compute. So that is an advantage. But more than that, because now what we do is like Postgres is like, you get the same experience when you're querying Postgres, then we actually have the ability to make Postgres available for you instantly. So if you think about it, well, because we have Postgres in like Postgres when it's on Neon, which is also open source, it's stateless. So we're able to essentially just spin up very quickly. So if, for example, you try spinning up a managed Postgres database with any other service, typically it takes several minutes, but with Neon it takes like a second. So I would say in a nutshell, Postgres on Neon autoscales, it instantly provisions, and that's kind of like the innovation side of the compute. But I think you might be familiar with Neon's branching feature. And that's kind of innovation on the storage side of things. So there's actually an interesting blog post that we have in the Neon blog. It's actually one of the first blog posts ever, and it's called Architecture Decisions in Neon, I believe. But the idea is essentially talks about design decisions that were made in Neon regarding actually architecting a service where Postgres can take full advantage of modern cloud infrastructure. And one of kind of like the innovations other than the compute side is actually on the storage side. So with Neon, you still work with Postgres the same way you work with Postgres you know and love with all of the tools that you're familiar with. But we have actually a custom storage layer because, well, since they're now separate, well, you can kind of get to innovate on that layer. And there are a bunch of features that are pretty much unheard of when you're doing traditional Postgres. And one of them is branching. And with branching, you can— I think it's kind of hard to explain, you kind of have to see it for yourself. But we ultimately make it easy for you to create isolated copies of your data. And you might be wondering, okay, why does this matter? Well, typically in your software development lifecycle, you will be using a database at every stage. And when you're developing locally, when you, if you have like a preview environment, or if you're doing any testing, you're going to need a database. And typically what you do is you seed this database with some data that's supposed to kind of mimic what you have in production. Well, what we make possible is essentially if you have a database, and let's say this database, you kind of tailor the data for, let's say, a staging environment, Well, you can create copies of this database and use a new workflow. So you can literally copy, let's say, I don't know, a terabyte. And this terabyte, you just have it in an isolated instance where you can essentially test things. So it's called branching because you can kind of imagine that you have your branch in Neon and then you kind of like create a separate branch and they kind of diverge at the point of creation. And that's kind of how you have this isolated instance. And you can think of it really as this maps to having a database for every environment, but the database is ready to use. So you don't need some sort of seed step. It also provisions instantly. And that I would say is, in my humble opinion, of course, I mean, I work there, but it's a very cool feature. And once you try it, you really can't stop using it because it's like the actual experience of you working with the database is just 10 times better. Um, but yeah, that's kind of like a Neon and I don't know how many minutes of that, but that was kind of like the Neon pitch pretty much.
Scott Keck-Warren
13:43
Yeah. And so I signed up for an account and I was, I was shocked. Like I was like, I like just like entered my information and I called the name, like my database and it was done. And I was like, I was like, okay, well, where's the next step? Right. I was just like, there must be another step in order to get like this thing going. Uh, but it wasn't, it was just like, here's your database. It was, I was surprised at that. And the other thing I find kind of interesting is like, it seems like it, like from this auto scaling, it scales all the way down to zero. So like, I'm not using it right now and it's not doing anything. And so it's not costing me anything. It's just sitting there in case like somebody comes along and says, hey, I need to use the database. And then the database will spin back up. It's really like, especially for, for me who has like applications that are used heavily between like 8:00 AM and 5:00 PM in a specific time zone, that could be like a huge cost savings. Right. To just go like zero. Yeah.
That's actually also another thing. So it, you can't, it actually shuts down when not in use. And of course you can customize this behavior if you want. But the idea is, well, if there are no incoming queries, depending on a specific time period, the default is 5 minutes. Well, the database just shuts down and whenever there's an incoming query, it spins back up and usually it takes like less than, or I'd say around 500 milliseconds, sometimes less. So you technically won't feel Uh, this sort of like, uh, slow performance only for like the first query. Um, but yeah.
Scott Keck-Warren
15:01
Yeah. And, and so from, from a branching, can I go back to the branching for a minute? Um, so, so when you, and I haven't played with this enough, I guess, to figure this out, but so when you, when you branch your database, is it making like a full copy of the, of the database or is it just recording the differences between them?
So, uh, it actually uses copy on write. Uh, so kind of like you, what you hinted at, it's like we keep track of like, essentially when you create a branch, you create another pointer in your storage and we keep track of the delta between the different branches. So this means that's why it's super fast. And I would say it's kind of like, it makes sense this way. And that's what we're like, that's why you're able to copy any database in like a second, regardless of the size. Like we have a video on like the YouTube channel that I like a while back where I copied like a 1 terabyte database. And I would copy it. So I would create a branch. I would drop the database from this branch. And then you would see like the actual main branch didn't, like wasn't affected. Which means you can essentially use this workflow to test any queries you might have or migrations or yeah, that's kind of like, or use it like in development. So you have actually like, you can have actual data to work with. You don't need to write a C script by hand, update it, all this stuff. You can just create a branch, which is super convenient.
Scott Keck-Warren
16:24
Yeah, I did. So like, in my day job, we have like a staging environment that's a copy of the production data from like, you know, within like the last week. And, um, it takes like forever to, to, to restore that. And it would be so cool to just be able to like reset, you know what I mean? Cause like sometimes it's like a half a day to reset the staging environment. Whereas like this would just be like, it would literally be like less than a second, right? It would just be like, remove what was there, create a new one. Yeah. That's really cool. Yeah. So do you have anything else that you'd like to direct our listeners to before, uh, I say goodbye today?
Um, I would say check out Neon. Uh, it's at neon.tech. Uh, you can again sign up for free or you can actually just go to like pg.new. This is our fancy, uh, short URL. It will take you through the same thing. Uh, but it's like, it's easier to remember. So yeah. Uh, go try it out. And if you have any questions, we also have a Discord. Uh, I hang out there all the time, so you can reach out, ask questions. And yeah, that's very much.
Scott Keck-Warren
17:19
Awesome. Thank you for your time today.
Thank you so much. Glad to be here.
Scott Keck-Warren
17:24
I have to say another heartfelt thank you to Mahmoud for all of their time, and thank you for listening to our podcast. If you're not already a subscriber, make sure you subscribe with whatever your preferred method is so you can get episodes as soon as they're released. Maybe share this with us, with a coworker so they can learn about it as well. This podcast is available both in an audio version on the PHP Architect website as well as a video version on our YouTube channel. We also now have a Discord which I will link to in the description below. This is Scott Kirk Warren for the PHP Architect Community Corner signing off and reminding you to keep listening, keep coding, and keep reading.