Crawl, Walk, Run...Fly - Take your build and test pipeline to the next level!

The full transcript

Kavya (LambdaTest) - Hello everyone! Welcome to another exciting webinar of our LambdaTest Experience (XP) Series. Through the XP Series, we dive deep into a world of insights and innovation, featuring renowned industry experts and business leaders in the testing and QA ecosystem. I'm Kavya, your host and Director of Product Marketing at LambdaTest, and it's an absolute pleasure to welcome you to today's XP webinar.

Before we begin today's episode, there has been a situation where teams find themselves stuck in reworking the same logic because of a feedback loop, which in turn, takes more time than the usual manual process. I know most of you could relate to this situation, but what if I told you today is all about unraveling the secrets that top tech teams use to build and test their applications at a faster speed without getting trapped in a vicious cycle.

So without any further ado, let me introduce you all to our guest speaker, Nick Sanjines, who is a Senior Solutions Engineer at Bitrise. Nick is not just passionate about improving the productivity of developers and testers; he's been a driving force in the realm of build, test, and deploy automation. With a background as a former iOS and Android Developer, Nick has a wealth of experience helping organizations, both large and small, in maturing their automation practices.

In today's session, Nick will unravel the benefits of a fully automated, mobile-specific CI/CD process and will discuss the strategies employed by top tech companies on how they build and test fast at scale. So let's embark on this journey with Nick. Nick, why don't you get started and let us know a bit about yourself?

Nick Sanjines (Bitrise) - Awesome!! Thanks, Kavya. I will share my screen, and we have a presentation to go through. I've got a quick agenda. First, I'll introduce myself. And then we'll go into how these organizations I've been working with go and progress their automation, their pipeline journey. And there's a step-by-step approach. So if you find yourself doing nothing today, you've got some steps to to grow there, as well as if you're already a little bit down the pipe, how can you advance even further?

Let's get into that. First about me, I spent about a decade doing iOS and Android development. But this has been mostly in startup culture. So that means that although you're doing development work, you're also doing a lot of the testing, a lot of the automation, a lot of the pipeline setup and deployment that led to my career as a Pre-Sales Engineer.

And I love working with organizations, learning what they're doing how they can get better than helping them mature in their practice. Now I work with Bitrise. So just a little about Bitrise. We're a mobile-specific DevOps platform. That means we're the platform where you orchestrate your build and test pipelines specific to mobile - Android, iOS, cross-platform. It is cloud-based. It could also be hosted with Bitrise or a big cloud partner like AWS. So a lot of options there.

We'll cover a few things that Bitrise would help in some of the things that we'll cover. So as we get into the heat of the presentation, why do we care about speed? What is important about the speed of your developer pipeline, actually your automation testing?

The first thing that I'll mention is as a developer, I like to use a story with placing my car keys, because I think it relates really well. As I come home from work, I set my car keys down on the counter; if I were just going to grab a gym bag and then leave again to go to the gym, I know exactly where my car keys are. It's easy to grab them, and then I'm on my way.

But if I set my keys down and then my kids have some homework to do, and we go have dinner, and maybe I have a beer with dinner, and the next morning I'm frantically looking everywhere for my keys, and I have no idea where they are. That's only because a little time has passed, and I forgot where I put them the day before.

This is the same thing that happens to developers as they're working through building new features. I am looking through a code base of hundreds of thousands or even millions of lines of code. I find exactly where I need to make a change and how that change is impacting other components of your application.

And if I have to wait an hour, if I go into another step of logic, it takes me so much longer actually to go in and fix it if something is wrong. So it's imperative for developer efficiency to ensure that my pipeline is testing everything. But it's also fast. I want to ensure speed is super important in ensuring I don't lose contact.

I've actually had some developers that I've worked with say that it's easier, especially in our kind of working remote culture, it's easier for me to go do some remedial task like go take the trash or go to the store than it is for me to wait for a build by working on something else. So where do you find yourself today?

The goal is how can we get you to the next step? So I have a section here, you know, crawl. Maybe you're not doing any automation today, and that's okay. How can we get you to a crawl stage so that you can continue that journey? Maybe you're already parallelizing your build tasks, and you wanna get to the next step. How can we speed up those builds even more? Or maybe you're doing really great, and what else is there to do? What are top tech companies?

Doing that, you could take advantage of today. So that's where this story is headed. I'll start with a few basics for those who may not be familiar with this topic. What is automated CI/CD? Why is it important? You've probably been familiar with this image on the right going through and really testing as a whole, we want to test in each section of this pipeline.

But how do you do that? If you're manually testing here, it's very difficult. It's very time-consuming. So we want to automate this process, not just the deployment, not just the building and the testing, but automating the whole process. So continuous integration, when we talk about that CI, it's really the process of integrating those new changes into a branch. So we're automating, integrating that new into a developed branch. And then run through a series of your validation tests. This can be something simple like a smoke test.

As you progress, you may want to add more functional testing in this stage to give your developers and your testers the fastest feedback possible. And then we get into the CD portion of that. So once you've progressed enough, we want to deliver continuously. So as code is built, it is then deployed to your test staging production environment. So when we talk about CI-CD, we're talking about building and deploying in an automated way to your test team or out to production.

Kavya (LambdaTest) - Interesting, Nick, because I did have a question on this slide. So what role does continuous testing play when it comes to mobile-specific CI/CD pipelines? And how does it contribute to achieving faster deliveries?

Nick Sanjines (Bitrise) - Absolutely. So continuous testing is super important. It is so much more expensive to find a bug in production or a staging environment because of the amount of work that has to go into isolating it and then fixing it and testing it again. It's so much easier to find a bug. It's so much easier and cheaper to find a bug earlier in the process. That's why it's important to test at every stage. That said, the speed also comes from, like we talked about, losing your key story.

If you can get feedback to that developer immediately, even in an automated way, minutes from the time they wrote that code, they know exactly where they place the keys. They know all the logic that was going in and out of that function or that method, and they're able to make that change relatively quickly and get a fix out. And it's, you know, the cost of speed, the cost of the developer is almost.

Kavya (LambdaTest) - That helps.

Nick Sanjines (Bitrise) - All right, so we talked a little bit about this, but why would you want to automate your CI process? I think at the top of every company, you want happy customers. The fewer bugs that get into production, the happier they are, right? The better user experience. We're in a day and age where every organization is compared; their mobile products, specifically, are compared to top tech companies.

Right? I use applications on my phone from a variety of organizations, but I compare my experience from a new app to those experiences that I have on ones with thousands and thousands of developers. So, how can we make sure that we're following a process that ensures our product is top quality and best user experience so that we don't have to work at, or we don't have to lose customers due to bugs.

We mentioned this also increased the productivity of the development team because of that lack of having to context switch and firefighting, and this is again that key story that we talked about. This is the reason why the developers are more productive. Typically, I find that adding test automation and automating this whole pipeline will increase your developer throughput by 30% to 40%. So pretty massive speed gain in the features that your team is able to deliver to the end user.

And then lastly, a quick recovery from incidents. If you have a largely manual process from a developer's machine to your end user, there's just time to get to each stage of the process. If we can automate that piece from a button, click out to your end users, that automation just gets it out to your users much faster, so that maybe if there was a bug, you can lessen the impact of that, shortening the mean time to resume.

So how do you go about building this? With building web and middleware, back-end components, the restrictions are much less. Where it becomes challenging is when we talk about mobile. And that's why Bitrise is mobile-specific. Mobile has a unique need because Apple requires hardware to build iOS apps. And then, typically, your mobile developers, Android and iOS, tend to use Apple hardware.

So if you're building iOS apps, it's necessary to use Apple hardware. You could potentially reuse that same hardware to build your Android app, although it's not necessary. There's also some networking, rack space, and power, like just some of the basics that you would need if we're building a CI setup.

Secondly, you need a consistent environment. I have some teams that come to us, and they're building on a Mac mini under somebody's desk. And that works to start with. However, there's often flakiness. These are flaky tests. This is a flaky build. Something worked once, and the next time, it didn't work, or vice versa.

The way that you maximize this is by adding a virtualization layer so that you have the same environment every time. Right. So consistency is really important. This comes from also at scale, orchestrating a fleet of machines. So this may not be just one Mac mini on your desk. It may be 10, or 50, or a hundred Mac minis in a data center.

Right? Distributing and creating VM images for new versions of Xcode, for example, has become important, and those release cycles have become more frequent. When I was developing iOS apps, you had a new version of iOS maybe once a quarter. Now it's almost weekly or bi-weekly.

So this becomes another thing. Also, the life cycle of the machine, these machines aren't meant to run continuously. So how can we ensure we power cycle them, keeping them fresh so that you get a consistent environment? These are things to keep in mind. This is why you may not be at the crawl stage because there's so much involved here.

Another thing we can mention is a build agent. So something to orchestrate your builds so that when code is committed, we kick off a build on this machine. And then how can we invoke reuse throughout the organization at scale so that each team doesn't have to build something novel for every app. And again, lastly, at scale, this usually takes a team of people.

Most large tech organizations have dozens of engineers who are just focused on maintaining this infrastructure. So that's one way you can do it. That said, Bitrise is also here to manage all this for you. So everything I mentioned before, Bitrise does for you out of the box. And then you just pay for your usage.

What does this look like? I've got a slide here from the infrastructure perspective. Bitrise has all of the hardware that you would need, Mac OS as well as Linux, if you prefer. We also have all of the different versions of Xcode, and new versions come out within 24 hours of release. So you always have the latest and greatest of what you need to build, and then essentially just pay for what you use.

Lastly, predefined steps. As you'll see here, as an example, you want to build a workflow that runs your test, runs your Lambda test. We have those pluggable steps already built, and you drag them into your workflow, and you can run those tests. So a couple of advantages of having something like Bitrise is that those things are already built for you. It's not required.

If you're not doing anything today, definitely get into starting that automation practice, even if you have to build it on just a Mac server on your desk. All right. So next step, how do we improve on this? We talked about, first, why do we want to improve? As we build automation, the team grows and builds start to take longer. Maybe it's costing more time than just waiting for that one Mac mini to become available for the next team to utilize.

Or teams are saying we don't have dozens of engineers to spend on hardware, but we still need to ship all the features we've planned to our shareholders. So, how do we improve on that? That's where we get into a couple of these steps. So parallelization would be the first step. If we're building in a single-threaded way now, I have to compile my app, run tests, then send a notification to my team and deploy. Those tests are probably the longest poll in that barn.

I'm from Tennessee so I might use some weird acronyms, but the longest piece of that build is usually the testing. It's super important. Don't cut those out because then you're essentially driving blind. But it usually is the longest part of that bill. And so, how can we orchestrate the resources that we have to make that much faster? And that's where parallelization comes in.

So parallelization is essentially running jobs concurrently with multiple workers, and multiple machines that are communicating with each other. This involves some organization of your CICD process. It also greatly improves the wall clock time of your build. How long from the time that a button is pushed to the time the build is done?

It doesn't improve the amount of compute minutes that your build would take, right? You're just splitting that out across multiple workers. So as an example, we would have your Git clone, maybe install CocoaPods, archive and export, building your binary for your iOS.

Right! And then we would have our tests run. Maybe there are a hundred tests. So I need to get that binary, install it on, uh, install it on a device, run all a hundred tests, and then publish the results. Or I can run 10 tests on one machine and 10 tests on another machine all the way through, in this case.

I can run 100 on one machine and 100 on another machine. This, in essence is cutting your build time in half. So depending on how you want to set this up, parallelizing your build across multiple workers, the more chunks you can cut that into, the faster your build will become.

What does this look like in Bitrise? Essentially setting up your stages. In this case, we have a pre-check stage for linting. That would be the basics covered. Then we want to run our UI and unit tests in parallel. This could be split up into five workflows of UI tests. We could run some stuff with LambdaTest is part of that.

Maybe you have some other, just a unit tests that are running that can run, uh, you know, on the machine completely up to you what happens there. But I would highly recommend once you get to this stage, split out your tests, and split out your build functions that can work independently of each other so that the build is much faster. So that your developers don't have to wait as long and can deliver more features much faster.

And I think we talked a bit about this just now, but why would you do this faster PR feedback, that wall clock time? It's also easier to debug. If you're looking through flaky tests, it's much easier to look through a subset of 10 tests than it is to look through your whole test suite.

Also, if you do have flakiness, identifying that, being able to rerun the build of a flaky test, this is something that Bitrise allows you to do, is rerunning just the portion that failed so that you can identify those flaky tests versus having to rerun the whole thing is just cheaper. This is gonna be, like I said, the same usage cost but substantially faster wall clock time.

I had a customer I was recently working with. Their full build took about 50 to 60 minutes. We added some parallelization to reduce the wall clock time. I think it was down to 12 to 15 minutes. So substantial speed gain in terms of build time just by splitting these out on parallel workers.

Kavya (LambdaTest) - Interesting. So, Nick, I wanted to double-click on that and understand, you know, how do flaky tests affect the overall efficiency and effectiveness of your testing pipeline, and how do you quantify the impact on the overall development cycle?

Nick Sanjines (Bitrise) - Great question. So the first thing that I'll mention there, with a flaky test, well, sorry, before I get there, as a developer, and a tester, automating or doing a manual process is the most robust plan you can do. A developer checks in code, and a tester will manually go through every functional requirement or test in the test plan. And then, when that's finished, the app can go to the next stage.

So it's rock solid. There are rarely things that get by, which is why that method was used for so long. When you start automating, there may be some challenges where a test is supposed to work, it should have worked, and it would have passed if somebody was manually testing it, but it fails in an automated way. And this is super dangerous because it causes developers and testers not to trust your pipeline. Ignore that. Ignore that failure, right? That can lead to bugs getting out to production because we ignored a test failure.

So identifying this can ensure that everybody is confident in the build pipeline. We're confident that if a build fails, it's because of an issue, but it can be incredibly time consuming to track down why did this test fail? What is causing this to fail? So identifying flaky issues.

We talked about parallelizing and branching these out so that you have small buckets of tests run separately. I can identify this machine had a failure and it had a failure because of this test, and when I re-ran it. It worked, so I needed to identify what was causing the failure of that test. And either pull the test out because it's not doing what it should or rework it so it passes 10 out of 10 times when it's supposed to.

So the impact that has on the whole team is the trust in the automation and the ability for a developer to know exactly when something failed, that it's on them to fix it, not on the QA team to rework the test. Does that answer your question?

Kavya (LambdaTest) - Yeah, absolutely. Thank you so much.

Nick Sanjines (Bitrise) - So we've done our crawl. We got started on building automation. We started then with the walking. How can we speed up those builds even more? What's next? The next logical step to building fast pipelines is setting up a dependency cache. So we'll talk a little bit about what cache is in general, and then how do we apply this to our builds? So first, cache, in general, vaguely or more generally, is software used to store something temporarily, mostly just in computing, right? It's faster, it's more expensive, but it's used more frequently.

And so it's something that I want to access quickly. I use the example, like a phone number. I grew up in a time when I had to remember everybody's phone number if I wanted to call them, right? Now, I don't even know my kid's phone number. It's just a contact stored in my phone. Right? But even then, I have people that maybe I contact more infrequently, and I have to go search for their name and, you know, to be able to find their contact information to send them, you know, call or text them.

But people that I contact more frequently, like my kids, they're a contact in my phone under favorites. And so it's one button push to contact them. It's the same thing with cash. I find something that I need to use a lot. It doesn't change very often, but I need to use it a lot. So let me store this in a known place so that I can get it as quickly as possible every time I need it. That's what this cache is all about.

So key-value cache, really, all caches a key value, but essentially it's assigning a unique identifier. In this case, it would be a key to differentiate it from other bits of information so I can more easily find it. The contact of my kids has a picture of their face by it so that I can easily identify who this belongs to and access it as quickly as possible.

And then what gets cached is anything, like we mentioned, used frequently and doesn't change much. This is all about task avoidance. I don't want to rebuild something or redo something that I have stored over here. This can be for build tools, it can be for dependencies, or some configuration, things that, again, are used frequently and don't change much.

Kavya (LambdaTest) - Now that you were on the topic of run, I wanted to bring up, and talk about scaling a bit. So when scaling, how do you sort of maintain a balance between the speed of the feedback loop and the requirement to deliver through, along with the passing of actionable feedback to developers?

Nick Sanjines (Bitrise) - So, balancing this is a very good question because the more you grow, the more needs are placed on a solution like this. Right, so a developer that is building only on their local machine or on a Mac mini on some of these desks is they needs less because they're not yet more mature. And the more you grow, the more you scale, the more imperative some of these solutions are, or else the developers will have merge conflicts all day and nothing will actually get delivered.

So I think this is important to set up in a way where you can; this is a big reason why Bitrise exists as a company, is we wanna make sure that you can scale without the heavy investment in tens or hundreds of machines and setting up all the infrastructure and the company knowledge that you would need to set this up the right way. How can you lower that bar to entry? Because it's super important to scale so that you're not holding your developers back.

We want to make sure, I know there's always this conflict, developers want to move fast, and testers want to ensure we're delivering a quality product. And sometimes those things are at odds. Scaling in the right way involves getting both of those things. How can we let developers be fast and have QA ensure that what's being delivered is bug-free?

Kavya (LambdaTest) - That's it.

Nick Sanjines (Bitrise) - So why would we add this next stage of caching? And I had a little picture to show kind of where we are at each stage. When you get to the crawl stage, we're going to do everything serially, one after another. So I have a few steps here. They're not all respective of their size. The Git clone may be super short, you know, maybe we're doing some installing some dependencies. I will run my Android tests. I'll build Android and run the functional, uh, like end-to-end Android test.

Then I go about doing my iOS build, uh, right. Cocoa pod, uh, unit testing, build my binary, run my end to end testing. Then I deploy it to the different app stores. This whole process may take 60 minutes. This is an example. I've seen organizations where this takes much longer than 60 minutes.

So we get to the next level of maturity, which is the walking phase, right? I still have two processes. My testing still has to happen, although I'm splitting it across parallel workers. I also have an Android pipeline and an iOS pipeline. So I'm doing those things in parallel, right? So you see, I have an Android release pipeline and an iOS one here.

And then lastly, the next phase would be how can I split out those dependencies. How can I ensure that my dependencies are not rebuilt every time, but I'm pulling them from a cache? That's where the KV portion of the slide comes from adding that key-value cache. So I don't have to wait for that. And you can see the respective speed improvements you would expect from those components, right? So now we're at a much more mature section. But we have another step, which we'll discuss in a moment.

Now, this could be a little challenging to build on your own. This is probably something that I would say utilize a vendor that does this. Building this on your own can get expensive. If you're just the developer, take the time to build it for all the different scenarios you may have. Most organizations, for example, are if you're gonna use Android and iOS, you're going to have a Gradle dependency cache.

You will have maybe CocoaPods and SPM together for iOS. Like there's a whole lot of options here. And then, if you are building like a React Native app, that adds another layer of dependencies that are needed. So utilize this from a vendor. It would make your life a lot easier there. OK, so the last stage, the last stage, how can we improve even more? And we'll, we'll move through this pretty quickly. I'll give you some examples first.

So why would we want to push this even further? We've gotten our bill down to 15 minutes. That's great. Developers are happy. Again, the faster we can go through these things, um, even at scale, because the 15 minutes was an example. As a simple example, I have some organizations I'm working with that. They're getting to the run stage is still 60, 70-minute build types. Um, they have a lot of tests happening. There's a lot of artifacts being pulled from locally hosted servers and those things take a long time.

As your development team, as the lines of code grow, your build and your tests grow exponentially, right? For every feature, you may have a dozen tests that are written. And so those grow exponentially. As you scale, sometimes that day-to-day job satisfaction is low because you're not making as much of an impact as you were when the team was small. Productivity is dropping because you're dealing with more things as a part of that build.

Obviously, organizations are looking to optimize. As I mentioned, I have some teams that get to this stage, and their build times are still excruciatingly long. So, what are teams doing to combat this? There are a number of tools that you'll see in this stage. And I'll give my opinion on some of these tools, depending on where you're at. There are some build-cache components. XCRemoteCache is one of them. This open source was developed by Spotify. It's like a plug-in that helps you reuse artifacts in CI.

It's not as well maintained anymore. I think Spotify ended up using a different tool now, but the tool is still usable, and still beneficial, depending on where you're at. The tuist is another, similar to XCRemoteCache. It's a command line tool to help your interaction with Xcode at a scale that is super simple to set up.

They were doing really well for a while, and in the last few months, they actually closed-sourced their application, which is super dangerous for the developer community. It means you're locked in, like vendor lock, which is scary. So, at present time, I don't recommend using that unless you're okay with being locked to that.

Gradle is a build automation tool. It is bundled with Android Studio. So, if you're building Android, you probably are already using Gradle. It comes with it. Open source tool. It's great. If you're already using that and looking for how we can advance, I have a couple of slides on why we would do that. Buck is a build system for iOS and Android is developed by Facebook. It wasn't very extensible. They're actually doing a rewrite currently.

So if you're already using this, great. Know that it's coming with some extensibility in the very near future. I wouldn't recommend moving to this currently just because of a lot of the changes that are happening. And Bazel is probably the most used tool in this space. Developed by Google, it was an internal project that they called Blaze, and then they changed the letters around to make Bazel. It's an advanced build system.

Teams use this to build iOS, Android, middleware, backend, and everything. This is the build system that they use for everything super extensible, heavily used and maintained by some of the largest tech companies. If you're at this stage and looking at how we can better manage our build system in a better way, this would be something I'd do.

Why do we want to look at an alternate build system? Well, if you're, if you're like, well, I use Xcode build, I use Gradle. Why would I want to? I'll tell a couple of stories here from the setup. So have a build we can use as this works with Gradle with Xcode build everything right part of my build, I have multiple projects right, and for each project to build each project I have a number of tasks, and each task is broken down, I have one or multiple inputs.

So, at a very basic level, what an alternate build system does is say, if the inputs didn't change, I can get the output of that build and not have to rebuild it. And it's much faster for me to pull that from a memory cache than rebuilding it. So the guiding principle, as you can see here, the best way to work faster is to avoid doing it if you don't have to. This is commonly known as building without the bytes. So that's where BuildCat, this is this philosophy came from.

Obviously, if an input does change, we do need to do something; we'll rebuild that. And at scale, you can see where this becomes really valuable. So, I have hundreds or thousands of tasks and inputs. If one changes, I don't have to rebuild everything. I can just rebuild a portion of it. And so, at scale, this makes a lot more sense as it requires some additional components to do effectively.

So, a remote cache is gonna be needed to be used by your team of developers and your CI system so that you can share those build outputs. You want your CI system to take advantage of it, but your developers can also take advantage of it. So, what does this architecture really look like? Right, you have your CI components here; you have your local developers. They're both feeding a build cache, which is central to your developers and to your CI. And we can download what we need and build the rest. And then obviously writing new outputs to that cache to be reused later.

So architecturally, this kind of shows you what it would look like. It is you will have exponentially faster build times. Those teams that I mentioned that have 80 90-minute plus builds have their builds dropped to like three minutes, two minutes, including the tests that need to run, including the parallelization that we mentioned because some things don't need to run over and over again. So, it's super consistent and reliable, especially for large teams. So you will have 10 out of 10 times the, the build will succeed if it's supposed to, and it will fail if it doesn't need to.

However, there are some bottlenecks that need to be addressed. Speed is the name of the game when it comes to this. And latency of retrieving cache entries, for example, can be important. So geographic distance from you has a team that's globally dispersed. We need to make sure that they all have the same speed impact and that they're not equally able to take advantage of the cache. The read speed of the cache becomes important, as well as a network protocol.

These are all things that we can cover in greater detail if you have questions about them. Happy to go into it in more detail. But for now, I'll leave it and say this is something that Bitrise does. So, we have a remote cache in the same data center as your CI builds. That's going to give you much faster builds.

We also have a CDN infrastructure that gets the points of presence of this cache close to where your developers are. Also, gRPC streaming is much faster and more economical compression, so this cache doesn't have to cost you an arm and a leg. And just like your CI system, it's fully managed.

Kavya (LambdaTest) - Thank you.

Nick Sanjines (Bitrise) - The best thing about this is that you are taking advantage of it. This, to build this cache usually takes months and months of engineering work. With Bitrise, it's adding a step into your build. So adding a step into your build, activating the build cache, and your builds would go from, you know, in this example that I had a very simple app that build was taking about 10 minutes down to three minutes. So as long as you're having a cache hit, we can see that a couple of times we had a cache miss, and the build took as long as it normally would. Maximizing those cache hits is gonna be great for the whole team.

So I guess I will wrap it there, but thank you for going through and seeing how we can get from no automation to crawling, walking, running, and getting the most speed so that the developers and test team can deliver quality products as fast as possible.

Kavya (LambdaTest) - Thanks Nick. Before we wrap, I have a couple of questions. So yeah, you spoke about remote cash. I wanted to understand how you currently optimize local builds and tests to ensure a swift development process. I mean, when should organizations consider automating?

Nick Sanjines (Bitrise) - Yeah, so local cache becomes really important at scale. If you are probably in the 10 to 20 developer range, I don't think local cache makes a lot of sense. The cost of that data to local developers is probably not worth it at that scale. I would say for those teams, you're not having enough dependency conflicts between developers where it makes sense for the speed.

So I would say dependency or build-cache is super important for your CI to take advantage of, probably not for your local developers at that scale, but beyond those numbers, when you start getting into the hundreds or thousands of developers, these conflicts necessitate a local build cache. And the way that we solve that is by getting points of presence in the city centers where your developers are to make sure that it's as fast as possible.

Kavya (LambdaTest) - Great, thank you. So essentially, based on the size of the organization or the use cases that they have to solve, they need to sort of choose the path forward. Moving on, so wanted to understand how you address and minimize bottlenecks that could otherwise slow down the developer feedback loop and release timelines.

Nick Sanjines (Bitrise) - Great question. So from a bottleneck perspective, these are, there's, depending on what we're doing, specifically with cache, there can be a lot of bottlenecks. Latency and geographic distance are big, so like the network, there are a lot of points.

When you talk about pulling in big chunks of data, maybe your organization's network could be a bottleneck, right? So there are a lot of things to consider. So the protocol that we're streaming over, the GRPC, would be like, it's the way that like Netflix does streaming to optimize your streaming protocol without completely crushing your network, right? So there are some options there that are super important.

Obviously, we mentioned some other bottlenecks in terms of distance from the cache. I think Gradle did a study, it's been probably a year or two now, but if you had, even in the US, if you had a cache, location in US West. Your West Coast developers are great. Their distance from that cache is, you know, they're right there. It's very fast. Your East Coast developers are almost as slow as if they didn't have the cache at all. And if you're in Europe or India, your builds are actually slower with the cache than if you rebuilt it. So, these are huge bottlenecks that can happen with globally distributed teams that necessitate a CDN to prevent those bottlenecks.

Kavya (LambdaTest) - Very interesting. Thank you so much for sharing that. Coming on to the last question of the day, what are the key advantages of prioritizing speed in builds and tests for mobile applications?

Nick Sanjines (Bitrise) - Yeah, I think this is what this presentation is all about. How do we make sure speed is important? And I'll go back to the key story that we had mentioned. Making sure that your developers or testers have the context of what's going on so that they don't have to go find that needle in a haystack over and over again is why speed is so important. It enables your teams to move more swiftly in the build process.

It also helps to minimize the number of bugs, not just get caught, but minimize the bugs that happen at all. Because I'm able to pay attention to what I need to do, and when I'm done with that thing, I know that it's good, and I can go to the next task. So it allows your teams to move more swiftly and make sure that the product gets all the testing attention that it needs to move confidently to the next stage and that to your customers, which is ultimately why we're building these products.

Kavya (LambdaTest) - Thank you so much, that really helps us. So, we are at the very end of the session. I just wanted to take a minute to say that I loved your storytelling skills and use of means, absolutely could relate to everything. It was amazing to follow through.

So as we wrap up today's session on this insightful journey of optimizing build and test pipelines. I would like to thank Nick for sharing his expertise and guiding us through this enriching experience. And to your audience, I hope you have gained knowledge and will implement them in your journey. Thanks for joining today's session.

Stay tuned with us for more XP Webinar Series, where we would continue to explore and elevate the latest cutting-edge technologies. Until next time, keep innovating and happy testing. Thank you so much. Thanks again, Nick.

Nick Sanjines (Bitrise) - Thanks, everyone!