Dynamics Corner

Episode 334: In the Dynamics Corner Chair: Business Central Experience - Security, Scalability, and Updates

• Dmitry Chadayev • Season 3 • Episode 334

Share episode

In this conversation, Kris and Brad join Dmitry Chadayev to discuss everything Business Central Online, including  Security, Scalability, and  Updates. Dmitry provided valuable insights into applying updates and hotfixes to the online service and the measures in place to ensure optimal performance and resource allocation. They also discuss the scalability of the database and compute capacity and the mechanisms to maintain fairness and prevent resource exhaustion. In this conversation, Dmitry discusses the benefits and features of Microsoft Dynamics 365 Business Central, the Business Central team's passion and dedication, and the product's value.

Send us a text

#MSDyn365BC #BusinessCentral #BC #DynamicsCorner

Follow Kris and Brad for more content:
https://matalino.io/bio
https://bprendergast.bio.link/

Speaker 1:

Welcome everyone to another episode of Dynamics Corner, the podcast where we dive deep into all things Microsoft Dynamics. Whether you're a seasoned expert or just starting your journey into the world of Dynamics 365, this is your place to gain insights, learn new tricks and how it all works in the back end. I'm your co-host.

Speaker 2:

Chris, and this is Brad. This episode was recorded on July 26th 2024. Chris, chris, chris, what an amazing discussion. My mind is overflowing with all of the information we learned Today. We had the opportunity to learn more about the Business Central performance, online backup, security, performance and partner tools, and much more. With us today, we have the opportunity to speak with Dmitry Chaydev. Good morning, gentlemen.

Speaker 3:

Good morning, good afternoon, how are?

Speaker 2:

you doing.

Speaker 3:

Yes, good time of the day yes.

Speaker 2:

Excellent time of the day and it's Friday.

Speaker 3:

Nice background bread there.

Speaker 2:

Yeah, you know, chris has been sitting outside the past couple discussions that we've had. So I figured today, since I'm, you know, in the northern part of the country, I might as well sit outside and appreciate it as well. It's finally a nice day out.

Speaker 3:

Fantastic Nice no great to see you, great to see you. Yes, nice to see you. The weather is great.

Speaker 2:

Yes, yes, yes. How was your vacation Perfect. Yes, yes, yes, how was your vacation Perfect perfect.

Speaker 3:

I spent the whole three weeks out there in Spain, with some of that time with my parents. So incredible to meet everybody. My brother lives there in Spain so we had some really good time there together and it's a nice area I can definitely recommend. It's not too far from Valencia, so the coastline is beautiful. Small towns along the coastline, nice mountains. For hiking I mean great area, so I could most definitely recommend that.

Speaker 2:

I'm jealous and one thing it's very, very few people here in the United States take vacations like they do in Europe, and I wish we did, because that's one thing I'm envious of is that you know, in Europe, most of the countries, most of the individuals that I speak with take the vacations and enjoy the vacations and appreciate the vacations like they should. You know, doing things, spending time with your family. I know here in the United States very few do it. We have that you know we'll take a Friday off and have a three-day weekend, which really doesn't do anything, because I think with a three-week vacation

Speaker 2:

you get to spend a couple days unwinding, you know, forgetting about work, kind of de-stressing. Then you get to have your vacation and enjoy it and then you know you have what I call, like you know, the sunday horrors. I guess you could call it, because then, a couple days before you go back, you start getting okay, what am I going to go back to, what do I have to deal with? So having that three week period of time gives you that, you know, chance to really detach and relax, whereas you know, if you do it for a long weekend or even sometimes just a week, you really can't let go.

Speaker 3:

So I'm glad you're absolutely right. Like, practically I only had one week of vacation because you're absolutely right in the mindset. The first week was all wind world of what's going on in the office where I am, you know logistics and still catching up with a few things. But then the second week, full relaxation, full disconnect, no emails, no teams messages. And then the the third week was the time when I actually started thinking about work a little bit like, okay, I'm starting to get there. What's cooking there in the office, what's going on there, what's the next step there? So yeah, so it's a good timeline, absolutely but yeah I thoroughly enjoyed it.

Speaker 3:

Yeah, really really beautiful place.

Speaker 2:

I'm glad that you had a great vacation. Looking forward to speaking with you about Business Central online. You know server performance, upgrades and security. But before we do, would you mind telling everyone a little bit about yourself?

Speaker 3:

Yeah, yeah, absolutely so. My name is Dmitry Chudaev. I'm a group product manager in Business Central team. I live here in Copenhagen with my family for quite some time for more than 20 years now and I'm embarrassed to mention that because all of my Danish colleagues they start asking me why don't I speak Danish to them, and, yeah, that is embarrassing. I do understand Danish, I do speak Danish, but I guess I'm too shy to use it in the daily circumstances and in my work life. I'm responsible for more of a technical part of the product, so I'm responsible for the online service, everything that happens in the online service for upgrade, availability, lifecycle of environments, things of that nature, for our server, a server runtime and for our development tools and partner experiences. So this side of the Business Central product is part of my responsibility. Yannick is my counterpart. I've seen a fantastic show with you with you, gentlemen and Yannick, so lots of great insights there. So we work together, yannick on application and client experiences, and I'm more on the online service, server and partner tooling experiences.

Speaker 2:

Excellent. It must be exciting to work with the online or the architecture, the technical platform of Business Central Online. Business Central Online is taking off. The product has taken off. It's an amazing product. I've seen some great news and information released about. You know the number of you know implementations there are. I believe I thought we saw like there was 40,000 plus or something, which is just amazing.

Speaker 2:

And I can't. You know, I'm happy for it. It's been my entire life and I'm happy for it. But that must present some challenges, or architectural challenges for you as well, because you have this online environment with a number of users and users using it through web application, and you must have some challenges with performance. I know that a lot of individuals have security considerations or questions about having their information in the cloud, and then also some information or questions that I hear often are also even about the upgrades the upgrades between the architectural, the technical upgrade. I don't even know how to call it these days. I guess we used to call it the platform, but I guess now it's in the cloud. I'm not really certain what you would call it, but these are quite a bit of questions that we receive and we hear, so I was hoping to have the opportunity to talk with you about them.

Speaker 3:

Excellent, excellent, very, very good topics, for sure. Yes, running service at that scale is an interesting experience for sure. For sure, and you just named all the core topics that we get to deal with quite often in our daily life.

Speaker 2:

It must be both a challenge as well as excitement as well for you to be able to see that it works. We'll start with kind of a small portion of with Business Central Online. Now, one of the benefits of having an online product is that you have the upgrades and the updates as they become available. And with Business Central, the application, microsoft has a schedule of two major wave releases as of this recording two major wave releases per year and then monthly updates of the application. With the platform, is there a set schedule for the platform updates? With the platform, is there a set schedule for the platform updates? Do they coincide with the application or is there a different schedule that is followed for that?

Speaker 3:

Yeah, yeah, so the way it works and you're completely right. So we have specific cadence with application updates and application updates apply to specific environments. So each customer have their own environment, which is a separate database of some sort that you can imagine, and application updates applied to each of these environments. And because these updates are applied to each of these environments, we provide this functionality for the environment owner to schedule these updates so they can decide when they want to apply the application update. Now you can imagine that on the technical level, on the server level, this infrastructure, the server itself which manages all of these environments, is shared across multiple environments. It's a multi-tenant system. So it's one NST managing multiple, one server tier managing multiple customer environments. So applying hotfixes and changes to the server cannot be scheduled by each individual environment owner because I mean that's a shared infrastructure provides resources for all of these shared resources. So we use different approach to that. So application platform hotfixes are rolled out as they come. So when developers produce certain changes, there is a process where they go through certain testing in different tiers of our infrastructure, so they obviously are tested by the developers themselves in their local environment. Then we have a so-called pre-production environment where the changes can be tested as well. Then they're merged into our official production branch and then they're scheduled to be deployed into production. So these changes also go through a thing which we call safe deployment, which is a very, very important practice for all of the cloud services.

Speaker 3:

The hot fixes don't go to all of the customers at the same time. They go through a set of stages. So first they get deployed to a small segment of our service, to a few customers out there in a specific region, and then we monitor the health of these environments. If anything suspicious happens in that system, we interrupt the update, we analyze the changes and we introduce new things. But if we see that everything is healthy and is proceeding as usual, then we expand the deployment to more regions and more clusters and nodes in our service, eventually deploying all of it into all of the regions. So that happens because it's a shared infrastructure. That happens typically during the night, so we will locate night hours in each of the zones where we operate. And it is a challenge because we now run in some 170 countries. Yet in each Azure region there's obviously a good time zone where very little business activity. So we apply these fault fixes during that time period.

Speaker 3:

Another dimension of kind of protection which we apply to this process is that our service nodes, where hosting our customers, they obviously don't consist of a single, you know, of a single node. They consist of multiple nodes. So when we apply our hotfixes, platform hotfixes, we apply them to a specific node at the time. So we start applying the hotfixes to one NST on one node and then this NST is taken out of rotation and all of the new connections are sent to the new environments and new NSTs. So the new customers logging in into BSC they go to that part of the service. We patch and update this NST and once it is healthy and we approve and the signals are coming in green, we open it up for our load balancer to send customers there as well. So this is happening face-by-face across these NSTs.

Speaker 3:

However, I also have to admit that sometimes on that specific NST there are some users, so users doing jobs even during the night. They don't get any vacations, they don't get any night times or automated processes. There's some bad jobs, web services and portals doing the job. So we give a certain period of time to kind of drain these connections. There's a notification sent to the users and to the processes that we're going to interrupt you. Please refresh your browser. And that is a signal for us applying these hot fixes. So we try to avoid the disruption as much as we can. We try to thread as safely as possible across all of these instances. But, yeah, sometimes you can get this notification. Please refresh your browser, please reconnect, and that's the most disturbance that we apply to these environments.

Speaker 2:

Yeah, that's basically the screen. That is a lot, and I've seen that message before too, and now it's nice to know and have an understanding of exactly what that means. You answered so many of the questions that I had in that explanation.

Speaker 1:

Thank, you for that.

Speaker 2:

Because having an online application that's available 24 hours a day, seven days a week, and if you have a shared update, as you had mentioned, across time zones, I know how difficult it is trying to coordinate with individuals over time zones. With you having users, it must be a challenge, but now I see that you isolate it by region to have their nighttime to minimize the disruption, which is good.

Speaker 3:

And so sometimes users can see the same message popping up oh, you need to refresh your browser. For different reasons, like it's not necessarily that when you see this message is the time when we deploy an update. It could be done for other resource governance reasons, like if that specific machine or that specific segment of our service is experiencing more than usual traffic, we try to distribute it to other machines. And that's where we start to gradually kind of close it down and inform users please refresh your session and you will be sent to another machine with plenty of resources, lots of things you can do on another place.

Speaker 3:

But yes, it may come in as a destruction, coming in when it comes during the day, for example during morning hours. But during morning hours and the daytime this should not be an update. So this is something else, this is resource governance and in fact we built a special chart which we monitor very tightly the number of times this notification is showed to the users. So in the live site, periodically we analyze this chart to see if any particular segment of our service is affected by this annoying dialogues more than usual, and then we try to understand what's going on and try to kind of check whether the resource situation is okay there or whether the updates are applying okay. So we monitor this as well, very, very tightly. But yes, sometimes it's unavoidable, so we'll have to gently push.

Speaker 1:

When I see this pop up now moving forward, I'll start thinking about you and your team. Now it's like, oh, they must be doing something.

Speaker 2:

Yeah, they see your blip on the chart, Like that's Christopher refreshing his session and also depending upon the time of the day, which also leads to another interesting point that you were talking about, where you may receive that notification if there's a lot of activity on a server or a service tier.

Speaker 2:

For the sake of this conversation, I'll refer to it as the service tier performance. All right, so now that I mentioned you had you had mentioned that you may get this notification because there's a lot of activity on a particular service tier and you need to balance the load to ensure optimal performance for the users of the application. When talking with potential customers, prospects or customers who are looking to move from an earlier version of Dynamics Nav or even from an on-premise version of Business Central, a lot of times the topic of performance comes into play. Performance is always going to be, you know, a challenge or a topic, because there are many facets to performance. So let's talk about all things being equal. Take out any potential issues with coding or from the application side, right, because those sometimes could cause performance issues.

Speaker 2:

From the platform side, or from the server side, the NST side, what considerations are in place for performance to allow for optimal use for a particular customer or user? And also, in the conversation we talked about how there is a shared environment. So with the shared environment, we can have many companies accessing that at the time of day. Um, so with the shared environment, we can have many companies, uh, accessing that at the time of day I may be a distributor and I need to have a high volume of transactions towards the end of the day because I need to fulfill my shipments, to get the trucks loaded in my packages out, and another retailer during a holiday season may have influxes during you know, uh, a time period. Uh, that's, um, you know, through a seasonal time period. Excuse me, what considerations or how do you address the performance when you have large loads coming at the NST in these shared areas?

Speaker 3:

Yeah, yeah, yeah, very excellent question and yeah, we get to discuss this a lot and there's always a balance between performance and throughput, so these two topics come very closely together. What's the performance of Business Central at scale and how much throughput we can push through? Business Central at scale? And performance is an interesting beast. Very early on, when we just started building an online service, we understood that performance is going to be a mutual responsibility.

Speaker 3:

I mean, I've also seen many fantastic webcasts from you gentlemen about performance and how much it is important to keep your code in a good shape and to use all of the tools which we provide to monitor performance. We provide many telemetry signals to understand where the bottlenecks may be in the code, in web service calls, in AL code, maybe there's some locking happening too much in specific tables. So we try to build a lot of tools for partners to understand where on which side of the performance stack, the problem may occur. So now you would like us to focus on kind of the resource governance part of this, because all of the other aspects were mentioned, but I would also want to sort of lean in just a little bit there, remind everyone that we did provide a lot of tools, fantastic tools to understand what is causing these issues. So there's an inclined performance profile at which you can run and see where the time is spent in certain process so you can understand whether a certain extension or certain piece of code or update changed the behavior. So I do want to kind of stress that performance is a multifaceted area and we take care of the resource governance part of it. Our goal and sole purpose of existence, work, life maybe there are a few other purposes, but our goal is to provide enough resources for all of these workloads. So, yes, so, while partner's responsibility and customer's responsibility to kind of keep the code sane, what we do, we designed the system in a way that it allows for scalability at many different layers. So the scalability bottlenecks may come, which has kind of presented itself in performance, may come from different places. So there's a scalability of database. So the database simply cannot cope with that many requests coming, for example, from a web store, from e-commerce solution. So there is a scalability of compute and scalability like the traffic which is going between these systems. So there's many aspects and on each of this level we build automatic scaling system.

Speaker 3:

So for database scalability we use a system of tiers. So there are several tiers in our infrastructure which we use to kind of move the database between these tiers as it exhausts its limits in the specific tier. So what happens is that this system is fully automated. There's no way any human being is capable of monitoring this in real time. There is automation which is monitoring each and every environment time. There is automation which is monitoring each and every environment and it tries to understand whether this environment is getting close to the limits of what this database can handle. So not when it crosses these limits, but when it's getting close to these limits. So when we detect this moment of time, we take this database and move it automatically into another tier. So it gets moved into a quieter place, into a place with more resources. It scales as well. Azure SQL has many, many different tiers. Azure Elastic Pools, which we also use on the back end, also have a lot of configuration settings to define how much capacity you want to allocate to each environment. So your database goes into another tier automatically and by that you get more throughput and thereby more performance for your processes.

Speaker 3:

Now, so that is the database situation. The same happens for the compute capacity. You said correctly that the compute is shared, so many environments using compute on our service nodes. So when we see that the traffic is approaching some limits of CPU and other capabilities of this node which are about to be uncomfortable, we start scaling out the service to provide more of these resources and we stop sending sessions to this segment of the service. So all of the new connections go to the new services, the new services.

Speaker 3:

So whenever you experience this slowness and something which is maybe caused by the resources like in the worst case, refreshing your browser, trying to kind of hard refresh your browser, close, reopen, so that should automatically redirect your session to another machine compute-wise. So that's another good advice. But all of it is happening automatically. We kind of redirect resources to machines with enough capacity and to database and pools with enough capacity. And also I wanted to say that, looking at our KPIs, so we try to measure how long time each database spends on, or each user or each session spends on the machine with insufficient capacity, how long time users are suffering, and that percentage of time is very, very small. So we are in 99.9x of sending sessions to the machines with enough capacity.

Speaker 3:

And what I'm trying to say by that is that when the theme comes to the performance and the throughput. It's rarely caused by insufficient CPU memory and database throughput. It's quite often in some other segments. Insufficient CPU memory and database throughput, it's quite often in some other segments. So the combination of database locking, extensions too many instructions issued at the same time. Sometimes it is throttling as well. So you kind of send so many signals to Business Central that at some point we start throttling and capping this kind of influx of requests.

Speaker 2:

That was the question that I had, if there was a ceiling. We talked about the scaling of performance and being able to work with processing and CPU and then also with the ability to move the database, and I do. I'll reiterate what you said is before. That's what I was trying to say at the beginning is is there are application considerations to performance and I can say from a lot of times I get a lot of performance issues and sometimes it could be, as you had mentioned, multiple extensions that are installed that may be competing with each other, or even if somebody had created some code that may not be the most efficient. You know, even if I put something in there and tell it to wait a minute before it does the next process, a user may feel that it's slow performing, when in reality it was. It was written that way.

Speaker 2:

And you mentioned the performance, the incline performance analyzer, which I think I did see in 2024, wave two, is going to also have some enhancements. I read the notes last week because I'm excited about that. I do use that regularly as well because it is a good tool to help pinpoint from the application side the amount of time that's spent in certain areas, to help you isolate your code. So I did, taking the code piece into consideration. That's where I was wondering. Taking the code piece into consideration, that's where I was wondering if there was a cap, because if someone does create an extension that could impact performance, as you had mentioned, maybe just send, you know, an over abundant amount of requests, where is that cap that it would actually now say, okay, well, you can't you know you don't have an infinite pool of resources, right?

Speaker 2:

I mean theoretically, within Azure and an online environment, it's supposed to scale up to give you the resources adequate for you to perform your functions. But if there is some incorrectness or something going on, that's, you know, inundating or creating excessive traffic, how do you determine the ceiling of that?

Speaker 3:

Yeah. So very, very good question and I think, like I mentioned, our goal and aspiration and the whole team is kind of rallying behind that is to provide enough resources to everyone. I mean, this is the sole purpose of the online service. You cannot starve on resources. So a lot of efforts were put into place to kind of automate this process and make sure that it doesn't happen. But for some scenarios we wanted to introduce a reasonable ceiling. So, for example, for web service requests due to just heavy workloads, like extremely heavy workloads or a programming mistake or inefficient code in this web service request, you may start consuming unusually high number of resources in Business Central. So what we apply here is a throttling limit which I believe is 6,000 requests per five-minute sliding window. So five-minute sliding window if you issue more than 6,000 requests, we start throttling them. So that's for web service requests.

Speaker 3:

And the way we arrived to that number is we kind of went through the same safe rollout process. We kind of went through the same safe rollout process. So we introduced the measurements in our system to understand how much throughput different workloads require. So we monitored the system for a very, very long period of time I would say even more than a year. So more than a year we were looking at how much resources and how many requests customers are sending to us and then we collected all of this telemetry and then we looked at the peaks, like what is the throughput that they actually use, and then we almost doubled it generally.

Speaker 3:

So we try to be super generous in this throttling threshold to make sure that in this throttling threshold, to make sure that going to these limits it really is something where it's either by mistake or something unusually high load coming from some e-commerce solution or some reporting solution. So that's how these limits were defined by us looking at telemetry, trying to see even the heaviest workloads and putting these thresholds even above these heaviest workloads so they should be sufficient for the vast majority of our customers. Another unit of measure which we're tracking is the number of scheduled tasks or job queues, so you can create some processes looping automatically and doing things. We also apply a limit of five scheduled tasks per user executed in parallel, so you cannot spawn too many tasks or execute too many per environment at the same time. So with these numbers, hopefully there's enough throughput for many, many, many workloads. Yeah.

Speaker 2:

Just to touch base on the workloads and the limits and one key point that you're mentioning that is a common practice for all online environments. I have had interfaced with other applications from Business Central, interfaced with other applications from Business Central. You mentioned web requests. Well, we had web requests where we had to, again just by the nature of business, restructure how we were sending information because they also had limits. So the limits are there and not to say that there's a limit to the application, it's to ensure that you have proper performance and throughput. And, as you had mentioned, in the case where we had ran into the problem, we found that they had a measure in place where, if you sent the transactions differently, you wouldn't hit that throughput. We were sending them individually instead of in batch.

Speaker 2:

It goes to the point where some of these I just want to just put people in mind we talk about limits and their ceilings. It's not a negative to me. It's a positive because it can help ensure that everybody has the equal processing to continue. It can help ensure that everybody has the equal processing to continue, but also it may be a signal that there may be something else that you should maybe look at if you're constantly hitting those limits and that's what we had found in this transaction is the web service application interface that we were tasked to take a look at. Somebody was just sending the same transactions multiple times individually, instead of in one batch, with like they thought they were. So the the throttling, as we'll call it, or the limits that they set. The rate limits, as you mentioned, usually are adequate for normal processing, and if you hit those limits a lot, maybe it's time to maybe take a look at the process to see if you're using it effectively will they get notified if they're hitting that threshold too many times?

Speaker 3:

Yes, yeah, they receive a response from the service. 429, if I'm not mistaken, like wait a little bit. Wait, you've sent in too many requests. So yeah, and that's absolutely a good point. Brett, this is the time to re-evaluate. Are you doing something not right or are you just? You can spread your workload over a longer period of time, move it to another, to offline mode, to kind of async mode, or batch different requests or make requests shorter. There's many ways of coping with this. There's many ways of coping with this, but, to your point, this is more of a limit to ensure fairness rather than not provide enough resources, because we absolutely do want to provide enough resources for all SMBs. And, by the way, when we talk SMBs, you would be surprised what kind of SMBs we run in Business Central Service. Yes, some years ago we started with smaller companies. You know 20, 15, 20, 30 users, companies joining the cloud. So that was years ago. These days, smbs come in really, really large. You know form and shape.

Speaker 2:

I saw an interesting statistic. I think it was at Directions, not last year but the year before. There was a presentation. I'm getting old so I forget a lot of things. Someone put a presentation for large implementations or you know, large business central implementations, and I think they had a statistic, and this is me saying from memory for what I thought I saw. If you can validate it, great, if you can't, great. I think they had mentioned, like the largest implementation at 8,000 users or some significant number, which I was quite impressed. You know, sometimes people think of the SMB market they think of, well, the application can only handle the 20 to 30. I mean, I know firsthand of somebody using it with, you know, 400 users, which is a nice size implementation it is. So it's not the small, but I was impressed to see that there was thousands of users, if I recall the statistic. And if I'm incorrect then you know, we'll just chalk it up to me being old.

Speaker 3:

Yeah, the spectrum is very wide and you're absolutely right 300 users, 400 users is absolutely not unusual. This is almost the usual situation in Business Central. So the question is what do the users do, how they interact with the service, how they work with it, like, what are the web service workloads, what are the end users workloads, what are the integrations doing the reporting doing? That's where most of the discussion is. The number of users is not really a limiting factor in Business Central.

Speaker 2:

You're absolutely correct. I've always had the question people say how many sales orders can you handle? And I said it's not the number of sales orders, it's, in essence, the number of lines you want to process. If you're processing one sales order with 20,000 lines, or if you're processing 20,000 sales orders with one line, it's different, but it's not the sales order. It's like you had mentioned. It's the workload that you're working on.

Speaker 3:

By the way, since we are on this topic, I also wanted to advertise one of the resources that we have online which I would definitely recommend everyone to scan and to read about the throughput and performance. We use this short URL, akams, and the URL is called akamsvcservice slash VC service. So if you navigate to that resource, we have three segments there One talking about the high availability and disaster recovery and business continuity, so how we make sure that all of your data remains accessible and available at all times. The second one is scalability, the things which we're talking about right now. What is the throughput? How we scale the resources With practical examples how much customers pushing through Business Central today.

Speaker 3:

So we collected some telemetry how many sales orders to your point with how many lines people post and create through Business Central, and we put these telemetry results on that page so you can see the examples from the live customer installations. They're all anonymized. We will never share any private information of our customer, but you can see examples from the real-life telemetry. And the third one is our live site, our operational processes, how we deal with incidents, with support and other aspects of running the live service. So that is a very, very good resource BC Service very easy to find akms forward slash BC Service.

Speaker 2:

I will definitely put that up and keep it up.

Speaker 3:

Is it a?

Speaker 2:

real-time update on the statistics. So should I keep it open?

Speaker 3:

Yeah, we need to do better job. But this, the last one, is fairly recent, so it it was put there within the time frame of directions. So, um, yeah, but as the service evolves, you can imagine that these numbers are already too weak. So it only goes in the, in the in the other direction, in the other direction, in the growing direction. No, it's good If you see the numbers there and they fit your business. So that's a safe choice to further assess Business Central. Try it out and do the implementation.

Speaker 2:

No, those are good statistics for somebody to look at to give a general idea if they have a good understanding of the business. But it should not be the only consideration they use if they want to use the product. It's just one signal that they can look at, because it's important to make sure that they're looking at the right information to make the right decision. So talking with somebody about it would make it a proper decision.

Speaker 3:

Sorry. I just wanted to say that the way we built this article is we have this program called Concierge. So this program is meant to provide some consultancy and help to larger customers do cloud implementation. So if you are a larger customer with more than I believe it's 50,000 revenue or something like that yearly revenue licenses so you can ask for concierge service to provide you with guidance what business central is, what it can handle, what's the scalability considerations. So for a long period of time that team was assembling all of the questions and you know from these larger companies. So we took all of this feedback and packaged into this article. So now everyone has access to the same answers that we were giving to our larger prospects.

Speaker 1:

So you get concierge service. That's awesome.

Speaker 2:

That is a great resource, then, to have all that information on the akams4.bc service. We have to look at that and use that, chris.

Speaker 2:

Chris, that's a good one to show prospects and customer potential customers, even customers looking to upgrade. I mean, there is uh. That's that that's a big portion of uh, the BC implementation as well. It's not only new prospects, but also customers looking to migrate to business central online uh with online services and um multi-tenancy shared environments. Uh, one of the questions that I mentioned when we have conversations with many individuals and questions that I have even more, so you start hearing about issues with technology on a wide scale is security. So then let's talk about, from the NST point of view, how can we ensure that individuals' information, individual businesses' data as you had mentioned, you can anonymize some statistics. But also, now that we have shared environments, we have users connecting online, what assurances do they have that this is a secure system, yeah, that their data is secure, that the information is secure, that if, um, you know, somebody can't compromise the system from the cloud-based uh avenue, um this is.

Speaker 3:

This is an excellent topic, I think, like these days, cyber cybersecurity is so high on every agenda With all the troubles happening in the world and all of the cyber attacks and businesses getting in trouble with cyber attacks. This is an important point to keep in mind. Cyber attacks this is an important point to keep in mind. So there are two ways of answering this. So there's an official way which is fairly standard. So Business Central undergoes certifications recurrently. So every year there's a certification process assessing the way Business Central manages the service and manages the customer data. So there is ISO certifications which Business Central goes through with different authorities and provides proof of how these processes are executed. So you can see some ISO 27001, see some ISO 27001, some standards mentioned on our akamsbc security page. So all of these certifications SOC compliance, iso compliance. So these certifications are listed on that page and in the scope of these certifications, there are questions about whether you use encryption, whether you use encryption of databases at rest, whether you encrypt everything that moves through the traffic. So all of the industry standard practices for security are not all, but many are kind of part of this ISO certification criteria, so NSOC compliance criteria. So that's a good resource for prospects and customers to assess.

Speaker 3:

Another part of this answer is quite interesting. So I don't know if you get this feeling, but it feels like Business Central Team produces a lot of new capabilities. You know, every release I don't know if you get this feeling Every time I go to directions and I see the keynote and concurrent sessions every half a year. Actually there's just so much. There's new capabilities, there's new features, there's new, you know, like online service capabilities as well coming out, and yet a lot of effort that is going on here internally is actually put on the work done in the security area. So I'm not going to call a percentage of the team resources working on security topics, but it's a huge percentage of the team working within the realm of security. So ERP is a jewel in the business world, so that data needs to be protected. Data needs to be protected, and the amount of work that the team here and across Microsoft teams puts into security is incredible.

Speaker 3:

And we are in a very good situation because in a large organization like ours, there's this principle that when one is attacked, everyone is attacked. So whenever a product from Microsoft portfolio gets exploited or even potentially exploited in any way, so this information is propagated automatically to many teams, to all teams within Business Central. So there are tools and there are processes in place to share this information, to react on these incidents or suspected incidents and apply changes rapidly across all of the other services. So this is what we monitor and do pretty much all the time. So there's always new attack vectors emerge and new approaches and new ways to penetrate systems, and the team is very agile in monitoring this situation and applying these changes to the system, and so this is a lot of work of Business Central.

Speaker 3:

And, yes, the way we design our service in the cloud, we make absolutely certain that this service is sealed from all of the external factors that we are aware of. First of all, there's no external connections. Everything is sealed within these nodes of the service. Only the services whitelisted and registered to work with resources have access to resources, also known as managed identity way to kind of authenticate to these resources. So there's a lot of work going on in this space, for sure. And of course, in multi-tenant system, each customer environment is a separate, database is a separate environment and the data is never in the place where one tenant can kind of somehow get access to the data of another environment. It's just by design sealed off into complete silos. The data is encapsulated in the database. Everything that manages that data is running in its own process in its own sealed kind of entity and is never shared across these environments. That is the main principle of designing multi-tenant services. Okay.

Speaker 2:

So they have the same shared application, but the database and their tenant is isolated, so it's separate, correct, as you mentioned, it's sealed, so the shared metadata all of the properties, the pages, the code units, the code.

Speaker 3:

So that is obviously shared, but when it comes to data, that is managed completely separately.

Speaker 2:

Okay, yeah, that is an important point to note. And again, everyone gets a little nervous with security, even recently this week, because I got stuck, I couldn't fly anywhere. I wasn't security-related directly but indirectly. Yeah, well, interestingly enough.

Speaker 3:

I would say that this was more of an availability incident than security.

Speaker 2:

Yes, yes.

Speaker 3:

But it kind of brings this point of awareness that that, uh yeah, we also need to remember about security as well as availability, high availability correct, correct and and sometimes, as you had mentioned, the lines get blurred between what it was, I mean we.

Speaker 2:

I think by this point, everybody understands, or has at least read, what had happened, but it still raises the question of okay, well, what would happen to my environment if something like that were to happen? You talk about the databases. Another thing that we often hear and are asked is now this database, my information's online, it's in the cloud, in this great big sky place. What about? We talked about the data security. What about backups and data redundancy, and we talked about the data protection in a sense, but is it user-based for the backups or data redundancy, so that if something were?

Speaker 2:

to happen in one particular Azure region to a center where maybe my data is stored? How do I ensure that I have access to that data if they were to have an incident at that facility? As well as now, just backups, if I have a problem with my system and I need to restore it on my own.

Speaker 3:

Yeah, yeah. So that's again the topic of availability of data, and you know this is also one of the top priorities of the team, obviously. So how can we make this data available to everyone running DC? So that is also kind of a multifaceted space. I don't even know where to start, but the amount of replication that we apply and redundancy that we apply to data to make sure that it gets accessible and available even in case where the Azure region goes offline, which is happening very, very rarely is enormous.

Speaker 3:

So there's many, many kind of levels to ensuring the data availability. First of all, we use, in the backend, we use the technology Azure SQL. So Azure SQL on its own provides a lot of redundancy on the physical level. So what we see is a logical database, but on a physical level, all of these files and the actual servers where the data is hosted and is run it's all replicated and kept up to date and kept highly available. Azure SQL Database on its own has 99.99, like four nines availability SLA. So Azure SQL Database has four nines availability SLA. So Azure SQL Database has four nines availability SLA. It's always highly available.

Speaker 3:

Now, what we also apply to Business Central is a feature called a point-in-time restore, which is one of my personal favorites. So with this feature, which is available in Business Central Admin Center, you can go back in time restoring your database to any point in time within the last 28 days. So if anything happens to your data, let's say that by coincidence or by chance or not by chance, you installed an app or you ran a process and it somehow damaged your data. So something went wrong and you cannot simply undo that. So you can go to any minutes before you know in the last 28 days and recover your data to that point in time. This is a mind-blowing capability.

Speaker 1:

That is mind-blowing and the first question.

Speaker 2:

If you can pick a minute up, I'm sorry I just get excited about some of these topics, and it's it's. I get excited because these are real world situations and and instances that occur, because I can't tell you how many implementations that I've gone through on premise that they've had an incident and they've had to restore to a backup from the day before. They've had to restore from a backup from if they have a valid backup. I'm not even going to get into those that think they're doing backups and they realize they don't have a backup.

Speaker 2:

Separate topic. You know to have a good disaster recovery plan, but to be able to go to the minute for 28 days is amazing.

Speaker 1:

The simplicity too. It's going to the admin center and choosing a specific date and time. And imagine if you have to manage that on-prem.

Speaker 2:

We'll pretend I'm a prospect. What does that cost? Is that an additional cost or is that included as part of the Business Central license to have that option? That is included, so, or is that included as part of the business central license?

Speaker 3:

to have that option. That is included. So that is all included. There's no, that's the magic right there.

Speaker 2:

Yeah, that's the magic right there that you can roll back.

Speaker 3:

I mean, there's just so much which is in that package. Yes, yeah, for sure.

Speaker 2:

Yeah, that's um so for 28 days.

Speaker 2:

You can go to a point in time. I'm sorry you got me excited now with this. Go on, go on. Yes, what about could you go to a previous point in time before the 28 days? Like if there's a weekly backup, a monthly backup or any other disaster recovery plan if the customer needs it for whatever reason? I mean, I think a minute to a specific minute for the previous 20 days should satisfy most, yeah, needs for recovery. But if somebody needs to go further, are there any options available?

Speaker 3:

yes. So another option which is available you can. You can download the entire database in the backpack format. So somewhat technical, but you can go and download the entire database into your storage and then store it forever for 10, five, 20 years. This is now your responsibility. Now, restoring data beyond 28 days is not a process which is happening a lot. So for these purposes, we recommend to use our cloud migration tool, which is a bit of a I would call it is a bit of a workaround, but again, it also happens in extreme circumstances where you need to recover environment which is more than 28 days old. So, yes, you can take that back up. You can create an on-prem environment or just install unravel this database to your local SQL server, connect to a new environment in the cloud, migrate the data, clean up the data and you can still recover your data even from beyond 28 days. So that is a possibility and hopefully not many customers will have to revert to that possibility.

Speaker 3:

But the way we came up with 28 days is that we, as we usually do with many of our features, we surveyed many partners. You know how far back are you recovering or restoring your data, and hardly anyone went beyond one week, like if you go beyond one week, it has to be something really dramatic. So we said, okay, 28 days should probably be enough, so not one week.

Speaker 2:

but we set it to 28 days, that's plenty, and that's also what.

Speaker 3:

Azure SQL Server kind of provides us with, so that's a very, very good availability option.

Speaker 2:

Any minute in time for four weeks is significant. Yeah, and I know many with disaster recovery plans, with rotations of tapes, with daily, with weekly, with monthly backups. Whatever plan they have, they're not going to get a point in time.

Speaker 1:

They're going to get a specific day.

Speaker 2:

In many cases.

Speaker 1:

If you need more than a week, it's because that's the last full backup you had, Not because that was the only option. Yeah, that's the only option.

Speaker 3:

Yeah, good point. Good point. That's not it, by the way. So, interestingly, this is a way for you to kind of recover your data if something happens within your environments, like something happened to your environment. But we also protected your customer data from Azure outages. So, like I said, they happen very rarely, but when they happen, we want to make sure the data still remains available, and it's a very interesting story.

Speaker 3:

So some time ago, we had this incident where one of the Azure regions had troubles with I believe it was something with the cooling system. So unfortunately, the environments in that region were not available for a continuous time, so the Azure region was not available and therefore all of the environments in that region were not available. So that was quite some time ago. I have to do this full disclaimer, but instantly following this incident, just like we do for any severe incidents like this one, the team redirected the efforts to build support for a capability called Azure Availability Zones. So the way the Azure is kind of laying out their Azure regions is that every Azure region let's just say East US or East US 2, it's not one data center, it's a collection of data centers. It's not one data center, it's a collection of data centers. There's many data centers, buildings hosting these machines or keeping these machines I use.

Speaker 2:

US2, by the way.

Speaker 3:

There you go, so let's use that as an example.

Speaker 2:

East east.

Speaker 3:

Yeah, so that's a collection of data centers and not only it's a collection of data centers.

Speaker 3:

It's a collection of availability zones, so groups of data centers, of which there is always three.

Speaker 3:

So there's a group of data centers, there's another group of data centers and there's a third group of data centers and they're positioned very close to each other, but not too far, like very close, high latency, high bandwidth connection. Close, high latency, high bandwidth connection. And what happens is our databases, our paid production environments, are replicated instantly across these availability zones. So if the entire availability zone goes offline due to some issues like it's very rare when it happens, but when it happens the entire availability zone goes offline Our database in another availability zone becomes the primary one and the traffic is redirected to it without us doing anything on our end and without customers doing anything on their end. So even when the entire data center becomes unavailable, business central environment will remain online from another Azure availability zone. So that was another big feature we built for the online service to make sure that databases never go offline for these reasons. There may be many other reasons for databases to experience issues, but that will not be the issue.

Speaker 1:

And that's included with their license.

Speaker 3:

And, by the way, it's included with the license. So if you have paid licenses, paid production environment. That is a part of the offering.

Speaker 2:

These are the subtle things. I call them subtle. They're significant but often overlooked when people look to see which and I'm not trying to sound like a sales pitch I'm just trying to make sure that individuals have the best implementation they can have and take many things into consideration. As you had mentioned, the Azure regions may go. It's rare that they go down, but it is a possibility. You could have a natural disaster. You could have something occur in an area where it does go offline. So knowing that there's already measures in place, that your business continuity is still in place is important, because that's you know. You put a server in your back office, for example. You know what happens if you have a natural disaster in the back. You know your back office the water pipe breaks.

Speaker 2:

I've gone through that too, by the way, you know, my server gets, you know, get all wet. It's underwater because the water pipe broke. You know what do we do and you know, and this is the. These are some of the benefits that I think, if they were highlighted more as as you, would be well received uh so I'm happy to hear about that because, um, it's a, it's a good selling point, yeah, um by the way, our sla is is three nines right now.

Speaker 3:

So nine 99.9 percent availability and in the observable past across the business central service we have not gone under this SLA. So for each particular environment it may have challenges, but the online service in total has never gone under 99.9 SLA. You know, that's also something we keep very, very close eye on and whenever anything happens which affects availability of an environment, that's a major investigation, that's a major, you know activity to understand what are the repair items to put into development and make sure it never happens again.

Speaker 1:

Yep, it's amazing.

Speaker 2:

No, there's a lot to it, and you also work with partner tooling. I recall, yep, it's amazing. No, there's a lot to it, and you also work with partner tooling. I recall, correct, yep, yes, so what does that encompass with your group?

Speaker 3:

So that's a fantastic team of individuals working with AL language and Visual Studio Code and GitHub and GitHub so some of the top-of-the-line developer tools and team working with AppSource and all of our ISVs. So it is, by the way, we kind of stopped talking about this number because we talked about it too much, but we have, I think, much more than 5,000 apps on AppSource right now ISV apps and they are not dead apps. These are ISV solutions and some reseller solutions live on AppSource, installed, maintained, managed and covering pretty much every industry segment, pretty much every niche. We have apps for anything. In Business Central it seems like, oh yeah, there's a lot of apps.

Speaker 2:

A lot of apps.

Speaker 3:

So that is the team which is building the language, building the compiler, building the Visual Studio Code extension and maintaining the DevOps. You may have heard about our solution for DevOps ALGo built on GitHub. So the tooling, the set of tooling to help you manage your code as well in GitHub. So that's the development tools experience.

Speaker 2:

No, it's a great tools. I'm very familiar with them, I use them. I use all of them.

Speaker 3:

And millions of developers around the world would also be agreeing with you. No, it's a great set of tools, familiar with them, I used them. I use all of them, and millions of developers around the world would also be agreeing with you. No, no it's.

Speaker 2:

It's a great set of tools. I remember app source back when it first started. I think you had it was all manually done. I think I remember the I forget the individual's name that I used to talk with when you put it in and he had to do everything manually uh early on. This is very early on many many ago. And now times have changed. You can submit it. It automatically goes through the entire process Once it's been in there. If you do an update, the validation. It's a process that I'm thankful is continuously improving.

Speaker 1:

Like anything.

Speaker 2:

It's time and use will drive change to improve the experience. At least we start out and we have that and you can have a number of applications in there.

Speaker 1:

Wow my mind is growing.

Speaker 3:

This automation, by the way, is also one of these wonders, similar to Point-in-Time Restore. Imagine that you have so many customers to support, so your solution is deployed across hundreds of customers or hundreds of customers, and then you need to deploy a hotfix or change to this environment. How do you go about this and anything else then? Then you know online environment. So with this, you submit your change, it gets processed, it gets validated across all of these country deployments. Business Central is Business Central has some 25 plus localizations, so your change gets validated against these localizations. Make sure that it compiles against all of them. It gets uploaded and available for all of your customers.

Speaker 3:

The customers can go into their admin center and see that there is an update from ICD app. Click a button and get it installed. I mean within a matter of an hour. You can roll out your update if necessary. You can go as fast as you want, you can go as slow as you want. Your customers can decide to update, to do this update a week later, a month later, wherever they choose. But it is incredible the flexibility and the easiness of deploying updates into all of these environments by ICs, not just by Microsoft.

Speaker 2:

I think it's a wonderful way of deployment because, as you mentioned, you just update the app, you say, okay, reinstall it, and they have the change. You don't have to send anybody anything. You don't have to get on the phone and walk them. Walk them through an install or an update. You know they can do it themselves. I mean, you may have to talk with them about the changes or, you know, I'm just saying for the implementation of the installation process, you may have another reason that you'd have to speak with them for it. But, uh, I don't know. It's been my life. So maybe maybe I, so maybe maybe I've. You know, I've been in it for too long. I can't think of, uh, any other ways, but I try to look at it, uh, you know, from outside eyes at times, just to, to, you know, make sure that I'm not missing anything. So, yeah, uh, well, well, dimitri, you, you filled my mind with a lot of, uh, great things. Uh, a couple new great links BC Security, bc Service Are those on the AKMS, bc All page as well.

Speaker 3:

They should be Like if they're not? If they're not, we'll have to make a note to get them added.

Speaker 2:

But I definitely watch out. I'll be posting those links later today, chris, don't do it before me, but I'll be sharing those.

Speaker 3:

Those are two great links to share and I think it's important for partners and customers to be aware of them and you can add akamsbcperformance for sure to it, like that's a very well-known resource. This is where we describe guidance, not just for technical specialists but for the end users. I mean, use the latest version of your browser or use you know faster machine or things of that nature also help. Yes.

Speaker 2:

Yeah, I'm just writing all these down. I'm actually I'm texting yeah, all of it is already written in that article DC performance.

Speaker 3:

There's lots of great guidance there. Yeah, no, it's great For sure.

Speaker 2:

Thank you again for taking the time to speak with us. We know that you have a busy schedule, you taking the time to speak with us. We know that you have a busy schedule. You must have a lot going on after coming off a vacation and you know, hearing all of the things that your amazing group is involved in and does, I'm sure you have a full schedule. So we appreciate you taking the time to speak with us.

Speaker 3:

As I always say.

Speaker 2:

I appreciate anybody spending time with me. Time is the currency of life. Once you spend it, you can't give it back. So anybody who gives me their time I'm grateful for and thank you again. Hope you have a great weekend.

Speaker 1:

You get caught up, Thanks.

Speaker 2:

Dimitri.

Speaker 3:

And I look forward to Thanks a lot for inviting me, and I just wanted to mention in the end is that I, first of all, I do feel privileged talking to you, gentlemen, and talking to your listeners, so you're doing an amazing job, so that's very appreciated. We do want to spread this message that BC is evolving year after year to what you heard today and many other incredible things in the space of AI and other areas. But one thing which I wanted to share is the Business Central team keeps me amazed every day. So the team is an incredible set of individuals. They're so committed to the product when things happen in LifeSite and we try to stay humble.

Speaker 3:

Yes, I may have mentioned a few incredible things that we do with the Business Central today, but things happen in the daily life and operation of the service which we want to improve, and every individual keeps accountable and bought in into the idea of making it happen, making it a success, and bought in into the idea of making it happen, making it a success, and that drive that's taking the responsibility, having no issues which are somebody else's issues, like everything is our own, everything is my own, like this mindset, this culture in the team, is such an incredible thing seem to see that I wanted to also send kudos to this unbelievable team of developers that are working on Business Central.

Speaker 3:

And, by the way, it's the same vibe I'm getting from all partners engaged with Business Central. Every time you meet people at the conferences, in online forums, on Yammer, on our social media, there's very few people just chagging along. Everyone is chipped in and active and bought in and passionate, and the discussions we're having in this forum they're so passionate, they're so lively that they're not dry. They're never dry, and that's what keeps our team and me personally going that engaged partner community budding passionate individuals and this engaged team behind me in this building. It's an incredible setup. So I just wanted to say thank you to you for giving me this opportunity, thank you to all of our partners and thank you to the team building this amazing product and we see that we see that passion everywhere.

Speaker 2:

Yes, it is this business central community. We do thank your team. We do recognize you have a team with everyone. We've spoken with the other product managers I forget the names, you guys, names change with Microsoft all the time Product group owners, product managers you know it's more than just you know the ones we get to speak with directly. You do have a great team behind you, because it does take many individuals to do all of these tasks. And one thing I can say about the business central community as well it is a passionate group. There's a lot of discussions that may be not always positive, but at least when I read them.

Speaker 2:

it's out of passion, it's not anybody being, you know, malicious or anybody being hurtful. They're doing it because they're passionate and they enjoy and appreciate what they do. I'm thankful to be part of it, and it's changed over the years too, just like anything else. I remember when I first got into it you know back in late nineties.

Speaker 2:

Um, you know as it was when it first came to the United States. The partner structure was so different. Now it is a true community. A lot of partners now have realized it's beneficial to help each other. You had mentioned there's so many updates to the products and I'll say every time how do you keep up? I have a hard time keeping up, and how you can keep up is you just need to have a good group of people that you work with or that you're associated with. That can help keep everybody moving forward and work with such a great product. Thank you again for your words and also give thanks to the team, tell, yannick and Vincent.

Speaker 2:

I didn't forget Tell them when you talk with them afterwards, tell them we had a conversation. I didn't forget. I'll send you the email afterwards too, and they'll know what I'm talking about.

Speaker 3:

Yes, thank you so much.

Speaker 2:

But thank you again. We appreciate it and we look forward to talk with you soon and seeing you at some of the upcoming conferences as well my pleasure, all right, thank you ciao.

Speaker 1:

Thank you see you take care, see you.

Speaker 2:

Thank you, chris, for your time for another episode of in the dynamics corner chair, and thank you to our guests for participating thank you you, brad, for your time.

Speaker 1:

It is a wonderful episode of Dynamics Corner Chair. I would also like to thank our guests for joining us. Thank you for all of our listeners tuning in as well. You can find Brad at developerlifecom, that is D-V-L-P-R-L-I-F-E dot com, and you can interact with them via Twitter D-V-L-P-R-L-I-F-E dot com, and you can interact with them via Twitter D-V-L-P-R-L-I-F-E. You can also find me at Mattalinoio, m-a-t-a-l-i-n-o dot I-O, and my Twitter handle is Mattalino16. And you can see those links down below in the show notes. Again, thank you everyone. Thank you and take care.

People on this episode

Head Shot

Kris

Host
Contributor

Brad

Co-host