domain + /API + /Version + /Entity + /ID Parameter [Optional] + /SubEntity + /ID Parameter + ? + fromDate=01/02/2022 + & + endDate=02/02/2022

For instance, if we want to get all the items of an order, where the order ID is 17, out API should look like this,

https://mydomain/api/v1/order/17/items

Request

Let’s debug the previous URL step by step,

Few conventions,

1. The api keyword should be somewhere as follows

   1. api.mydomain.com [Preferred]

   2. mydomainapi.com

   3. mydomain.com/api

2. API should support versioning,

   1. Inside request header [Preferred]

   2. Inside URL. In this case,

      1. Version starts after the api/

      2. Version should have a prefix v

      3. Version number should be a natural number, like 1, 2, 3, etc. No decimal should be used

   3. Should not put versioning in the query parameters

3. The entity should be one word as a name, but not a verb, since the HTTP method is taking care of that

4. ID Parameter: The ID parameter should come after the entity

5. Subentity: The subentity should come after the entity

6. Query Parameters: Should come after ? and concatenated by the &

Query Parameters vs ID Parameters

1. While nothing was found in query parameters, it’s fine. However, missing ID parameters should throw an error.

2. For an entity, there could be multiple query parameters. But there should be only one ID parameter for the entity.

3. To extend, query parameters may return one to multiple entities, but if the ID parameter finds nothing, it will throw an error.

Response Code

Five Groups

• 1xx: Avoid this one, for informational purposes

• 2xx: Success

• 3xx: Avoid this one, Redirection in OS low-level

• 4xx: Client Error

• 5xx: Server Error

2XX

• 200 - Ok: Default Status Code

  • Get request, get data

  • Post request created data

• 201 - Created: On Post request, entity created

• 202 - Accepted: Request accepted and under processing on the server

• 204 - No Content: Operation completed and no entity as response

4XX

• 400 - Bad Request: Client sent a request with invalid parameters or JSON

• 401 - Unauthorized: Server does not recognize the client

• 403 - Forbidden: Server recognizes the client, but the client does not have enough permission to access the resource

• 404 - Page Not Found:

  • The resource entity was not found.

  • Entity with the ID parameter not found

  • Not Applicable when query parameters are used

5XX

• 500 - Internal Server Error: Something is wrong with the server, and the client has nothing to do with it.

JWT in REST API

JWT Token should be passed using Authorization Header with a prefix bearer

Performance

To keep up the scalability and availability,

1. Using an asynchronous mechanism for long-term operation

2. Using caching for near-frequently accessed data with expiration

3. Use a rate limit to reduce the maximum concurrent requests

4. Rate limit for a specific user

Metrics to Monitor

1. Request/sec

2. Failure ratio

3. Latency between request and response

4. Memory, CPU Utilization

5. Users, sessions, geolocation distribution

HATEOAS

Stands for, Hypermedia as the Engine of Application State. The user should not have prior knowledge of resources. Other relevant resource links should be included in the response.

Spider Web

If the URL of one service is changed, all other services calling that service will go down. To resolve this, we can use Yellow Page Directories, where the service and mapping will be stored. Also, it is possible to use an Gateway as a middleman, it will handle the URL mapping of a specific service.

Consider these three operations on a web page:

• HTML Parsing

• JavaScript Download

• JavaScript Execution

We can manipulate the sequence of these phases using the following attributes in the <script> tag:

• async (<script async src=''></script>)

• defer (<script defer src=''></script>)

Default Behavior:

By default (<script src=''></script>), the script tag blocks HTML parsing until the JavaScript is downloaded and executed. This means that the browser will stop parsing the HTML, download the script, execute it, and then resume parsing the HTML.

async and defer Behavior:

Both async and defer do not block HTML parsing during the JavaScript download. The script download and HTML parsing happen simultaneously.

• defer ensures that the JavaScript is executed after the HTML parsing is completed. This means that all deferred scripts are executed in the order they appear in the document, but only after the entire HTML has been parsed.

• async ensures that the JavaScript is executed as soon as it is downloaded, without waiting for the HTML parsing to complete. This can result in the script executing before, during, or after HTML parsing, potentially interrupting it.

Default: JS Download → Execute → HTML Parse

Defer: (HTML Parse + JS Download in Parallel) → JS Execute (after parsing)

Async: HTML Parse + (JS Download → JS Execute when ready, possibly mid-parse)

Use Case

1. Use async, when a script has no dependency on others

2. Use defer when the script depends on other scripts

References:

1. Web dev simplified

2. Great Front End

static: Default position. It does not allow setting properties like top, bottom, left, right, z-index

relative: Acts the same as static. but allows positioning it relatively by putting properties like top, bottom, left, right, z-index

absolute: Remove the DOM from the DOM tree and position it to the root element unless the parent has some position other than static. Ideal way to implemet is, make the current postion absolute and parent element position as relative.

fixed: Move to the top, ignores all the parents, stays there even if it is scrolling

sticky: With sticky position while the parent is relative, the DOM fixed it to the top while scrolling. It is a combination of fixed and relative position. To implement this, the current element should have sticky position and the parent should have relative position.

static vs relative

Both align with regular DOM flow; however, unlike static, in relative We can set the top, bottom, left, right and z-index.

absolute vs fixed

Both ignore the Dom and move to the very top. However, unlike absoluteThe fixed does not scroll.

Combination of absolute and relative

absolute took the Dom to the very top, unlike there is a parent other than static, like relative.

absolute vs translate

absoulte remove the dom and as a reference use the parent element (other than the static). However, translate also moves the element but uses itself as a reference. Since, translate does not invoke the browser to repaint the DOM, it is more efficient.

References

1. The CSS Position Property Explained with Examples (freecodecamp.org)

2. Learn CSS Position In 9 Minutes (web-dev simplified)

When an element in the browser has a conflict of styles, the browser uses a set of rules to determine which style should be rendered. This set of rules is defined as CSS Selector Specificity.

For instance, for an element, if we have the following DOM and CSS classes,

.myClass {
    color: yellow;
}

<div style="color: red;" class="myClass">My Text</div>

In this case, the color will be red, since the inline style has higher weight.

Calculation

The browser uses an array of four elements, where the previous index has higher priority. Consider these four elements a a, b, c and d.

[a, b, c, d]

A = Inline styles

B = Number of selector IDs

C = Classes, Pseudo Classes, and Attributes

D = Elements and Pseudo-Elements

We calculate A, B, C, D as follows,

body.home #sidebar .menu li a:hover {
  color: gold;
}

Let’s break it down:

body → element → +1 to d
.home → class → +1 to c
#sidebar → ID → +1 to b
.menu → class → +1 to c
li → element → +1 to d
a → element → +1 to d
:hover → pseudo-class → +1 to c

Specificity: 0,1,3,3

Rules of Specificity

• Higher specificity wins.

• If specificity is the same, the last declared rule in the CSS wins.

• Inline styles have the highest specificity (except !important).

• !important can override normal specificity but should be used sparingly.

Best Practices

• Write CSS with low specificity, so we can override it later

• Avoid nested selectors and keep them shallow

Responsibilities of these values are described below,

• none: The current elements along with all its children will not be rendered in the browser

• block: Takes up the full width, forces a new line, and allows width/height to be set.

• inline: Takes up only as much width as needed, does not force a new line, and cannot have width/height set.

• inline-block: Combines the behavior of inline (doesn’t force a new line) and block (can have width/height set).

• flex: Use block-level flex container

• grid: Use block-level grid container

• table: Behaves like <table /> component

• table-row: Behaves like <tr /> component

• table-cell: Behaves like <tc /> component

• list-item: Behaves like <li />

Regarding flexbox and grid,

Use flexbox for one dimentional simple layout. However for multi-dimentional complex layout, where both row and column needs to be managed, use grid layout.

Understanding the display property and its various values is essential for crafting effective and responsive layouts. While simpler layouts benefit from inline, block, or flex, more complex, multi-dimensional layouts are best handled with the grid property. Choose the appropriate display value based on the layout requirements to create efficient and maintainable web designs.

Box models are the core concept of CSS. We can define box models,

• Rectangular Box

• Generated for the elements

• Reside in the document tree

Each of the CSS models has a content area along with the possible optional surroundings,

• Padding

• Border

• Margin

Now regarding margin, it is not included with the actual size of the box. Instead it grows outside the box. Only the padding and border are included with the box size

Rules

The rules for the box-sizing,

• Box models are calculated by width, height, padding, and border

• If no height is defined, it will take the height same as the content

• If no width or max width is defined, it will take the width of the parent. A few exceptions are input, table, figure, span, etc.

• The default value of the box-sizing is content-box. For the value content-border, the size of the box includes the padding and border.

Rules are not applicable for the float component.

In conclusion, understanding the CSS box model is essential for precise layout control. By mastering properties like box-sizing, padding, border, and margin, you can effectively manage the size and spacing of elements on a webpage.

Today, we will get the most basic problem and solve it step by step. We will start by using the recursion and later on top of the recursion problem, we will use the tabulation of dynamic programming. Finally, we will optimize the space.

The house robber problem is explained, as follows,

There are a few houses in a row and as a robber, I can not rob adjacent houses. This implies, if I rob house 1, I can not rob house 2. After house 1, I have to choose house 3 or 4 or so on. However, If I do not rob house 1, then I may rob house 2.

With this in our mind, the simplest solution is, to write a recursion and try all the possibilities.

Recursion

Recursion will be the brute force solution, where we will explore all the possibilities and figure out the maximum value. We can consider any of the sequences, from the first to the last index or vice-versa of the array. Here’s a solution, driving from the first to the last index.

function rob(nums: number[]): number {
    const helper = (index) => {
        if (index >= nums.length) {
            return 0;
        }

        const pick = nums[index] + helper(index + 2);
        const notPick = helper(index + 1);

        return Math.max(pick, notPick);
    }

    return helper(0);
};

DP Memoization

Now, we can see the sub-problems, where, we may calculate the index multiple times using the function. Instead, to persist the already calculated value, we can consider an array dp of the array size.

function rob(nums: number[]): number {
    const dp = new Array(nums.length).fill(-1);
    const helper = (index) => {
        if (index >= nums.length) {
            return 0;
        }

        if (dp[index] !== -1) {
            return dp[index];
        }

        const pick = nums[index] + helper(index + 2);
        const notPick = helper(index + 1);

        dp[index] = Math.max(pick, notPick);
        return dp[index];
    }
    return helper(0);
};

Now, if we already calculated the value, we skip the further calculation and get the calculated from the array.

We can calculate the the value from last to the first index as well as follows,

function rob(nums: number[]): number {
    const dp = new Array(nums.length).fill(-1);
    const helper = (index) => {
        if (index === 0) {
            return nums[index];
        }

        if (index < 0) {
            return 0;
        }

        if (dp[index] !== -1) {
            return dp[index];
        } 

        const pick = nums[index] + helper(index - 2);
        const notPick = helper(index - 1);
        dp[index] = Math.max(pick, notPick);

        return dp[index];
    }

    return helper(nums.length - 1);
};

Tabulation

If we observe closely, we can see, the base value of the recursion,

• If there is only one house, the result is the first house's value

• For the first two houses, the result is the maximum of the first two houses, since we can not take these consecutive two houses

• For the rest of the houses, if we pick the house, just can not pick the immediate previous one

Considering these, can calculate the final maximum results as below,

function rob(nums: number[]): number {
    const dp = new Array(nums.length).fill(0);
    dp[0] = nums[0];
    dp[1] = Math.max(nums[0], nums[1]);

    for (let i = 2; i < nums.length; i += 1) {
        const take = nums[i] + dp[i - 2];
        const notTake = dp[i - 1];
        dp[i] = Math.max(take, notTake);
    }

    return dp[nums.length - 1];
};

Space Optimization

From the last solution, it occurs, that the solution of the current index only depends on the previous two values. With this consideration, we can simply avoid the entire O(n) array with a just two variables,

function rob(nums: number[]): number {
    let rob1 = 0;
    let rob2 = 0;

    for (let i = 0; i < nums.length; i += 1) {
        const take = rob1 + nums[i];
        const notTake = rob2;
        rob1 = rob2
        rob2 = Math.max(take, notTake)
    }

    return rob2;
};

This is the most optimized solution with constant space and O(n) runtime complexity.

For practice: https://leetcode.com/problems/house-robber/

We will start solving the problem from a recursive solution and later step by step we will use memoization and space optimization.

The Fibonacci Number or Fibonacci Sequence is as follows,

0, 1, 1, 2, 3, 5, 8, 13 and so on.

After the first two numbers, each number is the sum of the previous two numbers.

Recursive Solution

So, the constant part of the base case of the sequence is first two numbers are always 0 and 1. To calculate the rest of the numbers we will consider,

f(n) = f(n - 1) + f(n - 2)

With the base case and driven formula, our recursive solution will be,

function fibonacciNumber(n: number): number {
    const dp = new Array(n + 1).fill(-1);
    const helper = (step) => {
        if (dp[step] !== -1) {
            return dp[step];
        }

        if (step === 0 || step === 1) {
            return step;
        }

        const first = helper(step - 1);
        const second = helper(step - 2);

        dp[step] = first + second;

        return dp[step];
    }

    return helper(n);
};

This is a bottom-up approach. However, if we go to a tabulation format, we can design a top-to-bottom approach.

Memoization

We are maintaining an array dp where we are keeping the sequence number. After the first two values, the next one is the sum of the previous two values. In a more simplified way, we can say,

dp[n] = dp[n - 1] + dp[n - 2]

Now, the first two values are known and we will update these two values initially. Then with a simple loop we will calculate the rest of the sequence.

function fibonacciNumber(n: number): number {
    const dp = new Array(n + 1);
    dp[0] = 0;
    dp[1] = 1;

    for (let i = 2; i <= n; i += 1) {
        dp[i] = dp[i - 1] + dp[i - 2];
    }

    return dp[n];
};

With the implementation, we can see the space complexity is O(n). With this in mind, can we do more space optimization?

Space Optimization

Since each of the values just depends on the previous two values, we can just keep these previous two values and calculate the current one.

function fibonacciNumber(n: number): number {
     if (n < 2) {
        return n;
    }
    let prev1 = 0;
    let prev2 = 1;

    for (let i = 2; i <= n; i += 1) {
        const temp = prev2;
        prev2 = prev1 + prev2;
        prev1 = temp;
    }

    return prev2;
};

With these constant values, now, we can now optimize the space to O(n)

Now, in 2024, one of my tour mates coincidentally booked the same cottage.

This time, eight of us decided to embrace the clouds in the beautiful Sajek Valley. Since it was just a week before the Eid festival, the valley was quite empty.

We reached Khagrachari early in the morning and after having breakfast, took the nicest jeep, called "Chander Gari" locally. On our way to Sajek, we enjoyed the amazing nature and during stops, tasted the local fruits.

Since my last visit, Sajek has changed a lot. There are a lot of cottages now and they have electricity. The Adrika Eco Cottage has merged with Lusai Cottage. I saw the old room and felt nostalgic.

During lunch, we tried their local bamboo-made chicken. Then we headed to a hill called Kanglak. We returned in the evening. At night, a barbecue was arranged for us.

The next day, we heard a warning that the road was blocked by locals for some of their demands. However, after a few hours of negotiation, the army managed to escort us to Khagrachari city.

There, after having our lunch, we first went to the beautiful Risang waterfall. Since the waterfall was slippery and sloped, we used a rope for support. Then we went to a cave and crossed it.

It was almost dark everywhere. We returned to Dhaka that same night.

From the very beginning, we wanted a relaxed trip to enjoy some leisure time. At the last moment, when we were looking for a resort, none of them were available, and some were beyond our budget. I had previously visited Rangamati three times and remembered a park called "Polwel Park," where I noticed some nice resorts. Suddenly, I made a call and found that cottages were available and within our budget. Soon, we bought tickets for a total of seven people.

We reached Rangamati early in the morning and headed to our cottage. Fortunately, the cottage was empty, and we checked in immediately. On our first day, we had no plans other than being lazy. We ordered our food from the park's cafe and then went to the swimming pool. We spent about three hours in the pool.

Returning from the pool, we took a nap for three to four hours. In the afternoon, we went for our lunch. The park had kayaking facilities, and we did not want to miss it. So, in the evening, we cruised over the beautiful Kaptai Lake with the kayak.

The next morning, we rushed to the cafe because of the time limit for the complimentary breakfast. Then we hired a wooden boat for the whole day. It took us to a few remote places around Kaptai Lake. On our way to Shuvlong Waterfall, we ordered our lunch at a cafe that served only local and traditional foods.

We were lucky to have a thunderstorm on our way to Shuvlong, which brought more water to the waterfall. From the waterfall, we visited a temple and then headed back to the restaurant for our lunch. The food items were both unique and delicious.

It was almost evening when we returned to Rangamati.

The most special aspect of this event was that it was a night run. Usually, we are used to participating in morning or daytime runs. This one started at 1 AM. Additionally, it was only a 20-minute walking distance from my neighborhood, so I did not want to miss the event.

A month before the event, we observed fasting for religious reasons. I had one month to prepare. I used to run slowly, covering 20-25 km a week. It turns out that was not enough.

During the race, I started very slowly, just jogging. We had four loops of 5 km and one very small loop, around 1.5 km. I completed three loops, totaling 12 km, with slow running. Later, I did not feel well and started walking. For the rest of the track, I alternated between walking and running 1 km each.

I took more than three hours to complete the run, whereas my best time was 2 hours and 18 minutes. This event highlighted my weaknesses, particularly my lack of endurance. I have now started a 10-week training program to increase my endurance.

When It was about to happen in the Alikodom, I knew I could not participate as I had an office meeting the previous day. However, when the location changed, I decided to take the shot towards an uncertain journey to reach the Bandarban. Previously I visited Bandarban over 15 times, so I know some routes, so I can reach to the city before 3 AM.

I reached Bandarban very early in the morning, around 2 am. There, I collected my BIB and then encountered some familiar faces.

One of my previous running partners was already there, and we surprisingly had a similar pace. So, we started running together. At around 3 km, we reached a U-turn leading to a hill. Although it was physically devastating, the natural beauty was serene. I encountered another mind-blowing view during the last U-turn.

My partner experienced muscle cramps twice and received long massages twice at two hydration points. Overall, it took us around 4 hours and 30 minutes.

After collecting the medal, I opted to visit Kutubdia Island. I had previously tried twice to go there, but due to very bad weather, I was unable to reach it. This time, the weather was nice, and I went there by a small boat. It is famous for its lighthouse and windmills. Although the lighthouse was broken, the windmills were still standing.

This race day adventure was not just another running event. The hill track was exceptionally challenging but worth every bit. Additionally, I enjoyed utilizing the second day on a nice island.

• Choosing the right certification

• Identifying the courses/resources you will need

In most cases, the first confusing part of certification for engineers/developers is whether to go for Solution Architect Associate or Developer Associate.

Now, some of you may wonder why I did not consider AWS Cloud Practitioner. There are two reasons for this:

• This Cloud Practitioner certification suits marketing roles more than developer roles.

• I assume you are a developer who has been using AWS services for at least a year.

Now, before you proceed, this article is not suitable for you if:

• You are not using AWS services regularly.

• You are more interested in obtaining a certificate than gaining hands-on experience.

Assuming you are still here, the perspective of having a certificate is to validate your existing knowledge. One key point to remember is that while you validate your knowledge, AWS acts as a marketing agent to sell its products and services.

Before we proceed further, what are my credentials to write this article? Okay,

• I hold two AWS certificates (Both AWS CSAA and CDA).

• I am a senior software engineer with over 5 years of experience.

Which certificate should I go for?

For the first certificate, we have two options, both at the associate level:

• AWS Certified Solution Architect - Associate (CSAA)

• AWS Certified Developer - Associate (CDA)

Now, although in my case, I took the CSAA first, I encourage you to go for CDA.

Reasons why I discourage going for CSAA:

• It requires knowledge of networking to a certain degree.

• Comparing study content, it is larger than the CDA.

• Challenges knowledge on migrating on-premise services to AWS Clouds, which is not a regular job for a developer role.

• Tons of hands-on work on the AWS console.

And why I prefer CDA for the first certification:

• This includes serverless services, which are pretty common for developer roles, as we use them in on-premise servers.

• Most of the hands-on work is related to existing development tasks.

• We have already used most of the tools and services in open-source versions (or from different vendors).

Overall, as developers, we use a lot of open-source tools/services. In the CDA exam, we will become familiar with the AWS versions of these tools. For example, for CI/CD, we use Jenkins, GitHub Actions, Travis CI, etc. Here, we will find the AWS versions of these tools. In our daily jobs, we develop APIs, build UIs, and deploy them. In the course curriculum, we will get used to AWS services for these tasks.

So AWS CDA is not just comparatively easier but also an interesting choice for the first certification.

However, if you are involved in building infrastructure, managing security, handling networking stuff rather than raw coding, I would suggest going with AWS CSAA first.

Courses/Resources

Stephane Maarek, Neal Davis, Adrian Cantrill, Jon Bonso, Wizlab have created great resources for both CDA and CSAA certifications. However, if you want a bold outline with minimum resources, you can go with the following:

Courses for AWS CDA

• Video course from Stephane Maarek: Ultimate AWS Certificate Developer - Associate

• Practice Exam from John Bonso: AWS CERTIFIED DEVELOPER ASSOCIATE PRACTICE EXAMS

Courses for AWS CSAA

• Video course from Stephane Maarek: Ultimate AWS Certificate Solution Architect - Associate

• Practice Exam from John Bonso: AWS CERTIFIED SOLUTION ARCHITECT ASSOCIATE PRACTICE EXAMS

Tips

Video Course

Video courses are the most boring but impactful resource for the exam. Personally, I could not concentrate for more than 5 minutes on video tutorials. To reduce this issue, I did the following:

• Ran these videos at 1.5x speed

• Paused and took personalized notes

• Went to the AWS console and played with the services

• Tried practice exams and to clarify specific exam topics watched the related video (or video section)

Practice Exam

Practice exams help identify the boundaries of the course. When I found multiple questions on a single topic, I put additional concentration on these topics' video courses.

During practice, even if you are sure about the answers, read the explanation part to learn additional information about the services.

In my real exam, I found around 10 questions pretty close to the practice exam. Although the practice exams are comparatively more challenging than the original exam, try to revisit them until you score over 80% in each exam.

Hands-On

Hands-on experiences are the keys to learning, experimenting, and memorizing the services. For CSAA, the most important hands-on part is creating a custom VPC that includes public and private subnets. Run servers in these subnets along with ALB, ASG, IG, NAT Gateway. This will help to cover a major portion of the syllabus.

For CDA, build a serverless application with SAM, API Gateway, Lambda, and DynamoDB. Use Code Pipeline to deploy code to the EC2 instance (or Beanstalk) using Code Build and Code Deploy. Play with different deployment configurations. This will not just cover the exam syllabus but also keep the certification journey interesting.

Conclusion

This is a personal opinion based on my journey with these two certifications. For queries, comment below or shoot me an email. Good luck on your cloud journey.

• Marine Drive Ultra Marathon (50 KM)

• Dhaka Marathon (42 KM)

• Rajkandi Hill 25K (25 KM)

• UCR Half Marahon (21 KM)

• And in-house practice 100 KM

As I am going through the recovery training, I wonder, did I push myself beyond my limits?

Marine Drive Ultra Marathon

It all started with the Marine Drive Ultra Marathon. Before this one, I never run a full marathon. Since it is one of the most prestigious events for runners and not easy to be selected for the program, I registered with no hope of getting a chance. To my surprise, I was selected for my previous month's consistency. I wrote them in detail, I participated in 7/8 half marathons in the past few months and I am capable of running this ultra distance.

After being selected, I started to run ~15 KM 3 days a week. My target was to increase my endurance so I could stay on track for over 8 hours. Usually, the cut-off time for 50 KM used to be 10 hours which was reduced this year to 8 hours. The week before the run, I stopped running and only did free training. It was one of these feelings, I practiced my best and there was nothing more I could do. No matter, if I can complete it or not, there will be no regret.

During the run day, I started slowly and on the track, I found a partner with the same pace. Or I should say, I found a mathematician. He opened the Strava app and calculated each KM with the time. We were together the entire 50 KM and his precious calculation helped me to complete the run in the cut-off time.

I went around 35 KM pretty much comfortably and after 35 KM I mixed both running and walking to the rest of the track. The amazing part was kids were standing by the side of the road and giving high-fives all the way.

I completed the run around 7 and a half hours duration. It was a mixed feeling when you saw lots of good runners who were not able to complete the run under the cut-off time for different challenges. Because this also glorifies the achievement.

Dhaka Marathon

In the immediate next week, there was the Dhaka Marathon. I was not quite confident about this run. Although my muscle memory now has a distance of 50 KM I have not completely recovered from the last Ultra distance run. In Bangladesh, a full marathon is not very frequent and I was desperate to put a full marathon in my resume.

Leaving all the confusion behind, I decided to participate in the event. For the first 21 KM, I run quite easily at a slow pace. This time the cut-off time was 6 hours. As I recall, I completed half of the distance in under 2 hours and 23 minutes. I kept the same pace till 30 KM. I had around 2 hours left for the last 10 KM. Since I run previous week a long distance and in the upcoming weeks I have a couple of running events, I did not push myself. I kept my heart calm by running and walking.

This time, I completed the full marathon in around five and a half hours. The consequence of two consecutive long-distance runs injured my left foot.

RajKandi Hill 25K

In Bangladesh, Rajkandi Hill 25K is the most beloved running event. Not only it is organized by the dedicated community runners, but also the track surprises the runners every 5 KM.

I was not fit to run this event due to the left foot injury. But hearing the stories from fellow runners about the track could not resist myself. This week, I did not practice and slept a lot. I was worried this time If I would be able to complete this hill track.

On the run day, surprisingly I was feeling very good. On the track, we started running in the rice field. Later, the track was going through the tea garden of the Dankan Brothers. The surprises start after the tea garden. There were rubber gardens on the hills and our tack was under these large rubber trees. Then comes the cascade all over the track. At the end of the run, I found a hill, that will be around 300 meters high.

All these beauties of the track and support from the locals made the run very easy and I completed the run in around three and a half hours.

We also visited few places in the Sylhet region after the run.

UCR Half Marathon (Edition 2)

Again, this is the event I should not participate in as I have not recovered yet. However, since I was in Dhaka I would retreat not participate in the event in terms of staying home idle during the weekend. So I decided to participate UCR Half Marahon before going into a long recovery training.

The best part was, I found the best pacer in BD with a close time to my target. I planned to complete the run in under 2 and a half hours. Since the pacer had a time of 3 hours, I decided to accompany him around 10-15 KM and then push myself. Since I was not completely fit this time, I started to struggle after 10 KM. After half of the distance, my leg, especially my injured left foot cramped a few times. This time, I could not walk because walking was causing more pain than running.

Along with the pain, I never left the pacer bus and after 15 KM pushed myself to my regular pace. This time, I met my personal best in half marathon distance, 2 hours and 27 minutes.

Back To The Answer

So, back to my early question, did I cross my limit? I thought and it appeared to me, we can never cross our limit because the term "LIMIT" does not exist in this context.

I am under recovery training and in the next month, I will be back on track. Till then, signed off.

To stay tuned, check out my running blog series Rising Beyond Miles.

For individuals:

• Allow searching tracks from source to destination.

• Allow direction on the track.

• Show the ETA (estimated time of arrival).

For companies:

• Allow driving the self-driving cars (Waymo's self-driving car).

• Allow ride-sharing services (Lyft).

• Allow parcel services (Uber).

Requirements

Functional Requirements

• Detect the current location: It requires finding out the latitude and longitude of the current location. This is the basis for providing other services.

• Provide a route: According to the source and destination, the application should provide multiple faster routes based on the vehicles.

• Directions: On the way from the source to the destination, the app will guide users in the right direction.

Non-Functional Requirements

• Availability: The system should be highly available.

• Scalability: The system should be scalable as both individuals and enterprise applications like Uber, Lyft, Waymo, etc., use the application internally.

• Performance: This includes the accuracy and latency of finding and providing routes.

Challenges

• Scalability: The map application will be dealing with billions of nodes and edges when finding the path all over the world. Typical pathfinding algorithms like Dijkstra's algorithm will not scale to this volume and may provide a poor experience. We will require a modified and improved version of the algorithm.

• ETA Computation: The ETA should consider different factors like road constructions, traffic volumes, and different incidents on the way to the destination and update accordingly.

Resource Estimation

Let's consider the map application has a total of 30 million daily users.

Servers:

If a single server can handle 8000 requests per second, and a user makes 50 requests per day, we will make a total of (50 30 10^6) requests per day.

This implies (50 30 10^6) / (24 60 60) requests per second.

TOTAL_REQUEST_PER_DAY = (50 30 10^6) / (24 60 60);

Total number of servers we will need is (TOTAL_REQUEST_PER_DAY/8000).

Storage:

Storage requirement for the map is pretty much consistent. Reading route data from various sources is persisted for one time.

Bandwidth:

Based on the following assumptions:

• Total active users: 30 million

• Each user makes 50 requests per day

• Each request size is 200 bytes

• Each response size will be 2050 bytes

Incoming Bandwidth:

INCOMING_BANDWIDTH = ((TOTAL_ACTIVE_USERS REQUEST_PER_DAY_BY_EACH_USER EACH_REQUEST_SIZE) / 24_HOURS_IN_SECONDS) bytes/s

In numbers: (30 10^6 50 200) / (24 60 * 60) bytes/s

Outgoing Bandwidth:

Unlike the incoming request, the response will have both textual and visual content to build up a map.

OUTGOING_BANDWIDTH = ((TOTAL_ACTIVE_USERS REQUEST_PER_DAY_BY_EACH_USER EACH_RESPONSE_SIZE) / 24_HOURS_IN_SECONDS) bytes/s

Building Blocks

Load Balancer: To distribute traffic between different services and servers.

Database: To store the location data and metadata in graph format.

Pub-Sub System: Navigate users on the right directions and also invoke related services.

Key-Value Storage: Also, store the metadata information.

High-Level Design

Components

• Location Finder: Reads the user's current position. May use GPS/WIFI/Cellular technology to find out the current accurate location.

• Distributed Search: This determines the latitude and longitude of a place entered by the user. It's a mapping service that translates the place name to the latitude and longitude.

• Route Finder: Using two positions, will generate all possible routes.

• Graph Processing Service: Between multiple routes, will determine the optimal route.

• Area Search Service: Orchestrates the distributed search, route finder, and graph processing service. At any time, using two places, get the actual position by invoking the distributed search. From these two locations, find the routes using the route finder. Later, use the graph processing service to get the optimal route.

• Database: Stores the routes, nodes, and edge data. Perhaps, utilize the graph database like DataStax.

• Pub-Sub System: While the user moves through the route, to navigate and guide, the pub-sub model will be used.

• Third-Party Data Receiver: Gets data from various sources about city, roads, and places. Pre-processes the third-party data so later these data can be used to store in the DB in native graph format.

• Graph Building: Graph building will be used to store the route data in the graph database that is suitable for making queries.

• User: The programs and people who are using the service.

• Load Balancer: Used to distribute traffic between different servers and services.

Workflow

• User enters the source (uses location finder for current location) and destination.

• The area search service will find the optimal route using distributed search, route finder, and graph processing service.

• Navigator will track the user's route. In case of distraction, the navigator will send an event to Kafka. Upon receiving the event, Kafka will recalculate the route using the area search service and send it back.

API Design

• Current location: Returns the user's current location.

• Get route: Based on source, destination, and transport type, return the optimal route.

• Get direction: At any point in time, based on the current location, give the next steps.

Challenges of Google Map Design

Scalability

We have a mesh of billions of nodes and edges. When we need to calculate the shortest path between two nodes, we cannot simply run the shortest path algorithm on the entire graph. Because, first of all, it is both impractical and impossible to load the entire nodes and edges into memory. Additionally, it will lead to performance issues for the users. Instead, we can divide the graph into different small segments. This will lead to:

• Smaller polygons to calculate the shortest path.

• We can query them in parallel, improving performance.

Segment

Calculate the shortest path between two nodes in a single segment:

• In a single segment, there will be a weight between each edge that connects two nodes.

• Weight will be calculated by the traffic density, average speed of the vehicles, and other criteria.

• Load the segment in memory and run the regular shortest path algorithm like Dijkstra.

• Cache the results to reuse in the future.

Calculate the shortest path between two places that are not nodes:

• Calculate the nearest nodes from the point of edge.

• Use the shortest path using these nodes.

Calculate the shortest path when the source and destination are from different segments:

• We will use the term 'Exit Point', where an exit point is the connecting point between two segments.

• Exit points will be shared by two or more segments.

• Using latitude and longitude, we can find the source and destination segments.

• Instead of going to all the segments, we will use a radius of approximate distance and discard any segment outside the radius during path computation.

• While finding the shortest path over segments, we only consider the exit point and do not consider the internals of the segments.

• For finding the shortest

segments path, we will do the rest of the computation inside the segments.

ETA Computation

• Using the pub-sub model, the navigation service will collect live location data.

• Considering the current location, traffic density (high, medium, low), traffic pattern (rush hour or not), find out the ETA.

Detailed Design of Google Maps

Segment Setup and Request Handling

Storage schema

We will use 3 types of data storage:

• Key-value store

• Graph database

• Relational database

After generating a segment, we will store the segment id and nodes (along with the edge) in the graph database.

We will utilize the key-value store to store the segment server id (where the segment is stored), boundaries (or exit points), and neighbor segment ids against the segment id.

In the traditional relational database, we will keep the information of the edges like weight, their traffic rush hours, and the related segment and node id.

Design/Create Segment

• First, get the data of the segment and get a segment id by a UUID generator.

• Store the segment in the server using a 'server allocator service'.

• Now we have saved the segment with the segment id.

• Create the key-value store using segment id and its boundaries, neighbors, and server id.

• Insert the edges' information of the segment into the relational database.

Handle User Request

• When the user gives us the source and destination, find the latitude and longitude using the 'distributed search' service.

• Find out the source and destination segments from the graph database.

• Find all the relevant neighbors/segments using a radius.

• The 'route finder' will find all the optimal routes using these segments.

• The 'graph processing service' will find the appropriate route from the list of optimal routes.

Improve estimation using live data

• Web-socket & Load-balancer: Using the WebSocket, the server can do two-way communication. A load balancer can be used to limit the number of socket connections on the server.

• Pub-Sub: The pub-sub model collects the data streams and puts them into a data analytics service like Apache Spark. The 'map update service' listens to the analytics result and updates the data storage.

Performance

Our target is:

• The system can handle millions of user requests.

• The system can handle a large amount of data.

Availability

• Instead of storing the graph on a single server, we are utilizing individual servers for each segment.

• This will help load the relevant segment into memory and query; otherwise, loading the entire graph will lead to out-of-memory errors.

• We will utilize lazy loading so only relevant segments will load up in memory.

Scalability

• Since we load segments only based on user requests, we can scale to a degree.

• Also, for more segment data, we will utilize more servers that will not degrade individual server performance.

Reduce response

• We will query smaller subgraphs (only relevant segments).

• Cache the results of the query.

Accuracy

• Use live location from the user to guide accordingly.

• Update the edge weight based on traffic patterns.

I reached the raceground quite early, around 4:30 PM. I collected the kit; however, the jersey was not available by then. In about an hour, I found some familiar faces and had some conversations about the previous events.

The starting point of the 21KM race was 1.5 KM away from the raceground. After reaching there, I found my favorite pacer. He was aiming to complete the run in 2 hours and 40 minutes. My plan was to run with him for at least 1 hour and later, if possible, increase the speed.

During the run, we discussed a lot of things, including the motive behind the run. It was great to hear the different backgrounds of other runners and their initial inspirations. All the way, he guided us with stepping and breathing instructions.

After crossing 15 KM, I decided to push myself further. Until that date, my personal best for the half marathon was 2 hours and 33 minutes. This time, I reduced it by 6 minutes, achieving a new personal best.

After the run, I met a few fellow runners, collected the medal, food, and returned home.

This was the last event of my intense month of running. Now, I am focusing on recovery training and will continue with recovery exercises until the end of the month.

I reached there very early in the morning by train. Moulavibazar is one of the most favorite places in Bangladesh, and I've always been fond of running through the tea gardens. Fortunately, this running track was also on the Dankan Brothers tea estate, and my surprises started along the route.

We started our run on time. After 200 meters on the road, we moved to an open rice field. Then, after one and a half kilometers, we went to a hill in a tea garden. According to my previous knowledge from other runners, this should be the rest of the track. To my surprise, after a few kilometers, we left the tea garden and entered a rubber tree garden. My reaction at the time was, "Nice one." But things changed dramatically when we were done completing the rubber garden area. There were cascades here and there. The beauty of nature confused me—was I in a running event or on another expedition in the Bandarban area? After 10 kilometers, we had to cross a 200-meter hill and found a very old temple.

The last time I completed a 25K distance in 3 hours, and initially this time, I planned to take more time as I was tired from the last two weeks of long-distance runs. But the road was too beautiful. In some places, instead of running, we were doing a photoshoot. When I reached the finish point, the organizers had arranged a feast just like a wedding festival.

A few of my friends were coming to the next district to travel, and I had plans to accompany them. I rushed to Sylhet from Moulavibazar. Their bus was late, so I reached there before them. In two days, we visited Lovachora, Dibir Hor, and Lalakhal. Among these places, I had visited Lalakhal before, but with the right company, travel is always fun.

It was a great memory, especially finding stories while running the Rajkandi Hill 25K and visiting Lovachora.

Functional Requirements

When designing a Q&A site, we only consider the following features and also ensure that, in the future, we can extend the features.

• Questions and answers functionality

• Commenting to the questions along with upvote and downvote capabilities

• Recommendation system for personalized home feeds and advertising

• Ranking mechanism for answers

Non Functional Requirements

The site should have the following characteristics due to the large number of users all over the world,

• Scalability: The architecture should accommodate additional features and support a growing user base seamlessly.

• Consistency: The questions and answers should be consistent with any set of users. This does not imply, that the newly added Q&A should be available to all the users right away.

• Availability: The system should be highly available. It should handle a large number of concurrent requests and also be able to perform in case of a few server failures.

• Performance: The system should serve the user without noticeable delay.

Resource Estimation

Consider 300 million active users. Each user will make 20 requests per day. With these assumptions, we are going to calculate the,

• Number of servers

• Storage size that includes database and blob storage

• Network bandwidth

Servers

• Each server can handle 8000 requests per second

• Total request per day, TOTAL_REQUEST_PER_DAY = (3x10^6) * 20

• Requests per second, TOTAL_REQUEST_PER_SECOND = (TOTAL_REQUEST_PER_DAY) / (24 60 60)

• Number of servers required (TOTAL_REQUEST_PER_SECOND / 8000)

Storage

• 15% Q&A will have the image

• 5% Q&A will have video

• For a Q&A there will be either an image or video or nothing. Both video and image can not be in a single Q&A

• Consider 1 question from each active user

• Two responses from each question

Calculate the storage

Let's assume

• Each image size is 250 KB

• Each video size is 5 MB

• Text content and metadata regarding a question is 100 KB

Image Storage

• 15% of total Q&A has image

• Size 15% of 300 Million * 250 KB = ~ 11 TB

Video Storage

• 5% of total Q&A has video

• Size (5% of 300 Million) * 5 MB = ~75 TB

Text Content

• 1 Q&A per active user

• Total 300 million active user

• Each Q&A has an estimated of 100 KB textual content and metadata

• Total 30 TB

Each of the day, we will require 11 + 75 + 30 = ~116 TB of storage

Bandwidth

Incoming Bandwidth

• we will send 116 TB = (116 * 8) GB data per day

• Bandwidth per second (116 * 8) / (24 hours in seconds) ~ 11 Gb/s

Outgoing Bandwidth

• Consider 300 million active user

• Consider each user sees 20 questions each day

  • 300 million * 20 Q&A, each Q&A has 100 KB of textual and metadata

  • ~600 TB

• Consider that 15% of Q&A has an image

  • 300 million active user

  • Each user fetches 20 Q&A

  • 15% of this Q&A have an image

  • Each image has a size of 250 KB

  • ~225 TB

• Consider 5% of Q&A has video

  • 300 million active user

  • Each user fetches 20 Q&A

  • 5% of these images have video

  • Each video has a size of 5 MB

  • ~1500 TB

• Now total size of data: 600 + 225 + 1500 = 2325 TB ~ 2500 TB ~ 2500000 GB ~ 20000000 Gb

• Outgoing bandwidth 20000000 / (24 hours in seconds) = 231 Gb/s

Building Blocks

• Load Balancer: Distribute traffic between servers and services

• Database: Store textual content in the DB

• Distributed Caching: Schedule tasks and reduce loads to db and services

• Blob Store: Store the images and videos

Web and Application Server

• To handle requests

  • web server for the manager processes

  • application server for the worker process

• Application servers maintain an in-memory queue to process different user requests

• A router library between the web and application server

• Manager process enqueued the tasks

• Application process dequeued the tasks

Data Stores

• A relational database MySQL for storing Q&A and comments, as it has a high level of consistency

• An hBase to store metadata as it has a very high throughput in storing and retrieving data. Use these

• Stats for recommendation later

• Blob storage for the images and videos

Distributed Cache

• memcacheD for critical data caching

• Redis for upvote type data as it has in-store increment

• CDN for serving images and videos

Computer Servers

• For recommendation engine or ranking

• The process will be online and offline

• Probably running some ML operations

• May have lots of memory and high processing power

Workflow

Posting Q&A and comments

• The web server receives the request and passes it to the application server

• Web servers also manipulate the web page like the request is in progress

• The worker process will manipulate the database, ex: fetch or save data

• The task will be prioritized by different queue

  • User requests will be served earlier

  • The weekly digest will have less priority

• Images and videos will be stored in the blob storage

• The answer ranking system will rank the answers based on upvotes, views, dates, and some other properties. An ML engine will back it up.

• Extract and store metadata of answers, comments, images, and videos in hBase and feed these metadata to ML and rank offline

Recommendation System

• Run both online and offline, used for

  • User feed

  • Find duplicate

  • Generate add

Search

• Build index and store in DB and keep frequently used in hBase

• Make an index by tokenizing from Q&A, level, and comments

Limitations

Latencies of web and application servers: Latency of web and application servers communications.

In Memory Queue Failure: Tasks are queued in the queue. If a queue is failed, a lot of manual engineering will be required. Replicating the queue can be a solution but it will require extra memory. Tasks like view count should not hamper comparatively more important tasks like saving answers or questions.

MySQL QPS: Since we offer a lot of services, it is possible to encounter a lot of queries in our MySQL server. This will result in a huge latency in getting the query results.

HBase Latency: Although the HBase has a high throughput, it has slow latency. On top of that, since, we rely on the ML, at one point, it will have a poor performance.

Adjustment for the mentioned limitations

Latencies of web and application servers: Using a service host. A large powerful machine to handle all web and application processes at once in a single place.

In Memory Queue Failure: Use Kafka instead.

MySQL QPS: Use vertical sharding. If there are joins involved between two tables, put them in the same shard.

hBase Latency: Use MyRocks instead. Allows improved latency and data transfer tools between RocksDB and MySQL.

Conclusion

Understanding the intricate system design of a Q/A site like Quora offers invaluable insights into creating scalable, efficient, and reliable platforms in the digital age. By addressing functional and non-functional requirements, designing robust building blocks, establishing efficient workflows, and implementing mitigation strategies for potential limitations, developers can craft high-performance systems capable of meeting the demands of millions of users worldwide.

From Ultra to Dhaka Marathon

Just a week prior to the Dhaka Marathon, I undertook the daunting challenge of a 50 KM Ultra Marathon. Acknowledging the limited recovery time between these two demanding races, I faced the dilemma of participating in both. The Marine Drive Ultra Marathon, a prestigious event in Bangladesh, presented an opportunity I couldn't ignore.

Despite the physical toll of the Ultra Marathon, the allure of the Dhaka Marathon was too strong to resist. With a mere week between the two events, I carefully gauged my fitness. Feeling surprisingly good after the Ultra, I made the decision to take on the Dhaka Marathon, fully aware of the challenges that lay ahead.

The Dhaka Marathon Experience

The Dhaka Marathon, renowned for its punctuality, commenced at 5:15 AM amidst the speeches of esteemed guests. The initial kilometers were invigorating, with the absence of scorching sunlight. However, the race presented its own challenges, including heavy winds that added an element of discomfort.

The first 20 KM of the race were filled with camaraderie as fellow runners cheered each other on. As the race progressed, energy began to wane, and temperatures soared. With a strategic plan in mind, I aimed to run until the 32 KM mark and then adopt a walking pace. Preserving energy was crucial, as another challenging event awaited—a 25 KM run in a reserve forest the following week.

Executing my plan, I walked the final 10 KM, completing the marathon in 5 hours and 35 minutes. Despite the inevitable fatigue, the sense of accomplishment overshadowed any minor injuries. The pain under my left foot was a testament to the challenges faced and overcome.

Post-Race Reflections

After crossing the finish line, I joined fellow runners, exchanged stories, and collected my well-deserved medal and refreshments. The post-race moments, captured in group photos, serve as lasting memories of conquering a remarkable long-distance running challenge.

Embark on a journey through the unique landscape of long-distance running in Bangladesh, as I navigate the challenges of back-to-back marathons, sharing the highs and lows of conquering extraordinary distances.

Preparation Challenges:

Participation in the Marine Ultra Marathon requires participants to meet rigorous prerequisites, including completing at least four half marathons in the four months leading up to the event. Despite my reluctance to adopt modern tracking apps like Strava, I successfully navigated these requirements by providing detailed documentation, certificates, and personal motivation in the application form.

Countdown to Race Day:

In the week leading up to the marathon, I pushed my limits with an almost 50 KM run and an eight-hour regular workout regimen. The logistical challenges were not to be underestimated, involving a 600 KM journey by bus and train. The pressure mounted, and sleep eluded me the night before the race.

The Marathon Journey:

Once the race commenced, the pre-race anxieties dissipated, and my focus shifted solely to the next steps. Unlike my usual strategy of adhering to a strict time frame per kilometer, this time I adopted a more intuitive approach – it was not a strategy, just running until I was alive and facing no major injury. Hydration stations every 5 KM became my lifeline, with brief 1-2 minute breaks to refuel. Surprisingly, an abundance of chocolate, water, saline, glucose, biscuits, and fruits provided the sustenance needed for the miles ahead.

The Challenge Unfolds:

The initial 25 KM unfolded seamlessly, but the true test began as my body entered uncharted territory. I always knew that after 20 KM, my energy started to drain linearly. However, after 30 KM, I discovered that it started to drain exponentially. Between 30 and 40 KM, running and walking sustained me. However, the last 10 KM proved to be a monumental struggle, with every bone and joint seemingly at its breaking point. Despite the physical toll, I pressed on, determined to reach the 50 KM mark before the cut-off time.

Triumph at the Finish Line:

After a grueling 7 and a half hours, I crossed the finish line. The overwhelming joy and sense of accomplishment brought tears to my eyes. The cheers of children and the celebratory atmosphere marked the culmination of a test of endurance at its peak.

Post-Marathon Reflection:

The undeniable toll on my body quickly altered post-run plans for a relaxing island visit. Despite the initial intention to get a massage, the fatigue was so profound that even walking 200 meters became a daunting task. I spent a few hours recuperating, had lunch, and made the journey back to Dhaka that same night. Like always, I took the same bus, at the same time while returning to Dhaka, It has a special 360-degree view (not exactly!). It was special, because, I took this bus for the 4th time in a row in the last 4 months.

The Marine Ultra Marathon was not just a race but a profound journey that tested my physical and mental resilience. Overcoming challenges, embracing the unknown, and triumphing at the finish line made it an experience worth every step.