Go to Swayam
- Home
- About
- Consulting
- Services
- Minfy Labs
- Industries
- Resonances
- Careers
- Contact
September 08, 2023 | Resonances type: Case study
Pos Malaysia is the national courier service provider and sole licensee for universal postal services in the country, delivering to more than 10 million addresses across the nation. With a track record of over 200 years, the Pos Malaysia Group has progressed from a traditional
postal service into a dynamic mail and parcel services, financial services and supply chain solutions provider with the largest delivery and touchpoint network in Malaysia. Pos Malaysia’s Integrated Parcel Centre (IPC) was upgraded to process over 300,000 items a day. With the growing demand, Pos Malaysia incorporates various technologies to deliver several services to its customers and to realize its vision, “We deliver. We connect. We improve lives.”
Pos Malaysia’s existing business model uses on-premises data centers for storing and processing all the data generated across different services. The company stores a large amount of data on-premises database which is used by different business units for data processing, transformation and further analysis. As the company is seeing an increase in orders, they are looking forward to leveraging cloud technology to build a data warehouse for data analytics. The company wants to build an end-to-end ML pipeline using this data warehouse to incorporate advanced ML algorithms that can predict ETA (Estimated Arrival Time). The company is also faced with challenges like scaling during data spikes and vertical scaling for heavy workloads. A secure solution to maintain and monitor different APIs for delivering integrated solutions. The company is therefore looking for a cost-effective solution that can address the above requirements and reduce its operational overheads. Being an AWS premier partner, Minfy brings in a range of deep cloud expertise in migration and modernization. Minfy believes that the client can modernize its database for scalability and security and leverage serverless architecture for its workflows. Minfy proposes various data analytics services offered by AWS for building advanced data analysis and ML pipelines enabling the company to be more agile in delivering its solution.
• The data resided in different databases. The client wanted a data warehouse solution to help its business units to access the right data and perform analytics in real time removing the existing manual interventions to fetch the data.
• The client wanted to enhance the user experience of their customers by reducing the time to predict the estimated arrival time. During peak time, the APIs took a long time to respond.
• Owing to the pandemic, there was a negative impact on the delivery time for parcels which significantly affected the estimated arrival time. The client wanted to revise and run an ML model to predict the ETA with a low margin of error.
• The client required a pipeline that can process data in batches and persist it to a NoSQL database for fast retrieval. Monitoring the APIs was becoming a challenge as the traffic in the overall application has increased to over a million requests daily.
• A robust and scalable infrastructure solution with minimal spending
• An architecture that can allow provision-on-demand spikes.
• Efficient and fast data warehouse solution for quick insights
• Low latency storage with minimum operational overheads
• Fast response and deployment times for APIs
• Security of data at rest and in transit across services
Transit Gateway was configured to establish a secure connection between the Virtual Private Connection in the cloud with the on-premises database/application.
• AWS API Gateway was configured to host and manage the private API service as a private API endpoint. It enables the client with functionalities like API keys for partner validation. Rate-limiting usage in order to implement throttling and handling unwanted spikes in usage.
• The basic level of validations of API requests was configured using API Gateway which also assists in maintaining multiple versions for the next phase releases.
• AWS Lambda was configured as a computer service triggered by API Gateway upon request. Lambda transforms the request into a valid API call for the Dynamo DB query and responds back with the relevant information.
• Dynamo DB was configured to store the connote ids of the parcel with an internal trigger to create a data lifecycle in order to purge legacy data from storage. Dynamo DB was configured as a key-value pair storage service which is a fast, scalable, and, flexible NoSQL database for low-latency data
• AWS Redshift was configured as the data warehouse system to store the data and create business reports.
• AWS Batch is a trigger-based service that will be used to create either event-based or time-based triggers in order to extract the delta data from Redshift on a periodic basis and process the data into the input set for Sagemaker
• AWS Sagemaker endpoint was configured for predicting ETA using ML algorithms.
• AWS Batch retrieves the output from SageMaker and pushes it into Dynamo DB for storage. The table structure was decided based on output structure from SageMaker.
• AWS S3 is the data source where the data extracted from the OAL database is stored.
• AWS Glue was configured to preprocess the raw data to relevant data structure for the data warehouse. It also helps in validating the data set and correcting any duplication-related errors.
• Key Management Service (KMS) was configured with the database services to ensure data encryption at rest. HTTPS API with certificate hosted in ACM was used to ensure data encryption in transit.
• CloudWatch was configured to monitor all the services being used in the architecture. It helped set up alerts in case infrastructure usage goes beyond expected parameters and assists in having automated resolutions for the same.
• Using API Gateway, the client was able to handle a huge number of requests. They were further able to reduce the cost using the tiered pricing model of this service. Since API Gateway is a fully managed service, the client was also able to reduce the operational efforts in managing the APIs and tracking requests.
• Since Lambda was querying results from the DynamoDB, the customers got the response from the API with very low latency even during peak traffic.
• Using AWS Glue, Redshift, Batch and Sagemaker, the client was able to implement the ML pipeline to deliver quick results. Using AWS Batch, the client was able to run Batch computing reducing the cost significantly
Minfy has a repository of learnings, competencies and an enviable track record of meeting customer needs. Advice and service, solutions and responsiveness work in tandem. Begin your cloud journey, accelerate it or optimise your cloud assets. Experience business impact.
© 2022 Minfy™. Minfy Technologies. All rights reserved. | Privacy Policy | Terms and Conditions
This website stores cookie on your computer. These cookies are used to collect information about how you interact with our website and allow us to remember you. We use this information in order to improve and customize your browsing experience and for analytics and metrics about our visitors both on this website and other media. To find out more about the cookies we use, see our Privacy Policy. If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference not to be tracked.