Improving File Management for a Data Management & Analytics Expert

Challenge

Since the business model deals with the storage of large amounts of structured and unstructured data in the form of documents, images and videos for analytics purpose, reliability, availability, and durability along with cost becomes a major challenge. Another challenge was accidental deletion of data with less or no option for retrieval.

Shore Group also wished to overhaul their existing document repository and discovery product to meet the growing market expectations related to search and performance.

Solution

1. Set up file management and content processing engine based on the Serverless Application Modelling Architecture of AWS
2. AWS DynamoDB has been configured to manage the workflow entities
3. AWS SQS has been configured to ensure parallel processing of file movement from one bucket to another and to extract content from each file
4. AWS Elasticsearch has been configured to allow a full-text search of the document
5. We used the S3versioning to preserve, retrieve, and restore every version of every object stored.

-SAP Solutions Manager 7.2 is to be Installed at AWS.

Benefits

Shore has adapted AWS to resolve existing challenges, also achieving durability of 11 9’s and availability of 99.9% through it. S3 versioning was leveraged to ensure data was never lost even if it is accidentally deleted.

The need to have SAN, NAS infrastructure for redundant backup storage and infrastructure was no longer necessary as buckets could be replicated. EBS based solution with EC2 instance running watcher functions was not an ideal solution for shore group as it did not ensure security or reliability.

Nodes with watcher functions on EC2 instances to watch for files to be processed were replaced with SQS along with the logic of processing of the input jobs automatically using Lambda. This has enabled shore to go serverless reducing overheads of SLA and increasing performance.

Data breach threat was solved by using Server Side Encryption and Key Management Service – KMS-SSE combination to protect S3 buckets.