Feat/parser in ecs by ranjan-stha · Pull Request #3 · the-deep/deepl-deep-integration

ranjan-stha · 2022-08-05T09:21:41Z

Addresses xxxxxx
Depends on xxxxxx

Changes

ECS to deploy the deepex parser
Lambda to use bucket for file conversion if not in pdf or html
Sentry integration.

Mention related users here if any.

This PR doesn't introduce any:

[ X] temporary files, auto-generated files or secret keys
[ X] n+1 queries
[X ] flake8 issues
print
[X ] typos
[ X] unwanted comments

This PR contains valid:

tests
permission checks (tests here too)

…ated the configs in gateway for prod, staging envs;

…files;

…no extraction happens);

…ated the configs in gateway for prod, staging envs;

…ile to find the extension of that file;

…bucket/key payload;

bimal125 · 2022-08-09T11:31:32Z

+    local_temp_directory = pathlib.Path('/tmp', file_name)
+    local_temp_directory.mkdir(parents=True) if not local_temp_directory.exists() else None
+    # Note: commented for now
+    # images.save_images(directory_path=local_temp_directory)
+


Comment these lines

bimal125 · 2022-08-09T11:32:33Z

+
+    s3_file_path = f"s3://{DEST_BUCKET_NAME}/{str(s3_path_prefix)}/{file_name}"
+    s3_images_path = f"s3://{DEST_BUCKET_NAME}/{str(s3_path_prefix)}/images"
+


Make a general utils for this

bimal125 · 2022-08-09T11:34:16Z

+
+    extracted_text = extracted_text.replace("\x00", "")  # remove null chars
+


Lets make a util function for this

bimal125 · 2022-08-09T11:36:38Z

+    s3_file_path = f"s3://{DEST_BUCKET_NAME}/{str(s3_path_prefix)}/{file_name}"
+    return s3_file_path, None, total_pages, total_words_count  # No images extraction (lib doesn't support?)


Lets make util function

bimal125 · 2022-08-09T11:46:59Z

+                url, file_name
+            )
+            if s3_file_path:
+                extraction_status = ExtractionStatus.SUCCESS.value
+            else:
+                extraction_status = ExtractionStatus.FAILED.value
+        except Exception:
+            logging.error(f"Error occurred during text extraction. {str(e)}", exc_info=True)
+            s3_file_path, s3_images_path, total_pages, total_words_count = None, None, -1, -1
+            extraction_status = ExtractionStatus.FAILED.value
+    elif content_type == UrlTypes.DOCX.value or content_type == UrlTypes.MSWORD.value or \
+            content_type == UrlTypes.XLSX.value or content_type == UrlTypes.XLS.value or \
+            content_type == UrlTypes.PPTX.value or content_type == UrlTypes.PPT.value:
+
+        ext_type = content_type
+        tmp_filename = f"{uuid.uuid4().hex}.{ext_type}"
+        flag = False
+        if upload_file_to_s3(url, key=tmp_filename, bucketname=DOCS_CONVERSION_BUCKET_NAME):
+            payload = json.dumps({
+                "file": tmp_filename,
+                "bucket": DOCS_CONVERSION_BUCKET_NAME,
+                "ext": ext_type,
+                "fromS3": 1
+            })
+
+            docs_conversion_lambda_response = lambda_client.invoke(
+                FunctionName=DOCS_CONVERT_LAMBDA_FN_NAME,
+                InvocationType="RequestResponse",
+                Payload=payload
+            )
+            docs_conversion_lambda_response_json = json.loads(
+                docs_conversion_lambda_response["Payload"].read().decode("utf-8")
+            )
+
+            if "statusCode" in docs_conversion_lambda_response_json and \
+                docs_conversion_lambda_response_json["statusCode"] == 200:
+                bucket_name = docs_conversion_lambda_response_json["bucket"]
+                file_path = docs_conversion_lambda_response_json["file"]
+                filename = file_path.split("/")[-1]
+
+                if download_file(file_path, bucket_name, f"/tmp/{filename}"):
+                    s3_file_path, s3_images_path, total_pages, total_words_count = get_extracted_content_links(
+                        f"/tmp/{filename}", file_name
+                    )
+                    if s3_file_path:
+                        extraction_status = ExtractionStatus.SUCCESS.value
+                    else:
+                        extraction_status = ExtractionStatus.FAILED.value
+                else:
+                    flag = True
+            else:
+                logging.error(f"Error occurred during file conversion. {docs_conversion_lambda_response_json['error']}")
+                flag = True
+        else:
+            logging.warn("Could not upload the file to s3.")
+            flag = True
+
+        if flag:
+            s3_file_path, s3_images_path, total_pages, total_words_count = None, None, -1, -1
+            extraction_status = ExtractionStatus.FAILED.value
+    elif content_type == UrlTypes.IMG.value:
+        logging.warn("Text extraction from Images is not available.")
+        s3_file_path, s3_images_path, total_pages, total_words_count = None, None, -1, -1
+        extraction_status = ExtractionStatus.FAILED.value
+    else:
+        logging.error(f"Text extraction is not available for this content type - {content_type}")
+        s3_file_path, s3_images_path, total_pages, total_words_count = None, None, -1, -1
+        extraction_status = ExtractionStatus.FAILED.value
+
+    logging.info(f"The extracted file path is {s3_file_path}")
+    logging.info(f"The extracted image path is {s3_images_path}")
+    logging.info(f"The status of the extraction is {str(extraction_status)}")


Lets improve this fucntion

thenav56 · 2022-08-10T04:23:34Z

+SENTRY_URL = os.environ.get("SENTRY_URL")
+ENVIRONMENT = os.environ.get("ENVIRONMENT")


Maybe use django-environ for env configuration and load all configurations from a single app like config.py?

https://django-environ.readthedocs.io/en/latest/
example: https://github.com/the-deep/server/blob/develop/deep/settings.py#L22-L85

I think this is not that important to use.. just adds an extra dependency. but moved all the envs to a config.py file.

thenav56 · 2022-08-10T04:23:54Z

+docs_convert_bucket_name = "deep-large-docs-conversion"
+
+# sentry url
+sentry_url = "https://[email protected]/1223576"


This is sensitive info.

stored in aws secrets.

thenav56 · 2022-08-10T04:25:16Z

+SENTRY_URL = os.environ.get("SENTRY_URL")
+ENVIRONMENT = os.environ.get("ENVIRONMENT")


Let's define all the config in a single file.

thenav56 · 2022-08-10T04:26:04Z

+    && apt-get autoremove -y \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY deepex_ecs/app.py content_types.py wget.py /code/


We don't need other files?

thenav56 · 2022-08-10T04:26:57Z

+    key="temporaryfile.pdf",
+    bucketname="deep-large-docs-conversion"


Let's not define the default value here or use the value from config.

thenav56 · 2022-08-10T04:31:42Z

+    key="temporaryfile.pdf",
+    bucketname="deep-large-docs-conversion"
+):
+    try:


Maybe define a class/object for handling upload/download/URL.

Something like this:
Definition: https://github.com/the-deep/serverless/blob/develop/src/common/s3.py
Usages: https://github.com/the-deep/serverless/blob/develop/src/functions/source_extract/models.py#L92-L109

thenav56 · 2022-08-10T04:37:37Z

+            elif url.endswith(".jpg") or url.endswith(".jpeg") or url.endswith(".png") or \
+                url.endswith(".gif") or url.endswith(".bmp") or content_type in self.content_types_img:


Maybe use any here.

Suggested change

elif url.endswith(".jpg") or url.endswith(".jpeg") or url.endswith(".png") or \

url.endswith(".gif") or url.endswith(".bmp") or content_type in self.content_types_img:

elif (

content_type in self.content_types_img or

any([

url.endswith(f".{extension}") for extension in [

"jpg", "jpeg", "png", "gif", "bmp"

]

])

):

thenav56 · 2022-08-10T04:46:00Z

+                if temp_filepath.endswith(".pdf"):
+                    return UrlTypes.PDF.value
+                elif temp_filepath.endswith(".docx"):
+                    return UrlTypes.DOCX.value
+                elif temp_filepath.endswith(".doc"):
+                    return UrlTypes.MSWORD.value
+                elif temp_filepath.endswith(".xlsx"):
+                    return UrlTypes.XLSX.value
+                elif temp_filepath.endswith(".xls"):
+                    return UrlTypes.XLS.value
+                elif temp_filepath.endswith(".pptx"):
+                    return UrlTypes.PPTX.value
+                elif temp_filepath.endswith(".ppt"):
+                    return UrlTypes.PPT.value
+                elif temp_filepath.endswith(".jpg") or temp_filepath.endswith(".jpeg") or temp_filepath.endswith(".png") or \
+                    temp_filepath.endswith(".gif") or temp_filepath.endswith(".bmp"):
+                    return UrlTypes.IMG.value
+                else:
+                    logging.warn(f'Could not determine the content-type of the {url}')
+                    return None


Maybe do this using dict.

Suggested change

if temp_filepath.endswith(".pdf"):

return UrlTypes.PDF.value

elif temp_filepath.endswith(".docx"):

return UrlTypes.DOCX.value

elif temp_filepath.endswith(".doc"):

return UrlTypes.MSWORD.value

elif temp_filepath.endswith(".xlsx"):

return UrlTypes.XLSX.value

elif temp_filepath.endswith(".xls"):

return UrlTypes.XLS.value

elif temp_filepath.endswith(".pptx"):

return UrlTypes.PPTX.value

elif temp_filepath.endswith(".ppt"):

return UrlTypes.PPT.value

elif temp_filepath.endswith(".jpg") or temp_filepath.endswith(".jpeg") or temp_filepath.endswith(".png") or \

temp_filepath.endswith(".gif") or temp_filepath.endswith(".bmp"):

return UrlTypes.IMG.value

else:

logging.warn(f'Could not determine the content-type of the {url}')

return None

EXTENSION_TO_ENUM_MAP {

"pdf": UrlTypes.PDF,

"docx": UrlTypes.DOCX,

"doc": UrlTypes.MSWORD,

"xlsx": UrlTypes.XLSX,

"xls": UrlTypes.XLS,

"pptx": UrlTypes.PPTX,

"ppt": UrlTypes.PPT,

# Images

"jpg": UrlTypes.IMG,

"jpeg": UrlTypes.IMG,

"png": UrlTypes.IMG,

"gif": UrlTypes.IMG,

"bmp": UrlTypes.IMG,

}

file_extension = temp_filepath.split('.')[-1]

if file_extension not in EXTENSION_TO_ENUM_MAP:

logging.warn(f'Could not determine the content-type of the {url}')

return None

return EXTENSION_TO_ENUM_MAP[file_extension].value

ranjan-stha added 19 commits April 29, 2022 11:52

lambda functions updated with sentry

adad0a6

terraform configs updated with sentry_url passed as env variable; upd…

423a13a

…ated the configs in gateway for prod, staging envs;

values updated for dev, staging, prod environments in their variable …

8235f16

…files;

added the missing package dependencies in terraform config;

150791b

updated the parser tool url;

1e6b69e

enabled stacktrace in sentry; handles image links(and discards them, …

f168556

…no extraction happens);

enabled err stack;

74e353a

updated the hashicorp aws version;

9e09b2a

terraform configs updated with sentry_url passed as env variable; upd…

5eb3862

…ated the configs in gateway for prod, staging envs;

enabled err stack;

4f98fa5

updated the hashicorp aws version;

54e86e7

if content type is not determined with head requests, downloads the f…

4274ac8

…ile to find the extension of that file;

user agent added while sending get or head request;

2ddbcdd

doc conversion is done by uploading in s3 and triggering lambda with …

4623d60

…bucket/key payload;

deploy deepex parser in the ecs;

db7b492

deepex parser dockerfile;

d73c26b

reduce the number of privisioned lambda;

2902c1f

try..except block used;

8399454

param values added for prod env

1914f13

ranjan-stha requested review from bimal125 and thenav56 August 5, 2022 09:21

ranjan-stha assigned thenav56 Aug 5, 2022

bimal125 reviewed Aug 9, 2022

View reviewed changes

thenav56 requested changes Aug 10, 2022

View reviewed changes

ranjan-stha added 2 commits September 1, 2022 14:14

config values for dev env;

4acc48a

code refactored in parser for text extraction;

708b73a

ranjan-stha added 24 commits September 1, 2022 16:34

env added;

dc0fd27

updated lambda concurrency for staging env;

72da790

sentry url fetched from secrets manager;

4b823a1

sentry secret name assign;

67e3324

updates to use cpu model for prediction;

4dd5d45

fixes on env vars usages; refactored code;

f89506a

removed unused dependency;

a3b9a04

updated env vars

5024243

moved env vars to a separate file;

b6daeb3

prod vars updated;

da2e6da

parsing lib updated; method updated;

b27963f

tf vars updated on ecr url;

46532b2

utf-8 added to content type while saving file in s3;

80fe746

added vpc endpoint for transferring data to s3;

1b1f94b

added variables for setting the lambda concurrency

86f2f8e

updated poetry lock file; added sentry pkg;

828bdb1

updated the methods for parser library;

b56e61b

ecs vcpu and memory reduced;

35af4fa

ecs vcpu and memory updated in prod;

f07893a

refactor and removed few non-required functions;

7e8ae4e

dockerfile updated;

6852eca

updated the poetry requirements;

f612614

updated the deep parser import;

503dc59

updated the mock data related to preds, thresholds and selected tags;

f13dbf4


		s3_file_path = f"s3://{DEST_BUCKET_NAME}/{str(s3_path_prefix)}/{file_name}"
		s3_images_path = f"s3://{DEST_BUCKET_NAME}/{str(s3_path_prefix)}/images"


		extracted_text = extracted_text.replace("\x00", "") # remove null chars

		s3_file_path = f"s3://{DEST_BUCKET_NAME}/{str(s3_path_prefix)}/{file_name}"
		return s3_file_path, None, total_pages, total_words_count # No images extraction (lib doesn't support?)

		SENTRY_URL = os.environ.get("SENTRY_URL")
		ENVIRONMENT = os.environ.get("ENVIRONMENT")

		key="temporaryfile.pdf",
		bucketname="deep-large-docs-conversion"

		elif url.endswith(".jpg") or url.endswith(".jpeg") or url.endswith(".png") or \
		url.endswith(".gif") or url.endswith(".bmp") or content_type in self.content_types_img:

-            elif url.endswith(".jpg") or url.endswith(".jpeg") or url.endswith(".png") or \
-                url.endswith(".gif") or url.endswith(".bmp") or content_type in self.content_types_img:
+            elif (
+                content_type in self.content_types_img or
+                any([
+                    url.endswith(f".{extension}") for extension in [
+                        "jpg", "jpeg", "png", "gif", "bmp"
+                    ]
+                ])
+            ):

Uh oh!

Conversation

ranjan-stha commented Aug 5, 2022

Changes

This PR doesn't introduce any:

This PR contains valid:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranjan-stha Oct 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ranjan-stha Oct 11, 2022 •

edited

Loading