React - Flask - RQ - Scrapy - MongoDB Pipeline Integration and Debug by Miclin1024 · Pull Request #28 · comp-strat/scraping_server

Miclin1024 · 2022-03-09T19:27:13Z

No description provided.

Miclin1024 · 2022-04-07T05:50:40Z

The workflow is functioning properly now. I'm able to use the UI to submit a job request through the API, have Flask process the request and send it to RQ whose worker will pick up the job and run Scrapy, and confirm that Scrapy will actually sync the job status and scraped data with the MongoDB instance.

I had to disable sign in due to some lingering issues with the authentication setup, but I think it would be a good idea to merge this branch first @jhaber-zz.

Miclin1024 · 2022-04-07T05:55:59Z

@@ -0,0 +1 @@
+from .run_spider import scrapy_execute


In this new setup, the Scrapy project becomes a sub-package under server. A driver function scrape_execute in run_spider.py was added to serve as an entry point for rq.

Miclin1024 · 2022-04-07T06:01:06Z

 #     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
 import os

+from scrapy.utils.project import get_project_settings


scrapy.settings was integrated into server.settings. Reference to google api token was removed for security. It is now in a .env file that's source controlled. A template file for .env was added in #29.

Miclin1024 · 2022-04-07T06:03:34Z

@@ -0,0 +1,43 @@
+const path = require("path");


The new CI setup using webpack and babel. This is being used in package.json's npm scripts. An update to our current README is needed.

Miclin1024 · 2022-04-07T06:04:31Z

@@ -1,123 +0,0 @@
-from pymongo.results import UpdateResult


Renamed to tracking.py

Miclin1024 · 2022-04-07T06:05:27Z

@@ -1,51 +0,0 @@
-{


Moved npm packages to project root.

Miclin1024 · 2022-04-07T06:07:05Z

@@ -1,372 +0,0 @@
-import React, {Component} from "react";


Renamed to JobsDashboard.js to better distinguish it from Job.js.

Miclin1024 · 2022-04-07T06:08:17Z

@@ -0,0 +1,43 @@
+module.exports = {


ESLint for enforcing coding style.

Miclin1024 · 2022-04-07T06:18:31Z

-import AutorenewIcon from '@material-ui/icons/Autorenew';
-import Box from '@material-ui/core/Box';
-import Paper from '@material-ui/core/Paper';
+import {


These new import statements are for migrating to material-ui v5 and react-router-dom v6.

In terms of actual code change, for mui most of the updates were due to the deprecation of makeStyle, and for react-router-dom it was the removal of the history state. This part was quite significant because of the extensive use of both APIs in our code.

Miclin1024 · 2022-04-07T06:20:42Z

@@ -0,0 +1,30 @@
+import os


For modularity the Flask backend is restructured using package and blueprints.

Miclin1024 · 2022-04-07T06:23:09Z

+        app.config.from_mapping(test_config)
+
+    from .home import views as home_views
+    app.register_blueprint(home_views.bp)


home/views.py serves various entry points to React. It renders an HTML template along with the javascript bundle from babel/webpack.

Miclin1024 · 2022-04-07T06:25:08Z

+    from .jobs import interfaces as job_interfaces
+    app.register_blueprint(job_interfaces.bp)
+    from .jobs import actions as job_actions
+    app.register_blueprint(job_actions.bp)


jobs/interfaces.py and jobs/actions.py serve the APIs interacting with the job dispatcher and tracker.

Miclin1024 · 2022-04-07T06:26:00Z

+    <title>Crawl4All: A Universal Web Crawling App</title>
+</head>
+<body>
+    <div id="react-app"></div>


All React content will be rendered inside this <div> box.

Miclin1024 · 2022-04-07T06:26:56Z

+    def decorator(*args, **kwargs):
+        token = None
+        if DEBUG_NO_AUTH_ENABLED:
+            g.user = "NO_AUTH_ENABLED"


The temporary backdoor to bypass user authentication until it is fixed.

Miclin1024 · 2022-04-07T06:27:53Z

+
+@bp.before_request
+@token_required
+def before_request():


Require auth for all APIs in this blueprint

Miclin1024 · 2022-04-07T06:29:48Z

+@validate_job_id
+def before_request():
+    """Validate user and job_id at all action endpoints."""
+    pass


Validate all job_id prior to all requests to server.actions and standardize error handling.

Miclin1024 added 14 commits February 14, 2022 12:41

Updated gitignore

013f5e5

Refresh project structure for a unified app

3565a3d

Update .gitignore

f42d449

Setup works, debugging APIs

be21d27

Clean up

1dd5bb9

WIP React Refactoring

26bc19c

Server side REST API migration

86b1318

Client side mui migration

8daf60a

Try integration

dec2e68

Set proper content type

4538a93

Finished frontend react-router-dom refactoring

4b232b4

Use localhost to allow GoogleAuth

7188c3b

Investigate router issue

af31620

Added tracking.py to replace crawl_task_tracker

85b82e6

Miclin1024 requested a review from jhaber-zz March 9, 2022 19:28

Miclin1024 added 7 commits March 30, 2022 19:13

Server migration complete

f962b89

Explicit None to silent warnings

8135ac7

Fixed module import

3f6ae9f

Use global context object to validate/pass user and job_id

2a29eb1

Fixed responsive navbar et al.

771066e

Frontend pipeline debugged, temporarily removed sign in

458345e

Test scrapy on more urls

cf6323f

Miclin1024 requested a review from hwarnuh April 7, 2022 05:31

Miclin1024 changed the title ~~[WIP] Package update, NodeJS removal, and workflow refactoring~~ Scrapy, Flask, React intergration and workflow debugging Apr 7, 2022

Miclin1024 changed the title ~~Scrapy, Flask, React intergration and workflow debugging~~ React - Flask - RQ - Scrapy - MongoDB Pipeline Integration and Debug Apr 7, 2022

Miclin1024 commented Apr 7, 2022

View reviewed changes

Comment thread client/package.json

@@ -1,51 +0,0 @@

{

Miclin1024 Apr 7, 2022

Copy link
Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved npm packages to project root.

Miclin1024 commented Apr 7, 2022

View reviewed changes

Comment thread .eslintrc.js

@@ -0,0 +1,43 @@

module.exports = {

Miclin1024 Apr 7, 2022

Copy link
Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ESLint for enforcing coding style.

Miclin1024 commented Apr 7, 2022

View reviewed changes

Miclin1024 requested a review from ThaiHipster April 13, 2022 20:36

Uh oh!

Conversation

Miclin1024 commented Mar 9, 2022

Uh oh!

Miclin1024 commented Apr 7, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant