Web Scraper Improvements

[aubundy]

The current web scraper works maybe 95% of the time. Sometimes a website detects the bot and won't display the article. Other times a user will save a PDF or video they watched, but the current scraper is built to only handle articles, so the content doesn't display correctly. For a start, I created a checklist below of these edge cases/other features that would make the scraper more robust.

• Work better with bot detectors - Web Scraper Improvements - Work Better with Bot Detectors #40
• Better error handling. Python errors should be passed into an array (along with the values array) that is printed at the end of reading_scraper.py to be picked up by nodejs
• PDFs
• Tests
• YouTube and Vimeo videos
• Web-based Podcasts
• Better error handling/general refactoring for better readability
• Working with articles written in other languages
• As improvements are made to the scraper to make it more accurate, we need a way to go back and update old readings

Feel free to break off one of the tasks above into its own issue for better collaboration

[aubundy]

Update article scraper

[hiki270bis]

Hi, I don't know if can discuss here. As a newbie, I would like to try and start working on some of these issues. Are the devs who wrote the scraper part still on the project?

[aubundy]

Yes, we'd love your help! I wrote the scraper and can try to answer any questions you have

[hiki270bis]

Oh, nice to hear! I don't know many of the packages, so maybe when I have a doubt about one of them maybe I can ask you fro clarifications? Obviously I'll first read the docs. I tried to run the script from the terminal and it works if I just pass the link as an arg in the cmd line, but in the app, where does it get the url from? Can I assume it does work as if I am passing an arg from cmd line?
And also the results can I assume that that's the way the script has to present them in order then to be picked up by (I guess) the rest of the javascript which then displays them on the site?
And, last thing. Why are the reading_scraper and the reading_summary in two different files, and do they communicate somehow? 'Cause I don't see one importing the other but maybe that's just me missing something?

[aubundy]

Good questions!

Oh, nice to hear! I don't know many of the packages, so maybe when I have a doubt about one of them maybe I can ask you fro clarifications? Obviously I'll first read the docs.

Yes, that won't be a problem.

I tried to run the script from the terminal and it works if I just pass the link as an arg in the cmd line, but in the app, where does it get the url from? Can I assume it does work as if I am passing an arg from cmd line? And also the results can I assume that that's the way the script has to present them in order then to be picked up by (I guess) the rest of the javascript which then displays them on the site?

Yes, it should. In the app, the client sends a post request to /api/users/:id/readings. The createReading function in controllers/readings receives the user id and article url and sends them as arguments to the reading_scraper.py that parses the article and returns the values array in line 153 to be inserted into the db. The db then saves the title, domain, description, image, word count, and url.

And, last thing. Why are the reading_scraper and the reading_summary in two different files, and do they communicate somehow? 'Cause I don't see one importing the other but maybe that's just me missing something?

reading_summary.py generates a 500 word summary of the article. The idea was for someone who was interested in an article that was posted, but didn't want to read the whole thing, they could click a "summarize" button and read a summary in the app. It's not in use right now - I removed the frontend code that sends a request to this. I probably should have removed the file by now.

[hiki270bis]

Ok, so which issue do you think we should tackle first?
I feel like the pdf one is challenging but not impossible. I googled around and there are few packages that allow you to scrape the pdf content, so I think one could come up with some sort of content to show in the preview, even if I don't know how easy would it be to separate the title from the rest. As for the picture to show, I didn't find anything about showing a smaller preview image of the pdf, so I don't know if I would be able to do it on my own, but there's still the option, when the article is on pdf, to show a custom icon image of pdf.
Working better with bot detector seems tempting as well but I am not confident I could solve it. Besides, I don't know how powerful beautiful soup is as a scraper. Maybe we could consider using scrapy but again, I have no experience, I once tried to install that but couldn't get it to work properly, in the end I resolved to use selenium, but I don't think it would be very good for this kind of project, although I'm still inexperienced
Videos and podcast maybe it's just me but it would be a feature to add after we solved the pdf thing.
As for the "better error handling/general refactoring for better readability", which are the main issues? I've only written scripts for myself (and just few), so I only know about the "good practice" what the ZTM course told.
Sorry I wrote a lot but I'm looking forward to work on this project

[aubundy]

I agree, the PDF viewer could be a good start. Feel free to use any package you choose. I've only come across pdfreader

I think the Goose package has built in video parsing, so that might be a quicker task as well.

And no need to worry about the refactoring yet. It'll probably be good to eventually put the scraper into it's own folder with the different functions separated into their own files for better readability

[hiki270bis]

ok, then I'll try to build a separate module for the pdf scraping. Don't know how long it will take me but I'll keep you updated

[maxemileffort]

How do we feel about using OCR API's? They can be used to pull text out of images and PDFs.

Google's: Link
OCR Space: Link

The difference between the two is that Google's handles larger files but lets you have fewer calls, while OCR Space does the opposite (smaller files, but more calls). Google's is usually a higher quality as well.

Both have really good documentation. The free tiers are pretty generous for development, but nothing else really.

I'd like to tackle bot detection, but I have a question before I get started.

Locally, I can run a browser where I can mimic user behavior (searching for article titles through Google, having extra tabs open, changing some fingerprints that bot detectors look for, random smooth scrolls, etc.). This gets me through Amazon and Google bot detection, but I haven't tried any other sites.

Will a production environment have the capacity to implement that? Or should we stick to libraries that grab html?

[hiki270bis]

But using an API wouldn't mean we have to download the file and then pass it through the app? Just asking