Electron Js Web Scraping



The good

Last week I felt pretty good. I got the Electronjs app up and running with the Electron Dead Link Checker v0.0.1. This week is a stark contrast as I feel like very little of good came out of the code this week.

Github

The coolest part is that I found and am starting to use parts of github that I have never used before. They have a trello board like thing that you can easily use to track tasks. I really like having it baked right into my repo where I am doing all my work.

From issues, you can assign a project. Then the board watches that issue and updates its position in the board based on what happens with the issue. Let’s say you resolve an issue (which I discovered can be done from the commit with just a fixes #<issuenumber> in the commit message; see this for more). That issue will automatically move from where it was to done.

Use the Electron Framework to build compelling cross-platform desktop applications with the latest web dev technologies What you’ll learn. Electron for Desktop Apps: The Complete Developer’s Guide Course Site. Learn how to make native-feeling applications using web development technologies; Master the intricacies of development with Electron JS. Javascript Web Scraping Guy. I got the Electronjs app up and running with the Electron Dead Link Checker v0.0.1. This week is a stark contrast as I feel like very.

You can manage the automation like the above picture shows and decide what triggers what events for the column. Pretty cool.

Electron is a framework for creating native Windows/Mac/Linux applications with web technologies (Javascript, HTML, CSS). It includes the browser Chromium, fully configurable. Is there a better way to code a portable application with a graphical user interface to scrape a given site? Script website interactions. A runtime, just like Node. Nightmare.js A high-level browser library https: //gi thub.com /segmentio.

Display progress

Displaying an elapsed time was a goal of mine with v0.0.2 and while I didn’t finish all I wanted for v0.0.2, I did finish this. An elapsed time serves as both a notice to the user that work is happening and a cool way to track how long it’s taking to complete the task.

Originally I was returning the elapsed time from the dead-link-checker module but then I realized I could just start a timer within my electron app and complete it when the request completed.

Install electron js

Pretty simple. I think it looked a bit better incrementing it every 10 ms and then use the simple math in my html of dividing by <h4 *ngIf='elapsedTime'>Elapsed time - {{elapsedTime / 100}} seconds.</h4>.

Don’t display 999 for timeouts

This is good and bad. Previously I was displaying 999 if I got something wrong without a timeout. Generally the error would be something like RequestError {name: 'RequestError', message: 'Error: read ECONNRESET', cause: Error: read ECONNRESET which didn’t have a status code. I have been treating them like timeouts and assiging a 999 status code.

For this…I cheated. I just looked at if I had a 999 and replaced it with an actual timeout status code, 408. It’s kind of misleading. I’ll talk more about this kind of stuff in the bad section.

Electron app is handling long tasks like a champ

I always worry with any kind of script that the longer it goes, the more likely something will break. I had proven that the script alone could handle long tasks, which one going up to somewhere in the 30 some hours.

I’ve improved the script to not check things like #comment so that makes it check a lot less links but it’s still hitting upwards of 10k+ links over 10-15 minutes. I’m really pleased with this.

The bad

Cancel button

Most of my focus was on this. I’m trying to do it with webworkers because this is an area that I would like to get better at. I feel like I spun my tires here. I barely made any progress before I was stopped by an erro that I still haven’t figured out.

If I call deadLinkChecker directly from my app, no problem. If I call it from a webworker, death:

Electron Js Web Scraping Github

I went through every piece deadLinkChecker commenting out parts until I could isolate the problem. It seems there is more than one problem but for sure requestPromise is a problem. It would be pretty tough to use this script without requestPromise.

This is something that I’m going to keep working on. There must be some way to do this kind of thing. Most of the people that have this problem have it when it’s related to Angular but for me the script works great if it’s called directly from the Angular side. It only crashes like this in the web worker.

Not very reliable yet

This should probably be the highest priority. Having a cancel button is important but a user could always just close the app and then reopen it. Right now it seems that bad links are returning, such as my faked out 408, that are actually fine. Having an occasional of these I think is probably okay. I think a low threshold like 10% may be acceptable and I’m way over that. As such it ruins confidence in the product and kind of makes it useless.

Electron Js Alternatives

I’m not sure why sometimes I will get a 503 from Amazon or a 403 from other random websites, in addition to the weird timeouts I get. It’s possible I’m getting blocked but I really think I can probably work that out in all situations except recaptchas and even a recaptcha should return a 200 status code.

So…that is what I will continue to work on.

Electron Js Web Scraping

Electron Js Web Scraping Example

To get started with Electron, check out the resources below. Learn how to wrap your web app with Electron, access all the APIs, and generate installers.

Explore the Electron APIs

Make A Browser Electron Js

The Electron API Demos app interactively demonstrates the most important features of the Electron API. See what's possible with Electron with sample code and helpful tips for building your app.

Try more powerful experiments with Electron Fiddle

Electron Fiddle lets you create and play with small Electron experiments. It greets you with a quick-start template after opening – change a few things, choose the version of Electron you want to run it with, and play around. Then, save your Fiddle either as a GitHub Gist or to a local folder. Once pushed to GitHub, anyone can quickly try your Fiddle out by just entering it in the address bar.

Spin up the Quick Start app to see Electron in action:

A minimal Electron app with helpful notations.

Electron Js Web Scraping Tutorial

Or dive deeper and read the documentation.

Electron Js Web Scraping Api

Need Help?

Electron Js Web Scraping Examples

Ask questions in the Discuss forum. Follow @electronjs on Twitter for important announcements. Need to privately reach out? Email info@electronjs.org.