I am currently checking this file: https://api.steampowered.com/IEconService/GetTradeOffers/v1/?key=700D84417970EEAE593ACB8BE455B16E&format=json&get_sent_offers=1 for a change every 5 seconds but Is there any sort of way that I can get notified E.g. a post request when this website changes? I had a look at google push notifications but It seems that it only works for their own api's not external api's
I don't know what do you mean by getting notified.
If you want to get an email when the page changes you can use something like this Google Chrome extension
If you mean it in an programmatically way, then you will have to GET the JSON object from the URL every 5 sec and compare it to the previous version, see this question regarding the compare part.
Related
For example there is a table like in this link https://leetcode.com/contest/weekly-contest-309/ranking
How to access the database from where it is coming. Like let's say to get whole ranking table at a place
I tried reading HTML file but didn't get it
One extension scrapes the table only
How can we achieve this?
There is no simple answer here - it really depends how is it implemented on the server side. As rv.kvetch pointed you can get part of result from url:
https://leetcode.com/contest/api/ranking/weekly-contest-309/?pagination=1®ion=global
You can notice pagination query parameter here, indeed you can access second page, third page and so on. Sometimes there is some parameter like page_size implemented on server but it doesn't look like that case.
So to access full table you probably need to iterate over that pages and glue the results.
EDIT: How to get such url for some page?
Open your favorive browser, run web inspector (usually right click - inspect) and go to network tab, where you can find all requests sent during page rendering.
I am currently working on a project of finding empty classrooms in our school in real time. For that purpose, I need to extract substitution published on our school page (https://ssnovohradska.edupage.org/substitution/?), since there might be any additional changes.
But when I try to extract the html source code and parse it with bs4, it cannot find the divs(class: "section print-nobreak") that contain the substitution text. When I took a look at the page source code(Ctrl+U) I found that there is only a javascript that prints it all directly.
Is there any way to extract the html after the javascript output has been already rendered?
Thanks for help!
Parsing HTML is unfortunately necessary to solve your problem. But I will explain how to find ways to avoid that in your future projects (not based on this website).
You've correctly noticed that the text is created by JavaScript code running on the page. This could also indicate that the data is either loaded from another resource (XHR/fetch call getting a response from an API) or is stored as a JSON/JS inside of the website's code. (Or is generated from an algorithm, but this is unlikely to be the case in such websites.)
The website actually uses both methods (initial render gets data stored inside of the website's code, but when you switch dates on the calendar it makes AJAX requests). You can see this by searching for ReactDOM.render(React.createElement( in the code. They're providing a HTML string to the createElement call, so I would suggest looking into the AJAX way of doing things.
Now, to check where the resource is located, all you need to do is opening Developer Tools in your favorite browser (usually Control+Shift+I) and navigating to the Network tab. Now that your network tab is open, you need to cause the website to load external data, for example, by pressing a date on the "calendar bar".
Here you will notice many external requests, but we're actually looking only for XHR calls. Click on the XHR button next to the "Filter" text field. That should result in only one request being shown:
Unfortunately for us, the response only contains HTML. Also, API calls are protected - they require a PHP session ID and some sort of a token (__gsh) to not fail. So, going back to step 1 - seems like our only solution is to use regular expressions to find the text between "report_html":"<div class and </div></div></div> from the source code, if you're interested in today's date only. If you want to get contents for tomorrow or any other date - you will need to either fetch the page, save the cookies and find the token to supply to the request and then make that request, or use something like puppeteer or pyppeteer (since you've mentioned BS4) and load the webpage in that. If you aren't doing the data fetching that often, you should be fine overall.
First of all, Hi to all of you (I'm new here).
I'm having a look on how Google Analytics works as I'm gonna develop a similar tracking js to collect all the data I need for my websites and, as far as I can see, the ga.js script send all the data (maybe not all but a good part of it) with a get request with a 1x1 gif and all the parameters following.
Seen here: How does google analytics collect its data?
So, on the server side It seems the only way to "read" all these parameters is going to analyze server logging and then collect everything on my database?
Is this the best option to get users data?
I think, server logging could "switch file" every 2 hours so you can analyze that file of the past 2 hours and show "not that old" data to your graph!
Of course will never be "realtime" graph but a 2 hours delay could be acceptable, I think.
I think you can simply put a script (PHP for example) at the image path, then through the script return as a response the image, by doing this you can act in real time, since using a script you can get all the data that would be present in your server log.
If you want to try my solution I think a good point to start (in PHP) would be this to create the GIF image and then you can use data located in $_SERVER to start gathering data!
I am trying to write my first chrome extension. The workflow goes something like this -When the extension is installed and active if a user hovers over a specific product/ID displayed on the page, the extension retrieves related vendor data about the product with the ID.
This is how I thought about this:
Use jQuery attr to access the ID on mouse over.
Post this ID to a retrieve.php file with .post() method
The retrieve.php file retrieves the data from database
Display the data in a tool tip on the web page.
I have some queries for the above process:
I am able to get this working on a local XAMPP server but how will it work online as the chrome extension will not have access to server. What is the way around to retrieve data without using PHP?
I am able to get the logic working but am unable to place these in respective files - Will all my logic reside in background.js ?
Any suggestions on getting this started will be much appreciated.
You could build a very simple API on your server that responds with JSON to any request it receives after processing it. Like this:
{"firstVar":"foo","secondVar":"bar" }
Your chrome extension can then make an xmlhttp request to this server and and process the returned data.(You could also use JSONP and wrap the response in a callback function which will execute as soon as you have the reponse)
The JS extension will be able to deal with the JSON nicely as it can understand that format so you can then choose to display the data in whatever way you want.
Essentially, what you want is a server that can take an ID posted to it and return the corresponding date in a nice and readable format. And a chrome extension that can make an request to a server and then process the response. Build and test them separately (keep positing an ID to the server and see the response and for your JS side at first instead of making requests to your unfinished API just set a static response to begin with which will be the same as an expected response.
Adding a Google Map plugin to our homepage, which updates a single marker dynamically whenever there is a new product search on our site (which we read from our database). So, "has there been a new search via our site? If yes, reposition the marker based on the new search's coords".
Currently every "n" seconds (haven't settled on a seconds value yet) an Ajax call is made (using SetInterval) to determine if there has been a new search, and if there has it returns a small JSON response. The script run via the Ajax call is a PHP script, which queries the database for the last row in our searches table (order by desc limit 1).
So, my question is (not being a sysadmin), could this setup put an undesirable strain on our server? Should i incorporate a timeout session, or something, which turns off the Ajax call after 100 goes, or after 15 mins (i mean, who sits for 15 mins looking at markers dynamically generate on a Google map?!).
Our homepage only receives roughly 200 visits a day.
As you have you given the statistics that your website gets 200 visits per day and that your server is a spitting JSON that you have to extract and display it on the UI, It is a normal practice to have a set up like this one. You can rather ping the server data using AJAX in every 5 sec to get more precise data but it wont cause any performance issue at this level.
Please be sure that you dont have servers that are separated geographically else you have to use some other synchronization mechanism to track users location based on there search.
For AJAX JQuery implementation details please see the following page.
For project implementation as a tutorial please visit this tutorial.