Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I am trying to scrape https://public.rts.iebc.or.ke/enr/index.html#/Kenya_Elections_Senator/1
with HtmlAgilityPack. It is a dynamic site. The content is shown after the page loads completely.My code returns the HTML of the loading bar through this method while this method throws TargetInvocationException. I don't know how to make it wait until the page loads completely and then scrape it.
HtmlAgilityPack is just a library for .Net. You make a request and the library allows you to easily parse HTML response. If it does not contain the data you want to scrape then you need to do a different request. In the case of the page you mention, it uses Ajax for updating the page but Html is generated dynamically from a Json response. HtmlAgilityPack doesn't parse json but Html and this is a problem. If your code repeatedly makes requests for the same Url, you're going to get a new page every time with the original Html, which doesn't solve your problem neither.
If you're using WebBrowser you can wait using a timer.
With Selenium driver for .Net you need to set the timeout so that it will keep trying to find an element for a while before raising not found exception.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I wrote a small code in c# that reads files from the hard drive and outputs that info to a webpage using asp.net.
Information about those files is being changed in the codebehind.
I managed to create variables to read and display the info on the webpage, but they change only after a refresh. When I used setTimeout it would always read the same variable, leaving me thinking that the codebehind does not get re-executed.
How would it be possible to have them updated live without needing to refresh the entire page?
This is somewhat vague but if you want updates to the values on your page after loading, you'll need to use some javascript to grab new values and then more javascript to update values on your page.
There are some great frameworks out there. Unless you have a lot of front-end logic you need to perform through javascript, I'd keep it as simple as possible and throw jQuery in there for the AJAX calls (fyi, this is not the only solution, there are 10+ different ways to skin this same cat).
If you want a bit more efficiency, you could look into using SignalR - which wraps long polling or web sockets (depending on browser capability) on the front end and signals those subscribing pages on the backend only when changes have occurred to data.
Link: http://www.asp.net/signalr
There is no "right" answer to your question so the best I or anybody can do is guide you in a direction. Hopefully this answer helps you.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have been searching across the Web to find a solution to this:
Basically what I want to setup is a way to have an HTML page on a live production site to kept updated. This is because I have setup a SilverStripe Application as part of a graphics system (green chroma key) that runs through normal HTML.
AJAX load doesn't seem to be an option here since the load would be every 1 second or half a second to make sure the page is live.
I looked into web socket quite vaguely but the whole concept of integrating it with my project confused me entirely.
Anyone got any ideas of what could be done?
To do this you'll need a combination of:
AJAX, you'll need to never do a full page load after the initial Web 1.0 request.
Session history management, you'll need this in order to not break the browser navigation (e.g. back/forward buttons).
Web Sockets, you'll need this so when new content arrives on the server it is received by the client without first having to poll the server.
Web sockets will be the greatest challenge. I listed them in the order in which you should approach the project, in three phases.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Hi I am trying to develop a mobile app. The problem I am having is this, when the user fills in a form on the html page I want the data to get written to an unordered list on the same page. I want when the next user fills data for that dat to get written to the second list item in the list and so on
.
Thanks in advance for any help you may be able to give with this problem
If I understood correctly, you're trying to make an app that stores the data of each submitting of the forum and returns them to all users that come across the page?
For such tasks you'll need, at least:
A web server that runs PHP, Python or other language to your liking*;
A database to connect the server-side with (MySQL or PostgreSQL usually);
What you need to do:
Learn about HTTP POST and basic SQL input/output if you haven't;
Learn AJAX if you haven't;
Store the user input in the database and have a server script retrieve it on call;
Update the dynamic page bit (the unordered list) with AJAX (loading a XML file with the PHP script that writes the database info into it should do) every minute or so using a timeout function.
I'll post links to all I've mentioned in a minute.
*NodeJS is getting popular for being asynchronous (making it easier for dynamic pages to interact with the server) and allowing programmers to use the same language client- and server-side, but I have never tried it to tell you how it fares. Edit: says John NaN: NodeJS is not recommended. Again, I don't know it myself; that is why it's on a note and not upper in the post. Good luck!
*Bonus note: don't trust W3Schools most of the time. The AJAX tutorial I linked to, however, isn't that bad and it's easy to follow.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
Is it possible to do Ajax Deep Linking without hash sign (#) in the url. Knowing that my urls don't point to any controller/action on the server.
For example How does Stackoverflow or Soundcloud to manage the url?
Any help would be appreciated
You could use a query string, but the # is better since it doesn't give the appearance of different resources on the server.
Stackoverflow doesn't use Ajax (afaik) to load pages (if they did, they would use the technique below). I've no idea what Soundcloud does as I don't use them.
Twitter and Github use the history api (pushState and friends) but populate the initial page load server side (i.e. they don't load a default page and then overwrite it using JavaScript).
Doing it server side in this fashion is the preferred approach. It loads the requested content faster (and without the flash of homepage problem), continues to work if JavaScript fails, and is search engine friendly.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I am currently working on a small nodejs project that is responsible for returning search terms from a twitter feed. I have the search working however because I am new to this I am having problems displaying this on my webpage. The search information I want to display is being held by a callback from my function and I want to output this information. Keep in mind this information is ascertained after the webpage is loaded and I think that is why response.write(information); is not working for me. However if I do console.log(information); I do get the information I want but it's just not where I want it. Can someone help me out with outputting please.
I ended up solving this by using JSON.stringify(information);
You are right, it's very likely that once you have returned your webpage the response will be closed.
2 options:
1) you load your webpage first on 1 static route (using static middleware) and this web page then does an ajax call to a second route which returns your information.
2) you don't return your webpage until you have retrieved your information. I'm assuming your information will need to be displayed on this webpage which means you'll have to turn your webpage in a template and merge it with your information. You can use a Jade template or underscore template for example.