We are using a third part web app which does not allow or have an API yet, this third party app is basically a membership registration website and each member belongs to a specific category.
I need to use these category in our internal system and so far I have been manually adding the category in a drop down menu of a form as soon as a new category in created in third party app.
Since there is no access to an api so I am wondering if it is possible to crawl the third party app where the dropdown menu is and copy the entire dropdown menu over to our internal site.
I wish i can show you the efforts I have made so far but I am stuck on how to even begin this. I did however search online but all I could find is how to copy a dropdown on a same page.
Any push to the right direction will really be helpfull, the technologies I am working with is PHP and JS
I don't think CORS is going to help you here, as it's function is to provide a legal/safe way of sharing web resources across different domains (i.e. images/css files/web fonts), not data.
If there is no API for the data you need, you are almost certainly limited to scraping the data out of the web page.
You can do this by first issuing a request for the page to obtain the html, then searching/parsing the html to find the drop-down menu, then finally parsing the menu items to obtain a list that you can use for your own drop-down.
So, some pointers:
Obtain page html - See PHP: how can I load the content of a web page into a variable?
Parse html - See PHP Parse HTML code
Of course how easy this ends up being depends on many factors, e.g.
Can you just request the page containing the drop-down, or does the
web app need authentication? You may need to refine the curl request
as appropriate.
Can you easily identify the html drop-down, e.g.
using a unique id tag. If so, you could use
DOMDocument::getElementById, otherwise you may need more complex
logic to parse the page html and find the menu.
Either way, it should be possible to achieve - just remember that the third-party app is not under your control, and as such may be subject to changes that break your program.
LATEST UPDATE:
Added in retrieval of value, and we hide parse warnings using internal_errors.
Here's a simple PHP script that will print out the text and value of each of the drop-down options:
<?php
libxml_use_internal_errors(true);
$html = file_get_contents('http://example.com/');
$domdoc = new DomDocument;
$domdoc->loadHTML($html);
libxml_clear_errors();
$menu = $domdoc->getElementById('tid');
$options = $menu->childNodes;
foreach ($options as $option) {
echo($option->nodeValue)." - ".$option->getAttribute('value')."<br>";
}
?>
Related
For example there is a table like in this link https://leetcode.com/contest/weekly-contest-309/ranking
How to access the database from where it is coming. Like let's say to get whole ranking table at a place
I tried reading HTML file but didn't get it
One extension scrapes the table only
How can we achieve this?
There is no simple answer here - it really depends how is it implemented on the server side. As rv.kvetch pointed you can get part of result from url:
https://leetcode.com/contest/api/ranking/weekly-contest-309/?pagination=1®ion=global
You can notice pagination query parameter here, indeed you can access second page, third page and so on. Sometimes there is some parameter like page_size implemented on server but it doesn't look like that case.
So to access full table you probably need to iterate over that pages and glue the results.
EDIT: How to get such url for some page?
Open your favorive browser, run web inspector (usually right click - inspect) and go to network tab, where you can find all requests sent during page rendering.
I want to know How to have pages like YouTube's watch page, http://www.youtube.com/watch?v=abc123 ?
I need it to do the following:
when just going to the main file watch.php, I want it to :show certain code for when they go to that page,
when they go to example /watch?v=abc123, I want it to : show different code only for that link.
I also don't want the main watch.php page to be all cluttered up with code as I'll be at nearly 50+ times using these ?v= I heard you can use Databases but I don't know.
Just whichever ones the best and easiest to make also please try and get it not to be clustered up and easy for me to edit. Thank you
Please Edit this if needed. My website and page i'll be using
This is only available via php. Using $_GET method.
Basically, the link is only www.mysite.com/watch.php. But if you use watch.php?v=123 then the $_GET['v'] will have the value of 123.
For more info: http://www.php.net/manual/en/reserved.variables.get.php
EDIT: basically your step would be like this
Set up a database. This depends on where you host your site. But most host service allow you to create and manage database.
Create a table with values you wanted to store.
Get the "v" value through $_GET
Run queries to connect to the database, and get the stored value. Ue echo to print it out on HTML
I'm writing a program in Python that needs to use a site's advanced search options. Specifically, the search page is the NVC advanced search page . I know the names of the projects and versions I need to search for, so ideally the program would select the project names and versions numbers from the dropdown lists, then return the results page(s).
I'm totally unfamiliar with HTML and Javascript, and I'm fairly new to Python, so I don't know if there's a way to 'click' these dropdown menus via Python, then return the results. The fact that the Javascript makes an Ajax call further complicates the situation, since I can't just load the page's source and parse out the list of project names and version.
Can anyone with some Python/Javascript/Ajax experience send me in the right direction?
An example use of this program would be that I start out with the project "glibc' and its version number '2.3.6' The program would make sure that this combination is listed at all (which isn't guaranteed), then return the results page (which has about 13 results).
The Mechanize Python library is perfect for form automation. There is an example of how to edit and submit forms on the examples page.
If a human user is using that search page, they click on one of the product links, which then load the list of products from another page, e.g.:
http://web.nvd.nist.gov/view/vuln/cpe/cpe-chooser?index=0&component=Vendor
This page is unfortunately not using JSON, so they have some custom javascript parsing for the response. The data from this response is then displayed as a drop-down for the user. When the user selects a product, the browser selects the correct value, so that when the form is submitted, it will be part of the query. e.g.:
http://web.nvd.nist.gov/view/vuln/search-results?adv_search=true&cves=on&cpe_vendor=cpe%3A%2F%3Aa-a-s_application_access_server
In this, cpe_vendor=cpe%3A%2F%3Aa-a-s_application_access_server is the important part. The part before the = sign is the field name, the part after is the selected value (which originally came from the ajax request). The funny %3A bits are URL-encoding.
So you don't actually need to interact with the page, since you know the names of the vendors and products for which you want to search; you just need to look up the field name (cpe_vendor for vendors) and the value for the specific products/vendors (cpe:/:a-a-s_application_access_server for my example above), then do a request to the normal search URL.
The advanced search options page sends the options via GET to the results page, giving you the URL (linebreaks mine to make it clearer):
http://web.nvd.nist.gov/view/vuln/search-results?
adv_search=true&
cves=on&
cve_id=&
query=&
cwe_id=&
cpe_vendor=cpe%3A%2F%3Aian_bezanson&
cpe_product=cpe%3A%2Fa%3Aian_bezanson%3Adropbox&
cpe_version=cpe%3A%2Fa%3Aian_bezanson%3Adropbox%3A0.0.3_beta&
pub_date_start_month=0&
pub_date_start_year=2005&
pub_date_end_month=2&
pub_date_end_year=2009&
mod_date_start_month=2&
mod_date_start_year=2007&
mod_date_end_month=9&
mod_date_end_year=2009&
cvss_sev_base=&cvss_av=&
cvss_ac=&
cvss_au=&
cvss_c=&
cvss_i=&
cvss_a=
It would then take a bit of sleuthing to figure out what bit of the url is what information from the form but that should let you then just scrape the results page.
How would I search an SQL database when a user makes a selection in a dropdown list. I know how to change/add text to areas using some javascript but not with a SQL search as well.
Ideally I don't want to be changing pages in this process as I'm thinking they won't be sticking on a single option in the drop down for long.
Cheers in advance. =)
You have to add an onchange event listener to the dropdown box. When the user selects an option, an AJAX (XmlHttpRequest) should be send to the server. A serverside script, a PHP page for example, should parse the parameters (after checking).
When these parameters have been checked, they should be escaped for this reason. Perform a search query, parse the results, and send the output back to the client (the user, browser).
Useful links:
http://www.php.net/manual/en/ref.mysql.php
https://developer.mozilla.org/en/xmlhttprequest
you should look into implementing Ajax for this.
Here's a simple example and tutorial utilizing a drop down list that fetches information from a database on selection, without changing pages :)
http://www.w3schools.com/php/php_ajax_database.asp
You should also look into using jquery for your Ajax requests as stated in the answer below. (Do more with less code).
See also:
http://15daysofjquery.com/quick-and-dirty-ajax/14/ for a simple jquery Ajax tutorial.
Before you render the page, make sure you do a grab the info you want presented from the database.
select distinct NAME, ID
from tableName
You can use the information obtained here to generate the html for a drop down box. Include something to the effect
<select onchange="doAction(this.value);">
<option value="userID">userName</option>
<option value="userID">userName</option>
<option value="userID">userName</option>
...
</select>
Then perform an AJAX request back to your site. Get your information from your second query to the database, and return with the information you need. When you perform your second query (and in general) you have to protect yourself against injection attack vectors. The best start for this is to make sure you have encoded(HTML, SQL) correctly.
Without more information I can only really give you the below as a guide. It uses the jQuery load function to put the contents of one page into an element. Note that .load cannot be used across domains.
$("#searchButton").live("click", function() {
$("#results").load("/SearchResults.php?s=" + $("#searchBox").val());
});
Also note that you should use something to filter out HTML tags to prevent SQL injection, as well as taking other measures.
I have a simple (yet somehow convoluted) issue. Basically I'm adding items to make my web app more "desktop-like". For instance, right now I'm trying to get a page to dynamically load info into a DIV based on previously selected items. I'm currently using a cookie to handle saving the data, but I can't for the life of me get my brain to work this problem out.
I have a scenario with the following relationships:
SITE has_many BUILDINGS
BUILDING has_many METERS
METER
All entities can have associated charts. So, in an effort to make it generic, I set up a "has_many" relationship for each to CHARTS and abstracted it like so.
SITE has_many CHARTS, as chartable
BUILDING has_many CHARTS, as chartable
METER has_many CHARTS, as chartable
Once the user selects an item from the menu on the left, I then use a method to determine what item needs charts found and I display the particular item's charts. That all works fine.
My issue now is working with cookies in order to either save data to independent keys (or perhaps Marshal objects) in order to dynamically reload the previously selected item's data whenever the page reloads. The ajax call requires several values in order for the "update" action to find the correct item and display it. I'm having trouble with whether to use Javascript directly, try to trigger an action, or use some kind of combination.
As I said, I'm sure the issue is rather simple or straightforward, but I'm just not seeing it. If this description is a bit vague, I do apologize. Feel free to ask for more info.
Best
When the user selects an item from the menu, save all the necessary information to re-select that item to a cookie. Bind a Javascript method to the page load and check the value of that cookie. If the information is there indicating that an item should be preselected, just call the same Javascript method that is called when the user selects a new item from the menu. If you're using JQuery, for example, you might do something like this to bind to the page load:
$(document).ready(function() { /* check cookie and do stuff */ }
Another thing you could do is pre-render that stuff in your RoR code if that cookie exists so you don't immediately execute an AJAX call on page load (since that is sometimes considered bad form due to the page load performance hit).
This is too big for storing in cookies, you should either:
Store an id cookie client-side and store the data on the server-side which can be accessed with a corresponding id cookie and valid authentication credentials.
Use HTML5 client-side storage such as localStorage or a local database.