When I browse, let's say, to example.com/page/name?source=illia I get to example.com/password page. This is an application set up.
In the dev tools, on the Network tab, I can see "resources" (not sure if I name it correctly) in the Name column.
So there are /name?source=illia and /password and all other items.
The question is how can I access /name?source=illia with js. Based on that I'd like to change the workflow.
document.referrer is an empty string
UPDATE:
Here is the screenshot from the devtools. Is it possible to get diagnostic?source=illia#example.com with javascript?
Here you can call javascript to get whole url:
window.location.href
Then you can extract what you need.
One easy way is to split:
var url = window.location.href;
alert(url.split('?')[1]);
Hope to be useful;
Related
I'm very new to JavaScript so I might did not get the basics right.
I'm trying to open a new page based on the currents page URL. But it seems that the function has no reference to the current URL.
It should be a simple Firefox Addon that looks at the URL, replaces a certain part and opens the "new" page in a new tap.
browser.browserAction.onClicked.addListener(function() {
var path = window.location.href
var newPath = path.replace('www', 'test')
var creating = browser.tabs.create({
"url": "${newPath}"
});
});
Being on www.google.com it should open me a new tap with test.google.com. Instead it opens me a Tab with the following URL=moz-extension://fde3def8-cf60-4536-b96b-1bf7ed91a8da/$%7BnewPath%7D.
Taking a look at the end of the URL I think it has no reference to the variable. When replacing the newPath variable in the last line with a static sample URL e.g. www.facebook.com it works fine.
Extension pages, such as an extension pop-up page, are separate pages. They run in a separate context that has no (or very little) connection to the regular currently open page.
Your path is the URL of the extension, which apparently happens to be "moz-extension://fde3def8-cf60-4536-b96b-1bf7ed91a8da/". You are trying to replace parts of that URL and open the result in a new tab.
Read up on architecture of extensions first:
Anatomy of an extension
Achitecture of extensions (for Chrome, but the concept is the
same)
To accomplish what you are trying to do, you will probably have to first query the active tab using tabs.query({active: true}) to get its URL.
Note on asynchronous execution:
Many extension related APIs (including tabs.query) are asynchronous (promise-based in Firefox). Promises might be a bit difficult to grasp for beginers.
You also confuse strings and template strings:
This is also incorrect: "url": "${newPath}". It should be simply "url": newPath. You seem to be confusing regular strings with template strings.
String interpolation in JS works only with backticks not with quotation marks as in: "url": `${newPath}`
I have an Html file containing the following code:
<script>
Object.defineProperty(document, "referrer", {get : function(){ return "myreferrer.com"; }});
//document.location="somelink.com";
</script>
From what I've read,maybe the thing I'm trying cannot be done,but I wanted to be sure.
I want to visit the site somelink.com but when my browser finishes the redirection to the location,the document.referrer value to be "myreferrer.com".
If I run the html with this format(document.location in comments)
the command in url --> javascript:alert(document.referrer) is the one I want.
But if I erase the comments and activate the document.location line,the above command will show up an empty document.referrer and not the one I want.
Can I achieve what I have in mind?
Some browser versions allowed you to customize the referer header using the approach of overriding the document.referer property in javascript, but that doesn't appear to be reliable. Even if some browsers still allow that, there's no guarantee it would work in future versions.
If you need a workaround, you could link to the desired referrer domain and serve up an intermediate page that performs the navigation to the final destination URL via an HTML form submission. That should preserve that intermediate page as the referrer.
Within the context of a browser extension however, you can alter the headers via onBeforeSendHeaders
I often use Chrome's debugger console for experimenting with javascript code fragments. When I got it right I usually want to copy the needed commands into my script, but here is where it gets messy. The is no filter options for commands and no way to call certain commands back (like with Ctrl-R in Bash) so you need to step through all the commands in the history and copy the commands you want one by one.
Instead, I think it should be possible to retrieve the command history from some file or Sqlite database. But I can't find it.
So my question is: Where is Chrome's debugger console command history stored?
I found an answer here: How to access firefox web console command history?
I had some trouble getting it working, but here is how I did.
Open the developer console (shift-ctrl-I). Then open that console in a new window if it isn't that already by using the menu in the upper right (the three dots).
When it is a separate window, press shift-ctrl-I again. Then paste something like this:
var hist = JSON.parse(localStorage.consoleHistory);
hist.forEach(function(command){
console.log(command);
})
Now, with all the commands in the console you can either copy them all to the clipboard or use the filter field above the console to do some filtering on them (you can use regex).
https://code.google.com/p/chromium/issues/detail?id=171386
Seems there was talk of such a feature which never came to fruition
You can collect some people and pressure the devs to put it in, or get it done.
Sounds really useful to me (:
For retrieving history :
https://developer.chrome.com/extensions/experimental_devtools_console#method-getMessages
How about developing an extension around that ?
Adding To marlars answer :
Actually i think its a little useful to not convert it into json. you can just keep it as a string so that you can use .indexof('yoursearchvalue')
And you don't have to always type that piece of code. you can just go to the Application tab -> Local Storage -> Devtools Entry and click on the consoleHistory
I have tried to scrape scoring/event time and also player name http://en.gooooal.com/soccer/analysis/8401/events_840182.html.However cannot work.
require(RCurl);
require(XML);
lnk = "http://en.gooooal.com/soccer/analysis/8401/events_840182.html";
doc = htmlTreeParse(lnk,useInternalNodes=TRUE);
x = unlist(xpathApply(doc, "//table/tr/td"));
normal html page doesn't show the details of the table contents.
the nodes only can get from
>>> open Chrome >>> click F12 >>> click Element
Can someone help? Thanks a lot.
If you reload the page while Chrome developer tools are active, you can see that real data is fetched via XHR from http://en.gooooal.com/soccer/analysis/8401/goal_840182.js?GmFEjC8MND. This URL contains event id 840182 which you can scrape from the page. The part after ? seems to be just a way to circumvent browser caching. 8401, again, seems to be just first digits of the id.
So, you can load the original page, construct the second URL, and get real data from there.
Anyway... In most cases it's a morally questionalble practice to scrape data from web sites. I hope you know what you're doing :)
It sounds as if the content was inserted asynchronously using javascript, so using Curl won't help you there.
You'll need a headless browser which can actually parse and execute javascript (If you know ruby you could start looking for the cucumber-selenium-chromedriver combo), or maybe just use your browser with greasemonkey/tampermonkey to actually mimic a real user browsing the score scraping.
The contents are probably generated (by Javascript, like from an ajax call) after loading the (HTML) page. You can check that by loading the page in Chrome after disabling Javascript.
I don't think you can instruct RCurl to execute Javascript...
I have a webpage that contains some information that I am interested in. However, those information are generated by Javascript.
If you do something similar like below:
browser = webdriver.Chrome()
browser.set_window_size(1000, 1000)
browser.get('https://www.xxx.com') # cannot make the web public, sorry
print browser.page_source
It only print out a few javascript functions and some headers which doesn't contain that information that I want - Description of Suppliers, etc... So, when I try to collect those information using Selenium, the browser.find_element_by_class_name would not find the element I want successfully either.
I tried the code below assuming it would has the same effect as typing document in the javascript console, but obviously not.
result = browser.execute_script("document")
print result
and it returns NULL...
However, if I open up the page in Chrome, right click the element and inspect element. I could see the populated source code. See the attached picture.
Also, I was inspired by this commend that helps a lot.
I could open up the javascript console in Chrome, and if I type in
document
I could see the complete html sitting there, which is exactly what I want. I am wondering is there a way to store the js populated source code using selenium?
I've read some posts saying that it requires some security work to store the populated document to client's side.
Hope I have made myself clear and appreciates any suggestion or correction.
(Note, I have zero experience with JS so detailed explaination would be gratefully appreciated!)