I'm looking for a way to automatically extract information from a web page, more specifically an online game (https://www.virtualregatta.com/fr/offshore-jeu/).
In the game, I want to extract/copy the position of the boat. With Mozilla and its debug tools, I used the network debugger and I saw an HTML POST request containing what I want.
It seems that we receive as a response a json containing a structure with latitude/longitude.
This is perfect to me, but I want a more user friendly way to get it and I would need advices. Problem is that I'm really a beginner in web development haha.
Is it possible to do this using a script ? (But I suppose it will be complicated to first log into the game)
Is it possible to create a basic Mozilla plugin which would be able to catch the request/response and copy the position to clipboard for me ?
anything else ?
EDIT:
I've tried using a Mozilla plugin, and I achieved to add a listener on POST request. I see the request to get the boat information but I can't find a way to get the json response in JS.
function logURL(responseDetails) {
console.log(responseDetails);
}
browser.webRequest.onResponseStarted.addListener(
logURL,
{urls: ["*://*.virtualregatta.com/getboatinfos"]}
);
In Chrome I use Broomo for this purposes. It helps you to add scripts in web pages, you can console.log the POST you found, and of course you can create functions and Use the webpage Backend.
In firefox I found this one js-injector. But I didn't use it before.
Update:
Now there are a new extension for both browsers:
Chrome: ABC JS-CSS Injector
Firefox: ABC JS-CSS Injector
I have a need to create a wallet (generate account address and private key) in the browser without connecting to a node. If I understand correctly, in order to use web3.js we need to set a provider (Metamask or localnode) to use web3.personal.newAccount("SEED", (response) => { console.log(response)})
I found ethereumjs-wallet perfect for my use case but it exists only as a node module. Any way to use it as a browser module or alternatives to that?
Another key management tool from EthereumJS community is the Keythereum
Keythereum is a JavaScript tool to generate, import and export Ethereum keys. This provides a simple way to use the same account locally and in web wallets. It can be used for verifiable cold storage wallets.
A minified, browserified file dist/keythereum.min.js is provided by them for use in the browser. Including this file simply attaches the keythereum object to window:
<script src="dist/keythereum.min.js" type="text/javascript"></script>
You can try with this one.
I am following a Django Tutorial by Marina Mele, which is pretty good but a bit outdated since it was last updated in 2016, I believe. I am now trying the Selenium testing and ran into the problem that I can send my e-mail address via Selenium but not the password. My code is
self.get_element_by_id("identifierId").send_keys(credentials["Email"])
self.get_button_by_id("identifierNext").click()
self.get_element_by_tag('input').send_keys(credentials["Passwd"])
self.get_button_by_id("passwordNext").click()
with these functions being defined as:
def get_element_by_id(self, element_id):
return self.browser.wait.until(EC.presence_of_element_located(
(By.ID, element_id)))
def get_element_by_tag(self, element_tag):
return self.browser.wait.until(EC.presence_of_element_located(
(By.TAG_NAME, element_tag)))
def get_button_by_id(self, element_id):
return self.browser.wait.until(EC.element_to_be_clickable(
(By.ID, element_id)))
Most advices that I read to this issue circled around waiting until the element appears. However, this is covered through these functions. And I am using by_tag since the current version of Google Authentication is using an input for the password field that has not an ID but is a div/div/div child of the div with the "passwordIdentifier"-id. I have also tried using Xpath but it seems that this does not make a difference.
Also, it seems like Selenium is capable of finding the elements...at least when I check with print commands. So, locating the element seems not to be the problem. However, Selenium fails to send the keys from what I can see when I look at what happens in the Firefox browser, while Selenium is testing. What could be the issue? Why is Selenium struggling to send the password keys to the Authentication form?
Thanks to everyone in advance!
When you search an input on google registration page, you will find 8 WebElements. I think it is the origin of your problem.
I would use another localizer such as an xpath = //input[#name='password'] or a By on the name instead of the tag name, as implemented below:
def get_element_by_name(self, element_tag):
return self.browser.wait.until(EC.presence_of_element_located(
(By.NAME, element_tag)))
and:
self.get_element_by_id("identifierId").send_keys(credentials["Email"])
self.get_button_by_id("identifierNext").click()
self.get_element_by_name('password').send_keys(credentials["Passwd"])
self.get_button_by_id("passwordNext").click()
Is it possible to show the documents from my drive on a webpage? I want the user to be able to click the document and download it, directly from my drive. How would I go about doing this? Thank you for your suggestions.
The fastest and easiest solution is to embed the folder using an iframe (no javascript needed). Obviously this is also the least flexible solution, although you can use CSS to change the layout of the iframe contents (see below).
Google Drive won't allow embedding of the url you would normally use. It has its X-Frame-Options header set to "SAMEORIGIN", preventing use in an iframe. So you have to use the following link, which will allow embedding:https://drive.google.com/embeddedfolderview?id=DOCUMENT_ID#VIEW_TYPE
DOCUMENT_ID is the id that is mentioned in the normal share link (which looks like https://drive.google.com/folderview?id=DOCUMENT_ID), so you can just copy that from there.
VIEW_TYPE should be either 'grid' or 'list', depending on your preference.
And if you need to change the style of the iframe content, take a look at this solution.
For HTML/JavaScript solution, look at the following links:
https://developers.google.com/drive/quickstart-js
https://www.youtube.com/watch?v=09geUJg11iA
https://developers.google.com/drive/web/auth/web-client
Here's the simplest way using JavaScript, most of the complexity is in
your WebApp authorization. The example below reads files IDs, names and description in a folder you specify.
- go to: https://cloud.google.com/console/project
and create a new project "xyz"
- Select "APIs & auth", disable the ones you don't need, enable "Drive API"
- Select "Credentials",
push "CREATE NEW CLIENT ID" button
x Web Application
Authorized Javascript origins: "https://googledrive.com/"
Authorized redirect URI: "https://googledrive.com/oauth2callback"
it will result in:
Client ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
Email address: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#developer.gserviceaccount.com
Client secret: xxxxxxxxxxxxxxxxxxxx
Redirect URIs: https://googledrive.com/oauth2callback
Javascript Origins: https://googledrive.com/
- in the code below, replace
CLIENT_ID with xxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
FOLDER_ID with the ID you see in the folder address line,
https://drive.google.com/?tab=mo&authuser=0#folders/xxxxxxxxxxxxxxxxxxx
- run it, authorize
I don't know if you read JS, the code can be followed from bottom up, I made is as simple as possible.
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
<script>
var FOLDER_ID = '.xxxxxxxxxxxxxxxxxx'; // the folder files reside in
var CLIENT_ID = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com';
var SCOPE = //'https://www.googleapis.com/auth/drive';
[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/drive.file', // for description,
];
function rsvpCB(resp) {
var picAlbumLst = '<ul>\n';
for (i=0; i<resp.items.length; i++)
picAlbumLst += (
' <li>'+resp.items[i].id+', '+resp.items[i].title+', '+resp.items[i].description+'</li>\n');
picAlbumLst += "</ul>\n";
$('#container').append(picAlbumLst);
}
function rqstCB() { //test # https://developers.google.com/drive/v2/reference/files/list
var rv = gapi.client.drive.files.list({
'q': '"'+FOLDER_ID+'" in parents and trashed = false',
'fields' : 'items(id,title,description)' //'items(id,title,description,indexableText)'
}).execute(rsvpCB);
}
// authorization server reply
function onAuthResult(authResult) {
var authButton = document.getElementById('authorizeButton');
authButton.style.display = 'none';
if (authResult && !authResult.error) { // access token successfully retrieved
gapi.client.load('drive', 'v2', rqstCB);
} else { // no access token retrieved, force the authorization flow.
authButton.style.display = 'block';
authButton.onclick = function() {
checkAuth(false);
}
}
}
// check if the current user has authorized the application.
function checkAuth(bNow) {
gapi.auth.authorize({'client_id':CLIENT_ID, 'scope':SCOPE, 'immediate':bNow}, onAuthResult);
}
// called when the client library is loaded, look below
function onLoadCB() {
checkAuth(true);
}
</script>
<script src="https://apis.google.com/js/client.js?onload=onLoadCB"></script>
<body style="background-color: transparent;">
<input type="button" id="authorizeButton" style="display: none" value="Authorize" />
<div id="container">
</div>
</body>
This should be done with Google API. You can search google drive php api list files on google. And also I found this and this on SO.
Here are some main points:
Do you want anyone with the URL to be able to see your document? You can share a document as public to anyone on the internet. Plus you can set read access to specific folders. Just right click a Google Doc file, and choose 'Share' from the short cut menu.
I'm assuming you want people to download your docs, even when you are not signed in. This is called 'Offline Access', and is one of many terms you'll need to figure out in order to do all of this with a program.
If you only want to give read access to the user, using JavaScript, jQuery, etc on the front end is a viable option. You can also do this in PHP, it's just a matter of personal preference.
To do all of this in code, you need to grant authorization to read your files. The oAuth2 process has multiple steps, and it's good to understand the basic flow. Setting up the code and the webpages to initially grant authorization, then retrieve and store the tokens can get confusing.
Your Google Project has a setting for where the origin of the authorization request is coming from. That is your website. But if you want to develop and test locally, you can set the Javascript Origins to http://localhost
How much time do you have, and how much programming experience? Would it be easier to give the user a few lines of instruction to "Manually" download your file, rather than program the authorization check?
Putting the document into your webpage is the easy part.
In order to embed a Google doc in your website, go to your Google Drive, open a document and choose File then Publish to Web, and you will be given an HTML iFrame Tag that can be embedded into you web page. You can change the height and width of the iFrame to match the document size. iFrame Instructions W3Schools
Downloading your document can be done very easily from the online version of a shared document just by choosing FILE and then DOWNLOAD AS from the menu.
To get up and running fast, just give the user a couple lines of instructions on how to download "Manually", then see if you can program the code.
Provide a link to your shared document instead of programming the button, and then work on the code.
Search Git Hub for Google Drive, you might find something there.
Some of the official Google code examples are way more complicated than you need, and will take a long time to figure out. The code examples in the documentation pages are simpler, but are almost never complete functioning code examples. You'll need to put lots of pieces of the puzzle together to make it work.
Basically I have created one simple search engine in php.
I want to create a firefox addon, after installing that it will automatically add my search engine URL to the search box. And this should be default one as now "Google" is the default. So when user type anything, the suggestions should be coming from my webservice.
I'm using FF 21
Any working code or documentation will be very helpful.
Simplest method to create a search plugin is to use it as OpenSearch plugin.
https://developer.mozilla.org/en/docs/Creating_OpenSearch_plugins_for_Firefox
And OpenSearch plugin supports 'search suggestions' using json. But, It'll not be a default one without manually changing it.
Next method is to create a extension using XUL & JS to do whatever you want.
https://blog.mozilla.org/addons/2009/01/28/how-to-develop-a-firefox-extension
http://www.codeproject.com/Articles/31736/Introduction-to-Building-FireFox-Add-ons-Extension
https://developer.mozilla.org/en-US/docs/Firefox_addons_developer_guide/Let%27s_build_a_Firefox_extension