I'm referencing this url: https://tracker.icon.foundation/block/29562412
If you scroll down to "Transactions", it shows 2 transactions with separate links, that's essentially what I'm trying to grab. If I try a simple pd.read_csv(url) command, it clearly omits the data I'm looking for, so I thought it might be JavaScript based and tried the following code instead:
from requests_html import HTMLSession
session = HTMLSession()
r = session.get('https://tracker.icon.foundation/block/29562412')
r.html.links
r.html.absolute_links
and I get the result "set()"
even though I was expecting the following:
['https://tracker.icon.foundation/transaction/0x9e5927c83efaa654008667d15b0a223f806c25d4c31688c5fdf34936a075d632', 'https://tracker.icon.foundation/transaction/0xd64f88fe865e756ac805ca87129bc287e450bb156af4a256fa54426b0e0e6a3e']
Is JavaScript even the right approach? I tried BeautifulSoup instead and found no cigar on that end as well.
You're right. This page is populated asynchronously using JavaScript, so BeautifulSoup and similar tools won't be able to see the specific content you're trying to scrape.
However, if you log your browser's network traffic, you can see some (XHR) HTTP GET requests being made to a REST API, which serves its results in JSON. This JSON happens to contain the information you're looking for. It actually makes several such requests to various API endpoints, but the one we're interested in is called txList (short for "transaction list" I'm guessing):
def main():
import requests
url = "https://tracker.icon.foundation/v3/block/txList"
params = {
"height": "29562412",
"page": "1",
"count": "10"
}
response = requests.get(url, params=params)
response.raise_for_status()
base_url = "https://tracker.icon.foundation/transaction/"
for transaction in response.json()["data"]:
print(base_url + transaction["txHash"])
return 0
if __name__ == "__main__":
import sys
sys.exit(main())
Output:
https://tracker.icon.foundation/transaction/0x9e5927c83efaa654008667d15b0a223f806c25d4c31688c5fdf34936a075d632
https://tracker.icon.foundation/transaction/0xd64f88fe865e756ac805ca87129bc287e450bb156af4a256fa54426b0e0e6a3e
>>>
I was using google finance api to get the stock quotes and display the contents on my site. All of a sudden from 6/september/2017 this stopped working. The url i used to get the stock quotes is https://finance.google.com/finance/info?client=ig&q=SYMBOL&callback=?.
Previously, i was using yahoo finance api and it was inconsistent. So, i switched over to google finance api.
Could you please help me on this?
Thanks,
Ram
This url works. I think just the url changed from www.google.com to finance.google.com
https://finance.google.com/finance/getprices?q=ACC&x=NSE&p=15&i=300&f=d,c,o,h,l,v
In the end i started using yahoo finance. The data is not live, there is a 20 minutes delay. I thought it will be helpful to people who are facing issues like me.
The yahoo api url is https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%3D%22MSFT%22&env=store://datatables.org/alltableswithkeys
This will return the stock data in xml format. You can parse the xml to get your desired fields.
Thanks,
Ram
We had a same issue & we found below alternative API provided by Microsoft Bing API for Stock Markets. Below API returns the stock data in JSON format.
https://finance.services.appex.bing.com/Market.svc/ChartAndQuotes?symbols=139.1.500209.BOM&chartType=1d&isETF=false&iseod=False&lang=en-IN&isCS=false&isVol=true
Thanks, Shyamal
I was dying to look for thread like this yesterday when I faced the issue!
Like Salketer said, Google Finance API was officially "closed" in 2012. However, for some reason it was still working until September 5, 2017. I built a program to manage my portfolio that uses GF API to get live quotes for US stocks. It stopped working on Sep 6, 2017, so I am assuming that engineers behind "secretly providing" API now "actually" stopped the service.
I found an alternative https://www.alphavantage.co/documentation/ , and this seems like the best alternative for free live US equity quotes. They just require your email, nothing else. It's a bit slow because it doesn't have multi-symbol query yet, but beggars can't be choosers.
I had to switch to Google finance after using Yahoo finance for a long time after Verizon bought yahoo this May and ended the free API service. I went back and re-researched this issue and someone created a new Yahoo finance API call that works with the new yahoo API. https://stackoverflow.com/a/44092983/8316350
The python source and installer can be found here: https://github.com/c0redumb/yahoo_quote_download
The arguments are (ticker, start_date, and end_date) where dates are yyyymmdd format and returns a list of unicode strings. The following test will download a couple weeks worth of data and then extract only the adjusted close price to return a list called adj_close:
from yahoo_quote_download import yqd
import string
quote = yqd.load_yahoo_quote('AAPL', '20170515', '20170530')
print(quote[0]) # print the column headers
print(quote[1]) # print a couple rows of data
print(quote[2]) # just to make sure it looks right
quote.pop() # get rid of blank string at end of data
quote = [row.encode("utf-8") for row in quote] # convert to byte data
quote = [string.split(row, ',') for row in quote] # split the string to create a list of lists
adj_close = [row[5] for row in quote] # grab only the 'adj close' data and put into a new list
print(adj_close)
Returns:
Date,Open,High,Low,Close,Adj Close,Volume
2017-05-15,156.009995,156.649994,155.050003,155.699997,155.090958,26009700
2017-05-16,155.940002,156.059998,154.720001,155.470001,154.861862,20048500
['Adj Close', '155.090958', '154.861862', '149.662277', '151.943314', '152.461288', '153.387650', '153.198395', '152.740189', '153.268112', '153.009140', '153.068893']
I was manually reading from Google Finance page for each stock before I got the ?info link. As this is not working anymore, I am going back to the webpage.
Here is my python snippet:
def get_market_price(symbol):
print "Getting market price: " + symbol
base_url = 'http://finance.google.com/finance?q='
retries = 2
while True:
try:
response = urllib2.urlopen(base_url + symbol)
html = response.read()
except Exception, msg:
if retries > 0:
retries -= 1
else:
raise Exception("Error getting market price!")
soup = BeautifulSoup(html, 'lxml')
try:
price_change = soup.find("div", { "class": "id-price-change" })
price_change = price_change.find("span").find_all("span")
price_change = [x.string for x in price_change]
price = soup.find_all("span", id=re.compile('^ref_.*_l$'))[0].string
price = str(unicode(price).encode('ascii', 'ignore')).strip().replace(",", "")
return (price, price_change)
except Exception as e:
if retries > 0:
retries -= 1
else:
raise Exception("Can't get current rate for scrip: " + symbol)
Example:
Getting market price: NSE:CIPLA
('558.55', [u'+4.70', u'(0.85%)'])
You can simply parse the result of this request:
https://finance.google.com/finance/getprices?q=GOOG&x=NASD&p=1d&i=60&f=d,c,o,h,l,v
(GOOG at NASDAQ, one day, frequency 60 seconds, DATE,CLOSE,HIGH,LOW,OPEN,VOLUME)
I had a same problem in PHP.
I replace the URL https://www.google.com/finance/converter?a=$amount&from=$from_Currency&to=$to_Currency
to
https://finance.google.com/finance/converter?a=1&from=$from_Currency&to=$to_Currency
Works fine for me.
I am programming in python a script to obtain statistical data of the public schools of the city in which I live. With the following code I get the source code of a page that shows, by pages, the first 100 results of a total of 247 schools:
import requests
url = "http://www.madrid.org/wpad_pub/run/j/BusquedaAvanzada.icm"
post_call = {'Public title': 'S', 'cdMuni': '079', 'cdNivelEdu': '6545'}
r = requests.post(url, data = post_call)
The page can be viewed here.
On that page there is a button that activates a javascript function to download a csv file with all 247 results. I was thinking of using Selenium to download this file, but I have seen, using Tamper Data, that when the button is pressed a POST call occurs, in which the parameter codCentrosExp is sent with the codes of the 247 colleges. The parameter looks like this:
CodCentrosExp = 28077877%3B28077865%3B28063751%3B28018392%3B28018393%...(thus up to the 247 codes)
This makes my work easier, since I do not have to download the csv file, open it, select the code column, etc. And I could do it with Tamper Data, but my question is: how can I get that parameter with my Python script, without having to use Tamper Data?
I finally found the parameter in the page's source code, and extracted them as follows:
schools = BeautifulSoup(r.content, "lxml")
school_codes = schools.findAll(Attrs = {"name": "codCentrosExp", "value": re.compile("^.+$")})[0]["value"]
school_codes = school_codes.split(";")
Anyway, if anyone knows how to respond to the original question, I would be grateful to know how it could be done.
INTRODUCTION
I am working on personal project and using Symfony3.
In order to upload files i am using OneUpUploaderBundle.
And I am not accepting file name that consists of characters with accents, Cyrillic characters, etc.
In order to do so - I am using function from CODE section
TARGET
I would like to use PHP function in CODE section in JavaScript!
CODE
// transliterate text
public function transliterateText($input_text)
{
$input_russian = transliterator_transliterate('Russian-Latin/BGN', $input_text);
$input_german_french = transliterator_transliterate('Any-Latin; Latin-ASCII', $input_russian);
$input_baltic = iconv('UTF-8', 'ASCII//TRANSLIT', $input_german_french);
$transliterated_text = preg_replace('/[^a-zA-Z_0-9\(\)\n]/', '_', $input_baltic );
$transliterated_text = strtolower($transliterated_text);
return $transliterated_text;
}
EXAMPLE
input: "12345 Rūķīši Проверка äöüß àâæçéèêëïîôœùûüÿ.txt"
output: "12345_rukisi_proverka_aouss_aaaeceeeeiiooeuuuy.txt"
QUESTION
I did not found many information about this problem on the Internet.
May be it is not a good idea to use JavaScript for this task...
Or maybe I should create service in Symfony3, that is accessible through AJAX and returns transliterated text instead?
CONCLUSION
Please advise.
Thank You for your time and knowledge.
UPDATE
I would like to use this function in JavaScript in order to show user what the filename would look like when on the server. (File names are going to be transliterated on server anyway). At the moment I am sending (in the UploadListener) following information for each file [{'error':'none'}{'orig':'my file name.txt'}{'t13n':'my_file_name.txt'}]. I would like to send as little as possible informātion from server to browser. So if there was "translation" of the CODE I would need only to send error for each file...
I would like to begin consuming VersionOne api to use as building block for presenting custom HTML page view of Story Names (expand details on this later once I got the initial thing working). I'd like to formulate this in HTML, javascript, JSON, OAUTH2 (no callback URL) implementation. For this implementation the query.v1 endpoint is required and the usage/setup instructions for me aren't as clear/easy to use as the rest-1.v1.
I did "build"/download my personal client secret("v1_client_secrets.json). Although I do not know what "using the scope query-api-1.0" from the documentation means or how it would get applied. Api documentation I have been reading is here: https://community.versionone.com/Developers/Developer-Library/Sample_Code/Tour_of_query.v1
I am fine with the json data GET pattern below.
{
"from": "Story",
"select": [
"Name"
]
}
Although there are examples of the JSON patterns, I can't seem to understand how to code it from start to finish. Is there a starter template script (includes all html/javascript) based on my details, that someone could pass along? I am assuming jquery, angular, ajax would be incorporated in the retrieval/parse/read process.
FYI, I am new to this coding, especially consuming api, so hoping this all makes sense.
Using the Javascript SDK you can communicate with a VersionOne instance. If you are using node you can install the package with npm install v1sdk. Otherwise the source code can be downloaded at https://github.com/versionone/VersionOne.SDK.JavaScript/ . An example of this.
import $ from 'jquery';
import sdk, {jqueryConnector} from 'v1sdk';
const jqueryConnectedSdk = jqueryConnector($)(sdk);
const v1 = jqueryConnectedSdk('www14.v1host.com', 'v1sdktesting', 443, true)
.withCreds('admin', 'admin'); // usage with username/password
// .withAccessToken('your token'); // usage with access tokens
v1.create('Story', {estimate: 5, status: 'Not Started'})
.then((story) => v1.update(story.oidToken, {estimate: 7}))
.then(v1.query({
from: 'Story',
select: ['Estimate', 'Status'],
where: {
Status: 'Not Started'
}
}))
.then(console.log)
.catch(console.log);
This example shows creating a v1 connection, creating a story, updating that story, and then querying for that same story and selecting the Estimate and Status where the Status is 'Not Started'. This is an article that gives an example of displaying the backlog using the SDK http://walkerrandolphsmith.com/blog/v1sdk/. It also has an accompanying git repo you can look at mentioned in the article.