I am new to elasticsearch and I am trying to build a movie search app. For this I plan to get data from kaggle and add to my elasticsearch which I have setup locally at localhost:9200. I see this in the localhost link:
name "bxiIZLL"
cluster_name "elasticsearch"
cluster_uuid "zc_JPmw4TQ2G5bvahEF6LQ"
version
number "5.6.14"
build_hash "f310fe9"
build_date "2018-12-05T21:20:16.416Z"
build_snapshot false
lucene_version "6.6.1"
tagline "You Know, for Search"enter code here
Now I need to add Kaggle data to this server. How can I do it?
I saw somewhere the curld -XPUT command. I am not sure how that can work with Kaggle.
A follow up question - if I want to publish my app later on, how can I host the elasticsearch ?
In order to upload a CSV file to elasticsearch:
download the file.
use logstash in order to read the file using file input
modify and transform the data as you need using logstash's CSV filter
output logstash to elasticsearch
For your follow up question - how can I host Elasticsearch - you can either run it by your own, in AWS EC2 for example, or use a managed service like Elastic cloud or AWS ES. good luck
Related
We build a NN model in tensorflow similar to
https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/census/tf-keras
Then we exported it to GCP using the following gcloud ai-platform models create commands
and predict using gcloud ai-platform predict works fine
Now we want to do online prediction using javascript, seems to have several options:
using https://www.npmjs.com/package/#google-cloud/automl, but
Checking https://googleapis.dev/nodejs/automl/latest/v1beta1.PredictionServiceClient.html#predict
it seems that it can not be used for neural networks(?)
When we run the example code
https://www.npmjs.com/package/#google-cloud/automl
with our parameters we get 'Invalid resource ID' error
Using REST API, curl works fine but how do we set permanent authorization in the server ?
Any other suggestion ?
Amir
I solved the problem by building Python container that run on gcloud
(https://cloud.google.com/run/docs/quickstarts/build-and-deploy)
The JS call the container using REST API, the python container do the prediction and reply to results the JS
I have been using AWS Firehose, Lambda and ElasticSearch to analyse and visualize Twitter data following this tutorial:
https://aws.amazon.com/blogs/big-data/building-a-near-real-time-discovery-platform-with-aws/
It works great and I can open the link to the Kibana dashboard and make the plots. However, I would like to display the plots in a custom built dashboard that I will build with HTML and javascript as part of a web app (I will use plotly.js to build plots).
How can I get the data from ElasticSearch into the javascript file to make the plots? Can I make the plots update in realtime like Kibana does
Is the endpoint an API? What javascript packages and commands do I need?
Do I even need ElasticSearch or could I just go straight from the S3 bucket which holds the streaming data into the app via a Lambda function? What are the advantages/disadvantages?
Please have a look at elastic apis
It's just a matter of consuming those APIs with any programmed ReST Client such as retrofit/okHttp etc.
A sample here : http://www.baeldung.com/elasticsearch-java
To implement this in JS , a simple call will serve your purpose :
**Using JQuery
$.ajax({
url: 'http://<yourElasticHost:port>/',
type: '<GET/PUT/POST etc>',
data: 'ID=1', //
success: function() { alert('done !!'); }
});
ElasticSearch includes lots of search options and it is Document-oriented.
If the access speed, Scalability, fine tune queries based on json dsl are your requirements then you can definitely go for elastic approach
I am trying to learn how to process payments with Square, and am trying to run their examples from GitHub to get a feel regarding how to structure the payments application. I built the node example from here: https://github.com/square/connect-api-examples/tree/master/connect-examples/v2/node_payment using npm install and npm build to get the app up and running.
I am using "4532759734545858" for the card number, "123" for CVV, "0120" for expiration, and "94103" for the zip. I got the card number from here: https://docs.connect.squareup.com/articles/using-sandbox where it states that this is a good number to use for a Visa sandbox.
Also, I have updated the config.json with properties from my developer settings.
When trying to process a payment a get a DOM element that says "Card Declined" without further specifying the error. Is there something I can do to parse the error?
Based on the documentation at: https://docs.connect.squareup.com/articles/using-sandbox#generatingerrorstates it seems the amount_money field of the request is not being populated, but I am having trouble confirming.
Ideally I would like to get to a point where I can add a card as a hash value to my db and use it for recurring billing...
That "card declined" message is actually the error you get back from Square's APIs. You can play around with the error messaging in the app.js file and the `error.jade. Try error.catagory, code, detail.
Keep in mind that this is just a sample app, to show that you can use the APIs with node.js, you probably don't want to use this code in your production system.
inside region_config.json in aws-sdk I can only see "eu-west-1"
Does AWS SDK for JavaScript support "eu-west-2" region ?
Regions endpoints are made as similar as possible to avoid multiple configuration keys.
In your case, you can just add the following line in the region_config.json to add the ability for s3:
"eu-west-2/s3": "s3dash",
Other service are generic enough to understand directly at call time how to route your request.
Suppose I want to execute 'db.things.insert({colors : ["blue", "black"]})' in browser. I can execute it in the Mongodb -shell but not yet understanding how to execute it like: open up the Google Chrome Console, initialize the DB -object with some connection and execute the command. Does there exist some plugin? Sorry I am totally new to MongoDB, trying to test just things fast with Browser shell. How can I do the initialization like that?
Trial 0: perhaps with REST -interface?
I have enabled the REST with "$ echo 'rest=true' > /etc/mongodb.conf;
$ sudo restart mongodb", works in Ubuntu. More about rest
here, not sure
yet whether needed here but perhaps with some POST/REST -method I can
do the init.
Trial 1: Oreilly's book about MongoDB and 50 Tips (page 47)
The book has some example
> db = connect ("ny1a:27017/foo")
> db = connect ("ny1a:27017/admin")
so now
> db=connect("localhost:27017/test")
ReferenceError: connect is not defined
Yes because I need to source the connect -command, some further examples here, where can I get it?
P.s. I am studying this tutorial here.
You cannot simply access mongodb from the browser console. Your browser is a client, and there isn't (as far as I know) a javascript client-side library. Any javascript library you will find will most likely be for Node.js (server).
The mongo console is its own type of compiled client. The native language is javascript, but those commands only pertain to the actual mongo command shell. connect is a command for the mongo command shell.
Enabling REST starts a port on your mongod that will accept REST http communication. You get a browser page here: http://localhost:28017/
This is a very basic page displaying data, but you can further run queries yourself. See http://www.mongodb.org/display/DOCS/Http+Interface#HttpInterface-SimpleRESTInterface
Note, the REST interface is READ ONLY. You cannot insert.
That being said, you need a proper driver for your language of choice.