I have been using AWS Firehose, Lambda and ElasticSearch to analyse and visualize Twitter data following this tutorial:
https://aws.amazon.com/blogs/big-data/building-a-near-real-time-discovery-platform-with-aws/
It works great and I can open the link to the Kibana dashboard and make the plots. However, I would like to display the plots in a custom built dashboard that I will build with HTML and javascript as part of a web app (I will use plotly.js to build plots).
How can I get the data from ElasticSearch into the javascript file to make the plots? Can I make the plots update in realtime like Kibana does
Is the endpoint an API? What javascript packages and commands do I need?
Do I even need ElasticSearch or could I just go straight from the S3 bucket which holds the streaming data into the app via a Lambda function? What are the advantages/disadvantages?
Please have a look at elastic apis
It's just a matter of consuming those APIs with any programmed ReST Client such as retrofit/okHttp etc.
A sample here : http://www.baeldung.com/elasticsearch-java
To implement this in JS , a simple call will serve your purpose :
**Using JQuery
$.ajax({
url: 'http://<yourElasticHost:port>/',
type: '<GET/PUT/POST etc>',
data: 'ID=1', //
success: function() { alert('done !!'); }
});
ElasticSearch includes lots of search options and it is Document-oriented.
If the access speed, Scalability, fine tune queries based on json dsl are your requirements then you can definitely go for elastic approach
Related
I am new to elasticsearch and I am trying to build a movie search app. For this I plan to get data from kaggle and add to my elasticsearch which I have setup locally at localhost:9200. I see this in the localhost link:
name "bxiIZLL"
cluster_name "elasticsearch"
cluster_uuid "zc_JPmw4TQ2G5bvahEF6LQ"
version
number "5.6.14"
build_hash "f310fe9"
build_date "2018-12-05T21:20:16.416Z"
build_snapshot false
lucene_version "6.6.1"
tagline "You Know, for Search"enter code here
Now I need to add Kaggle data to this server. How can I do it?
I saw somewhere the curld -XPUT command. I am not sure how that can work with Kaggle.
A follow up question - if I want to publish my app later on, how can I host the elasticsearch ?
In order to upload a CSV file to elasticsearch:
download the file.
use logstash in order to read the file using file input
modify and transform the data as you need using logstash's CSV filter
output logstash to elasticsearch
For your follow up question - how can I host Elasticsearch - you can either run it by your own, in AWS EC2 for example, or use a managed service like Elastic cloud or AWS ES. good luck
We build a NN model in tensorflow similar to
https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/census/tf-keras
Then we exported it to GCP using the following gcloud ai-platform models create commands
and predict using gcloud ai-platform predict works fine
Now we want to do online prediction using javascript, seems to have several options:
using https://www.npmjs.com/package/#google-cloud/automl, but
Checking https://googleapis.dev/nodejs/automl/latest/v1beta1.PredictionServiceClient.html#predict
it seems that it can not be used for neural networks(?)
When we run the example code
https://www.npmjs.com/package/#google-cloud/automl
with our parameters we get 'Invalid resource ID' error
Using REST API, curl works fine but how do we set permanent authorization in the server ?
Any other suggestion ?
Amir
I solved the problem by building Python container that run on gcloud
(https://cloud.google.com/run/docs/quickstarts/build-and-deploy)
The JS call the container using REST API, the python container do the prediction and reply to results the JS
I am trying to insert the record into my mongoDB. Here is what I tried:
$.ajax({
type: "POST",
url: "http://localhost:28017/messages_sent/better/?doc=",
data: {
thread_id: event.threadID,
message_id: event.messageID,
user_email: sdk.User.getEmailAddress(),
token: token
},
dataType: "JSON"
});
countUsage();
chrome.storage.sync.set({ "last_tracked_email_date": Date.now() });
alert("hello");
}
});
I am trying to look into the MongoDB, but could not find any thing in the database.
How I can insert the record into MongoDB using javascript and Rest API of MongoDB itself?
After a good discussion, I get to know that you were trying, is to just insert data in Mongo.
And this Mongo's rest API feature only supports retrieving data
from official docs
The mongod process includes a simple REST API as a convenience. With no support for insert, update, or remove operations, it is generally used for monitoring, alert scripts, and administrative tasks.
So, there are many ways you can insert data in Mongo:
Using APIs in server side code using any server side lang like Python/java/node
Using mongo shell, simple insert
using some GUI tool, like robomongo
and I'm sure there are more
But in comments, OP wanted some data in Mongo to test functionality, so writing APIs for just inserting is not a great solution.
So, if you just want to insert data and comfortable using CLI, then just go in mongo shell, and insert directly into the collection.
But if not, then Robomongo is very efficient. It's made for adding/removing and other CRUD operations.
Simple steps for using robomongo:
Download from official site and install
Open it and click on connect, add your port (default 28017) and host (localhost)
Now on lhs, click on MongoDB and you should see you dbs in it it, if you've created those.
Then you can see your collections inside it, and just right click and click on insert option and you can add your JSON into it and click save.
That's it, doc is saved in collection. Simple right.
Introduction
I have a SAP HANA Cloud Platform account. I have also deployed a Java application to the account and created a test servlet, which returns dummy data.
On the other side i have a SAP UI5 application which i develop on the WebIDE. I created also two destinations:
Destination "virtualTEST" is connected to a SAP Backend System (HANA Cloud Connector)
Destination "javaTEST" is connected to my Java servlet application
The neo-app.json is well configured and can obtain data from the test servlet (the dummy data) and data from the SAP Backend System (OData Gateway).
The problem
Now i want to pass variables to the SAP Backend System (virtualTEST) destination, which should not be visible in the frontend to avoid javascript manipulation.
My first thought
My first thought was that i create a new servlet which acts as proxy. In the SAPUI5 i call the servlet from javaTEST destination and pass the "hidden variables" like /testServlet?targetUrl=https://webide-xxx.hana.ondemand.com/sap/opu/odata/TEST_SRV/TEST?$filter=Var eq '{{MYVAR}}' and the Java application replaces {{MYVAR}} with my real variable. Then the target will be loaded (this is also a destination url to my SAPUI5 application). This does not work, i do not know why, but i think the proxy can not obtain data from the destination of an application.
Also, i think this is not the best solution. How can this be solved? Any ideas or best practices? Can destinations be used in Java application? :)
It is not really clear to me what you want to achieve. Of cause you can call destinations from Java.
<resource-ref>
<res-ref-name>myBackend</res-ref-name>
<res-type>com.sap.core.connectivity.api.http.HttpDestination</res-type>
</resource-ref>
import javax.naming.Context;
import javax.naming.InitialContext;
import com.sap.core.connectivity.api.http.HttpDestination;
...
// coding to lookup the destination "myBackend"
Context ctx = new InitialContext();
HttpDestination destination = (HttpDestination)
ctx.lookup("java:comp/env/myBackend");
// coding to call service "myService" on the system configured in the given destination
HttpClient createHttpClient = destination.createHttpClient();
HttpGet get = new HttpGet("myService");
HttpResponse resp = createHttpClient.execute(get);
from the official documentation. in the HttpGet you could set Params if you like.
In my opion your Backend should be so save you don't have to worry about Javascript manipulations, especially not for exposed OData services
Regards
Mathias
The Short
I have an existing WCF Data Service that I would like to wire up to use in an AngularJS SPA using Breeze.
Can anyone show a noobie level example of how to do that with out access to the actual database (just the OData Service)?
The Long
I have an existing WCF Data Service that is already in use by a WPF app.
I experimenting with web development and would like to wire up to that service using Breeze. In case it matters, I am using Angular (and am setting up via the HotTowel.Angular nuget package).
I have done a fair amount of Googling and I am stuck.
I can see two ways outlined from my searching:
The First
Make is to make a Breeze controller on the server side of my web app.
The problem I see with that is the metadata. From my limited understanding I need to tell breeze all the meta data of my WCF Data Service. I know how to get the meta from my WCF Data Service (the url + $Metadata), but I don't know how to tell this to Breeze.
The Second
This way is more vague in implementation. I got it from the accepted answer on this question: Breeze.js with WCF Data Service.
Basically the answer here does not seem to work. It relies on something called the entityModel that I cannot seem to find (I have an entityManager, but not an entityModel. And the entityManager does not have the properties that the entityModel is shown to have.
In the end I like the idea of the second method best. That way I can directly connect to my odata service with out needed my web app to have a "in-between" server component. But I would happily take anything that does not need entity framework to connect to my database.
I tried several variations of the second option, but I just can't seem to get it to work. I also tried the Breeze samples. It has one for OData, but it still relies on having Entity Framework hook up to the source database.
To to clearly summarize what I am asking: I am looking for a Breeze example that connects to an existing WCF Data Service.
We regret that you were mislead by that old StackOverflow answer which was way out of date and (therefore) incorrect. There is no longer a type called entityModel.
I updated the answer there and repeat here the same advice.
The recommended way to configure Breeze so that it talks to a standard OData source (such as a WCF OData service) is
breeze.config.initializeAdapterInstance('dataService', 'OData', true);
Here's how you might proceed with defining an EntityManager and querying the service:
// specify the absolute URL to the WCF service address
var serviceName = "http://localhost:9009/ODataService.svc";
var em = new breeze.EntityManager(serviceName);
var query = breeze.EntityQuery.from("Customers")
.where("CompanyName", "startsWith", "B")
.orderBy("City");
em.executeQuery(query).then(function(data) {
// process the data.results here.
});
There is some documentation on this subject here.
A Web API OData service differs from a WCF OData service in several respects. But you may still find value in the Angular Web API OData sample.