I'm building a simple search engine that takes a user submitted search query as input and outputs a list of appropriate search results. Search queries ultimately get sent to a third party API, which does the heavy lifting of generating search results.
There are 2 ways I can handle this workflow:
My server takes user requests, queries the third party API, and returns results to the user
or
Shifting this responsibility to the client-side; the client queries the third party API directly.
What are some considerations when choosing between these 2 methods?
Using option #1 gives you a few advantages like:
You have one API for one view. It brings some clarity in your architecture.
You might have a lot of pages/windows/whatever using same search API. If third-party API changes or moves to another domain or does something which makes you change your code, you can fix only one API method on your server instead of fixing all your clients.
You can perform some additional changes with your search query like translating it if third-party search engine can't do it itself. Formally, you can implement some additional logic.
Using option #2 reduces the burden on your server.
Related
I have struggled to find many resources on this online. I am developing an application that multiple users will be using at the same time. This means that one user may edit the database after another user has loaded the data from database. This means that this second user will not have an up to date view of the current state of the database. What is the best way to subscribe to database changes and deal with them. I am using a MEAN stack.
If you are trying to develop a real time system where changes are reflected instantly upon changes in database, you need to make use of web sockets. Since you are using Node.js as backend, see Socket.io
A good resource for implementation can be found here
However, if you plan on implementing web sockets, you will have to make significant changes to both your Node.js and Angular code.
Another method (which I would not recommend) is to make periodic api calls for those views which you want to reflect real time changes. You can make use of setInterval for this
I have built a web application using AngularJS (front-end) and PHP/MySQL (back-end).
I was wondering if there is a way to "watch" the MySQL database (without Node.js), so if one user adds some data to it, the changes are synced to other users too.
E.g. I know Firebase does that, but it's object oriented database and I am unable to do the advanced queries there like I do with SQL.
I was thinking to use $interval and $http and do ajax requests, so that way I could detect changes in the database. Well, that's possible, but it'll then do thousands of http requests to the server everyday and plus interpret php on each request.
I believe nothing is impossible, I just need an idea to do this, which I don't have, so that's why I am asking for a help here.
If you want a form of "real-time communication" you'll likely have to incorporate some form of long-polling from the client. Unless you use web sockets, but that's a big post about a bunch of different things. You're right to be concerned about bandwidth and demand on the DB though. So here's my suggestion:
If you don't have experience with web sockets then log your events in a separate table/view and use the pub/sub method to subscribe entities to an event, and broadcast that event to the table. Then long-poll against the watcher view to see when changes may have occurred. If one did occur then you query for the exact value.
Another option would be to use some query system with "deciders" that hold messages. Take a look at Amazon's SQS platform for a better explanation of how this could work. Basically you have a queue that holds messages and a decider chooses where to store the message using some hash or sorting method (to reduce run time). When the client requests an update, the decider finds any messages that would apply based on the hash/sort and returns them. Then you just have to decide how and when to destruct the messages.
The second option would require a lot more tinkering though, so it's really about your preference. I think what you'll find the difficulty to be is that most solutions have to deal with the fact that the message has to be delivered 1 or More times and you'll need to track when someone received the message and if it can now be deleted from the queue/event table or if you still need to wait. Otherwise you'll consume a lot of memory.
I am developing a web application using NodeJS & SailsJS frameworks. Now I am going to develop searching functionality. There are around 5000 records from which I want to search on one attribute.
I know I can search it using mogodb query. What if I get all the records in javascript at frontend and search from it? What is good way to search? At backend using db query or at fronend using javascript searching?
If you search in the frontend then you have to load the entire dataset into the frontend and keep it synchronised for every query. This is not a good idea.
Use database queries - that is what they are designed for, and you only need to transfer the results.
It's all about your app and users expectations on it. You definitely shouldn't use client-side search if you have:
Short-living data which couldn't be cached (like list of users who are online).
Huge dataset which a) couldn't be cached or b) wouldn't be cached (most visitors woudn't use search). But the size limit depends on the app.
Complex computation intensive search (like full-text search).
In other cases it can work. And searching even millions of data records could run under 100 ms, what is faster than common network delay required to receive a response from server.
Advantages of client search:
fast: no network latency.
powerful queries: query can use all JS capabilities with engine optimization advantages.
Disadvantages:
load full dataset (critical on huge amounts of data).
require synchronization strategy: full reload, partial updates, CRDT, etc.
Do it in backend only using db query, which is good practice.It will reduce execution time.
Should not do this kind of check in client side as you have to send the whole database to client and loop through the records several times to fetch the desired records.
Most 3rd party analytics services that rely on client-side tracking (i.e. Javascript tags) are built around the assumption that you want to push data to the third party's servers, and then use their reporting front-end.
For a company with a large data warehouse, there is often a requirement to join the web (or mobile) events data to offline data on a persistent unique user id. From there, the company will have it's data scientist/devs/analysts build custom reports and front end dashboards.
I'm looking for a solution that handles all client-side tracking, but assumes you want to push the data to a local data warehouse. The solution must allow for a persistent unique user id to be associated with each event (unlike Google Analytics). I am NOT concerned with a reporting front end.
EDIT: the real limitation seems to be exporting data at per-event, per-user granularity from an API call, intra-day. Mixpanel allows this level of granularity, but only every 24 hours as a data dump.
edit based on clarification in comments
A couple clarifications: 1. I'm looking to push web event data to my
own servers, not push offline data to Google's Servers. 2. Google does
not allow for export of data at per user id granularity. 3. Mixpanel
does allow for export at per user id granularity, but only every 24
hours as a giant data dump
Here are the 2 that I have seen come up most often in my web analytics adventures:
Piwik is an open source tracking solution that you install and collect data on your own server. The main advantage to this one is that it's free. But it is also really popular and has a robust community.
Alternatively, WebTrends, has a version that can be installed on your own server (Analytics on Premises). Personally I'm not a fan of WebTrends, but I've had some clients swear by it for reasons unknown to me.
For my CMS component I'm implementing integration with Twitter API to fetch and display list of tweets (either connected to user or search query). I'm using Twitter Restful API v1.1, since the 1.0 version is going to be dropped in two months. Two interesting requests for me are user_timeline and search one.
Since my technology strongly relies on caching I need to avoid server-side processing as much as possible providing static html and piece of javascript. I've done it already for old version API and it worked fine. New approach however requires providing authentication data via OAuth. One of the property (oath_signature) is a hash of other properties (in which there are oauth_timestamp and oath_nonce, which should (should they?) be unique per each twitter request) and secret keys, thus make it unsecure to generate it on client side.
Is there any secure way to get list of tweets on client-side using new API?
The simple answer to your question is, "no, there is no secure way of doing this without server-side code." What I would do is set up a service to poll Twitter every xxxx seconds and retrieve the desired tweets. You should cache or store the results and then empty them each time you make your next request. If you are using C#, I have been working on a C# Twitter library that replicates Twitter's API and already has support for grabbing a user's timeline. I will be adding support for filter and search within the next two days (each one should take no more than ten minutes to implement, excluding testing, if you decide to do it yourself). You can reference this library in the service that I mentioned before.
If you do not have the server resources that you need for this, then I strongly caution you against using solutions that circumvent Twitter's intended securities, as it could leave you or your client in a vulnerable position.
You'll have to write a proxy web service on your server side. And as you say caching will be critical to avoid the 15 requests per 15 minutes for basic stuff like pulling tweets.
Definitely avoid doing any auth stuff on the front end. The new "application only" auth using OAuth 2 would allow you to embed bearer tokens in JavaScript meaning you don't need to do any of the signature stuff you're talking about. But don't. Anyone could use your bearer token and if your own users didn't exhaust the rate limit, other people stealing your token might.
If you don't have the server side resources to do this yoursefl, you might want to look at Flamingo. It'll do the auth and the caching for you, so you only need to work in JS like you used to.