Develop Reference

Randomly occuring Unexpected end of JSON on GET request

I'm developing a simple web project where one of the features (Related to the error) will display a quote, credit and source. This feature works as intended using setInterval every 12seconds to call a function which will make a connection to my XAMPP hosted MYSQL server and return a random row as a JSON object which I can use to display on the page. below is the Quote retrieval script and PHP:
quotes.js
(function() {
ajax = new XMLHttpRequest();
var method = "GET";
var url = "data.php";
var asynchronous = true; // revise asynchronous
var quoteContainer = document.getElementById("quote");
var authorContainer = document.getElementById("author");
function getQuote() {
ajax.onreadystatechange = function(){
if (this.readyState == 4 && this.status == 200) {
var data = JSON.parse(this.responseText);
console.log(data)
quoteContainer.innerHTML = "\"" + data.Quote + "\""
authorContainer.innerHTML = data.Author + " - " + "<a href='"+ data.Source +"' target='_blank'>source</a>";
}
}
ajax.open(method,url,asynchronous);
ajax.send();
}
getQuote();
setInterval(getQuote,12000); //set to 100 for testing
}());
Data.php
<?php
// write secondry query for seasonal messages
$conn = mysqli_connect("localhost","root","","quotes_db");
$sql = "SELECT * FROM quote_table ORDER BY RAND() LIMIT 1;";
$result = mysqli_query($conn,$sql);
$data= mysqli_fetch_assoc($result);
echo json_encode($data);
?>
The issue is that on random occasions while this is running the server returns a 'null' packet (with the titled error) I have checked the network data using Chromes developer console and the effected packets are empty and take slightly longer to return. I have checked my database table for special characters and it all seems normal. due to the packet returning null the page is left empty
How can I stop this error appearing, so that I can get a continuous stream of random rows from my table?
If the answer is not an obvious one what bug-testing steps should I take to find a fix?
If more information is needed I can update this post.
Error log

This usually happens when the data returned is undefined or is not in a valid format. That is when JSON.parse() would fail. You can consider putting try-catch block.
try {
var data = JSON.parse(this.responseText);
} catch(e) {
console.log('Unable to parse the string.')
}
Also, consider wrapping the JSON.parse() statement in an if condition, if server sends an empty response.

You can do some kind of error handling to achieve your goal.
ajax.onreadystatechange = function(){
if (this.readyState == 4 && this.status == 200) {
try
{
var data = JSON.parse(this.responseText);
}
catch (error)
{
if (error instanceof SyntaxError)
{
// unable to parse the result to json
// maybe retry?
// do some kind of errer handling
}
}
}
Using the try-catch statement you simply catch the error instead of stopping the script from executing so you are able to handle errors yourself.
Best regards.
EDIT:
setInterval(function ()
{
var found = false;
var attemptCount = 0;
var allowedAttempts = 10; // your exit condition, so you wont get stuck in an infinite loop if the server is broken or something
while (!found && attemptCount < allowedAttempts)
{
try
{
getQuote();
}
catch (error)
{
if (error instanceof SyntaxError)
{
attemptCount++;
continue;
}
}
found = true;
}
}, 12e3);
EDIT No.2: (based on your comment on another answer)
If you want to do some server-side validation, you have to modify the PHP-Code:
$data = []; // set a default return value
$result = mysqli_query($conn,$sql);
if (mysqli_num_rows($result) === 1) // we found a result
{
$data= mysqli_fetch_assoc($result);
}
echo json_encode($data);
So, if you combine both, the client- and server-side validation you should be fine for sure:
What did we do?
we implemented a server side validation, so the result that is returned to the client should never throw an error when going through JSON.parse(). To make this work you have to implement a client-side validation on the result of JSON.parse() to make sure that you got a valid result (because it could be an empty array).
if any errors occur on the server (for whatever reason) and the returned result cannot be parsed, we simply retry the entire process for n times.
Best regards.

The issue is caused by your encoding: Jean de La Bruyère is transmitted as Jean de La Bruy�re by MySQL.
PHP json_encode() is not able to deal with the encoding of the French character as shown in this print_r() output of the mysqli_fetch_assoc() result:
Array
(
[QuoteID] => 6
[Quote] => Those who make the worst use of their time are the first to complain of its brevity.
[Author] => Jean de La Bruy�re
[Source] => https://www.goodreads.com/author/show/778012.Jean_de_La_Bruy_re
)
As json_encode() is only accepting UTF-8 encoded strings ("All string data must be UTF-8 encoded.", cf. PHP Documentation), you can tell MySQL to return every result in UTF-8 encoding by simply executing the following query once after opening the connection:
mysqli_query($conn, "SET NAMES 'utf8'");

Bing Web Search API GET request returning 404 not found

When submitting a search on my bing custom search API it is returning 404 not found.
I have changed the API key and endpoint to my details
My endpoint is: https://fruitbox-search.cognitiveservices.azure.com/bing/v7.0
// Replace with a valid subscription key from your Azure account.
$accessKey = 'MY-KEY-HERE';
$endpoint = 'https://fruitbox-search.cognitiveservices.azure.com/bing/v7.0';
$term = 'Microsoft Cognitive Services';
function BingWebSearch ($url, $key, $query) {
/*
* Prepare the HTTP request.
* NOTE: Use the key 'http' even if you are making an HTTPS request.
* See: http://php.net/manual/en/function.stream-context-create.php.
*/
$headers = "Ocp-Apim-Subscription-Key: $key\r\n";
$options = array ('http' => array (
'header' => $headers,
'method' => 'GET'));
// Perform the request and receive a response.
$context = stream_context_create($options);
$result = file_get_contents($url . "?q=" . urlencode($query), false, $context);
// Extract Bing HTTP headers.
$headers = array();
foreach ($http_response_header as $k => $v) {
$h = explode(":", $v, 2);
if (isset($h[1]))
if (preg_match("/^BingAPIs-/", $h[0]) || preg_match("/^X-MSEdge-/", $h[0]))
$headers[trim($h[0])] = trim($h[1]);
}
return array($headers, $result);
}
I am getting the following errors in my console:
GET https://fruitbox-search.cognitiveservices.azure.com/bing/v7.0?q=test&mkt=en-US&SafeSearch=strict&promote=webpages&answerCount=9&count=25&offset=0&textDecorations=true&textFormat=HTML 404 (Resource Not Found) - VM523:1
Refused to get unsafe header "BingAPIs-TraceId" - script.js:264

I have solved this, solution is below:
I had to set my endpoint to:
https://fruitbox-search.cognitiveservices.azure.com/bing/v7.0/search
It was missing the /search at the end.

Data not being sent via XMLHttpRequest

I have a weird issue with a script I am using to send some info to my PHP. I say weird because this code is a direct copy of some already working code on the same application, with new variable names. I need to upload a small amount of data to my server, and while the value is there and the code runs all the way through, the POST data is not being sent or something. I have checked, and it says it has been submitted to my PHP with the value but nothing happens. I am using Vue.js if there are questions about some of my formatting
I have tried looking at other examples online, but my thing is that this was a block of code that copied from a working part of my application. It works until the data transfer from JS -- PHP
JS
editDisplayName: function() {
var self = this;
var newName = prompt("10 Characters Max, 3 Min", "Enter Display Name");
if(newName.length <= 10 && newName.length >= 3 ) {
var sendData = new XMLHttpRequest();
sendData.open("POST", "primary.php", true);
sendData.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
sendData.send(newName);
self.username = window.localStorage.getItem('username');
window.localStorage.clear();
}
}
PHP
function changeUsername() {
// Connect to Server
$link = // Commented Out For Security;
if (mysqli_connect_error()) {
die ("DB Connection Error");
}
$newName = $_POST['newName'];
$newName = mysqli_real_escape_string($link, $newName);
$token = $_COOKIE['validToken'];
$token = mysqli_real_escape_string($link, $token);
$query = "UPDATE userData SET username = '$newName' WHERE token = '$token' LIMIT 1";
mysqli_query($link, $query);
mysqli_close($link);
echo("
<script>
var username = $newName;
window.localStorage.setItem('username', username);
</script>
");
}
if(isset($_POST['newName'])) {
changeUsername();
}
I'm expecting on my DOM that I would have the new username set, instead it is blank. The spot where stuff stops working is somewhere with the POST data being sent, since my PHP isn't picking anything up, but I can't figure out what is wrong.

Instead of just sending the value, you should send the parameter name as well.
So, instead of using sendData.send(newName);, try using:
sendData.send("newName=" + newName);
(as your Content-type is application/x-www-form-urlencoded)
Or if you want to use JSON:
Change your Content-type to application/json, then send the data as a JSON string, like so:
sendData.send(JSON.stringify({'newName': newName}));

How to send a POST message from MQL4 to NodeJS?

webrequest.mq4
#property copyright "Copyright 2013, apla"
#property link "-"
//+------------------------------------------------------------------+
//| expert start function |
//+------------------------------------------------------------------+
int start()
{
//----
// WebRequest
string cookie = NULL;
string headers = "Content-Type: application/x-www-form-urlencoded";
int res;
string url = "localhost:8080"; // url = localhost:8080
char post[], result[];
string signal = "account=" + AccountNumber() + "&balance=" + AccountBalance() + "&equity=" + AccountEquity();
StringToCharArray( signal, post );
Print( signal );
int timeout = 5000; // 5 sec
res = WebRequest( "POST",
url,
cookie,
NULL,
timeout,
post,
ArraySize( post ),
result,
headers
);
Print( "Status code: " , res, ", error: ", GetLastError() );
//----
return(0);
}
//+------------------------------------------------------------------+
I want to send a file from MetaTrader Terminal 4 webrequest.mq4 to a Node this Site section that can be given up, however.
MT4 >> Nodejs
??? POST[] ??? (JavaScript nodes)
account, balance, equity
how to convert file.php to nodejs
<?php
$myfile = fopen("newfile.txt", "w") or die("Unable to open file!");
$txt = "account ".$_POST['account']."\n";
fwrite($myfile, $txt);
$txt = "balance ".$_POST['balance']."\n";
fwrite($myfile, $txt);
$txt = "equity ".$_POST['equity']."\n";
fwrite($myfile, $txt);
fclose($myfile);
?>
For which I do not know how to get the POST.
writeFile.js
var http = require('http'); var fs = require('fs');
fs.writeFile("file.txt",??? POST[] ???, function(err,data) {
if (err) throw err;
console.log('The file was saved!');
http.createServer(function(req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('OK');
}).listen(8080);
console.log('Server running at http://localhost:8080/'); });

kindly be attentive to details:
Step 0 :MQL4 partshould follow the recent New-MQL4.56789copy/paste code fails
As a first sign of this, MetaTrader Terminal 4 code-base statically present on web does not reflect creeping syntax changes of MQL4 language. Recently MQL4 has moved closer to MQL5 ( reasoning for which is outside of this post, if interested, check other posts about New-MQL4.56789 ).
int start(){...} // cannot be used anymore,
// neither for EXPERT_ADVISOR
// nor for SCRIPT
Recent #property strict compilation mode imposes use of:
void OnTick(){ ...} // for EXPERT_ADVISOR type of MQL4-code
void OnStart(){ ...} // for SCRIPT type of MQL4-code
int OnCalculate(...){ ...} // for CUSTOM_INDICATOR type of MQL4-code,
// while,
// CUSTOM_INDICATOR has explicitly
// FORBIDDEN any attempt
// call to a WebRequest( ... ) et al
This said, your MQL4-part of the code shall be modified in it's principal structure so as to reflect these facts.
For any further tasks, related to MQL4, rather use localhost installed Help-service from the IDE, searching for "help" on web will most of all become a misleading source for un-edited copy/paste attempts due to above presented reasons.
Step 1 :POST http-syntax constructionought be conformant to RFC 7231, Section 4.3.3
as a minimum, your constructed text, being stored into a string signal ought look something like this:
User-Agent: aplaHTTP/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 54
account=123456789&balance=1234567.89&equity=1234567.89
Step 2 :Node.js partparse received parameters for whatever further post-processing
Similarly, the node.js part shall decipher the parameters delivered inside POST-url-encoded http-message.
And the job is done.
Welcome to the Wild Worlds of MQL4

How to set WebSocket Origin Header from Javascript?

I'm trying to use javascript to make a websocket request from a local test.dev page to a server running at ip 123.123.123.123 on behalf of test.com. The request goes through, but the 123.123.123.123 server sees the Origin: test.dev header in the websocket request and rejects the connection because it wants to see Origin: test.com.
Here is the javascript code for connecting the socket:
ws = new WebSocket("123.123.123.123");
How can I use javascript to start a websocket connection with a dishonest Origin header of Origin: test.com?
I was hoping something like this would work, but I can't find any such:
ws = new WebSocket("123.123.123.123", "test.com");

The simple solution would be to simply create an entry in your hosts file to map test.com to 123.123.123.123. You would need to remove this entry later when you want to connect the "real" test.com.
A less hacky solution would require the use of a proxy which can re-write your headers for you on-the-fly. Consider install nginx on your system, and then proxing the request to 123.123.123.123 keeping everything the same except for the Origin header. Here's the entry you would need in your nginx config file:
server {
server_name test.dev;
location / {
proxy_pass http://123.123.123.123;
proxy_set_header Origin test.com;
# the following 3 are required to proxy WebSocket connections.
# See more here: http://nginx.com/blog/websocket-nginx/
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}

How can I use javascript to start a websocket connection with a dishonest Origin header of Origin: test.com?
If we could forge the origin of requests in JavaScript, the same origin policy wouldn't be very good at keeping us safe. It exists solely to protect us from this and other potential attack vectors.
As this looks like dev work, have you considered using a web debugging proxy such as Fiddler (free) or Charles (paid)? With those you could modify the initial handshake request or response for the WebSocket for your own machine or any test machines that are proxied through the debugger.

A good solution is to override the WebSocket call with another websocket library, like https://github.com/websockets/ws. To use this node.js library in a browser, you just have to use http://browserify.org/.

If you really must forge a fake 'origin' header value from javascript - there is a way. Its not something you will find in your generally accepted principles handbook, but here it is:
Create a php file that invokes the socket, with a fake origin value. Now call the php file using ajax, from you javascript.
It may not be elegant, ethical or acceptable, but never accept anyone telling you that it cant be done.
'send.php' was called using ajax from javascript
send.php contents
require "websocket_client.php";
$client = new Client("IP_ADDR:PORT", $_GET[mobile] ."|".$_GET[login]);
$client->send('1112223333|sms');
$client->send(json_encode(array('login'=>$user,'msg'=>$msg)));
echo $client->receive();`enter code here`
websocket_client.php was a class file with basic websocket functions (including the custom origin values
/**
* Perform WebSocket handshake
*/
protected function connect() {
$url_parts = parse_url($this->socket_uri);
$scheme = $url_parts['scheme'];
$host = $url_parts['host'];
$user = isset($url_parts['user']) ? $url_parts['user'] : '';
$pass = isset($url_parts['pass']) ? $url_parts['pass'] : '';
$port = isset($url_parts['port']) ? $url_parts['port'] : ($scheme === 'wss' ? 443 : 80);
$path = isset($url_parts['path']) ? $url_parts['path'] : '/';
$query = isset($url_parts['query']) ? $url_parts['query'] : '';
$fragment = isset($url_parts['fragment']) ? $url_parts['fragment'] : '';
$path_with_query = $path;
if (!empty($query)) $path_with_query .= '?' . $query;
if (!empty($fragment)) $path_with_query .= '#' . $fragment;
if (!in_array($scheme, array('ws', 'wss'))) {
throw new BadUriException(
"Url should have scheme ws or wss, not '$scheme' from URI '$this->socket_uri' ."
);
}
$host_uri = ($scheme === 'wss' ? 'ssl' : 'tcp') . '://' . $host;
// Open the socket. # is there to supress warning that we will catch in check below instead.
$this->socket = #fsockopen($host_uri, $port, $errno, $errstr, $this->options['timeout']);
if ($this->socket === false) {
throw new ConnectionException(
"Could not open socket to \"$host:$port\": $errstr ($errno)."
);
}
// Set timeout on the stream as well.
stream_set_timeout($this->socket, $this->options['timeout']);
// Generate the WebSocket key.
$key = self::generateKey();
// Default headers (using lowercase for simpler array_merge below).
$headers = array(
'host' => $host . ":" . $port,
'user-agent' => 'websocket-client-php',
'connection' => 'Upgrade',
'upgrade' => 'websocket',
'origin' => $MY_CUSTOM_SHADY_VALUE,
'sec-websocket-key' => $key,
'sec-websocket-version' => '13',
);
// Handle basic authentication.
if ($user || $pass) {
$headers['authorization'] = 'Basic ' . base64_encode($user . ':' . $pass) . "\r\n";
}
// Deprecated way of adding origin (use headers instead).
if (isset($this->options['origin'])) $headers['origin'] = $this->options['origin'];
// Add and override with headers from options.
if (isset($this->options['headers'])) {
$headers = array_merge($headers, array_change_key_case($this->options['headers']));
}
$header =
"GET " . $path_with_query . " HTTP/1.1\r\n"
. implode(
"\r\n", array_map(
function($key, $value) { return "$key: $value"; }, array_keys($headers), $headers
)
)
. "\r\n\r\n";
// Send headers.
$this->write($header);
// Get server response.
$response = '';
do {
$buffer = stream_get_line($this->socket, 1024, "\r\n");
$response .= $buffer . "\n";
$metadata = stream_get_meta_data($this->socket);
} while (!feof($this->socket) && $metadata['unread_bytes'] > 0);
/// #todo Handle version switching
// Validate response.
if (!preg_match('#Sec-WebSocket-Accept:\s(.*)$#mUi', $response, $matches)) {
$address = $scheme . '://' . $host . $path_with_query;
throw new ConnectionException(
"Connection to '{$address}' failed: Server sent invalid upgrade response:\n"
. $response
);
}
$keyAccept = trim($matches[1]);
$expectedResonse
= base64_encode(pack('H*', sha1($key . '258EAFA5-E914-47DA-95CA-C5AB0DC85B11')));
if ($keyAccept !== $expectedResonse) {
throw new ConnectionException('Server sent bad upgrade response.');
}
$this->is_connected = true;
}