Array output problem at Webscraping with DOMparser - javascript

I'm facing some output trouble at the second part of my codes.
function getSiteContent($url)
{
$html = cache()->rememberForever($url, function () use ($url) {
return file_get_contents($url);
});
$parser = new \DOMDocument();
#$parser->loadHTML($html);
return $parser;
}
libxml_use_internal_errors(true);
$url = 'https://sumai.tokyu-land.co.jp/osaka';
$parser = getSiteContent($url);
$allDivs =[];
$allDivs = $parser->getElementsByTagName('div');
foreach ($allDivs as $div) {
if ($div->getAttribute('class') == 'p-articlelist-content-right') {
$allLinks = $div->getElementsByTagName('a');
foreach ($allLinks as $a) {
$getlinks[] = $a->getAttribute('href');
}
}
}
var_dump($getlinks);
At this var_dump I can see links that I scraped. No problem 'till here. And one more time. I want to go into those links. That's why I wrote the codes right below.
getSiteContent($getlinks);
$link = [];
$siteler = [];
foreach ($siteler as $site) {
if($site == 'https://sumai.tokyu-land.co.jp'){
$site = $getlinks->getElementsByTagName('div');
foreach ($site as $links) {
if($links->getAttribute('class') == 'pc_hnavi'){
$linker = $links->getElementsByTagName('a');
foreach ($linker as $a) {
$link = $a->getAttribute('href');
}
}
}
}
}
var_dump($link);
When I var_dump it. It says Array 0
I didn't understand why it doesn't go in those links with foreach
My codes are wrong? What am I missing here? Any idea for this?
Thank you helping me out.

As I said in the comments $siteler is empty when you try to loop through it, but there are a couple more problems:
First your code will only trigger once at most, when the link is exactly 'https://sumai.tokyu-land.co.jp' and I'm not sure that's what you want.
You are calling DOM functions on an array.
Only seem to care about links inside 'div' tags.
You redefine the $link variable on each loop, so the final result will be just one link.
This is the fixed code:
$link = [];
foreach ($getlinks as $site) {
// Any link in the domain, not just the homepage
if(strpos($site, 'https://sumai.tokyu-land.co.jp') === 0) {
$dom = getSiteContent($site);
$divs = $dom->getElementsByTagName('div');
foreach ($divs as $div) {
// Can have more than one class
$attrs = explode(' ', $div->getAttribute('class'));
if(in_array('pc_hnavi', $attrs)) {
$linker = $div->getElementsByTagName('a');
foreach ($linker as $a) {
// Add to the array
$link[] = $a->getAttribute('href');
}
}
}
}
}
However this doesn't check if the link already exists in the array and has the potential to process the same links over and over. I'd strongly suggest to use an existing crawler.
From the comments, turns out that pc_hnavi is an id and not a class and you are interested in the first link only. You can access that element directly without iterating the elements:
foreach ($getlinks as $site) {
// Any link in the domain, not just the homepage
if(strpos($site, 'https://sumai.tokyu-land.co.jp') === 0) {
$dom = getSiteContent($site);
$div = $dom->getElementById('pc_hnavi');
if ($div != null) {
$links = $div->getElementsByTagName('a');
if ($links->length > 0) {
$a = $links->item(0);
$link[] = $a->getAttribute('href');
}
}
}
}

It looks like your problem is here:
...
$siteler = []; // $siteler is set to an empty array ...
foreach ($siteler as $site) { // then you loop through the empty array which does nothing ...
...
}
...
Fixing that should get you started.

Related

PHP Array to Javascript Object no array after console.log

Now there are plenty of people with the same issue and their resolutions have not worked.
Problem
I have a list of Times coming back from a REST call. They are created exactly as they should, except I want them to be stored into a JavaScript Array.
PHP
function getOpenAppts()
{
global $wpdb;
$final_array = "";
$td = date('m/d/Y');
$datetime = new DateTime('tomorrow');
$tm = $datetime->format('Y-m-d');
$startDate = $td;
$endDate = $tm;
$apptTypeID = "23";
$get_loc = $wpdb->get_results('SELECT provid,id FROM location');
foreach($get_loc as $val){
// call to Athena to get open appointments
$res = getOpenAppointments($val->id, $val->provid, $startDate, $endDate, $apptTypeID);
//print_r($res);
// if we got some appointments back strip any that started before now
if (array_key_exists('totalcount', $res) && $res['appointments'] > 0)
{
$tzStr = "America/Los_Angeles";
$tzObject = new DateTimeZone($tzStr);
$nowDT = new DateTime();
$nowDT->setTimezone($tzObject);
//print_r($nowDT);
//print_r("\n");
$appts = array();
for ($i = 0; $i < count($res['appointments']); $i++)
{
$apptDT = new DateTime($res['appointments'][$i]['date']." ".$res['appointments'][$i]['starttime'], $tzObject);
//print_r($apptDT);
//print_r("\n");
if ($nowDT < $apptDT)
$appts[] = $res['appointments'][$i];
}
}
if (count($appts) > 0)
foreach($appts as $data) {
$final_array[] = $data;
}
else
$res; // something went wrong. return error message
}
echo json_encode($final_array);
}
Header.php
<script>
var times = <?php getOpenAppts(); ?>;
console.log(times); //Will display properly
</script>
That is exactly how it should come back!
But.. When I run a console on the variable times (which is in the header making it a global variable. I get this.
It should give me the exact same list that the console.log gave me.
What I Have tried
I ran:
PARSE.json(times);
No effect...
I did in PHP:
json_encode(json_decode($appts),true);
No effect...
What part of this process is incorrect?
You are using time as a global variable.
Since the console.log right after the declaration prints everything fine, you are probably overriding its value somewhere after.
Avoid the most you can global variables, they're evil :)

How should I keep track of for loop index while using ajax to insert php data into object?

Ok so I'm pretty much completely new to using ajax and I'm also still in the process of understanding effective implementation of php. Been a bit stuck recently because I feel like I'm forgetting or overlooking a really easy solution.
Right now the following code gets a trove api query with the help of php and appends 100 of these queries for each item (I know it's inefficient but right now I have a reason for this method). I use 'getperson' to grab the name from the object's own data and then for each I assign the relevant value to all items.
I want to do the same for a img url, however I don't have a good enough understanding of ajax syntax or possibilities to properly pass the correct index (which matches the getname index) into the ajax function.
I feel like there was a much more efficient way to do this, but would prefer if anyone could help explain the steps I can take to iterate on my current model.
var compilelist = [];
var promises = [];
for (i=0; i < <?php echo $rowcount ?>; i++) {
var getname = <?php echo json_encode($followarray) ?>[i];
var getimg = <?php echo json_encode($imgarray) ?>[i];
var getitem = Geturl(getname, 100);
var promise = $.getJSON(getitem);
$.when(promise).done (function(data) {
var getperson = data.response.query;
$.each(data.response.zone[0].records.article, function(index, value) {
value.name = getperson;
// This is what I want to do but I can't grab from the loop cause 'each' is inside the ajax query
value.img = getimg;
compilelist.push(this);
});
});
promises.push(promise);
}
Well this question was dumb anyway but I quickly found a messy solution if anyone ever has a similar problem, I just stored img var in array parallel to name array so that I could select index from the name and use it to grab img from php:
var compilelist = [];
var promises = [];
for (i=0; i < <?php echo $rowcount ?>; i++) {
var getname = <?php echo json_encode($followarray) ?>[i];
var getitem = Geturl(getname, 100);
var promise = $.getJSON(getitem);
$.when(promise).done (function(data) {
var getperson = data.response.query;
var getindex = <?php echo json_encode($followarray) ?>.indexOf(getperson);
$.each(data.response.zone[0].records.article, function(index, value) {
value.name = getperson;
value.imgurl = <?php echo json_encode($imgarray) ?>[getindex];
compilelist.push(this);
});
});
promises.push(promise);
}

Getting wrong array length with jQuery and mysqli_num_rows()

I'm adding new data to database table with jQuery. Then i'm getting all content from that table. Problem is i'm getting wrong array size every odd time.If i refresh the page it's getting right array size but when i'm clicking button with jQuery function it's all messed up.
I figure out that
$n = mysqli_num_rows($result);
already return wrong num of rows
Here is the script:
$(document).ready(function(){
$.getallentries = function(){
$.getJSON("entries.php",{action : "getall"},function(data) {
var content_array = $.map(data, function(e) { return e;});
console.log(content_array.length); // to check array size
});
$.addstatic = function(){
$.post("entries.php",{action : "addstatic"});
};
};
$("#adds").on("click",function(){
$.addstatic();
$.getallentries();
});
$.getallentries();
And here is entries.php:
function getall(){
$link = db_connect();
$query= "SELECT * FROM jq";
$result = mysqli_query($link,$query, MYSQLI_STORE_RESULT);// Smart people said MYSQLI_STORE_RESULT should help but it didn't
$n = mysqli_num_rows($result);
for($i = 0 ;$i<$n;$i++)
{
$row=mysqli_fetch_assoc($result);
$b[]=$row;
}
echo json_encode(array($b));
}
function addstatic(){
$link = db_connect();
$statin_Entry = "Static Entry";
$query = "INSERT INTO jq (Entry) VALUES ('$statin_Entry')";
$result = mysqli_query($link,$query);
}
if(isset($_GET['action']) && !empty($_GET['action'])) {
$action = $_GET['action'];
switch($action) {
case 'getall' : getall(); break;
}
}
else {
if(isset($_POST['action']) && !empty($_POST['action'])) {
$action = $_POST['action'];
switch($action) {
case 'addstatic' : addstatic();break;
case 'removelast' : removelast();break;
// ...etc...
}
}
}
This is log of array length
So the problem again, same array size 143,146,151 and where is 156??
You should call $.getallentries in the callback function of the $.addstatic AJAX call, so that it runs after $.addstatic has finished updating the database.
$.addstatic = function(){
$.post("entries.php",{action : "addstatic"}, $.getallentries);
};
There's no need to use mysqli_num_rows before fetching the rows. You should use a loop like this:
while ($row = mysqli_fetch_assoc($result)) {
$b[] = $row;
}
Also, you're wrapping your array in another array. Just do:
echo json_encode($b);
The way you've written it, console.log(content_array.length) should always log 1, I don't understand how you're getting higher numbers. Are you sure you posted the actual code?
There's no point in use $.map, all it's doing is making a copy of the data array. Just use data itself.
And in your PHP, you don't need to test both isset() and !empty(), because empty() checks if the variable is set first.
You don't need to use MYSQLI_STORE_RESULT, it's the default for that option.

JSON occasionally undefined - how do I check?

So the JSON object I'm trying to reach sometimes does not exist.
Notice: Undefined index: movies in C:\xampp\htdocs\example\game.php
I'm reaching for it in the Steam API with this code on game.php:
$GLOBALS['gameTrailer'] = $game_json[$trimmed]['data']['movies'][0]['webm']['max'];
echo json_encode(array(
'gameTrailer' => $GLOBALS['gameTrailer'],
//+ other variables
));
I'm using AJAX to poke game.php like so on index.php:
function returnGame () {
$.ajax({
url: "game.php",
type: "post",
dataType: 'json',
success: function(data){
console.log(data);
$('#video').removeAttr('src');
///// Game name /////
$('#gameName').html(data.gameName);
/////////////////////
////// Append and load video /////
var videoSrc = data.gameTrailer;
var video_block = $('#video');
if (videoSrc !== null && videoSrc !== undefined) {
video_block.load();
document.querySelector('video').src = videoSrc;
} else {
$("#gameTrailer").find("#gameScreenshot").attr("src", data.gameScreenshot);
}
//////////////////////////////////
},
});
}
When movies is null, the AJAX function does nothing. The video does not change to blank, and the movie title is not updated to something new. When movies is not undefined, it works perfectly.
I can't seem to catch $GLOBALS['gameTrailer'] as undefined and re-iterate or replace with screenshot instead of movie though, not on game.php or index.php. I've tried things like if(empty()) {} and if($GLOBALS['gameTrailer'] == NULL) {}, but even though the error code on game.php tells me that it is undefined, it seems to act like it's not.
Any ideas will be much appreciated. Thanks.
EDIT: Full game.php code:
<?php
if(isset($_POST)) {
fetchGame();
}
function fetchGame() {
////////// ID-picker //////////
$f_contents = file("steam.txt");
$url = $f_contents[mt_rand(0, count($f_contents) - 1)];
$answer = explode('/',$url);
$gameID = $answer[4];
$trimmed = trim($gameID);
////////// Fetch game //////////
$json = file_get_contents('http://store.steampowered.com/api/appdetails?appids='.$trimmed);
$game_json = json_decode($json, true);
////////// Store variables //////////
$GLOBALS['gameName'] = $game_json[$trimmed]['data']['name'];
$GLOBALS['gameTrailer'] = $game_json[$trimmed]['data']['movies'][0]['webm']['max'];
$GLOBALS['gameScreenshot'] = $game_json[$trimmed]['data']['screenshots'][0]['path_full'];
$GLOBALS['gameImage'] = $game_json[$trimmed]['data']['header_image'];
$GLOBALS['gameId'] = $trimmed;
$GLOBALS['free'] = $game_json[$trimmed]['data']['is_free'];
$GLOBALS['price'] = $game_json[$trimmed]['data']['price_overview']['final'];
if(!isset($GLOBALS['price']) && ($GLOBALS['gameTrailer'])) {
fetchGame();
}
if ($GLOBALS['free'] === TRUE) {
$GLOBALS['final_price'] = "Free";
} elseif($GLOBALS['free'] === FALSE || $GLOBALS['final_price'] != NULL) {
$GLOBALS['final_price'] = $GLOBALS['price'];
} else {
$GLOBALS['final_price'] = "-";
}
}
////////// Return to AJAX (index.php) //////////
echo
json_encode(array(
'gameName' => $GLOBALS['gameName'],
'gameTrailer' => $GLOBALS['gameTrailer'],
'gameImage' => $GLOBALS['gameImage'],
'gameId' => $GLOBALS['gameId'],
'finalPrice' => $GLOBALS['final_price'],
'gameScreenshot' => $GLOBALS['gameScreenshot']
))
;
?>
It breaks on line 23 ($GLOBALS['gameTrailer'] = $game_json[$trimmed]['data']['movies'][0]['webm']['max']; as an undefined index)
Okay, so here is an example update to the code provided going by the comments made on the question.
Notes:
1) Don't use $GLOBALS unless you know what you're doing and why you're using it. In this case, it doesn't appear required. Updated to $game as it's holding game details.
2) You need to check whether something exists before you try and access it. So, the isset you previously had is useless as you've already accessed a non-existing array item. Removed that check.
3) Recursion is good, but again, you need to know what you're doing, and why you're using it. I suspect this was unintentional anyway but was removed as part of 2.
4) You state that having a video (and price by the looks of it) is a requirement for this function, so you should check that up front. Note that the first thing after getting the ID, is to see if there's a video. Why do work just to find we can't use it?
5) You want to try a different game if there is no video, so we wrap the check and assignments in a loop, and continue if the video isn't there. Note that you might want to restrict this to n number of tries otherwise you might end up waiting ages for your script to find a game that has a video, but that's for you to decide.
<?php
if(isset($_POST)) {
fetchGame();
}
function fetchGame() {
$gameFound = false;
while(!$gameFound) {
////////// ID-picker //////////
$f_contents = file("steam.txt");
$url = $f_contents[mt_rand(0, count($f_contents) - 1)];
$answer = explode('/',$url);
$gameID = $answer[4];
$trimmed = trim($gameID);
////////// Fetch game //////////
$json = file_get_contents('http://store.steampowered.com/api/appdetails?appids='.$trimmed);
$game_json = json_decode($json, true);
if(!isset($game_json[$trimmed]['data']['movies'][0]['webm']['max']) || !isset($game_json[$trimmed]['data']['price_overview']['final'])) {
continue;
}
$gameFound = true;
////////// Store variables //////////
$game['gameName'] = $game_json[$trimmed]['data']['name'];
$game['gameTrailer'] = $game_json[$trimmed]['data']['movies'][0]['webm']['max'];
$game['gameScreenshot'] = $game_json[$trimmed]['data']['screenshots'][0]['path_full'];
$game['gameImage'] = $game_json[$trimmed]['data']['header_image'];
$game['gameId'] = $trimmed;
$game['free'] = $game_json[$trimmed]['data']['is_free'];
$game['price'] = $game_json[$trimmed]['data']['price_overview']['final'];
if ($game['free'] === TRUE) {
$game['final_price'] = "Free";
} elseif($game['free'] === FALSE || $game['final_price'] != NULL) {
$game['final_price'] = $game['price'];
} else {
$game['final_price'] = "-";
}
}
}
////////// Return to AJAX (index.php) //////////
echo
json_encode(array(
'gameName' => $game['gameName'],
'gameTrailer' => $game['gameTrailer'],
'gameImage' => $game['gameImage'],
'gameId' => $game['gameId'],
'finalPrice' => $game['final_price'],
'gameScreenshot' => $game['gameScreenshot']
));
?>
Note this code is untested, but I hope gives you an idea of how to proceed.

convert from json to array

I am trying to get data to display in a table. I don't know what I am doing wrong, but when I get the data from my page it is an array of single characters. I could parse this myself but would prefer to know what I am doing wrong.
I have this php to get the data:
function BuildViewerCombo($autocomplete) {
$wholeNumberCombo = array();
$dbhandle = DB_Connect();
$result = QueryForward($dbhandle, SQL_WholeNumbersPartial($autocomplete));
while($wholeNumber = sqlsrv_fetch_array($result))
{
$wholeNumberCombo[] = array($wholeNumber['DocumentNbr'] => 'Number', $wholeNumber['DocumentRevision'] => 'Revision');
}
//close the connection
sqlsrv_close($dbhandle);
return $wholeNumberCombo;
}
Which is called from this page
<?PHP
include "Scripts/DB_Functions.php5" ;
include "Scripts/SQL_Viewer.php5" ;
$wholeNumber = $_GET['wholeNumber'];
echo json_encode(BuildViewerCombo($wholeNumber));
?>
Which gets loaded from this function
function toggleDropdown()
{
var wholeNumberData
var wholeNumber = document.getElementById('WholeNumber').value;
if (wholeNumber != '') {
wholeNumberData = GetData('wholeNumber', wholeNumber);
var table = document.getElementById("wholeNumberDropdown");
alert ('WN = ' + wholeNumberData.length);
alert (wholeNumberData);
for (var i in wholeNumberData) {
alert(wholeNumberData[i]);
}
}
else {
alert("Please enter a whole number.");
}
}
By calling this function:
function GetData(getType, param) {
var http = new XMLHttpRequest();
http.open("GET", 'ViewerWholeNumbers.php?wholeNumber=' + param, false);
http.setRequestHeader("Content-type","application/json");
http.onload = function() {
}
http.send('wholeNumber=' + param);
return http.responseText;
}
The data that gets returned is:
[{"SS3999":"Number","A":"Revision"},{"SS3999":"Number","11":"Revision"},
{"SS3999":"Number","11":"Revision"},{"SS3999":"Number","11":"Revision"},
{"SS3999":"Number","":"Revision"},{"SS3999":"Number","11":"Revision"},
{"SS3999":"Number","":"Revision"},{"SS3999":"Number","11":"Revision"},
{"SS3999":"Number","11":"Revision"},{"SS3999":"Number","A":"Revision"},
{"SS3999":"Number","11":"Revision"},{"SS3999":"Number","A":"Revision"},
{"SS3999":"Number","11":"Revision"},{"SS3999":"Number","A":"Revision"},
{"SS3999":"Number","":"Revision"}]
But alert ('WN = ' + wholeNumberData.length); returns 546 and when I try to loop through the array I get a single character for each element instead of the values.
First off, your associative array is flipped. You need to change
array($wholeNumber['DocumentNbr'] => 'Number', $wholeNumber['DocumentRevision'] => 'Revision');
to
array('Number' => $wholeNumber['DocumentNbr'], 'Revision' => $wholeNumber['DocumentRevision']);
You need that in order to access the elements of the JSON. Then, in your loop, you would use wholeNumberData[i].Number to get the number and wholeNumberData[i].Revision to get the revision.
Update:
As #jeroen pointed out, you need JSON.parse() to convert the return string to JSON. In your GetData function replace your return with this:
return JSON.parse(http.responseText);

Categories

Resources