Develop Reference

Web Scraping with Javascript?

I'm having a hard time figuring out how to scrape this webpage to get this wedding list into my onepager. It doesn't seem complicated at first but when I get into the code, I just can't get any results.
I've tried ygrab.js, which was fairly simple and got me somewhere but then I can't seem to scrape the images and it only prints the output in the console (not much documentation to go on).
$(function() {
var $listResult = $('#list-result');
var kado = [];
var data = [
{
url: 'https://www.kadolog.com/fr/list/liste-de-mariage-laura-julien',
selector: '.kado-not-full',
loop: true,
result: [{
name: 'photo',
find: '.views-field-field-photo',
grab: {
by: 'attr',
value: 'src'
}
},
{
name: 'title',
find: '.views-field-title .field-content',
grab: {
by: 'text',
value: ''
}
},
{
name: 'description',
find: '.views-field-body .field-content',
grab: {
by: 'text',
value: ''
}
},
{
name: 'price',
find: '.price',
grab: {
by: 'text',
value: ''
}
},
{
name: 'remaining',
find: '.topinfo',
grab: {
by: 'text',
value: ''
}
},
{
name: 'link',
find: '.views-field-nothing .field-content .btn',
grab: {
by: 'attr',
value: 'href'
}
},
],
},
];
ygrab(data, function(result){
console.log(JSON.stringify(result, null, 2)); //photos = undefined
});
Then there's Node.js with Request and Cheerio (and I tried Crawler too), but I have no idea how node works.
var request = require("request");
This gives me an error in the console saying require is not defined. Fair enough, I added require.js to the scripts in my page. I got another error ("Uncaught Error: Mismatched anonymous define() module: ...").
My question is this: Is there a simple Javascript way (possibly without involving node?), to scrape the wedding list I'm trying to get? Or maybe a tutorial that resembles what I'm trying to do step by step ?
I'd be truly grateful for any help or advice.

i think your only issue is the img selector.
Change
{
name: 'photo',
find: '.views-field-field-photo',
grab: {
by: 'attr',
value: 'src'
}
},
To this
{
name: 'photo',
find: '.views-field-field-photo .field-content img',
grab: {
by: 'attr',
value: 'src'
}
},
I actually can't test this right now, but it should be working!!

Node.js is a seperate application that executes javascript independent of a web page.
require is Node's way of importing packages, and isn't defined by the browser, require.js is a javascript library for requiring packages, but it doesn't work the same way as Node's require function.
To use request and cheerio, you'd need to install Node.js from here, then install request and cheerio with the following commands:
npm install request --save
npm install cheerio --save
Then any code you write with Node.js in that directory will have access to the modules.
Here's a tutorial to web scraping in Node.js with cheerio.

AngularJs crashes when trying to access data in directive

So I'm new to Angular and trying to do an app. My server is very basic and written in go. This is my angular code-
(function() {
var app = angular.module('dashboard', []);
app.controller("JobsController", function() {
this.currentJob = 0;
this.jobs = jobs;
this.setCurrentJob = function(index) {
this.currentJob = index;
};
});
var jobs = [
{
'id': 1,
'requester': 'Prakash Sanker',
'description': 'I want you to give me the entire world'
},
{
'id': 2,
'requester': 'MJ Watson',
'description': 'Please give me Spiderman, preferably upside down...but Im not fussy'
},
{ 'id': 3,
'requester': 'Josh Lyman',
'description': 'Matthew Santos as president...or Jed Bartlet..but please not anyone Republican'
},
{
'id': 4,
'requester': 'Barry Allen',
'description': 'A frictionless suit that wont burst into flames when I run at a zillion miles an hour...its for a friend'
},
{
'id': 5,
'requeter': 'A. Bambaata',
'description': 'Boombox, prime condition, from the 80s. Go back in time if you have to'
}
];
})();
This is my index.html -
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js"></script>
<script type="text/javascript" src="/dashboard.js"></script>
<body>
<div class="dashboard">
<div ng-controller="JobsController as jobsCtrl">
<div class="jobs">
{{jobsCtrl.jobs[0]}}
</div>
</div>
</div>
</body>
This causes this error on my server
2015/07/29 12:18:02 http: panic serving [::1]:56857: runtime error: invalid memory address or nil pointer dereference
goroutine 18 [running]:
net/http.func·009()
/usr/local/go/src/pkg/net/http/server.go:1093 +0x9e
runtime.panic(0x20ed40, 0x4efaed)
/usr/local/go/src/pkg/runtime/panic.c:248 +0xda
html/template.(*Template).escape(0x0, 0x0, 0x0)
/usr/local/go/src/pkg/html/template/template.go:52 +0x30
Why is this happening?

You cannot use angular's default start {{ and end }} brackets in a go template because the go server interprets your angular as go template arguments and pipelines.
"Arguments" and "pipelines" are evaluations of data
To solve this you can make use of angular's $interpolate in your config to change the delimiters that define angular code:
var app = angular.module('dashboard', []);
app.config(function($interpolateProvider) {
$interpolateProvider.startSymbol('[{');
$interpolateProvider.endSymbol('}]');
});
Alternatively if your pages are static you can just use Go's built in file server to not cause confusion with templates:
http://golang.org/pkg/net/http/#example_FileServer

Nested attributes with Angular.js

I have been racking my brain and google all morning trying to figure this out but I have come to the conclusion that I need to ask the experts! I am trying to do nested attributes with Sinatra and Angular, don't worry about the Sinatra side of things I am just trying to get the data to the server side in the correct manner at the moment. Please see the code below for an explanation of
My Input:
<input type="text" placeholder="{{item.placeholder}}" ng-model="question.possible_answer_attributes[$index][title]" class="form-control" />
My model object:
$scope.question = new Question({
poll_id: parseInt($routeParams.id),
title: '',
kind: 'open',
possible_answer_attributes: [] // I believe the issue may lie here
});
My factory:
.factory('Question', function($resource) {
return $resource('/api/questions/:id', { id: '#id' }, {
'update' : { method: 'PUT' },
'get_by_poll' : { method: 'GET', url: '/api/questions_by_poll/:id', isArray: true }
});
})
My object at time of running save function:
{"poll_id"=>1, "title"=>"123123", "kind"=>"multiple", "possible_answer_attributes"=>[{"undefined"=>"412312"}, {"undefined"=>"1234124"}, {"undefined"=>"234235234"}]}
I do not understand why my "possible_answer_attributes" keys are coming through as "undefined". It may be something very simple that I have missed, but any feedback would be great!
Thanks in advance!

In order to address the title property, you would need to use a string to index into the object:
ng-model="question.possible_answer_attributes[$index]['title']"
This should hold as long as possible_answer_attributes array looks like:
[{ title: 'my title' }]

How to make Automated Dynamic Breadcrumbs with AngularJS + Angular UI Router

One key component to web applications is breadcrumbs/navigation. With Angular UI Router, it would make sense to put the breadcrumb metadata with the individual states, rather than in your controllers. Manually creating the breadcrumbs object for each controller where it's needed is a straight-forward task, but it's also a very messy one.
I have seen some solutions for automated Breadcrumbs with Angular, but to be honest, they are rather primitive. Some states, like dialog boxes or side panels should not update the breadcrumbs, but with current addons to angular, there is no way to express that.
Another problem is that titles of breadcrumbs are not static. For example, if you go to a User Detail page, the breadcrumb title should probably be the user's Full Name, and not a generic "User Detail".
The last problem that needs to be solved is using all of the correct state parameter values for parent links. For example, if you're looking at a User detail page from a Company, obviously you'll want to know that the parent state requires a :companyId.
Are there any addons to angular that provide this level of breadcrumbs support? If not, what is the best way to go about it? I don't want to clutter up my controllers - I will have a lot of them - and I want to make it as automated and painless as possible.
Thanks!

I did solve this myself awhile back, because nothing was available. I decided to not use the data object, because we don't actually want our breadcrumb titles to be inherited by children. Sometimes there are modal dialogs and right panels that slide in that are technically "children views", but they shouldn't affect the breadcrumb. By using a breadcrumb object instead, we can avoid the automatic inheritance.
For the actual title property, I am using $interpolate. We can combine our breadcrumb data with the resolve scope without having to do resolves in a different place. In all of the cases I had, I just wanted to use the resolve scope anyway, so this works very well.
My solution also handles i18n too.
$stateProvider
.state('courses', {
url: '/courses',
template: Templates.viewsContainer(),
controller: function(Translation) {
Translation.load('courses');
},
breadcrumb: {
title: 'COURSES.TITLE'
}
})
.state('courses.list', {
url: "/list",
templateUrl: 'app/courses/courses.list.html',
resolve: {
coursesData: function(Model) {
return Model.getAll('/courses');
}
},
controller: 'CoursesController'
})
// this child is just a slide-out view to add/edit the selected course.
// It should not add to the breadcrumb - it's technically the same screen.
.state('courses.list.edit', {
url: "/:courseId/edit",
templateUrl: 'app/courses/courses.list.edit.html',
resolve: {
course: function(Model, $stateParams) {
return Model.getOne("/courses", $stateParams.courseId);
}
},
controller: 'CourseFormController'
})
// this is a brand new screen, so it should change the breadcrumb
.state('courses.detail', {
url: '/:courseId',
templateUrl: 'app/courses/courses.detail.html',
controller: 'CourseDetailController',
resolve: {
course: function(Model, $stateParams) {
return Model.getOne('/courses', $stateParams.courseId);
}
},
breadcrumb: {
title: '{{course.name}}'
}
})
// lots more screens.
I didn't want to tie the breadcrumbs to a directive, because I thought there might be multiple ways of showing the breadcrumb visually in my application. So, I put it into a service:
.factory("Breadcrumbs", function($state, $translate, $interpolate) {
var list = [], title;
function getProperty(object, path) {
function index(obj, i) {
return obj[i];
}
return path.split('.').reduce(index, object);
}
function addBreadcrumb(title, state) {
list.push({
title: title,
state: state
});
}
function generateBreadcrumbs(state) {
if(angular.isDefined(state.parent)) {
generateBreadcrumbs(state.parent);
}
if(angular.isDefined(state.breadcrumb)) {
if(angular.isDefined(state.breadcrumb.title)) {
addBreadcrumb($interpolate(state.breadcrumb.title)(state.locals.globals), state.name);
}
}
}
function appendTitle(translation, index) {
var title = translation;
if(index < list.length - 1) {
title += ' > ';
}
return title;
}
function generateTitle() {
title = '';
angular.forEach(list, function(breadcrumb, index) {
$translate(breadcrumb.title).then(
function(translation) {
title += appendTitle(translation, index);
}, function(translation) {
title += appendTitle(translation, index);
}
);
});
}
return {
generate: function() {
list = [];
generateBreadcrumbs($state.$current);
generateTitle();
},
title: function() {
return title;
},
list: function() {
return list;
}
};
})
The actual breadcrumb directive then becomes very simple:
.directive("breadcrumbs", function() {
return {
restrict: 'E',
replace: true,
priority: 100,
templateUrl: 'common/directives/breadcrumbs/breadcrumbs.html'
};
});
And the template:
<h2 translate-cloak>
<ul class="breadcrumbs">
<li ng-repeat="breadcrumb in Breadcrumbs.list()">
<a ng-if="breadcrumb.state && !$last" ui-sref="{{breadcrumb.state}}">{{breadcrumb.title | translate}}</a>
<span class="active" ng-show="$last">{{breadcrumb.title | translate}}</span>
<span ng-hide="$last" class="divider"></span>
</li>
</ul>
</h2>
From the screenshot here, you can see it works perfectly in both the navigation:
As well as the html <title> tag:
PS to Angular UI Team: Please add something like this out of the box!

I'd like to share my solution to this. It has the advantage of not requiring anything to be injected into your controllers, and supports named breadcrumb labels, as well as using resolve: functions to name your breadcrumbs.
Example state config:
$stateProvider
.state('home', {
url: '/',
...
data: {
displayName: 'Home'
}
})
.state('home.usersList', {
url: 'users/',
...
data: {
displayName: 'Users'
}
})
.state('home.userList.detail', {
url: ':id',
...
data: {
displayName: '{{ user.name | uppercase }}'
}
resolve: {
user : function($stateParams, userService) {
return userService.getUser($stateParams.id);
}
}
})
Then you need to specify the location of the breadcrumb label (displayname) in an attribute on the directive:
<ui-breadcrumbs displayname-property="data.displayName"></ui-breadcrumbs>
In this way, the directive will know to look at the value of $state.$current.data.displayName to find the text to use.
$interpolate-able breadcrumb names
Notice that in the last state (home.userList.detail), the displayName uses the usual Angular interpolation syntax {{ value }}. This allows you to reference any values defined in the resolve object in the state config. Typically this would be used to get data from the server, as in the example above of the user name. Note that, since this is just a regular Angular string, you can include any type of valid Angular expression in the displayName field - as in the above example where we are applying a filter to it.
Demo
Here is a working demo on Plunker: http://plnkr.co/edit/bBgdxgB91Z6323HLWCzF?p=preview
Code
I thought it was a bit much to put all the code here, so here it is on GitHub: https://github.com/michaelbromley/angularUtils/tree/master/src/directives/uiBreadcrumbs

I made a Angular module which generate a breadcrumb based on ui-router's states. All the features you speak about are included (I recently add the possibility to ignore a state in the breadcrumb while reading this post :-) ) :
Here is the github repo
It allows dynamic labels interpolated against the controller scope (the "deepest" in case of nested/multiple views).
The chain of states is customizable by state options (See API reference)
The module comes with pre-defined templates and allows user-defined templates.

I do not believe there is built in functionality, but all the tools are there for you, take a look at the LocationProvider. You could simply have navigation elements use this and whatever else you want to know just inject it.
Documentation

After digging deep into the internals of ui-router I understood how I could create a breadcrumb using resolved resources.
Here is a plunker to my directive.
NOTE: I couldn't get this code to work properly within the plunker, but the directive works in my project. routes.js is provided merely for example of how to you can set titles for your breadcrumbs.

Thanks for the solution provided by #egervari. For those who need add some $stateParams properties into custom data of breadcrumbs. I've extended the syntax {:id} for the value of key 'title'.
.state('courses.detail', {
url: '/:courseId',
templateUrl: 'app/courses/courses.detail.html',
controller: 'CourseDetailController',
resolve: {
course: function(Model, $stateParams) {
return Model.getOne('/courses', $stateParams.courseId);
}
},
breadcrumb: {
title: 'course {:courseId}'
}
})
Here is an Plunker example. FYI.

Twitter's Bootstrap typeahead setup

I'm using the official examples from Twitter.
The main problem, I probably don't know how to use the Hogan monster. The JS side:
$("#search_name").typeahead({
name: 'name',
remote: {
url: '/entities/search_autocomplete.json?query=%QUERY',
template: '<p><strong>{{id}}</strong> – {{name}}</p>',
engine: Hogan
}
});
The server is returning the data in JSON, the structure is:
[{\"id\":1234,\"name\":\"Blah blah...\",\"tokens\":[\"blah...\",\"blah\"]}]

Just took this code from one of our projects, should help you understand the necessary markup of converting external JSON arrays and outputting in a custom autocomplete prompt:
$('input').typeahead({
header: 'Your Events',
template: [
'<img class="ta-thumb" src="https://graph.facebook.com/{{id}}/picture?type=square" />',
'<p class="ta-h1">{{name}}</p>',
'<p class="ta-p">{{start_time}}</p>'
].join(''),
limit: 3,
remote: {
url: 'https://graph.facebook.com/me/events?access_token=' + access_token,
filter: function(parsedResponse) {
var dataset = [];
for(i = 0; i < parsedResponse.data.length; i++) {
dataset.push({
name: parsedResponse.data[i].name,
start_time: parsedResponse.data[i].start_time,
id: parsedResponse.data[i].id,
value: parsedResponse.data[i].name,
tokens: [parsedResponse.data[i].id, parsedResponse.data[i].name]
});
}
return dataset;
},
},
engine: Hogan
});
You need to download the Hogan.js template compiler and include it in your markup (e.g. using it as an external script or via a module loader like Require.js). This will then set the Hogan variable.
I'd also recommend looking at that Graph API call to understand the array conversion better.
Hopefully this helps :)