Develop Reference

What's the best method to EXTRACT product names given a list of SKU numbers from a website?

I have a problem.
I have a list of SKU numbers (hundreds) that I'm trying to match with the title of the product that it belongs to. I have thought of a few ways to accomplish this, but I feel like I'm missing something... I'm hoping someone here has a quick and efficient idea to help me get this done.
The products come from Aidan Gray.
Attempt #1 (Batch Program Method) - FAIL:
After searching for a SKU in Aidan Gray, the website returns a URL that looks like below:
http://www.aidangrayhome.com/catalogsearch/result/?q=SKUNUMBER
... with "SKUNUMBER" obviously being a SKU.
The first result of the webpage is almost always the product.
To click the first result (through the address bar) the following can be entered (if Javascript is enabled through the address bar):
javascript:{document.getElementsByClassName("product-image")[0].click;}
I wanted to create a .bat file through Command Prompt and execute the following command:
firefox http://www.aidangrayhome.com/catalogsearch/result/?q=SKUNUMBER javascript:{document.getElementsByClassName("product-image")[0].click;}
... but Firefox doesn't seem to allow these two commands to execute in the same tab.
If that worked, I was going to go to http://tools.buzzstream.com/meta-tag-extractor, paste the resulting links to get the titles of the pages, and export the data to CSV format, and copy over the data I wanted.
Unfortunately, I am unable to open both the webpage and the Javascript in the same tab through a batch program.
Attempt #2 (I'm Feeling Lucky Method):
I was going to use Google's &btnI URL suffix to automatically redirect to the first result.
http://www.google.com/search?btnI&q=site:aidangrayhome.com+SKUNUMBER
After opening all the links in tabs, I was going to use a Firefox add-on called "Send Tab URLs" to copy the names of the tabs (which contain the product names) to the clipboard.
The problem is that most of the results were simply not lucky enough...
If anybody has an idea or tip to get this accomplished, I'd be very grateful.

I recommend using JScript for this. It's easy to include as hybrid code in a batch script, its structure and syntax is familiar to anyone comfortable with JavaScript, and you can use it to fetch web pages via XMLHTTPRequest (a.k.a. Ajax by the less-informed) and build a DOM object from the .responseText using an htmlfile COM object.
Anyway, challenge: accepted. Save this with a .bat extension. It'll look for a text file containing SKUs, one per line, and fetch and scrape the search page for each, writing info from the first anchor element with a .className of "product-image" to a CSV file.
#if (#CodeSection == #Batch) #then
#echo off
setlocal
set "skufile=sku.txt"
set "outfile=output.csv"
set "URL=http://www.aidangrayhome.com/catalogsearch/result/?q="
rem // invoke JScript portion
cscript /nologo /e:jscript "%~f0" "%skufile%" "%outfile%" "%URL%"
echo Done.
rem // end main runtime
goto :EOF
#end // end batch / begin JScript chimera
var fso = WSH.CreateObject('scripting.filesystemobject'),
skufile = fso.OpenTextFile(WSH.Arguments(0), 1),
skus = skufile.ReadAll().split(/\r?\n/),
outfile = fso.CreateTextFile(WSH.Arguments(1), true),
URL = WSH.Arguments(2);
skufile.Close();
String.prototype.trim = function() { return this.replace(/^\s+|\s+$/g, ''); }
// returns a DOM root object
function fetch(url) {
var XHR = WSH.CreateObject("Microsoft.XMLHTTP"),
DOM = WSH.CreateObject('htmlfile');
WSH.StdErr.Write('fetching ' + url);
XHR.open("GET",url,true);
XHR.setRequestHeader('User-Agent','XMLHTTP/1.0');
XHR.send('');
while (XHR.readyState!=4) {WSH.Sleep(25)};
DOM.write(XHR.responseText);
return DOM;
}
function out(what) {
WSH.StdErr.Write(new Array(79).join(String.fromCharCode(8)));
WSH.Echo(what);
outfile.WriteLine(what);
}
WSH.Echo('Writing to ' + WSH.Arguments(1) + '...')
out('sku,product,URL');
for (var i=0; i<skus.length; i++) {
if (!skus[i]) continue;
var DOM = fetch(URL + skus[i]),
anchors = DOM.getElementsByTagName('a');
for (var j=0; j<anchors.length; j++) {
if (/\bproduct-image\b/i.test(anchors[j].className)) {
out(skus[i]+',"' + anchors[j].title.trim() + '","' + anchors[j].href + '"');
break;
}
}
}
outfile.Close();
Too bad the htmlfile COM object doesn't support getElementsByClassName. :/ But this seems to work well enough in my testing.

read value from txt file in javascript

I have a simple html file in which there's javascript code referring to google charts.
The code I use is this (I'll show the important part):
function drawChart(){
var data = google.visualization
.arrayToDataTable([ ['Label', 'Value'],['Temp', 22.75],]);
// etc...
}
I use a bash command (sed) to replace that 22.75 value with a new one from the last line of a .txt file. However, this throws some errors which I haven't been able to neither correct nor ever identify.
So is there any javascript code that takes that file, extracts the last value and simply displays it on the right place of the code?
UPDATE:
Sorry for the lack of info in this question, I really appreciate all the people that took the time on reading my question. I'll try to fill with more information in the next minutes.
I am able to extract the last line of the .txt file, extract the value on the right part of the '-' symbol and store it in a variable. Then that value is taken to update the html file with a sed command. The error comes when the value is updated but with no value. I guess that happends due to a failed record of temperature in the txt file, then the extracted value is a null. Finally is the html fiel with javascrit code happens to be like this:
(...)['Temp', ],]);
Then the updater can't update the value since due to the way that sed command is written I guess there's no way that it can detect a no-number-value in there. So the html remains without a value all the time.
TXT File structure:
(...)
20:25:03-23.312
20:26:02-23.312
20:27:03-23.375
20:28:03-23.375
20:29:02-23.375
20:30:02-23.312
Bash script:
# (...code...)
lastRecord=`cat /home/pi/scripts/temp_control/logs/"$today".log | awk 'END{print}'`
function rightNow {
lastTemp=`echo $lastRecord | cut -d'-' -f2`
timeOfTemp=`echo $lastRecord | cut -d'-' -f1` # Not used yet
#Command used to update
sed -i "s/['Temp', [0-9]\{1,2\}.[0-9]\{1,3\}]/$lastTemp]/" /var/www/rightnow.html
}
rightNow

You cud get your file just like any other ajax request.
Using javascript
var request = new XMLHttpRequest();
request.open('GET', 'public_path_to_file.txt', false);
request.send();
var textFileContent = request.responseText
Using jQuery
var textFileContent;
$.get('public_path_to_file.txt', function(data) {
textFileContent = data;
});
Whats left is to get the right part from textFileContent. Dependent of the structure of the file we can do this in different ways. Without an example file you are on your own but here is some examples.
If you need the last line
var lines = textFileContent.split("\n");
var lastLine = lines[lines.length - 1];
If you need to use regex
var regex = //* some regex to get your content*//gm;
var result = regex.exec(textFileContent);
// result should now the content who matches your regex

First I'll assume that you ultimately want to read a local file with your browser and your current workflow is something like a local 'bash-script' that
first updates/modifies an inline piece of javascript (inside a locally stored html
file) with the last occurring value retrieved from a local txt-file (using sed)
opens the (just modified html-) file (via commandline) inside a common browser.
Then I assume the sed-route once worked but now doesn't work anymore (probably because the html file has changed?) and now you'd like the inline javascript (in the html file) to fetch that value from the textfile itself and subsequently use it (thus without the need for the 'bash-script'/sed solution.
Thus, the answer (based on above assumptions) to your final question: 'is there any javascript code that takes that file, extracts the last value and simply displays it on the right place of the code?', depends on your final requirement:
are you ok with a file-input where you select the text-file every time you view the html-file?
If your answer is YES, then, (depending on the browser you use) you can read a local file (and work your magic on it's contents).
In modern browsers, using the File API (which was added to the DOM in HTML5) it's now possible for web content to ask the user to select local files, then read the contents of those files.
For example, using FireFox's 'FileReader' you could do:
html:
<input type="file" id="fileinput" multiple />
javascript:
function readAllFiles(evt){
var files = evt.target.files, i = 0, r, f;
if(files){
for(; f = files[i++]; ){
r = new FileReader();
r.onload = (function(f){
return function(e){
alert(e.target.result);
};
})(f);
r.readAsText(f);
}
} else {
alert("Error loading files");
}
}
document.getElementById('fileinput')
.addEventListener('change', readAllFiles, false);
Note that for accessing local files in Chrome you must start Chrome with this switch: chrome --disable-web-security
However,
if the answer is NO (so you want to specify the file, and more importantly it's path, inside the 'code', so you don't have to select the text-file every time your local app runs) then you (usually) can't (because you can't get/set the path, thank the great maker)...
Unless you choose a specific older/unpatched browser (specifically for this task) where you know of a (hack) way to do this anyway (like the IE xml vulnerability or the XMLHTTP vulnerability or etc... you get the picture..).
Some alternative solutions (that don't require you to select the correct textfile over and over again)
Setup a fullblown web (LAMP) server (to use the XMLHttpRequest way as used in aross answer, but this might feel like shooting at a mosquito with a cannon..)
Explore different script languages (but effectively still do the same as your now broken sed-solution)
Combine 1 and 2, choosing from php (the latest version has a small webserver included, you might start/stop it when needed (even in the bash-script workflow) OR using node.js (which is 'javascript' and where you can program/control a small task-specific server in just a couple of lines).
Hope this helps!
Update:
Based on your updated question, comments and request for recommendation, I'd like to suggest to use PHP to dynamically fetch the value from your log txt file and have it generate your html code with inline javascript on the fly (every time you visit the page).
The browser will never see the php code, only what php inserted to your page (in this example the last found value or 0).
You'd rename the rightnow.html file to rightnow.php and modify it (something like) this:
<!DOCTYPE html>
<html><head>
<!-- your header code -->
<script type="text/javascript">
//parts of your javascript
<?php // start php script
$logFile= '/pathToYour/logFile.log'; // <-Modify
if( $fp= #fopen($logFile, "r") ){ // if logfile succesfully opened,
fseek($fp, -30, SEEK_END); // set pointer 30 chars from EOF
$val= array_pop(explode("-", rtrim(fread($fp, 30)))); //get last value
fclose($fp); // close file
}
?> // end php script
function drawChart(){
var data=google.visualization
.arrayToDataTable([ ['Label', 'Value'],
['Temp', <?php echo $val? $val : "0"; ?>],
]); // ^php above inserts value or 0
// etc...
}
//parts of your javascript
</script>
</head><body>
<!-- your body code -->
</body></html>
Note that fopen in combination with setting the filepointer via fseek and sequentially fread-ing from the pointer to EOF does not load the complete logfile (60min * 24hour=1440 lines of 16 bytes=22.5kB at the end of the day) into memory (good for this purpose), but only the last 30 chars (as in this example).
The variable to your logfile and path must still be modified to your situation (I don't know the format of your $today variable).
Depending on your further needs you might want to perform some extra checks/logic on the array of values that explode returns (instead of popping the last value). Or what about modifying the html a little so you could also include the last temperature's time reading, etc. (But this tested piece of code should get you started and explains the procedure of going the php way).
Update:
Since you have chosen to place the last known value of your logfile as in textfile placed inside your public www-root (with a bash script I assume, every minute of the day?), you can now indeed go the 'ajax' way, as answered by aross!
However I want to hint that the code/solutions in all current answers here could be mixed (since you now also have ajax working): instead of ajax-ing (loading) a txt file, you could have php fetch and send this value to the browser on-the-fly/on-demand!
So, instead of requesting http://url_to_my_rpi/file_to_download.txt, you could request http://url_to_my_rpi/read_last_temperature.PHP which should fetch the last known value out of the log-file (set proper security/access) and send it to the browser (set proper headers), just like your text-file did. You wouldn't have to change anything in the html/javascript except the url you request.
The advantage would be (depending on how your current bash-scripts works) that your PI now only does this 'work' (of getting the last value of your logfile) when you are viewing your monitor-page. And that you are not writing that file in your www-root every minute of every day (as I suspect).

The solution achieved, finally, was like this:
I did it with a jQuery statement and reusing the javascript code of Google Charts.
First I added javascript and jQuery tags in the html file:
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script type='text/javascript' src='https://www.google.com/jsapi'></script>
Then I merged jquery code and javascript code that I had in one script:
<script type='text/javascript'>
// Needed this var so that I could use it in other places of the code
var t;
jQuery.get('http://url_to_my_rpi/file_to_download.txt',function(data){
console.log(data)
t=data;
},'text');
google.load('visualization', '1', {packages:['gauge']});
google.setOnLoadCallback(drawChart);
function drawChart() {
t=eval(t);
var data = google.visualization.arrayToDataTable([
['Label', 'Value'],
['Temp', t],]);
// (... more javascript with Google Charts options, display parameters..)
</script>
Finally, and even if it's not listed as the main question, be sure to enable *mod_headers* on your apache and add Header set to apache2.conf file (In my case: /etc/apache2/apache2.conf)
1) Enable the module:
a2enmod headers
2) Add the line on your configuration file
Header set Access-Control-Allow-Origin "*"
3) Restart apache 2
4) In case the 3 steps above didn't work, follow the instrcutions by entering in this website or reinstall apache2.

Saving without dialog window

I'm trying to write a script that will automate a bunch of stuff for Photoshop CS5. Part of this involves saving a bunch of files. Is there a way to save a file in a way that doesn't open up a dialog window? I've been looking over the JavaScript Tools Guide, but I didn't see a way to do this. This suggested I used an action to deal with it but I'd really prefer not to do that.
EDIT: specifically I want to save the files as crytiff format but I'd just like to know how to save a file with whatever extension I want

The following saves the active document as PNG. You can change the type to save it as.
// reference open doc
var doc = app.activeDocument;
// set save options
var opts = new ExportOptionsSaveForWeb();
opts.PNG8 = false;
opts.transparency = true;
opts.interlaced = false;
opts.quality = 100;
opts.includeProfile = false;
opts.format = SaveDocumentType.PNG; // Document Type
// save png file in same folder as open doc
activeDocument.exportDocument(doc.path, ExportType.SAVEFORWEB, opts);

Try using Document.saveAs(). But, like El Cas said, you still have to pass in some kind of SaveOptions object. You don't necessarily have to specify all the options if you don't want. You can just use the generic object like this:
app.activeDocument.saveAs(new File(doc.path + "/myDocument"), TiffSaveOptions);
// or BMPSaveOptions or GIFSaveOptions or JPEGSaveOptions...
Here's a much more complete Photoshop CS5 Javascript Reference

Open:
Windows > Actions
You will find Toggle Dialog On/Off check box before every action. Turn it off.

NetUtil.asyncCopy from one file to append to another in Firefox extension

I'm trying to use NetUtil.asyncCopy to append data from one file to the end of another file from a Firefox extension. I have based this code upon a number of examples at https://developer.mozilla.org/en-US/docs/Code_snippets/File_I_O, particularly the 'Copy a stream to a file' example. Given what it says on that page, my code below:
Creates nsIFile objects for the file to copy from and file to append to and initialises these objects with the correct paths.
Creates an output stream to the output file.
Runs the NetUtil.asyncCopy function to copy between the file (which, I believe, behaves as an nsIInputStream) and the output stream.
I run this code as append_text_from_file("~/CopyFrom.txt", "~/AppendTo.txt");, but nothing gets copied across. The Appending Text and After ostream dumps appear on the console, but not the Done or Error dumps.
Does anyone have any idea what I'm doing wrong here? I'm fairly new to both Firefox extensions and javascript (although I am a fairly experienced programmer) - so I may be doing something really silly. If my entire approach is wrong then please let me know - I would have thought that this approach would allow me to append a file easily, and asynchronously, but it may not be possible for some reason that I don't know about.
function append_text_from_file(from_filename, to_filename) {
var from_file = Components.classes["#mozilla.org/file/local;1"].createInstance(Components.interfaces.nsILocalFile);
from_file.initWithPath(from_filename);
var to_file = Components.classes["#mozilla.org/file/local;1"].createInstance(Components.interfaces.nsILocalFile);
to_file.initWithPath(to_filename);
dump("Appending text\n");
var ostream = FileUtils.openFileOutputStream(to_file, FileUtils.MODE_WRONLY | FileUtils.MODE_APPEND)
dump("After ostream\n");
NetUtil.asyncCopy(from_file, ostream, function(aResult) {
dump("Done\n");
if (!Components.isSuccessCode(aResult)) {
// an error occurred!
dump(aResult);
dump("Error!\n")
}
});
}

asyncCopy() requires an input stream not a file.
you can do this:
var fstream = Cc["#mozilla.org/network/file-input-stream;1"].createInstance(Ci.nsIFileInputStream);
fstream.init(from_file, 0x01, 4, null);
NetUtil.asyncCopy(fstream, ostream, function(aResult)....

Auto Save a file in Firefox

I am trying to find a way where by we can auto save a file in Firefox using JS. The way I have done till yet using FireShot on a Windows Desktop:
var element = content.document.createElement("FireShotDataElement");
element.setAttribute("Entire", EntirePage);
element.setAttribute("Action", Action);
element.setAttribute("Key", Key);
element.setAttribute("BASE64Content", "");
element.setAttribute("Data", Data);
element.setAttribute("Document", content.document);
if (typeof(CapturedFrameId) != "undefined")
element.setAttribute("CapturedFrameId", CapturedFrameId);
content.document.documentElement.appendChild(element);
var evt = content.document.createEvent("Events");
evt.initEvent("capturePageEvt", true, false);
element.dispatchEvent(evt);
But the issue is that it opens a dialog box to confirm the local drive location details. Is there a way I can hard code the local drive storage location and auto save the file?

If you are creating a Firefox add-on then FileUtils and NetUtil.asyncCopy are your friends:
Components.utils.import("resource://gre/modules/FileUtils.jsm");
Components.utils.import("resource://gre/modules/NetUtil.jsm");
var TEST_DATA = "this is a test string";
var source = Components.classes["#mozilla.org/io/string-input-stream;1"].
createInstance(Components.interfaces.nsIStringInputStream);
source.setData(TEST_DATA, TEST_DATA.length);
var file = new FileUtils.File("c:\\foo\\bar.txt");
var sink = file.openSafeFileOutputStream(file, FileUtils.MODE_WRONLY |
FileUtils.MODE_CREATE);
NetUtil.asyncCopy(source, sink);
This will asynchronously write the string this is a test string into the file c:\foo\bar.txt. Note that NetUtil.asyncCopy closes both streams automatically, you don't need to do it. However, you might want to pass a function as third parameter to this method - it will be called when the write operation is finished.
See also: Code snippets, writing to a file

Every computer has a different file structure. But still, there is a way. You can save it to cookie / session, depends on how "permanent" your data wants to be.
Do not consider writing a physical file as it requires extra permission.