I can't find out the keyword I'm looking for. When I google anything with URL encoding or storing data, include data, whatever, I get all kinds of results except what I'm interested in. This is the only website I could find off the top of my head that shows what I'm looking for:
http://www.pathofexile.com/passive-skill-tree/AAAAAgMA37CCEEGWBUKusyycwbTk7HYRfq9JsgLjB6Vr230Y7IpzU8BU5oERUDGIeQMI9It6EHQOXEV-Va6X9JeVUlOPpkSrPV8EB0yzLR-NGeAS3Yy1heZM2V8ucA==
after tree/ it has a long code that pretty much is full of data. What should I look into to be able to do something like that? Is one supposed to create their own method according to what they need? Or is there a way one can just take one super long text and have a library encode it to make it smaller for the URL and then decode it when it loads?
I require tons of numbers, around 100. I figured it would be something like this, first off use a symbol to separate each 'variable', in this case let's use '-' and do something like this:
www.url.com/tree/1-1-1-0-3-2-1-3-4-5-2...total of 100 numbers..1-0-2, but then it gets encoded to be much smaller to something like
www.url.com/tree/xDgdmFdmnDfjSDfjSFdKflWepLS and this url gets decoded once loaded and the data retrieved and used behind the scenes.
Is there an easier way of doing this, or does one have to do it manually depending on their needs? By easier I mean, a way of encoding it, or does one have to do the encoding themselves? For example, make it so if there are more of the same numbers next to each other then it takes them and transforms them into letters, let's say there are five 3's next to each other, it would use the letter c to show what the number is, and a capital letter for the number of times it's repeated, so cE would mean five 3's in a row.
My question is, is there a way to encode it or do I have to think of a way to encode it myself like I was writing in the example?
Any information you have related to this subjecte is GREATLY appreciated!! Thanks so much in advance for taking the time to read all this and reply, sorry to bother
You are looking to base 64 encode data.
Related
What I am doing is saving and retrieving lot of Images on the client.
(Now indexedDB seemed to be overkill for this simple job, but since it was the only Cross-Browser solution with no limit(like localStorage), I had to use it ... and it works)
This is what my db looks like:
(more specific, the only objectstore of my db)
# | key(timeID) | value
0 | 812378123 | {data:¨....¨, tnData:¨...¨, timeID:812378123}
1 | 912378123 | {data:¨....¨, tnData:¨...¨, timeID:912378123}
2 ....
KeyValue is a unique TimeID, Data contains the Image as text and tnData contains the thumbnail of this Image as text
(when canvas.toBlob() is ready I will switch to that)
To retrieve a Image I just use store.get(id)
Now all of that works.
But now I just want to load the Thumbnail(¨tnData¨) - but NOT the real Image (¨data¨ which can be quite big).
So I hope there is something like store.get(id, ¨tnData¨) ...
But I did not found it, so far.
Does anyone know of a easy way of doing this, without having to rework my db?
Thanks in advance ... and sorry if I am not in the right place or have broken some other rule ... first question for me ;)
Does anyone know of a easy way of doing this, without having to rework my db?
No, IndexedDB doesn't let you return only part of an object.
Although reworking your db might not be too hard. You could break it into two object stores, one for thumbnails and one for whole images. Then you can query for specifically the one you want.
If you're often getting both at the same time, you'll have to do some benchmarking and see if it actually is faster to break them up, or maybe it could even make sense to duplicate thumbnails (one object store like you have now, and one with just thumbnails).
I want to use cmudict file in a web. It contains 170000 words with its phonetic transcription (in ARPAbet symbols).
http://www.speech.cs.cmu.edu/cgi-bin/cmudict
I want to use it in JSON format, search any word introduced by the user and return an explanation of how to pronounce it syllabe by syllabe. The second part is not very complex in search terms as there are only 39 different phonemes, but the first one with the 170000 entries may consume too much time if the user introduces a text instead of a single word to transcript.
I wonder if it's worth to split the JSON into for example 26 parts (one per initial letter) and search only in the corresponding file.
Also I don't know if JSON is the best format for this, but I want to use it in a free blog like Tumblr or Blogger ones (or similar, the thing is that I don't want to spend money in this) and Javascript is what they support. I would listen suggestions on this too.
Well, that is tough call since you must consider download size. I would shorten the names of all your properties to be as small as possible, so instead of repeating "description" : "the short description", I would go with "sd" : "the short description". You are trying to use javascript to serve a data file, which is okay since you can rely on caching and what not, but the initial download size may be rather large. I would do something like var myDictionary = { }; at the top of the file, that way you can reference the variable since it is in the global space. It is an interesting experiment for sure.
Before anyone jumps in and says, "Oh!! that's a bad idea", I know it is.
I want to keep both the key and value in the query string to be not easily visible to the end user.
I have something like this google.com/?category=textile&user=user1
I need to make it unintelligible like this: google.com/?kasjdhfkashasdfsf32423
Is there any way to achieve this in javascript. I have already seen this
I have already seen this
and this.
but I don't think encoding will solve the problem. Also, this code is entirely in client side. I know that it is not secure but I just need this is a naive, weak defense.
Please help.
Edit
I apologize if my question was not clear earlier.
The URL google.com/?category=textile&user=user1 is being passed on from a different application.
The values passed in the query string directly controls what is being displayed to the user. As is, anyone with no technical knowledge can easily change the value and view the data corresponding to a different category or user. I need to make this unintelligible so that it is not obvious. If a user is a techie and figures out the encryption used, then it is fine. I need a stop-gap solution till we have a better architecture in place
You can use base64. Javascript has native functions to do that :
alert(btoa("category=textile&user=user1")); // ==> Y2F0ZWdvcnk9dGV4dGlsZSZ1c2VyPXVzZXIx
and to reverse it :
alert(atob("Y2F0ZWdvcnk9dGV4dGlsZSZ1c2VyPXVzZXIx")); // ==> category=textile&user=user1
Be careful to read the doc if you have unicode strings, it's a little different : https://developer.mozilla.org/en-US/docs/Web/API/Window.btoa
If you don't looking for serious strong crypto, you can use ROT13:
http://en.wikipedia.org/wiki/ROT13
This is enough for slightly obfuscate keys/values in the your URLs.
I have a simple piece of data that I'm storing on a server, as a plain string. It is kind of ridiculous, but it looks like this:
name|date|grade|description|name|date|grade|description|repeat for a long time
this string can be up to 1.4mb in size. The idea is that it's a bunch of student records, just strung together with a simple pipe delimeter. It's a very poor serialization method.
Once this massive string is pushed to the client, it is split along the pipes into student records again, using javascript.
I've been timing how long it takes to create, and split, these strings on the client side. The times are actually quite good, the slowest run I've seen on a few different machines is 0.2 seconds for 10,000 'student records', which has a final string size of ~1.4mb.
I realize this is quite bizarre, just wondering if there are any inherent problems with creating and splitting such large strings using javascript? I don't know how different browsers implement their javascript engines. I've tried this on the 'major' browsers, but don't know how this would perform on earlier versions of each.
Yeah looking for any comments on this, this is more for fun than anything else!
Thanks
String splitting for 1.4mb data is not a problem for decent machines, instead you should worry about the internet connection speed of your users. I've tried to do spell check with 800 kb dictionary (which is half of your data), main issue was loading time.
But looks like your students records data could be put in database, and might not need to load everything at loading time, So, how about do a pagination to show user records or use ajax to request to search certain user names?
If it's a really large string it may pay to continuously slice the string with 'string'.slice(from, to) to only process a smaller subset, appending all of the individual items to the end of the output with list.push() or something similar might work.
String split methods are probably the most efficient way of doing this though, even in IE. Processing individual characters using string.charAt(x) is extremely slow and will often show a security error as it stalls the browser. Using string split methods would certainly be much faster than splitting using regular expressions.
It may also be possible to encode the data using a JSON array, some newer browsers such as IE8/Webkit/FF3.5 have fast JSON parsing built in using JSON.parse(data). But using eval(JSON) may overflow the browser if there's enough data, so is probably a bad idea. It may pay to compare for performance though.
A much better approach in a lot of cases is to use AJAX and only load some of the data at once from the server, which would also save download time.
Besides S. Mark's excellent comments about local vs. x-fer speed and the tip to re-encode using AJAX, I suggest a (longterm) move away from JavaScript in the Browser (assuming that's were it runs) to either a non-browser implementation of JS (or possibly another language).
A browser based JS seems a week link in a data-x-fer chain and nothing I would want to run unmonitored, since the browsers are upgraded from time to time and breaking your JS-x-fer might be an unanticipates side effect!
I have a web application where a client side editor is editing a really really large text which is known on the server side.
The client can make any kind of modifications to this text.
What is the most network-efficient way to transmit the result difference in a way that the server understands? Also, since this will happen on client side (Javascript), I would also like it to be 'fast' (or at least not noticeably slow)
Some scenarios:
User modifies ONE character
User modifies several sentences in random positions
User erases everything and results in a blank text.
I cannot use diff-like syntax since it's not network efficent, it checks lines, where examples 1 and 3 will produce horrible differences (especially the last one, where the result will be more than the old itself).
Anyone has experience in this matter? User operates on a really large set of data - around 3-5MB of text, and uploading the whole "new" content is a big no-no.
To be clear, I'm looking for a "protocol" of transfer, string comparison is not the issue.
I'm not very familiar with this topic but I can point you to an open source (Apache License 2.0) project which may be very useful.
It is a Diff, Match and Patch library written in several languages, including JavaScript, from a Google engineer and it is used in several online collaborative editing services.
Here are a list of resources:
The Diff, Match and Patch project
The MobWrite project (Editor implementation based on the above project)
"Differential Synchronization" (A Google Tech Talk by the engineer)
A simple approach, assuming that you know the copy on the server isn't going to change, would just be to send a list of edits (deletions and additions), with the deletions represented as a start and end index, and the additions represented as a start index and the text to insert.
If you have more than a simple diff algorithm to work with (I'm not sure exactly what you mean by "string comparison is not the issue"), you could also detect moved or copied chunks of text, and send those as the start and end index of the moved or copied piece of text, as well as the destination to insert it.
Note that you'll need to make sure to keep track of whether your indices refer to the original document, or the document as edited so far. An easy approach to avoid this problem is to always perform the edits from the end of the document towards the beginning; then earlier edits won't affect the offsets specified by later edits.
For an example of an approach like this, see the ed format that diff -e outputs. This is basically input that could be fed into the ed line-oriented text editor. If you want the absolute smallest diffs to send across you may want to do character based indexing rather than line based indexing, but the same basic approach could work.
Any edits the user's performing can be efficiently broken down into: delete from X for length Y; insert at X text "whatever". X and Y are offsets in characters from the start of the text; Y is a number of characters; "whatever" is any string of characters. You say you need no help computing the diff, but an example is here, except it's richer in its output than you need, but does identify "removals and insertions", so, just change the output part.
The exact format in which you send the data to the server can be tuned, but I don't think there's much mileage in doing that -- pending measurement, I'd start by sending the commands as D for delete or I for insert, the numbers in decimal, the inserted string in quoted form. Once you have some statistics on actual transfers being performed, you can see how much overhead is in the numbers (decimal vs binary) and quotes, but I suspect that may not be all that meaningful (if it proves to be, there are all sort of things you can try, such as giving offsets from the latest point of insertion or deletion, rather than always from the start, to make things faster).
You can sample what the user is doing every few seconds, and just send the incremental changes over those last few seconds (if any) -- this way, each packet you're sending will be small, and if the net connection or the user's computer/browser crash, the user won't have lost much work.
You could just send changes every 500ms, so, whatever changes were made in the last 500ms would be sent, but you only send data when there was a change.
In this you could then send the position of the changed word(s) and just send the entire word, but I would have the position be from the front of the text.
It won't be several sentences worth, but there may be several words involved, but, if you send them in order of change then the result should be consistent.
Because there are so many ways to do edits--even within short periods of time like 500ms--including dragging and dropping, or cutting and pasting, large sections of text around within the document or from outside it--I don't know if there's going to be something that will cover all scenarios really well. This is certainly a non-answer to your question at face value, but I would consider carefully the trouble of developing and maintaining something like this compared to changing the interface to restrict the text size and breaking existing texts into smaller pieces.
Maybe that's not possible in your situation, but if it is, I would guess it would be much less trouble in the end to dodge the issue in this way and just send full documents after an edit.