This pertains to cookies set inside a script (maybe inside a script tag).
System.Windows.Forms.HtmlDocument executes those scripts and the cookies set (like document.cookie=etc...) can be retrieved through its Cookies property.
I assume HtmlAgilityPack.HtmlDocument doesn't do this (execution). I wonder if there is an easy way to emulate the System.Windows.Forms.HtmlDocument capabilities (the cookies part).
Anyone?
When I need to use Cookies and HtmlAgilityPack together, or just create custom requests (for example, set the User-Agent property, etc), here is what I do:
Create a class that encapsulates the request/response. Let's call this class WebQuery
Have a private CookieCollection (in your case public) property inside that class
Create a method inside the class that does manually the request. The signature could be:
...
public HtmlAgilityPack.HtmlDocument GetSource(string url);
What do we need to do inside this method?
Well, using HttpWebRequest and HttpWebResponse, generate the http request manually (there are several examples of how to do this on Internet), create an instance of a HtmlDocument class using the constructor that receives an stream.
What stream do we have to use? Well, the one returned by:
httpResponse.GetResponseStream();
If you use HttpWebRequest to make the query, you can easily set the CookieContainer property of it to the variable you declared before everytime you access a new page, and that way all cookies set by the sites you access will be properly stored in the CookieContainer variable you declared in your WebQuery class, taking in count you're using only one instance of the WebQuery class.
Hope you find useful this explanation. Take in count that using this, you can do whatever you want, no matter if HtmlAgilityPack supports it or not.
I also worked with Rohit Agarwal's BrowserSession class together with HtmlAgilityPack.
But for me subsequent calls of the "Get-function" didn't work, because every time new cookies have been set.
That's why I added some functions by my own. (My solution is far a way from beeing perfect - it's just a quick and dirty fix) But for me it worked and if you don't want to spent a lot of time in investigating BrowserSession class here is what I did:
The added/modified functions are the following:
class BrowserSession{
private bool _isPost;
private HtmlDocument _htmlDoc;
public CookieContainer cookiePot; //<- This is the new CookieContainer
...
public string Get2(string url)
{
HtmlWeb web = new HtmlWeb();
web.UseCookies = true;
web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest2);
web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse2);
HtmlDocument doc = web.Load(url);
return doc.DocumentNode.InnerHtml;
}
public bool OnPreRequest2(HttpWebRequest request)
{
request.CookieContainer = cookiePot;
return true;
}
protected void OnAfterResponse2(HttpWebRequest request, HttpWebResponse response)
{
//do nothing
}
private void SaveCookiesFrom(HttpWebResponse response)
{
if ((response.Cookies.Count > 0))
{
if (Cookies == null)
{
Cookies = new CookieCollection();
}
Cookies.Add(response.Cookies);
cookiePot.Add(Cookies); //-> add the Cookies to the cookiePot
}
}
What it does: It basically saves the cookies from the initial "Post-Response" and adds the same CookieContainer to the request called later. I do not fully understand why it was not working in the initial version because it somehow does the same in the AddCookiesTo-function. (if (Cookies != null && Cookies.Count > 0) request.CookieContainer.Add(Cookies);)
Anyhow, with these added functions it should work fine now.
It can be used like this:
//initial "Login-procedure"
BrowserSession b = new BrowserSession();
b.Get("http://www.blablubb/login.php");
b.FormElements["username"] = "yourusername";
b.FormElements["password"] = "yourpass";
string response = b.Post("http://www.blablubb/login.php");
all subsequent calls should use:
response = b.Get2("http://www.blablubb/secondpageyouwannabrowseto");
response = b.Get2("http://www.blablubb/thirdpageyouwannabrowseto");
...
I hope it helps when you're facing the same problem.
Related
I use jquery.get() to make a call to the Spring-Boot backend, like this:
var url = "http://localhost:8080/api/v1/get_holdings?userId=1";
$.get(url, function(payload,status) {
if (status == "success") {
$.each(payload.data, function (key,entry) {
...
})
}
else {...
In SpringBoot I create a model that has an embedded model. In this case the model is called Holding and the embedded model is called Account, like this:
#Entity
#Table(name="holding")
public class Holding {
...
#ManyToOne //Many holdings to One Account
#JoinColumn(name="account_id", nullable=false)
private Account account;
...
//getters and setters...
The call to the database returns the data correctly.
Then I try to return it to the jquery ajax call like this:
JsonResponse response = new JsonResponse();
List<Holding> holdings = holdingRepo.getUserHoldings(userId);
response.setData(holdings); //response looks correct but browser shows a 500 error. Why?
return response;
JsonResponse is just a wrapper that I use for all my responses. I've used it many times and has worked fine for many other calls:
public class JsonResponse {
private String status = null;
private Object data = null;
//getters and setters...
It returns to the browser ok, but the browser gives a 500 error.
There are two ways that I can get the browser to accept the data.
create a separate model that contains just the data I need with no embedded models
create the json manually using the toString() method (pain in the butt)
I've tried converting the Holding object to JSON using Gson.toJson and JSONArray(holdings) but both of those methods fail. What am I missing?
You are using a relation. I suppose there happens an exception, like following:
org.hibernate.LazyInitializationException: could not initialize proxy
This can happen because Hibernate tries to optimize performance and does not load objects in the relation.
Then there are many answers to this question on SO.
One solution can be to force Hibernate to initialize the proxy via
Hibernate.initialize( entity )
Another approach can be not to send entities to the client, but to transform them into data transfer objects. If you do such transformation within the transaction, Hibernate will resolve all needed objects automatically and you will not need any tricks.
I'm fairly new to MVC, and I seem to be having a problem returning a view from JS. I'm working on a chatting site, and this code creates a new chatroom with the name you assign it. Here's my JS code:
function CreateConversation() {
var conversationName = document.getElementById("conversationName").value;
var username = document.getElementById("usernameLabel").textContent;
window.location.href = "/CreateNewChat/CreateConversation?username=" + username + "&convName=" + conversationName;
}
That should call this ActionResult method:
public IActionResult CreateConversation(string username, string convName)
{
ChatModel model = new ChatModel(username, convName);
model.Chatters.Add(username);
return RedirectToAction("Chat","Chat", model);
}
Which does, as long as I don't type any conversation name in the text box. If I check the JS code, both username and conversation name values are correctly set up, but the ActionResult method will not be hit if I fill in the conversation name.
Here's the Chat method in the ChatController:
public IActionResult Chat(ChatModel model)
{
return View(model);
}
If I don't put any conversation name in, it will come all the way to this method, but it won't work (as that field is obligatory and I need it to load the view). Any ideas? I've tried using ajax but while it will pass the conversation name along, it will get stuck at the RedirectToAction statement (will never hit the Chat method in the ChatController). I'm really lost here.
Generally, you shouldn't pass objects around like that - at best you should only pass around ViewModels to prevent someone from fudging data and potentially breaking your app.
The way you have it with your redirect will actually push the whole model into the URL parameters before stepping back into code, leaving it wide open for someone to change as they see fit.
Ideally you would post to create a persisted instance then redirect by passing only an identifier to the Chat() method like so:
CreateNewChatController.cs
public ActionResult CreateConversation(string username, string convname)
{
Guid chatId = yourServiceFactory.CreateNewChat(username, convname);
// You passed the service the username anyway so it can do the Chatters.Add() internally
return RedirectToAction("Chat", "Chat", new { id = chatId });
}
ChatController.cs
public ActionResult Chat(Guid id)
{
ChatModel model = yourServiceFactory.GetChatById(id);
return View(model);
}
However, if you really want to pass the model onto ActionResult Chat(ChatModel model), then you can execute and return the controller action directly.
public IActionResult CreateConversation(string username, string convname)
{
ChatModel model = new ChatModel(username, convname);
model.Chatters.Add(username);
ChatController chatController = new ChatController();
return chatController.Chat(model);
}
I've created a very simplistic working example of this based on what you've said above, so any validation you may have in your ChatModel constructor is omitted in my example.
For any other errors, I would suggest checking to see if the values you're passing in aren't being detected as "potentially harmful" by the app and being dropped.
Noting that the use of .textContent instead of value would get the full contents between the start and ending tags of a given element, I'm assuming the element is a textarea type not a normal text input. You could be getting extra elements between <textarea>...</textarea>
In my example, my javascript function just takes two local variables declared and set within the function. Without seeing your submit page, I can't really comment on why else setting a value would prevent a call from working.
Let's say, I have a class like this:
var block = class {
constructor(val1, val2) {
this.value1 = val1;
this.value2 = val2;
}
}
I want to send an instance of this class from one server to another. I would typically pass it through JSON.stringify() before sending it to the other server, then JSON.parse() once it reaches the other server. However, this would involve converting it to JSON and it would no longer be an instance of the 'block' class.
If I wanted to send it over in a way that keeps it as an instance of the 'block' class what would be best practice.
What your asking for here, is a kind of RPC (Remote procedure call).
And I think your right in making it so that when you deserialize the object at the server side, you can re-use the class that you used at client side.
Using your example class block, here I've added a method called display,. What's handy here is that display will work on both servers after sending the JSON.
Obviously here in an SO snippet it's a little awkward to show Server to Server comms, but what I have done is save to a String, and then regenerate the same class from the String, exactly what you would do if doing some RPC.
You could of course extend this so that even the class name gets sent, and this is used to re-generate the object, so that it could send lots of types of RPC commands.
var block = class {
constructor(val1, val2) {
this.value1 = val1;
this.value2 = val2;
}
display() {
console.log(this.value1, this.value2);
}
}
var b1 = new block(1,2);
b1.display();
//lets pretend our stream is what gets sent over network
var stream = JSON.stringify(b1);
var b2 = new block();
Object.assign(b2, JSON.parse(stream));
b2.display();
I am trying to build a console application using c# .net.. I need one particular functionality which is to retrieve values from an external website. The thing is we need to sign in to that website. I am able to open the page that i need using process.start in chrome signed in with the values i need.. but problem is when retrieving the value from page.. i thought of getting source code but every way i try it does not take the session and hence i am getting just error page source code as i am just entering the URL n not accessing the already opened tab? Is there any other way available either using JavaScript or c#?
Use WebClient API to login and download page data. You will need to make it cookie aware in order to maintain session. You will need to add reference to System.Web Assembly to use this API in console application.
public class CookieAwareWebClient : WebClient
{
public CookieAwareWebClient()
{
CookieContainer = new CookieContainer();
}
public CookieContainer CookieContainer { get; private set; }
protected override WebRequest GetWebRequest(Uri address)
{
var request = (HttpWebRequest)base.GetWebRequest(address);
request.CookieContainer = CookieContainer;
return request;
}
}
Now use it like
using (var client = new CookieAwareWebClient())
{
var values = new NameValueCollection
{
{ "username", "john" },
{ "password", "secret" },
};
client.UploadValues("http://domain.loc/logon.aspx", values);
// If the previous call succeeded we now have a valid authentication cookie
// so we could download the protected page
string result = client.DownloadString("http://domain.loc/testpage.aspx");
}
I borrowed this piece of code from WebClient accessing page with credentials
I'll preface this by stating that I know Java is not JavaScript and vice versa.
I've got a project where I need to count occurrences of words for each of 1750 document names and document contents. I've got some awesome JavaScript from a colleague that does exactly what I want from a form input on a web page.
I want to use Java's FileReader, BufferedReader, walkFileTree, etc. to traverse the directories in which the documents live.
I'm not sure if this is the most efficient or effective approach, but both the Java and JavaScript parts of the code are working independently of one another now, and I'd like to see if I can get them to pass data between them before I start re-inventing the wheel.
Here's where I am so far. I'm stuck at the CLParse method & have inserted pseudocode:
public static void main(String... aArgs) throws FileNotFoundException {
File startingDirectory= new File("CGT");
List<File> files = FileListing.getFileListingNoSort(startingDirectory);
for(File file : files ) {
CLParse(file.toString());
} }
static private List<File> getFileListingNoSort(File aDirectory) throws FileNotFoundException {
List<File> result = new ArrayList<File>();
File[] filesAndDirs = aDirectory.listFiles();
List<File> filesDirs = Arrays.asList(filesAndDirs);
for(File file : filesDirs) {
result.add(file); //always add, even if directory
if ( ! file.isFile() ) {
List<File> deeperList = getFileListingNoSort(file);
result.addAll(deeperList);
} }
return result;
}
/* is something like this doable and how would I do it?
*/
public static void CLParse(String fn) {
pass fn to JavaScript counter
return an array of {word,occurences} for the string
write array to file
}
I'll be creating another set of methods to extract and pass the document CONTENTS as a string as well. I'd love to know if anyone has any practical experience passing values back and forth between Java and JavaScript, and advice on a good/better way to do it.
You got 2 Options to let them interact with each other, which i know:
1.Applet <-> javascript
2.Serlvet <-> javascript
With option 1, you have to build a Communication with a JSObject: JSObject
or you cann call the Applets Method instanstly with document.appletname.methodname();
with this you can even Parse same simply Formats to each other.
With Option 2 you have to build a communication with a Servlet.
in here you have to send an Ajax request to the the Servlet:
$.post('login',{name:"Peter", pw:"123456"},function()
{
//do whatever
})
JavaServlet class
the first comment, has to written as an Servlet in your web.xml, it´s the servlet pattern.
the second ones, are the parameters which can be read in the servlet. the function describes the stuff, which can be done in the request.
The differences between these two Options are:
1.the Applets runs on the users Computer, so you can access his files. But for this your applet has to be signed.
2.the Servlet runs on the Server. Here you have got full file access(if the system allows you too have it).
I would try to investigate Mozilla Rhino.
http://en.wikipedia.org/wiki/Rhino_%28JavaScript_engine%29
Check out Rhino https://developer.mozilla.org/en-US/docs/Rhino
You can create java objects and use them in javascript. Integration is straightforward
You can use AJAX to send and receive values to server. You can send parameters or JSON to server and get response.
You can use JSONP to serve the data, or if you have no control of the second server, use a reverse proxy to proxy requests to the second server through the first.