maemo.org - Talk - Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

maemo.org - Talk (https://talk.maemo.org/index.php)

- Development (https://talk.maemo.org/forumdisplay.php?f=13)

Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

Hi!

I would like to create a application to store and manage my N900 / N9(50) bookmarks in "the cloud" - to be exact: at google bookmarks! ;)

Sadly there is no API and I've already searched for a good alterative, without success. :(

I would like to give google a second try. Because of the missing API and hence the missing OAuth I have to act like a webbrowser and after faking a logging in provide the stored cookies to let google believe I'm logged in into their website correctly.

I looked into the code from gmarks for android to have a working example how to do it:

Here is the login part - Have a look here if you like to see the whole source: BookmarksQueryService.java (Link to github)

Code:

public void login( String user, String passwd ) {

                try {

                        List<NameValuePair> queryParams = new ArrayList<NameValuePair>();

                        queryParams.add( new BasicNameValuePair("service", "bookmarks") );

                        queryParams.add( new BasicNameValuePair("passive", "true") );

                        queryParams.add( new BasicNameValuePair("nui", "1") );

                        queryParams.add( new BasicNameValuePair("continue", "https://www.google.com/bookmarks/l") );

                        queryParams.add( new BasicNameValuePair("followup", "https://www.google.com/bookmarks/l") );

                        HttpGet get = new HttpGet( "https://www.google.com/accounts/ServiceLogin?" + 

                                        URLEncodedUtils.format(queryParams, "UTF-8") );

                        HttpResponse resp = http.execute(get, this.ctx);

                        // this just gets the cookie but I can ignore it...

                        

                        if ( resp.getStatusLine().getStatusCode() != 200 )

                                throw new RuntimeException( "Invalid status code for ServiceLogin " +

                                                resp.getStatusLine().getStatusCode() );

                        resp.getEntity().consumeContent();

                        

                        String galx = null;

                        for ( Cookie c : cookieStore.getCookies() )

                                if ( c.getName().equals( "GALX" ) ) galx = c.getValue(); 

                        

                        if ( galx == null ) throw new RuntimeException( "GALX cookie not found!" );

                        

                        HttpPost loginMethod = new HttpPost("https://www.google.com/accounts/ServiceLoginAuth");

                        // post parameters:

                        List<NameValuePair> nvps = new ArrayList<NameValuePair>();

                        nvps.add(new BasicNameValuePair("Email", user));

                        nvps.add(new BasicNameValuePair("Passwd", passwd));

                        nvps.add(new BasicNameValuePair("PersistentCookie", "yes"));

                        nvps.add(new BasicNameValuePair("GALX", galx));                        

                        nvps.add(new BasicNameValuePair("continue", "https://www.google.com/bookmarks/l"));

                        loginMethod.setEntity(new UrlEncodedFormEntity(nvps));

                        resp = http.execute( loginMethod, this.ctx );

                        

                        if ( resp.getStatusLine().getStatusCode() != 302 )

                                throw new RuntimeException( "Unexpected status code for ServiceLoginAuth" +

                                                resp.getStatusLine().getStatusCode() );

                        resp.getEntity().consumeContent();

                        

                        Header checkCookieLocation = resp.getFirstHeader("Location");

                        if ( checkCookieLocation == null ) 

                                throw new RuntimeException("Missing checkCookie redirect location!");



                        // CheckCookie:

                        get = new HttpGet( checkCookieLocation.getValue() );

                        resp = http.execute( get, this.ctx );

                        

                        if ( resp.getStatusLine().getStatusCode() != 302 )

                                throw new RuntimeException( "Unexpected status code for CheckCookie" +

                                                resp.getStatusLine().getStatusCode() );

                        resp.getEntity().consumeContent();

                        

                        this.authInitialized = true;

                        Log.i(TAG, "Final redirect location: " + resp.getFirstHeader("Location").getValue() );

                        Log.i(TAG, "Logged in.");

                }

                catch ( IOException ex ) {

                        Log.e(TAG, "Error during login", ex );

                        throw new RuntimeException("IOException during login", ex);

                }

        }

I tried now to rebuild the same mechanism in Qt: (I shortened the code a little bit and removed the unessential parts)

.h

Code:

#include <QtNetwork/QNetworkAccessManager>

#include <QtNetwork/QNetworkReply>

#include <QUrl>

#include <QHttp>

#include <QNetworkCookie>

#include <QNetworkCookieJar>



[...]



private:

    QString galx;

    QNetworkAccessManager* nManLoginStep_1;

    QNetworkAccessManager* nManLoginStep_2;

    QNetworkAccessManager* nManLoginStep_3;



    QNetworkCookieJar* CookieJar;

init:

Code:

 galx = "";

    nManLoginStep_1 = new QNetworkAccessManager(this);

    nManLoginStep_2 = new QNetworkAccessManager(this);

    nManLoginStep_3 = new QNetworkAccessManager(this);



    CookieJar = new QNetworkCookieJar(this);



    nManLoginStep_1->setCookieJar(CookieJar);

    nManLoginStep_2->setCookieJar(CookieJar);

    nManLoginStep_3->setCookieJar(CookieJar);



  connect(nManLoginStep_1, SIGNAL(finished(QNetworkReply*)), this, SLOT(LoginStep_1_cookie(QNetworkReply*)));

    connect(nManLoginStep_2, SIGNAL(finished(QNetworkReply*)), this, SLOT(LoginStep_2_ServiceLoginAuth(QNetworkReply*)));

    connect(nManLoginStep_3, SIGNAL(finished(QNetworkReply*)), this, SLOT(LoginStep_3_check(QNetworkReply*)));

The essence of the problem:

Code:



void Application::Login()

{

    QUrl url("https://accounts.google.com/ServiceLogin");



    url.addQueryItem("service","bookmarks");

    url.addQueryItem("passive", "true");

    url.addQueryItem("nui", "1");

    url.addQueryItem("continue", "https://www.google.com/bookmarks/l");

    url.addQueryItem("followup", "https://www.google.com/bookmarks/l");



    nManLoginStep_1->get(QNetworkRequest(url));

}



void Application::LoginStep_1_cookie(QNetworkReply *r)

{

    QByteArray data = r->readAll();

    QString str(data);   //here I'm able to check the content of the website



    QUrl url1("https://accounts.google.com/ServiceLogin");

    QList<QNetworkCookie>  cookies = CookieJar->cookiesForUrl(url1);

   foreach (QNetworkCookie c, cookies)

   {

       QByteArray a(c.name());

       QString s(a);

              if (s.toStdString() == "GALX" ) {galx = c.value();}

}

    if (galx.trimmed() == "")

    {

    std::cout << "ERROR: No GALX cookie found!\n";

    }



    //--------------------------------------------------------------





    QUrl url("https://www.google.com/accounts/ServiceLoginAuth");

    QUrl postData;



    postData.addQueryItem("Email","EMAIL@EMAIL.COM");

    postData.addQueryItem("Passwd", "PASSWORD");

    postData.addQueryItem("PersistentCookie", "yes");

    postData.addQueryItem("GALX", galx);

    postData.addQueryItem("continue", "https://www.google.com/bookmarks/l");

    QNetworkRequest request(url);

    nManLoginStep_2->post(request, postData.encodedQuery() );



}



void Application::LoginStep_2_ServiceLoginAuth(QNetworkReply *r)

{

    QByteArray data = r->readAll();

    QString str(data);   

    //here I get a website with the error: "the cookies in your webbrowser are deactivated

}

Even before the verification I get a website with the error: "the cookies in your webbrowser are deactivated, please activate" - if I ignore this and proceed I get a "The document has moved Temporarily" website without layout and a redirectionlink to a website where I'm able to manage my google account data. :(

My code seems to doesn't provide the previously generated cookie to the website. I don't understand why. It is perhaps only a small mistake, a flag not set? The cookies are definitively generated and stored in the CookieJar.

Has anybody here some kind of experience with QNetworkAccessManager and cookies?
Please help! Any kind of hint is very welcome. :)

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

It's heavily advised to only use one QNetworkAccessManager.

It doesn't look like you've encoded the query, maybe you could try that.

I'll give it a better look when I can get to my laptop.

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

Quote:

Originally Posted by Reffyyyy (Post 1122102)

It's heavily advised to only use one QNetworkAccessManager.

Ouh, okay. Sometimes I guess I'm advertising-immune. :D
I will change my code now to use only a single QNetworkAccessManager. Thanks! :)

Quote:

Originally Posted by Reffyyyy (Post 1122102)

It doesn't look like you've encoded the query, maybe you could try that.

I thought this happens automatically when using QUrl... but I will check this. Thanks for the hint! :)

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

Time for bed now, I guess I'm working now to many hours on this login problem. A break will always be a good idea or I'm afraid google will lock my account - "suspicion of a account hacking attack" ;)

Quote:

Originally Posted by Reffyyyy (Post 1122102)

It's heavily advised to only use one QNetworkAccessManager.

I've changed my code and have now only a single QNetworkAccessManager for the communication. Still the same cookie problem.

Quote:

Originally Posted by Reffyyyy (Post 1122102)

It doesn't look like you've encoded the query, maybe you could try that.

If I understood the documentation correctly they are getting encoded: QUrl::addQueryItem

Quote:

Inserts the pair key = value into the query string of the URL.

The key/value pair is encoded before it is added to the query. The pair is converted into separate strings internally. The key and value is first encoded into UTF-8 and then delimited by the character returned by valueDelimiter(). Each key/value pair is delimited by the character returned by pairDelimiter().

Note: This method does not encode spaces (ASCII 0x20) as plus (+) signs, like HTML forms do. If you need that kind of encoding, you must encode the value yourself and use QUrl::addEncodedQueryItem.

But thanks for your hints. If you have further ideas please let me know! :)

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

You may find it quite PITA, due to google changing their logging method quite frequently. They do it, because they like break programs like gMail Drive (sometimes, new version of Gmail drive, just changing logging mechanism arrive every few weeks), so your application may be affected also.

I tried to check if Gmail Drive is Open source (so you could examine login method they use), but unfortunately, license isn't stated anywhere, and it's being released as binaries, so probably it's freeware closed source. Still, I've heard of phpGmailDrive for GNU (tool working just like gmail drive, but linking many accounts into one virtual big account) - You may want to search for it, it may be Open Source.

Hope it helps and is at least relevant to Your troubles ;)

/Estel

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

Quote:

Originally Posted by Estel (Post 1122187)

Yeah, because of this I searched (or I'm searching) for something "like" google bookmarks, but with a official and good documented API. You found my thread already... my current conclusion: The most webservices are overstating the scope of a bookmark storage solution in the cloud. The most are trying to build a social community around "bookmarks". To many features, toooo slow and colourful webservices, mostly only working on specific webbrowsers, only with a special installed "toolbar" or they concentrating themself on own mobile applications (for iPhone and Android - not more). To make it worse, some are storing the bookmarks only "public" - creating rankings who has the most, who has which stored and what is the "most popular" website.

I'm really afraid of the login problem at google. It is not official and as soon as google change their login method my whole work was for nothing. :(

I read google is using the same webservice to sync and store the bookmarks from their chromebook to "the cloud". Do you know something about this?

If I had the skills I would build such kind of website myself. It shouldn't be to hard. 3 Tables in a database, "user, bookmarks and a assistance table for a tag cloud"... [sigh]

Quote:

Originally Posted by Estel (Post 1122187)

I tried to check if Gmail Drive is Open source (so you could examine login method they use), but unfortunately, license isn't stated anywhere, and it's being released as binaries, so probably it's freeware closed source. Still, I've heard of phpGmailDrive for GNU (tool working just like gmail drive, but linking many accounts into one virtual big account) - You may want to search for it, it may be Open Source.

Hope it helps and is at least relevant to Your troubles ;)

Thanks for the hint, the Source is stored over here at SourceForge: Link

I found the relevant parts in the source (search for "GM_LNK_LOGIN"), but php is very different compared to C++ and java... from my current point of view it wont help me. It uses the same links to login. But what I find surprisingling good are the release dates of the files: the most files are from 2005 and the last release 2007. So, if this application still works google doesn't change the login method that often... [finger crossing] :rolleyes:

Have you used this application yourself?

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

I 100% agree with You, about currently existing bookmark services. The "social" things like ranking, who got more bookmarks sounds especially pathetic and ridiculous...

Unfortunately, I have no clue about chromebook (don't use anything chrome related, and don't plan to). I also haven't used phpGmailDrive - I used GmailDrive (basic) some time ago, and I remember downloading new version every time I tried to upload/download my files, because old version was already non-working :p I used it as some kind of p2m, to share my bootlegs video recordings.

Unfortunately, You're talking to non-coder here (although I'm very interested in coding, and hope to get any skills sooner or later...), so, probably, You're much more skilled than me.

---

As I said, I'm not coder, so i may be wrong, but it seems to me, that most time-effective thing You could do, is trying to write some small, but useful "bookmark site" - Honestly, I would use that myself. Even, if it ends up very niche, it seems more stable for future, than relying on non-standardized google bookmarks login. At least, there is no risk, that your work will be "wasted", when google change something. I agree, that writing Your own service for bookmarks may be more work (if You have not experience with it, otherwise, I agree that it should be simple), but You're 100% sure it's not going to be wiped out by other company decisions.

Sorry, for not being very helpful here

/Estel

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

Quote:

Originally Posted by Estel (Post 1122339)

Unfortunately, I have no clue about chromebook (don't use anything chrome related, and don't plan to).

I tried now the Chrome Browser for Windows. There is a sync function where I entered my google account data, but it doesn't import my google bookmarks... have not tested it that much but it seems it stores the Chrome bookmarks somewhere else.

And Google Bookmarks is still in a Beta stage. :mad:

Oh well... who knows if google wouldn't shut it down in several weeks like they do with google notes at the moment.

[own website]

Quote:

Originally Posted by Estel (Post 1122339)

but You're 100% sure it's not going to be wiped out by other company decisions.

Except a big company would buy it for a good offer. ;)

Sadly I would need to learn to much other stuff before I would be able to setup such a site... and I'm not sure about the legal requirements. What if someone has patented the idea of a site for bookmarks?

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

You can't patent such idea - you can patent certain way to achieve it (i.e. certain security methods, etc). Anyway, thankfully, there are many Open (licensed following terms of Open Software = can't be patented by some sh|t company) methods, that can be used without fear of patents. MySQL, PHP, just to name few.

Still, honestly, I understand that You're not willing to start such a project. It's quite a huge task, and Your starting idea was to write Open Application for using external possibilities, not creating new ones. On the other hand, many great projects with many people involved, started as small ideas or quasi-unrelated projects... ;)

/Estel

Re: Qt: fake the behavior of a webbrowser during logging in into Google (QNetworkAccessManager and Cookies)

Quote:

Originally Posted by Estel (Post 1122376)

Still, honestly, I understand that You're not willing to start such a project. It's quite a huge task, and Your starting idea was to write Open Application for using external possibilities, not creating new ones. On the other hand, many great projects with many people involved, started as small ideas or quasi-unrelated projects... ;)

At least I wouldn't want to do it alone. I know some html, have some SQL skills, I know some ASPX stuff and I'm this way able to write some small applications for SharePoint using DataView's. But a own SharePoint server (even to rent) is expensive, I don't know if access to the tables using external Qt applications is possible and I'm not a professional so I have to use mySQL and php... hmm... but I have no skills running my own server und I won't like to spend a lot of money to build something like this and pay for the needed infrastructure, only to publish a application for free or perhaps $0.99 at the OVI store.

I'm sure, the revenue would be to small... but I guess I have to think about this quite a little bit. This sounds to huge for me at the moment. Perhaps I have a different idea. Don't know at the moment... :(