Ask Question Forum:

Model Library:2025-02-08 Updated:A.I. model including DeepSeek is online for auto reply question page

Show

ASK RECENT

Curl access https/ssl site with password encryption

Asked by duncanb7

at 2024-02-14 14:55:47

Point:500 Replies:9 POST_ID:828884USER_ID:11059

Topic:

PHP Scripting Language;Apache Web Server;Secure Socket Layer (SSL) & HTTPS

During these two years, I've used the following php curl code to access https/ssl site that is successful and no issue at all, I could collect anything I want after logged into https site.

In the code, I have extracted all hidden keys and cookie info from login form and then
post submit with those key to final https security check sites with my login name and password. The key/query string for https site setup by php code is matched to what I see
on one of proxy debugger such as Chales or Fidder. And cookie transfer from one
site to next https site is also okay and saved it into one cookie file in my code.

But Now today, I could not access the https site, finally found the reason from proxy debugger,the password is encryption with 128bit . At the beginning or last two days, the password is not encrypted that is why I could access the site in last two days.

The question is what I should do:

Question-1: Who is responsible to do client password encryption on https site? Is it
from browser (Chrome or IE) with the trusted ssl cert or from javascript code in the login form ?

Question-2: How can I do the exact password encryption on my php code at my server
that is exactly same as it is done my browser so that I could pass the
security check for https site through php curl code

Question-3: there is a lots of option for curl_setopt about ssl option setup such as
CURLOPT_CAINFO, CURLOPT_SSLKEY, CURLOPT_SSH_HOST_PUBLIC_KEY_MD5, CURLOPT_SSH_PUBLIC_KEYFILE, CURLOPT_SSH_PRIVATE_KEYFILE ,etc... from http://hk1.php.net/curl_setopt. Those option is helpful to set password
encryption on my server ? Do I need to copy my https trusted ssl cert on my browser
and then used it with curl code ? How ?

I have read similar article at
http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_25111542.html

but that article is not dealing for the case when the password is with 128-bit encryption on https site, any other similar and good article is suggest and now I don't know what I should do next

Please advise
Duncan

<?phpchdir(dirname(__FILE__));error_reporting(E_ALL);$cookiefile="cookiefile.txt";@unlink($cookiefile);$handle=fopen($cookiefile, 'w+');login($cookiefile);function login($cookiefile){$url="https://example.com/login.htm?lang=en_US";  //-login-form$file_url=open_https_url($url,$cookiefile,0,0);  //curl and its setup;while($file_url==false){echo "Login Form-1 again....";$file_url=open_https_url($url,$cookiefile,0,0);}//-Extract all hidden key or cookie from the login form first before submit$data=explode("requestId = '",$file_url);$r=substr($data[1],0,stripos($data[1],"'"));$data=explode('name="t" value="',$file_url);$t=substr($data[1],0,stripos($data[1],'"'));$data=explode('name="s" value='."'",$file_url);$s=substr($data[1],0,stripos($data[1],"'"));//-Setup postfields for the new https site$postfields="lang=en_US&username=mylogin&password=mypassword";$postfields=$postfields."&r=".$r."&t=".$t."&s=".$s;//-Submit the form to next new https site with all key and cookie ready$url="https://example.com/securitycheck.htm";$file_url=open_https_url($url,$cookiefile,$postfields,1);$temp=stripos($file_url,"Welcome mylogin");while ($temp==false ||$temp==0 ){echo "Submit https site-2 again......";$file_url=open_https_url($url,$cookiefile,$postfields,1);$temp=stripos($file_url,"Welcome mylogin");}echo "Success Login initiation";//if the page has "Welcome mylogin" word that proved https access from curl is successful, otherwise looping;return "success";}function open_https_url($url,$cookiefile,$postfields,$postenable) {$ch = curl_init();curl_setopt($ch, CURLOPT_URL, $url);$header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";$header[] = "Cache-Control: max-age=0";$header[] = "Connection: keep-alive";$header[] = "Keep-Alive: 300";$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";$header[] = "Accept-Language: en-us,en;q=0.5";$header[] = "Pragma: ";curl_setopt( $ch, CURLOPT_URL,            $url  );curl_setopt( $ch, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  );curl_setopt( $ch, CURLOPT_HTTPHEADER,     $header  );curl_setopt( $ch, CURLOPT_REFERER,        'http://www.google.com'  );curl_setopt( $ch, CURLOPT_ENCODING,       'gzip,deflate'  );curl_setopt( $ch, CURLOPT_AUTOREFERER,    TRUE  );if ($postenable==1){curl_setopt($ch, CURLOPT_POST, 1);curl_setopt($ch, CURLOPT_POSTFIELDS,$postfields);}curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/ca.p7b");curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);curl_setopt($ch, CURLOPT_HEADER, 0);curl_setopt($ch, CURLOPT_COOKIEJAR, $cookiefile);curl_setopt($ch, CURLOPT_COOKIEFILE, $cookiefile);curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);$result =curl_exec($ch);$err = curl_errno($ch);$inf = curl_getinfo($ch);//print_r($inf);if ($result === FALSE){echo " curl fail: $url CURL_ERRNO=$err ";var_dump($inf);}curl_close ($ch);return $result;}?>

1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:28:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47:48:49:50:51:52:53:54:55:56:57:58:59:60:61:62:63:64:65:66:67:68:69:70:71:72:73:74:75:

Select All Code

Expert: Ray Paseur replied at 2024-02-15 08:05:35

Thanks for the points and best of luck with the project, ~Ray

Author: duncanb7 replied at 2024-02-15 07:28:25

Thanks for your detailed reply in this thread.
I will try Node.js first on my server.
If it is not successful, I will switch to do it on my PC browser
for data collection daily to remote https site with using VBA code
that must work but it is not good to do it on my PC instead of my remoter server

Memo for me: And third party SSL cert , its encyrption, public or private keys are other issues and not related to this thread topic

Have a nice day

Duncan

Expert: Ray Paseur replied at 2024-02-15 07:12:11

The first part is certainly correct. Not sure what node.js can do for you (I have not seen the login web page) but I'll take that on faith :-)

Author: duncanb7 replied at 2024-02-15 06:55:57

Dear Ray,

thanks for your reply and attached articles.
Now it seem It is better for me. if the following what I said it is correct.

php curl request or client PC browser request for https is nothing special and they
don't do any data info encryption during transmission. The transmission
encryption on the request data is done by SSL engine, for example, on
my server, Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0-fips mod_bwlimited/1.4 PHP/5.3.21,

the request data encryption is done by my server openSSL enginner or module (not by php curl code) when https request is enabled. And the remote site of https , pulishers'
server will de-crypte the data (encrypted from my server) from pubishers' SSL engine or module. Last words, both algorithm for encryption from my server and decryption from publisher https site are recognized and is one of standard of SSL engine with cryption algorithm. In othe words, there is no doubt my server could communicate with the
https site with SSL data encryption at both sides(my server and remote https site)

I will close this thread if what I said in this post is correct or close to correct

And the additional password encryption API javascript function is found on the remote site
login form page mentioned at top post , and probably I need Node.js help on my server to do the additional password encryption before curl https request on my server
that will simulate the same API action on login form with https request on my local PC browser such as on Chrome.

Please advise
Duncan

Accepted Solution

Expert: Ray Paseur replied at 2024-02-15 06:10:10

500 points EXCELLENT

The transmission is encrypted. The clear-text data is presented to the script (just as it is presented to the browser). This explanation from the security firm, Symantec, seems to be on point:
http://www.symantec.com/business/support/index?page=content&id=TECH180521

Nothing special is required by either the browser client or the cURL client. Please see http://www.laprbass.com/RAY_curl_get_example.php where we read the Twitter web site over HTTPS.

<?php // RAY_curl_get_example.phperror_reporting(E_ALL);// DEMONSTRATE THE BASICS OF CURL// SOMETHING LIKE RAY_curl_get_example.php?url=http://twitter.com// YOU COULD HAVE SOMETHING LIKE THIS$url = isset($_GET["url"]) ? $_GET["url"] : 'http://twitter.com';// BUT SINCE IT IS ON MY SERVER, I HAVE HARD-CODED THIS$url = 'https://twitter.com/RayPaseur';// TRY THE REMOTE WEB SERVICE$htm = my_curl($url);// SHOW THE WORK PRODUCT OR BARK OUT ERROR MESSAGESecho "<pre>";echo PHP_EOL . '<strong>' . $url . '</strong>';echo PHP_EOL . htmlentities($htm);echo PHP_EOL;// A FUNCTION TO RUN A CURL-GET CLIENT CALL TO A FOREIGN SERVERfunction my_curl( $url, $timeout=5, $error_report=TRUE){    $curl = curl_init();    // HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE    $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";    $header[] = "Cache-Control: max-age=0";    $header[] = "Connection: keep-alive";    $header[] = "Keep-Alive: 300";    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";    $header[] = "Accept-Language: en-us,en;q=0.5";    $header[] = "Pragma: "; // BROWSERS USUALLY LEAVE THIS BLANK    // SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php    curl_setopt( $curl, CURLOPT_URL,            $url  );    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  ); // ANCIENT HISTORY    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20100101 Firefox/22.0'  );    curl_setopt( $curl, CURLOPT_HTTPHEADER,     $header  );    curl_setopt( $curl, CURLOPT_REFERER,        'http://www.google.com'  );    curl_setopt( $curl, CURLOPT_ENCODING,       'gzip,deflate'  );    curl_setopt( $curl, CURLOPT_AUTOREFERER,    TRUE  );    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE  );    curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE  );    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout  );    // RUN THE CURL REQUEST AND GET THE RESULTS    $htm = curl_exec($curl);    // ON FAILURE HANDLE ERROR MESSAGE    if ($htm === FALSE)    {        if ($error_report)        {            $err = curl_errno($curl);            $inf = curl_getinfo($curl);            echo "CURL FAIL: $url TIMEOUT=$timeout, CURL_ERRNO=$err";            var_dump($inf);        }        curl_close($curl);        return FALSE;    }    // ON SUCCESS RETURN XML / HTML STRING    curl_close($curl);    return $htm;}

1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:28:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47:48:49:50:51:52:53:54:55:56:57:58:59:60:61:62:63:64:65:66:67:68:69:70:71:72:73:74:75:

Select All Code

Author: duncanb7 replied at 2024-02-15 04:56:24

Dear Ray,

I have No question on the publisher for additional encryption on password by javascript
API code in login form page.
But my concept to curl on HTTPS/SSL is still confusing even it seems simple.

1. There is nothing about HTTPS/SSL that will interfere with cURL. A cURL request is just like a browser request. It can be made over HTTP or HTTPS and both will work correctly.

if you said correct on pt-1, so the info sent by curl request is NOT encrypted/protected ? Is it dangerous ?
As I know when I access https/SSL site on my Chrome browser, the information
I submitted will be encrypted by the browser with the trusted SSL cert before
sending the info data to remote https site. Right ? and the pulisher will do
de-cryption on those info to collect final data at its side. If so, why we don't
need to do the same encryption for curl request on my server to achieve
the same action ?

Hope you understand what I am confusing on difference between curl request
and client browser request for HTTPS site

Duncan

Expert: Ray Paseur replied at 2024-02-15 04:33:49

1. There is nothing about HTTPS/SSL that will interfere with cURL. A cURL request is just like a browser request. It can be made over HTTP or HTTPS and both will work correctly.

2. "Why the company needs to do two times encryption on password..." That is a question you should direct to the publisher of the web site. My answer can only be speculative, but it would be "To reduce the risk of unwanted automated access to the web site."

Author: duncanb7 replied at 2024-02-15 04:01:31

Dear Ray,

thanks for your reply. I am getting confusing with https/SSL access. There is a few question
for two cases. Hope you can kindly answer this that I am appreciated

Case-1: successful logged in last years
=======================================
Supposed on browser will do information encryption according to the site SSL cert for https/SSL access, Right ? but why I could access that https site without password encryption in last two years on my server in which I just input php code for postfield as $postfields="lang=en_US&username=mylogin&password=mypassword"; ?
Whether php curl code NO need to do any encryption on password or login username for accessing https site succesfully ?

Like the case or code in
http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_25111542.html in which there is NO any password
encryption on php curl code for accessing http/SSL site, Why if it is https site ?

Case-2 Fail logged today
========================
On Charles proxy debugger, I just only found the password is encrypted but
other query string or form hidden key or username all are NOT encrypted. So
you might be right there is API javascript code in the login form to encrypt
password to 128 bit before accessing https. Why the company needs to do
two times encryption on passsword ? one from API javascript code on login page and second is from origin SSL browser encryption

Please help to point out my concept mistake if any , thanks

Duncan

Expert: Ray Paseur replied at 2024-02-15 03:22:54

I got a neglected question alert on this one, and I have a bad feeling about the approach you were using. It looks like you may have been "cut off." Here is what I would do:

The first thing I would do is contact the publisher of the web site and tell them about the trouble you are having because of their change. They may have a formal API that you can use to get the data you want, and if they do, that will be a better solution than trying to use an automated login. APIs are typically licensed, metered and versioned, so the publisher has a "contract" with the audience and will not break an API without formal notice, probably including deprecation. When you access a web site without an API, the "scraper scripts" often get broken by even small changes in the HTML document, so an API is the way to go. You may be required to pay a license fee to use the API.

It is possible that the publisher made this change to prevent automated access to its data. A few years ago, Google made a change like this, loading the DOM with JavaScript instead of producing clear-text HTML documents. This frustrated a lot of people who had been scraping Google pages (without permission) but it helped Google keep control of its data and preserve the value in its web service.

&nbps;

About Us|Service|Pricing

即時新聞:↓

即時新聞:↑

On Balance Volume (OBV):

Ask Question Forum:

Curl access https/ssl site with password encryption

Video♬:↓

Your Information

Members/Profolio