[squid-users] Gather POST request on HTTPS traffic?

[roee klinger]

Thanks for the replay Amos,

> You will notice when configuring SSL-Bump that you must install signing
> CA certificates used by your proxy into the clients software.
>

I understand, this is something I missed apparently.

Sometimes I am using proxies for scraping which detect if the scraping is
successful and run the request
from a different proxy if it isn't, they even go as far as automatically
solving captcha's for the client or changing
content on the page, I am pretty new to this but these feature seems
impossible to me on HTTPS connections
without having access to the client's machines.

Is there something I am missing or misunderstanding?
I cannot seem to find a good place to start reading about this.

Thanks.




On Tue, Nov 17, 2020 at 3:22 PM Amos Jeffries <squid3 at treenet.co.nz> wrote:

> On 18/11/20 1:41 am, roee klinger wrote:
> > Hey Amos,
> >
> > Thanks for your response, I will try to implement this today and check
> > if I can get the data I am looking for.
> >
> > I do however have a few questions regarding this approach:
> > 1. If I understand the docu currently, then the server is getting a
> > response which is identical to the client, meaning the server should not
> > detect anything unusual? The last thing I want is for Instagram to
> > detect something unusual and ban our clients Instagram accounts.
>
> That depends on the what you configure. Interception is always
> detectable, though most services have limited detection (if they care at
> all).
>
>
> > 2. You said I will need to figure out a way to identify accounts, in
> > Chrome the requests contain the info for both the accounts performing
> > the action and the account receiving the action, should I see the same
> > in these requests?
>
> Yes. That is what I mean by personal data *will* be gathered.
>
>
> > 3. By “personal” data we are referring to data generated by our clients
> > accounts, which are paying and willing for us to collect it to improve
> > our service, of course it will also contain data on the account which
> > they are performing the actions on, but this is not something that is
> > not visible on the Instagram app, is there anything else I should be
> > aware of that might be a privacy issue?
>
>
> That definition confirms the false nature of "No personal account data
> will be gathered." - having permission to gather does not negate the
> existence of gathering.
>
> Just make sure you have a real lawyers opinion / advice on the situation
> details.
>
>
> > 4. While this is great for my use case, is this something I should be
> > aware of when using outside proxies on our machine? Can any proxy
> > service simply decrypt and log our personal data? Seems like a security
> >   vulnerability I should be aware of.
> >
>
> You will notice when configuring SSL-Bump that you must install signing
> CA certificates used by your proxy into the clients software. Without
> that CA trust you cannot bump.
>
> The possibility of bumping (or lack of) is true for any intermediary
> software.
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20201117/ce781cc2/attachment.htm>