Subject: Re: [Blockchain-interop] Gatweay Crash Recovery Discussion #1

From: Thomas Hardjono <[email-masked]>
Date: Wed, 2 Dec 2020 00:17:52 +0000

Thanks Rafael.

Inline:

>>> > -- Which mode (self-healing mode, or primary-backup mode) do you
>>> >>> > recommend?  (Which one would be the simplest approach for now, and
>>> > what assumptions would we need to make).
>>> 
>>> The self-healing mode is simpler, as the same machine eventually
>>> recovers, continuing its operations since the latest log entry. It does
>>> not require, in principle, to read from the log storage API. However, we
>>> are assuming it eventually recovers, and while this happens the system
>>> is down, prejudicing availability.

For this scenario, there may need to be some assumptions about the gateway node.  

For example, (i) the gateway recovers to 100% without any internal losses (e.g. loss of private-keys); (ii) that the SSL connection has a longer life-time than the duration of crash (unavailability); etc.

For short-duration unavailabilities, would the gateway pick-up (restart) from the last message before crash, or does it start from the beginning of the Phase?

Best

-- thomas --

> ________________________________________
> From: Blockchain-interop [[email-masked]] on
> behalf of Rafael Belchior
> [rafael.belchior=[email-masked]]
> Sent: Monday, November 30, 2020 11:49 AM
> To: [email-masked]
> Subject: [Blockchain-interop] Gatweay Crash Recovery Discussion #1
>
> Dear All,
> Attached, the slides of the first discussion on the crash recovery
> mechanism for gateways, that took place during the last meeting.
>
>
> Cheers,
>
> --
> Rafael Belchior
> Ph.D. student in Computer Science and Engineering, Blockchain - Técnico
> Lisboa
> https://rafaelapb.github.io/
> https://www.linkedin.com/in/rafaelpbelchior/

--
Rafael Belchior
Ph.D. student in Computer Science and Engineering, Blockchain - Técnico
Lisboa
https://rafaelapb.github.io/
https://www.linkedin.com/in/rafaelpbelchior/

---

From: Rafael Belchior <[email-masked]>
Date: Wed, 02 Dec 2020 13:06:36 +0000

Thomas,
Yes, those assumptions are reasonable; i) we can also consider the case 
where a public-private key pair is lost, and thus a new pair needs to be 
generated. ii) I think we can assume a new SSL connection is created. Do 
you envision problems with this?

I think it makes sense for the gateway to resume operations from the 
last message before the crash because this mode is blocking, in 
principle - we can resume from the last event.

If anyone thinks this could be improved, please do not hesitate in 
providing feedback :)

Cheers,
Rafael

A 2020-12-02 00:17, Thomas Hardjono escreveu:
> For example, (i) the gateway recovers to 100% without any internal
> losses (e.g. loss of private-keys); (ii) that the SSL connection has a
> longer life-time than the duration of crash (unavailability); etc.
> 
> For short-duration unavailabilities, would the gateway pick-up
> (restart) from the last message before crash, or does it start from
> the beginning of the Phase?
> 
> Best
> 
> 
> -- thomas --
> 
> 
> 
>

---

From: Martin Hargreaves <[email-masked]>
Date: Wed, 2 Dec 2020 14:00:48 +0000

Hi both,

In terms of protocol messaging, how should we support this?

It sounds like, on recovery, the recovered gateway scans its logs and finds a set of uncompleted transactions, then need to send some kind of "recovery message" to its counterparties - offers to continue processing, with a list of transactions and phases in each transaction to pick up from.

The gateways that previously saw it time out as it crashed can then evaluate these and respond as to whether they wish to proceed or back out (or indeed don't recognise) the transactions.

What do you think?

Thanks

Martin

> -----Original Message-----
> From: Blockchain-interop <[email-masked]> On Behalf
> Of Rafael Belchior
> Sent: Wednesday, December 2, 2020 1:07 PM
> To: Thomas Hardjono <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Blockchain-interop] Gatweay Crash Recovery Discussion #1
>
> CAUTION: This email originated from outside of the organisation. Do not click
> links or open attachments unless you recognise the sender and know the
> content is safe.
>
>
> Thomas,
> generated. ii) I think we can assume a new SSL connection is created. Do you
> envision problems with this?
>
> I think it makes sense for the gateway to resume operations from the last
> message before the crash because this mode is blocking, in principle - we can
> resume from the last event.
>
> If anyone thinks this could be improved, please do not hesitate in providing
> feedback :)
>
> Cheers,
> Rafael
>
>
> >>>> recovers, continuing its operations since the latest log entry. It
> >>>> does not require, in principle, to read from the log storage API.
> >>>> However, we
> >>>> are assuming it eventually recovers, and while this happens the
> >>>> system is down, prejudicing availability.
> >
> >> Ph.D. student in Computer Science and Engineering, Blockchain -
> >> Técnico Lisboa https://rafaelapb.github.io/
> > Ph.D. student in Computer Science and Engineering, Blockchain -
> > Técnico Lisboa https://rafaelapb.github.io/
> Lisboa https://rafaelapb.github.io/
> Blockchain-interop mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/blockchain-interop
This message is intended solely for the addressee and may contain privileged and confidential information. If you have received this message in error, please send it back to us, and immediately and permanently delete it. Do not use, copy or disclose the information contained in this message or in any attachment. Quant Network does not guarantee that this email has not been intercepted and amended or that it is virus free.

---

From: Rafael Belchior <[email-masked]>
Date: Mon, 07 Dec 2020 07:17:57 +0000

Hello Martin,
Yes - on the crash recovery protocol we ought to support recovery 
messages that allow establishing to all parties what is the current 
state of an asset transfer.

We can specify this in the "3.3 Recovery procedure" section.

Cheers,
Rafael

A 2020-12-02 14:00, Martin Hargreaves escreveu:
> It sounds like, on recovery, the recovered gateway scans its logs and
> finds a set of uncompleted transactions, then need to send some kind
> of "recovery message" to its counterparties - offers to continue
> processing, with a list of transactions and phases in each transaction
> to pick up from.
> 
> The gateways that previously saw it time out as it crashed can then
> evaluate these and respond as to whether they wish to proceed or back
> out (or indeed don't recognise) the transactions.
> 
> What do you think?
> 
> Thanks
> 
> Martin
> 
>> last
>> providing
> This message is intended solely for the addressee and may contain
> privileged and confidential information. If you have received this
> message in error, please send it back to us, and immediately and
> permanently delete it. Do not use, copy or disclose the information
> contained in this message or in any attachment. Quant Network does not
> guarantee that this email has not been intercepted and amended or that
> it is virus free.

---

From: Thomas Hardjono <[email-masked]>
Date: Mon, 7 Dec 2020 14:06:05 +0000

We may have to consider the resumption at 2 levels, namely:

(a) Asset-transfer protocol (ODAP) resumption (i.e. which step of the transfer protocol does G1 and G2 restart from);

(b) TLS session resumption (i.e. can G1 and G2 re-use the existing TLS parameters, or should they establish a new TLS session).

A sketch would look something like this (assuming G2 crashed):

G2 ---> G1:  Transfer-Resume [SessionID; crash-error-code]

G1 ---> G2:  Transfer-Continue[SessionID; protocol-step-X; TLS-resume]

G2 <---> G1: (re-establish TLS channel)

G2 ---> G1:  (protocol-step-X)

It'd be nice if the recovery protocol can work both ways (reversible), in that it does not matter whether its G1 or G2 that crashes (recovery protocol runs the same).

-- thomas --

________________________________________
From: Rafael Belchior [[email-masked]]
Sent: Monday, December 7, 2020 2:17 AM
To: Martin Hargreaves
Cc: Thomas Hardjono; [email-masked]
Subject: Re: [Blockchain-interop] Gatweay Crash Recovery Discussion #1

Hello Martin,

Cheers,
Rafael

---

From: Rafael Belchior <[email-masked]>
Date: Mon, 07 Dec 2020 16:25:12 +0000

Hey,

I would only consider option (a), with a new TLS session, unless there 
are major reasons for resuming a session.

The protocol we are creating works both ways, but with differences 
depending on if the crashed party is the participant or the coordinator.

Cheers,
Rafael

A 2020-12-07 14:06, Thomas Hardjono escreveu:
> (a) Asset-transfer protocol (ODAP) resumption (i.e. which step of the
> transfer protocol does G1 and G2 restart from);
> 
> (b) TLS session resumption (i.e. can G1 and G2 re-use the existing TLS
> parameters, or should they establish a new TLS session).
> 
> 
> It'd be nice if the recovery protocol can work both ways (reversible),
> in that it does not matter whether its G1 or G2 that crashes (recovery
> protocol runs the same).
> 
> 
> 
> -- thomas --
> 
> 
>>> message before the crash because this mode is blocking, in principle 
>>> -
>>> we can

---

From: Thomas Hardjono <[email-masked]>
Date: Mon, 7 Dec 2020 16:39:41 +0000

Thanks Rafael,

>>> are major reasons for resuming a session.

Yes, this make sense (just assume a new TLS session after a crash).

Best

-- thomas -- 

________________________________________
From: Rafael Belchior [[email-masked]]
Sent: Monday, December 7, 2020 11:25 AM
To: Thomas Hardjono
Cc: Martin Hargreaves; [email-masked]
Subject: Re: [Blockchain-interop] Gatweay Crash Recovery Discussion #1

Hey,

Cheers,
Rafael