pyca/cryptography backwards incompatible since 35.0.0 / faulty country data?

[jsuesse]

Disclaimer:
I think this is lowest priority / documentation.
I did not modify any script-related file when this appeared.
Using no further parameters I was able to verify my German QR-code before.
./verify_ehc.py --image qr.jpeg

First appearance

I was playing around with --certs-from and --save-certs using different combination of countries, separately choosing json or cbor.
I ran ./verify_ehc.py --certs-from AT,DE,SE,NL --save-certs trust_list.json successful.
I got errors running ./verify_ehc.py --certs-from AT,DE,SE --save-certs trust_list.json (no NL).
Error:

Traceback (most recent call last):
  File "./verify_ehc.py", line 2435, in <module>
    main()
File "./verify_ehc.py", line 2373, in main
    save_certs(certs, certs_path, args.allow_public_key_only)
  File "./verify_ehc.py", line 1879, in save_certs
    'usage': sorted(get_key_usage(cert)),
  File "./verify_ehc.py", line 2022, in get_key_usage
    ext_key_usage = cert.extensions.get_extension_for_oid(ExtensionOID.EXTENDED_KEY_USAGE)
ValueError: error parsing asn1 value: ParseError { kind: EncodedDefault, location: ["BasicConstraints::ca"] } 

Investigation

Downloading JSON from all countries

I ran ./verify_ehc.py --certs-from XX --save-certs trust_list.json
with X being all countries one by one. (AT DE SE NL GB CH FR NO)

Failed with "No Token" (This is expected, see Readme)

CH FR NO

Failed with Type Error

DE, SE, AT
no json file was created

Successful

GB NL

I then tried verifying my qr-code with these trust_lists.
trust_GB:

KeyError: 'Key ID not found in trust list

ok - expected.

trust_NL:

 TypeError: can't compare offset-naive and offset-aware datetimes

that's weird! something for another time, moving on...

Downloading CBOR from all countries

I was wondering why ./verify_ehc.py --image qr.jpeg still worked.
Upon reading the README.md once again I realized JSON could be at fault.
I then did the same as before, collecting certs for all countries one by one but using cbor.

Successful

DE SE AT

faulty (expected)

NL GB
got a bunch of these or similiar looking.
ERROR: Cannot store entry 840d7ea7010ec422 / hA1+pwEOxCI= in CBOR trust list: cannot serialize certificate from public-key only
these countries provided JSON already. see README.md

no token (expected)

FR CH NO

Conclusion

might not be correct

DE, SE, AT provide CBOR
NL GB provide JSON
CH FR NO need token. I didn't bother to get them.

In my humble opinion this was not clearly stated in README.
Proposal: Include which countries provide which format in README.

Could you please have a look?
Thank you so much!

[panzi]

The datetime error is probably a bug in my code. Have to take a look at it.

DE, SE, AT provide CBOR
NL GB provide JSON

This is not correct. Every country provides its own weird format. I started with the AT trust list (I'm from Austria) and thus I just took their format to save trust lists. Later I also made up a JSON format that is useful when used with the WebCrypto browser API. I don't understand how you came to the conclusion that these formats depend on the trust list source?

To write multiline code here on GitHub enclose your code in triple backticks:

```
your code
```

You can also specify the language:

```Python
print("hello world")
```

[panzi]

Can you try if the datetime timezone issue still exists with what I've pushed just now?

[panzi]

Running ./verify_ehc.py --certs-from AT,DE,SE --save-certs trust_list.json did not produce an error for me. What Python and cryptography library versions do you have installed? For me:

$ python3 --version
Python 3.9.7
$ pip3 show cryptography
Name: cryptography
Version: 3.4.7

[jsuesse]

This is not correct. Every country provides its own weird format. I started with the AT trust list (I'm from Austria) and thus I just took their format to save trust lists. Later I also made up a JSON format that is useful when used with the WebCrypto browser API. I don't understand how you came to the conclusion that these formats depend on the trust list source?

yea, I didn't have time trying to make sense of it.

Can you try if the datetime timezone issue still exists with what I've pushed just now?

working!

Running ./verify_ehc.py --certs-from AT,DE,SE --save-certs trust_list.json did not produce an error for me. What Python and cryptography library versions do you have installed? For me:

This is my linux kubuntu VM.

Python 3.8.10

Name: cryptography
Version: 35.0.0
[..]
Home-page: https://github.com/pyca/cryptography

eh.. not quite right, is it?

This is my windows:

Python 3.6.5
cryptography
Version: 3.4.7

If I recall correctly, it all worked on the Windows environment before - will update this comment soon
Update: Indeed, smooth on windows. Timezone change also works.

[panzi]

Version: 35.0.0
eh.. not quite right, is it?

Looks like they did a Java and jumped from 3.4 to 35.0 (Java jumped from 1.4 to 5.0). But it would be weird of a newer version would fail to parse something an older version can parse. (Talking about the asn1 error.)

[jsuesse]

Source

BACKWARDS INCOMPATIBLE: Invalid ASN.1 found during X.509 parsing will raise an error on initial parse rather than when the invalid field is accessed.

Edit: same/similar Error

[panzi]

I've added some measures against broken ASN.1 encodings. It now prints an error message (for the Finnish certificate) and handles such broken certificates as if they wouldn't have any extensions. Does this fix everything for you?

[jsuesse]

Yes.
I now get the expected
ERROR: Parsing extended key usage: error parsing asn1 value: ParseError { kind: EncodedDefault, location: ["BasicConstraints::ca"] }
A Json-File exists and I was able to verify my qr-code with it.
Thanks!