I'm trying to decrypt encryted sni record using SSL_ESNI_dec function which is in ssl/esni.c line 2475.
The problem is in ssl/esni.c line 2648 ssl_md function.
2475 unsigned char *SSL_ESNI_dec(SSL_ESNI *esni,
2476 size_t client_random_len,
2477 unsigned char *client_random,
2478 uint16_t curve_id,
2479 size_t client_keyshare_len,
2480 unsigned char *client_keyshare,
2481 size_t *encservername_len)
2482 {
...
2647 const SSL_CIPHER *sc=cs2sc(esni->ciphersuite);
2648 const EVP_MD *md=ssl_md(sc->algorithm2);
2649 esni->Zx_len=0;
2650 if (esni->Zx!=NULL) OPENSSL_free(esni->Zx);
2651 esni->Zx=esni_hkdf_extract(esni->Z,esni->Z_len,&esni->Zx_len,md);
I checked that sc has right value but ssl_md always return NULL value. I step in to ssl_md(ssl/ssl_ciph.c line 587) while debuging. ssl_md receive argument named idx and returns ssl_digest_methods[idx] but ssl_digest_methods's value is initialized as NULL in ssl/ssl_ciph.c line 117 so ssl_md will always return NULL value.
117 static const EVP_MD *ssl_digest_methods[SSL_MD_NUM_IDX] = {
118 NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
119 };
...
587 const EVP_MD *ssl_md(int idx)
588 {
589 idx &= SSL_HANDSHAKE_MAC_MASK;
590 if (idx < 0 || idx >= SSL_MD_NUM_IDX)
591 return NULL;
592 return ssl_digest_methods[idx];
593 }
It doesn't matter md's value is NULL or not but in ssl/esni.c line 2651 esni_hkdf_extract tries to reference md->prov even md is NULL so program crashes with SIGSEGV. Does esnistuff/testserver.sh not using SSL_ESNI_dec while running?
I'm trying to decrypt encryted sni record using SSL_ESNI_dec function which is in ssl/esni.c line 2475.
The problem is in ssl/esni.c line 2648 ssl_md function.
I checked that sc has right value but ssl_md always return NULL value. I step in to ssl_md(ssl/ssl_ciph.c line 587) while debuging. ssl_md receive argument named idx and returns ssl_digest_methods[idx] but ssl_digest_methods's value is initialized as NULL in ssl/ssl_ciph.c line 117 so ssl_md will always return NULL value.
It doesn't matter md's value is NULL or not but in ssl/esni.c line 2651 esni_hkdf_extract tries to reference md->prov even md is NULL so program crashes with SIGSEGV. Does esnistuff/testserver.sh not using SSL_ESNI_dec while running?