BN_GENERATE_PRIME(3) | Library Functions Manual | BN_GENERATE_PRIME(3)

# NAME

`BN_is_prime_ex`, `BN_is_prime_fasttest_ex`, `BN_generate_prime_ex`, `BN_GENCB_call`, `BN_GENCB_new`, `BN_GENCB_free`, `BN_GENCB_set`, `BN_GENCB_get_arg`, `BN_GENCB_set_old` — generate primes and test for primality

# SYNOPSIS

library “libcrypto”
`#include <openssl/bn.h>`

int
`BN_is_prime_ex`(const BIGNUM *a, int nchecks, BN_CTX *ctx, BN_GENCB *cb);

int
`BN_is_prime_fasttest_ex`(const BIGNUM *a, int nchecks, BN_CTX *ctx, int do_trial_division, BN_GENCB *cb);

int
`BN_generate_prime_ex`(BIGNUM *ret, int bits, int safe, const BIGNUM *modulus, const BIGNUM *remainder, BN_GENCB *cb);

int
`BN_GENCB_call`(BN_GENCB *cb, int state_code, int serial_number);

BN_GENCB *
`BN_GENCB_new`(void);

void
`BN_GENCB_free`(BN_GENCB *cb);

void
`BN_GENCB_set`(BN_GENCB *cb, int (*cb_fp)(int, int, BN_GENCB *), void *cb_arg);

void *
`BN_GENCB_get_arg`(BN_GENCB *cb);

Deprecated:

void
`BN_GENCB_set_old`(BN_GENCB *cb, void (*cb_fp)(int, int, void *), void *cb_arg);

# DESCRIPTION

`BN_is_prime_ex`() and `BN_is_prime_fasttest_ex`() test whether the number a is prime. In LibreSSL, both functions behave identically and use the Baillie-Pomerance-Selfridge-Wagstaff algorithm combined with checks Miller-Rabin rounds. The do_trial_division argument is ignored.

It is unknown whether any composite number exists that the Baillie-PSW algorithm misclassifies as a prime. Some suspect that there may be infinitely many such numbers, but not a single one is currently known. It is known that no such number exists below 2^64.

In order to reduce the likelihood of a composite number passing the primality tests `BN_is_prime_fasttest_ex`() and `BN_is_prime_ex`(), a number of rounds of the probabilistic Miller-Rabin test is performed. If checks is positive, it is used as the number of rounds; if it is zero or the special value `BN_prime_checks`, a suitable number of rounds is calculated from the bit length of a.

If `NULL` is passed for the ctx argument, these function allocate a BN_CTX object internally when they need one and free it before returning. Alternatively, to save the overhead of allocating and freeing that object for each call, the caller can pre-allocate a BN_CTX object and pass it in the ctx argument.

`BN_generate_prime_ex`() generates a pseudo-random prime number of at least bit length bits and places it in ret. Primality of ret is tested internally using `BN_is_prime_ex`(). Consequently, for bits larger than 64, it is theoretically possible that this function might place a composite number into ret; the probability of such an event is unknown but very small.

The prime may have to fulfill additional requirements for use in Diffie-Hellman key exchange:

- If modulus is not `NULL`, a prime is generated that fulfills the condition ret % modulus = remainder. If the remainder argument is `NULL`, 1 is used as the desired remainder.
- If the safe argument is non-zero, a safe prime is generated, that is, (ret - 1)/2 is also prime.

If cb is not `NULL`, it is used as follows:

- `BN_GENCB_call`(cb, 0, serial_number) is called after generating a potential prime number.
- The state_code of 1 is reserved for callbacks during primality testing, but LibreSSL performs no such callbacks.
- When safe is non-zero and a safe prime has been found, `BN_GENCB_call`(cb, 2, serial_number) is called.
- The callers of `BN_generate_prime_ex`() may call `BN_GENCB_call`() with other values as described in their respective manual pages; see SEE ALSO.

In all cases, the serial_number is the number of candidates that have already been discarded for not being prime; that is, serial_number is 0 for the first candidate and then incremented whenever a new candidate is generated.

`BN_GENCB_call`() calls the callback function held in cb and passes the state_code and the serial_number as arguments. If cb is `NULL` or does not contain a callback function, no action occurs.

`BN_GENCB_new`() allocates a new BN_GENCB object.

`BN_GENCB_free`() frees cb. If cb is `NULL`, no action occurs.

`BN_GENCB_set`() initialises cb to use the callback function pointer cb_fp and the additional callback argument cb_arg.

The deprecated function `BN_GENCB_set_old`() initialises cb to use the old-style callback function pointer cb_fp and the additional callback argument cb_arg.

# RETURN VALUES

`BN_is_prime_ex`() and `BN_is_prime_fasttest_ex`() return 0 if the number is composite, 1 if it is prime with a very small error probability, or -1 on error.

`BN_generate_prime_ex`() returns 1 on success or 0 on error.

`BN_GENCB_call`() returns 1 on success, including when cb is `NULL` or does not contain a callback function, or 0 on error.

`BN_GENCB_new`() returns a pointer to the newly allocated BN_GENCB object or `NULL` if memory allocation fails.

The callback functions pointed to by the cb_fp arguments are supposed to return 1 on success or 0 on error.

`BN_GENCB_get_arg`() returns the cb_arg pointer that was previously stored in cb using `BN_GENCB_set`() or `BN_GENCB_set_old`().

In some cases, error codes can be obtained by ERR_get_error(3).

# SEE ALSO

BN_new(3), DH_generate_parameters(3), DSA_generate_parameters_ex(3), RSA_generate_key(3)

# HISTORY

`BN_generate_prime_ex`(), `BN_is_prime_ex`(), `BN_is_prime_fasttest_ex`(), `BN_GENCB_call`(), `BN_GENCB_set_old`(), and `BN_GENCB_set`() first appeared in OpenSSL 0.9.8 and have been available since OpenBSD 4.5.

`BN_GENCB_new`(), `BN_GENCB_free`(), and `BN_GENCB_get_arg`() first appeared in OpenSSL 1.1.0 and have been available since OpenBSD 6.3.

June 8, 2025 | Debian