Do I understand correctly that what you want, in the end, is a measure of acceleration? Be warned that measuring acceleration in this way is conceptually simple, but fraught with problems of unstable, noisy results. The basic idea is to measure position at two different times. The change in position divided by the change in time is velocity, right? Yes, but a small error in your measurement of time results in a larger error in your velocity calculation. And we're not done. We want to know acceleration, so we have to calculate velocity at two different times then calculate change in velocity divided by change in time to get acceleration. And again, a small error in time results in a large error in calculated acceleration. Combine that with the already large error in velocity, and it's easy to end up with a number for acceleration that is of little value. (Note that this method requires three position data points. It's easy to calculate acceleration directly from those three points, without calculating velocities, but this does not improve the quality of the result.)
The way to reduce the error is to use more than three data points to calculate acceleration. There are lots of ways to perform the calculations, but in the end, you can get arbitrarily clean results by combining an arbitrarily long string of data points in the calculation. The cost is that the calculation becomes more complicated with larger datasets and thus takes more clock cycles to perform. You have not told us the details of your requirements, but I'm thinking you'll need more than one data point per revolution of the motor.
So, my suggestion is to record data more frequently. You need both the position (angle) and the precise time that the measurement was made. I can think of two ways to do that:
1. Use the PWM output on the AS5600 and the RMT module in the ESP32. The RMT module will directly measure the duty cycle of the incoming PWM and dump the results in a memory block (aka buffer.) This is done entirely in hardware, so no attention is required by your code. There are a couple of things about this method that I would need to work out. One of them is deciding when to pull data from the buffer and calculate acceleration. The problem is that the only interrupt that the module can trigger in receive mode is when the end of the message is reached (basically, a time out.) But the pulse train from the AS5600 never stops, so the RMT never detects an end to the incoming message. I suppose you could use a periodic interrupt to stop the RMT, pull data from the buffer, then start the RMT again. The memory block can be configured as a ring buffer, and there is likely information in one of the RMT registers telling where the tail is; where the most recent sample is buffer. That way you could be sure of using only data that was acquired at consecutive time intervals. The other problem is determining precisely the time of each measurement. Really, we only need the time between samples, and that's determined by the rate that the pulses are coming in. Unfortunately, that rate is not well controlled by the AS5600. Maybe use the ESP32 to first measure the frequency of pulses, then collect a bunch of data? I don't know.
2. The other method that I can think of is using the analog output on the AS5600 and the ADC on the ESP32. This way the time between samples is controlled by the ESP32. Assuming you're using a stable clock, the time base can be controlled precisely. So, the basic routine would be to setup an ADC to convert continuously at some configurable rate, and then put a configurable number of samples into a global buffer. I'm not super familiar with ESP32, but I know that it's possible to configure an ADC to write directly to memory using DMA. In this way, all the work is done with hardware so that the processor is free to do other things while data is being collected. I'd have to figure out how, but it must be possible to have the ADC acquire a fixed number of samples, put them in a global array, and then trigger an interrupt. (Look for sample code somewhere.) I would expect this method to give more accurate time values, but less accurate angle values than method (1) above.
So how do you use the data to calculate acceleration? There are a number of ways. The easiest to understand might be to go through the array of position (angle) data and calculate another array of velocity values, one for each pair of angles. Then go through the velocity data and calculate another array of accelerations. Maybe then average all of the accelerations into one number. You might want to go through the position and velocity arrays and perform some kind of smoothing. That might help if the data is really noisy, but my intuition is that it wouldn't be very useful.
Another way that is a little more complicated conceptually, but would give better results, is to fit a straight line to the velocity data. The least squares line fit returns two numbers: slope and offset. The slope (of velocity vs time) is acceleration. Least squares fitting sounds scary, but there are libraries that could be used.
Or, we can eliminate the intermediate velocity calculations and fit a second order polynomial to the raw position data. For a second order fit, the least squares routine returns three numbers. The third one is the acceleration (really 1/2 the acceleration.) This can be computationally expensive, and the expense goes up fast with the number of data points involved. But this method will make the best possible use of the data. In fact, this the go-to method for analyzing noisy data. Doing it in real time may or may not be crazy depending on your processor, the number of data points in each set, and how often you need to perform the calculation. I think it could be made to work in your application with your hardware.
Let me know if you're interested in trying what I've just suggested. If you need help setting up the RMT module or the ADC, then a ESP32 forum would probably be the best place to go. If you want help performing least squares analysis on the raw data, then I can help with that.