Question An issue with Open AI API and nullable float variable(float?)

Hello! I am trying to use the OpenAI API in Unity with this library: https://github.com/RageAgainstThePixel/com.openai.unity I want to perform speech-to-text transcription with it. I've successfully managed to transcribe speech to text, but when I try to set the temperature parameter in the request, I encounter an error related to the float variable not being valid. The issue is that the request function in the library requires a nullable float, but even if I use a nullable float or just a float, I still encounter the same error.

Code (CSharp):

1. RestException: [400] https://api.openai.com/v1/audio/transcriptions Failed!
2. [Headers]
3. Date: Mon, 11 Sep 2023 09:31:46 GMT
4. Content-Type: application/json
5. Content-Length: 217
6. Connection: keep-alive
7. openai-organization: user-bm4bibgkqd6xzmawfuvsecxz
8. openai-processing-ms: 3
9. openai-version: 2020-10-01
10. strict-transport-security: max-age=15724800; includeSubDomains
11. x-ratelimit-limit-requests: 50
12. x-ratelimit-remaining-requests: 49
13. x-ratelimit-reset-requests: 1.2s
14. x-request-id: 20150ea72b83ae81c2615486f7f305c4
15. CF-Cache-Status: DYNAMIC
16. Server: cloudflare
17. CF-RAY: 804ee64daee2624f-OTP
18. alt-svc: h3=":443"; ma=86400
19. [Data] 217 bytes
20. [Body]
21. {
22.   "error": {
23.     "message": "1 validation error for Request\nbody -> temperature\n  value is not a valid float (type=type_error.float)",
24.     "type": "invalid_request_error",
25.     "param": null,
26.     "code": null
27.   }
28. }
29. [Errors]
30. HTTP/1.1 400 Bad Request
31.  
32.  
33. Utilities.WebRequestRest.Rest.Validate (Utilities.WebRequestRest.Response response, System.Boolean debug, System.String methodName) (at Library/PackageCache/com.utilities.rest@2.1.6/Runtime/Rest.cs:1071)
34. OpenAI.Audio.AudioEndpoint.CreateTranscriptionAsync (OpenAI.Audio.AudioTranscriptionRequest request, System.Threading.CancellationToken cancellationToken) (at Library/PackageCache/com.openai.unity@5.0.8/Runtime/Audio/AudioEndpoint.cs:72)
35. WhisperScript.TranscriptAudio (UnityEngine.AudioClip audioClip) (at Assets/Scripts/WhisperScript.cs:28)
36. System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__7_0 (System.Object state) (at <47fc8c70fa834cbf8141d7c1a7589125>:0)
37. UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <91dc399f41a440558c716315c7e834ab>:0)
38. UnityEngine.UnitySynchronizationContext.Exec () (at <91dc399f41a440558c716315c7e834ab>:0)
39. UnityEngine.UnitySynchronizationContext.ExecuteTasks () (at <91dc399f41a440558c716315c7e834ab>:0)

Here is the function where I make the request.

Code (CSharp):

1. using UnityEngine;
2. using OpenAI;
3. using OpenAI.Audio;
4.  
5. public class WhisperScript : MonoBehaviour
6. {
7.     private OpenAIClient api;
8.     private string result;
9.     private CanvasScript canvas;
10.     private string audioPath = "Assets/Audio/audioClip.ogg";
11.     float? temperatureValue = 0.1f;
12.  
13.     private void Awake() {
14.         api = new OpenAIClient(new OpenAIAuthentication());
15.         canvas = FindAnyObjectByType<CanvasScript>();
16.     }
17.     public async void TranscriptAudio(){
18.         AudioTranscriptionRequest transcriptionRequest = new AudioTranscriptionRequest(
19.             audioPath,
20.             model: "whisper-1",
21.             responseFormat: AudioResponseFormat.Json,
22.             temperature: temperatureValue,
23.             language: "en"
24.         );
25.  
26.         result = await api.AudioEndpoint.CreateTranscriptionAsync(transcriptionRequest);
27.         canvas.UpdateTranscriptedText(result);
28.     }
29. }

Here is the request function in the library;

Code (CSharp):

1. public AudioTranscriptionRequest(
2.             string audioPath,
3.             string model = null,
4.             string prompt = null,
5.             AudioResponseFormat responseFormat = AudioResponseFormat.Json,
6.             float? temperature = null,
7.             string language = null)
8.             : this(File.OpenRead(audioPath), Path.GetFileName(audioPath), model, prompt, responseFormat, temperature, language)
9.         {
10.         }
11.  

I even tried modifying the library files by moving them to the 'packages' folder. I changed all the nullable floats to floats, but I still encountered the same error.

By the way, I reported this issue on GitHub, but I haven't received any response yet.

I really need this to work so I can continue with my project. Thanks in advance.

 

I remember temperature being a thing in a DougDoug stream on twitch, as he does heavy use for multiple uses with AI. And I think the temperature became a standard 1.0f as max, so you attempting 0.1f should have no issues in that regard. I remember in that particular vid, not even adding the temperature worked(as whatever the default was would override).

But to be fair, I don't think it was the same app. Try adding just a float itself(instead of a float variable) within the call, and also try not adding the temperature at all. See if either of those help.

Still contact the maker of the app(if something helps), because he may just have a simple mistake within the hard code somewhere.

 

To debug any remote API / network call:

Networking, UnityWebRequest, WWW, Postman, curl, WebAPI, etc:

https://forum.unity.com/threads/using-unity-for-subscription-lists.1070663/#post-6910289

https://forum.unity.com/threads/unity-web-request-acting-stupid.1096396/#post-7060150

And setting up a proxy can be very helpful too, in order to compare traffic:

https://support.unity.com/hc/en-us/articles/115002917683-Using-Charles-Proxy-with-Unity

 

Yes, the temperature parameter can be set between 0 and 1, with the default value being 0. If I don't set the temperature in the request, transcription works fine, but there is a hallucination problem where random sentences are added at the end of the transcribed text. I read somewhere that lowering the temperature reduces this hallucination problem.

The owner of the library I mentioned said that you can use a regular float, but I encountered the same error when I tried that as well.

 

I had no idea that I could debug a network call. I'm a newbie when it comes to this stuff.
Thank you for the links; I'll try it as soon as possible!

 

All network stuff should always begin with curl / Postman / swagger until you have 100% success.

If you can't make that work, you certainly won't make anything else work!