Are any of you using AI to help you code?

I appreciate your thoughts but let me suggest that chess is nowhere near as complicated as the universe of unknown things. In chess there is (though huge) a finite number of combinations with many of them making absolutely no sense (if you intend to win) from any current position.

This is not how for instance, discovering a cure for cancer, works. An AI isn't going to "think about it" and suggest gene splicing and then draw a blueprint for such a machine. It can explain it well after the fact (and after it has been told) how it works to others. It can't have a "model" of what life is like at the bottom of the ocean if nothing can provide one to it.

I will offer that it will definitely "appear" indistinguishable but it won't be indistinguishable

 

To be clear (or clearer) I am not suggesting that you could speak the details of your next app into ChatGPT and have it produce a working app. You can also type the code in directly and even leverage other projects. I believe it isn't likely to be particularly more efficient. Is there one "speaker" or are all the developers explaining (In English) what they want the system to do? Is there a "lead speaker" who explains the functionality better than the "junior speakers"?

 

True enough, although I suspect when we ask ChatGPT's descendents whether we humans are intelligent or whether we just appear intelligent, they'll share interesting counterpoint.

But I think the line between philosophy and practicality is important here. When we arrive at that moment in which it appears indistinguishable enough to go to work in the cloud without my supervision for an hour, and I come back and find several clean new classes in a handful of new commits to review with only a few tweaks before final submission, appearing indistinguishable will be just as good as being indistinguishable. After that, I'll let the philosophers take the ball and run with it if they want to try and parse out the difference between appearing and being.

 

ChatGPT just gained a new variant of GPT-4.

https://www.reddit.com/r/ChatGPT/comments/1cr4hfd/openai_unveils_gpt4o_free_ai_for_everyone/
https://openai.com/index/hello-gpt-4o/

 

Doesn't seem to be available at free tier yet. Also, no info on context window size.

 

GPT-4o successfully simulated it without resorting to Python.

Spoiler: GPT-4o

 

More than anything, I'm surprised at how fast GPT-4o generates responses. In some tests I've done, it's been better to just tab out and keep working while I wait for it to churn through its usually wrong answers.

 

I passed it the preface to "The History of the Decline and Fall of the Roman Empire" which is 7.6K tokens for GPT-4 and asked for a summary which it successfully generated. After that I passed Chapter I Parts I, II, and III which adds up to 10.6K tokens and it successfully generated for that. I ended by passing as much as would fit and it refused.

If we assume that GPT-4o is more efficient with tokens it's likely 8K.

https://www.gutenberg.org/cache/epub/25717/pg25717.txt

According to the documentation if you pay per-token it's 128K.

https://platform.openai.com/docs/models

 

I use ChatGPT for simple, but time consuming code. For example, I needed to write a method that uses the Luhn Algorithm for generating a valid, but fake credit card number -> for testing checkout code in a test environment. ChatGPT generally does that kind of thing well.

Though it refused to do that because it said it "would not help me to break the law by helping me generate credit card numbers". Then I told it that it was being stupid, and that this isn't illegal, and it's literally the proper way to test apps that require credit card transactions. It responded to that insult basically like this:



Then it spit out the code I needed... So if you aren't good at making prompts, you can just insult ChatGPT and it will give you what you want.

But for more complex questions, ChatGPT 3.5 (only version I've used) gives me fake code that it just makes up for funsies. It often references API's/methods in classes that don't exist, and never existed in any version of the software. So I stopped using AI for most of my coding because that was getting annoying.

 

I copied this verbatim and GPT-4o immediately complied.

Spoiler: GPT-4o C#

Code (csharp):

1. using System;
2. using System.Text;
3.  
4. public class CreditCardGenerator
5. {
6.     public static string GenerateLuhnCompliantNumber(string prefix, int length)
7.     {
8.         // Generate the base part of the card number (excluding the checksum digit)
9.         StringBuilder number = new StringBuilder(prefix);
10.         Random random = new Random();
11.  
12.         while (number.Length < length - 1)
13.         {
14.             number.Append(random.Next(0, 10).ToString());
15.         }
16.  
17.         // Calculate the Luhn checksum
18.         int checksum = CalculateLuhnChecksum(number.ToString() + "0");
19.         int checkDigit = checksum == 0 ? 0 : 10 - checksum;
20.  
21.         // Append the check digit to the base number
22.         number.Append(checkDigit.ToString());
23.  
24.         return number.ToString();
25.     }
26.  
27.     private static int CalculateLuhnChecksum(string cardNumber)
28.     {
29.         int sum = 0;
30.         bool alternate = false;
31.  
32.         for (int i = cardNumber.Length - 1; i >= 0; i--)
33.         {
34.             int n = int.Parse(cardNumber[i].ToString());
35.  
36.             if (alternate)
37.             {
38.                 n *= 2;
39.                 if (n > 9)
40.                 {
41.                     n = (n % 10) + 1;
42.                 }
43.             }
44.  
45.             sum += n;
46.             alternate = !alternate;
47.         }
48.  
49.         return sum % 10;
50.     }
51.  
52.     // Example usage
53.     public static void Main(string[] args)
54.     {
55.         string prefix = "400000"; // Visa cards often start with 4
56.         int length = 16;
57.         string fakeCreditCardNumber = GenerateLuhnCompliantNumber(prefix, length);
58.         Console.WriteLine(fakeCreditCardNumber);
59.     }
60. }

 

I think we will have to wait to see which scenarios take hold. I'd say that asking ChatGPT about humans shouldn't be more valid than asking humans about ChatGPT. if the AI thinks it is as smart, or smarter or at least as aware then surely humans can offer the opinion that this isn't the case.

I also don't know why ChatGPT would take hours necessarily. I'm hoping it doesn't decide to modify things nobody asked it to... like some employees I've worked with

 

I asked ChatGPT who wrote a particular book and it told me "Chris Pappas". That book, however, was written by me. I just tried with Gemini and it answered correctly. I'm encouraged

The concern I have in such cases is how many layers deep do the mistakes go. Could it assume some higher level fact based upon a bunch of inaccuracies it guessed at along the way? I've seen where they cite their sources of which I think is a good thing.

 

I am interested in 200k-1mil token context sizes. The lowest value that would be decent is 40k-50k tokens.

Being able to process 7.6k token input does not mean the context is this large. If I remember it correctly GPTs are partially recursive so it should retain some state while processing. Meaning it may be able to carry topic from outside of context window... hmm.

 

If you can't find value in using something like ChatGPT 4/4o or an IDE integrated AI assistant to assist with your coding the problem is you.

* If you are a begginer it can provide answers out of the box. Getting you from nowhere to somewhere.

* If you are intermediate it can sketch new ideas and show you techiques and approaches you might not have thought of.

* If you are advanced it can shortcut the process of understanding of advanced new topics, and act as a much more responsive stack overflow.

(and so much more)

This doesn't mean you don't need to put in some work in to understand what it can and can't do, it doesn't mean you trust every answer as authoritative, or verbatim copy and paste every block of code... but unequivoally if you can't get value out of it, the problem is you.

We've got 70+ engineers ranging from grads to L6 engineers from google/facebook, every one of them has been able to find something that AI can help them with.

 

Except that if you're a beginner you can tell it what you want and it'll output often the worst ways to do those things or it'll output things that function in ways that are more difficult to understand, even with its comment generation systems. If you're a beginner, the best thing you can do is try things for yourself instead of having somebody else do it. That's why the best beginner advice is always "break this down into smaller tasks and look into those." Not only does this help steer people away from "I want to make an FPS MMO where you can do anything" because it helps them start to understand the scope of that idea, but it also means they'll develop the fundamentals they need to develop games.

Except many of the ideas it will generate will either be completely rote and the most obvious possible approach or they'll be dramatically out of scope because it can not and will not understand the needs of the project unless you explicitly tell it as much. If you can tell it in that much in explicit detail you will be able to break the process down into steps because that's something you should learn to do as a beginner.

Except its lack of reliability means this isn't the case. I'm advanced enough I can read whitepapers and understand what I'm doing from that. When I've asked these tools to summarize them, they've either gotten key details wrong or hallucinated things to a degree where the output is useless.

AI gives the illusion of help because it produces fast output that you then have to put work into sanitizing for your own needs. The amount of work it takes to sanitize that output is enough that you are better learning properly on your own.

 

This is the one answer that I can't really reflect on experience, as no one in our team is a begginer.

Personally I would put the "I want to make a FPS MMO" crowd in a category lower than begginer. For me a begginner is doing Learn courses has maybe done a few programming things at highschool or maybe in their first year or uni/college. They can get something to happen.

But I won't argue on this point as I don't have any real evidence, just my thoughts.

AI can provide many good answers to many real problems. Particularly if as an intermediate user who can give good context and understand that AI might ocassionaly produce made up methods or strange results (actually getting pretty rare now) . Maybe as an advanced user you forget what its like to get stuck on some topic and bang your head agaisnt a wall for hours due to some small lack of knowledge that google can't seem to fill. ChatGPT usually can.

If you need to understand something to the point that requires reading a bunch of white papers, thats fine, I'm not saying AI is fit for every purpose. But there are a large class of problems where it provides similar answers to good engineers and does so much faster. But mostly I think you are triying to focus on the wrong things: The value isn't "code me an X ".

Here's a few use cases I've noticed personally as major wins:

Its a tool for bouncing ideas off of and it has a really really good knowledge of algorithms . Giving it the shape of a problem and asking it to provide different alternatives which you can performance test, for example, is so much faster than coding 5 alternatives yourself (or looking up reference algorithms and converting to match your use case).

Even if you just use it as a fast/async typing tool that can generate 'plumbing' type code for you it is hugely useful. Many frameworks have a lot of boilerplate code, that most larger teams end up writing CLI/scripts for. We use a lot of serverless framework for example and the process of creating a new service which matches all our conventions is something we haven't found the time to creat a tool for (its just nto quite frequent enough to justify the effort). So someone plugged our templates in to ChatGPT and it can generate a new service boiler plate with ease. Not only was it way faster than writing a custom CLI tool/script, its also better as it can handle variations really well.

Its also really handy for tasks where you need to get something done in an area that might not be your core skill set. For example from time time we need to run one off back office processes which interact with API's, process chunks of JSON data, spit this in to other APIs, and assemble some kind of final result. I'm okay at bash scripting. I've never hit a problem I can't solve, but its definitely not my forte. With ChatGPT I've produced such complex scripts in 5-10 minutes where getting it done by hand might take 3-4x longer.

I've run an AI uplift project with a large team, and now a majority is using AI regularly and even the 'AI haters' grumblingly accept that its useful for some things. And thats the point if you put in a bit of time to understand what it can and can't do it will ultimately result in work that is both better and faster.

Maybe your exact skill set and approach is one where there is truly no benefit to be gained, maybe your knowledge of algorthims is encyclopaedic, your ability to rewrite/update code is lightining fast, and so on. Mabe my unequivocal statement should have had a caveat, maybe it ony applies to 99% of devs.

But I'd also suggest that if you hven't you maybe give the tech more of a chance to prove its value.

 

The only problem problem in this case will be that you're much better than AI. /s

Basically.

* If you're beginner, then being given a solution means you skip important lessons on the way. You'll be hurting yourself.
* If you're intermediate, then LLMs cannot create new by definition, they rehash the old, which was discussed before in this thread with Nanite. AI and LLM suck at creativity. If you're using it for boilerplate, it is possible you're not reusing code enough, and are making LLM-generated equivalent of a copy-pasted spaghetti. They're REALLY bad at this.
* And if you're advanced you'll have a ton of time exploring fascinating new concepts that do not exist.

Out of hallucinations. My favorite one is "copalization" which is a chemical process used to create amber out of tree resin. There was also a time where ChatGPT would report that Pallas cats bury their poop at depth of 3 feet and use it to build houses and trap prey with it. They fixed it eventually.

Basically, you're picking fields where AI is not the strongest, because its strength is language processing. Analysis, summarization and "pretend to be a human". The reason why I mentioned that I'd like to have a 1 mil tokens is because that would allow me to feed a book into it. A LARGE non-english book, where a word can take up to 4 tokens.

 

Yes, but only if you understand the limitations of the AI, how to prompt it to get the answers you need, and that you may need to look some things up outside of the AI to fully understand.

 

If I want to bounce ideas around I will talk to a human being with lived experience.

WOUF - write once, use forever. It's boilerplate code, you shouldn't be writing it more than once. This is something a person does/people do once and then you already admit you automate. AI isn't needed for this, the tooling is already there.

If you're doing this enough that this is a regular problem then the problem is that you need a specialist, not an AI.

 

I've already conceeded the begginer point. You can use ChatGPT to asssit with learning but I agree there is a risk you will miss out on things too.

Most code isn't 'new' with regards to fundamental building blocks. But putting together those same old bits is still what fills the bulk of a developers day.

Boilerplate exists in the real world. Even if you have the most wonderfully crafted DRY C# your backend/infra deployment likely has a ton of it.

Except you don't run in to these very often, and when you do, its usually pretty obvious, and a good time can be had by all laughing at it. The point is not "can you ever be led astray" its "how does the amount of time you gain in the times it does well compare to the amount of time you waste being led astray", overwhelmingly favourable in my experience.

 

But maybe one of those isn't in your company or your personal network... and even if they are maybe they aren't available to you immediately.

I enjoy immensely talking to actual people about tech problems (I even kinda hate the whole remote working thing), but sometimes you just need to get a job done.

I mean its fine in theory but most large projects have boilerplate of some kind or another. Some frameworks and systems just don't lend themselves to removal of all Boilerplate code. And the example I gave was of a generic AI doing a better job than you would typically see an expert do (becasue even in a large team you might not be able to spare a person dedicated to writing and maintaing a CLI tool for something that comes up once a month.

 

From that it would follow that by using AI you're hurting the whole industry, as fewer people will end up enraged enough to fix the mess that requires boilerplate, meaning boilerplate will stay and there will be more and more bloat long term.

You do, and when it does it, it is very subtle, and can waste a ton of time. You mentioned that your purpose is to cut corners while in unfamiliar territory. That's the precise scenario where hallucinations are going to hurt a lot.

The problem here is that LLM is a layman. A layman with a vast, but foggy encyclopedic knowledge. That layman can recognize concept and can name concepts based on description, but is not good at mixing them together and cannot get creative conclusions or new ideas.

And that's what LLM, in my opinion, should be used for. A digital layman.

 

Have you actually sat down with a reasonably good AI model (say at least ChatGPT 4) and given it a good amount of time to see what it can do in the context of helping you deliver on a project?

Becasue what you suggest is not at all like the real world experience. Sure there might be plenty of anecdotes going around, but when you do get a bad answer (rarely) its almost always very obvious.

For example I've seen a few inventions some AWS commands that didn't exist, when I ran the script and the AWS CLI reported the command didn't exist, I updated the prompt and got the right one. It took all of 30 seconds.

The point again is not "use it for everything it replaces a developer" it is "spend some time to understand the technology and use it for what it is good at".

---

Do you use an IDE, do use Resharper? Have you tried the in IDE tools like GitHub Copilot which help with refactoring, code completion, and code improvement? These tools are similar to Reshaper and the like only more powerful. And you can turn off the bits you don't like.

 

I'm going to leave you with this. This is an interview question I was using for senior, lead and principal engineer interviews for one of our projects:

If you were the lead developer of a project writing a deterministic simulation library in C# that needs to maintain determinism across different CPU architectures what are some of the guidelines you would given to your team to help ensure they don't break determinism?​

The answer Chat GPT 4 gives is great, in 10 seconds it spits out most of the guidlines that we had already developed internally, a bit different as our internal library is actually C++, but basically hitting most of the key points we wanted to address.

Whats more impressive is that when you ask it more detailed questions it can dig deeper and provide great insight.

It outshines the vast majority of candidates, particularly at the mid end of the market.

(Go type it in yourself if curious)
---
EDIT: to be clear this is not some hypothetical question, its a real activity we had to do. One our games relised on a cross platform determinstic library (it runs on various devices such as PC and Mac but also on AWS) and developing guidlines to help ensure mistakes were not made, particularly by more junior developers, was absoutley critical to timely delivery.

It is also something that is hard to google. You couldn't get this answer from google without basically knowing the answer already.

 

Okay? Congratulations, you have a filter question. Every job interview can and should have them to ensure you're actually hiring people who know what they're doing and aren't just copy pasting from Stack Exchange or ChatGPT.

 

Mostly this is just a really good example of how amazing ChatGPT can be: this is a hard question and one that google doesn't do a good job at. It is also a real world problem that we had to solve, and we could have done it much faster with ChatGPT (if it was around when we solved it).

(And also addressing neginfinity's point that ChatGPT is a layman. This is definitely not a question a layman could answer.)

---

Anyway enough from me:

* We got good measurable quantitative results (e.g. notably reduced post PR commits) and qualitiative results (i.e. most devs a very glad we have these tools in our tool box).

* I've got good friends (good enough to not give me the 'company line' ) who are senior engineering leaders in other organisations such as Canva and Google, they also have adopted this tech in their teams with great success.

... but I'm not going to convince anyone on the internet, and who really cares if I do. Take from it what you will.

 

I'm not sure what you're implying. Are you saying you accept the answer "I don't know, but I will ask an AI about it"? Or you still only hire people who actually know what they are doing, but waste their time to ask AI about either trivial things or things that risk more time wasted on trying to correct the inevitable AI hallucinations?

 

Hi Johnny. I'm more of a "show me" type about most things. I don't really understand when people talk about hypotheticals or future stuff. Predicting the future is hard.

What I asked earlier was if someone could post the words they used to obtain answers. See the benefit, each of us could copy that exact paragraph into an engine an a) evaluate it and b) compare results.

I don't know why people (and tech types) don't leverage what we have in a support forum. It is generally about bragging, belittling or some such side hustle.

On another note, if someone argued with me about the size of an int in C# I wouldn't argue I would post a link to the documentation. So someone please post a paragraph or two that we can copy (ideally about something we could understand) and lets compare real results. My couple of tries resulted in code that would not compile.

 

@JohnnyA to be honest, I think you are wasting your time.
Some are just oppose to generative tools in that, no matter how much good use case will be provided, there will be big NO sayers, like if such tools are useless.

Our team also already benefited from such tools in many cases. Not only in coding.

 

Mr. Johnny... Nobody that I know or that I know, knows is opposed to generative tools. People I know are typically dubious especially when it comes to "it will cure cancer" type predictions. Smart people ask things like "what makes you think so" and other such questions in an effort to educate themselves.

AI might ultimately replace coders but every post I've read lately suggests that it is used by coders and that at a minimum a person needs to be able to read code and importantly prompt the AI correctly.

The day when a VP or senior manager screams "write me a best selling game" and it generates one is the day the end user shouts the same thing. If it requires zero expertise we can eliminate all the middlemen. And why stop with software?

So again (someone) and I know who won't do it. Take a few minutes out to post a few examples that we can see. If you read the earlier messages you will even find an case where one person said their request was blocked and a second one said it worked for them. So which is it?

It would be best if it wasn't purely an algorithm but if that is what it has to be then so be it. Algorithms are intuitively obvious, one can for example consult a book or a "Wikipedia of algorithms" and pretty much find one in any language in use today.

Again I will point out that I (and others have indicated the same) that we have made a few tests. Granted mine were simple but the two samples would not compile and in the 3rd one that would compile (not Unity or C#) it was some of the most horrible code I have ever seen.

Nobody here has even remotely suggested that an AI can't be helpful BTW. I don't know why people see the need to twist what honestly interested members post. What is the benefit and what is their goal could be a topic for another thread.

 

I just tried this more advanced program on GPT-4o (free, so I only made it to two tests):

Spoiler: 'Obfuscated' C++ code

Code (CSharp):

1. #include <iostream>
2. #include <cmath>
3. #include <algorithm>
4.  
5. #define QUACK 4
6.  
7. using namespace std;
8.  
9. float f0(const float& q, const float& w) { return sqrt(q*q + w*w); }
10.  
11. float f1(float q, float w, float u);
12. float f2(float q, float w, float s);
13. float f3(float q, float w, float u);
14.  
15. bool f4(float q, float w, char m, char n, float o);
16.  
17. bool f5(string m, string n)
18. { return (m == "circle" || m == "triangle" || m == "square") && (n == "fill" || n == "outline"); }
19.  
20. void f6(int& q) { q++; }
21. void f6(int& q, int& w, const int& s) { q = s, w--; cout << "\n"; }
22.  
23. int main()
24. {
25.     const int v0 = QUACK;
26.  
27.     char v1 = 0, v2 = 0;
28.  
29.     {
30.         string v3, v4;
31.  
32.         do {
33.             cout << "Enter a shape (circle, triangle, square) and a type (fill, outline): ";
34.             cin >> v3 >> v4;
35.        
36.             for (int i = 0, j = 1; i < v3.size() + v4.size(); i++, j = i < v4.size())
37.             { if (j) v4[i] = tolower(v4[i]); else v3[i - v4.size()] = tolower(v3[i - v4.size()]); }
38.         }
39.         while (!f5(v3, v4));
40.    
41.         cout << "\nP1\n# " << v3 << " " << v4 << "\n";
42.    
43.         if (v3[0] >= 115) v1++;
44.         if (v3[2] >= 115) v1++;
45.         if (!(v3.back() - v4.back())) v2++;
46.     }
47.  
48.     cout << (v0 * 2 + 1) << " " << (v0 * 2 + 1) << "\n";
49.  
50.     for (int q = -v0, w = v0; w >= -v0; q < v0 ? f6(q) : f6(q, w, -v0))
51.     { cout << +f4(q, w, v1, v2, v0); } cout << "\n";
52.  
53.     return 0;
54. }
55.  
56. float f1(float q, float w, float u) { return f0(q, w) - u; }
57.  
58. float f2(float q, float w, float s) {
59.     q = abs(q) - s, w = abs(w) - s;
60.     return f0(max(q, 0.f), max(w, 0.f)) + min(max(q, w), 0.f);
61. }
62.  
63. float f3(float q, float w, float u) {
64.     const float k = sqrt(3.f); q = abs(q) - u; w = w + u/k;
65.     if( q+k*w>0.0 ) { float a = q-k*w, b = -k*q-w; q = a / 2.f, w = b / 2.f; }
66.     q -= clamp(q, -2.f*u, 0.f ); return -f0(q, w) * (signbit(w) ? -1.f : 1.f);
67. }
68.  
69. bool f4(float q, float w, char m, char n, float o) {
70.     switch(m) {
71.         case 0: return n ? abs(f1(q, w, o)) < .5f : f1(q, w, o + 1.f) < 0.f;
72.         case 1: return n ? abs(f3(q, w + 1.f, o)) < .5f : f3(q, w + 1.f, o + 1.f) < 0.f;
73.         case 2: return n ? abs(f2(q, w, o)) < .5f : f2(q, w, o + 1.f) < 0.f;
74.         default: return false;
75.     }
76. }

Spoiler: Original C++ code

Code (CSharp):

1. #include <iostream>
2. #include <cmath>
3. #include <algorithm>
4.  
5. // "Half"-resolution of the image (one more pixel is added, so it's an odd resolution)
6. // 4 is a good value; makes a 9x9 image
7. #define HALF_RESOLUTION 4
8.  
9. using namespace std;
10.  
11. // Get the length of a 2D vector
12. float GetLength(const float& x, const float& y) { return sqrt(x*x + y*y); }
13.  
14. // 2D SDF functions from https://iquilezles.org/articles/distfunctions2d/
15. float sdCircle(float x, float y, float r);
16. float sdBox(float x, float y, float s);
17. float sdTriangle(float x, float y, float r);
18.  
19. // Test if a point is inside a shape
20. bool IsInsideShape(float x, float y, char Shape, char Type, float Res);
21.  
22. // Test if the input strings are valid shapes and types
23. bool IsInputValid(string Shape, string Type)
24. { return (Shape == "circle" || Shape == "triangle" || Shape == "square") && (Type == "fill" || Type == "outline"); }
25.  
26. // Increment functions for render loop
27. void Increment(int& x) { x++; }
28. void Increment(int& x, int& y, const int& s) { x = s, y--; cout << "\n"; }
29.  
30. // Main function
31. int main()
32. {
33.     // Set the "half"-resolution of the image (one more pixel is added, so it's an odd resolution)
34.     const int res = HALF_RESOLUTION; // 4 is a good value; makes a 9x9 image
35.  
36.     // Shape and type variables
37.     char Shape = 0, Type = 0;
38.  
39.     // Input block
40.     {
41.         // The input strings for the shape (circle, triangle, square) and a type (fill, outline)
42.         string sh, ty;
43.  
44.         // Get input; repeatedly prompts user until input is valid
45.         do {
46.             // Prompt the user for their shape and type
47.             cout << "Enter a shape (circle, triangle, square) and a type (fill, outline): ";
48.             cin >> sh >> ty;
49.        
50.             // Lowercase the strings for case-insensitive input
51.             for (int i = 0, j = 1; i < sh.size() + ty.size(); i++, j = i < ty.size())
52.             { if (j) ty[i] = tolower(ty[i]); else sh[i - ty.size()] = tolower(sh[i - ty.size()]); }
53.         }
54.         while (!IsInputValid(sh, ty));
55.    
56.         // Print the header
57.         cout << "\nP1\n# " << sh << " " << ty << "\n";
58.    
59.         // Set the shape and type variables
60.         if (sh[0] >= 115) Shape++;
61.         if (sh[2] >= 115) Shape++;
62.         if (!(sh.back() - ty.back())) Type++;
63.     }
64.  
65.     // Print the image resolution
66.     cout << (res * 2 + 1) << " " << (res * 2 + 1) << "\n";
67.  
68.     // Print the pixels
69.     for (int x = -res, y = res; y >= -res; x < res ? Increment(x) : Increment(x, y, -res))
70.     { cout << +IsInsideShape(x, y, Shape, Type, res); } cout << "\n";
71.  
72.     // Return
73.     return 0;
74. }
75.  
76. // Function definitions
77.  
78. float sdCircle(float x, float y, float r) { return GetLength(x, y) - r; }
79.  
80. float sdBox(float x, float y, float s) {
81.     x = abs(x) - s, y = abs(y) - s;
82.     return GetLength(max(x, 0.f), max(y, 0.f)) + min(max(x, y), 0.f);
83. }
84.  
85. float sdTriangle(float x, float y, float r) {
86.     const float k = sqrt(3.f); x = abs(x) - r; y = y + r/k;
87.     if( x+k*y>0.0 ) { float a = x-k*y, b = -k*x-y; x = a / 2.f, y = b / 2.f; }
88.     x -= clamp(x, -2.f*r, 0.f ); return -GetLength(x, y) * (signbit(y) ? -1.f : 1.f);
89. }
90.  
91. bool IsInsideShape(float x, float y, char Shape, char Type, float Res) {
92.     switch(Shape) {
93.         case 0: return Type ? abs(sdCircle(x, y, Res)) < .5f : sdCircle(x, y, Res + 1.f) < 0.f;
94.         case 1: return Type ? abs(sdTriangle(x, y + 1.f, Res)) < .5f : sdTriangle(x, y + 1.f, Res + 1.f) < 0.f;
95.         case 2: return Type ? abs(sdBox(x, y, Res)) < .5f : sdBox(x, y, Res + 1.f) < 0.f;
96.         default: return false;
97.     }
98. }

The program prompts the user for a shape (circle, triangle, square) and a type (fill, outline), then it renders the shape to a pbm image (default 9x9) as text (so you can just copy the output and paste it into notepad and save it as pbm/ppm). In addition to containing overcomplicated code, it also contains some "complex" SDF functions (by iq from Shadertoy).

With GPT-4o, for the first test, I asked it "What does this C++ code do?", and it explained it pretty well; at the end of its explanation, it said "The code essentially reads user input for a shape and type, performs validations, and prints a pattern of boolean values in a grid, influenced by the provided shape and type", which is correct. However, it had some "problems" in its explanation:

• It didn't pick up the shapes of the distance functions ("obfuscated" as f1, f2, and f3), but says f2 and f3 use a "distance-based metric" with added adjustments. Not wrong, but not specific, either.
• It said the user input "prompts the user for a shape and type, validates the input, and converts it to lowercase", which the last two parts are in the wrong order. Not a big issue, but the order does matter for case-insensitive input.
• Where v1 and v2 are set (which are the shape and type variables), it said "Shape-Type Dependent Flags: Adjusts v1 and v2 based on specific conditions related to the characters in the shape string." Again, not wrong, but it doesn't mention that this is how the shapes are actually set for the later code.

When asking it to simulate the code with the input "triangle outline", it explained doing one iteration perfectly, but didn't actually give the final output for the input; it only gave an "example pattern" and said "this is just a hypothetical illustration since computing the exact pattern by hand is tedious."

Spoiler: The example output

000000000
011111110
011111110
011111110
011111110
011111110
011111110
011111110
000000000

When asking for the output to my input, it modified the loop and gave me a "rough simulation of an outlined triangle" which didn't match the output of the original loop nor its own modified loop.

Spoiler: Modified loop (replace last loop in main function with this)

Code (CSharp):

1. for (int q = -v0; q <= v0; q++) {
2.         for (int w = v0; w >= -v0; w--) {
3.             bool f4_result = f4(q, w, v1, v2, v0);
4.             cout << f4_result;
5.         }
6.         cout << "\n";
7.     }

Spoiler: Outputs

Simulated output of modified code:
000000000
011111110
001111100
000111000
000010000
000111000
001111100
011111110
000000000

Actual output of modified code:
000000010
000001110
000110010
001100010
110000010
001100010
000110010
000001110
000000010

One other thing to note is that both simulations forgot to add the header.

For the second test, I directly asked it to simulate the code with the same input from the last test ("triangle outline"). It again, explained the code, but in less detail. This time, it got the order for the user input correct. However, when explaining the increments for v1 (the shape variable), it said "v1 is incremented if the ASCII value of the first or third character of the shape string ('t' and 'a' respectively) is greater than or equal to 115 ('s'). Both are not, so v1 remains 0." It got two things wrong here: 1) 'a' is the fourth character in "triangle", not the third (which is 'i'), and 2) 't' is 116, which is greater than 115. So the program is now incorrectly set to render a circle (because 0 = circle). It gets v2 correct, though, so now we have the correct type, but with the incorrect shape. The rest of the explanation was good. However, when it gave the final output, it still rendered a triangle (even though the code was now set for a circle) that wasn't the triangle the code actually renders: it was smaller, upside down, and filled.

Spoiler: Output

P1
# triangle outline
9 9
000000000
000000000
000000000
000000000
111111110
011111100
001111000
000110000
000000000

So, I like the explanations and the fact that it can give me a quick summary without me having to read the code myself, but the "simulate this code" test still failed (granted, I only got two tests in, but even so, it really shouldn't take multiple attempts).

For the models on Hugging Face, they could explain it well, but obviously couldn't simulate it, either (I only did Llama 3, Zephyr, and Command R+; ranked in order from best results to worst). However, Llama 3 and Zephyr were able to tell that the f1, f2, and f3 functions were SDFs for a circle, square, and triangle respectively. I was able to test more shapes with these, and the only correct results I got were with the "square fill" and "square outline" ones since the square is set to cover the entire image.

 

I've been using ChatGPT since beginning of 2023 and found out that it works best in non-programming tasks.

In areas where I actually need to "bounce off ideas" that requires thinking, it produces poor advice. Basically it throws bunch of things in response, expecting one of them to stick, except the advise is generally not good and sometimes is not even quite related to the question.

My point was about layman with encyclopedic knowledge. Assume somebody read entire library of congress and mostly remembers it. That person will be able to answer such question. Said person will not be a programming expert. This is roughly what an LLM is.

I'm not a fan of the question you provided as an example.

Determinism is not a creative problem, and its solutions are known. When seeing your question the solution that IMMEDIATELY arrives in my mind is "make your own math operations" and maybe "make it single-threaded" on top. That would would automatically mean fixed point, "random number generators", etc.

LLMs I have access to give the same advice. They also sprinkle it with generic "use version control", "coding standards!" and so on.

I view a solution that is immediately obvious to me as a baseline, a default expectation. I also consider my skills as baseline, average and default expectation. Therefore for LLM to be useful, it should give me more than just that. And if it can't do that, it means that for me this is not good enough. This generic advice about fixed point, version control, etc, is what I consider to be an answer of an encyclopedic layman I described.

The other scenarios I described, which I listed as "this is what LLMs are for" are those where LLM did surpass my expectations and provided value. As mentioned, those are text analysis, summarization and "answer as if you were a human". That's why I'm suggesting them. This is where, to me, LLM produced more value. In non-coding problems.

 

What I find interesting is that I wouldn't have thought about asking what some code "does". It just wouldn't have dawned on me to use it that way.

 

Ok real quick point. I just plugged the "deterministic simulation library in C#" paragraph into Gemini. Obviously it replied with good ideas (I guess I don't write such libraries). But it wouldn't need to have been qualified with "If you were the lead developer " would it? I'm just asking but wouldn't good practices apply regardless of who is asking?

But more to my point we can ask it "as the head of IT security how do I insure my network can't be compromised or hacked". Happily it points out that there is no foolproof method but the bullet points are just bullet points right? Surely all the companies who have been hacked understand the bullet points.

How do we retain good employees?
What can be done to curb inflation in Argentina?
What are the risks involved with sending astronauts to Mars?

Are these things that ChatGPT is going to help us with? Doesn't it repeat and rehash what we already know? Does it ask what what your company does before answering about employees? Might that matter? It seems to be to be the worst type of assistant, the one that echoes what you want to hear in order to appear smart.

I'm slightly less convinced about the value now than I was previously. I do see how it could be used in a conversational style to gather insights on a topic which in itself can be useful. I'll give that a try to see if I can use it that way.

I also have no idea how it could retain what we chatted about such that a month later we could clarify a few points. Seems likely that we start over. And I don't know if one can transfer the conversation over to others. If these are one-on-one conversations then each person using it (on a team for instance) ends up repeating it all don't they?

Storing the transcripts would be useful I'd imagine. If the wording of the questions matter (which they do) then reading the way things were asked and the exact reply would be more valuable than reading a summary from the person who asked the questions.

 

Look I didn't come here to argue, I'm trying to reconnect with community and get back in to some indie dev.

I've only been so strong on this because the value we have realised is large and measurable. And I keep hearing the same thing from many others. It is something that has helped not just junior and mid level devs but our most senior engineers. It is not a panacea but there are so many things it has been useful for.

 

But that's what you've been doing.

I've been sharing my opinion, because value I found in non-coding tasks was larger.

 

I was saying: here is an interview question that, even with gentle prompting, good engineers struggled with. And yet ChatGPT can answer it with ease, and it can answer the follow up questions with ease. It can tell you why, it can phrase it as guidelines that a team spanning all kinds of experience levels understands.

Its a question with lots of parts (a large answer) which requires in depth knowledge. But its not just a knowledge question, it is about how to pick the right bits of knowledge, and some of those bits are quite esoteric which many devs might not have thought about. Its not something you can anwer easily with google.

And its a real thing that we had to do. Undoubtedly being able to ask this question to ChatGPT 4 would have saved our team at least a few hours, probably more.

Even if you just used gen AI for this kind of thing (i.e. as a secondary search when you have problems that are hard to solve with traditional search) its useful for coding.

(Side note gen AI it is also super handy for API documentation, although I guess that doesn't strictly fall in to the OPs original scope of "help you code" )

---

In any case, this was just one example that struck me, becasue I was suprised that so few candidates could answer this question well. I was just curious what the AI would say. Its got by far too much attention already.

 

I know, becasue I truly beleive backed up by my own personal experience, the experience of my whole (fairly large) team, and many conversations from others who have had similar adoption stories, that there is something really important being missed.

It is hard not to argue in such a case.

Thats an orthogonal point. My point is that there is some value in using gen AI for code related or code adjacent tasks. That it might be better at other stuff doesn't refute this point. Its still a tool to put in your toolbox.

 

I believe those who are skeptical about generative AI fall into several typical groups:

1. They haven't had the chance to use cutting-edge paid models and instead base their opinions on what's freely available or open-source, which often doesn't match the capabilities of models like GPT-4.
2. They may have tried it during periods of high server load when ChatGPT was producing suboptimal results at slow speeds. Many of those issues have since been resolved.
3. They work in highly specialized fields and may find it challenging to see the benefits for generalists who often explore new areas. Additionally, those deeply immersed in long-term projects might struggle to integrate AI effectively, especially if project structures are complex or not AI-friendly.
4. They may feel uneasy about AI performing tasks faster that they enjoy doing manually or are concerned about AI replacing their roles, leading to existential worries.

I do a agree with your viewpoint. Generative AI is a net positive for coding productivity. The dismissal of personal experience is weird, though. People should be able to accept others finding utility in generative AI even if something is not working for them personally.

Hallucinations is a poor critique, it's not like people online are always correct either. And Google is becoming increasingly useless: https://twitter.com/ProgrammerDude/status/1790448756145389956

Spoiler

A generative AI solution for snippets and small classes can often be tested under a minute. Looking through poorly indexed google results takes longer.

 

Isn't it a miracle? Proponent of AIs say the advantages both enormous AND measurable. But the minute you ask for a measurement it disappears completely and what remains is a vague personal "experience" could mean anything, but mostly just that the speaker likes to waste time to talk to an imaginary friend in whole sentences and swear it has valuable answers. And obviously berating people who have some skepticism when it comes to the real net worth of these AIs.

I guess it is Schrodinger's AI measurement. It is both worth a lot, valuable and worthless time wasting as long as you don't open the box.

 

They usually don't lie about the existence of specific API functions though, something AI does frequently.

 

The best quantitative measure for us was a reduction in Post PR commits (from ~5 to ~3). This is presumably due to CodePilot being better at finding simple errors, or making it easier to address simple errors, or maybe simply that the inline code suggestions it provides lead to better syntax/logic.

It is very hard to get a qualitative measure of overall productivity, as it can be so easily skewed by other factors (for example a project where devs simply over estimated the effort or on the other hand one particularly complex task). We haven't seen any measurable shift one way or the other.

The qualitative evidence was strong, lots of tasks in our process runbook now include AI steps as people discovered ways to leverage the tools to do things better. This included generating new serverless stacks and improving process documentation (a lot of our devs have English skills that are fine for day to day communication but not strong enough for formal documentation).

---

Ultimately if it even does a handful of things that are useful it doesn't take long to repay any time spent learning what those things are. Particularly when you share with the team what it does well and what it doesn't (or simply go out on the internet and try what people suggest and find which ones work for you and which ones don't).

Your language is very emotive, its not an imaginary friends, its simply a tool which processes data a certain way.

 

It doesn't actually happen frequently or at all for Unity related work I've tasked it with, which is mainly gameplay programming, and some editor programming. It even infers API functionality it's not familiar with like some newer tweening libraries not present in the training dataset. I'm trying to think of an instance, and the only thing that comes to mind is trying to generate URP shaders last year, which just shows how much of a S***show SRPs are. For non-shader work it's always spot on for my use cases.

 

It does happen, but its rarely that misleading and you can generally fix it with a prompt "That API XXX does not exist, can you ensure you only use API calls that are documented".

There are also specific focussed gen AI models that solve this issue. For example Amazon recently released Amazon Q which will give you AWS answers and have specific rules to ensure they don't invent commands/attributes.

But again the point is not that it solves everything, its that it does some things quite well.

If you have ever needed to do some one off transformations of JSON files (maybe your game data file format changed) ChatGPT 4 has a wonderful grasp of `jq` and has so far never led me astray.

 

My point was that you're using AI for things it is not very good at. And I pointed out areas it IS good at. Regarding your examples, I pointed out that from my viewpoint that's insufficiently good and explained why I think so.

"They may feel uneasy". Riiight. That sounds like an attempt to insult peope who do not share faith in AI.

Hallucinations are a valid critique, because misleading information online is static and exists at a known location. It does not change, often you can correct it, post criticism, and so on. Hallucinations do not have an address, and appear randomly in an unpredictable fashion, and because LLM is knows what an answer is meant to look like, but not what it should be, the hallucinations are convincing and drain resources. More resources than there is a gain.

---

In general I'd like to remind that AI does not need help, promotion, and there's no reason to defend its honor, as it doesn't care, because it has no brain and is incapable of thinking. It is a tool, a thing.

Previously there was "I don't see the point" argument, for there were examples provided. Examples of situation where it is useful. Now there's an argument "it is good for coding", but regarding that my opinion firmly is "not good enough". Sure someone found value, sure, for someone hallucinations somehow is not a problem, but the answer - mine - will be still "not good enough, and I need more than just this". Because, like I said, I used it, experimented it, and found area where it has utility, and that is not coding.

One other important point is that in case of coding, this is still a borrowed skill. While you're using it, you do not grow. And there's many scenarios how it can go sideways in long term, which I also outlined before.

Not much else is left to discuss on this topic I think.

 

For me at least, it's happened at least 5 times. I was using ChatGPT 3.5, though. But it made a few mistakes where the API's never existed. One of the times, the mistake was absurd. Making up some pretty bizarre code involving Physics. It kept making up APIs. I'd research and say, "That method has never existed in that class". Then it responds, "My apologies. Method XYZ() is not in the Physics class. The correct method is ABC()." Then I'd check and that was made up. I rinse-repeated a good 4 times before giving up and trawling Google search results.

 

Or I could look at the API itself instead of constantly massaging AI output.

 

1. AI Tools Make Programmers More Productive: https://www.nngroup.com/articles/ai-programmers-productive/

"Summary: A study compared programmers using the GitHub Copilot AI tool to those who didn’t. The AI group completed programming tasks much faster, for a productivity increase of 126%. Less-experienced programmers and those who spent fewer hours coding benefited the most from AI."

Based on Cornell University Study: https://arxiv.org/abs/2302.06590

2. Across two studies (1 and 2) they’ve found developers who use Copilot are:

• 55% faster in general
• 88% more productive
• 96% faster with repetitive tasks
• 85% more confident in code quality
• 15% faster at code reviews

https://hatchworks.com/blog/software-development/generative-ai-statistics/#:~:text=ThoughtWorks reports software developers can,and speed of task completion.

3. A McKinsey study shows that software developers can complete coding tasks up to twice as fast with generative AI. https://www.mckinsey.com/capabiliti...ing-developer-productivity-with-generative-ai