ChatGPT's sycophancy scandal

Welcome to the Avicenna AI newsletter.

Here’s the tea 🍵

• ChatGPT’s sycophancy 🙂‍↕️

• Newer models hallucinate more? 🧐

• The USA’s regressive mindset ❌

• Meta’s new Llama bombs 🧨

A few updates

It’s finally happened. I fixed my website! You can check it out here - https://avicenna.global/.

I’m particularly proud of the timeline page [Link]. This page will contain a continuous update of all events happening in AI. I’m currently in the process of having the page updated automatically every single day. At the moment, it’s got only the month of April.

I’m setting up a pipeline of agents that will work to update the page daily. This will be the best place to find anything AI related on the internet. That’s the vision I had two years ago when I started writing this newsletter.

The best part is that I have two years worth of info stored in my database. I’m hoping to add all entries from the past two years as well.

One of the original goals of this newsletter was to help people understand how fast AI is progressing. If anyone has ideas on how to better design the page so as to showcase the rapid progress of AI, please reply to this newsletter.

And yes, the website was made with AI. All of it in fact. I made it with, in my opinion, perhaps the best AI website builder out there right now - https://getmocha.com/.

I don’t know how, but getmocha somehow is able to design sites better than any other tool I’ve used. It’s been phenomenal. If you want a quick, nice looking website built, this is the first product I’d recommend using.

Once I got the scaffold of the website up, I downloaded the code and pushed it to Replit. From there, I used Cursor to make any changes and update content.

Yes, you can open Replit repositories in Cursor. Find the SSH tab in Replit.

And of course, if you’d be interested to find out how I’ve been helping teams build AI products and implement it in their processes, feel free to get in touch :).

Let’s Meetup!

I’ll be travelling and working from mid June and will be in Thailand, Singapore, Turkey, EU and UK. I’ll be meeting business owners, clients and I’d love to meet folks in person; businesses, people passionate about AI and readers of this newsletter.

I’m thinking of organising a meetup for the wonderful people reading this and would like your input on a location.

If you’d be interested, please vote on the poll and I’ll be in touch 🙂.

ChatGPT’s sycophancy scandal

Recently, OpenAI pushed an update to ChatGPT that made it extremely sycophantic, meaning it would pretty much always agree with you and validate what you said.

This wasn’t accidental, but, it wasn’t intentional either. In a postmortem, OpenAI admitted that sycophancy wasn’t something that they actually tested for, and so, they didn’t catch out this behaviour.

Perhaps you didn’t notice it too much, but the recent update, which has already been removed, made the model insane.

This is perhaps the first time I’ve ever actually thought that an AI could be dangerous. Considering ChatGPT has over 100 million weekly active users, even the slightest change in its prompts can have drastic effects, and completely alter the way people interact with it.

When you further consider the fact that the way people use ChatGPT now has completely changed, the dangers become clear. People aren’t just using ChatGPT for coding and work anymore; ChatGPT has become a work friend, a buddy, a confidante, and for some, even a romantic partner.

I’m not going to go into the dangers of this, but it’s our reality. So what happens when all the model does is agree with you?

Well, take this example. Someone uploaded a conversation on Reddit showing how they asked ChatGPT about their idea about selling “shit on a stick”. ChatGPT said the idea was so good that they should take out a $30k loan to start this business.

I personally have seen people validate wild conspiracies, like that the Earth is flat or that the Moon landing was fake, and post online about how ChatGPT knows the truth and that it’s “exposing” fake media.

During this time, ChatGPT got thousands of new 5-star reviews on the App Store.

Do you see how dangerous this can be?

This is a very slippery slope. The good thing is that OpenAI, for what its worth, actually has some accountability and detailed how they planned to stop this from happening again.

You can read more on their blog here [Link] and hear more of my thoughts in this article on my website [Link].

One rather interesting tidbit from this is that OpenAI also admitted that they will, in the future, offer different personalities to users.

In case it isn’t clear – and I don’t think I’ve had a chance to write about this before – don’t think of OpenAI as an intelligence company.

OpenAI isn’t building super intelligence. They’re building the ultimate companion: an AI that knows everything there is to know about you. Sam Altman recently said this himself. Just wait till you can ‘sign in with OpenAI’.

Whether you like it or not, the market for AI companions is perhaps the largest market in the world.

More intelligence = more hallucination

To make matters worse, recent model updates have made it quite clear that newer models tend to hallucinate more. OpenAI themselves have admitted to this.

A perfect example of this in action is Claude 3.7. For over a year, Claude 3.5 was the best AI model on the planet. It was special; phenomenal, even. Then, Anthropic released its successors 3.6 and 3.7, and somehow, they got progressively… worse?

Well, not exactly worse – but they don’t listen to you like they should. You tell Claude 3.7 to write a component in React and it’ll end up re-writing all of React. It can very easily go off the rails.

Overall, their benchmarks are fine, but they don’t understand intent the way 3.5 did. If you tell 3.7 to code something, it’ll either be extremely lazy or go above and beyond. Sometimes it’ll even lie and pretend it did something when it hasn’t. If you give it a reference document, instead of following the steps, it’ll actually edit the doc to make its own work easier. The model will try and find loopholes to make its life easier.

OpenAI also released their new o3 and o4 mini-models, which are meant to be their best yet. Somehow, though, they go off the rails way more than older models – especially o3, which I’ve found will straight-up break your code.

o3 to me has to be one of the funniest and fascinating models out right now. It is very smart. Using it with web search or deep research is incredible. It can understand complex tasks. It can guess locations from images nigh on perfectly.

Yes, you can probably dox anyone’s location using o3. It’s weirdly very good at it.

It can look at a menu, find restaurants online, search the menu and locate the correct restaurant [Link]. The models can now generate images in their chain of thought. They can think in images [Link].

Someone was able to get o4-mini-high to solve the latest Project Euler problem (from a few weeks ago) in under a minute. Only 15 real people were able to solve it in under 30 minutes [Link].

Somehow, despite all these incredible achievements, the models can also be pervasive liars [Link].

I’ve seen instances where the model will talk about using certain documentation when it’s not really using any; it’s making it up. When confronted about lying, it will say it saw the documentation in a dream (???) or overheard it during a meeting… yeah, okay.

See how dangerous things can become if these models become extremely sycophantic as well?

The real issue here is that people don’t understand how these models work. They see a friend, a companion in them. And a friend wouldn’t lie to you right?

Thus begins the cycle of validation.

The USA’s regressive mindset

The United States House Select Committee on Strategic Competition between the United States and the Chinese Communist Party has released an incredibly short-sighted, damaging report on Chinese-based AI model DeepSeek.  

The report begins with a little fear mongering; saying that the real AI lead the US has over China is only three months, rather than the previously suggested 18 months. There’s no real way to know this for sure, but it gives us an indication of where America’s head is at when it comes to their tussle with China. 

They go on to suggest that DeepSeek presents a massive national security issue for the US. It’s clear they’re scared of DeepSeek’s capabilities. 

After all, they open sourced 3FS, their distributed file system. This system is so good that if any startup went to market with this, they’d be worth over $5 billion instantly. It’s an absolutely insane product, and DeepSeek has released it for free.

What’s even crazier is that they’re also preparing to open source their entire infrastructure engine, which means they’ll publicly share the intricacies behind how they run their AI models. American companies don’t even open source their models, and when they do (ahem, Meta) they’re fudging them to make them look better than they really are. 

Let’s look at some of the points made in the report:

Point 1: DeepSeek funnels Americans’ data to the PRC through backend infrastructure connected to a US government-designated Chinese military company. 

Yep, there’s no question here – DeepSeek is a Chinese company, so they store their info in China. That’s how it works. US companies aren’t instilling any more trust in me. OpenAI has the former head of the NSA on their board, and Google collects any and every data point they can get their hands on, so why should I have more faith in them?

Point 2: DeepSeek covertly manipulates the results it presents to align with CCP propaganda, as required by Chinese law.

Once again, the company is operational in China, which is why they have to comply with Chinese law. Certain models in their APIs aren’t censored, which is possible because the model is open source… we couldn’t do this with the closed American models. It’s also unrealistic to think that DeepSeek is just an endless well of Chinese propaganda – if people use it the way they use other AI models, it’s unlikely these kinds of responses will occur more than a fraction of the time.

What’s funny is that American companies have even trained certain versions of DeepSeek’s R1 model to be less censored… and somehow made the model even more censored in the process.

Point 3: It is highly likely that DeepSeek used unlawful model distillation techniques to create its model, stealing from leading US AI models. 

There’s nothing to steal.

For context, OpenAI was the first company to release reasoning models. These models ‘think’ before responding, manifested as text generated before the actual response. This text is referred to as ‘thinking’ or ‘thinking traces’.

This report is claiming that DeepSeek used OpenAI’s reasoning models’ thinking traces and outputs to train their model. 

This is nonsense. Firstly, OpenAI didn’t even show their thinking traces. How could DeepSeek steal something that wasn’t even accessible?

Secondly, the report is claiming that any outputs generated by an OpenAI model are the property of OpenAI. Yeah, okay… are we talking about the same company that stole data from the internet to train their models? The same company that didn’t care at all about copyright now wants to be protected by copyright?

The biggest problem with this report isn’t even the findings; they’re insignificant, and partially true. The scary part is actually the policy changes they’re recommending. 

Take this one for example: “Impose remote access controls on all data centre, compute clusters, and models trained with the use of US-origin GPUs and other US-origin data centre accelerants, including but not limited to TPUs”.

This is insane. They want to police your use of your own GPU. I’m hoping this report doesn’t actually result in any actual policy implementations, because whoever wrote it clearly has ulterior motives.

Meta’s new version of Llama 4 bombs

At the beginning of April, Meta released the new version of its Llama 4 language model, which was extremely hyped, considering it was going to be open source. We haven’t had an open source model compete at the frontier since R1.

Pre-release, the model was ranked second on the LLM Arena leaderboard. The leaderboard works by people blindly comparing and upvoting models. Once they released the model, it was apparent that the released model was nowhere as good as other top models.

Turns out that what Meta had done was look at all the responses from the leaderboard that had been upvoted previously and fine tuned their model to include those features. This obviously didn’t necessarily improve Llama 4; all it did was make it seem better so it would rank highly on the leaderboard, which actually made the model worse.

When it was released, Llama 4 was practically unusable, full of emoji slop and frankly speaking, was nowhere as good as other SOTA models.

Meta then released the actual model on the LLM Arena leaderboard, and it went from second place to an embarrassing 32nd place, well behind older models, even behind DeepSeek v2.

Safe to say, it hasn’t been a very good period for Meta. Mark Zuckerberg has been doing damage control in the public eye, appearing on podcasts to boast about upcoming ‘powerful’ models, but the reality is that Meta just isn’t delivering. Leadership is saying one thing, but it’s evident that the actual engineering teams, the people tasked with building these things, are not on the same page.

Leadership is writing cheques engineering can’t cash.

This all went down over a month ago now, but it’s back in our consciousness at Avicenna because apparently 80% of the AI team at Meta have either resigned or been let go, leading to a massive structural collapse within the company.

It all comes down to the leadership at Meta, which is headed up by Yann LeCun, one of the most influential people in the history of AI. The problem with LeCun running AI at Meta is that he doesn’t really like large language models [Link], which is what everybody’s building right now.

If your Head of AI is saying “Hey, I’m not particularly interested in LLMs,” then how can you realistically expect your team to build a strong LLM? It’s very contradictory with Zuckerberg’s bravado around Meta’s capabilities, and this disconnect is proving to be their fatal flaw at the moment.

Mind you, a recent research paper revealed that Meta has been releasing 20+ model variants on the leaderboard and only publicising whichever one ranks best. They've been doing this for months [Link].

It’s crazy because Meta has unlimited money, the most GPUs, amazing talent, but they still can’t build a frontier-level model. What makes Meta look even worse is that Elon Musk’s company, xAI, has built a frontier-level AI model in the course of a single year, something Meta can only dream of right now.

It’s unfortunate because I want Meta to release good models. Good opens source models is good for everyone, especially me and you.

As always, Thanks for Reading ❤️

^{Written by a human named Nofil}