An opinion on AI

TLDR:

The 1-line summary of this article is that AI is amazing but is likely to make running a business a lot more risky / unpredictable. I have broken my reasoning into subpoints

1)

I have read articles about critical bugs that bancrupted whole businesses. The most serious bug I have discovered in my career was a zero-day that gave me admin rights to majority of the infrastructure. This was literally a bug which allowed a not-too-technical malicious agent to destroy the whole company and its clients in a matter of minutes. This was a financial institution with interests in security.

The point here is that critical failures are not just additional costs, they may destroy you or your clients

2)

There are multiple articles of AI uses that are unethical, malicious, unintended and / or beyond the control of the system owners.

I decided to not trust the web and test this myself. I built a minimalist web app with very obvious security flaw. I familiarized a popular AI model with my app. Later as another user, I asked the AI about the app and it explained to me how to hack it. The AI was oblivious that it was being used in malicious way

The point here is that it is trivial to use the AI to damage a business. At the very least it looks to me that we have built the perfect black box channel for leaking classified / confidential data.

3)

Deep neural networks are incredibly complex and non-transparent. They are like black boxes. Troubleshooting them is possible but a lot harder than troubleshooting traditional software.

The point here is that a reliable fix for the whole class of issues in 2) is beyond our reach for the immediate future

4)

In modern software development search engines are integral part of the workflow of a programmer. The expected workflow of the future is the AI being used like that.

The point here is — your devs will use AI to make their lives easier, including sharing cofidential data and code with the AI

5)

The Bullshit Asymetry Principle (a.k.a. Brandolini’s Law) is a common sense principle stating that creating a problem requires a lot less effort than fixing said problem

The point here is that it is realistic that destructive use of AI to outperform constructive use, because of assymentry of costs

This includes unintended destructive use

6)

The above points make me believe that AI is likely to increase the complexity of the software we build, decrease our deep understanding of said software, increase the likelyhood of critical system failures, decrease their traceability

So AI will increase dev output, of course, but will also increase variance — the chance that something random and completely unexpected destroys your business.

7)
There is a mathematical proof that in a complex economical system high variance is not zero-sum, but is negative-sum, that is — it is more destructive than the data suggests.

This proof is beyond the scope of this article, but if you are interested, feel free to ping me.

8)

Here we reach the conclusion of this article. AI is amazing, but also has a cost — difficult to measure, almost hidden, increase in variance, or unpredictability, or risk.

Dealing with risk is the most difficult part of running a business. We made our most difficult task harder.