mirth
@mirth@mastodon.sdf.org
and yet I participate
0
Followers
0
Following
Joined December 24, 2021
Web:
Posts
@dank I can't tell if I should block you or buy you several beers, but either way I've clearly been on the internet too long.
View full thread on mastodon.sdf.org
1
2
0
@budududuroiu @jenniferplusplus I wouldn't give Anthropic's motives a lot of credit here but LLMs do make bug hunting much easier.
View full thread on mastodon.sdf.org
0
2
1
@haubles I couldn't discern a real point of view in his thread. I don't even know if there is a "Mastodon community", or if the "Twitter community" even existed in the way he seems to think it did. The Mastodon experience I see is no doubt very different from what other people get, and that is mostly a feature rather than a bug.
View full thread on mastodon.sdf.org
0
0
0
@MichaelTBacon @alienghic @cwebber Between improved models, app designs, and hardware, it seems costs would come down. Do you have a source for operating costs being 10x greater than the current API pricing, or typical developer usage being 2B tokens a month? I have access to some developer usage data and I'd guess devs using every day average under 10M tokens/dev/month. I'm sure there are some wild outliers, but that is a typical problem many subscription businesses manage.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber Re: chips... That post about Azure is quite interesting but not related to AI accelerators. The core workload for these neural nets is almost totally unrelated to what a general purpose CPU does, and for large models the majority of energy consumption is DRAM and interconnect (e.g. not the arithmetic). So the answer to how much more customized they can get is "a lot" if you frame the problem as how to avoid interconnect and DRAM usage. Wafer scale etc.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber The subscription products are moving to multi-model hybrids under the heading of "model routing" and "sub agents" and related schemes. I think primarily motivated by cost although I don't know if they admit it in public. This already exists in the current products in small ways but they'll likely push it a lot farther.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber Regarding models, size for a given output quality has been falling fast for the last couple years. Well documented. IMO a major threat to OpenAI et al is if on-device models pass the "good enough" line for casual users and destroy the unit economics of their subscription businesses. Apple and Google have privileged access to user data via their OSes, but not yet good enough models. They're motivated to try.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber I read the Ed Zitron piece with some interest but it's weakly sourced and light on actual analysis (though it has a lot of links). I think he is right that AI companies are spending eye watering amounts of cash but misunderstands why or what will likely happen when the game of musical chairs stops. Massive layoffs, other financial carnage, yes, but the insiders will still be rich and maintain control of the post-restructuring profits. Corrupt.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber Combine a downward trend in average model complexity (by usage) and downward trend in energy consumption (on new hardware) on top of a typical usage that currently costs perhaps $100 - $1000 at the high end... I can easily see a world of $500/month/seat subscriptions without any structural changes. I'm not saying it's good or that I like it, but based on the best information I can find I don't think the "price explosion" scenario is plausible.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber The other thing happening is there are many efforts to build special-purpose chips for these workloads, and some will eventually pan out. Big neural nets on GPUs are extremely wasteful in energy terms, and even though many people seem to think that approach is horribly wrong it's become "too big to fail" in a way that will encourage investment into new chips until something sticks.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber That math sounds way off. Assuming a monthly usage of 5M tokens for day to day developer usage, at the current Claude API costs, and billing them all at the highest rate ($25 per M), that's $125 per month at current pricing. It's a long way from there to $50k, and surveying the trajectory over the last couple years as well as models from some of the Chinese labs it's pretty clear that model size necessary to do these tasks is trending down.
View full thread on mastodon.sdf.org
0
4
0
@MichaelTBacon @alienghic @cwebber I haven't seen any hard data, but spending enough time in tech industry circles it seems to be working.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber ... Right now the big firms are loading up on cash and I think working hard to cut the cost of their subscription products (e.g. ChatGPT or Claude) to the point where they'll be able to run unsubsidized in the near future at something not far from the current output quality and pricing. Their priority appears to be to sell more seats at low cost (Claude Enterprise starts at $20 a seat) and hope that they can get entrenched before starting to ramp prices up.
View full thread on mastodon.sdf.org
0
2
0
@MichaelTBacon @alienghic @cwebber I don't see a stopping in the near term. PE hasn't done a lot of real AI deals, I think that's blocked on a lack of proven playbooks. VCs are making bets but the actual end user value is pretty unclear. Having studied this area and its trajectory quite a bit over the last year I think the unit economics of API serving are already approximately sustainable, and the models and hardware designs continue to get cheaper for a given level of performance. ...
View full thread on mastodon.sdf.org
0
2
0
LLMs
Sensitive
@jamey@toot.cat @regehr@mastodon.social I share your concern in this respect but I don't think fuzzers and LLM-assisted search can be made equivalent. At the margin, many existing codebases are already heavily fuzzed and the surfaced issues fixed. What remains are "weird" issues that violate intended invariants in ways that don't generate a fuzzer-visible crash. Use LLM to add instrumentation, so the fuzzer can fuzz, and the combination finds new bugs.
View full thread on mastodon.sdf.org
0
1
0
@corbet I think only accepting large changes from users with a track record of smaller contributions is probably good policy regardless of how they are generated. In my own contributions I usually only add small features or bug fixes that don't incur maintenance overhead, knowing in advance that I'm not able to provide reliable support after the fact. Conversely if it's something nobody else will care about or contributions are too much work I fork and live with old code.
View full thread on mastodon.sdf.org
0
0
0
@hongminhee @Gargron Thank you for your thoughtful and well-articulated thoughts in this thread. Machine translation has given me access to my family and cultural heritage in a way I didn't have before, and I read Rochko's post the same way.
View full thread on mastodon.sdf.org
0
0
0