George Hotz's Tiny Corp: Taking on Nvidia, Google, and PyTorch
George Hotz's Tiny Corp is taking on Nvidia, Google, and PyTorch with a tiny team. Learn about Tinygrad, the hardware design constraints, and more.
swyx.goodenough
Anti-ego ideas for anti-ergodic life. Working on something smol. AI news & interviews: @latentspacepod Book on Principles: @coding_career
-
The @LatentSpacePod is excited to publish:
— swyx.goodenough (@swyx) June 20, 2023
Petaflops to the People:@realGeorgeHotz's first interview
on his new personal compute cluster company
the tiny corp.https://t.co/yYgSk1tOJm
We discuss how tiny is taking on Nvidia, Google, and PyTorch with a tiny team and go deep… -
GPT4 is 8 x 220B params = 1.7 Trillion params https://t.co/DW4jrzFEn2
— swyx.goodenough (@swyx) June 20, 2023
ok I wasn't sure how widely to spread the rumors on GPT-4 but it seems Soumith is also confirming the same so here's the quick clip!
so yes, GPT4 is technically 10x the size of GPT3, and all the small… pic.twitter.com/m2YiaHGVs4 -
since MoE is So Hot Right Now, GLaM might be the paper to pay attention to. Google already has a 1.2T model with 64 experts, while Microsoft Bing’s modes are different mixes accordingly https://t.co/7OJi10qF1m pic.twitter.com/eEZO1TIA2X
— swyx.goodenough (@swyx) June 20, 2023 -
someone on Discord mentioned that maybe this is why GPT4 costs ~10x more than GPT3.5turbo once you normalize for context length pic.twitter.com/V3sHqtHiUY
— swyx.goodenough (@swyx) June 21, 2023 -
some sleuthing by AI twitter:
— swyx.goodenough (@swyx) June 21, 2023
apparently OpenAI hired away an author on Routed Language Models https://t.co/c5TxZl9KQW and 2 on Switch Transformers https://t.co/SixECcuTld
(after gpt4 was done)
Branch-Merge-Train from @AllenInstitute was also brought up… -
my bad - Google has 1.6T model and it is open source!
— swyx.goodenough (@swyx) June 21, 2023
great if you can run JAX... https://t.co/Vt2aw6W5Ld -
Huawei’s 1T PanGu (a seriously cool chinese good, look him up frens) published a nice MoE diagram
— swyx.goodenough (@swyx) June 21, 2023
and @Yampeleg mocked up some demonstrative code that i’d love an expert critique onhttps://t.co/xTTn5BkDri pic.twitter.com/zKDzmHUwwV