EleutherAI announces it has become a non-profit

259 points by stellaathena 2 years ago | 105 comments
  • dvt 2 years ago
    Not sure why the AI community has a weird obsession with being non-profit (or not). It's not like non-profits can't be evil, and it's not like OpenAI has done anything particularly egregious, either.

    What we do need is more weights being released in the public domain (hard to find even on Huggingface), easier ways to train models locally, better pruned models for embedded device inference (e.g. running on a Jetson Nano), easier ways to fine-tune for specific contexts, and so on. My big gripe, and for obvious reasons, is that we need to step away from cloud-based inference, and it doesn't seem like anyone's working on that.

    • stellaathena 2 years ago
      You're in luck! EleutherAI has trained and released open source weights of several LLMs, including GPT-Neo (2.7B parameters), GPT-J (6B parameters), and GPT-NeoX (20B parameters). This last model is currently tied for second on the list of the largest open source LLMs in the world.

      We also developed VQGAN-CLIP and CLIP-Guided Diffusion, techniques for doing text-to-image synthesis that don't require training and can easily be run locally for inference.

      • ImprobableTruth 2 years ago
        I don't want to be too cynical, but OpenAI used to be more open too until they decided releasing weights was too dangerous (/not profitable enough?), what guarantee is there that Eleuther doesn't also close their doors at some point?
        • jerpint 2 years ago
          Yes but in their wake they still left great contributions (CLIP, whisper, etc). So far they still seem net positive, even if they’re now for profit
          • stellaathena 2 years ago
            I mean, ultimately there isn’t one. I’m just providing examples of how we fulfill the things that the OP says they want, as they seem unaware of our work.

            But I’m confused by the anti non-profit vibes in this comment section. We aren’t saying that becoming a non-profit makes us ethical people, that would be a silly argument. But people do realize that the alternative would be to become a for-profit entity right?

            We’re still the same community-driven open collaborative research lab we’ve always been. But incorporating allows us to do things like hire full time staff, enter organizationally binding legal agreements, and protect our members. Between the options of becoming a for-profit and a non-profit, the later seems clearly better suited for our goals.

            • jamilton 2 years ago
              There is no guarantee, but it sounds like if they didn't incorporate they wouldn't be doing much of anything, public or not.
              • 2 years ago
            • josephjacks 2 years ago
              Sam doing a 180 on non-profit to for-profit was egregious, IMHO. Having said that, plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.

              Also: https://twitter.com/elonmusk/status/1626516035863212034

              • versteegen 2 years ago
                It's disruptive technology and an enormous amount of value is likely to be created. I would much rather the profits go to a nonprofit which can distribute them charitably rather than them giving them away freely to the megacorps which actually have the resources to run these massive models (edit^2: at scale). I do think the 100x cap on return on Microsoft's $1B investment was way too high (we don't know what the cap is on later investments but it's meant to be reducing over time).

                Edit: Plus it's still the case that OpenAI aren't putting profit above all else. E.g. they just released the ChatGPT API and it's 10x cheaper per token than GPT3! And their Charter does count for something, it's binding on them.

                • ketzo 2 years ago
                  A nonprofit wouldn't have to distribute profits to charity. It just can't distribute profits to private or public shareholders (because it can't have them). A nonprofit is obligated to reinvest any earnings towards its mission.

                  Not remotely an expert, this is just my basic understanding. "Nonprofit" is a pretty abused term.

                  • photochemsyn 2 years ago
                    I just looked up the the top CEO salaries of non-profit organizations in the USA - as of 2020, they ranged from about $16 million to $8 million. A non-profit appears to simply be a 'non-stock corporation' and can have the same ridiculouly pyramidal compensation structures that shareholder corporations do.

                    Overhauling the legal definition of non-profit to require the highest-paid employees to make no more than ~10X as much as the lowest-paid employees would make a lot of sense.

                    • bayindirh 2 years ago
                      Well, "they are doing something amazing, so ethics can be suspended for T amount time in the name of value" is equally egregious from my point of view.
                      • 2 years ago
                      • dragonwriter 2 years ago
                        > Having said that, plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.

                        Non-profit doesn’t mean the organization isn’t supposed to have surplus revenue, it means it doesn’t exist to return profits (e.g., via a claim on the assets of the company) to stockholders or some other beneficiary with a claim on them.

                        • mb7733 2 years ago
                          >Having said that, plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.

                          That's not a contradiction. A non-profit can be profitable. The profit just needs to be reinvested back into the mission of the organization.

                          • gowld 2 years ago
                            OpenAI is a nonprofit that ownes a for-profit. Same as Mozilla. OpenAI was never a charity.

                            A privately owned for-profit company can do whatever it wants, good or evil. OpenAI is not publicly-owned.

                            Did you donate money to OpenAI?

                            A nonprofit is an organization that uses its income and profits for the organization's main goal that supports the mission. On the other hand, a charity is a type of nonprofit that engages in activities aimed at improving lives in the communities.

                            NFL was a non-profit trade association until ignorant whiners made it bad PR and they changed it to shut up the whiners.

                            Did https://openai.com/about

                            • RobotToaster 2 years ago
                              >plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.

                              Isn't Ikea technically a non-profit? through some weird structuring?

                              • leesec 2 years ago
                                Capped profit*
                                • nwoli 2 years ago
                                  Capped at 100x investment which would be the most profitable company in history lol
                                • zitterbewegung 2 years ago
                                  Sort of meta that Sam did something that Elon Musk would do…
                                • vikp 2 years ago
                                  If you want more open research and weights, you should be happy about this announcement. Incorporating as a nonprofit doesn't guarantee that an organization will act ethically, but it does make it more likely. Nonprofits have more restrictions around how they can spend their money. They also pay less, so the teams is more likely to be mission-aligned.

                                  The profit motive pushes organizations to keep their work secret (which is what has happened with OpenAI).

                                  • braingenious 2 years ago
                                    > Not sure why the AI community has a weird obsession…

                                    Can you elaborate on this “obsession”? Aren’t a lot of AI things made/implemented by for-profit companies? Google, Meta, Midjourney, etc?

                                    • __loam 2 years ago
                                      Yeah lol. The obsession is understanding this is going to be pretty impactful technology and thinking it might be nice to not have that be closed source for once
                                    • fl0ps 2 years ago
                                      I'm with you there on hoping for more concentration on local training and inference, especially on something like the new Orin (open to donations, NVidia!).

                                      I think there's still something of a financial disincentive to promote selfhost over cloud capabilities for just about every party involved except the selfhoster. NVidia loses out if they're selling a few Nano or Orin platforms relative to the much pricier datacenter cards sold in huge lots. The cloud hosters are making up for the cost of hosting in terms of end-user pricing, snarfing all that lovely customer data and likely selling analytics, and probably other measures I can't understand yet. And the large companies that fund research and initial model development want to know what's being done with their models so as to gain any possible competitive advantage. They can't necesarily guarentee that intel from a self-hoster. Almost nobody is willing to spend the time explicitly to make it easier for the individual dev at a small lab or at home to do this because that's essentially a donation and not a business expense that might yeild obvious returns.

                                      • bootsmann 2 years ago
                                        > easier ways to train models locally, better pruned models for embedded device inference (e.g. running on a Jetson Nano), easier ways to fine-tune for specific contexts

                                        I think there are steps being taken in this direction (check out [1] and [2] for interesting lightweight transpile / ad-hoc training projects) but there is a lack of centralized community for these constrained problems.

                                        [1] https://github.com/emlearn/emlearn [2] https://github.com/BayesWitnesses/m2cgen

                                        • underlines 2 years ago
                                          I was looking for open alternatives to self hosted, or crowd hosted finetuned LLMs like ChatGPT and found LAION Open Assistant. Then found resources to further optimize inference as well as training:

                                          - Open source fine tuned assistants like LAION Open-Assistant [1]

                                          - inference optimizations like VoltaML, FlexGenm Distributed Inference [2]

                                          - training optimizations like Hivemind [2]

                                          1 https://github.com/LAION-AI/Open-Assistant

                                          2 https://github.com/underlines/awesome-marketing-datascience/...

                                          • warent 2 years ago
                                            It seems like a rift between "Enlightenment" and "Authoritarian High Modernism." People want this progress to benefit reason and humanities, not contribute to our backsliding. Being nonprofit is at least symbolic of enlightenment.
                                            • xwdv 2 years ago
                                              > Not sure why the AI community has a weird obsession with being non-profit (or not)

                                              A lot of times, I find the obsession isn’t a desire for collaboration or anything noble, it’s as simple as this: people want to use the end products for free or very cheaply.

                                              • Bloating 2 years ago
                                                > It's not like non-profits can't be evil

                                                There's no shortage of non-profits that exist to pay saleries. As an example, growing up in the bible-belt, and I can confirm that religion can be good business.

                                                • chologrande 2 years ago
                                                  I can't help but wonder if non-profit makes it easier to get access to or train on certain datasets/models.
                                                  • dustingetz 2 years ago
                                                    taxes!
                                                  • rahidz 2 years ago
                                                    I hope Eleuther will continue to focus on the development of truly open-source models, which seems to be where the big gap is. The quality gap between Neo-20X and even the original GPT3 is still rather large, and making headway there would be amazing.
                                                    • riku_iki 2 years ago
                                                      there will always be gap, since corps can grab open model, and add resources to improvements on top of it.
                                                      • Taek 2 years ago
                                                        This is not the case with Stable Diffusion. The open source community is moving faster and innovating faster, and the open models are by far the best. Midjourney is a contender, but it doesn't hold a candle to the best open models
                                                        • wokwokwok 2 years ago
                                                          Let’s be honest though; the number of people willing to devote their time to tireless effort generating porn (or suggestive images) is >> the number who will ever be willing to do it for text models.

                                                          Add the size barrier (an order of magnitude > SD for text models) and it’s pretty questionable the open source text models ecosystem is going to go anywhere.

                                                          Hardly anyone has done anything interesting with any of the existing ones and they’ve been out for longer than SD has.

                                                          Text models just aren’t that interesting to people.

                                                          • riku_iki 2 years ago
                                                            I think this space is in initial choke chaos.

                                                            Eventually commercial entities will build pipelines which will absorb open models and build on top of them using proprietary tech, data and expensive compute.

                                                      • jackblemming 2 years ago
                                                        They're the real OpenAI! Great team.
                                                      • supermatt 2 years ago
                                                        While I appreciate what Eleuther have done - why are they highlighting "research" consisting of removing a bit of whitespace from some source code... is it April 1st already?

                                                        Heres my addition! Or subtraction, i guess... (-4 chars by changing call to AdaptiveAvgPool2d):

                                                          from torch.nn import*
                                                          def c(h,d,k,p,n):S,C,A=Sequential,Conv2d,lambda x:S(x,GELU(),BatchNorm2d(h));R=type('',(S,),{'forward':lambda s,x:s0+x});return S(A(C(3,h,p,p)),*[S(R(A(C(h,h,k,1,k//2,1,h))),A(C(h,h,1)))for _ in[0]*d],AdaptiveAvgPool2d(1),Flatten(),Linear(h,n))
                                                        • 2 years ago
                                                          • jamilton 2 years ago
                                                            Just having a bit of fun.
                                                          • bilsbie 2 years ago
                                                            I followed along on their discord for a while but I wasn’t sure how to gradually increase my engagement to get involved.
                                                            • victorbjorklund 2 years ago
                                                              Awesome. Remember your great work in GTP-Neo
                                                            • victor9000 2 years ago
                                                              I really appreciate how the work of EAI has truly focused on making models accessible to the general public.
                                                              • eachro 2 years ago
                                                                How do they pay for their compute?
                                                                • stellaathena 2 years ago
                                                                  We have a number of donors including Hugging Face, Stability AI, Nat Friedman, Lambda Labs, and Canva that make our work possible. We also have some orgs that provide sponsorship for computing resources specifically: Stability AI, CoreWeave, and Google Research.
                                                                  • ftxbro 2 years ago
                                                                    are you bottlenecked by engineering
                                                                    • stellaathena 2 years ago
                                                                      Yes, we are currently bottlenecked primarily by engineering manpower.
                                                                  • ronsor 2 years ago
                                                                    Donations and sponsors such as Stability AI, based on the announcement in their Discord server
                                                                    • mperham 2 years ago
                                                                      With money. A non-profit can still have plenty of revenue.
                                                                    • valzam 2 years ago
                                                                      Easy to become a non-profit if you've never made any money...
                                                                      • kobalsky 2 years ago
                                                                        bold claim to make on a site linked to growing companies by burning money.
                                                                      • hinkley 2 years ago
                                                                        Now I have Lenny Kravitz songs stuck in my head.
                                                                        • quartzbox 2 years ago
                                                                          Eleuther has been at the forefront of AI, without them none of this would be possible ,thanks and kudos
                                                                          • return_to_monke 2 years ago
                                                                            sorry for killing the mood. great article and great progress.

                                                                            but. didn't openai start as a nonprofit, too?

                                                                            • loa_in_ 2 years ago
                                                                              OpenAI has some complicated setup about this. Part of it is non-profit and part of it isn't and there's some rules they laid out on how to keep it non profit first from what I've gathered
                                                                              • infinityio 2 years ago
                                                                                to my knowledge - started 100% non-profit, the split into partial profit came later when they started having a chance to make money
                                                                            • whitten 2 years ago
                                                                              As a three letter acronym (TLA) aficionado, the page uses TPU wwhich isn’t explained.
                                                                              • inconceivable 2 years ago
                                                                                tensor processing unit. a processor designed to do large, fast matrix operations.
                                                                                • archgoon 2 years ago
                                                                                  TPU - Tensor Processing Unit. They're essentially Google GPU's.