Quick update on the current Odin Fun situation: 1. Transactions fail often or take a long time to go through 2. Deposits and withdrawals fail often or take a long time to go through 3. We've been working non-stop on this for the last few days, and we've been working closely with Dfinity engineers (low level ICP issue) 4. We're hoping we can get this resolved in the next 24 - 72 hours (still some work and testing to do) Here's a bit more detail on the issue: 1. The issue is at the Internet Computer Protocol (ICP) level, specifically on the memory access level. With ICP smart contracts you have multiple levels of memory. Heap memory is readily accessible and fast to access. Stable memory is a bit more expensive to access and takes a bit longer. But heap is limited to a few GBs while stable can theoretically support hundreds of GBs. 2. Memory access can be done a few different ways. It depends on whether you are using Motoko or Rust for your canister. And it depends on your memory management solution (there are many ways to do stable memory). We're currently using a map based approach in a Motoko smart contract. 3. We've been digging DEEP trying to find the issue. Hard problem. All kinds of things. Update methods and DDOS, deserialization time, cross-canister calls, method protections, subnet config, memory usage, cycles usage, query/update call stats, batched processes, queues, and cross-canister call sizing, accessing motoko runtime information for additional data, garbage collection (whether incremental or the older setup), etc. 4. We think we've FINALLY found the problem. The best library for stable storage memory for Motoko canisters is a map-based solution that has 0(n) complexity on map resizing, so when your stable memory gets too large you run into issues trying to continue to add things to storage. This is a low level primitive / dependency on the Motoko side. Fixing is possible, but hard. A few options (pre-hash things to get log(n) complexity, use different map primitive, etc) 5. We have a concrete path forward and we are implementing the fixes outlined in #4. This will fix the issue and help us to continue to scale up at Odin Fun. Thank you to all of you and our incredible community for your patience as we work through this. Feel free to reply with questions if you want to dig deeper anywhere.
