CCS Wallet Incident #916
Comments
|
Just to add to this, it's entirely possible that it's related to the ongoing attacks that we've seen since April, as they include a variety of compromised keys (including Bitcoin wallet.dats, seeds generated with all manner of hardware and software, Ethereum pre-sale wallets, etc.) and include XMR that's been swept. See tayvano's thread here. That hack recently started seeing some more sweeps happen (and they can tell that it's from the same hack since the surveillance-chain sweeps go to the same cluster of addresses). It's entirely possible that other wallets are at risk, which is why luigi1111 and binaryfate have taken additional precautions. I no longer have access to any of these wallets (although I do have large corp / treasury wallets on that laptop that pre-date Monero hardware wallet support and remain untouched), but I've taken similar precautions. |
|
It's also possible that the attacker isn't aware of what they've stolen, in which case I'd ask them to consider that they have stolen funds that are donated by individuals against specific things that Monero contributors are working on. This attack is unconscionable, as they've taken funds that a contributor might be relying on to pay their rent or buy food. I'd urge them to take action to make this right if they become aware of this😞 |
|
Shit, thats hard. We've stumbled upon one of the few bad things about crypto that it is irreversible. I can't think of anything other than replacing from the general fund. Also we should use open source hardware wallets like MoneroSigner from now on imo. |
|
"Luigi makes payments from the hot wallet and tops it up from the CCS Wallet (via SSH), occasionally as needed." Does this mean that the private keys for the CSS wallet were on an online Ubuntu server? If yes, thats where the compromise happened imo. |
|
What’s the balance of the general fund, will replenishing the CCS impact protocol development? |
|
Thank you for the transparency and closure about this issue.
So to clarify, @fluffypony never had access to the private keys to the hot wallet, but did have the private keys to the main CCS wallet post-arrest? Would the public be able to get transaction proofs (with addresses) to all nine of those transactions? If the hack was non-targeted, there's a good chance that the receive address gets re-used in someone else's hack, which would help us find the perpetrator. Going forward, I think that this scenario is an excellent exhibit on why the CCS should use multisig (at least for the main wallet). |
|
So sad to learn about this, please let me know if you need any help for the forensic part. |
Yes, as well as keys to the Bitcoin donation wallet, previous Monero GF wallet, etc. Post my release I nuked everything that could potentially be problematic as I was unsure as to what might happen next, and didn't want to put anything at risk.
I'm sure @luigi1111 can do that.
Yes definitely; multisig was not ready for this prior, but now it is. |
|
@johnalanwoods General fund is around 8k.
No, the general fund isn't usually used for funding active development but more for emergencies like this and other unexpected expenses. |
|
There's a clear suspect: https://xmrchain.net/tx/bb77d03cae08942f43cccd759ade505a1c9435470a4a2cabfa5e26d2c93d1a58 |
The hacks you mentioned @fluffypony were determined to be related to LastPass. This seems to be something different... |
A large number of them were, but there are a whole screed of sweeps from users that have never even downloaded LastPass. |
|
"The CCS Wallet was drained of 2,675.73 XMR (the entire balance) on September 1, 2023, just before midnight." Is this midnight UTC or another timezone? If UTC we can assign it low probability to be the same attacker referenced in tayvano's thread:
I'm guessing Core is already looking into hiring professional digital forensics specialists, but this could help with prioritizing what data to collect now that might still be around: https://owasp.org/www-pdf-archive//NetSecurity-RespondingToTheDigitalCrimeScene-GatheringVolatileData-TechnoForensics-102908.pdf |
|
Maybe I'm not understanding correctly, but aren't both of @luigi1111 wallets, Ubuntu and Windows, "hot" wallets? Both reside on machines connected to the internet with no hardware devices. Both had their respective spendkeys on them, yeah?
How was this performed? Did the Windows computer SSH into the Ubuntu computer, or vice versa? Was the node that the Ubuntu wallet ran on a pruned node or full node? |
CCS Wallet Opsec 2.0
The offline computer could be a scrappy $200 notebook, what's important is that it is offline forever. There is a burden when moving funds like this, but then again - this is a large amount of community funds. Having more "hot" buffers would spread out risk as well, and would speed up the payout latency for contributors, e.g, @plowsof could be given enough funds to pay out soon-to-be-finished CCS's (assuming he doesn't vanish)
Now that this is disclosed, current contributors who have been waiting for payment should be paid ASAP :) |
Core and their helpers have often been trying to pay things out over the years. But a combination of some people being unreachable, refusing payment, or other such circumstances means that funds often sit there. Many times for years. It may be wise to institute a form of expiration policy where unclaimed funds (x months or years after funding/project completion) go into a special "Fund other CCS projects" wallet or something. All of this Monero sitting there years after funding are a liability. |
Windows -> Ubuntu, once every 3 months or so. Full node. |
I am of the same opinion. All Tayvano's "OG" friends were also Windows users and considering the amount of well done and undetectable malware existing for that OS, I wouldn't be surprised if Luigi's Windows machine was already part of some undetected botnet and its operators performed this attack via SSH session details on that machine (by either stealing the SSH key or live using trojan's remote desktop control capability while the victim was unaware). Compromised developers Windows machines resulting into big corporate breaches is not something uncommon. A first step to investigate this is to log that machine's network traffic on the router that connects it to the Internet. A log time should be at least 48 hours (but more = better) with any software using network switched off to maximize the log's quality by reducing the noise to the possible minimum. Backdoors existing today are capable of being very low profile in terms of networking and detecting them isn't easy, therefore it will require some time and patience. This is the only possible realistic attack vector in this case, given that the timeline provided in the OP doesn't omit some more important information. P.S. beware that chances to discover the malware are 50/50, given that the attacker may track all the public communications related to this event including reading this thread, who could decide to detach/deactivate the backdoor to clear the evidence and avoid its disclosure. So consider making a full disk dump of that machine as well. P.P.S. stop using Windows for such projects. |
|
The attacker likely consolidated the funds again in these two transactions. Exchanges and services should check to see if they received these XMR deposits. https://xmrchain.net/tx/2c5b45bf398dcae482019a46fb2d06d334bf4260484dc4857fc35db3689ad5ec https://xmrchain.net/tx/06550272cdfa1eea98d288b2d57c272b5c52a2b195b4f808c8c03422a58ca47b |
|
I think that nobody asked that before, @luigi1111 I have few questions about the Ubuntu server
p.s. @SamsungGalaxyPlayer are you tracking monero 😕? |
|
If you are truly concerned about malware, simply switching to Linux isn't a great answer. Default Linux installations are not that great for security and not very hardened. You need a hardened system, preferably an immutable OS that has the root partition as read-only, IE Fedora Silverblue or any other OSTree based systems. Use https://cisofy.com/lynis/ to see any potential unnecessary security issues and things that weren't being used that can be turned off. Setup automatic updates. Only use Wayland, as X11 is easy to keylog. Use keys for SSH, not passwords. Or better yet SSH turned off. If you need to access it do it physically. Same thing goes for the CCS node/wallet server. Using UEFI Secure Boot, LUKS encrypted main, root, and GRUB partition. Wanna get crazy you can do coreboot with heads on some specific systems that support it. Don't use LTS kernels, use the latest one with grsecurity patches. Just suggestions. Also a given, these two devices should be VLAN'd from the rest of the network if not already. @hinto-janai 's model would already greatly improve what already exists, offline signing would take so many potential attack vectors away. Also secure the network if not already. Run an OPNSense firewall to VLAN and make sure no unnecessary ports are open. Use an OpenWRT router if you need wireless. Countless shitty consumer routers don't get updated ever, and many of them have severe vulnerabilities that don't get patched for a really long time. |
I didn't say one should use a default Linux installation. What you said should be already obvious to people with such responsibilities. What's surprising is that this is being explained to people from Monero team. |
Fluffy's setup was much better..
I think that this may be the most likely cause of the incident, I doubt someone 'guessed' the seed right. |
Yeah it corresponds to the industry standard where the threat agent is LE. |
Given that Windows was being used these things probably aren't obvious. Most people are not very knowledgeable on the inherent security issues with desktop operating systems, or basic hardening. |
|
I'm not 100% caught up on this thread yet (just getting back home) but here's some more specific details on the threat actors ive been chasing for a good while now: typically operate 1200 utc - 2300 utc, though all hours have been observed. least amt of activity 300-1000 utc observations we have on them for the time period mentioneed by op: 2023-Aug-30 21:50 for those above timestamps, all activity was via 2a00:1650:0:3:45::1 2001:ac8:23:3c:2d4::1 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 These are either HideMe VPN or residential proxies (5socks etc), as is usual for these actors. Victims have all sorts of devices. The only real thruline is the age of keys (created prior to 2022, some keys date back as far as 2012) and everyone we have talked to has used lastpass, most have confirmed the specific keys/seeds that were compromised were in lastpass, usually a secure note, at some point. most are longtime lastpass users but a few only used last past for a short period of time. those users confirmed the specific compromised keys were in lastpass. fwiw, these actors even push stolen XMR to BTC—we have observed them consolidating a victim's eth, btc, xmr to a btc address before pushing to wasabi/sinbad/cryptomixer/coinomize/etc. they use instaswappers to do so e.g. fixedfloat, simpleswap, sideshift, etc. The size of this theft should make the funds easily findable in the outgoing transactions from the hot wallets of those instaswappers on BTC if it is in fact these same threat actors. The first thing I would ask is if anyone who's ever had access to the keys that were compromised here has had other wallets drained in the last ~year. Even if the amt stolen from those wallets was small / dust. That will help determine source of compromise faster than anything else tbh. |
|
I can't understand why on Earth you would use less secure system (the Windows hot wallet) to SSH into the what is supposed to be the more secure system (Ubuntu). With a password no less. Neither do I understand why you would choose Windows or Ubuntu for either operation in the first place. If you're not an expert at sysadmin and security, then you should be using Qubes for this amount of funds, and/or offline key storage. |
so then make your own DIY HWW using ANONERO or Feather |
For a properly air-gaped cold wallet machine you can use whatever, even outdated Windows. I'm not using anything other than official CLI wallet for my cold wallet, in case of my usage I never had to use other than official wallet and I never use my phone for banking or cryptocurrencies as I don't trust android or custom made firmwares. But if I had to use other software for the sake of convenience, I would use it only for small change. |
|
Have RSA-2048 been used? |
|
What's the timeline for when host and network logs will be made available? Has the compromised machine been forensically imaged? |
|
After Thanksgiving. |
|
@luigi1111 may this be possible in your case? https://www.youtube.com/watch?v=3T2Al3jdY38 |
Earlier in the thread it was brought up that an ssh password was used instead of a key. Still an interesting read though. Yet another attack against RSA. Apparently DSA is the way to go. |
|
Sorry I didn't have the time yet to browse through all of the thread and do all of the research, but I am here to send here anything that may help |
|
It is unbelievable that what is referred to as a "cold" wallet, was still a box with network access (SSH). For the most OG blockchain network next to Bitcoin you would think the core maintainers would know the difference between a hot and a cold wallet. I am still grateful for all the contributors making Monero possible, but I hope that this incident will inspire more developers to take operational security serious. Why is it so easy to break in into Microsoft? Because of all the Windows..... |
|
If you have a server that only should be accessed by one machine then firewall should ensure that. |
|
Another helpful thing could be: didn't you accidentaly exposed ssh passwords (keys)? |
|
Sounds like in house to me so I can say if you work for me block chain it's possible then |
|
"Rethinking the Monero CCS: A cypherpunk proposal" https://monero.observer/cypherpunk-transmission-017-rethinking-monero-ccs-cypherpunk-proposal/ |
|
1 vote for Luigi1111 |
|
Like what
…On Wed, Dec 6, 2023, 7:05 AM and21togrowon ***@***.***> wrote:
1 vote for Luigi1111
—
Reply to this email directly, view it on GitHub
<#916 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BDHMSFSIMDVBW53TJUWC623YIBNQRAVCNFSM6AAAAAA63D5YLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBSG4ZTSMJUGM>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
sorry i'm no expert buy why not using a hw wallet? |
Operational Security is a security research topic. Blaming Luigi for using Windows with false claims such as Windows being inherently insecure compared to Linux is just annoying for him and the discussion. If you have no practical knowledge on this matter, stop proposing such insights. |
|
I admit there was a bit of frustration during the posting, but hey, it's not us who lost half a million in donations. A little bit of frustration was justified. If you can't stand the heat, stay out the kitchen. But ok, we can (and will) be constructive too. On the topic of Windows, it is inherently less safe than a minimal Linux system, for starters because of its closed source nature. The time whining about Windows is time you could be spending some time learning Linux thoroughly. It will greatly help in upping that OPSEC posture. Some good advice, now we are on it:
If multiple holders share the wallet, share the private key physically at inception. |
|
Hi, I was checking the monero github donation address (with the secret view key that is published there) and I saw someone deposited 2,696.73 xmr on dec 6 2023. An amount quite close to the extracted 2,675.73 xmr. Are those 2 transfers related? or is it just a coincidence? Be that as it may, that donation may help to recover the CCS fund right? Unless that money was already spent in something else (cause I cannot see the address spendings) |
You might not know but Trezor has a hidden wallet behind the visible wallet so the first point is not valid. |
Generous whale, remorseful thief, or intentional ruse with the goal of bringing attention to improving the security and structure of the CSS I don't think anyone knows for sure. Either way it was a fortunate surprise and we shouldn't let this second chance go to waste. |
Where are the logs? Maybe I am so dumb, but I can't find it in this thread. Is it somewhere else? |
|
Something which would give it away: what was the “gas” price for a transaction involving 2,696.73 XMR? If it was a “generous whale” that had over 2 and a half K in Monero to burn, it would make more sense for them to pay a number equivalent or higher to the amount taken. However, if it was a “remorseful thief”, the thief likely wouldn’t have extra XMR to use for fees, nor would they feel obligated to do so. In fact, if this was a case of “remorseful thief”, there could actually be a completely different reasoning behind the sudden return of funds: the thief already had a sizable amount of XMR in their possession, and was afraid that the theft may bring unwanted instability to the project (after all, it doesn’t matter how much XMR you have if the entire chain dies). Or, they were just bored and gave the money back (considering the prevalence of crypto casinos for a while, having a lack of practical uses for a large amount of crypto can lead to wacky decisions; basically imagine if you won a lottery for 1 million dollars, but it was only payable via gallon jugs of 1% milk) At the end of the day, those are all just theories, and because of the design of Monero, there won’t (or shouldn’t) ever be a way to determine which one really happened. What’s important is making sure this doesn’t happen again. |
|
I know this may be an unpopular take, as it may be better to keep everything in house. However, would it be worth considering using other solutions in the chain of custody.. Something like Rhino wallet that has 2FA via 1 in 2 multi sig. From what i understand, it is open source so can be looked over by the team. But if the ecosystem is growing and tools are being made, then it would not be too out-there to start using some of them in house. When you have half a mill in funds, it may be worth it to start thinking more like an org/enterprise. In the end of the day, a mine for iron may not make there own steal, and most certainly will not make there own equipment for mining it out. They will use the services of CAT, JCB or even an engineering firm to make and design the equipment. I'm just a Monero user with some technical knowledge on how things work to know it makes sense on the surface, but i have little dev / detailed knowledge to say this would be worth doing or not when deeper details are taken into account. so let me know kindly if this would not be a good idea. I know that another in house solution would be to have a policy where you hold a limited amount of total funds on a hot wallet, then use Monero's offline transaction feature to top it up periodically. Though I understand the Devs have a valid concern that they may become a target for theft, Should it be known that 1 of them are holding the cold wallet with significant funds. |
|
Could be an inside job. I mean we are talking about over 350,000 USD. |
and others
The CCS Wallet was drained of 2,675.73 XMR (the entire balance) on September 1, 2023, just before midnight. A second, hot wallet, used for payments to contributors, is untouched; its balance is ~244 XMR. We have thus far not been able to ascertain the source of the breach.
Timeline
ffc82e64dde43d3939354ca1445d41278aef0b80a7d16d7ca12ab9a88f5bc56a
08487d5dbf53dfb60008f6783d2784bc4c3b33e1a7db43356a0f61fb27ab90cc
4b73bd9731f6e188c6fcebed91cc1eb25d2a96d183037c3e4b46e83dbf1868a9
8a5ed5483b5746bd0fa0bc4b7c4605dda1a3643e8bb9144c3f37eb13d46c1441
56dd063f42775600adf03ae1e7d7376813d9640c65f08916e3802dbfee489e2c
e2ab762927637fe0255246f8795a02bd7bb99f905ae7afc21284e6ff9e7f73db
9bf312ed09da1e7dfce281a76ae2fc5b7b9edc35d31c9eb46b21d38500716b6b
837de977651136c18b0018269626be7155d477cc731c5ca907608a2db57ff6a8
9c278d1496788aee6c7f26556a3f6f2cbb7e109cd20400e0b2381f6c2d4e29f4
(wallet was then empty)
Open questions:
The text was updated successfully, but these errors were encountered: