My backup strategies for Kubernetes, Proxmox, Unraid, and Ceph.
46 watts.. but, yea, I expected lower.
But, suppose when its spinning 4x seagate exos, they like their juice.
It apparently doesn't allow HDD hibernation while containers are running, and doesn't appear to like to use any sleep states.
Key word, is idle.
Synology... and HDD hibernation don't really go together very well. If you have containers running, it won't let the HDDs hibernate at all. And- I have a minio instance running.
Cars, Computer, and Code.
Unless you have a wall of old nokia phones..... it should be quite scary.
Eh... Mutually assured destruction.
It's a very scary phrase.
That, is a pretty good deal. Better start picking up some MD1200s!
Nope, not at all.
Behind every successful story, is a lot of failures. (or- really rich parents).
I agree, I'd be picking up a bunch of those, if that were the case.
esp32-c6 (supports zigbee), is pretty cheap.
No.
I wouldn't vote for Hillary period for many reasons. Her sex is not one of them.
A random fact, I actually did vote for a woman to be president. But, it damn sure was not Hillary. There is too much stink associated with her. Too much shit swept under the rug.
The other admin now "owns" this instance, and hosts it in the EU.
I am just a glorified moderator now.
I'd say, you have a small instance.
I used to host lemmyonline.com, which has somewhere around 50-100 users.
It was upwards of 50-80g of disk space, and it used a pretty good chunk of bandwidth. CPU/Memory requirements were not very high though.
Nope.
Still just feel like a kid, with extra responsibilities, while raising my own kids. Guess sometime around 50 or so i'll start feeling like an "adult"
Although, at least I call myself a dumbass, after doing something stupid, or wasting money on crap.
I'm gonna wait a few years, until prices go waaay down.... and I plan on doubling/tripling the PV capacity, which will make everything much more effective, as well.
How did you get that rate? We pay 33 cents, and it was 24 cents just a few months ago… wouldn’t be surprised if it goes up again next year and the year after since even 33 cents is government subsidised (so - there’s no cheaper option available).
All about location. There are supposedly many in my area on a different coop utility, who are only paying 0.03c/kwh.
Ooof. Why’d you do that? We simply put (a bit over) 5kW of panels on the roof, and a good 5kW inverter. One day of sun generates about as much power as we use in a week, and even if it’s overcast we still come out ahead.
I had a few other goals I wanted to accomplish-
- Reliability. The grid here isn't the most stable, and blinks a few times per week. And, a time or two per year, we have an outage. This solution has handled this fantastically well, so well, that I don't even notice when the grid has dropped unless I specifically go for it.
- Apart of this, was bringing some of my wiring/electrical up to code. This accounted for 10k of the price-tag... I relocated/replaced the mains panel across the house to a location more suitable then my daughter's closet. Also- the panel itself, was pretty old, and needed to be modernized.
One more issue- my PV is undersized a bit. Adding another 3kw, would yield much better returns for me.
Its undersized, because if I oversized it, and sent more energy than I consumed, my lovely utility slaps on a 42$ fee.... which is no-bueno.
If the ROI => 25 years, then it's not worth it- because the hardware and equipment is considered deprecated at that point.
If it lasts 30 years, sure, its making good use of itself. But- everything is rated between 15-25 years. As such, after that period, it's considered end of life, and no longer supported.
Now- I will note, it is not worth it for the "Rate I currently pay", which is 0.08c/kwh. If next year, my electricity rates tripled, it would vastly reduce the amount of time until this solution reached ROI. And- I am betting that electricity does not get cheaper in the future, otherwise I would have not have pulled the trigger on a 50,000$ project, where the math told me it wasn't the best idea.
Also, if you really want to see everything quantified- I plan on publishing all of the math, and numbers at the one year mark... which will be around march. -> https://static.xtremeownage.com/pages/Projects/Solar-Project/
This platform* is getting overrun by trolls and tankies. I’m going back to Reddit
Coming from someone who owns them-
Nah, it's not worth it.... at least, if you strictly look at "saving money" overall.
ROI is on average 10-25 years, depending on your current cost of energy. The components/inverters/etc, are usually rated for 20-25 years.
At least- this applies if you have a properly licensed contractor install everything. If you do everything yourself, its extremely worth it, and would achieve ROI in a decade or less.
What will realistically happen?
Nothing. Companies will just spend a hair more money finding ways to circumvent the new taxes. And, if the new taxes were not easily circumvented- they would just relocate the company to another country with lower taxes.
In the end, the consumer is paying the taxes, and not the company itself, either way.
Lemmyonline.com, will go offline 2023-09-04
As promised, if I brought the instance offline, I would give you a heads up in advance.
Here are the reasons for me coming to this decision-
Moderation / Administration
Lemmy has absolutely ZERO administration tools, other then the ability to create a report. This, makes it extremely difficult to properly administer anything.
As well, other then running reports and queries against the local database manually, I literally do not have insight into anything. I can't even see a list of which users are registered on this instance, without running a query on the database.
Personal Liability
I host lemmyonline.com on some of my personal infrastructure. It shares servers, storage, etc. It is powered via my home solar setup, and actually doesn't cost much to keep online.
However- for a project which compensates me exactly $0.00 USD (No- I still don't take donations). It is NOT worth the additional liability I am taking on.
That liability being- currently trolls/attackers are lit
Pictrs disabled
A heads up....
Since, attackers/etc are now uploading CSAM (child porn....) to lemmy, which gets federated to other instances....
Because I really don't want any reason for the feds to come knocking on my door, as of this time, pictrs is now disabled.
This means.... if you try to post an image, it will fail. As well, you will notice other issues potentially.
Driver for this: https://lemmyonline.com/post/454050
This- is a hobby for me. Given the complete and utter lack of moderation tools to help me properly filter content, the nuclear approach is the only approach here.
CloudNordic said a ransomware attack destroyed customer data on its servers, including primary and secondary backups.
Both CloudNordic and Azero said that they were working to rebuild customers’ web and email systems from scratch, albeit without their data.
Yea.... Don't bother. But, do expect to hear from my lawyers.....
CloudNordic said that it “had no knowledge that there was an infection.” CloudNordic and Azero are owned by Denmark-registered Certiqa Holding, which also owns Netquest, a provider of threat intelligence for telcos and governments.
Edit-
Negativity on Lemmy
I am just wondering... is it me- or is there a LOT of just general negativity here.
Every other post I see is...
- America is bad.
- Capitalism is bad. Socialism/Communism is good.
- If you don't like communism, you are a fascist nazi.
Honestly, it's kind of killing my mood with Lemmy. There are a few decent communities/subs here, but, the quality of content appears to be falling.
I mean, FFS. It can't just be me that is noticing this. It honestly feels like I am supporting a communist platform here.
I am on social media to post and read about things related to technology, automation, race cars, etc.
Every other technology post, is somebody bashing on Elon Musk (actually- that is deserved), or talking about Reddit (Let it go. Seriously. We are here, it is there).
On my hobby of liking racecars, I guess, half of the people on lemmy feel it is OK to vandalize a car for being too big.... an
Looks like google domains is no more.
Knock on wood, I have not used them in quite a while.
Cars, Computer, and Code.
My adventures in building out a ceph cluster for proxmox storage.
As a random note, my particular instance (lemmyonline.com) is hosted on that particular ceph cluster.
I might have a problem......
I can't say for sure- but, there is a good chance I might have a problem.
The main picture attached to this post, is a pair of dual bifurcation cards, each with a pair of Samsung PM963 1T enterprise NVMes.
It is going into my r730XD. Which... is getting pretty full. This will fill up the last empty PCIe slots.
But, knock on wood, My r730XD supports bifurcation! LOTS of Bifurcation.
As a result, it now has more HDDs, and NVMes then I can count.
What's the problem you ask? Well. That is just one of the many servers I have laying around here, all completely filled with NVMe and SATA SSDs....
Figured I would share. Seeing a bunch of SSDs is always a pretty sight.
And- as of two hours ago, my particular lemmy instance was migrated
LemmyOnline Updated to 0.18.4
Sorry for the ~30 seconds of downtime earlier, however, we are now updated to version 0.18.4.
Base Lemmy Changes:
https://github.com/LemmyNet/lemmy/compare/0.18.3...0.18.4
Lemmy UI Changes:
https://github.com/LemmyNet/lemmy-ui/compare/0.18.3...0.18.4
Official patch notes: https://join-lemmy.org/news/2023-08-08_-_Lemmy_Release_v0.18.4
Lemmy
- Fix fetch instance software version from nodeinfo (#3772)
- Correct logic to meet join-lemmy requirement, don’t have closed signups. Allows Open and Applications. (#3761)
- Fix ordering when doing a comment_parent type list_comments (#3823)
Lemmy-UI
- Mark post as read when clicking “Expand here” on the preview image on the post listing page (#1600) (#1978)
- Update translation submodule (#2023)
- Fix comment insertion from context views. Fixes #2030 (#2031)
- Fix password autocomplete (#2033)
- Fix suggested title " " spaces (#2037)
- Expanded the RegEx to check if the title contains new line caracters. Should fix issue #1962 (#
Experiments in Ceph (with Promox)
So, last month, my kubernetes cluster decided to literally eat shit while I was out on a work conference.
When I returned, I decided to try something a tad different, by rolling out proxmox to all of my servers.
Well, I am a huge fan of hyper-converged, and clustered architectures for my home network / lab, so, I decided to give ceph another try.
I have previously used it in the past with relative success with Kubernetes (via rook/ceph), and currently leverage longhorn.
Cluster Details
- Kube01 - Optiplex SFF
- i7-8700 / 32G DDR4
- 1T Samsung 980 NVMe
- 128G KIOXIA NVMe (Boot disk)
- 512G Sata SSD
- 10G via ConnectX-3
- Kube02 - R730XD
- 2x E5-2697a v4 (32c / 64t)
- 256G DDR4
- 128T of spinning disk.
- 2x 1T 970 evo
- 2x 1T 970 evo plus
- A few more NVMes, and Sata
- Nvidia Tesla P4 GPU.
- 2x Google Coral TPU
- 10G intel networking
- Kube05 - HP z240
- i5-6500 / 28G ram
- 2T Samsung 970 Evo plus NVMe
- 512G Samsung boot NVMe
- 10G via ConnectX-3
- Kube06 - Optiplex Mi
Home Assistant 2023.8: Translated services, events, and wildcards!
Sentence triggers can now contain wildcards, a brand new event entity, all services are now translated into your language, generate images with OpenAI's DALL-E using Assist, and more!
Time to pull some more fiber
Since, my doctor recommend that I put more fiber in my diet- I decided to comply.
So.... in a few hours, I will be running a few OS2 runs across my house, with 10G LR SFP+ modules.
Both runs will be from my rack to the office. One run will be dedicated for the incoming WAN connection (Coupled with the existing fiber that.... I don't want to re terminate). The other, will be replacing the 10G copper run already in place, to save 10 or 20w of energy.
This, was sparked due to a 10GBase-T module overheating, and becoming very intermittent earlier this week causing a bunch of issues. After replacing the module, links came back up and started working normally.... but... yea, I need to replace the 10G copper links.
With only twinax and fiber 10G links plugged into my 8-port aggregation switch, it is only pulling around 5 watts, which is outstanding, given a single 10GBase-T module uses more then that.
Edit,
Also, I ordered the wrong modules. BUT... the hard part of running the fiber
Turbocharged, A community around talking automotive, racecars, etc..
A community based around cars and automotive. Specifically, around fast cars. If it involves 2-4 wheels, and can burn rubber, this is a good place for it. Turbochargers, Nitrous, Supercharged, Naturally aspired, Its all welcomed here. [https://lemmyonline.com/pictrs/image/82da012a-b24c-4726-83e8-642...
Surprisingly, I guess this didn't exist. Well, if you like talking automotive technology, it does now.
LemmyOnline Updated 0.18.3
## What is Lemmy? Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open, and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized into communities, so it is easy to subscribe to t...
In, addition to updating lemmy just now-
The storage issues have been resolved, the hosting issues have been resolved....
And things should return back to stable and reliable now.
The issue, and next steps
Turns out... its ceph storage.
Despite having 7x OSDs on bare metal NVMe... despite having DEDICATED 10G network connectivity.... Its having significant performance issues.
Any spikes in IO (Large file transfers, backups. Even copying files to a different server) would cause huge IO delays, causing things to break or drop offline.
There are no errors shown. The configuration is pretty standard. I have no idea why it is having so many issues.
I have cleared off a new NVMe, and will move this server to it tomorrow, and hopefully end all of the issues from this week... Assuming I have any users left here. (I wouldn't blame you for leaving, it has been a really bad week for LemmyOnline)
IF, my assumptions are incorrect, then f-it, I will just run lemmy on a bare metal server I have on standby.
Update
Server migrated to local storage. Was, nearly unnoticeable, unless you did something in the 3 minute window it took to clone/restore/etc.
Migrated to new server
Just finished migrating to a different server... hopefully this helps some.
Downtime Followup
As a continuation from the FIRST POST
As you have likely noticed, there are still issues.
To summarize the first post.... catastrophic software/hardware failure, which meant needing to restore from backups.
I decided to take the opportunity to rebuild newer, and better. As such, I decided to give proxmox a try, with a ceph storage backend.
After, getting a simple k8s environment back up and running on the cluster, and restoring the backups- lemmy online, was mostly back in business using the existing manifests.
Well, the problem is.... when heavy backend IO occurs (during backups, big operations, installing large software....), the longhorn.io storage used in the k8s environment, kind of... "dies".
And- as I have seen today, this is not an infrequent issue. I have had to bounce the VM multiple times today to restore operations.
I am currently working on building out a new VM specifically for LemmyOnline, to seperate it from
Single Threaded Workload
Yup. always gotta be that one single threaded program. In this case, appears to be frigate.
Electricity & Water I Feature Highlights Ep 6 I Cities: Skylines II
I don't know about y'all.... But, I am really looking forward to CS:II