We've successfully upgraded Lemmy from v0.19.3 to v0.19.5 and have a bunch of changes to share!

Bug Fixes

• Previously users were reporting that the web client kept logging them out. This was suspected to be due to a misconfiguration of the cookie storing the session information and should be fixed.
• Some communities were slow to load. Federation changes in v0.19.4 and v0.19.5 should rectify some federation issues. Additionally, we increased the number of database connections for Lemmy and will gradually increase this if we identify any further slowdowns.

(Longer term, we're planning to upgrade the database hardware which should improve the performance for all services)

Local Only Communities

When creating or editing your communities, you'll now see a "Visibility" option that allows you to choose between "Public" and "Local Only".

The "Public" setting will continue to operate as normal and federate your community to all other instances we federate with.

"Local Only" wi

We’re aware of an issue affecting our Lemmy service (pawb.social) that seems to be causing users to be randomly logged out. We believe we’ve identified the issue thanks to a bug fix in a newer version of the UI code.

We’re needing to verify there are no other breaking changes for the upgrade process, and expect to perform maintenance this Sunday, Sept 22nd at Noon US Mountain Time (https://everytimezone.com/s/7abd4955). We’ll provide an update tomorrow to confirm the maintenance window.

Thank you to everyone who brought this to our attention and we apologize for the inconvenience.

tl;dr summary furry.engineer and pawb.fun will be down for several hours this evening (5 PM Mountain Time onward) as we migrate data from the cloud to local storage. We'll post updates via our announcements channel at https://t.me/pawbsocial.

In order to reduce costs and expand our storage pool, we'll be migrating data from our existing Cloudflare R2 buckets to local replicated network storage, and from Proxmox-based LXC containers to Kubernetes pods.

Currently, according to Mastodon, we're using about 1 TB of media storage, but according to Cloudflare, we're using near 6 TB. This appears to be due to Cloudflare R2's implementation of the underlying S3 protocol that Mastodon uses for cloud-based media storage, which is preventing Mastodon from properly cleaning up no longer used files.

As part of the move, we'll be creating / using new Docker-based images for Glitch-SOC (the fork of Mastodon we use) and hooking that up to a dedicated set of database nodes and replicated st

We've updated to Lemmy v0.19.3!

For a full change log, see the updates below:

• v0.19.0
• v0.19.1
• v0.19.2
• v0.19.3

Major changes

Improved Post Ranking

There is a new scaled sort which takes into account the number of active users in a community, and boosts posts from less-active communities to the top. Additionally there is a new controversial sort which brings posts and comments to the top that have similar amounts of upvotes and downvotes. Lemmy’s sorts are detailed here.

Instance Blocks for Users

Users can now [block instanc

Currently, we’re running the Ubiquiti Dream Machine directly as the modem via PPPoE, but there appears to be an intermittent issue with the software implementation that results in periodic downtimes of a few minutes while it reconnects.

We’re looking at switching this back out with the ISP provided router in pass through mode to negate the PPPoE connectivity drop.

We don’t expect this to take longer than 1 hour to switch over and test for reliability before bringing the services back up.

We’ll be performing this maintenance around 11 AM US Mountain Time, and will provide updates via the Telegram channel at https://t.me/pawbsocial.

One of the data storage systems (CEPH) encountered a critical failure when Proxmox lost connection to several of its nodes, ultimately resulting in the CEPH configuration being cleared by the Proxmox cluster consensus mechanism. No data, except ElasticSearch, was stored on CEPH.

When the connection was lost to the other nodes, a split-brain occurred (when nodes disagree on which changes are authoritative and which should be dropped). As we tried to recluster all of the nodes, a resolution occured that resulted in the ceph.conf file being wiped and the data on CEPH being unrecoverable.

Thankfully, we’ve suffered no significant data loss, with the exception of having to rebuild the Mastodon ElasticSearch indexes from 6 AM this morning to present.

I’d like to profusely apologize for the inconvenience, but we felt it necessary at the time to offline all services as part of our disaster recovery plan to ensure no damage occurred to the containers while we investigated.

We've migrated all of the Pawb.Social (Lemmy) media from within the container to the new local NAS.

We'll be performing the same updates for pawb.fun and furry.engineer over the course of the next week, minimizing downtime where possible.

We will need to take the services offline once the initial transfer is completed to perform a final check for any new data, and transition to the local NAS, but we'll post an announcement prior to the final maintenance for each instance.

These changes are to allow us longer and cheaper retention of remote and local media across all of our services, and to allow us to begin backing up media in case of a catastrophic failure.

With the upgrade to Mastodon 4.2.0, the way the search system works is changing to allow more broad searches across posts made across the fediverse.

Cutting to the chase: Full-text search across opted-in posts will be possible in the next few days.

In more depth, the system will allow you to search across your own posts and posts you've interacted with from others irrespective of their opt-in status for full-text search, but will additionally allow you to search ANY posts made by users that opt-in to allow their posts to be indexed.

To that effect, a new setting was added to the Public Profile > Privacy and Reach page called "Include public posts in search results" which is disabled by default. Turning this on will allow any user across the Fediverse to search and find your posts using full-text search.

Shortcuts:

• furry.engineer users click here
• pawb.fun users click here

EDIT Aug 2, 1:53 PM US Mountain Time: Maintenance has now concluded and all services are back online!

We're preparing the server rack for an additional Proxmox node to be added to support HA failover and have identified the need to offline and move some equipment around to complete this work.

As such, we'll have a temporary downtime for up to 1 hour from 1 PM US Mountain Time, today August 2nd, or 1 hour from this post. Though, we expect the work to update the node, move it, and bring it back online to take no longer than 30 minutes.

We're hoping these changes will help us maintain a more stable service going forward, spread out the Sidekiq processes (the ones that receive / send data across the Fediverse), and allow us to perform maintenance as necessary without having to offline the service in the future.

We apologize for the short notice as we had hoped to perform this maintenance without any downtime.

Maintenance window: 1 PM - 2 PM US Mountain Time, Au

Around 11:30 PM, one of the core GCFI breakers tripped resulting in the UPS array running until around just after midnight before dropping entirely.

We had not set up monitoring on the APC UPS as we hadn’t anticipated that as a failure point due to the reliability of the Colorado electrical grid, and didn’t anticipate the breaker itself flipping after months of continued use.

All services should be online again and working to catch up on any dropped content overnight, and we apologize again for the inconvenience.

We’ll be trying to identify the root cause throughout the day, working on the electrical connections, and setting up alerting through the APC UPS to detect and persistent A/C loss.

EDIT Maintenance Complete (7/6 8:41 AM): We’ve finished troubleshooting with the ISP modem (and stumbled onto fixing the issue somehow), so we’ve brought pawb.social back online.

The Mastodon upgrade has completed and we’re now running Mastodon+Glitch 4.1.3!

…What’s Glitch? Glitch-soc is a fork of Mastodon that enables a ton of cool new features, accessibility options, and more!

You’ll see new things like:

• An option under the three-dots when posting on the web UI to: Enable threading, or keep your post entirely local to your instance (We’ll be using this going forward with an announcement bot on both pawb.fun and furry.engineer to avoid spamming the timeline)
• Formatted toots! (You can now use markdown language in your toots!)
• More skins and flavors of the web-UI, including a high contrast version

Let us know if you have any issues!

Following our previous post about ISP issues, we'll be performing tests on our ISP equipment tomo

EDIT Mitigated (7/3 10:14 AM): We’ve been temporarily downgraded to 100Mbps to lower the bandwidth stress on the network due to a damaged fiber trunk cable.

The service should be stable now but some requests may take longer than expected due to occassional ping spikes.

An engineer has been scheduled for Wednesday 7/5 and we’ll have an update then. We apologize for any inconvenience.

EDIT (7/3 @ 1:29 AM): The service appears to be mostly functional again and is processing through missed messages, so you may see delays in receiving notifications and feed posts from during the downtime.

We’re still seeing above normal pings and the occasional dropped packets, but the connection seems mostly stable.

We’ll check in again around 10 AM US Mountain Time.

EDIT (7/3 @ 12:52 AM): We've exhausted our troubleshooting at this time and are waiting a repair technician. While the service does appear to be more stable right now, we're still seeing higher than normal pings.

Applications are now closed! Thank you to everyone who applied, we’ll be reviewing the applications this week and be in touch soon!

EDIT: Due to issues with sending DMs between Lemmy and Mastodon, if you're on Pawb.Social submitting an application, please send it to @[email protected] via https://pawb.social/create_private_message/126149 instead.

Pawb.Social services have been steadily growing more and more over the past few months, from the debacle of Twitter and the meltdown of Reddit, we've seen users flocking to not only our instances but those of our wider fediverse community.

Seeking Content Moderators

We're seeking to bring on a small group of volunteer Content Moderators to help with addressing reports, identifying content that breaks the Code of Conduct, and assisting users with concerns and issues.

Please consider the following questions:

• Are you friendly, polite, and willing to be approached by use

We've just updated to the latest version of Lemmy which comes with a ton of bug fixes and an updated UI without websockets (yay, easier debugging).

You can find the full changes over at https://github.com/LemmyNet/lemmy/releases/tag/0.18.0.

This also appears to have fixed an issue with connecting to kbin instances. You should now be able to search those up in the same way as other Lemmy communities: !community@domain, but let us know if there's any issues!