Dev and Maintainer of Lemmy Userdata Migration

  • 5 Posts
  • 23 Comments
Joined 3 months ago
cake
Cake day: April 20th, 2024

help-circle

  • I’ve been tempted by Tailscale a few times before, but I don’t want to depend on their proprietary clients and control server. The latter could be solved by selfhosting Headscale, but at this point I figure that going for a basic Wireguard setup is probably easier to maintain.

    I’d like to have a look at your rules setup, I’m especially curious if/how you approached the event of the commercial VPN Wireguard tunnel(s) on your exit node(s) going down, which depending on the setup may send requests meant to go through the commercial VPN through your VPS exit node.

    Personally, I ended up with two Wireguard containers in the target LAN, a wireguard-server and a **wireguard-client **container.

    They both share a docker network with a specific subnet {DOCKER_SUBNET} and wireguard-client has a static IP {WG_CLIENT_IP} in that subnet.


    The wireguard-client has a slightly altered standard config to establish a tunnel to an external endpoint, a commercial VPN in this case:

    [Interface]
    PrivateKey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    Address = XXXXXXXXXXXXXXXXXXX
    
    PostUp = iptables -t nat -A POSTROUTING -o wg+ -j MASQUERADE
    PreDown = iptables -t nat -D POSTROUTING -o wg+ -j MASQUERADE
    
    PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT
    
    PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT
    
    [Peer]
    PublicKey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    AllowedIPs = 0.0.0.0/0,::0/0
    Endpoint = XXXXXXXXXXXXXXXXXXXX
    

    where

    PostUp = iptables -t nat -A POSTROUTING -o wg+ -j MASQUERADE
    PreDown = iptables -t nat -D POSTROUTING -o wg+ -j MASQUERADE
    

    are responsible for properly routing traffic coming in from outside the container and

    PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT
    
    PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -m addrtype ! --dst-type LOCAL -j REJECT
    

    is your standard kill-switch meant to block traffic going out of any network interface except the tunnel interface in the event of the tunnel going down.


    The wireguard-server container has these PostUPs and -Downs:

    PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

    default rules that come with the template and allow for routing packets through the server tunnel

    PostUp = wg set wg0 fwmark 51820

    the traffic out of the tunnel interface get marked

    PostUp = ip -4 route add 0.0.0.0/0 via {WG_CLIENT_IP} table 51820

    add a rule to routing table 51820 for routing all packets through the wireguard-client container

    PostUp = ip -4 rule add not fwmark 51820 table 51820

    packets not marked should use routing table 51820

    PostUp = ip -4 rule add table main suppress_prefixlength 0

    respect manual rules added to main routing table

    PostUp = ip route add {LAN_SUBNET} via {DOCKER_SUBNET_GATEWAY_IP} dev eth0

    route packages with a destination in {LAN_SUBNET} to the actual {LAN_SUBNET} of the host

    PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip route del {LAN_SUBNET} via {DOCKER_SUBNET_GATEWAY_IP} dev eth0

    delete those rules after the tunnel goes down

    PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark 0xca6c -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -I OUTPUT ! -o %i -m mark ! --mark 0xca6c -m addrtype ! --dst-type LOCAL -j REJECT
    PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark 0xca6c -m addrtype ! --dst-type LOCAL -j REJECT && ip6tables -D OUTPUT ! -o %i -m mark ! --mark 0xca6c -m addrtype ! --dst-type LOCAL -j REJECT
    

    Basically the same kill-switch as in wireguard-client, but with the mark manually substituted since the command it relied on didn’t work in my server container for some reason and AFAIK the mark actually doesn’t change.


    Now do I actually need the kill-switch in wireguard-server? Is the kill-switch in wireguard-client sufficient? I’m not even sure anymore.



  • Oh, neat! Never noticed that option in the Wireguard app before. That’s very helpful already. Regarding your opnsense setup:

    I’ve dabbled in some (simple) routing before, but I’m far from anything one could call competent in that regard and even if I’d read up properly before writing my own routes/rules, I’d probably still wouldn’t trust that I hadn’t forgotten something to e.g. prevent IP/DNS leaks.

    I’m mainly relying on a Docker and was hoping for pointers on how to configure a Wireguard host container to route only internet traffic through another Wireguard Client container.

    I found this example, which is pretty close to my ideal setup. I’ll read up on that.



  • To add to this:

    We have to differentiate between physical and cybersecurity.

    Are you more likely to physically lose your smartphone you carry around with you all day than your full ATX desktop standing in your office? Yeah.

    But let’s consider the consequences for a moment.

    If someone physically stole your desktop, chances are that at least a part of your data isn’t encrypted, the boot sequence probably isn’t (at least completely) verified, and your OS is wide open. There is little to no real isolation in most desktop setups. Once somebody managed to gain access to your system, it is outright trivial to steal your browser sessions, modify commands or run some code, at least in userland.

    Physically stealing your smartphone is easy. But a modern smartphone is usually protected by verified boot and a password+fingerprint/Face ID combo. Unless you take active steps to compromise the security of the phone like rooting/jailbreaking it, disabling verified boot or disabling the passcode, it’s pretty hard if not near impossible to gain access to your data or modify it in a harmful way. If you visit an infected site or install an infected app, the damage is usually confined to that app’s data and the data accessible to it by permissions you probably had to allow to be set in the first place.

    Now that’s speaking to your usual bad actors and usual setups. Exceptions, as always, make the rule. As soon as a sufficiently motivated and technically able actor with access to 0-day exploits, like a state actor, targets you for some reason, all bets are off. But even in this case, due to the advanced verified boot chain on most modern smartphones, those exploits rarely have the ability to survive beyond a reboot.



  • Sure, the code is completely client-side, simply clone it. If you’re running into CORS problems due to the file:// scheme Origin of opening a local file, simply host it as a local temporary server with something like python -m http.server .

    This is due to the two ways most instances validate Cross-Origin requests:


    • Sending Access-Control-Allow-Origin: * (allow all hosts)
    • Dynamically putting your Origin into the Origin header of the response to your requests by the backend

    file:// URLs will result in a null or file:// Origin which can’t be authorized via the second option, therefore the need to sometimes host the application via (local) webserver.



  • The whole point of this being a web app is to make it as easy as possible for the user to download/modify/transfer their user data. LASIM is a traditional app the user has to download and install, similar to a script this web app was developed to replace due to being too difficult to use for some users.

    The import functionality targeted by this API is additive and my app features a built-in editor to add, modify or remove information as the user sees fit. To achieve your stated goal, you’d have to remove anything except the blocked_users entries before importing, which my app supports, I added a wiki entry explaining the workflow in more Detail.

    I may add options to modify the exported data in some ways via a simple checkbox in the future, but I wouldn’t count on it. I’m always open for pull requests!



  • The export/import functionality is, yes. This implementation uses the same API endpoints, but the main reason for this existing:

    An instance I was on slowly died, starting with the frontend (default web UI). At least at the time, no client implemented the export/import functionality, so I wrote a simple script in Bash to download the user data, if the backend still works.

    Running a script can still be a challenge to some users, so I wrote a web application with the same functionality.

    It’s a bit redundant if we’re talking about regularly working instances, but can be of use if the frontend isn’t available for some reason.


  • The export/import functionality is, yes. This implementation uses the same API endpoints, but the main reason for this existing:

    An instance I was on slowly died, starting with the frontend (default web UI). At least at the time, no client implemented the export/import functionality, so I wrote a simple script in Bash to download the user data, if the backend still works. Running a script can still be a challenge to some users, so I wrote a web application with the same functionality. It’s a bit redundant if we’re talking about regularly working instances, but can be of use if the frontend isn’t available for some reason.






  • An dieser Stelle reposte ich nochmal zwei einfache Wege, um seinen User (Settings und abonnierte/geblockte Communities) von einer Lemmy Instanz auf eine andere umzuziehen, beispielsweise von feddit.de auf feddit.org, von meinem ursprünglichen Post unter feddit.de/c/main ( https://alexandrite.app/feddit.de/post/11325409)


    Weg 1, falls man noch einen Browser mit aktiver Session auf feddit.de hat:

    Lemmy bietet seit Version 0.19 eine Funktion an, um die user data zu ex- und importieren. Das geht normalerweise über einen Button in den Settings des Webinterfaces, das geht aktuell bei feddit.de nicht.

    Aber der zugrundeliegende API-Aufruf funktioniert noch, solange man noch mit einem Browser auf feddit.de eingeloggt ist:

    1. Man gehe auf https://feddit.de/api/v3/user/export_settings und speichert die zurückgegebene Datei als irgendwas.json
    2. Man nehme einen (neuen) Account auf einer stabilen Instanz der Wahl, gehe auf /settings und lade irgendwas.json über den Import-Button hoch.
    3. Voilà, man genieße die neue Instanz.

    Das funktioniert mit jeder Instanz >=0.19, man muss lediglich das “feddit.de” in der URL ersetzen. Und wenn das Webinterface funktioniert, geht das auch über den Export- Button in den Settings.


    Weg 2:

    Für die Leute, die keine offene Browser Session haben, hier ein kleines, aber funktionales Bash Script, welches im Ausführungsverzeichnis eine myFedditUserData.json erstellt, welche bei anderen Instanzen importiert werden kann.

    Anforderungen:

    • Linux/Mac OS X /Windows mit WSL
    • jq installiert (Unter Ubuntu/Debian/Mint z.B. per sudo apt install -y jq

    Anleitung:

    • Folgendes Script unter einem beliebigen Namen mit .sh Endung abspeichern, z.B. getMyFedditUserData.sh
    • Script in beliebigen Textprogramm öffnen, Username/Mail und Passwort ausfüllen (optional Instanz ändern)
    • Terminal im Ordner des Scripts öffnen und chmod +x getMyFedditUserData.sh ausführen (Namen eventuell anpassen)
    • ./getMyFedditUserData.sh im Terminal eingeben
    • Nun liegt im Ordner neben dem Script eine frische myFedditUserData.json

    Anmerkung: Das Script ist recht simpel, es wird ein JWT Bearer Token angefragt und als Header bei dem GET Aufruf von https://feddit.de/api/v3/user/export_settings mitgegeben. Wer kein Linux/Mac OS X zur Verfügung hat, kann den Ablauf mit anderen Mitteln nachstellen.

    Das Script:

    #!/bin/bash
    
    # Basic login script for Lemmy API
    
    # CHANGE THESE VALUES
    my_instance="https://feddit.de"			# e.g. https://feddit.nl
    my_username=""			# e.g. freamon
    my_password=""			# e.g. hunter2
    
    ########################################################
    
    # Lemmy API version
    API="api/v3"
    
    ########################################################
    
    # Turn off history substitution (avoid errors with ! usage)
    set +H
    
    ########################################################
    
    # Login
    login() {
    	end_point="user/login"
    	json_data="{\"username_or_email\":\"$my_username\",\"password\":\"$my_password\"}"
    
    	url="$my_instance/$API/$end_point"
    
    	curl -H "Content-Type: application/json" -d "$json_data" "$url"
    }
    
    # Get userdata as JSON
    getUserData() {
    	end_point="user/export_settings"
    
    	url="$my_instance/$API/$end_point"
    
    	curl -H "Authorization: Bearer ${JWT}" "$url"
    }
    
    JWT=$(login | jq -r '.jwt')
    
    printf 'JWT Token: %s\n' "$JWT"
    
    getUserData | jq > myFedditUserData.json
    

    @[email protected] hat mein Script auch in PowerShell nachgebaut, welches unter Windows ohne WSL auskommt: https://gist.github.com/elvith-de/89107061661e001df659d7a7d413092b

    # CHANGE THESE VALUES
    $my_instance="https://feddit.de" # e.g. https://feddit.nl
    $target_file = "C:\Temp\export.json"
    
    ########################################################
    #Ask user for username and password
    $credentials = Get-Credential -Message "Logindata for $my_instance" -Title "Login"
    
    $my_username= $credentials.UserName
    $my_password= $credentials.GetNetworkCredential().Password
    
    # Lemmy API version
    $API="api/v3"
    
    # Login
    function Get-AuthToken() {
        $end_point="user/login"
        $json_data= @{
            "username_or_email" = $my_username;
            "password" = $my_password
        } | ConvertTo-Json
    
        $url="$my_instance/$API/$end_point"
    
        (Invoke-RestMethod -Headers @{"Content-Type" = "application/json"} -Body $json_data -Method Post -Uri $url).JWT
    }
    
    # Get userdata as JSON
    function Get-UserData() {
        $end_point="user/export_settings"
    
        $url="$my_instance/$API/$end_point"
    
        Invoke-RestMethod -Headers @{"Authorization"="Bearer $($JWT)"} -Method Get -Uri $url
    }
    
    $JWT= Get-AuthToken
    
    Write-Host "Got JWT Token: $JWT"
    
    Write-Host "Exporting data to $target_file"
    Get-UserData | ConvertTo-Json | Out-File -FilePath $target_file
    

  • Ein Kollege hat sich sein “normales” Fahrrad selbst mit einem Chinesischem Nabenmotor elektrifiziert, dort ist aus Bastler Sicht alles sehr viel zugänglicher. Für ein zukünftiges E-Bike würde ich persönlich also auf jeden Fall in die Richtung tendieren.

    Dazu sei auch gesagt, dass man gute Teile, um ein bestehendes Rad inklusive Akku auf Pedelec umzurüsten, bereits verhältnismäßig günstig für um die 500€ bekommt. Das sieht dann am Ende nicht so sexy wie ein fertiges E-Bike aus, da die Kabel normalerweise am statt im Rahmen verlegt werden, aber so ein Umbau ist naturgemäß modular und auch transferierbar. Der Umbau ist eigentlich für jeden machbar, Videos gibt es genug.

    Man sollte sich aber mit den Vor- und Nachteilen der verschiedenen Antriebsarten auseinandersetzen, wesentlich hier Mittel- und Nabenmotor. Auch werden viele Kits so ausgeliefert, dass theoretisch eine in Deutschland für Pedelecs nicht zugelassene Konfiguration möglich ist, beispielsweise durch Gashebel und höherer Unterstützung als 25 km/h.


  • Great synopsis!

    The cool thing about GrapheneOS: It provides basically all the comforts and usability as any Android (stock) ROM minus some compatibility issues with a portion of Google Apps and services (Google Pay doesn’t and probably will never work, for example) while providing state-of-the-art security and privacy if you choose to utilize those features. A modern Pixel with up-to-date GrapheneOS, configured the right way, is literally the most secure and private smartphone you can get today.