feature RQ -- Bedrock Worlds backup

My server's been been working great on my MacMini, thanks to this repo! With few friends we've started building a village, with people connecting at various times. It would be a shame if all wonderful builds they create got destroyed.
Bedrock server has the built-in backup functionality which works as I tested. But it is far from automated.
It involves tying in the server console save hold, save query, copy world files and save resume.
I was wondering if it would be somehow possible to automate it. Run with an interval. Perhaps a shell script in the container sending mentioned commands to the Bedrock console and then copying files to an external volume.
I could work on this but don’t have clue where to start...
Maybe I could use something like Expect?

I agree that it would be really nice to automate that process, but yes it would need to be something that could inspect the save query and react accordingly.
Right now the server process is wrapped by this helper tool I developed
https://github.com/itzg/entrypoint-demoter
That tool is currently generic to any type of containerized process; however, I'm thinking I need to spin off a bedrock server specific version of that. With that it could expose bedrock-specific operations such as doing a backup.
Enhancing that Go-based tool might have been more than what you were offering, but that's where I would see the enhancement happening. I have been wanting to expose console access via a web API anyway, so I might prioritize working on both of these changes soon myself.

Great stuff! thanks!

btw.. If you play MC yourself it would be a pleasure to invite u to our lil server, it’s great fun. Let me know!

I'm kind of a solitary LAN-server player, but yeah I should really get out there in the "world" ;) Shoot me your server address in email<EMAIL_ADDRESS>or DM me on Discord and I'll jump on sometime.

I’ve created a little script that seems to be working fine, though I share.
I requres expect and can be run with cron or lunchd every half hour or so.
(You might need to verify your docker path by running which docker)
#!/usr/bin/expect --

log_file <path>backup-world.log
set world_dir <dirs>/worlds
set docker_container <container name>
set world_name <world name>
set backup_dest “<dirs>"
set timestamp [timestamp -format {%Y-%m-%d--%H-%M}]
set world_compress_timeout 240
set world_arch_mv_timeout 30

spawn /usr/local/bin/docker attach --detach-keys=Q $docker_container
expect "DEBU*"
send "save hold\r"
expect "Saving...*"
expect "The command is already running*"
sleep 1

while true {

  send "save query\r"
  expect "Data saved.*" { break }
}

send "Q"
expect eof

spawn bash
send "cd \"$world_dir\"\r"
expect "*$ "

set timeout $world_compress_timeout
send "tar -zcf $world_name.$timestamp.tgz $world_name\r"
expect "*$ "

set timeout $world_arch_mv_timeout
send "mv $world_name.$timestamp.tgz \"$backup_dest\"\r"
expect "*$ "
send "exit\r"
expect eof

set timeout 10
spawn /usr/local/bin/docker attach --detach-keys=Q $docker_container
expect "DEBU*"
send "save resume\r"
expect "Changes to the level are resumed.*"

send "Q"
expect eof
exit

Here is my simple backup-script. Works since one year!
`#!/bin/bash
docker-compose -f "/home/daniel/dropbox/data/docker/3-minecraft/docker-compose.yml" down
_day="$(date +'%A')"
_file="minecraft-data-${_day}.tar"
_path="/home/daniel/dropbox/data/docker-backup/${_file}"
tar -P -cf ${_path} --exclude='bedrock_server*' /home/daniel/docker-data/minecraft --overwrite
docker-compose -f "/home/daniel/dropbox/data/docker/3-minecraft/docker-compose.yml" up -d --build`

I'm sort of new to Minecraft, and I'm investigating setting up a Bedrock server to run on our LAN for our kids and a few of their friends. I don't have a clear understanding of the process of backing up worlds. When you speak of commands like "save hold" and "save query", is that to facilitate backing up a running server? Would it be an acceptable workaround to periodically stop the server, make a backup of the worlds folder and then start the server, assuming this is done when the server is definitely not in use?

@wedgef5 yes, doing the backup while the server is stopped would be a great alternative.

Just wanted to comment that I've been playing around with a tool that can do these sort of backups. I started by creating a CLI tool I could run via systemd on the docker host, and I recently containerized it with a bit of help from reading through the docker-mc-backup scripts. It is able to backup a running server just fine, but it does require access to the host's docker.sock file in order to accomplish this.
It's a bit rough around the edges at the moment because it's not what I'd consider fully containerized and so the config file contains values it doesn't actually need, but it is what I've been using for about 3 weeks now (via systemd) and the last few days (via docker) on my own two-server host.
Docker Hub: https://hub.docker.com/repository/docker/kaiede/minecraft-bedrock-backup
GitHub: https://github.com/Kaiede/docker-minecraft-bedrock-backup
CLI Tool: https://github.com/Kaiede/BedrockifierCLI

Awesome. @Kaiede when you're ready for it, I would be happy to add a "Backups solutions" section to the README that links to your image.

I've done the cleanup I want to do around the configuration. I think I've managed to get it to the "minimum viable product" state that I'm happy with, and the changes have shown to be stable on my home server. So feel free to add the reference.
Mostly worried that the documentation needs some revision, but at this point, will have to see where folks get hung up on configuring things.

There it is:
https://github.com/itzg/docker-minecraft-bedrock-server#solutions-for-backing-up-data

This is awesome. I’m just wondering how that compares performance-wise vs just making dumps of a world via server’s CLI and just tar-ing them? My server runs on a quite low-RAM machine stretched up to limit so wondering if another docker container would take-up much resources.

Containers don't add any memory overhead, so @Kaiede 's backup container would only occupy the memory for the bash script to sleep most of the time. Since the tool is compiled from Swift, I'm guessing it has very low memory footprint for those moments it runs.

The processes running in the container on my server are eating about 4KB of RAM total while sleeping. When backing up my two containers and trimming the backup list (one 80MB world, and a 300MB world), the peak RAM stayed under 60MB (RSS) and took less than one second (NVMe SSD).

Since the tool is compiled from Swift, I'm guessing it has very low memory footprint for those moments it runs.

Depends on what you compare it to. Something like the POSIX suite of tools are hard to beat. But compared to say, the node process for manymine, or dockerd, it uses less. The Swift runtime isn’t quite as good as the C/C++ runtime in memory usage.

I've done the cleanup I want to do around the configuration. I think I've managed to get it to the "minimum viable product" state that I'm happy with, and the changes have shown to be stable on my home server. So feel free to add the reference.
Mostly worried that the documentation needs some revision, but at this point, will have to see where folks get hung up on configuring things.

Just setting a new world for my server so decided to finally give it a go :-)
Hunged-up on paths config... :-|
docker-compose for my server for reference:
`version: '3.4'
volumes:
emer2_vol:
external: true
services:
bds2:
container_name: emer2
image: itzg/minecraft-bedrock-server
environment:
  # blah blah
ports:
  - 19133:19132/udp
  - 39134:39134
  - 39134:39134/udp
  - 39135:39135
  - 39135:39135/udp
sysctls:
  net.ipv4.ip_local_port_range: 39134 39135

volumes:
  - emer2_vol:/data

stdin_open: true
tty: true
restart: unless-stopped

backup:
image: kaiede/minecraft-bedrock-backup
name: emer2_backup
restart: always
depends_on:
- "bds2"
environment:
BACKUP_INTERVAL: "3h"
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - /opt/bedrock/backups:/backups
    - /opt/bedrock/server:/server`

I would like to save the backup in e. g.  /Volumes/Storage/mine/emer2/backup/world. Do I need to create a external volume and define it in docker-compose ?
(also not sure if should just post this in https://github.com/Kaiede/docker-minecraft-bedrock-backup)

@MattrCoUk since you designated emer2_vol external, then yes you'll need to create that yourself. You also need to attached emer2_vol to the backup container at /data in order to point it at the content to backup.

Thanks for working on this @Kaiede
Will this solution also work on a Win10 host with appropriate path changes?

I would like to save the backup in e. g. /Volumes/Storage/mine/emer2/backup/world. Do I need to create an external volume and define it in docker-compose, similarly as I do for the data volume? or do I just replace the /opt/bedrock/backups bit. Also is the external data volume going to work with the above setup?

@MattrCoUk : My example for the backup is mostly there to give an idea what's possible. Whatever you map to /backups is where backup files get written to, and where config.json is expected to be, by default. So replace /opt/bedrock/backups with whatever you want so you can access things externally, such as /Volumes/Storage/mine/emer2/backup/world. I will say one thing though, is that I have never quite figured out how to deal with named volumes for restoring backups. It's honestly easier to restore while the server and backup containers are shut down, which makes it harder to access named volumes (something I never figured out how to do). So I'd generally recommend using external volumes for both containers so it's possible to shut down the containers, unzip the *.mcworld file into the world folder, and then start the containers back up.

Will this solution also work on a Win10 host with appropriate path changes?

In principle, it should, the same way running the container on macOS works. They both have to virtualize Linux to run either the backup container or the bedrock server container. So just keep in mind that paths inside the container are still going to be *nix paths.

So I’ve set it up on both of my worlds and it seems to be working splendidly!
here’s my docker-compose for reference
uses 2 external volumes
  version: '3.4'

  volumes:

    # create volumes before running docker-compose:
    # docker volume create --opt device=path/to/server/data --opt o=bind bs_data_volume --opt type=none
    # docker volume create --opt device=path/to/backup --opt o=bind bs_backup_volume --opt type=none

    bs_data_volume:
      external: true

    bs_backup_volume:
      external: true

  services:

    bedrock_server:

  		container_name: bedrock_server

      image: itzg/minecraft-bedrock-server

      volumes:
        - bs_data_volume:/data

      # ...

    bs_backup:

        container_name: bs_backup

        image: kaiede/minecraft-bedrock-backup

        restart: always

  			depends_on:
          - "bedrock_server"

  			environment:
          BACKUP_INTERVAL: "3h"

        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - bs_backup_volume:/backups
          - bs_data_volume:/server

server's container_name and the service name has to be the same for backups to work correctly

docker-compose will name a container with a prefix if you don't specify the container_name manually. So if I have my compose file at: /opt/mygreatserver/docker-compose.yml and name the service bedrock_server the container winds up being named mygreatserver_bedrock_server, and other containers get the same prefix, effectively grouping them together. As long as the name docker itself sees is in the config.json, it will work.
So yeah, naming it explicitly is a good way to keep things easier to remember and work with for sure.

would be nice to have an option to run backups on schedule (such as 6:00AM, 12:30PM etc.)

Feel free to open an issue on my github repo for this. I intentionally left it out of the first iteration because:

Making users enable cron on the docker host is effectively a support time sink.
Integrating something cron-like in the container itself is pretty heavyweight.
I have ideas that would require turning the backup tool itself into an always-running service monitoring the bedrock server (triggering backups on logout/login). If I go that route, then cron behaviors really should be handled by this service directly.

I figured being able to trim the backups, and run them a bit more frequently would help address some of the need for cron scheduling in the short term at least.

I agree with @Kaiede about the hesitations for directly supporting cron schedules. At the risk of being a heavy answer to an easy question, but kubernetes CronJob workloads are a robust way to support scheduled container activities
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

I’ve noticed that on one of my servers the backup produces some additional files. Other server with a similar backup settings  doesn’t have them. Wonder if those are important.

I’ve noticed that on one of my servers the backup produces some additional files.

Right now, when a backup is fired, all worlds detected in the worlds folder for a server are backed up. A “valid” world is one where there is a folder with “levelname.txt” inside, and the tool can extract a useful string from that file.
The main issue here is that since the tool is unaware of what world the server is using, or if the user might have swapped out the worlds between backups firing, it just backs it all up. This is something I can probably document better.
So what’s likely happening here is that you’ve got a second folder in your server’s worlds folder, and it’s levelname.txt file contains the string “PMC87f…”. I can’t really say anything about what created that folder in your case, but if you were to dig into your server directory and find that folder, and clean it up, these extra backups would stop being made. Being only 6KB in size, it almost seems like a world that was “created” but never properly named or logged into.

I’ve noticed that on one of my servers the backup produces some additional files.

Right now, when a backup is fired, all worlds detected in the worlds folder for a server are backed up. A “valid” world is one where there is a folder with “levelname.txt” inside, and the tool can extract a useful string from that file.
The main issue here is that since the tool is unaware of what world the server is using, or if the user might have swapped out the worlds between backups firing, it just backs it all up. This is something I can probably document better.
So what’s likely happening here is that you’ve got a second folder in your server’s worlds folder, and it’s levelname.txt file contains the string “PMC87f…”. I can’t really say anything about what created that folder in your case, but if you were to dig into your server directory and find that folder, and clean it up, these extra backups would stop being made. Being only 6KB in size, it almost seems like a world that was “created” but never properly named or logged into.

Ahhhh, I forgot that I had to change the server's name temporarily in order to verify it on planet.minecraft. It generated a new world..... Silly me!

Would anybody be able to help me understand the backup process a bit better? I've been reading about "save hold", "save query", screen, expect, etc. for a couple of hours now and still not quite comprehending everything.
Taking the docker commands (docker stop, docker start, docker attach) out of the picture; if we wanted to take a backup from INSIDE the running container is there a straightforward process for doing so?
@Kaiede I've looked at your BedrockifierCLI tool and that looks like it is used outside of the docker container, am I interpreting that correctly?
Thank you for any guidance or information.

Taking the docker commands (docker stop, docker start, docker attach) out of the picture; if we wanted to take a backup from INSIDE the running container is there a straightforward process for doing so?

Not really. You need to have access to the input/output of the server's console to issue commands to do this safely. So while you could write a script that runs inside the container, it would look a lot like a script that runs outside the container. The crux of taking a backup while the server runs relies on issuing the "save hold" and "save resume" commands. Those commands are there to tell the server to stop writing to disk for a moment so the backup can be made ("save hold"). Once the backup is made, you can issue "save resume" again to tell the server it's safe to write to disk again. Needing access to the server's console is what makes it a bit of a pain to get backup scripts working right.
The alternative is to shut down the server, make the backup and start it again instead, which achieves the same result, but brings down the server for anyone logged in while it does it. (This is the docker stop/start approach)
Docker itself isn't necessarily why it is complicated, and makes both approaches above a bit easier, to be honest.

I've looked at your BedrockifierCLI tool and that looks like it is used outside of the docker container, am I interpreting that correctly?

It currently assumes that the server to be backed up is in a docker container, but the tool itself can be used outside a container. I used it what way to backup my server container for a bit while I was working on the containerized version of the backup tool. But it doesn't support being used either in the same container as the server, or if the server isn't in a docker container at all.
The catch is that BedrockifierCLI can only really be built for Mac/Linux (Swift on Windows is still pretty experimental). So the containerized backup tool is honestly more compatible, and more "set and forget".

I've gotten this solution working on Win10 Pro. Thanks to @Kaiede for putting it together! Below is my docker-compose.yml which is using a mount to the host NTFS filesystem. Docker will issue a warning about this due to relatively poor performance of using Windows host FS mounts like this. I figure with the relatively low amount of IO that it shouldn't be an issue. I've also been investigating using a WSL2 Ubuntu instance to host the backup data, but I haven't actually tried it yet. It seems that it should be possible, though. My primary goal is to have the backed up worlds out of the container where they can be picked up by my cloud backup solution (iDrive). The easiest way to achieve that is for the backup to be on the host FS.
version: '3.4'

services:
  bedrock:
    image: itzg/minecraft-bedrock-server
    container_name: bedrock_server
    restart: always
    environment:
      EULA: "TRUE"
      SERVER_NAME: "Home Server"
      WHITE_LIST: "TRUE"
      WHITE_LIST_USERS: "wedgef5"
      GAMEMODE: creative
      DIFFICULTY: normal
      ONLINE_MODE: true
    ports:
      - 19132:19132/udp
    volumes:
      - bds:/data
    stdin_open: true
    tty: true
  backup:
    image: kaiede/minecraft-bedrock-backup
    container_name: bedrock_backup
    restart: always
    depends_on:
      - "bedrock"
    environment:
      BACKUP_INTERVAL: "3h"
      TZ: "America/Chicago"
    volumes:
      - //var/run/docker.sock:/var/run/docker.sock
      - G:/Minecraft-Backup:/backups
      - bds:/server
volumes:
  bds: {}

@wedgef5 Plumbing data around is always a pain, for sure.
Since I use B2 for my NAS backups already, I added a duplicati container to handle uploading my backups directory to a B2 container that houses my VM backup data. It looks like duplicati can be configured to upload to iDrive as well. That would let you wrap the whole thing up in either a Ubuntu VM or WSL2 setup.
I generally found a Hyper-V or VMWare VM was a bit more efficient (power, performance) than using Docker Desktop.

First and foremost, I'd like to thank. Thank you very much @itzg and @Kaiede for your efforts!
I know this is an old issue, but I still have questions ;-)
@Kaiede Am I assuming right that both the cli and container of your backup utility expect to run on a docker host and to have access to the docker socket? If true, I unfortunately can't use it, as I run my servers on Kubernetes.
My second assumption is that the way @DocBrown101 is saving his worlds (backing up the data directory) only works reliably if you stop the servers. I assume otherwise I can end up with a corrupt backup. Therefore that's also not an option. If the server doesn't need to be stopped, I could easy backup the PV containing the data.
I then read trough @itzg backup section of the bedrock helm-chart readme. And here I end up with my (little) Minecraft understanding. The readme say basically to save hold, save query and then to backup the data directory. The command save query tells me this:
[2023-10-24 07:08:35:392 INFO] Data saved. Files are now ready to be copied.
world/db/CURRENT:16, world/db/000016.ldb:208360, world/db/000015.log:0, world/db/MANIFEST-000013:354, world/level.dat:2733, world/level.dat_old:2733, world/levelname.txt:5

I read somewhere else that the server will continue writing to the files and I need to backup only the parts save query tells me and truncate (really ?!) the files accordingly. This is something else than the readme tells me.  I'm confused, but would trust @itzg and it's readme a bit more.
I guess the final goal is to end up with a .mcworld file. Of all things above, only the toll @Kaiede made would create that.
To cut a long story short, my question:

How would the correct way to backup be on Kubernetes?

Thanks a lot for inputs!

I will have to defer to @Kaiede since the extent of my bedrock backup knowledge is captured in that section you linked, @tuxpeople .

Am I assuming right that both the cli and container of your backup utility expect to run on a docker host and to have access to the docker socket? If true, I unfortunately can't use it, as I run my servers on Kubernetes

Currently, yes. As you have noted there’s a corruption risk without calling save hold first. So a backup tool needs to be able to talk to the server console. Rcon isn’t supported on bedrock, so there’s not much else that can be done other than talk to the console directly last time I looked at things.
If something like rcon existed in the bedrock container then I could probably make changes to support it and use a private network between the containers to issue the commands instead. That should be more agnostic to the container engine being used, and more secure when used with the private network.

@itzg No worries. I could ask my son if it's a question concerning in-game knowledge, but not for server stuff 😂
@Kaiede I think I understand: There's currently no way to do the necessary commands over the network, therefore you have to rely on attaching to the container to directly issue commands to it.
There's something for Kubernetes similar to docker attach: kubectl attach. I can use it to isse commands interactively, however, I don't see a way to script it atm. Am I assuming right that there's also no way to issue the neccessary commands from inside the bedrock container itself? And I also assuming correctly that the "normal" client protocol is unsuitable for this?
I have a way to issue commands to the server from within the running bedrock container. And also to read the response without asking @itzg to add additional tools to the container. Let's see how far I can go with this approach. Can you (@Kaiede) tell me how you create a .mcworld file from this:
[2023-10-24 07:08:35:392 INFO] Data saved. Files are now ready to be copied.
world/db/CURRENT:16, world/db/000016.ldb:208360, world/db/000015.log:0, world/db/MANIFEST-000013:354, world/level.dat:2733, world/level.dat_old:2733, world/levelname.txt:5

Thanks!

I think I understand: There's currently no way to do the necessary commands over the network, therefore you have to rely on attaching to the container to directly issue commands to it.

Correct. And this process is specific to the container engine being used. Kubernetes, being built around orchestration across a cluster of nodes, introduces wrinkles. I haven’t worked with it, so I’m not sure how straight-forward it would be to support. Not only does the backup container need access to the host’s docker instance, it needs to be able to share the data volume with the minecraft container. This gets messy in cases where the two containers might wind up on different nodes when using Kubernetes. I’m not sure how orchestration works when dealing with shared volumes.
I only run two nodes (one x86 and one arm64), so haven’t really needed something with cluster capability.

Am I assuming right that there's also no way to issue the neccessary commands from inside the bedrock container itself? And I also assuming correctly that the "normal" client protocol is unsuitable for this?

The normal client protocol is unsuitable because it means implementing a full client login with a Microsoft account, which is built for interactive logins. Note that logging into a server assumes that the client has already performed authentication and provides the token (the JWT listed in the protocol spec you shared) to the server. Also, I don’t remember if save commands are available to operators via the Bedrock client. So this would also be messy for an end user to configure and operate.
Keep in mind, the requirement of host docker access is needed for scripting of any kind from another container. For the backup container it’s just easier to attach and interact with the console directly, as the requirements are the same as asking the container to run scripts for you.

Can you (@Kaiede) tell me how you create a .mcworld file from this:

An .mcworld file is just a zip file of the world. It’s pretty simple once you know what the “root” of the zip file needs to be. If you really want to know more about how it works. you can export a single-player world from Bedrock on Windows, and rename the .mcworld to .zip and take a look inside. Or examine an existing mcworld file on linux using the zip tool to list the contents.
I wasn’t aware of the truncation requirements (Mojang doesn’t document this stuff), so my tool is rather crude here. It just bulk grabs everything on both Java and Bedrock. My backups have restored properly when I needed to do so, and I didn’t encounter corruption, so it just slipped under the radar. I’ll have to take a look to see if I can make the backups more correct going forward.
But thinking about this longer term, depending on how reliable the Bedrock console behavior is, I kinda think wrapping the server in a tool that translates to Rcon (or at least provides some sort of semi-secure console access over a docker/kubernetes private network) is the right approach. It would require integrating the work into the server container, but it is the cleanest way, and supporting Rcon would make it easier for tools to support both Bedrock and Java like mine does.

Correct. And this process is specific to the container engine being used. Kubernetes, being built around orchestration across a cluster of nodes, introduces wrinkles. I haven’t worked with it, so I’m not sure how straight-forward it would be to support. Not only does the backup container need access to the host’s docker instance, it needs to be able to share the data volume with the minecraft container. This gets messy in cases where the two containers might wind up on different nodes when using Kubernetes. I’m not sure how orchestration works when dealing with shared volumes.

I do Kubernetes for living, therefore I know a thing or two. But you're right, it would be a bit messy. Underneath Kubernetes, you can have different container runtimes and also different storage plugins. Therefore, I can see only three approaches to Kubernetes backup which do make sense:

Rely on the Kubernetes API to interact with the pod (container)
Run the backup from inside the minecraft-bedrock-server container (would also work on Docker)
Interact with the Minecraft server directly

The last thing is not possible on Bedrock due to the lack of rcon.

Can you (@Kaiede) tell me how you create a .mcworld file from this:

An .mcworld file is just a zip file of the world. It’s pretty simple once you know what the “root” of the zip file needs to be. If you really want to know more about how it works. you can export a single-player world from Bedrock on Windows, and rename the .mcworld to .zip and take a look inside. Or examine an existing mcworld file on linux using the zip tool to list the contents.

Yep, I see it now. Thanks!

I wasn’t aware of the truncation requirements (Mojang doesn’t document this stuff), so my tool is rather crude here. It just bulk grabs everything on both Java and Bedrock. My backups have restored properly when I needed to do so, and I didn’t encounter corruption, so it just slipped under the radar. I’ll have to take a look to see if I can make the backups more correct going forward.
But thinking about this longer term, depending on how reliable the Bedrock console behavior is, I kinda think wrapping the server in a tool that translates to Rcon (or at least provides some sort of semi-secure console access over a docker/kubernetes private network) is the right approach. It would require integrating the work into the server container, but it is the cleanest way, and supporting Rcon would make it easier for tools to support both Bedrock and Java like mine does.

My current status on backup is the following: I do have a working backup script to be run inside the container to issue the commands needed and copy/truncate the files. The script will compress the backup and write it into a folder. I was able to successfully import a backup of my server into my iPads Minecraft. The script gets automatically added to the container by Kubernetes and I can set an optional env-var to specify a name for the backup.
As @itzg makes good containers (they do not contain unnecessary tools), I had some issues :-) The current version of script has the following caveats:

I need to do some nasty stuff to get the PID of the Minecraft server process (as I can't be sure that it's always PID 11). This could be made more mature if @itzg would write the PID of the server process into a file.
I can't create .mcworld files as there's no zip utility in the container. I'm currently doing .tar.gz. That's good enough for backup, but not if one would like to use the backup for importing it into a Minecraft client like the iPad Minecraft.
It can't be run regularly based on a schedule (but can be run manually if a backup should be made)
Script needs some cleanup and a few more variables to be configured using env vars.
Script has currently clue of retention periods and deletion of old backups.

@itzg would you consider merging a PR for adding zip to the container? It would allow the creation of .mcworld files, which are ZIP files.
With ZIP I can conduct some testing and I'd be happy to open another PR later to discuss the possibility of adding a backup script to the container.

@tuxpeople I would prefer backup operations to be separate from this image since the final step exec's off to the bedrock server executable. With that said, I don't mind zip being added to the packages, but be aware that I may choose to reject a PR that overall extends the container's functional scope.

@itzg I completely see your point. What I propose is far from what's considered best practices. It's not in alignment with general container best practices and also not with Kubernetes best practices. Therefore, I'm not happy with as well. But as far as I can see, it's the best of all the bad options. But maybe I'm missing something.
Access to the files is not a problem. Regardless of the storage solution used for the Kubernetes deployment. The big issue is to speak with Bedrock Server. What I'm doing now in the script is to echo the commands into the server process's stdin and read the answers from its stdout. This can only be done from inside the Bedrock container.
I can do the same stuff with speaking to the Kubernetes API. Probably similar to what  @Kaiede does with the Docker socket. But I'm not a programmer. Therefore I can't do some programming magic and would need to rely on the standard Kubernetes CLI: kubectl attach <pod> gives me access to the I/O of the container. But it's not directly scriptable. I would have to use something like expect. But I'm not sure how mature and reliable this is.
May I ask you for your opinion on this? Not necessary on the expect thing, more on the issue in general.

@tuxpeople you beat me to my follow-up / redaction on some of my comments and concerns 😄 . The idea of a bundled script that is exec'ed is growing on me -- I was jumping to a conclusion that the periodic scheduling would be part of the introduced behavior, but that can be driven externally. Like what you're saying, there's not really an ideal solution without a remote access interface on Bedrock's part. So co-execution makes the most sense given the constraints.
So, go for it! We can iterate and discuss more via a PR when you're ready.

Personally, I’d still vote for some sort of console shim rather than baking functionality.
But that’s because it means I can move my container to it, which makes docker/kubernetes support optional for containers trying to do something similar (like mine), rather than fracturing stuff into different techniques.
I can’t guarantee much, but I could try to make time to whip some sort of prototype up using python or the like.

Thanks @Kaiede, that's where I'm still wanting to lean in order to keep the separation of concerns between containers.
Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

I stumbled upon this, which says Bedrock has rcon: https://help.craftingstore.net/games/minecraft-bedrock But I assume that's wrong.
@itzg Currently, I use your Helm chart to deploy a Kubernetes CronJob which starts the script within your container. The script is being added to the container via a ConfigMap. But using a Kubernetes CronJob to do kubectl exec is also something not winning a beauty contest, IMHO. But it's not our fault Bedrock can't speak with us :-) Btw. I didn't reinvent the wheel for the script, some of the logic is borrowed from here.
Curent version of my script is here. CronJob for scheduling and conversion to zip (.mcworld) is here.
I also found this: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ which would allow moving the whole backup stuff into its own container to be added as a sidecar. But this needs elevated privileges as normally different containers in a pod have segregated process namespaces. The question would be: what about users who do not have the possibility to grant such privileges?
@Kaiede There's an official Python client library for Kubernetes (the normal CLI client is written in Go) here: https://github.com/kubernetes-client/python there's a short note about attach calls at the end of the readme (Why Exec/Attach calls doesn't work), just in case you're curious.

Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

Yeah, going after the stdout/in file descriptors does complicate things a little. It's great for something like what you are doing here, but it's probably easier to let a process simply sit between the container's out/in and the server's out/in and expose a socket from there. Go would probably be perfect for this, and maybe it's the excuse I need to give it a try. Swift would be easy for me, but the binaries aren't nearly as portable and easy to integrate into a container right now.

I stumbled upon this, which says Bedrock has rcon: https://help.craftingstore.net/games/minecraft-bedrock But I assume that's wrong.

It's wrong. I suspect someone copy-pasted some Java content or maybe even asked GPT to write it (you never know these days).

@Kaiede There's an official Python client library for Kubernetes (the normal CLI client is written in Go) here

I was talking about writing a shim that implements rcon so that you can use it with Bedrock. The main thing having rcon built in provides you is that you get a bit more clarity on what command triggered what output. If you don't care about that, you can use rcon as a pure console.
I could directly support Kubernetes in my backup tool, but if I'm going to spend effort to help enable Kubernetes, I think I'd rather spend effort making the backup process free from Docker-isms. I have gotten reports from users that are in situations where they can't give my tool access to Docker/Kubernetes on the host in order to do the attach, but they can configure a private network between containers. So "fixing" the lack of rcon in Bedrock has wider impacts and helps more folks in the long run, based on what I've seen so far.

Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

Yeah, going after the stdout/in file descriptors does complicate things a little. It's great for something like what you are doing here, but it's probably easier to let a process simply sit between the container's out/in and the server's out/in and expose a socket from there. Go would probably be perfect for this, and maybe it's the excuse I need to give it a try. Swift would be easy for me, but the binaries aren't nearly as portable and easy to integrate into a container right now.

I also just access the processes stdin and stdout to do the communication, what you're describing here would be much better. I'm a big fan of go binaries when creating containers etc. Can't speak for the language though, as I'm not a programmer. I tried to learn go, but not doing programming stuff it's hard. I need something "real" to learn something, I'm not good at learning things the academic way.

@Kaiede There's an official Python client library for Kubernetes (the normal CLI client is written in Go) here

I was talking about writing a shim that implements rcon so that you can use it with Bedrock. The main thing having rcon built in provides you is that you get a bit more clarity on what command triggered what output. If you don't care about that, you can use rcon as a pure console.

Sorry, my bad. As you may have noticed, Engish is not my primary language and sometimes I missunderstand things while reading.

I could directly support Kubernetes in my backup tool, but if I'm going to spend effort to help enable Kubernetes, I think I'd rather spend effort making the backup process free from Docker-isms. I have gotten reports from users that are in situations where they can't give my tool access to Docker/Kubernetes on the host in order to do the attach, but they can configure a private network between containers. So "fixing" the lack of rcon in Bedrock has wider impacts and helps more folks in the long run, based on what I've seen so far.

I'd like the idea. Sounds good to have a runtime-agnostic, stable solution that does not need elevated privileges. Networking access between Bedrock container and a "backup" container would be easy in many environments. Whether it's Docker, Kubernetes, or anything else running containers.

Speaking of a shim, perhaps there's room for improvement in the "console shim" I have so far: https://github.com/itzg/docker-minecraft-bedrock-server/blob/master/bin/send-command . Reading out from the process' stdout seems non-trivial since I'm not sure it'll multiplex across the container's stdout and an arbitrary read from there.

I wasn't aware of that and reinvented that in my script 😇

Yeah, going after the stdout/in file descriptors does complicate things a little. It's great for something like what you are doing here, but it's probably easier to let a process simply sit between the container's out/in and the server's out/in and expose a socket from there. Go would probably be perfect for this, and maybe it's the excuse I need to give it a try

Funny thing is that https://github.com/itzg/mc-server-runner (used by the Java edition image) partly exists to wrap the stdin when rcon isn't enabled. It only writes a "stop" line, but perhaps that would be an enhancement vector to add a simple  REST API to enable remote stdin/stdout access.

So, I've added initial support for rcon into my backup container. It will let you use rcon for both Bedrock and Java containers, but clearly Bedrock won't work. The downside to rcon is that it doesn't support the server pushing anything to the client. It doesn't really need to, and so that's generally fine, but it does mean triggering backups on events doesn't work. Once I get a chance to let the validation run a bit, I'll merge the PR and update the documentation: https://github.com/Kaiede/Bedrockifier/pull/73
Thinking on it more, I wonder if the better approach is use rlogin for the wrapper? It at least aligns properly with exposing stdout/in directly which would simplify things considerably. It's not uncommon for folks to use the server username field as the password field for rlogin, which would make it about as secure as rcon in this case (i.e. terrible). It would also make it possible to monitor the output of the server in ways that rcon doesn't allow or make sense for, meaning it would be something that makes sense to include in mc-server-runner for both bedrock/java and enable it if a password is provided to configure it?
REST is also an idea, although if we wanted to support monitoring of the service output for certain events, that's a whole thing that we could bike shed for a while.

Funny thing is that https://github.com/itzg/mc-server-runner (used by the Java edition image) partly exists to wrap the stdin when rcon isn't enabled. It only writes a "stop" line, but perhaps that would be an enhancement vector to add a simple REST API to enable remote stdin/stdout access.

Just to make sure I understand, you are suggesting maybe forking this for building out the wrapper and then folding it back via a PR once it works?

So, I've added initial support for rcon into my backup container. It will let you use rcon for both Bedrock and Java containers, but clearly Bedrock won't work. The downside to rcon is that it doesn't support the server pushing anything to the client. It doesn't really need to, and so that's generally fine, but it does mean triggering backups on events doesn't work. Once I get a chance to let the validation run a bit, I'll merge the PR and update the documentation: https://github.com/Kaiede/Bedrockifier/pull/73
Thinking on it more, I wonder if the better approach is use rlogin for the wrapper? It at least aligns properly with exposing stdout/in directly which would simplify things considerably. It's not uncommon for folks to use the server username field as the password field for rlogin, which would make it about as secure as rcon in this case (i.e. terrible). It would also make it possible to monitor the output of the server in ways that rcon doesn't allow or make sense for, meaning it would be something that makes sense to include in mc-server-runner for both bedrock/java and enable it if a password is provided to configure it?
REST is also an idea, although if we wanted to support monitoring of the service output for certain events, that's a whole thing that we could bike shed for a while.

Funny thing is that https://github.com/itzg/mc-server-runner (used by the Java edition image) partly exists to wrap the stdin when rcon isn't enabled. It only writes a "stop" line, but perhaps that would be an enhancement vector to add a simple REST API to enable remote stdin/stdout access.

Just to make sure I understand, you are suggesting maybe forking this for building out the wrapper and then folding it back via a PR once it works?

Just to make sure I understand, you are suggesting maybe forking this for building out the wrapper and then folding it back via a PR once it works?

No forking needed -- that one is my project also 😀. I'm saying, it could be enhanced with a REST endpoint and I'll bump the version used by the image to pick that up.

I think we are saying the same thing. The fork would be mine so I can add the functionality before creating a PR to merge it back into your repo.
I'm playing around a bit with a copy of mc-server-runner at the moment. I've got the skeleton of a "RESTCON" server implemented that uses basic auth and re-uses the RCON password (it ignores the username for now). The bit that I need to figure out at the moment is when to consider the response from stdout "complete" and send it back to the client for Bedrock. Need to experiment with that a bit locally.
I've also drafted up a small API for registering listeners with the REST API. This would let containers like mine provide matching patterns to mc-server-runner and have it issue a callback to a URL when the pattern is matched with the content of the matching line. This would allow my event-based backups to still work with this approach, unlike RCON.

Yes we are 😀. Awesome!

I’m bailing on the REST approach BTW. The issue is race conditions. Because I’m trying to wrap the server in/out directly, most of the time the next output is going to be what is returned from the command, but that’s not a guarantee. You can get races where some other logged event makes it to the console first. So you either need to do some sort of mapping, or risk that sometimes you’ll get something entirely different than expected and get random “failures” from issuing a command. Not a great foundation for things that need to be reliable over days and weeks, unfortunately. Ultimately, you need such functionality baked into the game itself (i.e. rcon) to avoid this.
So really some way to export the interactive session over the network is needed. I see that gliderlabs has an SSH package to make writing a server or client as easy as using net/http, which is promising. And it should be possible to disable/ignore things like port forward requests from the client to minimize the exposure of the container’s internals.

...and I think that's why I never started on that endeavor 😀. I think anything stream/ssh, etc based is going run into challenges of non-deterministic delineation of the output.

...in the grander scheme of things, I'm starting to recommend people just move away from the bedrock server software and use Java edition with Geyser for equivalent client support
https://docker-minecraft-server.readthedocs.io/en/latest/misc/examples/#bedrock-compatible-server

...and I think that's why I never started on that endeavor 😀. I think anything stream/ssh, etc based is going run into challenges of non-deterministic delineation of the output.

Which is why my service has an expect-like processing flow with timeouts.
I’m trying to also be cognizant that anything added to mc-server-runner should be light-weight. Not looking to add a ton of complexity that has to be managed/maintained. One advantage of rlogin/ssh and pushing the complexity to the client is that the server is essentially just exposing the in/out streams as a connectable PTY, and keeps things fairly clean on your end.

...in the grander scheme of things, I'm starting to recommend people just move away from the bedrock server software and use Java edition with Geyser for equivalent client support
https://docker-minecraft-server.readthedocs.io/en/latest/misc/examples/#bedrock-compatible-server

I might have to play with that, but it does still mean that rcon + Bedrockifier can’t do things like wait for your server to be empty to kick off backups which works well in closed servers with a small number of users. So I’m still interested in something like the above for Java as well, to be able to start phasing out the need for Docker integration. That would solve something like 80% of my support requests right there.

Just as an FYI, I setup a draft PR here with a couple questions since I am relatively new to Go: https://github.com/itzg/mc-server-runner/pull/56
It enables SSH for remote console access on the container as part of mc-server-runner. In principle it should handle multiple simultaneous clients (including someone attached via docker as normal). It's a little over-engineered, but at the very least this should be easier to keep secure than rcon.

Thanks for the quick review @itzg. Now we just need a means to trigger the command-line argument for mc-server-runner on the bedrock/java container and we should be good to go there.
I've made changes to my container to support connecting via SSH, which means the next release of my container will support docker, ssh, and rcon as means to talk to the Minecraft server. rcon just isn't fully featured like the other two.
I just need to resolve an issue where ssh + entry point demoter don't like each other this weekend, and that will be ready to test the whole E2E on my local Linux VM. That should help unblock Kubernetes users like @tuxpeople, folks on NAS boxes, and really just anyone where the docker method is tricky or just not feasible.

@Kaiede let me know if I can be helpful. Time is currently limited, but I'm happy to help as good as I can.

@Kaiede let me know if I can be helpful. Time is currently limited, but I'm happy to help as good as I can.

I got lucky last night and was able to do some end-to-end testing of all this and was able to successfully backup Java and Bedrock servers, and trigger it via player logout over SSH, demonstrating that both listening and control functionality worked as expected between my backup container and the changes I made to @itzg’s containers.
What’s left is:

Do some validation on the bedrock container changes to be more confident that adding mc-server-runner to the container doesn’t regress anything there.
Do some validation on the backup container changes to ensure a fix I had to make for SSH to work (SSH requires carriage-return + linefeed, but rcon-cli and docker do not) doesn’t regress the docker and rcon functionality.

Once that’s done, then I should have a build on my test tag you can try with some docker-compose examples checked into the repo’s Examples folder of how to configure things for SSH. Unfortunately, translating that to Kubernetes will be up to you.
I’ll need a bit of time to update my wiki before I move the changes from test to latest, as I will be recommending people use SSH by default going forward, with docker as a fallback for a while. As docker amounts to around half of my container size, I may even look at building a “slim” tag that omits docker support entirely.

@Kaiede let me know if I can be helpful. Time is currently limited, but I'm happy to help as good as I can.

I'll just ping this one last time to say this work is now in the latest tag for itzg/minecraft-bedrock-server, itzg/minecraft-server and Kaiede/minecraft-bedrock-backup (which is increasingly poorly named these days). So it should be possible to configure something using SSH with Kubernetes.

@Kaiede Sorry for the delay. I haven't seen it. I'm currently trying to set it up. Not yet successful, but I'm working on it.
I've some feedback/observations. My logs currently looks like this:
/usr/sbin/deluser: Only root may remove a user or group from the system.
useradd: Permission denied.
useradd: cannot lock /etc/passwd; try again later.
adduser: Only root may add a user or group to the system.
time="2024-02-24T12:28:58+01:00" level=debug msg="Using /backups to match uid and gid"
time="2024-02-24T12:28:58+01:00" level=debug msg="Resolved UID=1024 from match path"
time="2024-02-24T12:28:58+01:00" level=debug msg="Resolved GID=100 from match path"
time="2024-02-24T12:28:58+01:00" level=info msg="Skipping uid/gid change since current user is not root"
[12:28:58.496][info    ] Initializing Bedrockifier Daemon
[12:28:58.507][info    ] Configuring Bedrockifier Daemon
[12:28:58.513][info    ] Loading Configuration From: /backups/config.yml
[12:28:58.546][info    ] Configuration Loaded, Running Service...
[12:28:58.560][info    ] Host PTY Handle Opened: /dev/pts/0  (minecraft-server-insel - 6)
[12:28:58.561][info    ] Child PTY Handle Opened: /dev/pts/0 (minecraft-server-insel - 7)
[12:28:58.564][info    ] Checking for servers that might not be cleaned up
[12:28:58.569][info    ] Backup Interval: 10800.0 seconds
[12:28:58.571][info    ] Service Started Successfully.
[12:28:58.574][info    ] Starting Full Backup
[12:29:58.586][error   ] SSH connection doesn't seem to have been made properly.
[12:29:58.588][error   ] Docker process didn't start successfully, or has died
[12:29:58.589][error   ] Container minecraft-server-insel failed to backup properly
[12:29:58.589][warning ] Attempted to stop a terminal process that isn't running. (container: minecraft-server-insel, kind: ssh)
[12:29:58.589][info    ] Performing Trim Jobs
[12:29:58.729][info    ] Full Backup Completed

Last line says Full Backup Completed but it hasn't done one (because of the errors earlier in the log, which I'm working on). I assume the last line should be something like Backup failed.


Export personalised visuals

Are there any plans to make this feature work when you export?
We are currently trying to release a standard report to 200 users which they will then go onto personalise, however if we are unable to do this will we have to go  back to square 1 or worse have to write 200 bespoke reports

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 615021e0-5504-046d-d6ab-c2d8dee0656f
Version Independent ID: d9c6ad5a-139f-716e-eb59-473a1f8d73de
Content: Let users personalize visuals in a report - Power BI
Content Source: powerbi-docs/create-reports/power-bi-personalize-visuals.md
Service: powerbi
Sub-service: pbi-reports-dashboards
GitHub Login: @maggiesMSFT
Microsoft Alias: maggies

Thanks for your comment, @kmarspence . This looks like a support request or product question rather than documentation feedback. You might find a solution in the Power BI Community, at https://community.powerbi.com. If not, you can get support by going to https://support.powerbi.com.


Requirement Analysis.

Hi Chris,
I have initiated scan on multiple filesystems, which is around 500 to 800TB and in future we are planning to use this tool to scan around 7 to 8PB every 3 months. In this case, can you suggest me the environment requirement. I mean to say, what was the hardware requirement for 8PB scan. Current environment details are mentioned below.
Redis Server : Red Hat Enterprise Linux 7.2 (with 128GB RAM, 12 Core * 2 CPUs)
Elasticsearch : Ubuntu 16.04 ( 3 Node Cluster with 32GB RAM, 8 Core CPU and 1TB Storage for elasticsearch data and log on each node)
BOT Count : 600
Thanks & Regards,
RaviTeja

How did your initial scan go?
Based on your configuration and the scan timing/storage/cpu stats you should be able to get a rough idea on how much you'd need to scale up in order to scan 8PB.
There are a lot of variables that only you know about your environment, like speed of the hardware hosting the filesystems you are scanning or the protocol your scanning over (NFS, CIFS, Direct Attached)

I have initiated scan for 450 to 500 TB of data. I can observe that RAM 128GB almost occupied by redis and scan failing with redis memory issues. (it's all 40G pipe network and storage is NetApp and Isilon)

How do you have redis configured? If it's setup to be an LRU cache it seems you'd just configure redis to have the available amount of memory that your system supports, and then it will age out anything that it needs to?
We are just spinning up a cluster to test with, and will a have similar sized dataset so will report back with our findings.
I'll have to look through the code to see how it traverses folders, but you can try turning down the TTL for directory caching in redis to slow the growth. It also might make sense to use a hash of the filepath for the redis key instead of the full string to slow the growth of the redis dataset especially for long filepaths/filenames.
From a quick look If you aren't passing a -I / --index2 argument than you could comment out the redis setting code and avoid the writes to it (or better yet put the same if statement around it that the get code uses:
Set code that doesn't have the if:
https://github.com/shirosaidev/diskover/blob/master/diskover_worker_bot.py#L216
Get code that has the if:
https://github.com/shirosaidev/diskover/blob/master/diskover_worker_bot.py#L112

Hi RaviTeja,
Are you running 600 worker bots all on 1 host, on the rhel server? Is this all on bare metal or vm?
You may see diminishing returns running that many bots using a single diskover.py crawl. Are you running crawls in parallel? You could try running rqinfo -i 2 during crawl to see how idle the bots are. If they are all busy most of the time that's great :)
-I / --index2 cli arg is used to get meta data from an existing index in ES when the mtimes of directories are the same. This is to reduce amount of stat calls to the disk/network. Mtimes for each directory are stored in Redis for a default of 1 week (setting in diskover.cfg).

it's bare metal with 2 * 12 care CPU's and 128GB RAM.
from redis point on you, i have not changed anything after instillation. If we required to configure something from redis side, please share with me.
I am running parallel crawls with each mount separate index name.
Please let me know if you required to change any config to perform well or do the crawler more faster.

I have written an optimization guide, this may help you to get the best performance when crawling
https://github.com/shirosaidev/diskover/wiki/Optimizing

You mentioned the only issue you were having was with Redis memory? Which version of Redis are you running? I've been testing on 4.0.8 and haven't seen any issues yet, but I'm not testing with as big as data set as you are...

Also what is the full command you are using to start the crawl? Are you using -a or -b ?

im using "-a"

Are you still having any issues crawling? It might be helpful to install x-pack to look at ES monitoring in Kibana during crawl..  Also this redis monitoring guide might help if you need to see what's going on with Redis
https://scalegrid.io/blog/6-crucial-redis-monitoring-metrics/

@ravitejamandapati any more issues with redis and crawling in latest release of rc10? Please try with the latest patches to rc10 up on github and let me know.


Add version 3 Fist of the North Star resolution pack

it just useful in the version lower than cemu 1.13x！i like this game
[Definition]
titleIds =<PHONE_NUMBER>116600
name = "beidou - 1080p"
version = 2
[TextureRedefine] # tv
width = 1280
height = 720
overwriteWidth = 1920
overwriteHeight = 1080
[TextureRedefine] # q-res
width = 320
height = 180
overwriteWidth = 480
overwriteHeight = 270
[TextureRedefine] # q-res 2
width = 322
height = 182
overwriteWidth = 483
overwriteHeight = 273
#o-res
[TextureRedefine] # o-res
width = 160
height = 90
overwriteWidth = 240
overwriteHeight = 135
[TextureRedefine] # o-res 2
width = 162
height = 92
overwriteWidth = 243
overwriteHeight = 184
[TextureRedefine] # gamepad
width = 854
height = 480
#overwriteWidth = 1920
#overwriteHeight = 1080
how to change this code to the high version，for example to cemu 1.15.7c。
Would you give me a hand ？

Sadly enough, it's far more complicated then just using the old pack in the new version. I'll look into creating a resolution pack for it today or tomorrow.

可悲的是，它比使用新版本中的旧包装要复杂得多。我会考虑今天或明天为它创建一个解决方案包。

You are so kind,Thanks a lot!

Should be available now! 👍


Unable to read complex data types(e.g. Map, Struct) after upgrading to latest (0.10.0) version

The extension is able to read complex data types using duckdb 0.9.0 -

We could not read complex data types after upgrading to duckdb 0.10.0 -

The database is throwing below error -
Error: IO Error: Invalid field found while parsing field: type

Thanks for reporting this! I think it is a duplicate of https://github.com/duckdb/duckdb_iceberg/issues/39

Yes, i ran into #39 after adding a temp workaround for this one.

@devendrasr Can you provide more details on how you overcame the issue? How did you disable schema parsing? Thanks

@harel-e  i made few changes like this - https://github.com/devendrasr/duckdb_iceberg/commit/d9e5d720b319e4e474a39373d8d43781a0a3b028

@devendrasr will PR#43 that you created solve this issue? Thank you

It just allows you to skip schema inference and fallback to parquet extension to infer schema. In this way we are able to read complex data types via iceberg extension.

@devendrasr - I'd like to test your changes locally, especially with issue #39.  Is it possible for you to provide a binary since I'm having issues building DuckDB locally? thanks.

@harel-e  You can try them out using -
docker run --rm --name=duckdb -it devendrasr/duckdb:0.0.1 bash

@devendrasr - thank you for providing an image for testing.
I tried the scenario described here: https://github.com/duckdb/duckdb_iceberg/issues/39#issuecomment-1943244889
Unfortunately, it still fails with:
D SELECT * FROM iceberg_scan('data/iceberg/lineitem_iceberg', allow_moved_paths = true);
Error: IO Error: Invalid field found while parsing field: required
I understand you were not trying to address issue #39. I thought they were somewhat related.
Thanks again.

My bad, I forgot to add skip_schema_inference = true.
I tested it again with both lineitem data and my own dataset (minio/nessie) and it works !
Thank you very much !
I'll wait for the PR to be merged.

The issue has been resolved in pull request - https://github.com/duckdb/duckdb_iceberg/pull/45
The request is now merged in trunk. Closing this one.


fix output directory

change output/debug to output/release when compiling in release mode

bump

ok but who cares like fr


get_extended_metadata

this seems to be a stub currently:
https://github.com/librespot-org/librespot/blob/4d35c5ffe222c839d1532d8afac78fe4c37043b3/core/src/spclient.rs#L703-L707

Yes, what is your point?
I believe I put it there just to cover all "known" endpoints.

What do you think my point is?

It's really hard to guess what people want. Please can you redefine this issue so it's easier to solve for the people giving their time to help you.

It's really hard to guess what people want.

is it really? it seems pretty obvious what the ask is here


Add IntervalChange, LocalDateChange, LocalTimeChange.

Current coverage is 80.86% (diff: 82.35%)

Merging #223 into master will increase coverage by 0.12%

@@             master       #223   diff @@
==========================================
  Files           147        148     +1
  Lines          5540       5618    +78
  Methods           0          0
  Messages          0          0
  Branches        627        653    +26
==========================================
+ Hits           4473       4543    +70
- Misses          952        953     +1
- Partials        115        122     +7

Powered by Codecov. Last update 57b6307...e18aa3e

LGTM. Please merge.


Home Assistant Discovery can't use SSL?

It appears that the Home Assistant device discovery is http:// only and SSL fails to work?

Looks like I've hit this issue too.  The code does look to have http hardcoded.  Could you add optional support for https?

Just curious, are these home assistant instances remote? If not, why not use the local ip address without SSL, unless there is a reason you would need SSL on a local network.

If you've set up home assistant directly with an SSL certificate, it only responds via SSL AFAIK.  There's no way to access it via IP directly.
FWIW, I ended up just setting up a reverse proxy with SSL to local subdomains and getting rid of Home Assistant's built in use of certificates.  I'd run into a few other headaches like this which seemed to necessitate this.
But if you're trying to use the tools Home Assistant provides, it will be a stumbling block, so it still seems like it may be a worthwhile feature for a lot of folks.
I look forward to using HA-Bridge now that I can get it talking directly to my HAss setup.  Appreciate the fast and good work on it!

Same issue as previously posted. I'm using my own wildcard SSL cert and I'm unable to access any Home Assistant entities with SSL enabled via Home Assistant config.

After upgrading to 4.1.0 and importimg the cert chain for my cert into the java keystone, I'm able to discover my home assistant devices.  Thanks @bwssytems.
sudo keytool -trustcacerts -keystore /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/security/cacerts -storepass changeit -noprompt -importcert -file /etc/letsencrypt/live/blah.com/chain.pem

I am unclear as to whether this is meant to be fixed or not, however I am having the same issue.  I enable SSL on Home Assistant using a legitimate SSL certificate.  This certificate works without issue for any browser, the AWS skills, even iPhones !!!  When I enable an SSL HA Instance in HA Bridge, it fails with the following error(s).  I've tried a few workarounds (temporarily disabling SSL, setting up an HASS device in HA Bridge, discovering - works fine - and setting up a "switch" - also works fine - then leaving the switch in situ, deleting and recreating the HASS device (back with SSL) in the bridge config, restart etc.  Still fails, though I think it's trying to use SSL as the messages below show.  I am happy to be a tester of this
For what it's worth (and apologies, I'm no expert on any of this), I think the SSL call is going out okay but it's something to do with not being able to validate the authenticity of the certificate.  As it's a legitimate (not self certified) version, this is something I wouldn't have expected to happen.
2017-02-25 07:12:03,596 [qtp19327982-17] WARN  com.bwssystems.HABridge.plugins.http.HTTPHandler - Error calling out to HA gateway: IOException in log
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
at sun.security.ssl.Handshaker.processLoop(Unknown Source)
at sun.security.ssl.Handshaker.process_record(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at com.bwssystems.HABridge.plugins.http.HTTPHandler.doHttpRequest(HTTPHandler.java:109)
at com.bwssystems.HABridge.plugins.hass.HomeAssistant.callCommand(HomeAssistant.java:65)
at com.bwssystems.HABridge.plugins.hass.HassHome.deviceHandler(HassHome.java:144)
at com.bwssystems.HABridge.hue.HueMulator.changeState(HueMulator.java:979)
at com.bwssystems.HABridge.hue.HueMulator.lambda$35(HueMulator.java:392)
at spark.RouteImpl$1.handle(RouteImpl.java:58)
at spark.webserver.MatcherFilter.doFilter(MatcherFilter.java:162)
at spark.webserver.JettyHandler.doHandle(JettyHandler.java:61)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:302)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:245)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Unknown Source)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(Unknown Source)
at sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
at sun.security.validator.Validator.validate(Unknown Source)
at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
... 44 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
at java.security.cert.CertPathBuilder.build(Unknown Source)
... 50 common frames omitted


Login - Blank space in magic code page & in login page on going back

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.45-0
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause internal team
Slack conversation:
Action Performed:

Navigate to staging.new.expensify.com
Enter new account & tap on continue
Observe blank space in magic code page
Click go back link
Observe blank space in login page

Expected Result:
No visual issues appear in log in flow. Blank space should not be displayed in magic code & login page
Actual Result:
Blank space in magic code page & in login page on going back
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[x] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/38435837/98ce9b02-f230-4199-87fc-6eeab1b61054
Add any screenshot/video evidence

View all open jobs on GitHub

@jliexpensify  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors.

@kadiealexander and I have an Android/iOS swap going on, ao assigning her this one!


Apply Task should be able to use "--force"

What would you like to be added:
A boolean "force" flag on the apply-task
Why is this needed:
Certain updates to resources are not allowed by k8s. The kubectl apply provides a --force flag to still deploy those changes. In the background it schedules a delete/create for this instead of applying a patch.
We could have the same flag on an "Apply" Task to allow disruptive updates to resources. At the moment the only way to update those attributes is to manually write an "update" plan that deletes and recreates the resources.

I'm not sure that adding a force flag is a good idea. My concern is that an operator developer might not be aware that this is in fact causing a delete+re-create - which has potentially undesired consequences such as cascading deletion.
Perhaps if we name the flag such that it's explicit... :thinking:

Another option would be to have the --force flag on the k kudo update level, and not in the operator itself. This would require the user to explicitly use it.

When the instance controller starts the plan execution, it clears this flag

Which flag? This is the first time a flag is being mentioned in this description.
Do you mean the field in the spec:

a new field on the InstanceSpec

?
In that case I think it would be quite surprising and contrary to established conventions, if a controller was to mutate the spec of a resource it manages.

Yes, I meant the field.
And changing the spec... It's what's currently happens with the PlanExecution field in the InstanceSpec as well if I read that correct.
I do agree that it's a bit unconventional. Any other ideas to pass a one-off flag for a parameter update to the controller would be very welcome


JavaScript heap out of memory with v3.6.2

Environment

Operating System: Linux
Node Version:     v18.15.0
Nuxt Version:     3.6.2
Nitro Version:    2.5.2
Package Manager<EMAIL_ADDRESS>Builder:          vite
User Config:      app, alias, content, nitro, modules, components, tailwindcss, pinceau, github, colorMode, runtimeConfig
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>pinceau/nuxt<EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
On the latest version of Nuxt have a large /content file (about 200 .md files), run yarn generate
Describe the bug
I have quite a few .md files using nuxt-content, yarn generate usually took a few minutes, but after upgrading to the latest version of nuxt after a while it throws a JavaScript heap out of memory error. Why is nuxt now consuming so much memory, how to check the memory consumption and can it be optimized somehow
...
  ├─ /api/_content/query/wjRchTaiX1.1688647063500.json (22634ms)                                                                              nitro 2:40:03 PM

<--- Last few GCs --->

[630312:0x55d628acb1c0]   173933 ms: Scavenge 3999.5 (4112.0) -> 3996.2 (4121.8) MB, 18.1 / 0.0 ms  (average mu = 0.219, current mu = 0.093) allocation failure;
[630312:0x55d628acb1c0]   174982 ms: Mark-sweep 4015.2 (4128.0) -> 3984.6 (4117.6) MB, 959.9 / 0.7 ms  (average mu = 0.215, current mu = 0.211) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
...

Additional context
No response
Logs
No response

@gitFoxCode You can disable nitro.prerender.failOnError, or add the failing paths to nitro.prerender.ignore, which is safer.

We could add a pre-render section to the docs, either in nuxt or in nitro. What do you think @danielroe ?

I think that would be great! There's enough here that is unique about Nuxt that I think it makes sense to have in our docs, and we can link out to Nitro where it makes sense

same issue!!!
{"log":"FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory\n","stream":"stderr","time":"2023-07-06T21:55:07.07458664Z"}
The solution is in nuxt.config.ts add nitro.prerender.concurrency=1?????

nothing anomalous prior to the error, there just comes a time when it fails

@agracia-foticos Yes,
[nuxt.config.ts]
nitro: {
    prerender: {
        concurrency: 1
    }
}

It gets the job done


Remove mention of custom BOM for trying out canary

Page: /setup/canary/
Feedback:
Now that we've mainline released kayenta in 1.7.x, no need to direct users to custom BOM.

Good catch.
@duftler , is it sufficient to remove this entirely, or do we want to specify a minimum version?

We should probably specify the minimum version of Spinnaker, just like we do for Halyard itself just above. Kayenta is available in v1.7.0 or higher.


Release/v1.0.0

Reached stable state :raised_hands:

Moved node-sass^3.0.0-alpha.0 to peerDependencies #28
Using webpack's module resolver as custom importer #39
Add synchronous compilation support for usage with enhanced-require #39

I've introduced a changelog. From now on, we will also tag every release.

@jhnns Wow! Awesome job!
Thanks @jhnns and @jtangelder for maintaining this loader so well!

This is pretty great news!


Where's the api key and is the backend opensource?

Unresolved reference: APIKEY
Please provide api key or opensource the backend and provide the compilation steps if possible.

Backend code is there in https://github.com/Vaibhav2002/Spring-Boot-Notes-REST-API, you need to connect backend to your database.
The Api key is custom , u can create any key and just paste it in the app code and the backend code in the correct files. See the code for app and backend , you will understand where u need to paste  the your own API key


Replace own vector search with Chromem-go

Because my implementation is apparently not producing the expected
results and this project fits the bill quite nicely.
Most of the tests are failing, but the core functionality appears to
work.

[x] Set
[ ] Get
[ ] Delete (we have to delete the whole collection presently)
[ ] Query (it returns normalized embeddings, not the originals)
[x] Query Normalized

Some features are missing from Chromem, but we don't strictly need
them. Meanwhile there are maybe some minor things that need adding on
our side to get the basic functionality working.

re: deletion, we might want to implement a .Reset() to discard and cleanup the memory?

Yes, I think so. Also I think we need to add IDs to the API (so the vector is not being used for that).
part of #1792


Minimalize the Git & Dictionary Plugin

Added support for the new Git View and Dictionary Plugin

Dictionary:

Git:

The screenshots look great! Thank you. I will test this out shortly.


[TF-TRT] Add DataFormatVecPermute converter

Adds a TF-TRT converter for the DataFormatVecPermute operation.
Example of a model which can benefit from this: Inception v4 with dynamic shapes (2 candidate segments previously, 1 with this change).

@bixia1 I have squashed the commits.

@Nyrio Can you check my comment and fix the #if line?


fix: proxy calls to aws fail with newer AWS SDK version

Issue:  #41716

Problem
When upgrading the AWS SDK from version 1.44.253 to 1.44.266 this caused the proxy calls to AWS to fail with SignatureDoesNotMatch error. This was caused as a new partition was added (aws-iso-e) which currently only has 1 service listed and does not contain IAM.
The logic loops around all the partitions trying to get the service and region. As the new partition is last then the value for service is overwritten with a blank string and the signing fails as the service isn't included in the signature.
AWS add partitions occasionally and these new partitions gradually introduce services into the regions of the partition.
Solution
This change checks if a partition returns a global service. If it does, then it stops processing partitions and breaks out so the existing logic then takes over for global services.

If we felt this was too risky, then downgrading the AWS SDK would also fix the problem short term. Long term we  don't want to be in a position where we can't upgrade the AWS SDK.

Testing
The repo steps are valid
Engineering Testing
Manual Testing
Tested manually.
Automated Testing

QA Testing Considerations
The fix is in a area that all communication with AWS occurs. So we should test a variety of cluster creation and operations for AWS clusters (EKS and non-EKS).

Added rancher/rancher-team-2-hostbusters-dev to reviewers to review as well since this code is used for EC2 node provisioned clusters. Pinging @kinarashah @jakefhyde more specifically, but anyone else from the team who understands this code is welcome to approve too.


Suggestion : Skipping broken images option

Having a single image that can't be downloaded stops the whole program and getting a newer archive didn't seem to fix the link.

I have always wondered if that ever happens.
Could you share the URL of the image, or maybe even the part of the data file that surrounds it (if you know how to find it – I can help)? Would you have any idea about why it’s missing?

URL of the image is https://pbs.twimg.com/media/Ck4fwiaXIAAUqs_.jpg:orig
I could give part of the data file, just needs to know what it is and where to find it. I'm on windows if that helps.

Thanks! Try to go into data directory and then js directory in your archive.
Then search for Ck4fwiaXIAAUqs in all the files with .js extension.
You should find it and see a structure wrapped in {}, starting with "source". If you could just copy/paste this whole thing, that’d be great.

Added a little more as it appeared 4 times in the archive, hopefully that's enough https://pastebin.com/zcemSSSE
The tweet itself doesn't have the first image either

Thanks for this. Weird. So you posted with two images, but only the second one survived?
Any ideas why the first one was gone? Censorship?

This is a retweet, and as there is nothing political or offensive in game sprites I don't see how it could be censorship.

Interesting. Thanks! I’ll add an option to skip and update this issue later today.

Thank you!

Okay, this should be done now. Try --continue-after-failure. Would appreciate checking it out and seeing if it outputs the right thing at the end!

Everything worked fine, The archive doesn't bring up any error and the script properly linked to the tweet causing issues.

Thanks for checking and filing the bug in the first place!