indexer-agent: Agent deploys subgraphs multiple times

When I add multiple offchain rules to indexer-agent, deployments are created multiple times and might be assigned to multiple nodes which leads to subgraphs failing. I'm running indexer-agent v0.20.3 with graph-node v0.27.0
Steps to reproduce:

Run indexer-agent with multiple nodes configured (INDEXER_AGENT_INDEX_NODE_IDS=index_node_0,index_node_1,index_node_2,index_node_3)
Add multiple offchain rules without delay

graph indexer rules offchain QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee
graph indexer rules offchain QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA
graph indexer rules offchain QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7
graph indexer rules offchain QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW
graph indexer rules offchain QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs
...

Result:

indexer-agent tries to deploy the subgraph multiple times
Different index nodes start syncing the same deployment (in my example, subgraph got reassigned 5 times to 3 different nodes)
Indexing fails with there are most likely two (or more) nodes indexing this subgraph error

Logs:
# docker logs graph_indexer_agent -t | grep QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee | grep -A10 'Deployment changes' | grep -v 'Reconcile deployments'
2022-10-06T19:09:21.061374954Z {"level":30,"time":1665083361061,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xd05877e0acdcc6e321787d584f0ffa2b695d1af57ef6f12ef5c4db68f816ed54","ipfsHash":"QmcMwkbXhthjpPn91nWNPaLZC9Vz5vCRwKRMTmoV7kqy3H"},{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:21.063686345Z {"level":30,"time":1665083361063,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:21.179530392Z {"level":30,"time":1665083361179,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:21.181528197Z {"level":30,"time":1665083361181,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:21.318661553Z {"level":30,"time":1665083361318,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:21.394692627Z {"level":30,"time":1665083361394,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:22.954917883Z {"level":30,"time":1665083362954,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:22.954921833Z {"level":30,"time":1665083362954,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_0","msg":"Reassign subgraph deployment"}
2022-10-06T19:09:23.155471475Z {"level":30,"time":1665083363155,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:23.155475504Z {"level":30,"time":1665083363155,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_1","msg":"Reassign subgraph deployment"}
2022-10-06T19:09:40.876018564Z {"level":30,"time":1665083380875,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:40.877586602Z {"level":30,"time":1665083380877,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:41.026132078Z {"level":30,"time":1665083381025,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xc7c272666799ff602fd8fae8ab986f1b052f8873891ef29364aca09286ab08fa","ipfsHash":"QmbnRp17QCAKsmS7js8n99F2UZTQkXWQ6ePi6ekdL5f45P"},{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:41.027515047Z {"level":30,"time":1665083381027,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:41.154043888Z {"level":30,"time":1665083381153,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:41.171100073Z {"level":30,"time":1665083381170,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:42.501661296Z {"level":30,"time":1665083382501,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:42.501666386Z {"level":30,"time":1665083382501,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_2","msg":"Reassign subgraph deployment"}
2022-10-06T19:09:42.518375413Z {"level":30,"time":1665083382518,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:42.518378643Z {"level":30,"time":1665083382518,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_2","msg":"Reassign subgraph deployment"}
2022-10-06T19:10:20.027238150Z {"level":30,"time":1665083420027,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xc7c272666799ff602fd8fae8ab986f1b052f8873891ef29364aca09286ab08fa","ipfsHash":"QmbnRp17QCAKsmS7js8n99F2UZTQkXWQ6ePi6ekdL5f45P"},{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:10:20.029254924Z {"level":30,"time":1665083420029,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:10:20.178769502Z {"level":30,"time":1665083420178,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:10:21.040350122Z {"level":30,"time":1665083421040,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:10:21.040472301Z {"level":30,"time":1665083421040,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_2","msg":"Reassign subgraph deployment"}

In the logs I've also found that every reconcile there is an error with one of the subgraphs, so agent tries to retry an entire queue which leads to problems.
Full logs: https://pastebin.com/QK9C3TEH
2022-10-06T19:09:21.350344021Z {"level":50,"time":1665083361350,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/p5hqPvjYeA","deployment":{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},"err":{"type":"IndexerError","message":"Failed to deploy subgraph deployment","stack":"IndexerError: Failed to deploy subgraph deployment\n    at indexerError (/opt/indexer/packages/indexer-common/dist/errors.js:161:12)\n    at Indexer.<anonymous> (/opt/indexer/packages/indexer-agent/dist/indexer.js:694:63)\n    at Generator.next (<anonymous>)\n    at fulfilled (/opt/indexer/packages/indexer-agent/dist/indexer.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","code":"IE026","explanation":"https://github.com/graphprotocol/indexer/blob/master/docs/errors.md#ie026","cause":{"code":0,"message":"subgraph validation error: [the graft base is invalid: deployment not found: QmQCH1zo9JPFhqEJSHUziWu3TtDFHWWAB4bNdBwhm2ua14]"}},"msg":"Failed to deploy subgraph deployment"}

Hey @cause1ove, thanks for the report.  We have a few related updates in the works that should resolve the issues you experienced.

For the situation where the agent repeatedly adds actions to the queue that can be expected to fail (have failed recently).
Additionally we are working on an update to allow the graph-nodes to decide which index node a deployment is assigned to, so the agent will no longer try to specify or reassign deployments.

@fordN just to clarify: in my case the subgraph was deploying successfully every time but was redeployed at every reconcile. As far as I understood it was because the other subgraph was failing to deploy. Will these fixes cover these cases?

Resolved by #868


Tsm

Can you please refer to the changed code(what was not copy-paste)?

@akpsgit
What was changed:
cmd folder removed
Added github folder for workflows
In grpc proto: remove ComponentID enum, instead using string
in initHostToTrace: Dont return error if secret not found (dont force to use secret)


How to delete a Token from the server-side (mutation). when a user gives a logout the token will be destroyed from server-side by using mutation? is it possible? If so. please explain.

JWT is stateless authentication approach, ie server-side does not store any info about token, so nothing to remove. Either you need another authentication appoach or you need to add a DB and respecive code to store info about tokens (in order to have something to delete).

Okay. I got a better approach without  creating the DB for jwt. https://github.com/flavors/django-graphql-jwt/issues/11#issuecomment-616806837


voxelization

Hi, could you post the voxelization code? Or tell more about the used algorithm. I tried some tools but they generate a volume model with artifacts like this:

Nearly there it seems. :) Check Figure 2 in the paper - we rotate to frontal first using the 3DDFA pose parameters and then rotate the entire 3D volume back so that spatial correspondence is preserved. If you can fill holes in advance, such as the ones on the nose and mouth, you can probably use something like this, but it will be slower than filling from the front. I am sorry but we have decided not to release any of the code required for training just yet.

Hi, I just have one question on the coordinate frame that the voxelisation is performed, do you perform it on the original image frame (arbitrary width and height) or do you perform the voxelisation in the coordinate frame of 192x192 image (which means that you scale and translate the mesh)? Thanks!

All faces are normalised using a landmark detector to fit into the 192x192 space. The same transformation is always applied to the mesh, hence, the voxelisation is done in the image space.

I don't remember sorry. I think I used black backgrounds during training, and the MATLAB version of this code should also be using black backgrounds. The python version might be doing something else. :man_shrugging:

Cool! Thx for the clarification! I am just having another question is that would you mind providing a rough estimate on how long it would take to train the model from scratch using 300W-LP? Thanks.

Training is usually a couple of days on a pair of Titan X or 1080 TIs. The part that takes the longest is voxelising your dataset.

Cool! Thanks for the clarification! So you are using multi-GPU training?


Dropdown is not clickable when debug set to false

The dropdown can't be opened when debug is set to false.
When the web.config file has the tag:
<compilation debug="false" targetFrame...

the dropdown menu will not be opened when the user click on it.. however it work fine when the debug tag set to true:
<compilation debug="true" targetFrame...

Any Idea ?

Does this work with a regular Bootstrap dropdown?

Will it appear i need to call
$('.dropdown-toggle').dropdown();

in javascript to solve this.


Add knob to enable system pod metrics

This is not yet used anywhere so this change is no-op. In the next
steps, this override will be added to test-infra tests as specified in
clusterloader2/docs/experiments.md.

/ok-to-test
/lgtm
/approve

/lgtm

/approve


Proposal: check for uninstall stanza in brew cask audit

Note the suggestion here by @vitorgalvao https://github.com/caskroom/homebrew-cask/pull/17342#issuecomment-172948903 "All of these are missing uninstall. There are scripts and documentation geared for that."

Already covered in https://github.com/caskroom/homebrew-cask/issues/7683.
Not every cask requires uninstall.

@joshka To clarify, casks which contain a pkg or installer stanza must have an accompanying uninstall.

Gotcha. Makes sense that you guys have thought of it already.


waybar tip is broken for me recently

I don't think I changed anything here related to waybar config, just updates to tip as usual.
Suddenly I can't start waybar:
Jan 30 20:56:15 zeph waybar[3264]: [2024-01-30 20:56:15.572] [info] Using configuration file /home/cole/.config/waybar/config
Jan 30 20:56:15 zeph waybar[3264]: [2024-01-30 20:56:15.574] [info] Unable to receive desktop appearance: GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: No
such interface “org.freedesktop.portal.Settings” on object at path /org/freedesktop/portal/desktop
Jan 30 20:56:15 zeph waybar[3264]: [2024-01-30 20:56:15.574] [info] Using CSS file /home/cole/.config/waybar/style.css
Jan 30 20:56:15 zeph waybar[3264]: [2024-01-30 20:56:15.574] [error] in Json::Value::resolveReference(key, end): requires objectValue
Jan 30 20:56:15 zeph systemd[2283]: waybar.service: Main process exited, code=exited, status=1/FAILURE
Jan 30 20:56:15 zeph systemd[2283]: waybar.service: Failed with result 'exit-code'.
Jan 30 20:56:15 zeph systemd[2283]: waybar.service: Scheduled restart job, restart counter is at 5.
Jan 30 20:56:15 zeph systemd[2283]: waybar.service: Start request repeated too quickly.
Jan 30 20:56:15 zeph systemd[2283]: waybar.service: Failed with result 'exit-code'.
Jan 30 20:56:15 zeph systemd[2283]: Failed to start Highly customizable Wayland bar for Sway and Wlroots based compositors..

jq and jless don't complain about my config:
[
  {
    "backlight": {
      "format": "nit {percent}",
      "on-scroll-down": "/nix/store/c0xk815ibsbz09q9cf0kdnaa6cayk9m5-brightnessctl-0.5.1/bin/brightnessctl set 2%-",
      "on-scroll-up": "/nix/store/c0xk815ibsbz09q9cf0kdnaa6cayk9m5-brightnessctl-0.5.1/bin/brightnessctl set 2%+"
    },
    "battery": {
      "format": "bat {}",
      "states": {
        "critical": 15,
        "warning": 25
      }
    },
    "clock": {
      "format": "{:%b-%d %H:%M}"
    },
    "cpu": {
      "format": "cpu {usage}",
      "interval": 2
    },
    "custom/ppp": {
      "exec": "/nix/store/x97klsl0km7nfr18080sgr2fv9n6h1zb-waybar-ppp.sh",
      "interval": 10,
      "return-type": "json"
    },
    "custom/rog": {
      "exec": "/nix/store/spdfmzf631ca547k1mq77vxqhlyl2cls-waybar-rog.sh",
      "interval": 10,
      "return-type": "json"
    },
    "idle_inhibitor": {
      "format": "iil {icon}",
      "format-icons": {
        "activated": "[x]",
        "deactivated": "[ ]"
      }
    },
    "layer": "top",
    "memory": {
      "format": "mem {}"
    },
    "modules-center": [
      "wlr/workspaces"
    ],
    "modules-left": [
      "sway/mode",
      "sway/workspaces"
    ],
    "modules-right": [
      "pulseaudio",
      "backlight",
      "tray",
      "custom/ppp",
      "custom/rog",
      "cpu",
      "memory",
      "network",
      "temperature",
      "battery",
      "clock"
    ],
    "network": {
      "format-ethernet": "eth",
      "format-wifi": "net {signalStrength}"
    },
    "pulseaudio": {
      "format": "vol {volume}",
      "on-click-middle": "/nix/store/h4bkx908ihvfaszgi3s3kwsh4wd23p5z-sway-+c8676fa/bin/swaymsg exec \"/nix/store/inj6ly5g2siq1632lh3mq9xszw8yvgjw-pavucontrol-5.0/bin/pavucontrol\""
    },
    "sway/mode": {
      "tooltip": false
    },
    "sway/workspaces": {
      "all-outputs": true,
      "disable-scroll-wraparound": true
    },
    "temperature": {
      "format": "tmp {temperatureC}"
    },
    "tray": {
      "spacing": 10
    },
    "wlr/taskbar": {}
  }
]

i too have been experiencing this, it seems too be related too the outer braces
removing [] should fix the issue, but this means 2 bars in one config is impossible which im unsure is intended

i too have been experiencing this, it seems too be related too the outer braces removing [] should fix the issue, but this means 2 bars in one config is impossible which im unsure is intended

I confirm, same bug, same workaround, but this still looks like an issue/regression 👍🏻

Same here since this commit, configuration of multiple outputs fails

Same error here.

Actually the faulty commit is https://github.com/Alexays/Waybar/commit/20fa578b12212591299d2010f1b6f8d5c117dcde which introduces the following code:
if (config.getConfig()["reload_style_on_change"].asBool()) {
    m_cssReloadHelper->monitorChanges();
}

The ["reload_style_on_change"]  throws since it expects the result of getConfig() to be either a JSON null or a JSON object, but with multiple outputs, it's an array.
The ideal solution would be for the config format to be changed so that there is a possibility for global config instead of just per-bar config, i.e.
{
    "reload_style_on_change": false,
    "bars": [
        { ... first bar ...  },
        { ... second bar ... }
    ]
}

This would also help pave the way for defining modules globally so they can be shared between bars.

Should be fixed with https://github.com/Alexays/Waybar/pull/2874


Verify RHODS version before and after upgrade

Refactor upgrade test to check if version after upgrade is greater than the initial version. The previous approach that checked if the pods in the operator namespace were Pending after 10s was causing the test to fail, because the pods were already running before 10s, and playing with that sleep timer might make the test flaky. I think the right way to confirm if the upgrade has been successful is to check the CSV version and that the pods in the operator namespace are running once the CSV version has been updated
rh-pre-commit.version: 2.2.0
rh-pre-commit.check-secrets: ENABLED

I think the right way to confirm if the upgrade has been successful is to check the CSV version and that the pods in the operator namespace are running once the CSV version has been updated

if something is wrong with upgrade, the pods may remain up and running. So, even if the CSV is updated, the upgrade  process may still be blocked and components not upgraded.
I think one possible option we have is to check that each deployment under redhat-ods-* namespaces got rollout, in other words the pods creation date should be more recent than the time upgrade was initiated.


Check that there's an href to .charAt()

The 'fix' in #2932 missed the existence of the JS controlled segmented controls. This reverts the change, but adds a check for the presence of an href to do the .charAt() test on, and returns early if it isn't there.

✨


Corrupted world on Windows 11

Describe the problem
After server restart, the server world files are corrupted. Everything works well while running pure Paper server without docker and also running same container on macOS or Linux. Recreating container doesn't help anything, after server restart I'm getting the same errors.
Container definition
version: "3.8"

services:
  mc:
    image: itzg/minecraft-server:java16
    ports:
      - "25565:25565"
    environment:
      TYPE: PAPER
      VERSION: 1.17.1
      EULA: "TRUE"
      ONLINE_MODE: "FALSE"
      COPY_CONFIG_DEST: /data
    volumes:
      - ./data/mc:/data
      - ./mc:/config

Container logs
Full server log is available here https://teksusik.pl/docker.log

I'm sorry to hear that happened. I'm afraid that is beyond the control of this image/container since ultimately it exec's to the requested server process. The logs provided do not show any of the container startup activity. Without those logs prior to the restart I don't know how you attempted to shutdown the server. Other external factors include the state of the attached ./data/mc host directory and any content augmented in from the ./mc host directory.
I would recommend discussing with the PaperMC team since it attempted but failed to recover the corrupted regionfile.

I'am restarting my server via stopping container from IntelliJ services tab, so the server is saving world, also there aren't any errors at this phase. I also probably know what's wrong. When I remove volume bindings from my docker-compose file and recreate container everything starts to work well.

Can this be closed now?

Well, I don't think this should work like this, but I don't know it's my fault or not, someone must try to reproduce it

I'm sorry I misunderstood your comment.
In any case, this is not a problem with the image. It's either something external or the PaperMC software itself. We can leave this issue open for others to find, but there's nothing I can do to fix it.

and just to make sure it wasn't a combination of those volume paths I used your compose file and tried a couple ways of restarting it in IntelliJ's service tab:
https://user-images.githubusercontent.com/988985/139678944-cfde1ee5-5832-4773-b37b-7a56e06ae8d5.mp4

Okay, so thanks for help. I will have to check what is wrong in my case.

Same issue but only for minecraft version 1.18.1
docker-file
`version: "3"
services:
mc:
image: itzg/minecraft-server:latest
ports:
- 25565:25565
environment:
EULA: "TRUE"
VERSION: 1.18.1
TYPE: "BUKKIT"
MEMORY: "10G"
tty: true
stdin_open: true
restart: unless-stopped
volumes:
- ./server:/data
attach a directory relative to`

@Aniello12 or @TeksuSiK are either of you running on Windows 11? I was chatting with someone yesterday with similar file corruption type issues and narrowed down that it was Windows 11 as the primary difference.

Yes, I'm working on Windows 11

I got the same issue here in Windows 11 but even with the default version (not paper). Whenever I chose a volume path for the container it will result in corrupted chunks.
As I haven't tested this on Windows 10 I can't say if this is an issue with W11.
How to "fix" it:
When not using the volume parameter (default is docker generates a anonymous docker volume for the container) or using a named docker volume it will work and I can still access data with the windows explorer at
\wsl.localhost\docker-desktop-data\version-pack-data\community\docker\volumes
without the chunk generation issues.

Thanks @MarkusKluge, your finding about the anonymous and named volume confirms another pattern I was starting to see. Also, thanks for the advice on how to access docker desktop file system. I never realized it was that easy even though I knew how to access regular distro file system.

To help others see it, I'm going to re-open this issue but change the title to mention Windows 11. I'll plan on submitting an issue with docker desktop; however, I would need those of you with Windows 11 to possibly help guide troubleshooting from there.

After some trying on #1341 the approach of @MarkusKluge about using named volumes didn't work for me. I make it working by disabling the wsl backend and using hyper-v backend, so to me this is the only working work-around.
@itzg maybe the problem is not docker but wsl itself.

@AlbertoPimpo I'm really surprised naked volumes didn't make difference.
Speaking of which, I thought I'd share this article I just saw that explains why host binding acts different than names volumes (in apparently not all cases)
https://www.docker.com/blog/file-sharing-with-docker-desktop/
Yes, it probably is a problem with WSL on Windows 11. I'm wondering if anybody has yet contacted Docker or Microsoft about this issue?

I am experiencing this also with our server manager.

Docker Desktop on WSL2 On Windows 11, completely shreds the world on shutdown or restart :(
On bare metal linux with Docker, it operates just fine :S

It's a peculiar one for sure, I wonder also if anyone has reached out to Docker or Microsoft.

I have this same problem on Ubuntu server. Like 30% of the chunks are misplaced. Only in 1.18.X, using any java 17 image.
The error happens on all the server types, including vanilla.
I took the server to Windows 11 and only worked using named volumes.
Using named volumes could patch this issue. (Tested on W11)

aight work around for now on windows if you 100% must have access to config files

in your mounts, put it in this order

":/data/world"
"./local/path/:/data"

start container
access container shell as root
as root run chown minecraft:minecraft /data -R
restart container

version: "3"

volumes:
  lobby-world:

services:
  mc:
    image: itzg/minecraft-server:java17
    container_name: mc
    user: "1000:1000"
    stdin_open: true
    tty: true
    ports:
     - "25565:25565/tcp"
     - "25565:25565/udp" # for things that use UDP
    volumes:
      - "mc-world:/data/world"
      - "./mc:/data"
    environment:
      SYNC_SKIP_NEWER_IN_DESTINATION: "false"
      COPY_CONFIG_DEST: "/data"
      PUID: "1000"
      PGID: "1000"
      TYPE: "fabric"
      VERSION: "1.18.1"
      PACKWIZ_URL: "https://gitlab.com/Merith-TK/vanillia-modpack/-/raw/server/server/pack.toml"
      EULA: "true"
      MEMORY: "3G"

I disable the 'Use the WSL 2 based engine' option in Docker Desktop settings to use Hyper-V as backend, and it just works fine.
I think it's a bug of WSL 2.

has anyone found a solution to this project that doesnt require disabling WSL? This may be related? https://github.com/microsoft/WSL/issues/7977

@quiquelhappy there's workaround mentioned in the comments above

@quiquelhappy there's workarounds mentioned in the comments above

Thanks, somehow I completely ignored one of the answers 🤔.
Hoping this gets fixed soon, seems like a WSL bug. Thanks for replying and for keeping up the repo! Have a nice day.

Luckily my PC doesn't have compatible hardware to run Windows 11 😉
Agreed, I'm surprised they haven't resolved this issue by now since it seems like quite a fundamental issue.

@itzg not sure if this is still relevant, but I found that if I interacted with the docker commandline directly from my Ubuntu WSL2 machine I did not get any corrupted data. If I stopped a container using the windows command line I suddenly got issues.

I'm not sure if it is (I'm avoiding upgrading to Windows 11 because of things like this 😀), but thanks for the info.

I think I got it. It's because the volumes are stored in the windows filesystem rather than the linux filesystem. When using a WSL2 backend with files stored in the linux filesystem there are no issues. Ps. windows 11 is awesome!

Thanks @langestefan for confirmation on both parts 😀

I think I got it. It's because the volumes are stored in the windows filesystem rather than the linux filesystem. When using a WSL2 backend with files stored in the linux filesystem there are no issues. Ps. windows 11 is awesome!

Pretty sure this was the case. For some reason, every time I installed Docker Desktop on my Windows computer, with a single WSL OS installed, it never automatically enabled Docker support for such OS. My recommendation to anyone having this issue is to use Docker from a WSL OS directly.
I'll double check later if this works fine with the /mnt/c/ path later and update this comment. If everything works correctly, I'll delete this paragraph and keep the following part:
If you are new to Docker on Windows 11:

install Docker Desktop
if you get a prompt saying WSL isn't enabled on your system, you can enable it on the most recent versions of Windows 11 by looking for the 'WSL 2' app on the Microsoft Store; if that doesn't work, follow the guide linked by the prompt on the Docker Desktop installation
install any distro of your liking, for example Debian or Ubuntu; you can find these on the Microsoft Store
after installing and configuring your distro (simply search for the distro name and open it as you would do with any other application), go to Docker Desktop > Settings > Resources > WSL Integration, and enable the integration for the distro you just installed
when running commands, you can open your distro, or open the newly released Terminal app on Windows 11, making sure you select a docker-enabled distro when opening a tab.
you can find your files by going to /mnt/c/. If you want to be able to access the files from windows directly, make sure to use a volume pointing to a folder within /mnt/c/

Small note if you are transitioning to Windows 11 and are worried about your privacy (@itzg ✌). There is no straight forward way of creating a local account while installing Windows 11 anymore, you'll have to enter<EMAIL_ADDRESS>as your email when the setup asks you to login with your Microsoft Account, you should get an error and then be allowed to create a local account. When installing apps from the Microsoft Store, if you get a prompt asking you to login, just close it, the apps should install nevertheless.

I've tried to do what @langestefan,  @quiquelhappy and @Merith-TK said, but does not work.

@itzg not sure if this is still relevant, but I found that if I interacted with the docker commandline directly from my Ubuntu WSL2 machine I did not get any corrupted data. If I stopped a container using the windows command line I suddenly got issues.

Pretty sure this was the case. For some reason, every time I installed Docker Desktop on my Windows computer, with a single WSL OS installed, it never automatically enabled Docker support for such OS. My recommendation to anyone having this issue is to use Docker from a WSL OS directly...

aight work around for now on windows if you 100% must have access to config files

in your mounts, put it in this order

"<volume name>:/data/world"
"./local/path/:/data"

start container
access container shell as root
as root run chown minecraft:minecraft /data -R
restart container
...

I can not disable WSL and use Hyper-V as backend.
I've tried to mount the container in a named volume and worked, but I can not access there easily from win11. Is there any way to mount a named volume in a specific volume of win11 or to do a reference of the named volume into win11?
What I have now is this, but does not work:
version: "3"

services:
  mc:
    image: itzg/minecraft-server:latest
    container_name: mc-Final-Paradox
    ports:
      - 25565:25565
    expose:
      - 25565
    environment:
      EULA: "TRUE"
      DIFFICULTY: 'hard'
      VERSION: 1.16.5
      WHITE_LIST: "TRUE"
      GAMEMODE: survival
      MEMORY: '8G'
    tty: true
    stdin_open: true
    restart: unless-stopped
    volumes:
      - FP-data:/data
volumes:
  FP-data:
    driver: local
    driver_opts:
       o: bind
       type: none
       device: C:\Users\user\Desktop\minecraft_server\mincreft-data

If I leave it like this, works well, but I want to access the volume from win11:
...
volumes:
      - FP-data:/data
volumes:
  FP-data:

I found it, but I want to have the volume in the desktop, not as a direct access.

Coming from https://github.com/itzg/docker-minecraft-server/issues/1247. Ran into this exact issue when setting up a Ubuntu container environment for my server on Windows 11. Suggested solutions didn't work for me, found a workaround that may present as a new alternative for some. Utilizing the --world-container flag when starting up the server addresses incompatibility issues between NTFS and ext4 filesystems by introducing a separate Docker volume for storing worlds. Worlds are stored in a named Docker volume, the path of which is specified by the argument pointed out earlier, while the server directory remains a shared mapped volume with the host's filesystem.

@qruet can you point to documentation for --world-container?

The best documentation I could dig up can be found here.


[feature] add "repo" into top level of vendor file to redirect all paths to local repos

We plan to use golang in our team, and want to create a local repo mirror that copy all dependencies we used in our private network. But after playing with dependency manager tools like godep and govender, we realized it is difficult work.
I suggest add "repo" into top level of vender file, to redirect all origin to repo/origin, like this:
{
    "comment": "Note the use of a non-standard crypto package.",
    "repo":"privatedomain",
    "package": [
        {
            "path": "rsc.io/pdf",
            "revision": "3a3aeae79a3ec4f6d093a6b036c24698938158f3",
            "revisionTime": "2014-09-25T17:07:18-04:00",
            "comment": "located on disk at $GOPATH/src/github.com/kardianos/mypkg/vendor/rsc.io/pdf"
        },
        {
            "path": "golang.org/x/net/context",
            "revision": "25f1feceb5e13da68a35ee552069f86d18d63fee",
            "revisionTime": "2015-04-09T05:06:17-08:00",
            "comment": "Use the 25f1 version. Located on disk at $GOPATH/src/github.com/kardianos/mypkg/vendor/golang.org/x/net/context"
        }
    ]
}

This will tell vendor tools to fetch "privatedomain/rsc.io/pdf" as "rsc.io/pdf", "privatedomain/golang.org/x/net/context" as "golang.org/x/net/context". I think this will be a great idea providing golang dependency management the ability of private mirror repo.

+1 we are struggling with the same issue as well

I'm open to this.
In your opinion, would this be per project, or a global setting for all govendor use in that env? Would this private repo have a http meta-tag setup for it or would we need to record the full vcs info. I assume any credentials would be taken care of in the existing env.

I'm definitely interested.
A global setting or project-wide setting would be convenient, but it's hard for me to be sure right now that it would be flexible enough for all corporate settings. I think it makes sense to start with a per-project setting, and then later think about if there is a project or global setting that could be used to reduce the repetitive config.
I think the simplest thing that may work is:
        {
            "path": "rsc.io/pdf",
            "revision": "3a3aeae79a3ec4f6d093a6b036c24698938158f3",
            "revisionTime": "2014-09-25T17:07:18-04:00",
            "mirroredAt": "privatedomain.org/repos/thirdparty/rsc.io/pdf"
        }

For even more flexibility you may support multiple named mirrors:
        {
            "path": "rsc.io/pdf",
            "revision": "3a3aeae79a3ec4f6d093a6b036c24698938158f3",
            "revisionTime": "2014-09-25T17:07:18-04:00",
            "mirroredAt": {
              "private-us": "privatedomain.org/repos/thirdparty/rsc.io/pdf",
              "private-uk": "privatedomain.co.uk/repos/thirdparty/rsc.io/pdf"
             }
        }

This could then be selected on the command line with a --mirror flag, e.g.:
$ govendor sync --mirror private-us

URL templates may be useful for reducing the amount of config so you don't need to put a mirroredAt element on every package. For example, this config section would match the section above.
(Note: I'm not proposing adding the authentication settings at this point, just showing that mirrors may eventually need some other settings beyond just the urlTemplate)
{
  "mirrors": {
    "private-us": {
      "urlTemplate": "privatedomain.org/repos/thirdparty/{path}",
      "username": "foo",
      "password": "<hidden>"
    },
    "private-uk": {
      "urlTemplate": "privatedomain.co.uk/repos/thirdparty/{path}",
      "username": "foo",
      "password": "<hidden>"
     }
}

@maxlinc I completly agree that having some type of global config would be good.
I'm thinking of the current design:
New config file a $HOME/.config/govendor.json
{
    mirror_list: [
        {name: "project_mirror", url: "git+ssh://my-server.com/{.Package}", user: "optional", pass: "optional"}
    ],
    mirror: "project-mirror",
    mirror_require: true|false
}

New fields in vendor.json: "mirror": "project-mirror", "mirror_require": true|false", mirror_list: [].
If "mirror-require" is set to true, govendor must NOT fall back to a non-mirror source. All fields are optional in both "vendor.json" and "govendor.json". Each project uses the combination of the two sets of fields. First looks in user specific "govendor.json", then adds/overwrites fields from project specific "vendor.json".  This allows you to:

Specify at a company policy repos are only hit internally.
Only use internal repos internally to the company, but then publish some of the projects as open source.
Specify and require region specific mirrors, but allow each user to choose which mirror to use.

Too complicated? Meet your needs?


Tar paths starting with ./ confuse the app

Depending on how a tar file was created, the tar -tf output might look like
./README.txt
./image.jpeg

list_contents() has no problem with this, and the index page gets generated with links to /HASH/./README.txt etc. However, because of browser URL resolution, when the user clicks on the link, the request comes in as /HASH/README.txt. The contents list does not contain README.txt so we return an error.
I can see a couple of ways to deal with this:

list_contents() could strip initial ./ off the path.
When looking up the file (the details.contents.indexOf(file_path) call in app.js), we could do a fallback check for './'+file_path if the first lookup fails.

(This is a rare problem -- low priority. I first noticed it with http://ifarchive.org/if-archive/games/pc/mansion-19.2.tar.gz . That was when testing my fix for https://github.com/iftechfoundation/ifarchive-unbox/issues/38 , so you won't be able to observe this until that fix it in. I haven't hunted for other cases.)

list_contents() should definitely normalise paths. If we do that I'm not sure if anything else would be required.


Signed indexed_iter?

I often use signed values for computing locations on the grid, and one annoyance is that indexed_iter() forces usize. It would be nice to have some way to control the type of the values is produces. I think it would be invasive to add <T> to indexed_iter(), though, as Rust would start asking users to add the template param if you have a situation where the result could not be inferred. This is what I tried:
    pub fn indexed_iter<I>(&self) -> impl Iterator<Item = ((I, I), &T)>
    where
    I: TryFrom<usize> + Div<I> + Rem<I> + Copy,
    I: From<<I as Div>::Output>,
    I: From<<I as Rem>::Output>,
    {
        let cols: I = self.cols.try_into().ok().expect("Type too small to hold cols");
        let rows: I = self.rows.try_into().ok().expect("Type too small to hold rows");
        self.data.iter().enumerate().map(move |(idx, i)| {
            let idx: I = idx.try_into().ok().unwrap();
            let position: (I, I) = match self.order {
                Order::RowMajor => ((idx / cols).into(), (idx % cols).into()),
                Order::ColumnMajor => ((idx % rows).into(), (idx / rows).into()),
            };
            (position, i)
        })
    }

Note that the bounds checking only happens once, rather than every time, as it would for a user converting to the type later. But call sites can't always infer I, requiring a type here sometimes where it wasn't required before. Would this make sense as indexed_iter_t, perhaps?

Hm... I am not so sure about adding another _t function. My goal with this library was to more or less mimic the features of rust's std vec type. Since as far as I know a vec can also only be accessed with an unsigned type, I don't really think it does make sense to add the featuer for grid. But maybe I am also wrong and std vec supports and inferred type by now?
See failing playground: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&code=fn+swap(x%3A+i32%2C+y%3A+i32)+->+(i32%2C+i32)+{
++++return+(y%2C+x)%3B
}

fn+main()+{
++++let+v+%3D+vec![1%2C2%2C3]%3B
++++let+idx+%3A+i32+%3D+1%3B
++++println!("{}"%2C+v[idx])%3B
}

For std::vec, you can do vec.iter().enumerate().map(|(i, v)| ...), which is conceptually the same as grid.indexed_iter.map(|ind, v| ...). In the docs for enumerate, it says "if you need a custom type, use zip(0..) instead. So I'm looking for the equivalent to vec.iter().zip(0..).map(|(v, i)| ...), which allows you to either pick a type (0i32.., for example), or will infer the type if possible. I don't see any equivalent currently for grid.
Not particularly happy with indexed_iter_t, but not sure how it could be elegantly done. I've seen a trick mentioned that I could try.


version 3.1

Changes:

adds city

congrats @isabarros 👏  version 3.1 is out with your new city method!!


Can't install extensions

I get the message "Installation Failed" : "Unknown internal error."
Xubuntu 16.04, radeon driver
Console output:
Download the React DevTools for a better development experience: https://fb.me/react-devtools /thirdparty/react.js:6541
PreferencesManager.convertPreferences() has been deprecated. Please upgrade to the current Preferences system (https://github.com/adobe/brackets/wiki/Preferences-System#conversion-from-the-pre-36-preferences-system).
    at Object.convertPreferences (/preferences/PreferencesManager.js:325:28)
    at Object.eval (/utils/Resizer.js:560:24) /utils/DeprecationWarning.js:90
9
WebSocket connection to 'ws://localhost:34839/' failed: Unexpected response code: 401 /utils/NodeConnection.js:89
WebSocket connection to 'ws://localhost:34839/' failed: Unexpected response code: 401 /utils/NodeConnection.js:89
[Extension] Error -- could not read native directory: /home/noneatme/.config/Brackets/extensions/user /utils/ExtensionLoader.js:346
Fail to nomalize Cmd-Shift-] /command/KeyBindingManager.js:580
Fail to nomalize Cmd-Shift-[ /command/KeyBindingManager.js:580
NodeSocketTransport - start /LiveDevelopment/MultiBrowserImpl/transports/NodeSocketTransport.js:87
WebSocket connection to 'ws://localhost:34839/' failed: Unexpected response code: 401 /utils/NodeConnection.js:89
Unknown installation error extensionManager domain is undefined /extensibility/Package.js:411

What can I do?

Tagging @swmitra

same issue on ubuntu 16.04

Elementary 0.3.2
Same issue

Same issue here,
> Distributor ID:	Ubuntu
> Description:	Ubuntu 16.04 LTS
> Release:	16.04
> Codename:	xenial

same issue on ubuntu 16.04

We are currently looking at it. Sorry for the inconvenience. We should be able to fix this and refresh the build in a day or two.

Same issue on debian sid

There was an issue with 1.7 Linux 64 builds. It was to do with brackets unable to connect to Node. We have now fixed the problem and refreshed the builds. Could you guys download the Linux 1.7 builds again from brackets.io and check?

thanks for fix.

Hi,
I've downloaded latest Brackets from the website (Release 1.7 experimental build 1.7.0-16898 (release b0a363b71) - 64 bit build ) and updated to the latest nodeJS 4.4.7 LTS and the problem still persists for me.
I can't even check for updates in Brackets or install the extensions from URL.
I am running Linux Mint 17.3 if it helps.
Should I wait for the next release or will it take some time to fix?
Thanks for any answer

thanks @nethip, worked flawlessly for me.
Linux 3.13.0-91-generic #138~precise1-Ubuntu SMP Fri Jun 24 18:50:48 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Closing since Linux build should have been fixed.
@CrNix if you are still experiencing some issue please open a new issue.


[BUG] java.lang.NullPointerException when exporting delta table

Describe the bug
When saving a data frame using the delta format, I get the the following exception:
Py4JJavaError: An error occurred while calling o431.save.
: java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:201)
	at org.apache.spark.sql.execution.collect.DatasetSparkResult.toArray(DatasetSparkResult.scala:61)
	at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3714)
	at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2954)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3702)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$5(SQLExecution.scala:116)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:249)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:101)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:836)
	at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:77)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:199)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3700)
	at org.apache.spark.sql.Dataset.collect(Dataset.scala:2954)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.$anonfun$x$4$1(SnapshotEdge.scala:102)
	at com.databricks.sql.acl.CheckPermissions$.trusted(CheckPermissions.scala:1116)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.x$4$lzycompute(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.x$4(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge._metadata$lzycompute(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge._metadata(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.metadata(SnapshotEdge.scala:143)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.toString(SnapshotEdge.scala:233)
	at java.lang.String.valueOf(String.java:2994)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at com.databricks.sql.transaction.tahoe.Snapshot.$anonfun$new$1(Snapshot.scala:297)
	at com.databricks.sql.transaction.tahoe.Snapshot.$anonfun$logInfo$1(Snapshot.scala:274)
	at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
	at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
	at com.databricks.sql.transaction.tahoe.Snapshot.logInfo(Snapshot.scala:274)
	at com.databricks.sql.transaction.tahoe.Snapshot.<init>(Snapshot.scala:297)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.<init>(SnapshotEdge.scala:67)
	at com.databricks.sql.transaction.tahoe.SnapshotManagementEdge.createSnapshot(SnapshotManagementEdge.scala:85)
	at com.databricks.sql.transaction.tahoe.SnapshotManagementEdge.$anonfun$update$3(SnapshotManagementEdge.scala:111)
	at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:377)
	at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:363)
	at com.databricks.spark.util.SparkDatabricksProgressReporter$.withStatusCode(ProgressReporter.scala:34)
	at com.databricks.sql.transaction.tahoe.util.DeltaProgressReporterEdge.withStatusCode(DeltaProgressReporterEdge.scala:30)
	at com.databricks.sql.transaction.tahoe.util.DeltaProgressReporterEdge.withStatusCode$(DeltaProgressReporterEdge.scala:25)
	at com.databricks.sql.transaction.tahoe.DeltaLog.withStatusCode(DeltaLog.scala:65)
	at com.databricks.sql.transaction.tahoe.SnapshotManagementEdge.$anonfun$update$2(SnapshotManagementEdge.scala:92)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:431)
	at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
	at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
	at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
	at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
	at com.databricks.spark.util.PublicDBLogging.withAttributionTags(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:412)
	at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338)
	at com.databricks.spark.util.PublicDBLogging.recordOperation(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.spark.util.PublicDBLogging.recordOperation0(DatabricksSparkUsageLogger.scala:55)
	at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:98)
	at com.databricks.spark.util.UsageLogger.recordOperation(UsageLogger.scala:71)
	at com.databricks.spark.util.UsageLogger.recordOperation$(UsageLogger.scala:58)
	at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:67)
	at com.databricks.spark.util.UsageLogging.recordOperation(UsageLogger.scala:346)
	at com.databricks.spark.util.UsageLogging.recordOperation$(UsageLogger.scala:325)
	at com.databricks.sql.transaction.tahoe.DeltaLog.recordOperation(DeltaLog.scala:65)
	at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:108)
	at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:94)
	at com.databricks.sql.transaction.tahoe.DeltaLog.recordDeltaOperation(DeltaLog.scala:65)
	at com.databricks.sql.transaction.tahoe.SnapshotManagementEdge.$anonfun$update$1(SnapshotManagementEdge.scala:92)
	at com.databricks.sql.transaction.tahoe.DeltaLog.lockInterruptibly(DeltaLog.scala:167)
	at com.databricks.sql.transaction.tahoe.SnapshotManagementEdge.update(SnapshotManagementEdge.scala:91)
	at com.databricks.sql.transaction.tahoe.SnapshotManagementEdge.update$(SnapshotManagementEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.DeltaLog.update(DeltaLog.scala:65)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge.$anonfun$doCommit$1(OptimisticTransactionImplEdge.scala:304)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at com.databricks.sql.transaction.tahoe.DeltaLog.lockInterruptibly(DeltaLog.scala:167)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge.doCommit(OptimisticTransactionImplEdge.scala:273)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge.doCommit$(OptimisticTransactionImplEdge.scala:269)
	at com.databricks.sql.transaction.tahoe.OptimisticTransaction.doCommit(OptimisticTransaction.scala:84)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$2(OptimisticTransaction.scala:536)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:431)
	at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
	at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
	at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
	at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
	at com.databricks.spark.util.PublicDBLogging.withAttributionTags(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:412)
	at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338)
	at com.databricks.spark.util.PublicDBLogging.recordOperation(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.spark.util.PublicDBLogging.recordOperation0(DatabricksSparkUsageLogger.scala:55)
	at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:98)
	at com.databricks.spark.util.UsageLogger.recordOperation(UsageLogger.scala:71)
	at com.databricks.spark.util.UsageLogger.recordOperation$(UsageLogger.scala:58)
	at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:67)
	at com.databricks.spark.util.UsageLogging.recordOperation(UsageLogger.scala:346)
	at com.databricks.spark.util.UsageLogging.recordOperation$(UsageLogger.scala:325)
	at com.databricks.sql.transaction.tahoe.OptimisticTransaction.recordOperation(OptimisticTransaction.scala:84)
	at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:108)
	at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:94)
	at com.databricks.sql.transaction.tahoe.OptimisticTransaction.recordDeltaOperation(OptimisticTransaction.scala:84)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImpl.$anonfun$doCommitRetryIteratively$1(OptimisticTransaction.scala:532)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at com.databricks.sql.transaction.tahoe.DeltaLog.lockInterruptibly(DeltaLog.scala:167)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImpl.doCommitRetryIteratively(OptimisticTransaction.scala:527)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImpl.doCommitRetryIteratively$(OptimisticTransaction.scala:524)
	at com.databricks.sql.transaction.tahoe.OptimisticTransaction.doCommitRetryIteratively(OptimisticTransaction.scala:84)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge.$anonfun$commit$1(OptimisticTransactionImplEdge.scala:222)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperation$4(UsageLogging.scala:431)
	at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:239)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:234)
	at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:231)
	at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:276)
	at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:269)
	at com.databricks.spark.util.PublicDBLogging.withAttributionTags(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:412)
	at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:338)
	at com.databricks.spark.util.PublicDBLogging.recordOperation(DatabricksSparkUsageLogger.scala:18)
	at com.databricks.spark.util.PublicDBLogging.recordOperation0(DatabricksSparkUsageLogger.scala:55)
	at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:98)
	at com.databricks.spark.util.UsageLogger.recordOperation(UsageLogger.scala:71)
	at com.databricks.spark.util.UsageLogger.recordOperation$(UsageLogger.scala:58)
	at com.databricks.spark.util.DatabricksSparkUsageLogger.recordOperation(DatabricksSparkUsageLogger.scala:67)
	at com.databricks.spark.util.UsageLogging.recordOperation(UsageLogger.scala:346)
	at com.databricks.spark.util.UsageLogging.recordOperation$(UsageLogger.scala:325)
	at com.databricks.sql.transaction.tahoe.OptimisticTransaction.recordOperation(OptimisticTransaction.scala:84)
	at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordDeltaOperation(DeltaLogging.scala:108)
	at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordDeltaOperation$(DeltaLogging.scala:94)
	at com.databricks.sql.transaction.tahoe.OptimisticTransaction.recordDeltaOperation(OptimisticTransaction.scala:84)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge.commit(OptimisticTransactionImplEdge.scala:168)
	at com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge.commit$(OptimisticTransactionImplEdge.scala:164)
	at com.databricks.sql.transaction.tahoe.OptimisticTransaction.commit(OptimisticTransaction.scala:84)
	at com.databricks.sql.transaction.tahoe.commands.WriteIntoDelta.$anonfun$run$2(WriteIntoDelta.scala:74)
	at com.databricks.sql.transaction.tahoe.commands.WriteIntoDelta.$anonfun$run$2$adapted(WriteIntoDelta.scala:70)
	at com.databricks.sql.transaction.tahoe.DeltaLog.withNewTransaction(DeltaLog.scala:203)
	at com.databricks.sql.transaction.tahoe.commands.WriteIntoDelta.$anonfun$run$1(WriteIntoDelta.scala:70)
	at com.databricks.sql.acl.CheckPermissions$.trusted(CheckPermissions.scala:1116)
	at com.databricks.sql.transaction.tahoe.commands.WriteIntoDelta.run(WriteIntoDelta.scala:69)
	at com.databricks.sql.transaction.tahoe.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:155)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:91)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:195)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$3(SparkPlan.scala:247)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:165)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:187)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:158)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:157)
	at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:999)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$5(SQLExecution.scala:116)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:249)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:101)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:836)
	at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:77)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:199)
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:999)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:437)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:415)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)

Steps/Code to reproduce bug
I haven't been able to reduce the issue to a small reproducible example yet. The Python stacktrace highlights the following location:
df.write.format('delta').partitionBy('aaa').save(path)

Interestingly, the Spark UI does not show any failed jobs or stages and the actual data write seems to have happened. Judging from the above stacktrace the delta log/metadata may be corrupt though since the issue seems to happen when writing the delta log.
Expected behavior
Data frame export to delta succeeds without exception.
Environment details

Environment location: Azure Databricks 7.3 LTS using Databricks Container Services docker image with CUDA 10.2
Recent rapids-4-spark 0.3.0-SNAPSHOT

$ git rev-parse HEAD
b2f03e0eae642c7999122c00127cd045627520dc

Spark configuration settings

"spark.plugins": "com.nvidia.spark.SQLPlugin",
"spark.sql.parquet.filterPushdown": "false",
"spark.rapids.sql.incompatibleOps.enabled": "true",
"spark.rapids.memory.pinnedPool.size": "2G",
"spark.task.resource.gpu.amount": 0.1,
"spark.rapids.sql.concurrentGpuTasks": 2,
"spark.locality.wait": "0s",
"spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version": 1,
"spark.kryo.registrator": "com.nvidia.spark.rapids.GpuKryoRegistrator"
"spark.databricks.delta.optimizeWrite.enabled": "false",
"spark.databricks.delta.autoCompact.enabled": "false",

Additional context
There seems to be no issue with rapids-4-spark 0.2.0.

thanks @martinstuder for reporting we haven't seen this before so I'll investigate.

@martinstuder  I'm trying to reproduce this, but haven't been able to yet.

df.write.format('delta').partitionBy('aaa').save(path)

Was this on the first write to this path?   The table didn't exist previously? can you tell me the schema for the partitionBy column - is it just a timestamp?
Based on your comment that no jobs failed, this exception happened on the driver side then.

at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:201)
at org.apache.spark.sql.execution.collect.DatasetSparkResult.toArray(DatasetSparkResult.scala:61)
at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3714)

Looking at the last few lines here it seems its just trying to collect it back to the driver but the deserializer doesn't like the what it got back. the DatasetSparkResult is a databricks specific class which we don't have source for.  Would you be able to perhaps tell me the last operation before the write? I'm wondering if that operation produced results that doesn't match what databricks expected.

@tgravescs Reconstructing the code path it basically comes down to the following:
df1 = spark_session.read.format('parquet').schema(schema).load(path1)
df2 = spark_session.read.format('parquet').schema(schema).load(path2)
df = df1 \
    .withColumn('extra', F.lit(0).cast('byte')) \
    .unionByName(df2.withColumn('extra', F.lit(1).cast('byte')))
df.write.format('delta').partitionBy('aaa').save(path)

It's basically unioning two source data frames from parquet, adding an extra discriminator column to both and then writing to delta with a different partitioning scheme. The issue happened on the first write to path with no table existing there. The partition-by column (aaa) is an integer.
We've recently changed things to have the source data frames in delta as well where we read them without specifying a schema explicitly. I'll check whether this has any impact.

I was just able to reproduce this with delta as the source for input data frames:
Py4JJavaError: An error occurred while calling o409.save.
: java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Deserializer.apply(ExpressionEncoder.scala:201)
	at org.apache.spark.sql.execution.collect.DatasetSparkResult.toArray(DatasetSparkResult.scala:61)
	at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3714)
	at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2954)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3702)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$5(SQLExecution.scala:116)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:249)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:101)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:836)
	at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:77)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:199)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3700)
	at org.apache.spark.sql.Dataset.collect(Dataset.scala:2954)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.$anonfun$x$4$1(SnapshotEdge.scala:102)
	at com.databricks.sql.acl.CheckPermissions$.trusted(CheckPermissions.scala:1116)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.x$4$lzycompute(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.x$4(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge._metadata$lzycompute(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge._metadata(SnapshotEdge.scala:90)
	at com.databricks.sql.transaction.tahoe.SnapshotEdge.metadata(SnapshotEdge.scala:143)
	[...]

Below is the last portion of the driver log4j output. The first three "Finished task" lines are from the write stage:
20/12/02 19:35:24 INFO TaskSetManager: Finished task 245.0 in stage 2.0 (TID 322) in 241069 ms on <IP_ADDRESS> (executor 0) (301/303)
20/12/02 19:35:27 INFO TaskSetManager: Finished task 261.0 in stage 2.0 (TID 338) in 243903 ms on <IP_ADDRESS> (executor 6) (302/303)
20/12/02 19:35:37 INFO TaskSetManager: Finished task 279.0 in stage 2.0 (TID 356) in 253963 ms on <IP_ADDRESS> (executor 7) (303/303)
20/12/02 19:35:37 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool<PHONE_NUMBER>745350677
20/12/02 19:35:37 INFO DAGScheduler: ResultStage 2 (write at TransactionalWriteEdge.scala:183) finished in 461.573 s
20/12/02 19:35:37 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
20/12/02 19:35:37 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
20/12/02 19:35:37 INFO DAGScheduler: Job 2 finished: write at TransactionalWriteEdge.scala:183, took 461.591995 s
20/12/02 19:35:37 INFO FileFormatWriter: Write Job cf9aeeb1-fdd0-49d3-94c9-9c9bf4a39dc5 committed.
20/12/02 19:35:37 INFO FileFormatWriter: Finished processing stats for write job cf9aeeb1-fdd0-49d3-94c9-9c9bf4a39dc5.
20/12/02 19:35:37 INFO OptimisticTransaction: [tableId=2c4f3ede-e0b7-4eb3-afc6-c1e1389d121c] Attempting to commit version 0 with 158343 actions with WriteSerializable isolation level
20/12/02 19:35:38 INFO AzureBlobFileSystem: FS_OP_CREATE FILE[/aaa/_delta_log/__tmp_path_dir/.00000000000000000000.json.5daef307-543a-4a02-8426-a852a621641a.tmp] Creating output stream; permission: rw-r--r--, overwrite: false, bufferSize: 33554432
20/12/02 19:35:38 INFO RetryTolerableRenameFSDataOutputStream: Writing atomically to dbfs:/mnt/output/aaa/_delta_log/00000000000000000000.json using temp file dbfs:/mnt/output/aaa/_delta_log/__tmp_path_dir/.00000000000000000000.json.5daef307-543a-4a02-8426-a852a621641a.tmp
20/12/02 19:35:38 INFO AbfsOutputStream: FS_OP_CREATE FILE[/aaa/_delta_log/__tmp_path_dir/.00000000000000000000.json.5daef307-543a-4a02-8426-a852a621641a.tmp] Closing stream; size: 95766506
20/12/02 19:35:39 INFO AbfsOutputStream: FS_OP_CREATE FILE[/aaa/_delta_log/__tmp_path_dir/.00000000000000000000.json.5daef307-543a-4a02-8426-a852a621641a.tmp] Upload complete; size: 95766506
20/12/02 19:35:39 INFO AzureBlobFileSystem: FS_OP_RENAME [/aaa/_delta_log/__tmp_path_dir/.00000000000000000000.json.5daef307-543a-4a02-8426-a852a621641a.tmp] to [/aaa/_delta_log/00000000000000000000.json] Starting rename. Issuing rename operation.
20/12/02 19:35:39 INFO AzureBlobFileSystem: FS_OP_RENAME [/aaa/_delta_log/__tmp_path_dir/.00000000000000000000.json.5daef307-543a-4a02-8426-a852a621641a.tmp] to [/aaa/_delta_log/00000000000000000000.json] Rename successful.
20/12/02 19:35:39 INFO RetryTolerableRenameFSDataOutputStream: Renamed temp file dbfs:/mnt/output/aaa/_delta_log/__tmp_path_dir/.00000000000000000000.json.5daef307-543a-4a02-8426-a852a621641a.tmp to dbfs:/mnt/output/aaa/_delta_log/00000000000000000000.json
20/12/02 19:35:39 INFO DeltaLog: Loading version 0.
20/12/02 19:35:39 INFO DeltaLogFileIndex: Created DeltaLogFileIndex(JSON, numFilesInSegment: 1, totalFileSize: 95766506)
20/12/02 19:35:39 INFO FileSourceStrategy: Pruning directories with:
20/12/02 19:35:39 INFO FileSourceStrategy: Pushed Filters: Or(IsNotNull(protocol),IsNotNull(metaData))
20/12/02 19:35:39 INFO FileSourceStrategy: Post-Scan Filters: (isnotnull(protocol#417) OR isnotnull(metaData#416))
20/12/02 19:35:39 INFO FileSourceStrategy: Output Data Schema: struct<metaData: struct<id: string, name: string, description: string, format: struct<provider: string, options: map<string,string>>, schemaString: string ... 6 more fields>, protocol: struct<minReaderVersion: int, minWriterVersion: int>>
20/12/02 19:35:39 INFO CodeGenerator: Code generated in 29.995099 ms
20/12/02 19:35:39 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 351.8 KiB, free 15.2 GiB)
20/12/02 19:35:39 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 19.6 KiB, free 15.2 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:41649 (size: 19.6 KiB, free: 15.2 GiB)
20/12/02 19:35:39 INFO SparkContext: Created broadcast 6 from execute at GpuRowToColumnarExec.scala:811
20/12/02 19:35:39 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
20/12/02 19:35:39 INFO SparkContext: Starting job: collect at SnapshotEdge.scala:102
20/12/02 19:35:39 INFO DAGScheduler: Got job 3 (collect at SnapshotEdge.scala:102) with 23 output partitions
20/12/02 19:35:39 INFO DAGScheduler: Final stage: ResultStage 3 (collect at SnapshotEdge.scala:102)
20/12/02 19:35:39 INFO DAGScheduler: Parents of final stage: List()
20/12/02 19:35:39 INFO DAGScheduler: Missing parents: List()
20/12/02 19:35:39 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[28] at mapPartitions at GpuColumnarToRowExec.scala:294), which has no missing parents
20/12/02 19:35:39 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 33.3 KiB, free 15.2 GiB)
20/12/02 19:35:39 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 9.9 KiB, free 15.2 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:41649 (size: 9.9 KiB, free: 15.2 GiB)
20/12/02 19:35:39 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1547
20/12/02 19:35:39 INFO DAGScheduler: Submitting 23 missing tasks from ResultStage 3 (MapPartitionsRDD[28] at mapPartitions at GpuColumnarToRowExec.scala:294) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
20/12/02 19:35:39 INFO TaskSchedulerImpl: Adding task set 3.0 with 23 tasks
20/12/02 19:35:39 INFO TaskSetManager: Jars for session None: Map()
20/12/02 19:35:39 INFO TaskSetManager: Files for session None: Map()
20/12/02 19:35:39 INFO FairSchedulableBuilder: Added task set TaskSet_3.0 tasks to pool<PHONE_NUMBER>745350677
20/12/02 19:35:39 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 380, <IP_ADDRESS>, executor 8, partition 0, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 381, <IP_ADDRESS>, executor 1, partition 1, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 382, <IP_ADDRESS>, executor 6, partition 2, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 383, <IP_ADDRESS>, executor 9, partition 3, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 4.0 in stage 3.0 (TID 384, <IP_ADDRESS>, executor 4, partition 4, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 5.0 in stage 3.0 (TID 385, <IP_ADDRESS>, executor 7, partition 5, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 6.0 in stage 3.0 (TID 386, <IP_ADDRESS>, executor 3, partition 6, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 7.0 in stage 3.0 (TID 387, <IP_ADDRESS>, executor 2, partition 7, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 8.0 in stage 3.0 (TID 388, <IP_ADDRESS>, executor 5, partition 8, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 9.0 in stage 3.0 (TID 389, <IP_ADDRESS>, executor 0, partition 9, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 10.0 in stage 3.0 (TID 390, <IP_ADDRESS>, executor 8, partition 10, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 11.0 in stage 3.0 (TID 391, <IP_ADDRESS>, executor 1, partition 11, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 12.0 in stage 3.0 (TID 392, <IP_ADDRESS>, executor 6, partition 12, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 13.0 in stage 3.0 (TID 393, <IP_ADDRESS>, executor 9, partition 13, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 14.0 in stage 3.0 (TID 394, <IP_ADDRESS>, executor 4, partition 14, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 15.0 in stage 3.0 (TID 395, <IP_ADDRESS>, executor 7, partition 15, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 16.0 in stage 3.0 (TID 396, <IP_ADDRESS>, executor 3, partition 16, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 17.0 in stage 3.0 (TID 397, <IP_ADDRESS>, executor 2, partition 17, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 18.0 in stage 3.0 (TID 398, <IP_ADDRESS>, executor 5, partition 18, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 19.0 in stage 3.0 (TID 399, <IP_ADDRESS>, executor 0, partition 19, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 20.0 in stage 3.0 (TID 400, <IP_ADDRESS>, executor 8, partition 20, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 21.0 in stage 3.0 (TID 401, <IP_ADDRESS>, executor 1, partition 21, PROCESS_LOCAL)
20/12/02 19:35:39 INFO TaskSetManager: Starting task 22.0 in stage 3.0 (TID 402, <IP_ADDRESS>, executor 6, partition 22, PROCESS_LOCAL)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:41817 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:37389 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:37481 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:41215 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:35131 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:41067 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:40297 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:42369 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:36759 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on <IP_ADDRESS>:45565 (size: 9.9 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:40297 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:41817 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:41215 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:41067 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:42369 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:36759 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:37481 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:45565 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:39 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:37389 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:40 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on <IP_ADDRESS>:35131 (size: 19.6 KiB, free: 66.3 GiB)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 18.0 in stage 3.0 (TID 398) in 4693 ms on <IP_ADDRESS> (executor 5) (1/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 8.0 in stage 3.0 (TID 388) in 4695 ms on <IP_ADDRESS> (executor 5) (2/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 17.0 in stage 3.0 (TID 397) in 4742 ms on <IP_ADDRESS> (executor 2) (3/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 7.0 in stage 3.0 (TID 387) in 4743 ms on <IP_ADDRESS> (executor 2) (4/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 12.0 in stage 3.0 (TID 392) in 4769 ms on <IP_ADDRESS> (executor 6) (5/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 22.0 in stage 3.0 (TID 402) in 4768 ms on <IP_ADDRESS> (executor 6) (6/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 2.0 in stage 3.0 (TID 382) in 4769 ms on <IP_ADDRESS> (executor 6) (7/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 9.0 in stage 3.0 (TID 389) in 4817 ms on <IP_ADDRESS> (executor 0) (8/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 19.0 in stage 3.0 (TID 399) in 4816 ms on <IP_ADDRESS> (executor 0) (9/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 4.0 in stage 3.0 (TID 384) in 4821 ms on <IP_ADDRESS> (executor 4) (10/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 14.0 in stage 3.0 (TID 394) in 4821 ms on <IP_ADDRESS> (executor 4) (11/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 16.0 in stage 3.0 (TID 396) in 4862 ms on <IP_ADDRESS> (executor 3) (12/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 6.0 in stage 3.0 (TID 386) in 4862 ms on <IP_ADDRESS> (executor 3) (13/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 10.0 in stage 3.0 (TID 390) in 4942 ms on <IP_ADDRESS> (executor 8) (14/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 20.0 in stage 3.0 (TID 400) in 4941 ms on <IP_ADDRESS> (executor 8) (15/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 380) in 4947 ms on <IP_ADDRESS> (executor 8) (16/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 3.0 in stage 3.0 (TID 383) in 4999 ms on <IP_ADDRESS> (executor 9) (17/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 13.0 in stage 3.0 (TID 393) in 4999 ms on <IP_ADDRESS> (executor 9) (18/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 381) in 5035 ms on <IP_ADDRESS> (executor 1) (19/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 21.0 in stage 3.0 (TID 401) in 5034 ms on <IP_ADDRESS> (executor 1) (20/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 11.0 in stage 3.0 (TID 391) in 5035 ms on <IP_ADDRESS> (executor 1) (21/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 5.0 in stage 3.0 (TID 385) in 5296 ms on <IP_ADDRESS> (executor 7) (22/23)
20/12/02 19:35:44 INFO TaskSetManager: Finished task 15.0 in stage 3.0 (TID 395) in 5296 ms on <IP_ADDRESS> (executor 7) (23/23)
20/12/02 19:35:44 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool<PHONE_NUMBER>745350677
20/12/02 19:35:44 INFO DAGScheduler: ResultStage 3 (collect at SnapshotEdge.scala:102) finished in 5.319 s
20/12/02 19:35:44 INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job
20/12/02 19:35:44 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished
20/12/02 19:35:44 INFO DAGScheduler: Job 3 finished: collect at SnapshotEdge.scala:102, took 5.328319 s
20/12/02 19:35:45 INFO AzureBlobFileSystem: FS_OP_DELETE  [/checkpoints_d398a706784a44b7b08488ec65e0f2ed] Starting delete - recursive true
20/12/02 19:35:45 INFO AzureBlobFileSystem: FS_OP_DELETE  [/checkpoints_d398a706784a44b7b08488ec65e0f2ed] Delete successful.
20/12/02 19:35:45 INFO ProgressReporter$: Removed result fetcher for<PHONE_NUMBER>745350677_8580777726221396950_job-23307-run-1-action-23295
20/12/02 19:35:46 INFO DriverCorral$: Cleaning the wrapper ReplId-31d72-d58ad-abd21-5 (currently in status Idle(ReplId-31d72-d58ad-abd21-5))
20/12/02 19:35:46 INFO DriverCorral$: sending shutdown signal for REPL ReplId-31d72-d58ad-abd21-5
20/12/02 19:35:46 INFO PythonDriverLocal$RedirectThread: Python RedirectThread exit
20/12/02 19:35:46 INFO PythonDriverLocal$RedirectThread: Python RedirectThread exit
20/12/02 19:35:46 INFO PythonDriverLocal$Watchdog: Python shell exit code: 143
20/12/02 19:35:46 INFO PythonDriverLocal$Watchdog: Strace is not enabled. To turn it on please set the spark conf spark.databricks.daemon.driver.python.withStrace to be true.
20/12/02 19:35:46 INFO DriverCorral$: sending the interrupt signal for REPL ReplId-31d72-d58ad-abd21-5
20/12/02 19:35:46 INFO DriverCorral$: waiting for localThread to stop for REPL ReplId-31d72-d58ad-abd21-5
20/12/02 19:35:46 INFO DriverCorral$: ReplId-31d72-d58ad-abd21-5 successfully discarded

When checking the delta table output, I can see the data files are there. What is strange though is the _delta_log directory:

The metadata file (*.json) is fairly large and the *.crc file is missing. Usually, I would expect the metadata file to be in the low MB range, together with a *.crc file:

The size of the metadata file may be explained by the fact that many more smaller parquet files are being written, together with appropriate column stats.

Just a guess: is it trying to read the metadata (*.json) back in to generate some log/diagnostic message but failing with the generated metadata?

thanks for the update, I'll give it another try to reproduce using delta file input and see what is going on. I suspect you are right about the metadata or its recording something like input version or something since it knows its delta format

yeah its likely your write wasn't happening on the GPU anyway at least my testing with writing to delta uses  SaveIntoDataSourceCommand com.databricks.sql.transaction.tahoe.sources.DeltaDataSource which GPU doesn't know how to do delta format.  I'm still trying to reproduce.
Are any of the column types complex types - like array?  Or is it all Long, Int, String types.  We are adding some support for complex types in 0.3 and since you said this worked with 0.2 I'm wondering if something there is not complete yet or causing incompatibilityes with databricks.

Sorry missed it earlier, I think its actually in your log and it looks like you are using structs:
20/12/02 19:35:39 INFO FileSourceStrategy: Output Data Schema: struct<metaData: struct<id: string, name: string, description: string, format: struct<provider: string, options: map<string,string>>, schemaString: string ... 6 more fields>, protocol: struct<minReaderVersion: int, minWriterVersion: int>>

The actual data frames are all just int, long, double and byte. The schema you are referring to seems to be the schema that delta uses for its internal metadata.

@martinstuder  unfortunately I still can't reproduce, would you be able to try to get a small reproducible case? I"m not sure if its the data or perhaps something in delta configuration or setup.   I tried both our latest 0,3,0 and went back to the commit you had.
I've tried a bunch of different things.  Perhaps you could try this sample I found on databricks to see if it reproduces for you?
I tried with both saveAsTable and just save/load so feel free to change those below and update the paths as needed.
from pyspark.sql.functions import *

events = spark.read.json("/databricks-datasets/structured-streaming/events/")
eventsWithDate = events.filter(col("time") <<PHONE_NUMBER>).withColumn("date_col", from_unixtime(col("time")))
spark.conf.set("spark.rapids.sql.format.parquet.enabled", "false")
eventsWithDate.write.format("delta").saveAsTable("deltaeventsPartitioned2Table")
eventsWithDate.write.format("delta").saveAsTable("deltaeventsPartitioned3Table")

from pyspark.sql import functions as F
df1 = spark.read.format('delta').load("/user/hive/warehouse/deltaeventspartitioned2table")
df2 = spark.read.format('delta').load("/user/hive/warehouse/deltaeventspartitioned3table")

df = df1 \
    .withColumn('extra', F.lit(0).cast('byte')) \
    .unionByName(df2.withColumn('extra', F.lit(1).cast('byte')))
df.write.format("delta").partitionBy("time").saveAsTable("deltaeventsPartitioned2TableWriteAgain3")

Also just to verify, if you use the 0.2.0 build with exact same setup you don't see the error?
Do you see this error every run?

also if its possible could you send me the explain output for the failure?  You can go to the SQL tab in UI and at the bottom is the text representation

@martinstuder  Sorry you can ignore the above questions I was finally able to reproduce it, will investigate more tomorrow.

I'm working on a workaround for this where we have the delta log queries use the CPU.

@martinstuder I checked in a fix for this would you be able to try it out?  You would have to build with the latest branch.

Hi @tgravescs, I can confirm that I can successfully export delta tables using a current build of the 0.3 branch. Thanks a lot for your efforts!


How to launch a TigerEvent at server startup?

I am using a script to have persistent cars, but I need that when the server is restarted, either manually or automatically, it launches a tigerEvent to save the position of the cars.
I need to know in which file and line to be able to add that tigert event for it to work, since it is something very important for me

Ressource start event
Or retard,  put it in the code anywhere without any function. Then it will auto-trigger 🤦♂

Ressource start event
O retard, póngalo en el código en cualquier lugar sin ninguna función. Entonces se disparará automáticamente🤦♂

this code has to be started before the server stops

Ressource start event
Or retard, put it in the code anywhere without any function. Then it will auto-trigger 🤦♂

I have put the triggerevent in this part of the server file, but when I restart the server it does not launch.
It has to be on the server side, on the server restart command line, either manual or automatic

wow.

Ofcourse. u trigger ur stuff when u kick all the players. wtf.
Check out OnResourceStop & OnResourcestart event. first event which u get on the fivem docs


Rename old create/alter table statement classes to proper names

See commits.

@kovrus add more related commits, pls check (sry missed them).


Use in memory config and do not save to home in guids

This PR removes the usage of Config() in favor of qcode.config and removes save_to_home() calls in guids.py. This PR is necessary for #2016 to pass. With these changes test_database_creation_and_upgrading.py passes with and without xdist-parallel without the need of a temp config.
@jenshnielsen

Tests fails as they call for Config() which is fixed in #2016. Maybe I should close this PR and push the changes to same branch? @jenshnielsen

@GateBuilder Yes that sounds good


[stable/postgresql] initContainer: also chown mount point itself.

Avoids error where start script can't touch .initialized* files.
What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/chart])

/ok-to-test
/lgtm


File eccodes.h not found when building the package in Windows/Linux manually

What happened?
As the wheels for Linux or Windows are missing from PyPi for the latest versions, they need to be created manually from the repository.
When doing so, the following error can be read (in Linux or Windows):
fatal error: eccodes.h: No such file or directory
What are the steps to reproduce the bug?
From the repository, execute, in Windows or Linux:
python setup.py build
Version
0.5.7
Platform (OS and architecture)
Windows/Linux
Relevant log output
No response
Accompanying data
No response
Organisation
No response

Same issue.

Same here as well


First steps to make a utils library

Restructuring the project into three (four actually) distinct projects:

scalacheck-datetime - the one we know and love
magic - inclusion of generators for "magic" strings (drop database STUDENTS etc)
combinators - generators for common use-cases (ie, a map and a list of elements guaranteed to be in the map)
docs - the fourth project, which is for the Jekyll site

magic and combinators are still in a skeleton form for this PR.
Also todo following this PR is the renaming in Github, docs and elsewhere to reflect use as a utils library rather than just datetime.
@raulraja would you be able to take a look please?

Thanks!
Re the microsite changes, I'll do this in a separate PR before a release.


Reject identical targets in the same family

Mostly because I added this for myself and I'm running into stuff while building, not while CI is running the tests.