Catboost Classifier - IndexError: list index out of range

Hi all. I am using the latest version of shap 0.39.0 and catboost 0.26.0.  I am running this code snippet.
import shap
shap.initjs()

explainer = shap.TreeExplainer(cat)
shap_values = explainer.shap_values(original_Xtrain)

# visualize the first prediction's explanation
shap.force_plot(explainer.expected_value, shap_values[0,:], original_Xtrain.iloc[0,:])

I stumble upon this issue. Any idea what is the reason to fail?
IndexError                                Traceback (most recent call last)
<ipython-input-117-900c1f23736e> in <module>()
      2 shap.initjs()
      3
----> 4 explainer = shap.TreeExplainer(cat)
      5 shap_values = explainer.shap_values(original_Xtrain)
      6

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/shap/explainers/tree.py in __init__(self, model, data, model_output, feature_perturbation, **deprecated_options)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/shap/explainers/tree.py in __init__(self, model, data, data_missing)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/shap/explainers/tree.py in get_trees(self, data, data_missing)

IndexError: list index out of range

Hi @beliakon,
On my side, shap 0.39.0 is compatible with catboost 0.26.0.
import shap
from catboost import CatBoostRegressor

train_data = [[1, 4, 5, 6], [4, 5, 6, 7], [30, 40, 50, 60]]
train_labels = [10, 20, 30]

model = CatBoostRegressor(
    iterations=2,
    learning_rate=1,
    depth=2
)
model.fit(train_data, train_labels)

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(train_data)
shap.force_plot(explainer.expected_value, shap_values[0,:], train_data[0], matplotlib=True)

What is containing your "cat" variable? Could you please send us a way to reproduce the issue?

Please close issue. I update correctly the version of libraries and everything is working.


Bugfix/watchlist errors

Description of Change
The included-users/add & included-users/delete endpoints don't actually 404 on bad watchlistId or userId values. They return a 400 instead. Added logic to catch those and raise appropriate "not found" exceptions instead.

I know when I was testing the get-watchlist-member() command it returned a 404 without specifying which resource it couldn't find.  Although I should've accounted for the user_ids in the original error handling - it is odd that the included-users endpoints return something completely different. 🤔


Zero-config

Describe the feature you'd like to request
Go back to zero config
Describe the solution you'd like

[ ] Remove the need for turbo.json or any config and assume every task in topological by default (like lerna)
[ ] Make caching opt-in (remove the defaults) or detected

[ ] 	Write a codemod to take existing undefined outputs in each pipeline key and set them as ['dist/**', 'build/**'] (which is the current default)

Describe alternatives you've considered
N/A

I do actually like that turbo fails when a command is not in the turbo.json.  It ensure I set the dependencies right and rename my scripts correctly.


iOS chip tool - Scanning QR code doesn't always start pairing process (CHIP Error 0x00000003: Incorrect state at ../../../../../../../../../../../connectedhomeip/src/controller/SetUpCodePairer)

Problem
Scanning QR code using the iOS chip tool app doesn't always start pairing process
SHA: 983fedfcbc38909d73b80f8b5c1d2d8ac163d0ff

Build iOS chip tool app using Xcode + load on iOS device
Use QACode scanner option to scan in a QR code eg. from ESP32 board

The step 2, occasionally eg. 25% of times, App gets into a state where it doesn't start the pairing process. Following error is seen:
2022-02-09 09:51:23.904861-0800  localhost CHIPTool[6690]: (CHIP) [com.zigbee.chip:all] ð´ [1644429083904] [6690:247785] CHIP: [-] ../../../../../../../../../../../connectedhomeip/src/controller/CHIPDeviceController.cpp:789: CHIP Error 0x00000003: Incorrect state at ../../../../../../../../../../../connectedhomeip/src/controller/SetUpCodePairer
2022-02-09 09:51:23.905582-0800  localhost CHIPTool[6690]: (CoreBluetooth) [com.apple.bluetooth:CoreBluetooth] Received XPC message 5: (null)
2022-02-09 09:51:23.905632-0800  localhost CHIPTool[6690]: (CoreBluetooth) [com.apple.bluetooth:CoreBluetooth] XPC connection finalized
2022-02-09 09:51:23.905719-0800  localhost CHIPTool[6690]: (CoreBluetooth) [com.apple.bluetooth:CoreBluetooth] XPC connection invalid
2022-02-09 09:51:25.353261-0800  localhost CHIPTool[6690]: (CHIP) PersistentStorageDelegate Set Key StartKeyID
2022-02-09 09:51:25.353699-0800  localhost CHIPTool[6690]: (CoreFoundation) [com.apple.defaults:User Defaults] setting {
StartKeyID = "AgA=";
} in CFPrefsPlistSource<0x28171eb80> (Domain: com.apple.chiptool, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No)
2022-02-09 09:51:25.355357-0800  localhost CHIPTool[6690]: (CHIP) DevicePairingDelegate Pairing complete. Status ../../../../../../../../../../../connectedhomeip/src/controller/CHIPDeviceController.cpp:1399: CHIP Error 0x00000032: Timeout
2022-02-09 09:51:25.355769-0800  localhost CHIPTool[6690]: (CoreFoundation) [com.apple.CFBundle:resources] Resource lookup at CFBundle 0x10cd04b90 </private/var/containers/Bundle/Application/BA5EB36A-1B92-4C79-8430-BADCFA9EB7F6/CHIPTool.app> (executable, loaded)
Request       : Localizable type: strings
Result        : None
2022-02-09 09:51:25.355922-0800  localhost CHIPTool[6690]: (CoreFoundation) [com.apple.CFBundle:resources] Resource lookup at CFBundle 0x10cd04b90 </private/var/containers/Bundle/Application/BA5EB36A-1B92-4C79-8430-BADCFA9EB7F6/CHIPTool.app> (executable, loaded)
Request       : Localizable type: stringsdict
Result        : None
2022-02-09 09:51:25.356006-0800  localhost CHIPTool[6690]: (CoreFoundation) [com.apple.CFBundle:strings] Hit last resort and creating empty strings table
2022-02-09 09:51:25.356123-0800  localhost CHIPTool[6690]: (CoreFoundation) [com.apple.defaults:User Defaults] found no value for key NSShowNonLocalizedStrings in CFPrefsSearchListSource<0x281711380> (Domain: com.chip.CHIPTool-Kean, Container: (null))
2022-02-09 09:51:25.356238-0800  localhost CHIPTool[6690]: (CoreFoundation) [com.apple.CFBundle:strings] Bundle: CFBundle 0x10cd04b90 </private/var/containers/Bundle/Application/BA5EB36A-1B92-4C79-8430-BADCFA9EB7F6/CHIPTool.app> (executable, loaded), key: Undefined error:%u., value: , table: Localizable, localizationName: (null), result: Undefined error:%u.
2022-02-09 09:51:25.356476-0800  localhost CHIPTool[6690]: Got pairing error back Error Domain=CHIPErrorDomain Code=1 "Undefined error:50."
To recover from this state, you need to force close the chip tool app

unable-to-start-pairing-on-chiptool.txt

still observed on today's SHA: 17dc4b003f2a433d2220cb3c22829d6336360e02 fairly frequently

still seeing this very frequently

not seeing anymore with iOS chiptool since changes went in to re-start the stack when opening QR code scanner


Inlay hints not showing up

My inlay hints stopped working for a while now and I don't know why, as all other rust-tools commands and lsp features seem to work fine. :RustSetInlayHints doesn't seem to do anything. Neither does calling :lua require('rust-tools.inlay_hints').set_inlay_hints() directly.
For the record, I use nvim-lsp-installer with the config inspired by #89 . Here is my config:
local global_opts = {
        on_attach = on_attach,
        settings = {
            ["rust-analyzer"] = {
                completion = {
                    postfix = {
                        enable = false
                    }
                },
                checkOnSave = {
                    command = "clippy"
                },
            }
        }
}

local rust_opts = {
    tools = {
        -- rust-tools options
        autoSetHints = true,
        hover_with_actions = true,
        inlay_hints = {
            only_current_line = false,
            show_parameter_hints = true,
        },
    },
    server = global_opts,
}

local lsp_installer = require("nvim-lsp-installer")

lsp_installer.on_server_ready(function(server)
    if server.name == "rust_analyzer" then
        local server_options = server:get_default_options()
        server_options['on_attach'] = on_attach
        rust_opts.server = vim.tbl_deep_extend("force",server_options, global_opts)
        require('rust-tools').setup(rust_opts)
        server:attach_buffers()
   elseif server.name == "volar" then
          local volar_options = server:get_default_options()
          volar_options["cmd"] = {"vls","--stdio"}
          volar_options["on_attach"] = on_attach
          volar_options["init_options"] = {
              serverPath = "~/.npm/lib/node_modules/typescript/lib/tsserver.js"
          }
          server:setup(volar_options)
   else
       server:setup(global_opts)
   end
end)

neovim version: v0.8.0-dev+150-g6c39f9d42 (nightly)

nvim-lsp-installer recently adjusted their setup call, so it no longer calls lspconfig's setup for you, instead you just call lsp-installer's setup function first, then start lsp servers with lspconfig like normal (or with rust-tools in this case)

Right here on nvim-lsp-installer's README
require("nvim-lsp-installer").setup {}
After calling this method you can just setup servers as you would normally.

I updated my config, still no change
require("nvim-lsp-installer").setup()
require("lspconfig")

local servers = {'sumneko_lua', 'tsserver', 'pyright', 'clangd', 'rust_analyzer', 'volar', 'jdtls'}

for _,lsp in pairs(servers) do
    if lsp == "rust_analyzer" then
        require('rust-tools').setup(rust_opts)
   elseif lsp == "volar" then
          local volar_options = vim.deepcopy(global_opts)
          volar_options["cmd"] = {"vls","--stdio"}
          volar_options["init_options"] = {
              serverPath = "~/.npm/lib/node_modules/typescript/lib/tsserver.js"
          }
          require("lspconfig")[lsp].setup(volar_options)
   else
          require("lspconfig")[lsp].setup(global_opts)
   end
end

Anything after this commit e29fb47326093fb197f17eae5ac689979a9ce191 did not work for me.

This seems to solve it.
Plug 'simrat39/rust-tools.nvim', { 'commit': 'e29fb47326093fb197f17eae5ac689979a9ce191' }

I wish there is a stable branch or version tags that I can rely on, idk im new to this whole vim plug in thing.

I had this issue as well but updated rust-analyzer with LspInstall and it works again.

I had this issue as well but updated rust-analyzer with LspInstall and it works again.

Confirmed. Updating rust-analyzer works for me (without changing anything)

I am also having the same problems. I was snooping around, it looks like the nvim-lsp-installer creator used to haev a page specifically for rust, but unfortunately they removed the wiki page (https://github.com/williamboman/nvim-lsp-installer/wiki/Rust)
For reference, I think this is the correct way to do things now with the nvim-lsp-installer, but again it produces in inlay_hints even with the latest rust-analyzer installed.
require('packer').startup(function(use)
    use 'simrat39/rust-tools.nvim'
    use 'neovim/nvim-lspconfig'
    use 'williamboman/nvim-lsp-installer'
end)

require('nvim-lsp-installer').setup({})

local rust_opts = {
    tools = {
        autoSetHints = true,
        inlay_hints = {
            only_current_line = false,
            show_parameter_hints = true,
        }
    },
}

for _, server in ipairs(require('nvim-lsp-installer').get_installed_servers())
do
    if server == 'rust_analyzer' then
        require('rust-tools').setup(rust_opts)
    else
        require('lspconfig')[server.name].setup({})
    end
end

I was having the same issue eventually got it inlay hints working by using the config @ephoris posted. However, now my lsp keymaps are getting wiped for rust files. I have these set in the on_attach method being passed to lspconfig's setup function. Anyone have any idea of why?

@nkhlmn solution worked for me. require('rust-tools').setup(...) behind lsp's setup(), passing on_attach and capabilities.

@nkhlmn solution worked for me. require('rust-tools').setup(...) behind lsp's setup(), passing on_attach and capabilities.

Please don't call lspconfig's setup as rust tools does it already

Closing since fixed


Missing boots: Angry Rain Boots

I'm trying to check out the stats on what I currently have, and I am currently using Angry Rain Boots obtained from Salmon Run. However, I can't seem to find them after mousing over each and every set of boots.

Missing gear has been addressed in commit 1989a7fe0e7f35769e074f877de2043b94099acc. Thank you for your patience.


Add developer documentation

Branch structure, squash merge.

This issue seems a duplicate of #481

This issue seems a duplicate of #481

Issue #481 is a duplicate of this issue 😜

Closing issue as #481 has been done and dev guide is in place.


Fix some colors

I think the interface is a bit too red now, we should keep some things blue (eg. links, or facebook logo on homepage).

Which should probably be the case, but Chrome render different colors for eg. links:
Chrome

Safari

Firefox


Handle content requests longer than<PHONE_NUMBER> bytes

There is a limitation in the ResourceHttpRequestHandler from Spring where the setHeader method limits the content length to Integer.MAX_VALUE<PHONE_NUMBER>) bytes.
	protected void setHeaders(HttpServletResponse response, Resource resource, MediaType mediaType) throws IOException {
		long length = resource.contentLength();
		if (length > Integer.MAX_VALUE) {
			throw new IOException("Resource content too long (beyond Integer.MAX_VALUE): " + resource);
		}
		response.setContentLength((int) length);
		...
	}

This looks to mostly be a limitation of the pre 3.1 servlet specification for setContentLength. People got around it by setting the Content-Length header directly as a string instead of an integer. Servlet specification 3.1+ has a setContentLengthLong API. Since Genie 3 uses Tomcat 8 embedded this API is available to us. As such if someone recompiles Genie as a war to run in Tomcat 7 or earlier this will break. We could patch to use If/Else at that point but I currently don't see issue going with Tomcat 8 as baseline as it is already mature and most people are moving up from 7 at this point.

Coverage decreased (-0.006%) to 80.877% when pulling 2c8e7b6296bd5ae417fab78ada3d84ef48fb8d27 on tgianos:develop into 17ee4e9f236c699369f1b52f664ad25f41d577a1 on Netflix:develop.


Rename dfs function

When I first read
df.select { dfsOf<Type> { it.size() > 10 } }

I thought it meant "Select dataframes of given type where their size is larger than 10".
Asking a friend of mine, they recognized "dfs" as "depth first search" but then they assumed it would return the deepest nested column, since a search is done when a result is found.
Plus, since this function traverses all column groups, it could just as well have been named bfs (breadth first search) and give the same result.
Clearly, this function needs renaming to avoid further confusion.
The functions that need to be renamed include dfs {}, dfsOf<> {} and allDfs(includeGroups: Boolean). Plus, in the internal parts of the library, the name DFS is also sometimes used to just mean a "recursive flattening" of columns, which is definitely not a "search".
Candidates for a rename include:
df.select { traversingGroups { it.size() > 10 } }
df.select { traversingGroupsColsOf<Type> { it.size() > 10 } } // ??
df.select { allTraversingGroups(includeGroups = false) } // ?

or
df.select { cols(traversingGroups = true) { it.size() > 10 } }
df.select { colsOf<Type>(traversingGroups = true) { it.size() > 10 } }
df.select { all(traversingGroups = false, includeGroups = true) } // ?

or
df.select { allLevels { it.size() > 10 } }
df.select { allLevelsOf<Type> { it.size() > 10 } } // ?
df.select { allLevels(includeGroups = false) } // ?

Other ideas are welcome :)

Plus many. I thought the same thing, dfs = dataframes so probably something to do with frame columns. And I was always thinking "if only there would be a way to get all the columns recursively from all the nested groups". I was doing some workarounds and planning to try to implement it at some point and make a PR.
Until I finally saw "depth first search" mentioned once somewhere in the docs at the bottom of the page.
But now that I know it, it is hard to beet the name dfs in conciseness. And I use it quite a lot, so every letter counts. Having traversingGroups everywhere would look awful and be a big bummer for me.
What would help is a dedicated page in the docs visible in the index tree or a more prominent section with the words "traverse" or "recursive". I would certainly remember that such functionality exists when reading the docs for the first time. But at that point dfs was mentioned only once in some table with other functions and I missed it.

deepCols etc. however makes the api less readable out loud which is one of the strengths of the Dataframe DSL.
I also came up with
select withinGroups (but that leaves out the top level), alsoInGroups, andInGroups, inGroupsToo, recursively.

And if possible
select { cols { it.size() > 10 }.recursively() }

What is the difference between select { all() } and select { cols() } ?
My point is that all or allCols() is the perfect name for dfs(). That what I would expect from allXXX functions, recursive behavior. Perhaps we could deprecate and re-purpose all. I know it is a breaking change, but that's why we are at 0.x.x
Then we can also have allGroups() in addition to allCols() etc.
I also find dfs() and allDfs() a little bit confusing. Why can't we just have includeGroups: Boolean parameter in dfs()?  Or maybe even just drop allDfs(). Currently if the user want to select with condition, they have to use dfs and include !it.isColumnGroup() in their predicate anyway.
This and if we rename allUntil/Before/After/Since will allow us to use all uniformly across the api to indicate recursive operations.

Working on it in https://github.com/Kotlin/dataframe/tree/dfs-rename
The basics work, along with @Deprecation annotations. However, some tests seem to fail due to changed orders.

How about going one level higher? I mean, we could have different top-level functions, like selectAll {} or deepSelect {} or even selectRecursively {}, although I really don't like the last one. The point is to change the meaning of the functions depending on the context, like select { all() } will select all top level but deepSelect { all() } or  select { recursive { all() } } to select all recursively. We can have a separate DSL for deep selections.
Just brainstorming, did not use it enough yet to really get an idea of what is better.
But I wanted to say that I do really like and fully embrace the hierarchical nature of kdf. So almost all of my columns are neatly organized in nested groups and I use recursive selectors most of the time. They do tend to get rather big though and .recursively(includeGroups = false) all over the place scares me a bit. It does look good as an API design, but I am trying to find a more concise way.
I am already used to dfs and it is very short. Do we need to remove it or can we keep it for "backwards compatibility" ;-) and have both?

Hmm, a level higher will probably be a bit too much, plus, since we have so many functions, hard to maintain.
We could probably also make an overload for .rec() or something short if it's used often, but probably dfs will have to go. It won't be immediate of course :) @Deprecated with level Warning and ReplaceWith etc.

Currently it is easy to write own recursive helper extensions, for example
fun <T> DataFrame<T>.deepSelectGroups(predicate: (ColumnGroup<*>) -> Boolean) =
    select { dfs { it.isColumnGroup() && predicate(it.asColumnGroup()) } }

I hope it will be the same easy with your new query wrappers when they are finished.

Maybe we should also consider https://github.com/Kotlin/dataframe/issues/270

I vote for .recursively() modificator that can be applied to any columns selector.
select { cols { it.size() > 10 }.recursively() }

One note for .recursively(): It changes cols {} from taking (AnyCol) -> Boolean to taking (ColumnWithPath<*>) -> Boolean aka ColumnFilter<*>. Otherwise, you won't be able to reason with depth etc.
This doesn't make much difference in the cols {} API, however it does change exact equality, making:
df.group { cols { it != name } }.into { "nameless" } -> df.group { cols { it.data != name } }.into { "nameless" }

also here parameter "dfs" https://kotlin.github.io/dataframe/reorder.html#reordercolumnsby

Reworking it from the ground up in https://github.com/Kotlin/dataframe/tree/dfs-rename2. This time I do modify ColumnSet to be able to transform the previous call in the chain before resolving the current.
Essentially allowing modifiers on ColumnSet returning calls, such as "colsOf().recursively()"

https://github.com/Kotlin/dataframe/pull/363

Merged :) While it might be a bit more wordy in some cases, at least at every step it's clear what you're doing IMO, especially with which types of columns are included and which aren't. Let me know what you think!
Currently busy reworking the ColumnsSelectionDsl further.


Python or Java interface for LevelDB?

Hi Level DB masters,
Are there any Python or Java interface for LevelDB?
Thanks in advance,
Lin

Forum is at https://groups.google.com/forum/#!forum/leveldb


Feature/add compagnons

Description

Added companion, linked to user, has a name, skin color and potential accessories
Jira Issue

https://etuduo.atlassian.net/browse/ETD-156
https://etuduo.atlassian.net/browse/ETD-158
Motivation and Context

This is to be used with the companion made on the front end
How Has This Been Tested?

Still Postman
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation accordingly.
[x] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Ne pas merge, il manque un commit


when using handle --use-euler-rotation Y-up flips to Z+ axis

@kcoley When using the --use-euler-rotation handle the Y-Up flips to Z+ axis. I've included the input GLTF for with and without euler handle.
using_euler_Y-up-Flips+Z.zip

@wave-electron I see. I need to find out what is causing the jittering to occur. It could be an issue with my interpolation code in Animation. As for the rotation, I saw an answer posted in the USD forums that might provide another workaround. Another thought would be to use animated transform matrices for rotation and see if that works per key frame


fix map layout id of underwater maps

fixes a mistake on my part from #439
back when i wrote it, i falsely assumed the route data entry numbers were map ids instead of layout ids

I'm not sure how to understand what the issue is here. Working with Emerald is difficult for me, as I do not play it enough to know what is or isn't correct.
From my understanding, we had the correct route data already.
Can you share some information to explain why its incorrect? Help me understand why this change is required?

at the time i wrote #439, i wasn't made aware of the fact that each map in the gen 3 games uses a layout id in their headers; many maps share layouts, such as generic houses, pokemon centers, and poke marts. looking at the route data now i realise that each route entry uses the layout id number (https://github.com/CyanSMP64/pokeemerald/blob/master/include/constants/layouts.h)
the underwater maps for routes 124, 126, 127, 128 follow a sequential order in the map groups (https://github.com/CyanSMP64/pokeemerald/blob/master/include/constants/map_groups.h) whereas the route 124 underwater layout has id number 274 while the others have 51-53.
this was correct until #439 was merged


[KYLIN-5090] isExactlyMatchSet should keep same size with cuboidIds

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Github Branch
As most of the development works are on Kylin 4, we need to switch it as main branch. Apache Kylin community changes the branch settings on Github since 2021-08-04 :

The original branch kylin-on-parquet-v2 for Kylin 4.X (Parquet Storage) has been renamed to branch main, and configured as the default branch;
The original branch master for Kylin 3.X (HBase Storage) has been renamed to branch kylin3 ;

Please check Intro to Kylin 4 architecture and INFRA-22166 if you are interested.
Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have create an issue on Kylin's jira, and have described the bug/feature there in detail
[x] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[x] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@5fa5a0b). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1790   +/-   ##
=======================================
  Coverage        ?   24.58%
  Complexity      ?     4430
=======================================
  Files           ?     1094
  Lines           ?    61856
  Branches        ?     8867
=======================================
  Hits            ?    15207
  Misses          ?    45040
  Partials        ?     1609

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5fa5a0b...b791129. Read the comment docs.

@hit-lacus @zhangayqian @zhengshengjun could you help take a look when you have time? Thanks.

LGTM


[Hub Generated] Public private branch 'dev-azure-kusto-Microsoft.Kusto-2021-08-27'

This PR moves the private branch that was created into public. The branch has already been reviewed, to view the status, please go to the OpenAPI Hub.

/azp run

/azp run


[WIP] [breaking] Pure rust implementation of the protocol

This is a mega-WIP-PR tracking the rust implementation of the protocol in wayland-rs. Once this is finished and landed, the native_lib flag will actually be meaningful.
General ideas of the implementation::

a lot of the message serialization logic is common to client and server, and will live in wayland-commons
from an API point of view, activating the native_lib feature will only add a few methods to access the C pointers of the underlying objects. As a result, any dependency that works without this flag should continue working with.
client-side, for now the egl and cursor features depend on native_lib. While the cursor feature could be implemented in pure rust, it is unlikely the egl feature could easily, unless some radical changes are accepted upstream in wayland-egl, regarding the backend API.
all tests must pass both with and without the native_lib feature (of course)

Progress:

[x] commons:

[x] message serialization and deserialization logic
[x] object map abstraction
[x] socket abstraction

[x] scanner:

[x] types and traits generation to encode the protocol
[x] API generation of the objects
[x] RequestsTrait generation for client side

[x] client:

[x] connection logic to find the server socket (must match the C libs rules wrt environment variables)
[x] event queue & message dispatching logic
[x] display implementation, delete_id and protocol errors
[x] blocking dispatch
[x] & prepare_read API : The prepare_read API is not essential for now, it still works while currently mocked though slightly suboptimal. This can be improved later and will not block this PR.

[x] server:

[x] general event loop & sources logic
[x] wayland socket listening logic
[x] wayland messages dispatching logic
[x] globals management logic
[x] completed API

Coverage increased (+1.4%) to 53.425% when pulling 22848eed48cee55ba5ce7773033ba68a2b3fbb75 on rust_impl into fc2164335cf584718469fc6d163f8bd4933d800c on master.

Well, here we go :tada:

This looks like a seriously impressive amount of work @vberger, epic stuff! Looking forward to having a closer look next time I get a chance. :heart:

You can expect a blog post about it soon-ish ;)


Rename userscript.js to Userscript.user.js

Userscript managers will detect these automatically when viewing the file raw

If doing this, rename it to something like arrpc.user.js instead of a generic name?

Agreed! 👍


✨ Place c ffi behind feature flag

This enables human-name to be compiled on wasm32-unknown-unknown
I didn't really check nothing but I don't think this should break anything.

Makes sense, thanks! And sorry for the slow response.


Host Lobby: Larger Player Cards

Currently, when the Host is in the Lobby, the players are displayed in a bulleted list. This works, but it does not look too good. Update the page so that all the names show up larger, and in a layout of cards. The MUI Paper element may be helpful for this.
It could end up looking something like this:

https://github.com/hto-projects/friendpardy/pull/55


Live lock in p2p-master-1.34.1

A possible live lock in cardano-node p2p-master-1.34.1.
The last block synced was https://adatools.io/blocks/7145191.
cardano-node is running at between 200-300 % CPU but it makes no progress and nothing is written to the logs.

We need more data to analyse this issue.  In the future we will be able to turn on logging tracer dynamically, which will help us analyse such problems.

The node kept syncing after a while.  We haven't reproduced the issue.


Errors with Docker Swarm Cluster

Hi,
I just followed the example but although the cluster is UP, the replica nodes connects correctly to the bootstrap node, I got this errors:
[warning] can't connect to cluster node<EMAIL_ADDRESS>due to timeout
for every replica node trying to connect to other replica nodes.
Here is my docker-compose:
`version: "3.7"
services:
vmq0:
image: erlio/docker-vernemq
environment:
DOCKER_VERNEMQ_SWARM: 1
vmq:
image: erlio/docker-vernemq
depends_on:
- vmq0
deploy:
replicas: 3
ports:
- "11883:1883"
- "18888:8888"
environment:
DOCKER_VERNEMQ_SWARM: 1
DOCKER_VERNEMQ_DISCOVERY_NODE: vmq0
DOCKER_VERNEMQ_ALLOW_ANONYMOUS: "on"`

Sounds like a networking/firewall issue and the cluster nodes are not able to establish connections between each other. Can you check if that's the case and perhaps try allowing everything between all cluster nodes just to see if that changes things.
Hope this helps.

Hi, all the three nodes are in the same LAN with all opened ports, is just a test lab. It seems the replica nodes listens on the wrong interface.

Ah, ok - then you probably need to specify on which interface to use. You can set the ip with DOCKER_IP_ADDRESS, I believe.

But I'm inside swarm and ip are dynamic

Right, you can also set the interface using DOCKER_NET_INTERFACE. See also https://github.com/vernemq/docker-vernemq/blob/master/bin/vernemq.sh#L3

It worked with this copose:
`
version: "3.7"
services:
vmq0:
image: vernemq/vernemq:1.9.1-alpine
environment:
DOCKER_VERNEMQ_SWARM: 1
vmq:
image: vernemq/vernemq:1.9.1-alpine
depends_on:
- vmq0
deploy:
replicas: 3
ports:
- "11883:1883"
- "18888:8888"
environment:
DOCKER_VERNEMQ_SWARM: 1
DOCKER_VERNEMQ_DISCOVERY_NODE: vmq0
DOCKER_VERNEMQ_ALLOW_ANONYMOUS: "on"
DOCKER_NET_INTERFACE: eth1
`

Great!

@webfrank Thanks! I had been trying to deploy a Vernemq cluster running in Docker Swarm using a custom overlay network with little success. The inclusion of the DOCKER_NET_INTERFACE: eth1 environment parameter was the missing piece of the puzzle.

@webfrank Thanks! , but still, I have an issue
my-vernemq-stack_vmq.2.r8sdetcwsx86@worker1    | 16:38:19.736 [info] successfully connected to cluster node<EMAIL_ADDRESS>my-vernemq-stack_vmq.2.r8sdetcwsx86@worker1    | 16:38:29.599 [warning] can't connect to cluster node<EMAIL_ADDRESS>due to timeout
my-vernemq-stack_vmq.2.r8sdetcwsx86@worker1    | 16:38:40.604 [warning] can't connect to cluster node<EMAIL_ADDRESS>due to timeout
my-vernemq-stack_vmq.2.r8sdetcwsx86@worker1    | 16:38:51.607 [warning] can't connect to cluster node<EMAIL_ADDRESS>due to timeout
my-vernemq-stack_vmq.2.r8sdetcwsx86@worker1    | 16:39:02.609 [warning] can't connect to cluster node<EMAIL_ADDRESS>due to timeout
my-vernemq-stack_vmq.2.r8sdetcwsx86@worker1    | 16:39:13.612 [warning] can't connect to cluster node<EMAIL_ADDRESS>due to timeout

it is working now with
version: "3.7"
services:
  vmq0:
    image: vernemq/vernemq
    environment:
      DOCKER_VERNEMQ_SWARM: 1
      DOCKER_VERNEMQ_ACCEPT_EULA: "yes"
      DOCKER_VERNEMQ_USER_QANAH: "password"
      DOCKER_NET_INTERFACE: eth0
  vmq:
    image: vernemq/vernemq
    depends_on:
      - vmq0
    environment:
      DOCKER_VERNEMQ_SWARM: 1
      DOCKER_VERNEMQ_DISCOVERY_NODE: vmq0
      DOCKER_VERNEMQ_ACCEPT_EULA: "yes"
      DOCKER_VERNEMQ_USER_QANAH: "password"
      DOCKER_NET_INTERFACE: eth1
    ports:
      - 18883:1883
      - 8888:8080
    deploy:
      replicas: 2

I modified the /usr/sbin/start_vernemq
Original (line 8 ~ 18)
# Ensure the Erlang node name is set correctly
if env | grep "DOCKER_VERNEMQ_NODENAME" -q; then
    sed -i.bak -r "s/-name<EMAIL_ADDRESS>VerneMQ@${DOCKER_VERNEMQ_NODENAME}/" /vernemq/etc/vm.args
else
    if [ -n "$DOCKER_VERNEMQ_SWARM" ]; then
        NODENAME=$(hostname -i)
        sed -i.bak -r "s/VerneMQ@.+/VerneMQ@${NODENAME}/" /etc/vernemq/vm.args
    else
        sed -i.bak -r "s/-name<EMAIL_ADDRESS>VerneMQ@${IP_ADDRESS}/" /vernemq/etc/vm.args
    fi
fi

Modified:
# Ensure the Erlang node name is set correctly
if env | grep "DOCKER_VERNEMQ_NODENAME" -q; then
    sed -i.bak -r "s/-name<EMAIL_ADDRESS>VerneMQ@${DOCKER_VERNEMQ_NODENAME}/" /vernemq/etc/vm.args
else
    if [ -n "$DOCKER_VERNEMQ_SWARM" ]; then
        IP_ADDRESS=$(hostname -i)
    fi
    sed -i.bak -r "s/-name<EMAIL_ADDRESS>VerneMQ@${IP_ADDRESS}/" /vernemq/etc/vm.args
fi

and all is fine.

How do you connect to an instance, once you deploy the stack using `docker stack deploy -c <docker-compose-file.yml>  my-vernemq-stack? I followed Qanahs configuration and the deployment was successful, each node joined cluster. But since I initialized a swarm, I cannot subscribe to any of the VerneMQ nodes.
I tried finding the IP Address / IPv4 address and connecting to 1883 port:
docker inspect <vernemq_node> | grep \"IPAddress\"
but I couldn't connect to that IP?
`


fix(cluster): fix internal validation failure

Fixes #101
tf plugin fails on internal validation:
resource scylladbcloud_cluster: All fields are ForceNew or Computed w/out Optional, Update is superfluous

Reason for it is that there are no fields that could be updated not triggering renew, which is correct, since siren have no api to update cluster.
But #88 left update-related fields in the cluster resource, which triggers tf validator to fail

I think I would be more comfortable if we just reverted some of the changes done in #88 instead of patching the codebase even further.
Namely I would revert this change:
diff --git a/internal/provider/cluster.go b/internal/provider/cluster.go
index 91ee52b..0388f47 100644
--- a/internal/provider/cluster.go
+++ b/internal/provider/cluster.go
@@ -135,8 +135,6 @@ func ResourceCluster() *schema.Resource {
 				Description: "Whether to enable CNAME for seed nodes",
 				Optional:    true,
 				Type:        schema.TypeBool,
-				// NOTE(rjeczalik): ForceNew is commented out here, otherwise
-				// internal provider validate fails due to all the attrs
 				// being ForceNew; Scylla Cloud API does not allow for
 				// updating existing clusters, thus update the implementation
 				// always returns a non-nil error.

And this:
diff --git a/internal/provider/cluster.go b/internal/provider/cluster.go
index 0388f47..1ccc513 100644
--- a/internal/provider/cluster.go
+++ b/internal/provider/cluster.go
@@ -138,8 +138,8 @@ func ResourceCluster() *schema.Resource {
 				// being ForceNew; Scylla Cloud API does not allow for
 				// updating existing clusters, thus update the implementation
 				// always returns a non-nil error.
-				// ForceNew:    true,
-				Default: true,
+				ForceNew: true,
+				Default:  true,
 			},
 			"request_id": {
 				Description: "Cluster creation request ID",

Currently Update is not implemented for the Cluster resource and I would want to make that explicit via those methods. If someone adds new attribute which is not ForceNew suddenly everything will break again. That's why I left one parameter which was not ForceNew, to not leave a "trap" in the code for the next person that modifies it.
So let's bring back previous setup and let's not modify things that are known to be working for no reason.

I don't see these commits on the main anymore, did you delete them ?

Currently Update is not implemented for the Cluster resource and I would want to make that explicit via those methods. If someone adds new attribute which is not ForceNew suddenly everything will break again. That's why I left one parameter which was not ForceNew, to not leave a "trap" in the code for the next person that modifies it.

tbh, I don't understand rational that, having no update method tells you exactly the same.
If we want to make sure that it works, let's have a test on CICD.
creating runtime panic to ease development is not a way out.

@rjeczalik , I don't agree with it, but I changed the code accordingly, please take a look


Warn the user of explicit use of any

I wrote an explicit any in a project, which uses the default unbrace eslint config and didn't get a warning

#4

PR #4 marks this as an error.


Extract channel graph reliability estimation into Mission Control probability machine

Background:
Currently reliability info is used during path finding in an all or nothing fashion. Either an edge can be used in the path or it is skipped completely. We think routing behaviour can improve by making this decision more nuanced.
Proposal
The proposal is to use not only fee and time lock information during pathfinding, but also a gradual measure of probability that indicates how likely it is that a candidate hop will be successful during execution of the payment. There will be a trade-off. It could be that a more reliable path is chosen even though it is not the lowest fee path.
Implementation
The source of these probabilities is the so called "probability machine". The probability machine will serve as the starting point for the next version of Mission Control. The concern of the probability machine is to only keep track of the set of nodes and edges in the graph with probabilities for successful routing through them, that can be consulted during path finding.
We also plan to remove the distinction between payment-level and global-level reliability data. There will be one global probability machine that provides state to all (concurrent) payments being made.
NOTE: at first we aim to keep the current Mission Control behavior, with focus on abstracting the Mission Control/PaymentSession logic into its own independent module, that can be used by the routing algorithm. This means that the exposed probabilities will initially be 0% and 100%. Improvements to make routing “smarter” can later be achieve by improving the probability machine, and we can also make the weighting of fees vs reliability tunable during path finding.
Steps to completion:

[ ] Define a new Mission Control interface, with methods for querying for routing success probabilities, and for reporting new information about the graph (routing failures/successes, graph updates etc)
[ ] Make the current MissionControl/PaymentSession implementation adhere to this interface, with pruned edges/nodes having reported success probability of 0.
[ ] Make the path finding algorithm consult these probabilities during path finding, in addition to fees, striking a configurable tradeoff between the two. For now we’ll aim for the resulting routes to be more or less equivalent to what is chosen by the current implementation, where reliability info currently is leading.

Fixed by https://github.com/lightningnetwork/lnd/pull/2802


Proposal: Configuration block directives for loading tailscale configuration

I've seen a few comments in some PRs or issues about switching to a configuration block for loading tailscale configuration. Since it's come up multiple times I figured a GitHub issue would help track potential approaches. I had some thoughts and will include them here but we can always drop it if you've already a potential approach
Global Configuration
Anything that needs to be global across all tailscale servers would end up in a global configuration block. This lets the plugin also implement setup and teardown functionality (but only when the global options are specified). Anything that sets up a bare listener without specifying any tailscale block would not get the plugin teardown called on it.
Some examples could be:

whether servers should be ephemeral
External storage for tailscale server state
a global auth key

Per server/listener configuration
The bigger question that I might have is where to store per listener configuration (such as auth keys) particularly when http servers declare conflicting configuration. My thought is to include individual server configuration as individual blocks inside the global configuration.
The final configuration would look something like this:
tailscale {
    authkey = "TS_AUTHKEY_global"
    state {
      kubestore "secret_name"
    }
    a { # this matches tailscale/a
      authkey = "TS_AUTHKEY"
      ephemeral = true
    }
}

:80 {
  bind tailscale/a
}

:80 {
  bind tailscale/b  #tailscale/b inherits global configuration
}

Please let me know your thoughts

I don't love having to separate the config (into the global section) from where it's used, but I think this is probably the best we can do.  However, I do like that this can make it a lot clearer how a single tsnet instance can be used as a listener, a transport, and/or an auth handler.  Configure it once globally, and then reference it where needed.  So that's actually really nice.

Configure it once globally, and then reference it where needed.
That's what I was thinking as well. It ensures that there can't be conflicting configuration on the same tsnet instance as well.

If there's a big desire to keep configuration where it's used, the method proposed here could be the way to go. Then when configuring the tsnet instances we'd want to throw an error if we run into any conflicts (e.g. on control url, ephemeral state, storage, etc)
The tsnet instance configuration would match between each Module + implementation specific configuration. That might look something like this
:80 {
  listen_wrappers {
    tailscale {
       host = "a"
       authkey = "TS_AUTHKEY"
       control_url = "https://example.com"
    }
  }
}

:90 {
    reverse_proxy / {
      to "ts-backend:5000"
      transport "tailscale" {
        host = "a"  # You probably need a host to do a transport?
        authkey = "TS_AUTHKEY"
        control_url = "https://c.tailscale.com"  # oh no! a conflict. Log a warning and use the first one? Crash? Create a new tsnet instance with different state (a-1)?
      }
    }
}

Like you said, everything is used where it's defined. My personal opinion is that when we identify a shared piece of data that can be redefined in multiple places it's better to configure them separately than redefine them multiple times. Conflicts can create user confusion (though cleared up through documentation) and less clean behaviors on those conflicts depending on decided on behavior.

yeah, I definitely agree... when we are reusing a single tsnet instance, it totally make sense to separate it to avoid having to interpret conflicts.
I suppose the global config would basically provision a caddy app that would maintain the usage pool of tsnet instances, configuring appropriately as needed.  I've never created a caddy app like that, so will need to look at some examples.  But does that sound like the right approach here, @mholt ?

Yeah, if you want to centralize configuration that a lot of handlers might use, you could use a Caddy app and stick it in the global options of the Caddyfile.
There are also some config primitives you can take advantage of depending on your needs:

Remember that the native config format is JSON, and you can use that to elegantly define variables that are used for all the HTTP routes, even routes for different sites, on the same server (server = set of listeners).
Environment variables are supported placeholders in a lot of Caddy config. Adding support for placeholders in a plugin like this is very easy (1 line to get or make the Replacer, then one line to call Replace).

oh no! a conflict. Log a warning and use the first one? Crash? Create a new tsnet instance with different state (a-1)?

This kind of thing is why we tend to avoid global config in Caddy, even the global config we do have is riddled with bugs here and there. Just something to keep in mind.

This kind of thing is why we tend to avoid global config in Caddy, even the global config we do have is riddled with bugs here and there. Just something to keep in mind.

In my mind the conflicts would mostly show up in the listen_wrapper case and the limited configuration surface on the Network module would prevent conflicts (beyond someone just creating the same server twice globally)
Does your experience say to go with a listen_wrapper instead? If I understand correctly, you would need to define each wrapper for each port you want caddy to bind to? Wouldn't that open more configuration surface and more possibilities for conflicts?

I've created an initial PR to support using the Caddy configuration file to setup the tailscale plugin here #30. Based on the initial response here I can extend it to support more configuration or change the approach.

Closed via #30


vmware controller: error message in email could be improved

[Version]
V16.0 Fix 6
[Date]
2016/06/26
[Test category]
Creative
[Description]
call action on a instance where the back end is already deleted
[Steps to reproduce]
To simulate errors:
subscribe to vmware image
delete the running instance on the backend (ess) via vShere client
call on CT MG side any action i.e.: Restart of virtual machine
Error is thrown (correct) but the wording in the email is 'very basic':
Dear user,
The current operation on subscription 'VMware_Original_With_ALL' for customer 'd5b8f512' has been suspended (Instance: '1467730620987').
Reported problem: Failed to check VM run state of xy. OSCM-Instance-ID:<PHONE_NUMBER>987
To continue automatic processing, please click the following URL:
https://Time.DarkSide.com:28881/oscm-app/notify?sid=1467730620987&cid=ess.vmware&_resume=yes
"Failed to check VM run state" is not very (lets say) elegant. The corresponding error in the log file states:
[2016-07-05 17:14:19] [__ejb-thread-pool9] ERROR Actions checkVMRunning Failed to check VM running state of instance<PHONE_NUMBER>987
java.lang.Exception: Failed to retrieve information of VM xy
Failed to check VM running state of instance.... is better then the message in the email
[Expected result]
the error message in the email should contain pain correct English
[Observed result]
Failed to check VM run state of xy
[Details]
See above;
[Messages and logs]

The error message for email is changed according to the comment.

ok with 2016/07/22


Create ID's for Header elements so they can be referenced in anchor tags

Credit for these changes goes to the work done in #765 by @netniV and @Ayesh.
This PR rephrases the changes made there toward the 2.0.x branch.

Thank you, this is nice and looking forward to the 2.x releases. I also think it's better to make the slug function configurable. This library ideally should come from a DI container, and it makes sense that we make the slug function customizable with a sane default.

If I'd suggest one thing if I may, I think the regex would be better with p{Nd}\p{Nl} instead of p{N}.
\p{N} includes superscript numbers and divisions, which ideally should not make to a URL, and neither does in GFM.
So instead of $slug = \preg_replace('/[^\p{L}\p{N}\p{M}-]+/u', '', $slug);, we could use $slug = \preg_replace('/[^\p{L}\p{Nd}\p{Nl}\p{M}-]+/u', '', $slug);. This converts 测试 x² 标题 to 测试-x-标题 (notice the ² gone in slug).

So instead of $slug = \preg_replace('/[^\p{L}\p{N}\p{M}-]+/u', '', $slug);, we could use $slug = \preg_replace('/[^\p{L}\p{Nd}\p{Nl}\p{M}-]+/u', '', $slug);.

Done, 99dd44e :)

@netniV RE your comment

Looks good. I added one comment as per our discussions here over str_replace. More involved is your change but I was thinking it should be an optional addition as it could affect other elements due to a duplication of ID
The only other thought is with trimming a trailing and leading hyphen as that seems wrong to keep those to me.

Trimming the leading and trailing hyphens seems sensible, I think I can also deal with the ID duplication too—GitHub seems to resolve this by appending -n to duplicated IDs (where n is an increasing counter per duplication), and I think this is a sensible approach to use. I'll probably generalise it a bit so that custom handling of de-duplication is possible (e.g. maybe you want to use a prefix, or a different separator).

It’s amazing how few lines of code can balloon ;)
Work is looking good.

It’s amazing how few lines of code can balloon ;)

Indeed 😉
I moved some of the irrelevant test fixes out of this PR and have now rebased, so it is looking a little more focused now. Still at ~350 additions though :)
Fortunately for anyone using these changes, they'll be able to modify slug behaviours (if they wish) with essentially a one-liner—which is quite nice :)

I moved the MutableConfigurable changes into a separate PR (#768) and have rebased this a final time (so now only the changes directly related to adding the slug are in this PR :) )


Can I make index with my ttl data?

Hi, I'm new to qlever, and I have few questions to index my ttl data.
In my lab, we build about 2 billion triples related to Korean address, and we want to test qlever performances for querying.
Questions are below:

I have 196 turtle files (about 90G, not compressed), and How can I index the turtle files? As far as I understand, qlever get-data downloads data from an external source, but which function I don't need and all files are in local. Do I need to compress in one ttl.gz file?
This is my Qleverfile for my data, but it doesn't work. I want to index all .ttl files in the INPUT_FILES path, and how can I write other settings?

# Qleverfile for hike, use with https://github.com/ad-freiburg/qlever-control
#
# qlever index
# qlever start

[data]
NAME         = hike
DESCRIPTION  = hike address data

[index]
INPUT_FILES     = /home/hike/qlever/qlever-indices/address/*.ttl
SETTINGS_JSON   = { "ascii-prefixes-only": true, "num-triples-per-batch": 10000000, "parallel-parsing" : true", "locale": { "language": "ko", "country": "KR", "ignore-punctuation": true } }
STXXL_MEMORY    = 10G

[server]
PORT               = 7001
ACCESS_TOKEN       = ${data:NAME}
MEMORY_FOR_QUERIES = 20G
CACHE_MAX_SIZE     = 10G

[runtime]
SYSTEM = docker
IMAGE  = docker.io/adfreiburg/qlever:latest

[ui]
UI_CONFIG = hike

It would be so pleasure if you answer !

You can use qlever get-data but it's not required. You can also just download and prepare the turtle files manually. To use this feature specify a shell command in GET_DATA_CMD in the [data] section. This command should then download and unzip the files into the current working directory. This enables other people who also use the Qleverfile to download the data easily.

You don't need to generate one complete turtle file or even a single archive. You just have to ensure that the files you want to index are present in the directory of the Qleverfile (either using qlever get-data or manually) and then specify them in INPUT_FILES. Because you are running QLever in docker, the working directory is mounted as /index in the container. So you have to adjust the paths, e.g. to *.ttl.
Have a look at the example Qleverfiles. Have a look at the options of the individual commands with --help. Almost all of the options correspond to an option in the Qleverfile, though I am afraid that the names in the config files are not really documented. Finally all the options are defined in code here.

Please also provide the concrete error messages in the future.

Thanks for your response. I successfully index my ttl data, but I got a new error when I run qlever start.
After qlever index, I got {host}.index-log.txt, {host}.settings.json, {host}.tmp.partial-vocabulary.*,{host}.unsorted-triples.dat files. After running qlever start, I got an error below:
Command: start

docker run -d --restart=unless-stopped -u $(id -u):$(id -g) -v /etc/localtime:/etc/localtime:ro -v $(pwd):/index -p 7001:7001 -w /index --init --entrypoint bash --name qlever.server.hike docker.io/adfreiburg/qlever:latest -c 'ServerMain -i hike -j 8 -p 7001 -m 20G -c 10G -e 1G -k 200 -s 30s -a hike_v78bB7fQfEvl > hike.server-log.txt 2>&1'

Follow hike.server-log.txt until the server is ready (Ctrl-C stops following the log, but not the server)

2024-05-30 06:13:44.922 - INFO: QLever Server, compiled on Thu May 23 14:26:32 UTC 2024 using git hash f9c313
2024-05-30 06:13:44.923 - INFO: Initializing server ...
2024-05-30 06:13:44.923 - ERROR: Could not open file "hike.meta-data.json" for reading. Possible causes: The file does not exist or the permissions are insufficient. The absolute path is "/index/hike.meta-data.json".

The error message said I have no hike.meta-data.json.
hike.meta-data.json file was not generated automatically, after qlever index.

The index step didn't complete successfully. {host}.tmp.partial-vocabulary.* and {host}.unsorted-triples.dat are intermediate files created during indexing and deleted afterwards. In the end you should have {host}.index.*, {host}.meta-data.json and {host}.vocabulary.* files. Try re-running the index step (qlever index --overwrite-existing) and posting the index-log if that does not help.

I reindex my test file to run qlever index, and it seems that all index process doesn't work well.
My index-log is below (after qlever index and never forced stopped):
2024-05-31 06:05:32.274 - INFO: QLever IndexBuilder, compiled on Thu May 23 14:26:32 UTC 2024 using git hash f9c313
2024-05-31 06:05:32.274 - INFO: You specified the input format: TTL
2024-05-31 06:05:32.274 - INFO: Processing input triples from /dev/stdin ...
2024-05-31 06:05:32.274 - INFO: Locale was not specified in settings file, default is en_US
2024-05-31 06:05:32.274 - INFO: You specified "locale = en_US" and "ignore-punctuation = 0"
2024-05-31 06:05:32.274 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files
2024-05-31 06:05:32.274 - INFO: You specified "parallel-parsing = true", which enables faster parsing for TTL files with a well-behaved use of newlines
2024-05-31 06:05:32.274 - INFO: You specified "num-triples-per-batch = 10,000,000", choose a lower value if the index builder runs out of memory
2024-05-31 06:05:32.274 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2024-05-31 06:05:32.327 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2024-05-31 06:06:19.858 - INFO: Triples parsed: 7,905,220 [average speed 0.2 M/s]
2024-05-31 06:06:38.983 - INFO: Number of triples created (including QLever-internal ones): 7,905,220 [may contain duplicates]
2024-05-31 06:06:38.983 - INFO: Merging partial vocabularies ...

It has not been progressed since then. As you can see below, the tmp files are not removed, and {host}.meta-data.json, {host}.index.* files are not generated. I didn't forced terminate the process, but qlever status said there is no processes.
(A test file is datahub-full-rna-*.ttl.)

Thanks for your kindness answers 😺

The progress messages (Triples parsed ...) are printed every 10 Mio triples. Your dataset is smaller, so no such message will be printed. At this point I am out of obvious ideas for the problems cause.

Can you try to index the Olympics dataset and report how long it takes? qlever setup-config olympics generates a Qleverfile for the dataset. On my laptop it takes about 3 seconds to index the 1.7 Mio triples running in docker.
Is the dataset public?

Jumping in here because I'm trying to do something similar. How do you make qlever index use your own Qleverfile?

@givemetarte Can you provide a link to your data, so that we can check if we can reproduce the error?
You can also try the following yourself, if you are sufficiently computer-savvy: When calling cmake, change -DCMAKE_BUILD_TYPE=Release to -DCMAKE_BUILD_TYPE=RelWithDebInfo (probably sufficient) or -DCMAKE_BUILD_TYPE=Debug (will be significantly slower). Either make that change in the Dockerfile and rebuild the image, or when you compile the binaries natively. Then run the index build in the debugger. When it comes to the point, where the execution hangs, check in which part of the code you are and report back to us.

@givemetarte Here is a simpler thing which you can try first. In the Qleverfile, in the value for SETTINGS_JSON, replace "parallel-parsing" : true by "parallel-parsing" : false. Also not the stray " you had after the true, but maybe that was just a copy&paste error on your part.

@Qup42 I tried to test the Olympic datasets, but I encountered the same issue. qlever setup-config, qlever get-data runs well, but the index process ended up with no {host}.index.* and {host}.meta-data.json.
My server spec is Ubuntu 20.04, 64G memory, 2TB hard disk, which seems to enough for running qlever.

I tested the same indexing work in my local laptop (MacOS), I got the same issue when I indexed the Olympic datasets.

@hannahbast I changed SETTINGS_JSON config ("parallel-parsing" : false), but there is no change.
I tested the Olympic datasets on both Ubuntu and MacOs, the indexing did not well done. There is no host.index* files and host.meta-data.json.

When I do the same work with the Olympic datasets, I got the same issue (cannot indexing completely).
I have no idea how to debug with cmake. I tried but I don't know where I need to change the options, and when I git clone qlever-control, there is no CMakeLists.txt...
You can download my data via here. This link is running temporarily, so the link will be gone in a week. The final Qleverfile is below:
# Qleverfile for hike, use with https://github.com/ad-freiburg/qlever-control
#
# qlever index
# qlever start

[data]
NAME         = hike
DESCRIPTION  = hike address data

[index]
INPUT_FILES     = *.ttl
CAT_INPUT_FILES = cat ${INPUT_FILES}
SETTINGS_JSON   = { "ascii-prefixes-only": true, "num-triples-per-batch": 10000000, "parallel-parsing" : false}

[server]
PORT               = 7001
ACCESS_TOKEN       = ${data:NAME}_VEjVfajs2n1C
MEMORY_FOR_QUERIES = 20G
CACHE_MAX_SIZE     = 10G

[runtime]
SYSTEM = docker
IMAGE  = docker.io/adfreiburg/qlever:latest

[ui]
UI_CONFIG = hike

@givemetarte I just tried it with your dataset and your Qleverfile and it works without problems. It is also very unusual that the olympics dataset does not work. So there must be something unusual about your machine and we have to find out what it is.
Is there anything that comes to your mind?

@tomersagi Why not just call your Qleverfile Qleverfile? Then you don't need that option.

That's great. Would be nice if that information was in the README or anywhere.

@hannahbast Thanks for testing!
I've tried running qlever indexing on three different servers. 2 servers (Ubuntu, and MacOS mentioned before) did not work, but the other ubuntu server worked well (I tested right after your answer).
I have no idea about the difference between a server where qlever works and a server where it doesn't.
Python version is the same (3.8.10)... Anyway, thanks to this, I have at least one server qlever is working.

@givemetarte Thanks for the update. That is really strange and it would be great to find out what the problem is. Can you provide information about each of the two machines with Ubuntu: which Ubuntu version is it, how much RAM does the respective machine have, and which version of Docker is installed on the respective machine?

Ubuntu servers' spec is below:

Ubuntu 20.04, RAM 16G+32G, 2TB disk storage, docker version 20.10.22 (which qlever is working)
Ubuntu 20.04, RAM 64G, 2TB disk storage, docker version 24.0.2 (which qlever is not working)

I'll downgrade the docker version and try running qlever again.

@hannahbast I changed SETTINGS_JSON config ("parallel-parsing" : false), but there is no change. I tested the Olympic datasets on both Ubuntu and MacOs, the indexing did not well done. There is no host.index* files and host.meta-data.json. ```
Hi,
For me what seems to have worked is to decrease the num-triples-per-batch. After reducing this to 100K the indexer worked. I guess there might be an unreported out of memory issue? Because the output suggests choose a lower value if the index builder runs out of memory and this fixed the problem, but there was no out of memory reported.

Thanks for your answers. @dssib
I reset num-triples-per-batch to 100000, and reindexed my data (about 9,450 triples).
Something different, but I got a new error.
Command: index

echo '{ "ascii-prefixes-only": true, "num-triples-per-batch": 100000, "parallel-parsing" : false}' > hike.settings.json
docker run --rm -u $(id -u):$(id -g) -v /etc/localtime:/etc/localtime:ro -v $(pwd):/index -w /index --init --entrypoint bash --name qlever.index.hike docker.io/adfreiburg/qlever:latest -c 'cat *.ttl | IndexBuilderMain -F ttl -f - -i hike -s hike.settings.json --stxxl-memory 5G | tee hike.index-log.txt'

2024-08-12 18:45:06.811 - INFO: QLever IndexBuilder, compiled on Thu May 23 14:56:04 UTC 2024 using git hash f9c313
2024-08-12 18:45:06.826 - INFO: You specified the input format: TTL
2024-08-12 18:45:06.826 - INFO: Processing input triples from /dev/stdin ...
2024-08-12 18:45:06.838 - INFO: Locale was not specified in settings file, default is en_US
2024-08-12 18:45:06.838 - INFO: You specified "locale = en_US" and "ignore-punctuation = 0"
2024-08-12 18:45:06.838 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files
2024-08-12 18:45:06.838 - INFO: You specified "num-triples-per-batch = 100,000", choose a lower value if the index builder runs out of memory
2024-08-12 18:45:06.838 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2024-08-12 18:45:07.158 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2024-08-12 18:53:59.097 - INFO: Triples parsed: 94,589,734 [average speed 0.2 M/s, last batch 0.2 M/s, fastest 0.2 M/s, slowest 0.1 M/s]
2024-08-12 18:53:59.270 - INFO: Number of triples created (including QLever-internal ones): 94,589,734 [may contain duplicates]
2024-08-12 18:53:59.271 - INFO: Merging partial vocabularies ...
2024-08-12 18:53:59.282 - INFO: Finished writing compressed internal vocabulary, size = 0 B [uncompressed = 0 B, ratio = 100%]
2024-08-12 18:53:59.762 - ERROR: ! ERROR opening file "hike.vocabulary.external.words" with mode "w" (No such file or directory)

tmp files are not deleted, and also hike.vocabulary.external.words file is not created. Also, there is no docker error (in docker logs).
However, when I reindexed with fewer triples (about 12,300,000 triples), it worked well.
I think the merging partial vocabularies seems to be not working properly, with a huge increase in memory usage.

I think it's indeed due to a memory error. On my side to avoid this I ended up merging the source .ttl files by hand first (I had a few hundreds of them), this will avoid having to merge vocabularies after the indexing. You could try this if that's also an option on your side. But perhaps the developers have also some suggestions, this one is based only on my trials and errors (in the end I did succeed to get everything indexed and running).

@dssib Thanks for sharing your trials and errors. Unfortunately, Merging all ttl files is not worked for me. I still got the same error, and I'll try to run qlever in another machine (much higher memory).
@hannahbast Could you recommend proper machine memory per triples? What is the minimum required to create an index on about 90 million triples?

I changed Qleverfile options like wikidata.
I run qlever index in mac (m1, 16G memory, 500GB), and the indexing was completed in about 25 mins.
@dssib You could try setting STXXL_MEMORY for better indexing. Thanks for your all comments!
# Qleverfile for hike, use with https://github.com/ad-freiburg/qlever-control
#
# qlever index
# qlever start

[data]
NAME         = hike
DESCRIPTION  = hike address data

[index]
INPUT_FILES     = *.ttl
CAT_INPUT_FILES = cat ${INPUT_FILES}
SETTINGS_JSON   = { "languages-internal": [], "prefixes-external": [""], "locale": { "language": "en", "country": "US", "ignore-punctuation": true }, "ascii-prefixes-only": true, "num-triples-per-batch": 500000, "parallel-parsing" : false}
STXXL_MEMORY    = 10G

[server]
PORT               = 7001
ACCESS_TOKEN       = ${data:NAME}_VEjVfajs2n1C
MEMORY_FOR_QUERIES = 10G
CACHE_MAX_SIZE     = 6G

[runtime]
SYSTEM = docker
IMAGE  = docker.io/adfreiburg/qlever:latest

[ui]
UI_CONFIG = hike

Command: index-stats

Breakdown of the time used for building the index, based on the timestamps for key lines in "hike.index-log.txt"

Parse input           :   10.5 min
Build vocabularies    :   11.0 min
Convert to global IDs :    0.8 min
Permutation SPO & SOP :    0.5 min
Permutation OSP & OPS :    1.0 min
Permutation PSO & POS :    0.8 min

TOTAL time            :   24.6 min

Breakdown of the space used for building the index

Files index.*         :    1.0 GB
Files vocabulary.*    :    1.2 GB

TOTAL size            :    2.2 GB


Fix st2ctl status for st2chatops

Addresses #2779

As @LindsayHill noted, one of the fast fixes is to use hubot instead of st2chatops for pid discovery.
$ st2ctl status
##### st2 components status #####
st2actionrunner PID: 929
st2api PID: 843
st2api PID: 1192
st2stream PID: 846
st2stream PID: 1193
st2auth PID: 832
st2auth PID: 1191
st2garbagecollector PID: 830
st2notifier PID: 835
st2resultstracker PID: 833
st2rulesengine PID: 838
st2sensorcontainer PID: 824
hubot PID: 9907
mistral-server is not running.
mistral-api is not running.

Good catch @LindsayHill and thanks @armab!

On a related note - we also need to add some tests for this to st2tests and or robotfm tests suite.
/cc @emedvedev

Btw, maybe this was done (semi) intentionally? Do we support installations without ChatOps / hubot?
If we do, we also need to handle that case as well - make sure st2ctl doesn't fail if hubot is intentionally not installed. IIRC, that was the scenario we indeed supported at some point, but not 100% sure right now.
/cc @lakshmi-kannan @DoriftoShoes @emedvedev

make sure st2ctl doesn't fail if hubot is intentionally not installed

It doesn't fail, but it does give an error that can be confusing. We had this covered, but apparently that code wasn't working for RHEL6 (IIRC), so it would make sense to rewrite it and include the fix into st2ctl again.
I'm closing this in favor of https://github.com/StackStorm/st2chatops/pull/45 which is somewhat hackier but better from the UX perspective.


Assertion failures prevent reorganizing chain

On 2 nodes in different places I'm seeing the same error repeated: the node attempts to reorganize the chain and fails with a Number exceeds 2^53-1 assertion error.
Both systems run Ubuntu Server 17.04.
Sometimes they run for a while without attempting a reorganization. Other times they try repeatedly in rapid succession with the same error every time.
Pasting a sequence of 2 of these errors here:
[warning] WARNING: Reorganizing chain.
[error] Number exceeds 2^53-1
{ AssertionError [ERR_ASSERTION]: Number exceeds 2^53-1
    at Object._read64 (/home/berry/sft/land-bronze/bronzeage-node/lib/utils/encoding.js:161:3)
    at Object.read64 (/home/berry/sft/land-bronze/bronzeage-node/lib/utils/encoding.js:175:19)
    at BufferReader.read64 (/home/berry/sft/land-bronze/bronzeage-node/lib/utils/reader.js:332:18)
    at Output.fromReader (/home/berry/sft/land-bronze/bronzeage-node/lib/primitives/output.js:259:15)
    at Object.decompressOutput [as output] (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/compress.js:74:10)
    at UndoCoin.fromReader (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:298:14)
    at Function.fromReader (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:321:25)
    at UndoCoins.fromRaw (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:100:30)
    at Function.fromRaw (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:112:26)
    at ChainDB.getUndoCoins (/home/berry/sft/land-bronze/bronzeage-node/lib/blockchain/chaindb.js:973:20)
  generatedMessage: false,
  name: 'AssertionError [ERR_ASSERTION]',
  code: 'ERR_ASSERTION',
  actual: false,
  expected: true,
  operator: '==' }
[info] Connected to <IP_ADDRESS>:2301.
[info] Attempting BIP151 handshake (<IP_ADDRESS>:2301).
[info] Received version (<IP_ADDRESS>:2301): version=70014 height=1426 services=111 agent=/decentraland:1.0.0-alpha/
[info] Received 1 addrs (hosts=21, peers=5) (<IP_ADDRESS>:2301).
[debug] Refilling peers (5/8).
[debug] Connecting to <IP_ADDRESS>:2301.
[debug] Connecting to <IP_ADDRESS>:2301.
[debug] Connecting to <IP_ADDRESS>:2301.
[debug] Sending 20 addrs to peer (<IP_ADDRESS>:2301)
[debug] Received verack (<IP_ADDRESS>:2301).
[info] Initializing compact blocks (<IP_ADDRESS>:2301).
[debug] Requesting headers packet from peer with getheaders (<IP_ADDRESS>:2301).
[debug] Height: 1303, Hash:<PHONE_NUMBER>3fcc90092dc5588c6e64192d04b0470248ee0ead4560e99f0143d8, Stop: null
[info] Connected to <IP_ADDRESS>:2301.
[info] Attempting BIP151 handshake (<IP_ADDRESS>:2301).
[debug] Received headers packet with 123 items (<IP_ADDRESS>:2301).
[debug] Received 123 headers from peer (<IP_ADDRESS>:2301).
[debug] Requesting 1/1 blocks from peer with getdata (<IP_ADDRESS>:2301).
[debug] Refilling peers (6/8).
[debug] Connecting to <IP_ADDRESS>:2301.
[debug] Connecting to <IP_ADDRESS>:2301.
[info] Connected to <IP_ADDRESS>:2301.
[info] Attempting BIP151 handshake (<IP_ADDRESS>:2301).
[warning] WARNING: Reorganizing chain.
[error] Number exceeds 2^53-1
{ AssertionError [ERR_ASSERTION]: Number exceeds 2^53-1
    at Object._read64 (/home/berry/sft/land-bronze/bronzeage-node/lib/utils/encoding.js:161:3)
    at Object.read64 (/home/berry/sft/land-bronze/bronzeage-node/lib/utils/encoding.js:175:19)
    at BufferReader.read64 (/home/berry/sft/land-bronze/bronzeage-node/lib/utils/reader.js:332:18)
    at Output.fromReader (/home/berry/sft/land-bronze/bronzeage-node/lib/primitives/output.js:259:15)
    at Object.decompressOutput [as output] (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/compress.js:74:10)
    at UndoCoin.fromReader (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:298:14)
    at Function.fromReader (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:321:25)
    at UndoCoins.fromRaw (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:100:30)
    at Function.fromRaw (/home/berry/sft/land-bronze/bronzeage-node/lib/coins/undocoins.js:112:26)
    at ChainDB.getUndoCoins (/home/berry/sft/land-bronze/bronzeage-node/lib/blockchain/chaindb.js:973:20)
  generatedMessage: false,
  name: 'AssertionError [ERR_ASSERTION]',
  code: 'ERR_ASSERTION',
  actual: false,
  expected: true,
  operator: '==' }

Confirmed it happens on my nodes as well

Current workaround for me is deleting the data/testnet/chain.ldb directory

Will controlled tiles be OK if you do that?

yes, make sure you don't delete your private keys, they sit in data/testnet/walletdb.ldb

OK, going to try it.

Worked for me too. The chain synced very quickly and got past where it was stuck.


Migrate theme settings to the same API we use for localization

Currently, we use the shared preferences plugin through Norbert Koszir's dynamic_theme package, It would be very nice to actually have global color theming, with more colors than red and blue, since his plugin is a bit limited.
I would like use to migrate theme to Hive, and have the global theme color changed between all of the material colors, when the theme button is pressed.

you may wanna add these colors to the terminal

Is @larsb24 working on this in the settings branch?

Is @larsb24 working on this in the settings branch?

Yes

he did it


Handle punctuation in column names for plotting

Is your feature request related to a problem? Please describe
H2O Driverless AI and MLOps return predictions of binary and multi-class in the form of name.0, name.1, ..., name.n. UI plots fail if the x or y has a decimal in the name, without an error message (the page is just white). The user needs to replace the decimal in the name for the plot to work as expected.
Describe the solution you'd like
Automatically handle decimals in the feature names
Describe alternatives you've considered
Require all developers to handle this themselves and just add more documentation
Additional context
Example that works
from h2o_wave import site, ui, data
import pandas as pd
import numpy as np

page = site['/demo']

n = 100
df = pd.DataFrame(dict(
    length=np.random.rand(n),
    width=np.random.rand(n),
    data_type=np.random.choice(a=['Train', 'Test'], size=n, p=[0.8, 0.2])
))

# Plot two numeric columns by each other and color based on a third, categorical column
page['scatter'] = ui.plot_card(
    box='1 1 4 5',
    title='Scatter Plot from Dataframe',
    data=data(
        fields=df.columns.tolist(),
        rows=df.values.tolist(),
        pack=True,
    ),
    plot=ui.plot(marks=[ui.mark(
        type='point',
        x='=length', x_title='Length',
        y='=width', y_title='Width',
        color='=data_type', shape='circle',
    )])
)

page.save()

Add a point to fail:
from h2o_wave import site, ui, data
import pandas as pd
import numpy as np

page = site['/demo']

n = 100
df = pd.DataFrame(dict(
    length=np.random.rand(n),
    width=np.random.rand(n),
    data_type=np.random.choice(a=['Train', 'Test'], size=n, p=[0.8, 0.2])
))

df.columns = ['length', 'width', 'data.type']

# Plot two numeric columns by each other and color based on a third, categorical column
page['scatter'] = ui.plot_card(
    box='1 1 4 5',
    title='Scatter Plot from Dataframe',
    data=data(
        fields=df.columns.tolist(),
        rows=df.values.tolist(),
        pack=True,
    ),
    plot=ui.plot(marks=[ui.mark(
        type='point',
        x='=length', x_title='Length',
        y='=width', y_title='Width',
        color='=data.type', shape='circle',
    )])
)

page.save()

@lo5 Thank you for fixing this in the color parameter! We noticed that the plot still fails with no error message if punctuation is in the column name for other parameters like x or y. Can this fix be applied to all inputs that can be parameterized by feature="=column"?
Repo of failing in the x and y:
from h2o_wave import site, ui, data
import pandas as pd
import numpy as np

page = site['/demo']

n = 100
df = pd.DataFrame(dict(
    length=np.random.rand(n),
    width=np.random.rand(n),
    data_type=np.random.choice(a=['Train', 'Test'], size=n, p=[0.8, 0.2])
))

df.columns = ['length.1', 'width.1', 'data.type']

# Plot two numeric columns by each other and color based on a third, categorical column
page['scatter'] = ui.plot_card(
    box='1 1 4 5',
    title='Scatter Plot from Dataframe',
    data=data(
        fields=df.columns.tolist(),
        rows=df.values.tolist(),
        pack=True,
    ),
    plot=ui.plot(marks=[ui.mark(
        type='point',
        x='=length.1', x_title='Length',
        y='=width.1', y_title='Width',
        color='=data.type', shape='circle',
    )])
)

page.save()

CC @mturoci

@mturoci Hit this again today on dataset with a column called Churn?
It would be great if Plot could support all punctuation in column names, but if this is not feasible, could we please add a check and present the user with an error message?

I guess it shouldn't be a problem for any of the options, just not sure what is the general practice - cc @lo5


1.5.1. logging error

Operating environment (HassOS, Virtual Machine, Device/platform running Hass.io):
Hassio with Homeassistant 0.96.2
Zigbee2mqtt on Hassio 1.5.1
Description of problem:
zigbee2mqtt:info 7/21/2019, 8:38:40 AM Starting zigbee-shepherd
events.js:174
throw er; // Unhandled 'error' event
^
Error: ENOENT: no such file or directory, open '/share/zigbee2mqtt/log/2019-07-21.08-38-40/log.txt'
Emitted 'error' event at:
at WriteStream. (/zigbee2mqtt-1.5.1/node_modules/winston/lib/winston/transports/file.js:491:16)
at WriteStream.emit (events.js:198:13)
at lazyFs.open (internal/fs/streams.js:277:12)
at FSReqWrap.args [as oncomplete] (fs.js:140:20)
npm
ERR! code ELIFECYCLE
npm ERR! errno 1
npm
ERR<EMAIL_ADDRESS>start: node index.js
npm ERR! Exit status 1
npm ERR!
npm ERR!
Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Your entire configuration from the frontend (with sensitive fields redacted)
{
"data_path": "/share/zigbee2mqtt",
"devices": "[]",
"groups": "groups.yaml",
"homeassistant": true,
"permit_join": false,
"mqtt": {
"base_topic": "zigbee2mqtt",
"server": "mqtt://homeassistant",
"user": "my_user",
"password": "my_password"
},
"serial": {
"port": "/dev/ttyACM0"
},
"advanced": {
"pan_id": 6754,
"channel": 11,
"network_key": [
1,
3,
5,
7,
9,
11,
13,
15,
8,
2,
4,
6,
8,
10,
12,
13
]
}
}

I have same, error.. with same configuration...

It looks to me it cannot open the log file for writing, which would point to a permission issue but given nothing on my side has changed between 1.4.1 and 1.5.1, I am thinking it is something with the Hassio implementation?

I seem to have the same issue.

i also have same problem
./run.sh: line 15: [Info] Configuration backup found in /share/zigbee2mqtt/.configuration.yaml.bk. Skipping config backup.: No such file or directory
2019-07-21T20:04:36: PM2 log: Launching in no daemon mode
2019-07-21T20:04:36: PM2 log: App [npm:0] starting in -fork mode-
2019-07-21T20:04:36: PM2 log: App [npm:0] online
><EMAIL_ADDRESS>start /zigbee2mqtt-1.5.1
> node index.js
  zigbee2mqtt:info 7/21/2019, 8:04:37 PM Logging to directory: '/share/zigbee2mqtt/log/2019-07-21.20-04-37'
  zigbee2mqtt:info 7/21/2019, 8:04:37 PM Starting zigbee2mqtt version 1.5.1 (commit #unknown)
  zigbee2mqtt:info 7/21/2019, 8:04:37 PM Starting zigbee-shepherd
/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/components/controller.js:429
                    throw new TypeError('net.precfgkey should be an array with 16 uint8 integers.');
                    ^
TypeError: net.precfgkey should be an array with 16 uint8 integers.
    at /zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/components/controller.js:429:27
    at Object._.forOwn (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/node_modules/busyman/index.js:242:27)
    at Object._.forEach (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/node_modules/busyman/index.js:415:18)
    at Controller.setNvParams (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/components/controller.js:414:7)
    at new ZShepherd (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/shepherd.js:28:21)
    at Zigbee.start (/zigbee2mqtt-1.5.1/lib/zigbee.js:57:25)
    at startupLogVersion (/zigbee2mqtt-1.5.1/lib/controller.js:167:25)
    at utils.getZigbee2mqttVersion (/zigbee2mqtt-1.5.1/lib/controller.js:193:13)
    at git.getLastCommit (/zigbee2mqtt-1.5.1/lib/util/utils.js:102:9)
    at /zigbee2mqtt-1.5.1/node_modules/git-last-commit/source/index.js:44:5
npm
 ERR! code ELIFECYCLE
npm ERR! errno 1
npm
 ERR<EMAIL_ADDRESS>start: `node index.js`
npm ERR! Exit status 1
npm ERR!

npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-07-21T19_04_37_672Z-debug.log
2019-07-21T20:04:41: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
2019-07-21T20:04:41: PM2 log: App [npm:0] starting in -fork mode-
2019-07-21T20:04:41: PM2 log: App [npm:0] online
><EMAIL_ADDRESS>start /zigbee2mqtt-1.5.1
> node index.js
  zigbee2mqtt:info 7/21/2019, 8:04:42 PM Logging to directory: '/share/zigbee2mqtt/log/2019-07-21.20-04-42'
  zigbee2mqtt:info 7/21/2019, 8:04:42 PM Starting zigbee2mqtt version 1.5.1 (commit #unknown)
  zigbee2mqtt:info 7/21/2019, 8:04:42 PM Starting zigbee-shepherd
/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/components/controller.js:429
                    throw new TypeError('net.precfgkey should be an array with 16 uint8 integers.');
                    ^
TypeError: net.precfgkey should be an array with 16 uint8 integers.
    at /zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/components/controller.js:429:27
    at Object._.forOwn (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/node_modules/busyman/index.js:242:27)
    at Object._.forEach (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/node_modules/busyman/index.js:415:18)
    at Controller.setNvParams (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/components/controller.js:414:7)
    at new ZShepherd (/zigbee2mqtt-1.5.1/node_modules/zigbee-herdsman/dist/lib/shepherd.js:28:21)
    at Zigbee.start (/zigbee2mqtt-1.5.1/lib/zigbee.js:57:25)
    at startupLogVersion (/zigbee2mqtt-1.5.1/lib/controller.js:167:25)
    at utils.getZigbee2mqttVersion (/zigbee2mqtt-1.5.1/lib/controller.js:193:13)
    at git.getLastCommit (/zigbee2mqtt-1.5.1/lib/util/utils.js:102:9)
    at /zigbee2mqtt-1.5.1/node_modules/git-last-commit/source/index.js:44:5
npm
 ERR! code ELIFECYCLE
npm ERR! errno 1
npm
 ERR<EMAIL_ADDRESS>start: `node index.js`
npm ERR! Exit status 1
npm ERR!

npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

You error seems to be different, you are missing a network key (see other thread about this).
Network key is now compulsory under 1.5.1

@danielwelch could this be similar to this? https://github.com/danielwelch/hassio-zigbee2mqtt/issues/54

I solved this issue by fill mqtt with the mqtt://ipaddress:port
Username and password, before I had my_user and my_password.

And no issues with creating the logs?

@EmmanuelLM have you tried defining the log directory explicitly in the config? I'm assuming you didn't specify it before, and still are not specifying it now.

@danielwelch @EmmanuelLM adding  "log_directory": "data/log/%TIMESTAMP%", under advanced fixes the issue.

Works perfectly, thank you!


CVE-2019-11254 is reported on a module  dependency gopkg.in/yaml.v2

https://access.redhat.com/security/cve/cve-2019-11254
The upstream Kubernetes fix for this vulnerability is to update the version of the Go dependency, gopkg.in/yaml.v2. This issue affects OpenShift Container Platform components that use versions before 2.2.8 of gopkg.in/yaml.v2 and accept YAML payloads.
A vulnerability was detected from go-autorest that depends on an yaml version that needs to be upgraded.
Please bump the module dependency of yaml to resolve this CVE.
Dependency: vendor/github.com/Azure/go-autorest/vendor/github.com/Azure/go-autorest/go_dep/gopkg.in/yaml.v2:v2.2.7

Fixed in<EMAIL_ADDRESS>


R48 Editor doesn't get past bootup

I'm unsure what the issue is,

I found out that version 1.4-3 works on my system however,

Huh! Can you please try LIBGL_ALWAYS_SOFTWARE=1 java -jar (etc...) when running R48? I'll admit, I don't understand how this happened, but that might provide an acceptable workaround for the moment. I would also like to see the output of the eglinfo command.

Oh! It's possible you might not have the package libgles1.
R48 uses OpenGL ES 1 (and desktop OpenGL 1 on Windows) in order to ensure compatibility with the widest range of hardware possible. However, it appears that expecting all Linux systems to support OpenGL ES 1 may have been a little hasty.

I installed libgles1, but I still got the same error,
Same with running the app with software rendering

Alright, I think I'll need the output of eglinfo on your system to really make any headway here, sorry!

No worries! Thank you so much for helping me through this
I hope this helps!
eglinfo.txt

It works! Thank you so much! ^^

It worked when I used the workaround ^^

Hmm. What GPU are you using? While it's good this worked, it also seems weird?

I'm using an Integrated Intel UHD Graphics 620! ^^

Oops, my phone glitched out and deleted a previous comment

...huh.
Well, in any case, I guess it works now and I'll make a new release?
The comment you deleted was the one with the eglinfo.txt results; if you don't mind I can reupload them for future reference?

Closing for now, though this might come up again later, hmm.

Oh! I don't mind ^^ sure thing!

eglinfo.txt
attached eglinfo results for the device in question

An update on this: I forgot eglBindAPI and later versions of Ubuntu don't ship an OpenGL ES 1 implementation. There was supposed to be a fallback path to use desktop OpenGL, but, well, again, I forgot eglBindAPI.


Comcol resource policies

References
https://github.com/DSpace/dspace-angular/issues/848
Description
This PR adds implementation for collections/communities resource policy management.
Implementation is similar to the existing Item-level feature.
Instructions for Reviewers

login as admin to the application
open a community edit item page
go to new authorizations tab and try to add/edit a new resource policy
do the same for a collection

List of changes in this PR:

added a new authorizations tab in the community edit page

added a new authorizations tab in the community edit page

Checklist
This checklist provides a reminder of what we are going to look for when reviewing your PR. You need not complete this checklist prior to creating your PR (draft PRs are always welcome). If you are unsure about an item in the checklist, don't hesitate to ask. We're here to help!

[ ] My PR is small in size (e.g. less than 1,000 lines of code, not including comments & specs/tests), or I have provided reasons as to why that's not possible.
[x] My PR passes TSLint validation using yarn run lint
[x] My PR doesn't introduce circular dependencies
[x] My PR includes TypeDoc comments for all new (or modified) public methods and classes. It also includes TypeDoc for large or complex private methods.
[x] My PR passes all specs/tests and includes new/updated specs or tests based on the Code Testing Guide.
[ ] If my PR includes new, third-party dependencies (in package.json), I've made sure their licenses align with the DSpace BSD License based on the Licensing of Contributions documentation.

@atarix83 : Just a reminder, this PR has a merge conflict... it also should be rebased so that the GitHub CI tests run against this branch.  Thanks!

@tdonohue this should be ok now

NOTE for @artlowel : As I know you'll likely need to update #739 anyway after this merger, keep in mind there's a TODO in this PR in resource-policy-create.component.ts -- it might be unnecessary code once #739 is applied.

You're right, it was no longer necessary. Updated in #961


BUG: center of mass

It is not really a bug but an issue of typical int32 overflow.
Below is an image, which, if converted to a numpy matrix is int32 and the scipy center of mass function delivers coordinates outside the grid, which is wrong. This happens, because in scipy:
normalizer = sum(input, labels, index)
overflows. Especially as 16bit images contain large numbers.
There are many methods to avoid, like making the matrix float, etc, but can it maybe be handled inside the scipy functions?

Reproducing Code Example
from PIL import Image
from scipy import ndimage
import numpy as np

im = Image.open('nf_2.png')

npa = np.array(im)

print(npa.shape,npa.dtype) # is int32
print('min/max of array',npa.min(),npa.max())
print(ndimage.center_of_mass(npa))

com_x = 0.0
com_y = 0.0
sum = 0.0 # float
for x in range(npa.shape[1]):
    for y in range(npa.shape[0]):
        sum += npa[y][x] # overflow if sum is int32
        com_x += float(npa[y][x])*float(x)
        com_y += float(npa[y][x])*float(y)
com_x /= sum
com_y /= sum
print('should be:',com_y,com_x)

Error message
(1216, 1936) int32
min/max of array 96 65520
(2645.1217782736508, 4121.233298317583)
should be:<PHONE_NUMBER>060187<PHONE_NUMBER>837329

SciPy/NumPy/Python version information
1.7.1 1.20.3 sys.version_info(major=3, minor=7, micro=7, releaselevel='final', serial=0)

Thanks for the report, @TomFD. I suspect you are on Windows, where the default integer type in NumPy is int32.  I can reproduce the problem on systems where the default integer is int64 with this simple example:
In [72]: import numpy as np

In [73]: from scipy.ndimage import center_of_mass

In [74]: m = 2**63-1

In [75]: x = np.array([[m, m], [m, m]])

In [76]: x
Out[76]:
array([[9223372036854775807,<PHONE_NUMBER>854775807],
       [9223372036854775807,<PHONE_NUMBER>854775807]])

In [77]: center_of_mass(x)
Out[77]: (-4.611686018427388e+18, -4.611686018427388e+18)

The result should be (0.5, 0.5):
In [78]: center_of_mass(x.astype(np.float64))
Out[78]: (0.5, 0.5)

Previously reported in #16821. My proposed fix is to try to make the normalization constant calculation use a float accumulator the same way that we use floats later in the calculation. Unfortunately, that requires making API changes through a moderately long chain of of functions.

We end up creating a bunch of float64 temporaries later when we multiply the input against the grids, so one more temporary (sum(input.astype(float), labels, index)) would probably not be too onerous.

Thanks @rkern.  I'll close this as a duplicate.


Fix invoke http service route '/'

Description
Why not support invoke route '/'? root route api maybe need be call
Checklist
Please make sure you've  completed the relevant tasks for this PR, out of the following list:

[x] Code compiles correctly
[x] Created/updated tests
[x] Unit tests passing
[ ] End-to-end tests passing
[ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Thanks @rrylee! Can you please open an issue for it so we can track this in release notes? Thanks!

@yaron2 here https://github.com/dapr/dapr/issues/2571


Connection issues when running action on custom container

Hi,
I don't know if this is a configuration issue or something that needs to be fixed in the action itself.
I switched from installing all dependencies in the github action to running the action in a custom container with all dependencies preinstalled. Now, my mqtt client can not connect to the broker in your container anymore.
I tried connecting to localhost and by setting the hostname.
Is this a limitation in github actions?
Is something wrong with my configuration?
See https://github.com/bertmelis/espMqttClient/actions/runs/9938521654/job/27451130098

Yes, a service container is the solution. Not a straightforward one though...


[Feature Request] ACERAC

🚀 Feature
Implementation of the ACERAC algorithm (Actor-Critic with Experience Replay and Autocorrelated Action). Will include implementation of replay buffer that supports returning of n-step trajectories, as it is required by the ACERAC algorithm.
Paper with the algorithm is avaliable here (Open Access)
Motivation
ACERAC is an off-policy Actor-Critic algorithm with well-adjustable hyperparameters for fine-time discretization and good results on PyBullet robotic environments.
Pitch
I will implement this feature by myself, if approved.
Alternatives
Original implementation is available here
However, I believe it would be easier for potential users if this algorithm was part of SB3 suite with unified interface.
Additional context
No response
Checklist

[X] I have checked that there is no similar issue in the repo
[X] If I'm requesting a new feature, I have proposed alternatives

hello,

fine-time discretization

Could you give a quick example/short explanation of what exact problem it solves that it not solved by other methods?

ACERAC is an algorithm designed to perform well in fine-time discretization environments.
Fine-time discretization environments include environments where a single time step corresponds to relatively short part of the whole MDP. Such environments include robotic control environments with high control frequency.
The research presented in the ACERAC paper uses PyBullet robotic environments (Ant, HalfCheetah, Hopper, Walker2D) with 3 and 10 times increased control frequency for experiments in such setting.
"Making deep q-learning methods robust to time discretization," by C. Tallec, L. Blier, and Y. Ollivier describes difficulties in using common RL algorithms in such environments. To summarize:

Action-value function degrades, to value function as control frequency increases, as each action becomes shorter and less significant
Structured exploration such as action autocorrelation is required to enable efficient exploration, as unstructured action noise may get filtered by momentum of the underlying system.

Experimental results in the ACERAC paper further suggest that using n-step return estimation is beneficial in such environments.


Return to work sound issue

I love this app - it runs my day - looking forward to any updates.
I would like to be able to set the sound for the end of my break to be louder. I hear the "take a break" sound just fine but for some reason, I am not hearing the "return to work" sound. I don't think it's working at the moment, but it could be that I am just not hearing it.
Mac Sierra 10.12.6
Chrome Version 61.0.3163.100 (Official Build) (64-bit)

@kleinash Thanks for the feedback.
In the settings, which sound are you using for the end-of-break audio notification?
I'm using various Creative Commons sounds that aren't normalized to the same volume. Some sounds are too quiet, some are too loud (see issue #71). I believe the fix is to just normalize the volume levels.

@schmich At the moment its the battle horn, but I go to the ship bell sometimes. I haven't quite found a sound that I hear through my thoughts.  So it will continue to sound until I get back to my work and shut it off.

I addressed volume issues in c1917a95c495e8a2fc20a68ec62ea57ec6e6ab2b. As I mentioned in #71, I don't think the results are perfect, but it should be better than before. I've also added a few more alarm sounds that are a bit more intrusive: analog alarm clock, digital watch alarm, and digital alarm clock. These changes will be present in the next release.
As for looping alarm sounds, I created #95 to track that feature.


1.3.1 on npm

Hi @nyxtom,
I saw you released a version to npm, but it seems the assume-no-change feature is not include?
I am not so familar with npm release, did I missed something?
Sincerely,

Oh the issue is the changes were made to reconcile.umd.js rather than lib/reconcile.js

Oh, ok. I'll create a new PR to revert and change and apply the change to lib/reconcile.js

A new version is released with the feature. Closing issue.
Thanks, @nyxtom


Mocks for client.go

Please use the following questions as a guideline to help me answer
your issue/question without further inquiry. Thank you.
Which version of Elastic are you using?
[ X] elastic.v7 (for Elasticsearch 7.x)
[ ] elastic.v6 (for Elasticsearch 6.x)
[ ] elastic.v5 (for Elasticsearch 5.x)
[ ] elastic.v3 (for Elasticsearch 2.x)
[ ] elastic.v2 (for Elasticsearch 1.x)
Please describe the expected behavior
Are there any mocks available for client.go for unit testing?
Please describe the actual behavior
There are no mocks that I see for client.go.

When I do mocking, I do so at a higher level like described here.
If I want to do integration tests, it works quite nice with the dockertest library.

Thank you!


haskellPackages.sym: mark broken on darwin

Build https://hydra.nixos.org/build/277976033
#ZurichZHF
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[x] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Can you please rebase this PR so that it online contains the commit in question? The diff is confusing me.