[Bug] LightGBMError: bin size 257 cannot run on GPU

I'm getting the following error while running the latest LightGBM GPU using these params:
params = {
            'device_type': 'gpu',
            'gpu_device_id': 0,
            'gpu_platform_id': 0,
            'gpu_use_dp': 'false',
            'max_bin': 255
}

on Google Colab using this Kaggle dataset: https://www.kaggle.com/c/ieee-fraud-detection. I'm dropping all the categorical variables:
LightGBMError: bin size 257 cannot run on GPU

I also have the same problem. Is there any way to solve this problem?

sorry for missing this issue.
The max-bin actually cannot limit the number of bins for categorical feature.
there are two workarounds:

use the categorical encodings, converting categorical features to numerical ones.
split one categorical feature to multi categorical features, and make sure the number of categories in each splitted feature smaller than 256.

I hit this randomly for no reason with categorical_features as explicitly empty.  Has nothing to do with that.  The test that hit this normally has passed 1000 times before.
File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/lightgbm_gpu/sklearn.py", line 794, in fit
    categorical_feature=categorical_feature, callbacks=callbacks, init_model=init_model)
  File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/lightgbm_gpu/sklearn.py", line 637, in fit
    callbacks=callbacks, init_model=init_model)
  File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/lightgbm_gpu/engine.py", line 230, in train
    booster = Booster(params=params, train_set=train_set)
  File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/lightgbm_gpu/basic.py", line 2104, in __init__
    ctypes.byref(self.handle)))
  File "/opt/h2oai/dai/cuda-10.0/lib/python3.6/site-packages/lightgbm_gpu/basic.py", line 52, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: bin size 257 cannot run on GPU

The number of bins was 256 and there are no categorical features as explicitly chosen.

Same issue happened for me:
  File "/usr/local/lib/python3.6/dist-packages/lightgbm/engine.py", line 228, in train
    booster = Booster(params=params, train_set=train_set)
  File "/usr/local/lib/python3.6/dist-packages/lightgbm/basic.py", line 2237, in __init__
    ctypes.byref(self.handle)))
  File "/usr/local/lib/python3.6/dist-packages/lightgbm/basic.py", line 110, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: bin size 257 cannot run on GPU

Some of the values are categorical in my case but not as many as 257 different ones, combined with @pseudotensor comment, I assume this is something else.

I am also getting the same error.
lightgbm.basic.LightGBMError: bin size 257 cannot run on GPU

What causes this error？Is the bin_size of a categorical feature bigger than the max_bin that causes the error? Or the memory is not enough? Thank you!

lightgbm.basic.LightGBMError: bin size 407 cannot run on GPU

lightgbm.basic.LightGBMError: bin size 407 cannot run on GPU

this is a bug for lightGBM run on GPU,when use CPU,it is OK. SO ,LGBM on GPU need improve.

Same error encountered, any update?

Same here.
lightgbm.basic.LightGBMError: bin size 670 cannot run on GPU

It seems that I've identified the cause of the error: The calculation method for num_total_bin used during Exclusive Feature Bundling https://github.com/microsoft/LightGBM/blob/665c47313d6938a8d80559c824073a142c8bd870/src/io/dataset.cpp#L134 doesn't align completely with the way num_total_bin is calculated during the creation of a FeatureGroup https://github.com/microsoft/LightGBM/blob/665c47313d6938a8d80559c824073a142c8bd870/include/LightGBM/feature_group.h#L68 As a result, the max_bin_per_group (=256) is working during Bundling, but it is not working when creating the FeatureGroup. When I replaced the GetDefaultBin() at dataset.cpp#L134 with GetMostFreqBin(), the issue was resolved.

Same issue here. Can we prioritize the fixing MR?

I also encountered the same error in this situation in fold 3th (total 5 fold), when my category characteristic had many NAN values.
Can do value np.NAN, np.int in feature category -> max_bin
I have use labelencoder convert feature category -> maxbin
But after i use
df_train[cat_cols] = df_train[cat_cols].astype(str)
df_train[cat_cols] = df_train[cat_cols].astype("category")
then the training process will good!


Build dependency issues

Describe the bug

C:\Users\dthal\git\ebpf\ebpf-for-windows\tools\bpf2c\bpf2c.cpp(28,10): fatal error C1083: Cannot open include file: 'bpf2c_driver.template': No such file or directory (compiling source file elf_bpf.cpp)

Alan reports this is because bpf2c_tests is missing a dependency on bpf2c

05/18/2022  02:19 PM             6,520 ebpf_bind_program_data.h
05/18/2022  02:19 PM             7,750 ebpf_sample_ext_program_data.h
05/18/2022  02:19 PM             6,523 ebpf_sockops_program_data.h
05/18/2022  02:19 PM             6,573 ebpf_sock_addr_program_data.h
05/18/2022  02:19 PM             6,813 ebpf_xdp_program_data.h

When adding a new helper function or some other change that would result in changes to the above generated headers, things can get out of sync since the build seems to not notice that the files are out of date and uses the old ones if they exist.
OS information
No response
Steps taken to reproduce bug
Build clean
Make a change that would result in changing the generated headers
Build incremental
Expected behavior
Everything should build and be in sync.
Actual outcome
Either build fails, or files are mismatched.
Additional details
No response

Check cmake build as well


Switch to Gulp 4 broke the Command Palette availability

Hi,
When I switched to Gulp 4 I could no longer use the Command Palette to access the commands. When I chose Gulp in the palette I got this error in the log:
/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:64
    gulp.tree({ deep: true }).forEach(function(task) {
                              ^

TypeError: gulp.tree(...).forEach is not a function
    at _forEachTask4x (/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:64:31)
    at forEachTask (/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:50:9)
    at Object.<anonymous> (/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:88:5)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)
    at Function.Module.runMain (module.js:430:10)
    at startup (node.js:141:18)
    at node.js:980:3

Is the plugin suppose to be compatible with Gulp 4?

It should be compatible and that's a bug! I'm going to upgrade a local project and give it a try.
In the meantime, the log is pointing to a line that does not exist in this package alert(gulp.tree({ deep: true })); on Line 63 of write_tasks_to_cache.js
Have you modified the package by hand? deleting the cache and trying again works? and, lastly, reinstalling the package does the same thing?
Thanks a lot for the report I'll let you know how it goes

Oh yeah, sorry about that. It was just me trying to debug the problem. Here's the actual message:
/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:64
    gulp.tree({ deep: true }).forEach(function(task) {
                              ^

TypeError: gulp.tree(...).forEach is not a function
    at _forEachTask4x (/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:64:31)
    at forEachTask (/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:50:9)
    at Object.<anonymous> (/Library/Application Support/Sublime Text 3/Packages/Gulp/write_tasks_to_cache.js:88:5)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)
    at Function.Module.runMain (module.js:430:10)
    at startup (node.js:141:18)
    at node.js:980:3

Perfect, I'll give it a try as soon as I can (it seems the API changed)

Cool and thanks for the great work! In the meantime I'll use the terminal =)

@jay-zahiri Status update
The gulp API changed in the last few commits. I fixed it locally but I want to test it out a bit more, I'll try to make a release with the fix by Thursday

@jay-zahiri The patch should be up (on v5.2.3) it may take a while to update if you're using Package Control.
I'll close this issue but if the error persist please let me know


arm64: Add stack guard for v8r

Add stack guard as an implementation of HW_STACK_PROTECTION, needs find a better way to handle

Depends on the safe exception stack (https://github.com/zephyrproject-rtos/zephyr/pull/48297).
Enable MPU based stack guard
Tested on fvp_baser_aemv8r board with HW_STACK_PROTECTION enabled.

$ ./zephyr/scripts/twister -p fvp_baser_aemv8r
Renaming output directory to /src/zephyrproject/twister-out.1
INFO    - Using Ninja..
INFO    - Zephyr version: zephyr-v3.3.0-494-ge460f91a379d
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testsuite list...
INFO    - Writing JSON report /src/zephyrproject/twister-out/testplan.json
INFO    - JOBS: 8
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue
INFO    - Total complete: 2018/2018  178%  skipped: 1663, failed:    0, error:    0
INFO    - 2209 test scenarios (2018 test instances) selected, 1663 configurations skipped (1582 by static filter, 81 at runtime).
INFO    - 355 of 2018 test configurations passed (100.00%), 0 failed, 0 errored, 1663 skipped with 0 warnings in 1318.53 seconds
INFO    - In total 2674 test cases were executed, 6498 skipped on 1 out of total 543 platforms (0.18%)
INFO    - 296 test configurations executed on platforms, 59 test configurations were only built.
INFO    - Saving reports...
INFO    - Writing JSON report /src/zephyrproject/twister-out/twister.json
INFO    - Writing xunit report /src/zephyrproject/twister-out/twister.xml...
INFO    - Writing xunit report /src/zephyrproject/twister-out/twister_report.xml...
INFO    - Run completed

Fix some testcases by increasing the stack guard which the stack guard detects

Limitation:

SMP NOT fully supported for now
MMU based stack guard does NOT support, and might open another PR in the future.

You have:
        /* load primary stack and entry point */
-       ldr     x24, =(z_interrupt_stacks + CONFIG_ISR_STACK_SIZE)
+       ldr     x24, =(z_interrupt_stacks + ARCH_KERNEL_STACK_RESERVED + CONFIG_ISR_STACK_SIZE)

You could use __z_interrupt_stack_SIZEOF here instead. It was
created for that purpose. See commit 1f362a81f172.

@carlocaione do you have any clue on this unexpected eof? INFO - 656/656 qemu_cortex_a53_smp samples/subsys/portability/cmsis_rtos_v1/philosophers/sample.portability.cmsis_rtos_v1.philosopher.semaphores FAILED unexpected eof (qemu 0.027s) https://github.com/zephyrproject-rtos/zephyr/actions/runs/4551908320/jobs/8044448572?pr=55207

Uhm, IIRC this problem could be due to some binary / non-ascii data that is being mis-interpreted by the test framework. Do you have anything like that when running locally?

Uhm, IIRC this problem could be due to some binary / non-ascii data that is being mis-interpreted by the test framework. Do you have anything like that when running locally?

Yes, I tested on my local. The weird thing is:
When I run west twister -p qemu_cortex_a53_smp -s samples/subsys/portability/cmsis_rtos_v1/philosophers/sample.portability.cmsis_rtos_v1.philosopher, it PASS without unexpected eof
But if I run west twister -p qemu_cortex_a53_smp, this testcase fails with unexpected eof

@povergoing this is failing locally as well for me:
west build -b qemu_cortex_a53_smp samples/subsys/portability/cmsis_rtos_v1/philosophers -T sample.portability.cmsis_rtos_v1.philosopher.semaphore
and then:
west build -t run

May I see your output? Cause after I run west build -t run, I got the following infinite output which is expected I think? I didn't see any of EOF things?

Weird, for me the output is stuck. Are you sure you are trying with sample.portability.cmsis_rtos_v1.philosopher.semaphore and not with sample.portability.cmsis_rtos_v1.philosopher (note the semaphore part).

Weird, for me the output is stuck. Are you sure you are trying with sample.portability.cmsis_rtos_v1.philosopher.semaphore and not with sample.portability.cmsis_rtos_v1.philosopher (note the semaphore part).

Yes, my cmd: west build -b qemu_cortex_a53_smp zephyr/samples/subsys/portability/cmsis_rtos_v1/philosophers -T sample.portability.cmsis_rtos_v1.philosopher.semaphores

Yeah, there is something strange going on. Sometimes it works, sometimes it doesn't.
Can you try with:
west build -b qemu_cortex_a53_smp samples/subsys/portability/cmsis_rtos_v1/philosophers -T sample.portability.cmsis_rtos_v1.philosopher.semaphores --pristine ?

I have the normal output even using --pristine. I need to try to upgrade the SDK

@povergoing any news?

@povergoing any news?

While I am still looking into the CI issue, bu I found the main branch is already broken. When we run twister, some of the time-sensitive test cases fail with qemu_cortex_a53_smp (this is already known right?), and further, the assert will be false and call thread_abort, but some test cases need some clean up, for example:
https://github.com/zephyrproject-rtos/zephyr/blob/main/tests/kernel/sched/schedule_api/src/test_sched_timeslice_and_lock.c#L262
However, the assert happens, then the current thread is aborted, thus the teardown_thread() will not run, and the "zombie" thread still lives in runq.
What's more, because of "zombie" thread will be reused many times, the same thread will be re-added into runq which will cause an unexpected exception during the next tests and then cause a psci_down. At this time, the Qemu will exit, and the twister thinks it's an unexpected eof.
Any suggestions? But I am now on vacation and will keep on fixing it when I am back.


Update column names in Meraki collectors to match the API results

I.e., 'org_id' should be 'orgId', etc.

Completed with one exception. The meraki_get_organizations "id" column is renamed to "org_id" to avoid conflicting with various database systems.


WebSocketDisconnect not caught while waiting on async tasks

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the FastAPI documentation, with the integrated search.
[X] I already searched in Google "How to X in FastAPI" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to FastAPI but to Pydantic.
[X] I already checked if it is not related to FastAPI but to Swagger UI.
[X] I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code
@app.websocket("/ws")
async def my_websocket(websocket: WebSocket):
    await websocket.accept()

    try:
        await websocket.send_text("Hello!")
        await asyncio.sleep(5)
        await websocket.send_text("Finished")
    except WebSocketDisconnect:
        print("Client disconnected")

Description
Example first, background below.
Steps to reproduce using provided example

Connect to the /ws WebSocket endpoint.
Confirm client receives "Hello!" message from the server.
Disconnect the client within 5 seconds.

Expected result
The WebSocketDisconnect error is caught as soon as the client closes the connection and "Client disconnected" is printed in the console.
Actual result
No WebSocketDisconnect exception is raised and instead, websockets.exceptions.ConnectionClosedError is raised, only when the server attempts to send a message back to the client.
With debug logging enabled, I can see the following at the exact moment the client disconnects:
DEBUG:websockets.protocol:server - event = data_received(<6 bytes>)
DEBUG:websockets.protocol:server < Frame(fin=True, opcode=<Opcode.CLOSE: 8>, data=b'', rsv1=False, rsv2=False, rsv3=False)
DEBUG:websockets.protocol:server - state = CLOSING
DEBUG:websockets.protocol:server > Frame(fin=True, opcode=<Opcode.CLOSE: 8>, data=b'', rsv1=False, rsv2=False, rsv3=False)
DEBUG:websockets.protocol:server x half-closing TCP connection
DEBUG:websockets.protocol:server - event = eof_received()
DEBUG:websockets.protocol:server - event = connection_lost(None)
DEBUG:websockets.protocol:server - state = CLOSED
DEBUG:websockets.protocol:server x code = 1005, reason = [no reason]

So, the websockets library does receive an event as soon as the connection is closed. But this is not propagated to a WebSocketDisconnect error and I've found no way to catch this as it happens.
Background
In my real code, at some point, the server must await a long-running task and send a message to the client when it has finished. This task is potentially expensive - as such, it is strongly preferable to cancel it as soon as a client disconnects, instead of waiting for it to finish only to find out at that point that the client is no longer there.
In my testing, I've seen that the WebSocketDisconnect error is raised in some cases, of course, for example:
@app.websocket("/ws")
async def my_websocket(websocket: WebSocket):
    await websocket.accept()

    try:
        await websocket.receive_text()
    except WebSocketDisconnect:
        print("Client disconnected")

If the client connects and then disconnects before sending text, then the disconnection is caught successfully. But, not if the client disconnects during a task that the server is waiting on.
Operating System
macOS
Operating System Details
No response
FastAPI Version
0.68.0
Python Version
3.9.5
Additional Context
No response

I think you might be able to use asyncio.create_task or one of the other convenient asyncio builtins, to do this?
Something like:

@app.websocket("/ws")
async def my_websocket(websocket: WebSocket):
    await websocket.accept()
    task = None

    try:
        text = websocket.receive_text()

        task = asyncio.create_task(some_long_running_task, **kwargs)

    except WebSocketDisconnect:

        if task:
            task.cancel()
        print("Client disconnected")

I think you might be able to use asyncio.create_task (https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task)

Yes, ideally I would use an approach similar to this in order to cancel the task if a client disconnects midway through. But the issue is that the exception is not raised if the client disconnects at that point, meaning code like your example wouldn't work.
I've been unable to find any method of detecting the disconnect until after the task finishes (no difference when using create_task unfortunately), at which point cancelling becomes irrelevant. So your example is exactly what I would like to achieve, it just doesn't work.

I just had the same issue. What you can do, is to create two tasks: one for your long-running thing and one for awaiting client text. The latter will raise an exception the moment the client disconnects, so you can use asyncio.FIRST_COMPLETED to return out of the task awaiter. Here's how it could look (my long-running task is a blocking redis call that waits for a list):
async def get_receive_text(websocket) -> Tuple[Union[str, None], bool]:
    """The second return value bolean will return True if the client disconnected"""
    try:
        return await websocket.receive_text(), False
    except WebSocketDisconnect as e:
        return None, True

async def get_redis_data(list_name) -> Tuple[Union[str, None], bool]:
    """The second return value bolean will never return true, just there for task consistency"""
    redis = aioredis.from_url("redis://localhost:6379")
    async with redis.client() as conn:
        result = await conn.blpop(keys=[list_name], timeout=30)
        if result:
            return result[1].decode(), False
        return None, False

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    conn_id = random.randint(0, 999)
    print(f'{conn_id=}')
    client_is_connected = True
    while client_is_connected:
        # Wait for either text from the websocket channel, OR the redis list (which will ultimately time out)
        task1 = get_receive_text(websocket)
        task2 = get_redis_data(list_name=conn_id)
        if websocket.client_state.name != 'CONNECTED':
            client_is_connected = False
            break
        # asyncio wait will return when either text is received from the client,
        # or the redis blocking call returned data
        # or the redis blocking call timed out
        # or the client disconnected
        done, pending = await asyncio.wait([task1, task2], return_when=asyncio.FIRST_COMPLETED)
        done_task = done.pop()
        while len(pending) > 0:
            # cancel the other tasks in the waiter
            pending_task = pending.pop()
            pending_task.cancel()
        result, connection_closed = done_task.result()
        if connection_closed:
            client_is_connected = False
            break
        if result:
            print(result)
            await websocket.send_text(result)

There is no correct solution if your coroutine only send data and not receive
Only trying to receive data to sync websocket state?..
But this is not good solution because why more simpler raise cancell task exception for this coroutine
And debug logs know about connection lost
Is there a plan to fix this issue in the future?
@tiangolo
Thanks for answer in advance.

Having the same issue

Having the same issue, tooo

This works well! One concern I am confused about is how to ensure all clients subscribed to the same websocket receive all messages; currently only alternating clients receives a new message

This works well! One concern I am confused about is how to ensure all clients subscribed to the same websocket receive all messages; currently only alternating clients receives a new message

you need to track subscribed clients in a db such as redis etc, and have each message "broadcast" to each subscriber (aka a for each loop basically)

I cannot provide many details, but this implementation misses sending messages when get_recieve_data adds a message to the Redis list being read by get_redis_data. Still investigating...

@joshjhans that sounds like another issue than what this issue is relating to. If you have more information, I would recommend opening another issue (if it is FastAPI related).


feat: Add scanning of local files and folders (single scan)

Pull Request
Description
Current PR contains:

implementation of the new feature - scanning of local files and folders with the possibility to generate reports in the console or an HTML file
update and extension of unit tests
fixes for several bugs that were found during the creation of new unit tests and local tests of the new feature
update of the documentation

Type of change
Please delete options that are not relevant.

[X] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[X] Code cleanup/refactoring
[X] Documentation update
[X] This change requires a documentation update
[ ] CI system update
[X] Test Coverage update

Testing
Unit tests and local functional tests.
Checklist:

[X] My code follows the style guidelines of this project
[X] My code meets the required code coverage for lines (90% and above)
[X] My code meets the required code coverage for branches (80% and above)
[X] I have performed a self-review of my own code
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have made corresponding changes to the documentation
[X] My changes generate no new warnings
[X] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes

Codecov Report
Attention: Patch coverage is 85.58559% with 16 lines in your changes missing coverage. Please review.

Project coverage is 92.48%. Comparing base (dccafc6) to head (60ec69c).

Files
Patch %
Lines

.../java/com/lpvs/service/scan/LPVSDetectService.java
85.93%
7 Missing and 2 partials :warning:

src/main/java/com/lpvs/util/LPVSFileUtil.java
80.95%
0 Missing and 4 partials :warning:

...va/com/lpvs/service/LPVSQueueProcessorService.java
0.00%
1 Missing and 1 partial :warning:

src/main/java/com/lpvs/util/LPVSPayloadUtil.java
95.00%
0 Missing and 1 partial :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #558      +/-   ##
============================================
- Coverage     92.57%   92.48%   -0.10%
- Complexity      529      545      +16
============================================
  Files            48       48
  Lines          1737     1809      +72
  Branches        208      223      +15
============================================
+ Hits           1608     1673      +65
- Misses           64       67       +3
- Partials         65       69       +4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@tiokim Please take a look at this PR. I would be grateful for your review and feedback.


Extension

Steps to reproduce
Manganato extension seems to broke in the app..but in manganato website it just fine

Expected behavior
Working
Actual behavior
Not loading picture
Crash logs
No response
Yōkai version
New version
Android version
14
Device
Realme c3
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] If this is an issue with an extension, or a request for an extension, I should be contacting the extensions repository's maintainer/support for help.
[X] I am reporting an issue exclusive to this fork. I have also checked that is not an issue on the main version of Mihon
[X] I have tried the troubleshooting guide.
[X] I have updated the app to version <IP_ADDRESS>.
[X] I have updated all installed extensions.
[X] I have filled out all of the requested information in this form.

Please confirm that it's in fact an issue with the app with the extension maintainer before reporting a bug regarding extension, this is not where you report extension issues as Yokai is not associated with any extensions.


Do Something Smart with Mage:: Calls

After converting, many classes are left with calls to the old static helper
Mage::app()->getConfig
Mage::app()->getLayout
Mage::app()->getWebsite
Mage::app()->getWebsites
Mage::getBaseDir
Mage::getConfig
Mage::getModel
Mage::getSingleton
Mage::getStoreConfig
Mage::getUrl
Mage::helper
Mage::log

While 100% coverage would be impossible, it'd be nice if some of these were converted to their Magento 2 equivalents.

Mage::log, Mage::helper, Mage::getSingleton, Mage::getModel, Mage::getStoreConfig should be fully covered.
Mage::app()->getConfig calls were split to multiple classes, some of them handled.
Could you please give examples of not converted code with these?

There were Mage::log calls in the setup resource data scripts (data/foobar_setup/data-*.php).
There were Mage::getSingleton  calls in the setup resource data scripts (data/foobar_setup/data-*.php).
There were Mage::getModel calls in the  setup resource data scripts (data/foobar_setup/data-*.php).
There were Mage::getStoreConfig calls in the setup resource data scripts  (data/foobar_setup/data-*.php).
There were Mage::helper calls in
app/code/Foo/Baz/Helper/Header.php
app/code/Foo/Baz/Helper/Template/Page/Html/Header.php
app/code/Foo/Baz/Helper/Data

These calls took the form
return Mage::helper($this->_menuModuleNameShort)->getMobileMenuThreshold();
return Mage::helper($this->_menuModuleNameShort)->getMobileMenuThreshold();
$theme = Mage::helper($packageName);
$theme = Mage::helper($packageName);

Some Mage::app() calls that wern't converted.
$now = strtotime(Mage::app()->getLocale()->date()->setTime('00:00:00')->toString(Date::DATETIME_INTERNAL_FORMAT));
$websites = Mage::app()->getWebsites(false, true); //IMPORTANT: without admin website. Get websites by code.
Mage::app()->getConfig()->reinit();

Thank you for details. The problem with variables in Mage::helper($var) is that it is impossible to convert them directly to class names as in case of static references Mage::helper('core/data').
Using ObjectManager::getInstance()->get($var) will not really fix anything because $var is generated not as class name but Magento1 alias.

All other cases handled in latest commit

Closing this and other issues, please reopen if there's a problem.


Changed Nav to be icons

Made app not render calculator that was not being used
Changed lodash imports to only bring in used functions
Centered content in browser window
Upgraded react-scripts mainly to get rid of webpack licence iframe
Moved some dev dependencies down in package.json
Added hillarious comment to commented out test

Thanks for the pull request!

Changed Nav to be icons

Can we have hovertext "Shoot" and "Fight"?  I'm addicted to hovertext for icons?  Also, are the engage and conceal order icons just for possible future use?

Made app not render calculator that was not being used

Unfortunately that change also makes the shoot/fight section lose state when you swap to the other section.  Ex: you are in shoot section, change MWx to 1, then go to fight section and back to shoot section, and you'll see that MWx is back to X (the default).  I'd rather that users not lose work when toggling between shoot and fight sections.

Changed lodash imports to only bring in used functions

Good idea.  Sidenote: you missed Util.ts, line 59, which uses _ and caused a syntax error for me.

Centered content in browser window

I like it.

Upgraded react-scripts mainly to get rid of webpack licence iframe

Could you walk me through how you did this?  I think at one point I tried to do some upgrading of eslint and got so many issues I undid it.  At another point, I did some other upgrade (react itself?) and then I couldn't use tsconfig.json's "baseUrl": "." to get import paths more absolute (within the project), so I undid that too.  So, I have to ask if you did anything other than a npm update react-scripts?
Also, webpack license iframe, is that this thing?

Moved some dev dependencies down in package.json

Interesting.  I can see why TypeScript and types packages are a dev dependency and not a runtime dependency, but now I'm confused why create-react-app would not put those things in devDependencies.  Any insight?

Oh, also, the icons shouldn't be in src/, right?  maybe public/ or a subfolder of public?

Can we have hovertext "Shoot" and "Fight"? I'm addicted to hovertext for icons? Also, are the engage and conceal order icons just for possible future use?

Done

Good idea. Sidenote: you missed Util.ts, line 59, which uses _ and caused a syntax error for me.

Fixed.

Unfortunately that change also makes the shoot/fight section lose state when you swap to the other section

Oops, this is better now.

Images should be in src because they are imported, so they will be part of the build process.

Thanks again!


Update the default GPU driver to 410.72

Also disable Module.symvers file check which causes installation
failure in the new driver version.

/cc @rkolchmeyer


”Florești”, ”Copăcioasa”, ”Gârdoaia”, ”Livezi”, ”Moșneni” are listed in the drop-down menu

Description:
When user writes ”Florești”, ”Copăcioasa”, ”Gârdoaia”, ”Livezi”, ”Moșneni” the locations appear in the drop-down menu next to the comune and the county which is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Florești"

Expected results
The location appears in the drop-down menu as "Sat Florești, MEHEDINȚI (Florești)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Copăcioasa"

Expected results
The location appears in the drop-down menu as "Sat Copăcioasa, MEHEDINȚI (Florești)".

Step 4
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 5
Write in the search bar "Gârdoaia"

Expected results
The location appears in the drop-down menu as "Sat Gârdoaia, MEHEDINȚI (Florești)".

Step 6
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 7
Write in the search bar "Livezi"

Expected results
The location appears in the drop-down menu as "Sat Livezi, MEHEDINȚI (Florești)".

Step 8
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 9
Write in the search bar "Moșneni"

Expected results
The location appears in the drop-down menu as "Sat Moșneni, MEHEDINȚI (Florești)".


Failed to install gotch on MacOS

I encountered problems when I was trying to install gotch on MacOS
I've installed libtorch at /usr/local/lib/libtorch and executed setup-gotch.sh after  setting environment variables. Here is the result:
GOPATH:'/usr/local/Cellar/go/1.15.6'
GOTCH_VERSION: 'v0.3.8'
CUDA_VERSION: '10.1'
go: creating new go.mod: module github.com/sugarme/gotch-test
<EMAIL_ADDRESS>existing. Deleting...
<EMAIL_ADDRESS>existing. Deleting...
creating<EMAIL_ADDRESS>for GPU
creating<EMAIL_ADDRESS>for GPU
sudo: ldconfig: command not found

however, I cannot use gotch in my go project.
I also tried to install gotch using go get, but I got
$ go get github.com/sugarme/gotch
go: github.com/sugarme/gotch upgrade => v0.3.11
# github.com/sugarme/gotch/libtch
ld: warning: directory not found for option<EMAIL_ADDRESS>ld: warning: directory not found for option '-L/usr/local/cuda/lib64'
ld: library not found for -lcuda
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@HaoboGu,
As you install for your Mac OS, I don't think you can install with CUDA support, however you can try CPU version of libtorch. The installation guide is just for Linux and there might be some changes for MacOS. We have limited experience with Mac OS and gotch requires C++17 support, so make sure your xcode version support C++ 17. However you can try it and let us know how you go: https://github.com/sugarme/gotch#cpu
Please ignore ldconfig warning/error as it is not applicable for MacOS. Also make sure you have GOPATH set in your machine that can be acccessible for current user without sudo.
I hope I can find my time to update my box MacOS and update the installation but please give further feedback so that we can update installation for MacOS. Thanks.

@sugarme

@HaoboGu,
Such error means your libtorch is not in the PATH. Can you printenv and check? You can set your environement for current session by doing:
export GOTCH_LIBTORCH="/usr/local/lib/libtorch"
export LIBRARY_PATH="$LIBRARY_PATH:$GOTCH_LIBTORCH/lib"
export CPATH="$CPATH:$GOTCH_LIBTORCH/lib:$GOTCH_LIBTORCH/include:$GOTCH_LIBTORCH/include/torch/csrc/api/include"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GOTCH_LIBTORCH/lib"

@sugarme ,
I added those environment variables and another error occurs
go build main.go
# github.com/sugarme/gotch/libtch
ld: warning: directory not found for option '-L/lib64'
ld: warning: directory not found for option '-L/libtorch/lib'
ld: warning: ignoring file /usr/local/lib/libtorch/lib/libtorch.so, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 )
ld: warning: ignoring file /usr/local/lib/libtorch/lib/libc10.so, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 )
ld: warning: ignoring file /usr/local/lib/libtorch/lib/libtorch_cpu.so, building for macOS-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x03 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 )
Undefined symbols for architecture x86_64:
  "at::__lshift__(at::Tensor const&, c10::Scalar)", referenced from:
      _atg___lshift__ in _x010.o
  "at::__lshift__(at::Tensor const&, at::Tensor const&)", referenced from:
...
...
...
...
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for torch::optim::RMSprop", referenced from:
      torch::optim::RMSprop::RMSprop(std::__1::vector<torch::optim::OptimizerParamGroup, std::__1::allocator<torch::optim::OptimizerParamGroup> >, torch::optim::RMSpropOptions) in _x010.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for torch::optim::Optimizer", referenced from:
      torch::optim::Optimizer::Optimizer(std::__1::vector<torch::optim::OptimizerParamGroup, std::__1::allocator<torch::optim::OptimizerParamGroup> >, std::__1::unique_ptr<torch::optim::OptimizerOptions, std::__1::default_delete<torch::optim::OptimizerOptions> >) in _x010.o
      torch::optim::Optimizer::~Optimizer() in _x010.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@HaoboGu ,
can you just try go run .  instead of go build? and try to setup a simple example to test that gotch is working in your system first.
Something like:
mkdir test && cd test
go mod init github.com/SOME_USER/test
go get<EMAIL_ADDRESS>touch main.go

package main
import (
	"fmt"

	"github.com/sugarme/gotch"
	ts "github.com/sugarme/gotch/tensor"
)

func main() {

xs := ts.MustRand([]int64{3, 5, 6}, gotch.Float, gotch.CPU)
fmt.Printf("%8.3f\n", xs)
fmt.Printf("%i", xs)

}

go run .

@sugarme
I got the same error when I ran go run .

@HaoboGu,
Do you happen to have Pytorch Python installed in your system?

@HaoboGu,
Do you happen to have Pytorch Python installed in your system?

@sugarme
Sure, I installed pytorch 1.7.0

@HaoboGu,
I had library linking incorrectly but the other way around, occuring with Python instead of Go. Anyway, I realize that there's DYLD_LIBRARY_PATH in MacOS instead of LD_LIBRARY_PATH so can you try to export:
export DYLD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GOTCH_LIBTORCH/lib"

@sugarme ,
I fixed previous problems by reinstall libtorch mac cpu version, but when I ran go run . or go build, another error occured:
/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.10/libtch/tensor.go:100:20: cannot use c_ptr (type *_Ctype_long) as type *_Ctype_longlong in assignment
/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.10/libtch/tensor.go:142:54: cannot use cindexes (type *_Ctype_long) as type *_Ctype_longlong in assignment
/usr/local/Cellar/go/1.15.6/pkg/mod/github.com/sugarme/gotch@v0.3.10/libtch/tensor.go:151:53: cannot use cindexes (type *_Ctype_long) as type *_Ctype_longlong in assignment

@HaoboGu,
Those errors actually bugs in library although it compiled Okay in Linux. Please do change the following in your machine to see how you go. I will fix and release new version if running well.
Change the following lines in<EMAIL_ADDRESS>// line 99: c_ptr := (*C.long)(ptr)
c_ptr := (*C.longlong)(ptr)

// line 140: cindexes := (*C.long)(indexes)
cindexes := (*C.longlong)(indexes)

// line 149: cindexes := (*C.long)(indexes)
cindexes := (*C.longlong)(indexes)

Please let me know whether this works? Thanks

@sugarme

It works! Thank you for your kind and patient reply!
I'm closing this.


pythonPackages.seekpath: mark as broken

Motivation for this change
it's breaking a few other packages
  Long log
builder for '/nix/store/ha7m938bngggxm9n0k0dgczk2j454l59-python2.7-seekpath-1.8.4.drv' failed with exit code 1; last 10 log lines:
    File "/build/seekpath-1.8.4/seekpath/test_paths_hpkot.py", line 366, in base_test
      self.assertEqual(res['bravais_lattice_extended'], ext_bravais)
  AssertionError: 'oI3' != 'oI2'

  ----------------------------------------------------------------------
  Ran 69 tests in 1.138s

  FAILED (failures=1, skipped=2)
  Test failed: <unittest.runner.TextTestResult run=69 errors=0 failures=1>
  error: Test failed: <unittest.runner.TextTestResult run=69 errors=0 failures=1>
builder for '/nix/store/ihi4s44w9fbidxjfkgrgcnxfz9xjh2n8-python3.7-aiohttp-2.3.10.drv' failed with exit code 1; last 6 log lines:
  unpacking sources
  unpacking source archive /nix/store/5dzzs2c8hhrmychljldmjrkgcf3hilgd-aiohttp-2.3.10.tar.gz
  source root is aiohttp-2.3.10
  setting SOURCE_DATE_EPOCH to timestamp<PHONE_NUMBER> of file aiohttp-2.3.10/setup.cfg
  patching sources
  substitute(): ERROR: file 'pytest.ini' does not exist
cannot build derivation '/nix/store/z6zlh6hgj2088zvfwzcjdy03yl3j2p9m-python3.7-pytest-aiohttp-0.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/1ba1k9sxgs5sy17jxgxvvvr1r7dmgpvl-python3.7-aiohttp-jinja2-0.15.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/w5kizcvjky785w5w41yifyvndhgijd7c-appdaemon-3.0.5.drv': 2 dependencies couldn't be built

I tried to fix, but.. not really sure if the package is actually in an invalid state or not, so I'm just marking it broken.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

We typically don't mark it broken in case a test fails. When we can't fix it we just skip it. Yes, there's a risk, but since test coverage is hardly ever 100% that risk is always present.

Closing for preference of other PR


Update pom.xml

Added support for javadoc:javadoc goal

@mgrand Can you add this in build.gradle rather?
The pom.xml file is autogenerated so this change will be overwritten.
+1 on adding JavaDoc

I don't do gradle. Someone else will have to do that


slack multi-config

slack now accepts + generates request for multiple configurations.
Required for all non-trivial PRs

[X] Rebased/mergable
[X] Tests pass
[X] CHANGELOG.md updated

Oh and it looks like there is one more Fatal -> Fatalf change that needs to be made.


[Improvement] Optimize count operation for iceberg

Proposed changes
Iceberg has its own metadata information, which includes count statistics for table data. If the table does not contain equli'ty delete, we can get the count data of the current table directly from the count statistics.
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.16 seconds
stream load tsv:          513 seconds loaded<PHONE_NUMBER>9 Bytes, about 139 MB/s
stream load json:         20 seconds loaded<PHONE_NUMBER> Bytes, about 112 MB/s
stream load orc:          65 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select:          29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size:<PHONE_NUMBER>1 Bytes

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.44 seconds
stream load tsv:          515 seconds loaded<PHONE_NUMBER>9 Bytes, about 138 MB/s
stream load json:         20 seconds loaded<PHONE_NUMBER> Bytes, about 112 MB/s
stream load orc:          65 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select:          29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size:<PHONE_NUMBER>7 Bytes

run buildall

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.77 seconds
stream load tsv:          516 seconds loaded<PHONE_NUMBER>9 Bytes, about 138 MB/s
stream load json:         21 seconds loaded<PHONE_NUMBER> Bytes, about 107 MB/s
stream load orc:          65 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size:<PHONE_NUMBER>9 Bytes

run buildall


Added Face emotion training and testing files

Pull Request for PyVerse 💡
Requesting to submit a pull request to the PyVerse repository.

Issue Title
Please enter the title of the issue related to your pull request.
Face Emotion Recognition
closes #988

[X] I have provided the issue title.

Info about the Related Issue
What's the goal of the project?
The goal of this project is to develop a comprehensive face emotion detection and classification system. This system will classify facial expressions into seven categories (sad, happy, surprise, angry, neutral, disgust) and work with both live camera feed and uploaded video files. This aims to enhance the application's ability to analyze and interpret human emotions in real-time.
Describe the aim of the project.
The aim of the project is to create a robust face emotion detection and classification system that can identify seven key facial expressions: sad, happy, surprise, angry, neutral, disgust, and fear. This system is designed to work with both live camera feeds and uploaded video files, providing a real-time analysis of human emotions. By integrating face detection and multi-face tracking capabilities, the project aims to offer accurate and dynamic emotion interpretation, which could have applications in user experience research, mental health monitoring, and interactive systems.

[X] I have described the aim of the project.

Name
Please mention your name.
Shiven Yadav S

[X] I have provided my name.

GitHub ID
Please mention your GitHub ID.
https://github.com/shivenyadavs

[X] I have provided my GitHub ID.

Email ID
Please mention your email ID for further communication.
<EMAIL_ADDRESS>
[X] I have provided my email ID.

Identify Yourself
Mention in which program you are contributing (e.g., WoB, GSSOC, SSOC, SWOC).
GSSOC

[X] I have mentioned my participant role.

Closes
Enter the issue number that will be closed through this PR.
*Closes: #988 *

[X] I have provided the issue number.

Describe the Add-ons or Changes You've Made
Give a clear description of what you have added or modified.
Describe your changes here.
Model Training (train.py):
Implemented a script to train the facial emotion recognition model using a dataset with labeled images for various emotions. The model learns to classify expressions by analyzing the features in the training images.
Added data preprocessing steps, such as resizing images and normalizing pixel values, to prepare the dataset for training.
Configured model architecture, training parameters (like batch size and epochs), and an optimizer to maximize accuracy.
Face Detection and Emotion Classification (test.py):
Integrated face detection using haarcascade.xml in test.py, enabling real-time detection of faces through a camera feed.
Loaded the trained model to classify detected faces into emotion categories based on the trained model’s predictions.
Developed a display feature that shows the recognized expression label (e.g., "happy" or "sad") directly on the screen above the detected face.
Dataset Utilization:
Set up a structured train and test dataset with labeled images for different facial expressions to support model training and testing.

[X] I have described my changes.

Type of Change
Select the type of change:

[] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[] Code style update (formatting, local variables)
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[] This change requires a documentation update

How Has This Been Tested?
Describe how your changes have been tested.
*Model Training Validation:
After training the model in train.py, evaluated its performance on the test dataset by calculating accuracy and checking confusion matrices to ensure the model accurately classifies each emotion.
Adjusted parameters as necessary to optimize accuracy, then retrained and re-evaluated.
Real-Time Emotion Detection Test (test.py):
Ran test.py to test real-time face and emotion detection using a live camera feed.
Tested the system with various facial expressions to verify that the correct emotion label appears on the screen.
Adjusted face detection and classification parameters to ensure smooth and accurate detection in different lighting conditions and with multiple faces.
Edge Cases:
Tested edge cases, such as partial face visibility and different angles, to ensure consistent and reliable emotion recognition.
Verified the model’s stability by running extended tests with continuous camera feed to check for performance issues or crashes..*

[X] I have described my testing process.

Checklist
Please confirm the following:

[X] My code follows the guidelines of this project.
[X] I have performed a self-review of my own code.
[X] I have commented my code, particularly wherever it was hard to understand.
[X] I have made corresponding changes to the documentation.
[X] My changes generate no new warnings.
[X] I have added things that prove my fix is effective or that my feature works.
[X] Any dependent changes have been merged and published in downstream modules.

@UTSAVS26 please check the changes.

@UTSAVS26 if there are any changes please let me know

@shivenyadavs why the code is commented?

those are alternate codes which will work.

The main working code is not commented please check

those are alternate codes which will work.

So create a separate file and add a readme file for telling the different ways to do the same work.

I have made the changes in emotion detection readme.md please have a look

@shivenyadavs don't add the project in the main folder, move it in some dedicated folders.

done please check

@shivenyadavs add the project in dedicated folders not into the main repo.


[BEAM-111] Move WritableCoder to hadoop-common

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @amitsela
This is an oldie, but now with the module for hadoop-common we can move this one there.
Please check the jackson dependency, I am not sure if I should make it transitive as I do.
Thanks.

Changes Unknown when pulling 14dfc21567464fbc62a332b97f67bad928d50a18 on iemejia:BEAM-111-move-writablecoder into ** on apache:master**.

Ok fixed after comments and rebased.

Coverage increased (+0.003%) to 69.187% when pulling 799cc6f32aadfab12c56c26bde1d006ddefcd1aa on iemejia:BEAM-111-move-writablecoder into 2a23374601d8f6079d025f41dc281a302a865512 on apache:master.

Rebased because there was a merge issue after changes in master.

Coverage increased (+0.007%) to 69.196% when pulling da8b0ad39d2d71cfc7fa9a90e90aa6be474d6de5 on iemejia:BEAM-111-move-writablecoder into d66029cafde152c0a46ebd276ddfa4c3e7fd3433 on apache:master.

Coverage increased (+0.01%) to 69.203% when pulling 5eb24223afadd1ebb169426a188796068631793d on iemejia:BEAM-111-move-writablecoder into d66029cafde152c0a46ebd276ddfa4c3e7fd3433 on apache:master.

@iemejia before I merge this, are you and @diptikul synchronized about #1994 ?
It looks as if Coders are moved around there as well.
CC: @dhalperi @ssisk @jbonofre

We are not synchronized because these are different subjects, and this PR is not a blocker for that PR, what I am doing for the moment is just looking for Hadoop classes that are needed in more than one project and moving those to hadoop-common.
What I saw in the review is that the reviewers proposed to do this changes (use or enhance hadoop-common) afterwards, to add less noise to the current InputFormat PR.

Ok, I guess it'll get merged into #1994 eventually. Thanks @iemejia
Merging.

Thanks.


The .desktop spec requires keeping comments, so I can use a Python library that supports them listed below.

It's this one:
https://codeberg.org/JakobDev/desktop-entry-lib
I had to go through the developer's desktop file editor project to find the library that was in use but I don't remember the code, to be fully transparent on this. Hopefully it's ok because I don't remember any of the rest of the code I read even though the project is under GPLv3 (I'm planning on switching to it where necessary, anyway). Even then, I purposefully skimmed it to not internalize it.
This library is under the BSD 2-clause license.
I may have to figure out how it's keeping comments and extract that so I can have my config files keep their comments if I can't figure out anything else.

Or maybe I could use ConfigUpdater? https://github.com/pyscaffold/configupdater
Only issue is it doesn't support duplicate sections and keys, a major issue considering one of the Gnome .desktop files has a duplicate key.


Update custom set test order

fixes #283
Note that I've done this as 2 commits:
Commit 1 does not break existing solutions and does as much as possible to conform to the canonical tests. This also uncovered a bug in the example solution, which this fixes.
Commit 2 will break existing solutions, as it changes #empty to have the meaning from the canonical tests, and renames the existing method to #clear
The non-canonical tests for this track have been left (they are positioned after the canonical tests)
I've stayed with the style of the test suite to keep things consistent
Happy to discuss, squash or change any of this!

this isn't quite right - see comment on issue #283

see issue #283 for discussion

Nice! Great job! I think changes are ok to be merged. I'll wait a little bit for you to squash your commits and it'll be merged.

I would like to split each test case, to reflect better the test cases exposed in the canonical tests document custom-set.json
Let me explain: in the canonical tests, there are three cases for the contains method, with these descriptions:

nothing is contained in an empty set
when the element is in the set
when the element is not in the set

But, in the test suite, there is only a spec (it block) that covers the three cases:
xit('can test for a member', function() {
    var actual = new CustomSet().contains(1);
    expect(actual).toBe(false);

    var actual2 = new CustomSet([1, 2, 3]).contains(1);
    expect(actual2).toBe(true);

    var actual3 = new CustomSet([1, 2, 3]).contains(4);
    expect(actual3).toBe(false);
});

I think the test suite will be clearer for exercism users if we split it blocks in this way:
xit('nothing is contained in an empty set', function() {
    var actual = new CustomSet().contains(1);
    expect(actual).toBe(false);
});
xit('when the element is in the set', function() {
    var actual = new CustomSet([1, 2, 3]).contains(1);
    expect(actual).toBe(true);
});
xit('when the element is not in the set', function() {
    var actual = new CustomSet([1, 2, 3]).contains(4);
    expect(actual).toBe(false);
});

But, test cases weren't splitted in the begginning, so I think this code is good to be merged. You've made a great job. Splitting the test cases could be a totally different PR. It'd be nice if you can work on it, if you want to.

Yes, I came to the same conclusion about the need to split the test cases - I also think it's out of scope of the original issue. I'll raise an issue so that it's visible, and I'm more than happy to work on it.

squashed and ready to go

Great!!!!
Thanks so much @ireddick


add zw0610 to kubeflow member

Hi, as I would like to keep contributing to kubeflow, please consider my request to join the org. Here is my contribution:
https://github.com/kubeflow/mpi-operator/pull/332
https://github.com/kubeflow/mpi-operator/pull/335
@terrytangyuan

/lgtm
/approve
Thank you for your contribution!


Agents issuing noop commands stop working

It's been reported that noop commands are broken in that adding them to a working multi-agent games stops agents from progressing further.

Possibly a problem with the turn based protocol where turn key is not updated on commands that are not legal.


pilot: enable e2e test for validating HTTPS_Gateway with v1alpha3

We currently mount in certificates for the HTTPs section of Gateways , but only test HTTP

may be blocked on https://github.com/istio/istio/issues/5995

Fixed in https://github.com/istio/istio/pull/6402


Added hooks support #836

Pull Request Checklist
Introduces support for hooks
Description
Commit addes the hooks parameter to the sensu::check define
Related Issue
Fixes #836
Motivation and Context
How Has This Been Tested?
Spec tested.
Integration test on PSICK setup.
General

[ ] Update README.md with any necessary configuration snippets

[x] New parameters are documented

[x] New parameters have tests

[x] Tests pass - bundle exec rake validate lint spec

Can you please add a test for us to verify this works in vagrant.

@dzeleski Could you please test that this works for you before we release it. Thanks!!!

Yep will do, working on that now. Might have to get back to you first thing in the morning though as I have to leave around 6pm EST tonight.

Inputs and outputs below, let me know if you need anything else tested. @ghoneycutt
Input:
sensu::check{ 'cl-pup-healthcheck-puppet_pemaster':
    command     => '/opt/sensu/embedded/bin/ruby /opt/sensu/embedded/bin/check-puppet-healthcheck.rb',
    handlers    => 'linux_handler',
    interval    => 300,
    subscribers => 'puppet_pemaster',
    standalone  => false,
    occurrences => 1,
    notify      => Service['sensu-enterprise'],
    hooks       => {
      critical => {
        command => 'test command',
        },
    },
  }

Output:
{
    "checks": {
        "cl-pup-healthcheck-puppet_pemaster": {
            "command": "/opt/sensu/embedded/bin/ruby /opt/sensu/embedded/bin/check-puppet-healthcheck.rb",
            "handlers": [
                "linux_handler"
            ],
            "hooks": {
                "critical": {
                    "command": "test command"
                }
            },
            "interval": 300,
            "occurrences": 1,
            "standalone": false,
            "subscribers": [
                "puppet_pemaster"
            ]
        }
    }
}

@ghoneycutt Added vagrant test manifests and check on the name of the hooks (I didn't realise that only some specific values are accepted) with relevant spec test.
Rebased and ready to merge (if travis won't surprise me).

Thank you everyone for your help getting this implemented!!
Released in v2.39.0


prometheus-node-exporter: 1.7.0 -> 1.8.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/prometheus/node_exporter/releases.
meta.description for prometheus-node-exporter is: Prometheus exporter for machine metrics
meta.homepage for prometheus-node-exporter is: https://github.com/prometheus/node_exporter
meta.changelog for prometheus-node-exporter is: https://github.com/prometheus/node_exporter/blob/v1.8.0/CHANGELOG.md
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.8.0 with grep in /nix/store/dbwlm5bh2ad67qksbpx8izsffj5mal7h-node_exporter-1.8.0
found 1.8.0 in filename of file in /nix/store/dbwlm5bh2ad67qksbpx8izsffj5mal7h-node_exporter-1.8.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

prometheus-node-exporter

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/dbwlm5bh2ad67qksbpx8izsffj5mal7h-node_exporter-1.8.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A prometheus-node-exporter https://github.com/r-ryantm/nixpkgs/archive/81590a410f3201d206b0276c24d91d304d4f939c.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/81590a410f3201d206b0276c24d91d304d4f939c#prometheus-node-exporter

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/dbwlm5bh2ad67qksbpx8izsffj5mal7h-node_exporter-1.8.0
ls -la /nix/store/dbwlm5bh2ad67qksbpx8izsffj5mal7h-node_exporter-1.8.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

prometheus-node-exporter

Maintainer pings
cc @benley @fpletz @globin @Frostman for testing.

Duplicate of #306540 which has been merged


The type for options.services.unbound.settings is too narrow

Describe the bug
The type for options.services.unbound.settings is too narrow, it precludes valid configs. Case in point,
{
config.services.unbound.settings.server.view = [{
  name = "foo";
  local-data = [
    "\"example.com 100500 IN A <IP_ADDRESS>\""
  ];
}];
}

breaks with

A definition for option services.unbound.settings.server.view."[definition 1-entry 2]" is not of type signed integer or string or boolean or floating point number.

Overriding by something like this fixes it:
{
options.services.unbound.settings.server.view = lib.mkOption {
  type = lib.types.anything;
};
}

I'm sure there are other examples. The problem is the type definition doesn't allow nested attrsets beyond the top level, despite the rest of the code handling that just fine:
https://github.com/NixOS/nixpkgs/blob/3030f185ba6a4bf4f18b87f345f104e6a6961f34/nixos/modules/services/networking/unbound.nix#L125-L135
Unfortunately, I'm not proficient enough with nix modules to fix this properly (and by "properly" I mean use a more sensible type than anything)
Steps To Reproduce
Steps to reproduce the behavior:

Try to add views config to unbound (see silly example above)
Attempt to build the configuration, or simply deep-evaluate (e.g. with :p) config.services.unbound.settings in REPL.
Observe the failure

Expected behavior
I expect my config to work without nix getting in my way.
Screenshots
N/A
Additional context
Reproduced on d946372f5aec57986015fd001d4958a3cac74a0f, but master has the same narrow option type definition.
Notify maintainers
I couldn't find who's supposed to be maintaining the module, so here goes...

@rissson, as you've introduced the type in question
@SuperSandro2000, as you're the last one to touch the module
@ajs124, @das_j since you're listed as unbound package maintainers, thus this may (or may not) be of interest

Metadata
$ nix run nixpkgs#nix-info -- -m
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.77, NixOS, 23.11 (Tapir), 23.11.20240213.d946372`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
---

Note: channel nixpkgs are irrelevant, I'm on flakes.
Add a :+1: reaction to issues you find important.

We probably need to add listOf (attrsOf validSettingsPrimitiveTypes) to https://github.com/NixOS/nixpkgs/blob/3030f185ba6a4bf4f18b87f345f104e6a6961f34/nixos/modules/services/networking/unbound.nix#L131****

I think that approach will inevitably have us playing whack-a-mole with edge cases.
In theory, I would suggest something to the tune of
  options.services.unbound.settings = lib.mkOption {
    type = with lib.types;
      let validTypes = oneOf [ int str bool float (listOf validTypes) (lazyAttrsOf validTypes)];
      in attrsOf validTypes;
  };

But IIUC that has potential to break stuff because lazyAttrsOf is weird.
There's a version of this approach that limits recursion to some arbitrary depth, then we could use attrsOf, but that's very ad-hoc...

Perhaps really something similar to anything is the right type, considering the config generator will bail if something's fatally not right
https://github.com/NixOS/nixpkgs/blob/3030f185ba6a4bf4f18b87f345f104e6a6961f34/nixos/modules/services/networking/unbound.nix#L22
and, furthermore, there's now an additional safety net with unbound-checkconf.
That said, anything's merge behaviour isn't quite what we'd want, so a custom type would be preferable. I can't suggest one, though.

To be clear, just musing out loud, feel free to choose whatever approach seems right. As I said, I'm not too proficient with nix modules.


Second and further file changes not detected on MacOS Mojave

Possibly related to https://github.com/bazelbuild/bazel-watcher/pull/139 and https://github.com/angular/angular/issues/21770
I've tested this using bazel-watcher version 0.7.0 and 0.5.0, on MacOS Mojave.
As an example project I picked java-tutorial from https://github.com/bazelbuild/examples.git .
Checkout that repository, change into the java-tutorial directory and execute ibazel build //:ProjectRunner. Then open the files src/main/java/com/example/{Greeting.java,ProjectRunner.java} in an editor.

Enter an empty line into the first file and save.
ibazel notices and rebuilds the target.
Enter a second empty line into the first file and save.
ibazel does not notice the modification and does not rebuild the target.
Any further edits to this file are not noticed.
Enter an empty line into the second file and save.
ibazel notices and rebuilds the target.
Enter a second empty line into the second file and save.
ibazel does not notice the modification and does not rebuild the target.
Any further edits to this file are not noticed.

We have observed the issue on two separate MacOS Mojave machines. We have observed it with version 0.5.0 and 0.7.0. We have not seen an instance of ibazel working properly on MacOS.

Update: The issue was observed on MacOS Sierra (10.12.6) and Mojave (10.14).
The issue was not observed on Linux, bazel-watcher seems to be working fine on Linux.

The real problem is that I don't have a Mac to debug with. Any ideas/information you have is a thing I would be very happy to help with, I just don't have a box to test on :frowning_face:

@achew22 FreeBSD might be a good alternative as a development platform. It also uses kqueue and you can run it in a VM. At a glance it seems like Bazel and fsnotify support FreeBSD.

Awesome, thanks @dougkoch and @achew22 . I tested this on a Mojave machine and the issue seems to be fixed. :tada:

Woohoo! I'm cutting a v0.8.0 release right now.


yankpad--trigger-snippet-function: First block in snippet must be an org-mode src block

Example yankpad file:
* Category 1
** elfeed                                            :func:e:

when calling with M-x yankpad-insert RET elfeed or M-x yankpad-map RET e, it throws error:

yankpad--trigger-snippet-function: First block in snippet must be an org-mode src block

Okay, I found it.
When calling with function (defun yankpad--trigger-snippet-function (snippetname content) the arguments are:

snippetname: "elfeed"
content: "\n"

So the content should be trim whitespaces first or do not add newline(s) inside ** elfeed   :func:e heading.

Thanks! Should be fixed now.


Implement Scottish Limited Partnership name options

JIRA link
https://companieshouse.atlassian.net/browse/LP-193
Change description
Implement 'Scottish Limited Partnership name' options on 'Limited Partnership Name' page
Work checklist

[x] Tests added where applicable
[ ] UI changes look good on mobile
[x] UI changes meet accessibility criteria

Merge instructions
We are committed to keeping commit history clean, consistent and linear.  A BREAKING CHANGE can be part of commits of any type,
types other than fix: and feat: are allowed, for example build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others,
footers other than BREAKING CHANGE: <description> may be provided.

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found

CI: No security warnings found


luci-mod-admin-full: check for password length on change

Add configrable password length check on password change.
The default password length is 6, because the busybox passwd command has the same length check.
Signed-off-by: Florian Eckert<EMAIL_ADDRESS>

There could be an optional password length checker, but having that on by default would deviate from the behaviour on console.
This PR would change LuCI to require 6 chars, but the user could still use console to change password to be just one character. (I tested and to my surprise a password of just "a" was accepted...)
Yes, busybox warns about a weak or short password, but "root" is allowed to have a weak password and busybox does change the password for "root" despite the displayed weak/short warning. As most users run Openwrt/LEDE/LuCI as "root", the check on console is just for display.
https://git.busybox.net/busybox/tree/loginutils/passwd.c#n73
https://git.busybox.net/busybox/tree/libbb/obscure.c#n174
One solution might be that

default min length is 1
the min. length requirement is displayed only if that config option is set to something > 1

In practice, the translatef("Minimal password length is %s characters", pw_length) should be moved away from the password option itself, to be inside an if statement and probably be a read-only dummy option field.

Or the text might be changed to something like guidance "Password should be at least %s characters." and the actual length check dropped. Then the default could be 6.
In any case, displaying and checking the minimum length twice makes no sense. The two passwords need to match, so just the first password needs to checked, if any.
As the "root" can change the config option itself, I am not quite sure how useful this check in LuCI would be in the big picture.

If the user will start the router for the first time.

A normal user will configure the router over the web. He will be prompted to change the password on first login. I think it is good practice to help him to select a good password. A check for the password length in the web would be a good help at first. In the future, if it is wanted, it is also possible to check the strength of the password with an password checker.

On the other side a power user will configure the router over the console and i think he knows the importance for a strength password.

@hnyman i would prefer your first solution suggestion

- default min length is 1
- The min. length requirement is displayed only if that config option is set to something > 1
- In practice, the translatef("Minimal password length is %s characters", pw_length) should be moved away from the password option itself, to be inside an if statement and probably be a read-only dummy option field

Does this mean this config option gets ignored?
config BUSYBOX_DEFAULT_PASSWORD_MINLEN
	int
	default 6

Source: https://github.com/lede-project/source/blob/84da2a677a7133426829a05954af64dd533bf7a5/package/utils/busybox/Config-defaults.in#L184

Does this mean this config option gets ignored?

Yes and no.

Busybox uses that to decide when to raise the warning about bad password
But for "root" the result of the warning is ignored (although the warning is displayed), and the password is changed despite being weak.

And if we patch busybox? We already have quite some patches to busybox, so another one wouldn't hurt. I think it would be a nice security improvement.

	if (ENABLE_FEATURE_PASSWD_WEAK_CHECK
	 && obscure(orig, newp, pw)
	 && myuid != 0
	) {
		goto err_ret; /* non-root is not allowed to have weak passwd */
	}

I'd like to path out the line reading && myuid != 0 from  busybox/loginutils/passwd.c to have a consistent behaviour on LuCI and CLI. Objections?

Sure you can propose that to the main source in LEDE (and Openwrt?).

I proposed patching passwd here: http://lists.infradead.org/pipermail/lede-dev/2017-February/006032.html

Did you check that "obscure_msg" function more closely?
It is rather complicated, and the actual enforced minimum length depends on the complexity of the password. Based on my own current root password, the minimum length would be 8, if I look at the rules correctly.

That is correct, my test cases did not cover a complex 6 char password, only 8 and longer. That is bad.  Let's raise minimum length to 8 chars (if the change ever gets accepted, I have my doubts).

Ok, after the ML discussion, I am in favor of rejecting this PR for the sake of consistency with the CLI.
Citing myself from the ML:

My goal was to have consistency at LuCI and CLI. I see how enforcing passwords of a particular kind, as well as enforcing passwords at all, is not an engineering decision. I have no problem with this patch being rejected.
So, since we decided to not enforce anything regarding passwords for the root user on CLI,  I think we should also not enforce anything regarding passwords for the root user on LuCI. Still, I am concerned about consistency foremost.

Let's close this. Not going to get merged, based on the active LEDE mailing list discussion.

@hnyman ok i agree too.
But would it not a good practice to show a notice that the password is weak or to short on password change?

That is shown when you change password in console, but I don't think that the warning from busybox can be brought back to LuCI.
I will probably add a guidance text in Luci, that password should be at least 6-8 chars long and contain various character types. Or something like that.


Bug: Platform specification is not complete

In runtime I see that Platform constant has multiple useful properties that are missing from the specification, e.g. Platform.isWin
From the other hand, Platform.isMacOS is mentioned in the specification.
Such kinds of checks are required sometimes to filter out invalid filename characters. Windows' set of invalid characters are bigger than on other platforms.

Properties missing from the spec means it's not intended to be used by plugins - either we anticipate that it can change in the future, or we don't feel like it's robust enough to support it indefinitely.
I've decided to add some of the variables here for public use, but not all of it.


initial

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@61bd0f4). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main       #1   +/-   ##
=======================================
  Coverage        ?   82.74%
=======================================
  Files           ?        5
  Lines           ?      226
  Branches        ?        0
=======================================
  Hits            ?      187
  Misses          ?       39
  Partials        ?        0

Flag
Coverage Δ

unittests
82.74% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 61bd0f4...0137fee. Read the comment docs.


Use URL and published date to generate ID

Addressing issue #354.

This project is not maintained anymore. You should consider forking the project if you still need it.


chore(docs): fix typo in union document page

As it's a tiny copy paste miss, I didn't open a issue for it.

Thank you!


Hook in distributed framework

Our goal is to be able to run a number of transformations simultaneously.

Addressed


use where inside of with

I have a Model and I want to use where inside it.
for example I have a User Model and Post Model and I want to use this query
.query()
.with("user")
.where("user.username","like","%john%")
and my result must be All post that owner username is john

See Adding Runtime Constraints
const posts = await Post
  .query()
  .with('user', (builder) => {
    builder.where('username', 'like', '%john%')
  })
  .fetch()

That's wrong sorry, I think you'll need whereHas
const posts = await Post
  .query()
  .whereHas('user', (builder) => {
    builder.where('username', 'like', '%john%')
  })
  .fetch()

tnx @webdevian for your answer.
know how can orderBy whit username in the same example.


Server Output text doesn't refresh

Description
Hi.  When I execute a PL SQL Code, the server output doesn't refresh the output (text).  I have to run another query and then when I go to server output I can see the log.
https://github.com/dbeaver/dbeaver/assets/37661233/d56d2607-4ba2-4c23-aa71-98c1bde12d6e
DBeaver Version
Community Edition 23.2.5
Operating System
Windows 10 Pro
Database and driver
Oracle 12c  -  Oracle JDBC driver
Steps to reproduce
Run a call to a procedure that writes to dbms and then see the output.  I maked a video =)
Additional context
No response

Hello @renezito
Thanks for the bug report.
Could you please check the latest Early Acces version? https://dbeaver.io/files/ea/
Issue is fixed here: https://github.com/dbeaver/dbeaver/issues/21902


Fixing linkage error on Linux (issue #103).

@malespiaut
CMake Error at CMakeLists.txt:98 (find_package):
Sorry but I can't merge with errors.


Error: send ENETUNREACH <IP_ADDRESS>:5353

Hi, I have installed the latest version from Homebridge (Docker) and from Homebridge-miot.
For some time now, the Homebridge shuts down and restarts.
These errors are then visible in the log.
[7/6/2021, 10:35:20 PM] [miot] [MIVentilator] Device setup finished! Device ready, you can now control your device!
[7/7/2021, 12:31:25 AM] Error: send ENETUNREACH <IP_ADDRESS>:5353
    at doSend (dgram.js:714:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:435:18)
    at afterDns (dgram.js:660:5)
    at processTicksAndRejections (internal/process/task_queues.js:83:21)
[7/7/2021, 12:31:25 AM] Got SIGTERM, shutting down Homebridge...

[MIVentilator] Device setup finished! Device ready, you can now control your device!
[7/9/2021, 1:50:32 PM] Error: send ENETUNREACH <IP_ADDRESS>:5353
    at doSend (dgram.js:714:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:435:18)
    at afterDns (dgram.js:660:5)
    at processTicksAndRejections (internal/process/task_queues.js:83:21)
[7/9/2021, 1:50:32 PM] Got SIGTERM, shutting down Homebridge...

[miot] [MIVentilator] Device setup finished! Device ready, you can now control your device!
[7/9/2021, 1:52:46 PM] Error: Device has no identifier yet, handshake needed
    at DeviceInfo.enrich (/homebridge/node_modules/homebridge-miot/node_modules/miio/lib/network.js:273:10)
    at Socket.<anonymous> (/homebridge/node_modules/homebridge-miot/node_modules/miio/lib/network.js:165:13)
    at Socket.emit (events.js:375:28)
    at UDP.onMessage (dgram.js:941:8)
[7/9/2021, 1:52:46 PM] Got SIGTERM, shutting down Homebridge...

Looks like the ip you specified there cannot be reached, this looks to me like a network issue. Maybe the docker container is in a different subnetwork?

Thank you for your answer.
The Docker container and devices are all on the same subnet.

Well the error clearly states that an ip cannot be reached, not sure from what the ip is but you need to investigate why it cannot be reached.

All right, thank you.


Added Template Types for SPA projects

Description
Fixes for issue #435. Introduces two template types for SPA App and SPA Page. Provides additional cleanup based on what frontend module is chosen.
Related Issue
Issue #435
Motivation and Context
Provides appropriate template types for SPA projects.
How Has This Been Tested?
Built and deployed sample projects from archetype
Screenshots (if appropriate):

Above is a screenshot of the create template wizard for a SPA project.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

Looks good to me


INSTALLATION

I found your documents very insufficient and confusing to install.
What are the exact steps?
Which "setup.py" must be run?
the one in OpenPCDet repo, or the one in this repo?
Should I replace the files in your repo with the same ones in OpenPCDet?
please explain it in more detail

like I can install OpenPCDet with the following tutorial, and it installs pcdet version 0.6.0.
https://medium.com/@alifyafebriana/setting-up-3d-open-source-openpcdet-with-anaconda-a-step-by-step-guide-66126107215
but according to your INSTALL.md, should I first install OpenPCDet and then run the setup.py in your repo? or should I even install OpenPCDet? buy running setup.py in your repo it will be installed, please explain it in more detail and guide me through it step by step.
thank you very much.

Hi,
Make a conda environment, install OpenPCDet following the tutorial, then run setup.py in this repo.
Thanks.


Test fails for Django 1.10a1 - default django auth backend treats credentials of inactive user as wrong ones

To reproduce install latest django and run:
pip install -U django==1.10a1
and then run tests.

See #152.

Pulled #152, closing.


Support for additional ANSI String types for client-side certificate validation

Issue # (if available):
UTF8String is the recommended type, but not the only type that is used. Adding support for the other common types.
Description of changes:
Add other types that are supported, the test for checking several top level domains would have caught this and is probably a good test to add.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Created #761 to address the encoding question


Install failed on Ubuntu server 16.04.1

Hi there,
Is this a result of me using 16.04.1?
TASK [azuracast-db : Install Initial Database] *********************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "unable to find /home/engineering/.my.cnf. Exception message: (1045, "Access denied for user 'root'@'localhost' (using password: NO)")"}
It was the only thing to fail
If not let me know if I can help at all?
Cheers
Colm

@colmrs Based on the error message, it looks like you're not running the initial installer as the actual root user. Unfortunately, due to a quirk in how Ubuntu 16.04 handles the MySQL root account, you have to actually be running as the root user (not just a sudo-capable user) to do the initial installation. Any updates from then on can be done as a sudo-capable user.

Yep I have just came to the same conclusion, just creating a new VM to try that on, as trying to then run the retry as root fails. Will close the issue if all is well. (Also sorry for not following the instructions) (would it be a good idea to add the "sudo passwd root" and "su" commands to the install so people dont make my same mistake?)
Thanks!

User error, sorry about that!


update to new version of GalSim

This uses a new version of the stack in order to update the underlying GalSim. This should be a significant improvement to memory use.

https://github.com/LSSTDESC/imSim/commit/e18c357685e9b37887cd0b845d09246cb908f3bf
Stack update was following the procedure in this imSim commit.


API_KEY issue

Describe the bug
Since the version changed, I decided to add my API_KEY as suggested (I got it from speedtest-tracker.dev).
So here is my Portainer stack :
version: '3.8'
services:
speedtest-tracker:
image: lscr.io/linuxserver/speedtest-tracker:latest
container_name: speedtest-tracker
ports:
- "8091:80"
environment:
- PUID=1033
- PGID=1000
- DB_CONNECTION=sqlite
- API_KEY=base64****
volumes:
- /volume1/docker/speedtest-tracker:/config
labels:
- "com.centurylinklabs.watchtower.enable=true"
security_opt:
- no-new-privileges:true
restart: on-failure:5
Here is the result :

I also tried to add the API_KEY directly in Portainer without success.

I also tried a fresh install but the result is the same.
It looks like whatever I do, the API_KEY is not used. Is there any other manipulation to do ?

I was able to get it working following the note in https://github.com/alexjustesen/speedtest-tracker/releases/tag/v0.20.0
 speedtest-tracker:
    image:  lscr.io/linuxserver/speedtest-tracker:latest
    container_name: speedtest-tracker
    networks:
      - misc
    volumes:
      - /portainer/Files/AppData/Config/speedtest-tracker-alexjustesen:/config
    ports:
      - 8765:80
    environment:
        - PUID 1000
        - PGID 1000
        - OOKLA_EULA_GDPR=true
        - DB_CONNECTION=sqlite
        - PUBLIC_DASHBOARD=true
        - APP_KEY=base64:clCteBaLyIwfNFKMejtH+KezhXR***********

Do check your naming , the env variable must be named APP_KEY.  It looks correct in your screenshot, but is not correct in your "portainer stack"

I was able to get it working following the note in https://github.com/alexjustesen/speedtest-tracker/releases/tag/v0.20.0
 speedtest-tracker:
    image:  lscr.io/linuxserver/speedtest-tracker:latest
    container_name: speedtest-tracker
    networks:
      - misc
    volumes:
      - /portainer/Files/AppData/Config/speedtest-tracker-alexjustesen:/config
    ports:
      - 8765:80
    environment:
        - PUID 1000
        - PGID 1000
        - OOKLA_EULA_GDPR=true
        - DB_CONNECTION=sqlite
        - PUBLIC_DASHBOARD=true
        - APP_KEY=base64:clCteBaLyIwfNFKMejtH+KezhXR***********

Do check your naming , the env variable must be named APP_KEY. It looks correct in your screenshot, but is not correct in your "portainer stack"

I was having the same issue, this seemed to have fixed it - thanks!

I was able to get it working following the note in https://github.com/alexjustesen/speedtest-tracker/releases/tag/v0.20.0
 speedtest-tracker:
    image:  lscr.io/linuxserver/speedtest-tracker:latest
    container_name: speedtest-tracker
    networks:
      - misc
    volumes:
      - /portainer/Files/AppData/Config/speedtest-tracker-alexjustesen:/config
    ports:
      - 8765:80
    environment:
        - PUID 1000
        - PGID 1000
        - OOKLA_EULA_GDPR=true
        - DB_CONNECTION=sqlite
        - PUBLIC_DASHBOARD=true
        - APP_KEY=base64:clCteBaLyIwfNFKMejtH+KezhXR***********

Do check your naming , the env variable must be named APP_KEY. It looks correct in your screenshot, but is not correct in your "portainer stack"

I was having the same issue, this seemed to have fixed it - thanks!

 Thank you very much for the tips.

Thanks @mitchv2020


system: add new subcommand "migrate"

it is useful to migrate existing containers to a new version of
podman.  Currently, it is needed to migrate rootless containers that
were created with podman <= 1.2 to a newer version which requires all
containers to be running in the same user namespace.
Closes: https://github.com/containers/libpod/issues/2935
Signed-off-by: Giuseppe Scrivano<EMAIL_ADDRESS>

Need to update command.md with an entry for this command.

do I need to add it?  It is a subcommand of system

Any chance to add a test? Seems hard to pull off, but thought I'd ask.

we would need to have an old version of podman preinstalled

I'll let @mheon make the call on the "should it be on commands.md" or not.  I think yes.  Currently we have separate listings for "podman container *" and "podman volume *", but we don't do the same for "podman system *".
The test sounds like a headache, I'd not worry about it then.  Sounds like you were considering the same roadblock that I was.

@giuseppe @TomSweeneyRedHat If system renumber is in the commands page, we add this; if it's not, we don't. I think we ought to match what the existing similar commands do.

Is this something we should run in the postinstall?  How are we going to tell individual users that we need to run this.  Can we do a check in the podman command?

We want this only for rootless, I think

rebased and addressed all the comments

commit message/PR summary says podman <= 1.12 -- probably meant 0.12 ?

I think that should be 1.1.2?

Possibly 1.2

Is this something we should run in the postinstall? How are
we going to tell individual users that we need to run this. Can
we do a check in the podman command?

RPM scriptlets are run server-side on Silverblue, and even on the legacy Workstation there's the problem of migrating rootless containers. However, we can assume that updates are applied offline (ie., the podman binary won't get swapped out on a running system). Silverblue guarantees it, and is what's encouraged on legacy Workstation.
I was wondering if I could call podman system migrate before podman start? Do I need to be careful about calling it only once after the update? Or can I hard code a call in there, and rely on subsequent calls turning into NOPs?

We want to make sure that no containers are running when it does run, but it seems like it ought to be safe to run more than once given that remains true.

Needs a rebase to get the SELinux fixes.

tests are happy!

I'll hit my comments once this merges in a separate PR, let's get this landed.
LGTM

@giuseppe  You picked up merge conflicts.

@giuseppe You picked up merge conflicts.

rebased again

LGTM and happy green test buttons.

tests are green again

/lgtm
I'll submit a followup over the weekend or on monday to hit my nits


APIs, Runtimes and Data Planes should always be sorted in the same order

APIs, Runtime and Data Planes come in varying orders in the Control Plane. It would be nicer if they would also be shown in the same order (easier to understand and use).

This has been completed and tracked through https://itrac.eur.ad.sag/browse/WCO-2747


Index issue with generating data script [discussion]

See https://github.com/dask/dask/issues/8983

Thanks for bringing this up, I had forgotten about this issue. Do you have a sense for how big a problem this is for us when running these benchmarks? My guess is that if, for example, partitions are all 103 MB instead of 100 MB, that won't impact us significantly.

When we took a look at this with Ian, it was between 5-10% increase in memory footprint. See comment https://github.com/dask/dask/issues/8983#issuecomment-1110062646

Okay, good to know. My question is still does this lead to problems when running h2o benchmarks? Do we need to devote time to resolving https://github.com/dask/dask/issues/8983 in order to make progress on h2o benchmarks, or is it just a related issue to be aware of (i.e. a valid bug, but not mission critical)? I think it's not a blocking issue, but I could be wrong

My question is still does this lead to problems when running h2o benchmarks?

Hard to know, we haven't compared runs with and without this problem. Things will still run, as seen in the runtime, but we don't know how it affects.

Do we need to devote time to resolving https://github.com/dask/dask/issues/8983 in order to make progress on h2o benchmarks, or is it just a related issue to be aware of (i.e. a valid bug, but not mission critical)? I think it's not a blocking issue, but I could be wrong

I don't think it's a blocking issue either, to get started, but it's something to be aware.


Allow partial translate file reads.

Overview
This commit fixes an issue where translation LogEntry must be read in its entirety, however, large entries can exceed the buffer size. This has been changed so that partial entries reads are allowed.
The LogEntry.ReadFrom() may still generate large byte slices during reads of large individual fields or keys.
Fixes #1860
Pull request checklist

[x] I have read the contributing guide.
[x] I have agreed to the Contributor License Agreement.
[x] I have resolved any merge conflicts.
[x] I have included tests that cover my changes.
[x] All new and existing tests pass.
[x] Make sure PR title conforms to convention in CHANGELOG.md.
[x] Add appropriate changelog label to PR (if applicable).

Code review checklist
This is the checklist that the reviewer will follow while reviewing your pull request. You do not need to do anything with this checklist, but be aware of what the reviewer will be looking for.

[ ] Ensure that any changes to external docs have been included in this pull request.
[ ] If the changes require that minor/major versions need to be updated, tag the PR appropriately.
[ ] Ensure the new code is properly commented and follows Idiomatic Go.
[ ] Check that tests have been written and that they cover the new functionality.
[ ] Run tests and ensure they pass.
[ ] Build and run the code, performing any applicable integration testing.
[ ] Make sure PR title conforms to convention in CHANGELOG.md.
[ ] Make sure PR is tagged with appropriate changelog label.

@jaffee Yes, binary.Read() will wait until it receives all the bytes necessary. I feel like there was a reason I went with this approach originally but honestly I can't remember now. I think I wanted readers to always start fetching from the start of an entry but that's really not necessary.

@jaffee Can you +1 this updated branch and I'll merge it in.


Eliminate trailing space on some OCLC numbers

We find that when OCLC numbers with prefixes in field 035 are converted, they have a trailing space.  This does not happen with OCLC numbers without a prefix.
035 __ |a (OCoLC)ocm04212209
bf:identifiedBy [ a bf:Local ;
            bf:source [ a bf:Source ;
                    rdfs:label "OCoLC" ] ;
            rdf:value "ocm04212209 " ],

Please eliminate this space, e.g. by using normalize-space() on the output.
The space causes problems for us when we query OCLC numbers in SPARQL.  Typically, BF triples for a converted record contain multiple instances of the OCLC number, often with and without a prefix.  This causes duplicate lines in the output, and when we try to use DISTINCT to eliminate dups, this fails.  It is easy enough to remove the prefix, which is rightly part of the data string, but the trailing space causes problems.  As strings, "04212209" and "04212209 " are not identical and won't de-dup.  Since the trailing space is not part of the data, it would be best to remove it during conversion.

I can't reproduce this and the chopPunctuation template already removes a trailing space so that's not the solution. How are you executing the converter? Also, you posted Turtle, are you sure this is in the RDF/XML output?
Note: there's a unit test for 035$a so I changed the data to use your example above (see /test/data/ConvSpec-010-048/marc.xml)  and ran the 035 scenario in /test/ConvSpec-010-048.xspec looking for the new value "ocm04212209" (no trailing space) and it passed.
xspec uses Saxon9he.jar - I did noticed other chopPunctuation call-templates have this line:
<xsl:with-param name="punctuation">xsl:text:,;/ </xsl:text></xsl:with-param> while the 035 one is relying on the default value of that parameter (which is the same text value).
You could go in ConvSpec-010-048.xsl, go to line 935, press enter and paste that into line 936 and see if it makes a difference.

I convert using Oxygen with Saxon-PE <IP_ADDRESS>.  The trailing space is in the RDF/XML, not an artifact of the conversion to Turtle.
I'm not following how you want me to test.  Open a new line after 935 and paste in "<xsl:with-param name="punctuation">xsl:text:,;/ </xsl:text></xsl:with-param>"? Or replace line 935 with this line?

The first, new line, paste in that with-param. Can you attach the XML you are transforming?

Line 918 in marc2bibframe2/xsl/ConvSpec-010-048.xsl
 <xsl:param name="pChopPunct" select="false()"/>
Change that to true() and it will remove trailing punctuation incl. spaces.


Options documentation

I cannot find information  in the documentation about all the options that can be passed to the open
function like custom button settings  buttons: [{type: '...', text: '...', className: '...'}].
For instance I know that there is a confirm type because I found that somewhere on jsfiddle but I don't know the other types.

The type option is used to set the DOM attribute type. Read more about button attributes: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/button
type, text, and className are the only three button options and they're used as you expect. I added some documentation in 8a914d9, does it look good?

Yes, thanks for adding this to the docs.


You can (attempt to) manipulate your inventory without any hands.

Description
It just drops on the floor, though.

Replicated.
steps:
spawn urist > equip urist with a hat or some other equipment > remove urists hands via smite > control urist > remove urist's hat and sulk now forever hatless.