Added tests for cat_years_to_hooman_years

I am following your PyCon 2016 tutorial on YouTube:
https://www.youtube.com/watch?v=UPanUFVFfzY
I hope it's ok to do a pull request.
Martin

Coverage increased (+1.1%) to 37.931% when pulling<PHONE_NUMBER>c63ec295560027c6d637ad84003575 on starz-blaze:tutorial_wip into cad3d2ba400a71b24a6aee9b7ecf44938b97fa0a on keeppythonweird:master.

I hope it's ok to do a pull request.

Of course @starz-blaze ! Hope you enjoy the tutorial :D

Coverage increased (+5.7%) to 42.529% when pulling f719d0ca4c4c1c31428d2cd65255bb7360fffb33 on starz-blaze:tutorial_wip into cad3d2ba400a71b24a6aee9b7ecf44938b97fa0a on keeppythonweird:master.

Coverage increased (+5.7%) to 42.529% when pulling c71863a7cc1dbf82a5e2110905941cf20ab92a25 on starz-blaze:tutorial_wip into cad3d2ba400a71b24a6aee9b7ecf44938b97fa0a on keeppythonweird:master.


NameError: name 'project' is not defined on pkg push

Error thrown
env_sessions = lifecycle_cli.build_sessions_for_project(project.config, environment, pwd, armname, config)
NameError: name 'project' is not defined

Fixed.


[Performance]: Image preprocessing is executed twice for same image during VLLM(Qwen2-vl) inference

Proposal to improve performance
Only perform image preprocessing once
Report of performance regression
No response
Misc discussion on performance

test code: https://github.com/vllm-project/vllm/pull/7905
Your current environment (if you think it is necessary)
The output of `python collect_env.py`

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

cc @ywang96 @DarkLight1337

Are you inputting more than one image at a time? I'm not sure under what circumstances the profiler would actually show two separate instances of the same function being called...

Are you inputting more than one image at a time, or using multiple processes? I'm not actually sure under what circumstances the profiler would show two separate instances of the same function being called...

Use batch inference to infer multiple requests at a time, but each request only contains one image.
The problem is that for the same picture, the same picture preprocessing is performed in the add_request stage and execute_model stage.
test code:
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '5'
import sys
sys.path.append("/data0/zhenglin/src/qwen2vl_vllm/vllm-add_qwen2_vl_new")
from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from qwen_vl_utils import process_vision_info
import vllm
print(f'vllm version:{vllm.version}')
MODEL_PATH = "."
llm = LLM(
model=MODEL_PATH,
limit_mm_per_prompt={"image": 10, "video": 10},
)
sampling_params = SamplingParams(
temperature=0.7,
top_p=0.9,
top_k=50,
repetition_penalty=1.2,
max_tokens=20,
stop_token_ids=[],
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)
directory = "./imgs"
llm_input_list = []
import time
start = time.time()
cnt = 0
for item in os.listdir(directory):
item_path = os.path.join(directory, item)
if os.path.isfile(item_path):
print(f"File: {item_path}")
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{
"role": "user",
"content": [
{
"type": "image",
"image": item_path,
"min_pixels": 840 * 420,
"max_pixels": 840 * 420,
},
{"type": "text", "text": "What does this diagram illustrate?"},
],
},
]
    prompt = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    image_inputs, video_inputs = process_vision_info(messages)

    mm_data = {}
    if image_inputs is not None:
        mm_data["image"] = image_inputs
    if video_inputs is not None:
        mm_data["video"] = video_inputs

    llm_inputs = {
        "prompt": prompt,
        "multi_modal_data": mm_data,
    }
    llm_input_list.append(llm_inputs)
    cnt = cnt + 1
    if cnt > 30:
        break

outputs = llm.generate(llm_input_list, sampling_params=sampling_params)
from torch.profiler import profile, record_function, ProfilerActivity
with profile(
activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
with_stack=True,
) as prof:
outputs = llm.generate(llm_input_list, sampling_params=sampling_params)
prof.export_chrome_trace("trace.json")
for output in outputs:
print(f'{output.outputs[0].text}')
end = time.time()``

I see. In this case it seems that we may add another layer to cache the images. @ywang96 do we have an existing benchmark for multi-modal models?

I took a look at the code path in the PR.
The first call of the processor processes the input sequence (I'm not sure why the image processor is also called here, and it shouldn't be called, so we need to take a look at this processor to see what's necessarily needed.)
The second call actually processes the input images

I see. In this case it seems that we may add another layer to cache the images. @ywang96 do we have an existing benchmark for multi-modal models?

We don't unfortunately, but it shouldn't be hard to add it to the existing benchmark framework since output is always text.

I took a look at the code path in the PR.
The first call of the processor processes the input sequence (I'm not sure why the image processor is also called here, and it shouldn't be called, so we need to take a look at this processor to see what's necessarily needed.)
The second call actually processes the input images, as expected.

Thank you for your comments! I will fix this asap.

@ZhangYaoFu Hi, this commit has removed redundant image transforms in input_processor_for_qwen2_vl, please check it again.

@ZhangYaoFu Hi, this commit has removed redundant image transforms in input_processor_for_qwen2_vl, please check it again.

Thank you so much！ I'll try it.

@ZhangYaoFu Hi, this commit has removed redundant image transforms in input_processor_for_qwen2_vl, please check it again.

I've tested this feature and it's fine. Awesome

The main issue has been solved so I'm closing this. We can revisit image caching when its performance becomes a greater concern.


chore(dep) Switch to Cicero 0.20.1

Signed-off-by: Jerome Simeon<EMAIL_ADDRESS>

Does this also require changes in each template's package.json?

The templates haven't changed, and already worked on 0.20.0. They still will.


Fixes #3236 - Adding Maven Central fallback for gradle

Fixes #3236
By following what Maven ecosystem is doing by always adding Maven Central repo to the list.

Thank you for demonstrating a bias for action and for your enthusiasm in getting this resolved. From a quick glance of the changes this looks like a great start.
I have a couple of questions that I'd like to discuss and to collect some additional feedback on such as:

Is there a way to detect name squatting collisions?
Should falling back to public registries be a default behaviour or an opt-in option?

Please say more about this, I'll be honest I'm not sure I follow. Are you saying in one public repo (let's say jcenter for example) someone has taken a package name version set and hijacked it but in maven Central it's a valid package??

Sure I'm happy to make this opt in but should maven follow the same behavior then as well??

Please say more about this, I'll be honest I'm not sure I follow. Are you
saying in one public repo (let's say jcenter for example) someone has taken
a package name version set and hijacked it but in maven Central it's a valid
package??

Let me try to provide an example for clarification:
Private registry: (https://acme.example.org/)

package name: acme-logger
package version: 0.1.0

Public registry: (https://repo1.maven.org/maven2/)

package name: acme-logger (malicious package not owned by acme corp)
package version: 0.1.1

In the example above an internal company package named "acme-logging" is
hosted and served from a private package registry. At the same time a package
named "acme-logging" is also hosted on a public registry with the same name
but owned by an entity not belonging to acme corp.
In this example, falling back to the public registry would produce
a pull request from Dependabot that suggests installing a malicious
package.
My question is to see how we might be able to identify and prevent
this behaviour.
More info can be found in Substitution attacks.

Sure I'm happy to make this opt in but should maven follow the same
behavior then as well??

I like consistency. So if this is what we're doing with Maven today
then it would be great to make sure we align the behaviour between
the two.
I raised the question of what the default behaviour
should be to make sure the default is the less risky option.
However, this might also mean adding a way to opt in or choose a specific
behaviour.
If this is the default behaviour for how mvn and gradle behave then
perhaps this is less of a concern.
I'll need some input from other colleagues on this one.
I might be over thinking this but I'd like to try to prevent introducing
risky packages if possible.
Also, thanks for taking the time to consider the
different security implications. I'd like to see if we can find a way
to cater to the scenario resolved in this PR while also providing safe
defaults for everyone.
What do you think?

@xlgmokha Thanks for clarification. That makes sense as far as the attack vector goes. And I agree if we can figure out a way to do that or at least make the behavior opt in that would probably better.
I did find this while researching substitution/brandjacking attacks.
https://blog.sonatype.com/malware-removed-from-maven-central

When a new publisher comes along requesting access to publish to Central, the requirements enforce that you can verify control of either the DNS for the Group ID, or for control over the account/repo for coordinates derived from platforms such as GitHub. As a part of this screening, users are asked to verify their GitHub account before they are assigned a Group ID, such as ‘com.github.codingandcoding’, as is the ID in this case.

Sounds like maven central might make it a little harder to do b/c of the way it validates and requires Group ID. Similar to the article you linked using scopes with npm packages can help protect you there.
In my case and maybe others packages look something like so
<groupId>com.bigcompany</groupId>
<artifactId>custom-package</artifactId>

So you would have to prove you have DNS access to com.bigcompany or rights to github. Probably doable still just harder to do.
Opt-in still feels like maybe the right choice. With the findings below about how the dependency managers work below it might make sense to have maven behave like gradle and not fallback to central unless you opt in.

More information on why dependabot probably works the way it does for maven vs gradle.
I started by creating this branch in my test repo.
https://github.com/zendern/private-repo-dependabot-fail/tree/default-repo-setup
Where maven and gradle both have no repositories or plugin repositories defined. In the maven case with nothing defined its behavior is to default to maven central. In the gradle case it throws an exception b/c no repositories are defined.
So I'm guessing the "defaulting/fallback" for maven was to simulate that. But maybe the maven code should only "default/fallback" if no repositories are defined instead of always doing it??

So I'm guessing the "defaulting/fallback" for maven was to simulate that. But maybe the maven code should only "default/fallback" if no repositories are defined instead of always doing it??

I'm checking to see if there is a way to specify opt-in behaviour via the dependabot.yml or if there's another way to do this. Meanwhile, I'm also looping in friends from security to make sure I understand the impact of using this as a default. I'll keep you posted as much as I can. Thanks for your patience @zendern.

A review from friends in #security suggests that it would be best to avoid falling back to the public registries by default due to recent naming/typo squatting.
However, our roadmap has some exciting features related to better support for private registries. I think this may also cover being able to specify public registries as a fallback mechanism.
I appreciate the effort made to ship this idea. Unfortunately, we wont be able to accept this contribution at this time.
Thank you very much @zendern and keep them coming.

@xlgmokha cool. That's fair and I understand. I will close this pr. I assume with the support of private registries there would have to be line of sight from GitHub to that registry??

I assume with the support of private registries there would have to be line of sight from GitHub to that registry??

Yep, that's right. Dependabot would need to be able to resolve the IP address for the private registry host and be able to connect to it with or without credentials. More details to come soon.


Multi-page annotation

How to create backend & labelling interface for multipage annotation?

hey @juhonkang
can you add more details about your case? what data you're annotating, how much data each annotation have, etc.

<View>
  <Repeater on="$document" indexFlag="{{idx}}" mode="pagination">
      <Image name="page_{{idx}}" value="$document[{{idx}}].page" inline="true"/>

    	<Labels name="labels_{{idx}}" toName="page_{{idx}}">
            <Label value="Text" background="green"/>
            <Label value="Handwriting" background="blue"/>
      	</Labels>

      <Rectangle name="bbox_{{idx}}" smart="true" toName="page_{{idx}}" strokeWidth="3"/>

      <TextArea name="transcription_{{idx}}" toName="page_{{idx}}"
            editable="true"
            perRegion="true"
            required="true"
            maxSubmissions="1"
            rows="5"
            placeholder="Recognized Text"
            displayMode="region-list"
            />

      <View style="display: flex; justify-content: center;">
        <Header value=" Rate this page:"/>
        <Rating name="rating_{{idx}}" toName="page_{{idx}}"/>
      </View>
  </Repeater>
</View>

<!-- {
"document": [
    {
      "page": "https://htx-pub.s3.amazonaws.com/demo/images/demo_stock_purchase_agreement/0001.jpg"
    },
    {
      "page": "https://htx-pub.s3.amazonaws.com/demo/images/demo_stock_purchase_agreement/0002.jpg"
    },
    {
      "page": "https://htx-pub.s3.amazonaws.com/demo/images/demo_stock_purchase_agreement/0003.jpg"
    }
  ]
} -->

I want to make with ~8 images at once
This one is the view I want to use, how can I modify the tesseract.py API of the example?
The most problem I'm facing is finding the idx of the image, and returning the result

Hi @nicholasrq,
Have you checked it?

I want to annotate the pdf files, converted to image.
It's multi page doc annotation, and can be ~8 pages

Hi @

Hi @nicholasrq, any progress for that?

@nicholasrq @makseq Hope that label studio expert help me soon because I need this feature for one of my project :(

I'm facing a similar issue where I can only annotate the first page. Rectangles for the other pages are not sent to the back-end (not http calls to /predict)


Substation batteries drain even if the substation is not connected to (or powering) anything.

How to reproduce:

Go into sandbox mode (not mapping) and build a room off the station that is not connected to the station's power grid.
Spawn a substation. It'll spawn with a full battery that will deplete pretty rapidly even if it's not connected to anything.

Can't replicate. Placing a substation in a grid far from the station doesn't drain the BatteryComponent CurrentCharge

this predates pow3r so not surprised it got fixed inbetween


about the rendered novel views

Hi, thanks for your excellent work. I have some questions about the rendered novel view images by the command in README.md.

The rendered images (ImagesRefined0) are some inconsistencies in continuous frames, just like this,

and,

The rendered images (ImagesGSplat) have some artifacts.

Could you tell me some reasons above abnormal results?
Thanks.
Best.

Hi, @szhang963, thanks for your interest in our work.
The 'ImagesRefined0' results are refined by the SVD, which can lead to some unexpected generated contents due to "hallucination", especially in regions unobserved in the inputs. You can check out the limitation section in Appendix C. of the paper for more details. One of the potential solutions for this issue is to replace SVD with a more advanced video diffusion model, e.g., CogVideoX or HunyuanVideo. Still, there is no guarantee of thoroughly addressing this issue since maintaining 3D consistency in unobserved generated regions is ill-posed and challenging.
The 'ImagesGSplat' results are of low quality, as expected, which is why we used the following SVD module to refine these outputs further. You can take a look at the motivation presented in the Introduction of our paper for more details.
Hope it helps. Cheers.


Fixed #1425: Full application crash at startup on version 0.8.7 (F-droid)

Probably fixed this #1425 crash, but we need double check.
@ross-holloway94
@Karljoones

Hey @l-o-b-s-t-e-r,
Something went wrong with the build.
TravisCI finished with status errored, which means the build failed because something unrelated to the tests, like a problem a dependency or the build process itself.

Hey @l-o-b-s-t-e-r,
Something went wrong with the build.
TravisCI finished with status errored, which means the build failed because something unrelated to the tests, like a problem a dependency or the build process itself.

thanks @l-o-b-s-t-e-r


Blockchain Progress: X restarts (endless loop)

A while ago I realized that my node is facing an issue (status screen) and decided to reboot (the counter is going up after some seconds).
Since then I'm facing endless restart loops to no avail, what I tried so far:

Flashing the SD card (v1.8a)
Using another SD card (only 16GB) since the other faced errors when flashing with balenaEtcher
Using another slot of my card reader via an SD-Card-Adapter
Flashing with v1.8c
Updating LND
Updating Bitcoin Core
Deactivating TOR

Stiil the same issue, I hope someone has an idea?
I'd also like to check if the SSD is still ok - but apart from the restarts everything seems to work fine...
To check if the blockchain etc. is still ok could also be useful since starting from scratch takes quite some time...
Thanks

Here the log some minutes and some loops after restart) - hope this helps to find the problem, thanks for your support!

RASPIBLITZ DEBUG LOGS

blitzversion: 1.8.0
chainnetwork: bitcoin / main
20:07:06 up 11 min,  2 users,  load average: 2.31, 2.31, 1.50
*** SETUPPHASE / BOOTSTRAP ***
see logs: cat /home/admin/raspiblitz.log
setupPhase--> done
state--> ready
*** BACKGROUNDSERVICE ***
to monitor Background service call: sudo journalctl -f -u background
*** BLOCKCHAIN (MAINNET) SYSTEMD STATUS ***
● bitcoind.service - Bitcoin daemon
Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled)
Active: activating (start) since Tue 2023-05-09 20:06:55 BST; 10s ago
Process: 77299 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log blockchain STARTED (code=exited, status=0/SUCCESS)
Process: 77304 ExecStartPre=/bin/chgrp bitcoin /mnt/hdd/bitcoin (code=exited, status=0/SUCCESS)
Cntrl PID: 77305 (bitcoind)
Tasks: 13 (limit: 4531)
CPU: 10.855s
CGroup: /system.slice/bitcoind.service
├─77305 /usr/local/bin/bitcoind -daemonwait -conf=/mnt/hdd/bitcoin/bitcoin.conf -datadir=/mnt/hdd/bitcoin -debuglogfile=/mnt/hdd/bitcoin/debug.log
└─77306 /usr/local/bin/bitcoind -daemonwait -conf=/mnt/hdd/bitcoin/bitcoin.conf -datadir=/mnt/hdd/bitcoin -debuglogfile=/mnt/hdd/bitcoin/debug.log
May 09 20:06:55 raspberrypi systemd[1]: Starting Bitcoin daemon...
*** LAST BLOCKCHAIN (MAINNET) ERROR LOGS ***
sudo journalctl -u bitcoind -b --no-pager -n8
-- Journal begins at Wed 2022-07-20 12:10:49 BST, ends at Tue 2023-05-09 20:07:06 BST. --
May 09 20:06:54 raspberrypi systemd[1]: bitcoind.service: Control process exited, code=exited, status=1/FAILURE
May 09 20:06:55 raspberrypi systemd[1]: bitcoind.service: Failed with result 'exit-code'.
May 09 20:06:55 raspberrypi systemd[1]: Failed to start Bitcoin daemon.
May 09 20:06:55 raspberrypi systemd[1]: bitcoind.service: Consumed 15.606s CPU time.
May 09 20:06:55 raspberrypi systemd[1]: bitcoind.service: Scheduled restart job, restart counter is at 38.
May 09 20:06:55 raspberrypi systemd[1]: Stopped Bitcoin daemon.
May 09 20:06:55 raspberrypi systemd[1]: bitcoind.service: Consumed 15.606s CPU time.
May 09 20:06:55 raspberrypi systemd[1]: Starting Bitcoin daemon...
*** LAST BLOCKCHAIN (MAINNET) 20 INFO LOGS ***
sudo tail -n 20 /mnt/hdd/bitcoin/debug.log
2023-05-09T19:06:55Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
2023-05-09T19:06:55Z HTTP: starting 4 worker threads
2023-05-09T19:06:55Z Using wallet directory /mnt/hdd/bitcoin
2023-05-09T19:06:55Z init message: Verifying wallet(s)…
2023-05-09T19:06:55Z Warning: Skipping -wallet path that doesn't exist. Failed to load database path '/mnt/hdd/bitcoin/wallet.dat'. Path does not exist.
2023-05-09T19:06:55Z Using /16 prefix for IP bucketing
2023-05-09T19:06:55Z init message: Loading P2P addresses…
2023-05-09T19:06:56Z Loaded 16406 addresses from peers.dat  236ms
2023-05-09T19:06:56Z init message: Loading banlist…
2023-05-09T19:06:56Z SetNetworkActive: true
2023-05-09T19:06:56Z Cache configuration:
2023-05-09T19:06:56Z * Using 2.0 MiB for block index database
2023-05-09T19:06:56Z * Using 127.8 MiB for transaction index database
2023-05-09T19:06:56Z * Using 8.0 MiB for chain state database
2023-05-09T19:06:56Z * Using 886.2 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2023-05-09T19:06:56Z init message: Loading block index…
2023-05-09T19:06:56Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2023-05-09T19:06:56Z Opening LevelDB in /mnt/hdd/bitcoin/blocks/index
2023-05-09T19:06:56Z Opened LevelDB successfully
2023-05-09T19:06:56Z Using obfuscation key for /mnt/hdd/bitcoin/blocks/index:<PHONE_NUMBER>000000
*** LND (MAINNET) SYSTEMD STATUS ***
● lnd.service - Lightning Network Daemon
Loaded: loaded (/etc/systemd/system/lnd.service; enabled; vendor preset: enabled)
Active: inactive (dead)
May 09 19:56:36 raspberrypi systemd[1]: Dependency failed for Lightning Network Daemon.
May 09 19:56:36 raspberrypi systemd[1]: lnd.service: Job lnd.service/start failed with result 'dependency'.
*** LAST LND (MAINNET) ERROR LOGS ***
sudo journalctl -u lnd -b --no-pager -n12
-- Journal begins at Wed 2022-07-20 12:10:49 BST, ends at Tue 2023-05-09 20:07:06 BST. --
May 09 19:56:36 raspberrypi systemd[1]: Dependency failed for Lightning Network Daemon.
May 09 19:56:36 raspberrypi systemd[1]: lnd.service: Job lnd.service/start failed with result 'dependency'.
*** LAST 30 LND (MAINNET) INFO LOGS ***
sudo tail -n 30 /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log
*** CORE LIGHTNING (MAINNET) SYSTEMD STATUS ***

not activated -

*** BLOCKCHAIN (TESTNET) SYSTEMD STATUS ***

OFF by config -
*** LND (TESTNET) SYSTEMD STATUS ***
OFF by config -

*** CORE LIGHTNING (TESTNET) SYSTEMD STATUS ***

not activated -

*** BLOCKCHAIN (SIGNET) SYSTEMD STATUS ***

OFF by config -
*** LND (SIGNET) SYSTEMD STATUS ***
OFF by config -

*** CORE LIGHTNING (SIGNET) SYSTEMD STATUS ***

not activated -

*** NGINX SYSTEMD STATUS ***
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/nginx.service.d
└─raspiblitz.conf
Active: active (running) since Tue 2023-05-09 19:57:11 BST; 9min ago
Docs: man:nginx(8)
Process: 8613 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 8664 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 8670 (nginx)
Tasks: 5 (limit: 4531)
CPU: 128ms
CGroup: /system.slice/nginx.service
├─8670 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─8671 nginx: worker process
├─8672 nginx: worker process
├─8673 nginx: worker process
└─8674 nginx: worker process
May 09 19:57:10 raspberrypi systemd[1]: Starting A high performance web server and a reverse proxy server...
May 09 19:57:11 raspberrypi systemd[1]: Started A high performance web server and a reverse proxy server.
*** LAST NGINX LOGS ***
sudo journalctl -u nginx -b --no-pager -n20
-- Journal begins at Wed 2022-07-20 12:10:49 BST, ends at Tue 2023-05-09 20:07:07 BST. --
May 07 17:59:49 raspberrypi systemd[1]: Starting A high performance web server and a reverse proxy server...
May 07 17:59:53 raspberrypi nginx[664]: nginx: [emerg] host not found in upstream "mempool.space" in /etc/nginx/snippets/mempool.conf:39
May 07 17:59:53 raspberrypi nginx[664]: nginx: configuration file /etc/nginx/nginx.conf test failed
May 07 17:59:53 raspberrypi systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
May 07 17:59:53 raspberrypi systemd[1]: nginx.service: Failed with result 'exit-code'.
May 07 17:59:53 raspberrypi systemd[1]: Failed to start A high performance web server and a reverse proxy server.
May 09 19:57:10 raspberrypi systemd[1]: nginx.service: Scheduled restart job, restart counter is at 1.
May 09 19:57:10 raspberrypi systemd[1]: Stopped A high performance web server and a reverse proxy server.
May 09 19:57:10 raspberrypi systemd[1]: Starting A high performance web server and a reverse proxy server...
May 09 19:57:11 raspberrypi systemd[1]: Started A high performance web server and a reverse proxy server.
--> CHECK CONFIG: sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
*** BLITZAPI SYSTEMD STATUS ***
● blitzapi.service - BlitzBackendAPI
Loaded: loaded (/etc/systemd/system/blitzapi.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2023-05-07 17:59:51 BST; 2 days ago
Main PID: 823 (python)
Tasks: 7 (limit: 4531)
CPU: 13.397s
CGroup: /system.slice/blitzapi.service
└─823 /usr/bin/python -m uvicorn app.main:app --port 11111 --host=... --root-path /api
May 09 20:07:04 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:07:06 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
*** LAST BLITZAPI LOGS ***
sudo journalctl -u blitzapi -b --no-pager -n20
-- Journal begins at Wed 2022-07-20 12:10:49 BST, ends at Tue 2023-05-09 20:07:07 BST. --
May 09 20:06:28 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:30 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:32 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:34 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:36 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:38 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:40 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:42 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:44 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:46 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:48 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:50 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:52 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:54 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:56 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:06:58 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:07:00 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:07:02 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:07:04 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error
May 09 20:07:06 raspberrypi python[823]: ERROR:root:Unknown answer from Bitcoin Core. Reason: Internal Server Error

TOUCHSCREEN is OFF by config
Loop is OFF by config

*** LND-RTL ***
● RTL.service - RTL Webinterface
Loaded: loaded (/etc/systemd/system/RTL.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-05-09 19:56:39 BST; 10min ago
Main PID: 5589 (node)
Tasks: 11 (limit: 4531)
CPU: 6.373s
CGroup: /system.slice/RTL.service
└─5589 /usr/bin/node /home/rtl/RTL/rtl
May 09 19:56:36 raspberrypi systemd[1]: Starting RTL Webinterface...
May 09 19:56:39 raspberrypi systemd[1]: Started RTL Webinterface.

 --
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.274 [DBG] GRPC: [core] Subchannel picks a new address "...:10009" to connect
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.275 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {...:10009 <IP_ADDRESS>:10009  0 }. Err: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:10009: connect: connection refused". Reconnecting...
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.275 [DBG] GRPC: [core] Subchannel Connectivity change to TRANSIENT_FAILURE
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.275 [DBG] GRPC: [core] pickfirstBalancer: UpdateSubConnState: 0x4000619370, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp ...:10009: connect: connection refused"}
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.275 [DBG] GRPC: [core] Channel Connectivity change to TRANSIENT_FAILURE
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.275 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {...:10009 <IP_ADDRESS>:10009  0 }. Err: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:10009: connect: connection refused". Reconnecting...
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.276 [DBG] GRPC: [core] Subchannel Connectivity change to TRANSIENT_FAILURE
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.276 [DBG] GRPC: [core] pickfirstBalancer: UpdateSubConnState: 0x4000619680, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp ...:10009: connect: connection refused"}
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.276 [DBG] GRPC: [core] Channel Connectivity change to TRANSIENT_FAILURE
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.276 [DBG] GRPC: [core] Channel Connectivity change to SHUTDOWN
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.276 [DBG] GRPC: [core] Subchannel Connectivity change to SHUTDOWN
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.277 [ERR] LITD: Could not start subservers: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:10009: connect: connection refused"
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.277 [DBG] GRPC: [core] Channel Connectivity change to SHUTDOWN
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.277 [DBG] GRPC: [core] Subchannel Connectivity change to SHUTDOWN
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.277 [INF] SGNL: Received shutdown request.
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.277 [INF] SGNL: Shutting down...
May 09 20:06:47 raspberrypi torsocks[76175]: 2023-05-09 20:06:47.277 [INF] SGNL: Gracefully shutting down.
May 09 20:06:47 raspberrypi torsocks[76175]: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp ...:10009: connect: connection refused"
May 09 20:06:47 raspberrypi systemd[1]: litd.service: Main process exited, code=exited, status=1/FAILURE
May 09 20:06:47 raspberrypi systemd[1]: litd.service: Failed with result 'exit-code'.

BTCPayServer is OFF by config

*** LAST* BTC-RPC-Explorer LOGS ***
sudo journalctl -u btc-rpc-explorer -b --no-pager -n20
-- Journal begins at Wed 2022-07-20 12:10:49 BST, ends at Tue 2023-05-09 20:07:07 BST. --
May 09 20:06:25 raspberrypi npm[6425]:     at Client.command (/home/btcrpcexplorer/btc-rpc-explorer/node_modules/btc-rpc-client/dist/src/index.js:169:24)
May 09 20:06:25 raspberrypi npm[6425]:     at processTicksAndRejections (node:internal/process/task_queues:96:5)
May 09 20:06:25 raspberrypi npm[6425]:     at async Object.rpcCall (/home/btcrpcexplorer/btc-rpc-explorer/app/api/rpcApi.js:452:20)
May 09 20:06:25 raspberrypi npm[6425]: 2023-05-09T19:06:25.766Z btcexp:error Error RpcError-001: RpcError: -28 Loading block index…, json: {"message":"Loading block index…","code":-28,"name":"RpcError","userData":{"request":"getblockchaininfo"}}, userData: [object Object] (json: {"request":"getblockchaininfo"})
May 09 20:06:25 raspberrypi npm[6425]: 2023-05-09T19:06:25.766Z btcexp:errorVerbose Stack: RpcError: Loading block index…
May 09 20:06:25 raspberrypi npm[6425]:     at getRpcResult (/home/btcrpcexplorer/btc-rpc-explorer/node_modules/btc-rpc-client/dist/src/parser.js:39:11)
May 09 20:06:25 raspberrypi npm[6425]:     at Parser.rpc (/home/btcrpcexplorer/btc-rpc-explorer/node_modules/btc-rpc-client/dist/src/parser.js:82:14)
May 09 20:06:25 raspberrypi npm[6425]:     at Client.command (/home/btcrpcexplorer/btc-rpc-explorer/node_modules/btc-rpc-client/dist/src/index.js:169:24)
May 09 20:06:25 raspberrypi npm[6425]:     at processTicksAndRejections (node:internal/process/task_queues:96:5)
May 09 20:06:25 raspberrypi npm[6425]:     at async Object.rpcCall (/home/btcrpcexplorer/btc-rpc-explorer/app/api/rpcApi.js:452:20)
May 09 20:06:55 raspberrypi npm[6425]: 2023-05-09T19:06:55.753Z btcexp:app Verifying RPC connection...
May 09 20:06:55 raspberrypi npm[6425]: 2023-05-09T19:06:55.759Z btcexp:error Error RpcError-001: Error: connect ECONNREFUSED ...:8332, json: {"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"<IP_ADDRESS>","port":8332,"userData":{"request":"getnetworkinfo"}}, userData: [object Object] (json: {"request":"getnetworkinfo"})
May 09 20:06:55 raspberrypi npm[6425]: 2023-05-09T19:06:55.759Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED ...:8332
May 09 20:06:55 raspberrypi npm[6425]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16)
May 09 20:06:55 raspberrypi npm[6425]: 2023-05-09T19:06:55.760Z btcexp:error Error 32ugegdfsde: Error: connect ECONNREFUSED ...:8332, json: {"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"<IP_ADDRESS>","port":8332,"userData":{"request":"getnetworkinfo"}}, userData: [object Object] (json: {})
May 09 20:06:55 raspberrypi npm[6425]: 2023-05-09T19:06:55.760Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED ...:8332
May 09 20:06:55 raspberrypi npm[6425]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16)
May 09 20:06:55 raspberrypi npm[6425]: 2023-05-09T19:06:55.761Z btcexp:error Error RpcError-001: Error: connect ECONNREFUSED ...:8332, json: {"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"<IP_ADDRESS>","port":8332,"userData":{"request":"getblockchaininfo"}}, userData: [object Object] (json: {"request":"getblockchaininfo"})
May 09 20:06:55 raspberrypi npm[6425]: 2023-05-09T19:06:55.761Z btcexp:errorVerbose Stack: Error: connect ECONNREFUSED ...:8332
May 09 20:06:55 raspberrypi npm[6425]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16)

LNbits is OFF by config
Thunderhub is OFF by config
SPECTER is OFF by config
SPHINX is OFF by config

*** MOUNTED DRIVES ***
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4       15G   14G  351M  98% /
devtmpfs       devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs          tmpfs     1.9G   16K  1.9G   1% /dev/shm
tmpfs          tmpfs     769M  1.1M  768M   1% /run
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs      32M   20K   32M   1% /var/cache/raspiblitz
/dev/mmcblk0p1 vfat      253M   31M  222M  12% /boot
/dev/sda1      ext4      916G  577G  293G  67% /mnt/hdd
tmpfs          tmpfs     385M   16K  385M   1% /run/user/1000
tmpfs          tmpfs     385M   16K  385M   1% /run/user/1001
*** DATADRIVE ***
RASPIBLITZ DATA DRIVE Status
BASICS
isMounted=1
isBTRFS=0
hddRaspiData=1
hddRaspiVersion='1.8.0'
isSSD=1
datadisk='sda'
datapartition='sda1'
hddCandidate='sda'
hddPartitionCandidate='sda1'
hddBlocksBitcoin=1
hddBytes=1000141693440
hddGigaBytes=931
hddUsedInfo='577G (67%)'
hddDataFreeBytes=313883238400
hddDataFreeKB=306526600
hddDataFreeGB=292
hddAdapterUSB='174c:55aa'
hddAdapterUSAP=1
RAID
isRaid=0
raidCandidate[0]='zram0 4 GB zram0'
raidCandidates=1
SWAP
isSwapExternal=1
SwapExternalPath='/mnt/hdd/swapfile'
*** NETWORK ***
localip=...
localiprange=.../24
dhcp=1
network_device=eth0
*** HARDWARE TEST RESULTS ***
UndervoltageReports in Logs: 0
*** SYSTEM CACHE STATUS ***
system_up="701"
system_count_longscan="1"
system_temp_fahrenheit="124"
system_cpu_load=" 2.31, 2.31, 1.50"
system_ups_battery=""
system_count_undervoltage="0"
system_init_time="1683478796"
system_ram_mb="3844"
system_ram_gb="4"
system_count_start_blockchain="39"
system_temp_celsius="51"
system_count_start_lightning="0"
system_ups_status="OFF"
system_count_start_tui="0"
system_vm_vagrant="0"
system_board="rp4"
system_ram_available_mb="2775"
ln_default_error_full=""
ln_default_running="0"
ln_default_locked="0"
ln_default_ready="0"
ln_default_error_short=""
ln_default_activated="1"
ln_default_sync_initial_done="0"
ln_default_version="0.15.4-beta"
ln_default_online="0"
btc_default_error_full=""
btc_default_activated="1"
btc_default_sync_initial_done="0"
btc_default_sync_percentage="0"
btc_default_peers="0"
btc_default_error_short=""
btc_default_ready="0"
btc_default_version="v23.0.0"
btc_default_online="0"
btc_default_port=""
btc_default_running="0"
*** LOGFILES ***
Archived and active journals take up 320.0M in the file system.
332M    /var/log

I can see the bitcoind restarting, but would need a bigger chunk of the bitcoind logs to see why. Can you post the output from this command also sudo tail -n 200 /mnt/hdd/bitcoin/debug.log

Thanks, I'll need some days to check and come back then...

Best to update to latest v1.9.0 raspiblitz and then use command repair to get access to the reindex options.

The update to v1.9.0 worked - I only was confused since a key has changed for the ssh login...
I tried the reindex option via the menu, but it did not work.
The "fast" version told me it's not possible due to -txindex and to I tried the first entry which gave me the following error message: Job for bitcoind.service failed because the control process exited with error code. See ... for details.
I'll have a closer look the next days, but if it's already clear what can be done without further infomation an answer would be appreciated...

With the two mentioned commands I get those logs:
systemctl status bitcoind.service
● bitcoind.service - Bitcoin daemon
Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; vendor preset: enabled)
Active: activating (start) since Wed 2023-05-31 12:47:15 BST; 7s ago
Process: 50943 ExecStartPre=/home/admin/config.scripts/blitz.systemd.sh log blockchain STARTED (code=exited, status>
Process: 50956 ExecStartPre=/bin/chgrp bitcoin /mnt/hdd/bitcoin (code=exited, status=0/SUCCESS)
Cntrl PID: 50964 (bitcoind)
Tasks: 13 (limit: 4531)
CPU: 7.874s
CGroup: /system.slice/bitcoind.service
├─50964 /usr/local/bin/bitcoind -daemonwait -conf=/mnt/hdd/bitcoin/bitcoin.conf -datadir=/mnt/hdd/bitcoin >
└─50983 /usr/local/bin/bitcoind -daemonwait -conf=/mnt/hdd/bitcoin/bitcoin.conf -datadir=/mnt/hdd/bitcoin >
journalctl -xe
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.
May 31 12:46:34 raspberrypi systemd[46016]: Started Sound Service.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 19.
May 31 12:46:34 raspberrypi systemd[46016]: Reached target Main User Target.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 1.
May 31 12:46:34 raspberrypi systemd[46016]: Startup finished in 681ms.
░░ Subject: User manager start-up is now complete
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The user manager instance for user 1001 has been started. All services queued
░░ for starting have been started. Note that other services might still be starting
░░ up or be started at any later time.
░░
░░ Startup of the manager took 681615 microseconds.
May 31 12:46:34 raspberrypi sudo[46162]:    admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/bin/df
May 31 12:46:34 raspberrypi sudo[46162]: pam_unix(sudo:session): session opened for user root(uid=0) by admin(uid=1001)
May 31 12:46:34 raspberrypi sudo[46162]: pam_unix(sudo:session): session closed for user root
lines 1091-1119/1119 (END)
Hope that helps, thanks!

Sadly it does not work - but it's likely another issue now.
I'll close the topic and open another now.


python311Packages.django-stubs: 4.2.3 -> 4.2.4

Changelog: https://github.com/typeddjango/django-stubs/releases/tag/4.2.4
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 261363 run on x86_64-linux 1

4 packages failed to build:

python310Packages.djangorestframework-stubs
python310Packages.djangorestframework-stubs.dist
python311Packages.djangorestframework-stubs
python311Packages.djangorestframework-stubs.dist

4 packages built:

python310Packages.django-stubs
python310Packages.django-stubs.dist
python311Packages.django-stubs
python311Packages.django-stubs.dist

The failing test in python3Packages.djangorestframework-stubs is fixed in https://github.com/NixOS/nixpkgs/pull/263704.

Part of #271586


Heavy lag when typing

The previous versions worked well (subtle lag was tolerable), but this version started creating annoying amount of lag between typing and display on the screen.
Version details:
Version 1.24.1
Commit 24f62626b222e9a8313213fb64b10d741a326288
Date 2018-06-13T17:47:35.732Z
Shell 1.7.12
Renderer 58.0.3029.110
Node 7.9.0
Architecture x64

Does it happen if you start with --disable-extensions?


post-process: allow run checksum at last and restrict the number of checksum jobs

What problem does this PR solve?

Also allow run checksum jobs at last
Restrict the number of post-process jobs by config field checksum-table-concurrency and set its default value to 4 since 16 is a very big value.
Change default config distsql-scan-concurrency from 100 to 15

What is changed and how it works?
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

change Some default config, this may cause checksum and analyze to run slower in cluster with abandunt CPU resources.
Breaking backward compatibility

replace config field analyze-at-last with checksum-at-last

Related changes

Any related issues for this PR?

Any related issues for this PR?

Related to https://github.com/pingcap/tidb-lightning/issues/518
root case (maybe one part of it): Lightning will sent many coprocessor requests to TiKV (one for each region) for checksum and analyze, the number of which is equal to distsql-scan-concurrency. If TiKV can not consume these requets as soon as possible, the latency of some requests may exceed 10min, which is the timeout duration of gRPC client in Lightning, so lightning will get an error of TiKV server timeout.

Related to https://github.com/pingcap/tidb-lightning/issues/518
root case (maybe one part of it): Lightning will sent many coprocessor requests to TiKV (one for each region) for checksum and analyze, the number of which is equal to distsql-scan-concurrency. If TiKV can not consume these requets as soon as possible, the latency of some requests may exceed 10min, which is the timeout duration of gRPC client in Lightning, so lightning will get an error of TiKV server timeout.

LGTM

LGTM

/merge

/merge


Implement Node API

Description
Implement Node API (see https://docs.docker.com/engine/api/v1.41/#tag/Node) with full test coverage.

LGTM, thanks !


Still having problems with db-migrate and ESM node projects...

I'm submitting a...

[X] Bug report
[ ] Feature request
[ ] Question

Current behavior

I'm still having trouble running db-migrate in an ESM project / module (on the latest beta 1.0.0-beta.18). I keep receiving this error:
[ERROR] Error [ERR_REQUIRE_ESM]: require() of ES Module /home/kyle/web/server/migrations/some-migration.js from /home/kyle/web/server/node_modules/db-migrate/lib/file.js not supported. Instead change the require of some-migration.js in /home/kyle/web/server/node_modules/db-migrate/lib/file.js to a dynamic import() which is available in all CommonJS modules.
Minimal reproduction of the problem with instructions
My migration file looks something like this:
`
const Table = require('../dist/database/tables/SomeTable.js').SomeTable;
const table = new Table();
...
exports.up = function (db) {
return table.up(db);
};
...
`
I've also tried swapping the first line for something like this:
import {SomeTable} from '../dist/database/tables/SomeTable.js'; ...
but neither have worked. Does it have something to do with me importing a file into the migration file? The imported table is from an ESM module that also has dependencies to other ESM (local) projects.
I've recently switched my projects to ESM, so let me know if I'm missing something.
Environment

db-migrate version: 1.0.0-beta.18
plugins with versions: none
db-migrate driver with versions: db-migrate-mysql 2.2.0

Additional information:
- Node version: 16.16.0
- Platform:  Ubuntu Server 18.04.6

Related Links
#721 & #784
Supposedly solved in #724

#724 is a proven fix for this. However, this fix expects you to use the version with that fix in the first place. The routine added here runs only when you run the db-migrate create command once. It adds a package.json in your migrations folder and after that works. Just create a dummy migration with the command and delete it and check for the package.json created in the migrations folder, after that it should work fine.

Thanks for the quick response. However, I'm still getting the same error even with the package.json file in the migrations folder
Now, if my migration file looks like this:
const Table = require('../dist/database/tables/SomeTable.js').SomeTable;
const table = new Table();

...

exports.up = function (db) {
  return table.up(db);
};

...

I get the same error as above (Error [ERR_REQUIRE_ESM])
But if my migration file looks like this:
import {SomeTable} from '../dist/database/tables/SomeTable.js';
...

I get this error:
[ERROR] unhandledRejection
[ERROR] /home/kyle/web/EbayServer/migrations/some-migration.js:3
import {SomeTable} from '../dist/database/tables/SomeTable.js';
^^^^^^

SyntaxError: Cannot use import statement outside a module

Is there something else I need to do? Thanks!

Well please read the models docs about esmodules for that.
Sent from Nine
--
Best
--
Tobias Gurtzick
CEO
WizardTales GmbH
Additional Contacts:
@github | @.***
Website: https://www.wizardtales.com
@.***
nmlpm: -- coming soon & open source --
Please tag your mails with "request", "issue" or "proposal", unless none of these fit or there already exists a thread.
-- Why should one reach for what one may be capable reaching, that's awfully boring and after all one still be ones yesterday being. --
Sitz der Gesellschaft: Ratingen
Kokkolastr. 5
40882 Ratingen
Germany
Registergericht: Amtsgericht Düsseldorf, HRB 84140
Geschäftsführer: Tobias Gurtzick

Von: Kyle Elmore @.***>
Gesendet: Samstag, 23. Juli 2022 17:57
An: db-migrate/node-db-migrate
Cc: Tobias Gurtzick; State change
Betreff: Re: [db-migrate/node-db-migrate] Still having problems with db-migrate and ESM node projects... (Issue #794)
Thanks for the quick response. However, I'm still getting the same error even with the package.json file in the migrations folder
Now, if my migration file looks like this:
const Table = require('../dist/database/tables/SomeTable.js').SomeTable;
const table = new Table();
...
exports.up = function (db) {
return table.up(db);
};
...
I get the same error as above (Error [ERR_REQUIRE_ESM])
But if my migration file looks like this:
import {SomeTable} from '../dist/database/tables/SomeTable.js';
...
I get this error:
[ERROR] unhandledRejection
[ERROR] /home/kyle/web/EbayServer/migrations/some-migration.js:3
import {SomeTable} from '../dist/database/tables/SomeTable.js';
^^^^^^
SyntaxError: Cannot use import statement outside a module
Is there something else I need to do? Thanks!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you modified the open/close state.Message ID: @.***>

I don't mean to be a bother, but what models doc? I don't think the db-migrate docs mentions ESM...
Appreciate your time btw, Tobias!

b/c this has nothing to do with us. We support for db-migrate, not for understanding nodejs.
as a little favor:
https://nodejs.org/api/esm.html#interoperability-with-commonjs
https://pencilflip.medium.com/using-es-modules-with-commonjs-modules-in-node-js-1015786dab03

Ah!
Thanks Tobias

Okay, I'm pretty sure now that what I'm trying to do won't work. I.e. I don't believe you can import a file from an ES module into the migration file
Meaning, anything I import into the migration file has to come from a CommonJS module, and that if I want to do above^, I'd have to convert at least those imports into CJS files.
Makes sense! Just leaving a comment in case anyone else needs it. Thanks again for the help!


es5 ?

Does it work with es5 ?

Not yet.

Seems like all thats needed is to replace elasticsearch version to 5.1.* in requirements.txt

No, there are a few changes in elasticsearch-py library, which should be implemented.

You are right i also had to comment      search_type='scan' and was able to run with elasticsearch 5.1.0 just fine..

You did not check all of arguments. For example fields param is not present in search method.

Correct. I did not use fields param. Thank you, Taras. Its a very nice utility. ES 5 is becoming more mainstream - elastic.co hosting ES5 with their cloud offering

Works for me, too. Thank you!

ES 5 support has been added in 5.2.0 release. Please check.


enhance: Default merge policy should handle race conditions

BREAKING CHANGE: Entity.merge() default behavior changed.
Use return { ...existing, ...incoming }; to keep existing behavior.
Motivation

https://github.com/coinbase/rest-hooks/pull/1653 enables full race condition handling when server sends its date. However, without such special handling common cases were subject the client race conditions that were previously avoided.
Solution

This closely mimicks the existing behavior by relying on client timings out of the box to handle out of order conditions. This still cannot handle cases were server -> client latency varies widely; however this is less common so things should mostly work out of the box.

Codecov Report

Merging #1771 (1a579cd) into master (f510abe) will decrease coverage by 0.04%.
The diff coverage is 92.30%.

:exclamation: Current head 1a579cd differs from pull request most recent head 9f0bac9. Consider uploading reports for the commit 9f0bac9 to get more accurate results

@@            Coverage Diff             @@
##           master    #1771      +/-   ##
==========================================
- Coverage   98.46%   98.41%   -0.05%
==========================================
  Files         102      102
  Lines        1884     1891       +7
  Branches      379      382       +3
==========================================
+ Hits         1855     1861       +6
- Misses         14       15       +1
  Partials       15       15

Impacted Files
Coverage Δ

packages/normalizr/src/entities/Entity.ts
97.76% <75.00%> (-0.72%)
:arrow_down:

packages/core/src/controller/createOptimistic.ts
100.00% <100.00%> (ø)

packages/core/src/state/createReducer.ts
97.43% <100.00%> (+0.13%)
:arrow_up:

packages/normalizr/src/normalize.ts
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f510abe...9f0bac9. Read the comment docs.


Unable to Install Specific PyTorch Versions via Conda MacOS Sequoia 15.1.1 (intel, AMD)

Description:
The required PyTorch packages (pytorch==2.4.1, torchvision==0.19.1, torchaudio==2.4.1) are not found in the specified conda channels (pytorch, defaults). The installation fails with PackagesNotFoundError.
System Details:

OS: macOS
Processor: Intel Xeon W-2140B, 8 cores @ 3.20 GHz
RAM: 32 GB
GPU: Radeon Pro Vega 56, 8 GB VRAM

Steps to Reproduce:
Run the following command:
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 -c pytorch
Expected: Successful installation.
Actual: Installation fails with PackagesNotFoundError.
Suggested Fix:
Verify if the specified versions are supported for macOS in the pytorch channel or provide alternative installation instructions (e.g., using pip).
But this works
conda install pytorch torchvision torchaudio -c pytorch
Details:
% conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 -c pytorch
Channels:

pytorch
defaults
Platform: osx-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

torchvision==0.19.1
torchaudio==2.4.1
pytorch==2.4.1

Current channels:

https://conda.anaconda.org/pytorch
https://repo.anaconda.com/pkgs/main
https://repo.anaconda.com/pkgs/r

To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.

hi，you can check this page: https://pytorch.org/get-started/previous-versions/#osx-2, the command is no problem.  can you try   conda update conda  first ? or reinstall miniconda3 refer here: https://docs.anaconda.com/miniconda/install/

Description: The required PyTorch packages (pytorch==2.4.1, torchvision==0.19.1, torchaudio==2.4.1) are not found in the specified conda channels (pytorch, defaults). The installation fails with PackagesNotFoundError.
System Details:

OS: macOS
Processor: Intel Xeon W-2140B, 8 cores @ 3.20 GHz
RAM: 32 GB
GPU: Radeon Pro Vega 56, 8 GB VRAM

Steps to Reproduce: Run the following command:
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 -c pytorch
Expected: Successful installation. Actual: Installation fails with PackagesNotFoundError.
Suggested Fix: Verify if the specified versions are supported for macOS in the pytorch channel or provide alternative installation instructions (e.g., using pip).
But this works conda install pytorch torchvision torchaudio -c pytorch
Details:
% conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 -c pytorch Channels:

pytorch
defaults
Platform: osx-64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

torchvision==0.19.1
torchaudio==2.4.1
pytorch==2.4.1

Current channels:

https://conda.anaconda.org/pytorch
https://repo.anaconda.com/pkgs/main
https://repo.anaconda.com/pkgs/r

To search for alternate channels that may provide the conda package you're looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.

hi，you can check this page: https://pytorch.org/get-started/previous-versions/#osx-2, the command is no problem.  can you try   conda update conda  first ? or reinstall miniconda3 refer here: https://docs.anaconda.com/miniconda/install/ ?


Mousify the somatic workflows

Key Differences

mouse PON
no gnomAD
mutect inputs for mouse?
no hotspot list

This is largely complete


Compare string type and numerical type will cause full table scan

Describe the bug
The sql like SELECT * FROM tbl WHERE olap_date='20200501' will not only scan partiiton p20200501 but scan the full table, it is because olap_date is int type and '20200501' is string literal, in following logic, olap_date and string literal will convert to double and compare, which lead to a full table scan.
https://github.com/apache/incubator-doris/blob/d64704599d338de290972c1324903cf03d7bab5c/fe/src/main/java/org/apache/doris/analysis/BinaryPredicate.java#L280-L310
Solutions:
I think we should throw a DdlException when compare string literal to a numerical type to avoid this situation, Is there any solutions better?

@WingsGo Hi, I think throw a DdlException is not acceptable.
Most of query engine support int compare string. But maybe don't choose double as the common type. You could investigate Presto, Hive Or Spark. Thanks.

First of all, throwing an exception is not acceptable. Because it will make many already exist queries fail.
I think it is viable approach to handle this case in a specified case.

@imay @kangkaisen can we check the binary compare type, if one type is string and the other type is numerical such as int, bigint...., we just return the numercal type in getCmpType() function?

There is another case cause by the problem.for example: if we convert string and bigint to double, it may cause query result not match , because
Double d1 = new Double(2882303761517473127L);
Double d2 = new Double("2882303761517473267");
System.out.println(d1.equals(d2));    // true

MySQL [(none)]> select<PHONE_NUMBER>517473127 = "2882303761517473127";
+-----------------------------------------------+
| 2.8823037615174733E18 = 2.8823037615174733E18 |
+-----------------------------------------------+
|                                             1 |
+-----------------------------------------------+
1 row in set (0.07 sec)

MySQL [(none)]> select<PHONE_NUMBER>517473127 = "2882303761517473267";
+-----------------------------------------------+
| 2.8823037615174733E18 = 2.8823037615174733E18 |
+-----------------------------------------------+
|                                             1 |
+-----------------------------------------------+
1 row in set (0.07 sec)

@imay @kangkaisen can we check the binary compare type, if one type is string and the other type is numerical such as int, bigint...., we just return the numercal type in getCmpType() function?

It is OK to optimize the case when one side of binary predicate is SlotRef and the other is a StringLiteral. After that work, the problem you mentioned will be resolved.


Replaced eslint and prettier with biome

I had to modify the modify-openapi-operationids.js script because biome was giving me a lint error because of the use of forEach instead of for...of
I also disabled some other rules in the linter that were making the project fail without some changes, because I'd prefer the maintainers decide if we want to keep it as it is or not.

I see that this other PR has been merged: #714
I see two alternatives:

Modify my PR to instead use biome in pre-commit: https://biomejs.dev/recipes/git-hooks/#pre-commit
Revert the changes made in #714 and keep just my changes

The only reason why I prefer 2 over 1 is that using that approach the git-hooks on the frontend are installed automatically when we run npm install, while pre-commit hooks are installed only if we run pre-commit install.
I think that we should consider which trade off we want to have, either have different ways we define linting git-hooks for the frontend and backend or we have a standarized way but are forced to run pre-commit install everytime we want to setup the project (currently we never mention in the docks that the pre-commit hooks should be installed)

Awesome! This looks cool @santigandolfo! :rocket:
We need to keep pre-commit because that's what's used to format the backend and the rest of the code, and Husky expects to be the only git hook handler, disabling pre-commit. :nerd_face:
I'll ask @alejsdev to tweak this PR a bit with some commits on top to adjust the rules to the current code style, to minimize the changes, but taking in a few of them, etc. And then it will be ready, thank you! :cake:

Awesome, thanks @santigandolfo and @alejsdev! :cake:


Doesn't handle scroll for content

In case inner content has size bigger then window height, there is no possibity to scroll it.

Hello there.
A11y-dialog is not applying any styles or doing any styling consideration whatsoever. You have to implement the styling layer, including the handling of overflow.
I understand how it feels less convenient having to handle styles yourself, but it makes a11y-dialog a good low level library to integrate with any front end framework.


feat: post tweet V.1.

Solving https://github.com/redouane59/twitter-client/issues/37

A quick note, you could use closing keywords instead of managing issues workflow manually.


Does not run with pytorch 1.8

Error is this:
INFO:RIMEngine:Cannot recover from exception aten::record_stream is not yet supported with named tensors. Please drop names via tensor = tensor.rename(None), call the op with an unnamed tensor, and set names on the result of the operation.. Exiting.

Note for future solution.
Seems like torch.multiprocessing.spawn fails. It's still unclear if there is a mismatch between fork() and spawn() in the backend or DDP forward is called twice recursively. Meaning that the tensor on GPU 0 is unnamed but on GPU 1 still named.


API does not work with Azure App Service Authentication enabled

Describe the bug
The API does not authenticate with API Keys when Azure App Service Authentication is enabled.
To Reproduce
Steps to reproduce the behavior:

Enable App Service Authentication in Azure
Restart the App Service
Try calling API using curl / python or any tool

Expected behavior
Json response with name and status code.
Additional context
The API works perfectly fine when Azure App Service Authentication is disabled.

Hi @ArlanNugara,
Thank you for your feedback. When using Azure AD Authentication for the App Service, you have to set OAuth 2 authentication in order to access the API. There are lots of ways to do this, depending on how you are calling the API. This is not specific to the Naming Tool, btu rather OAuth 2.0.
The overall process:

Create an app registration for your client
Create a secret for your client
Generate a token using your client/secret/scope details
Pass the token in an Authorization header for every API request

As an example, here is a good guide on accessing an Azure AD-protected API using Postman. The guide is good, but has a very areas that are may be a little confusing. I've tried to clarify this below.
https://dev.to/425show/calling-an-azure-ad-secured-api-with-postman-22co
For the guide, a few tips to help you get going:

You will need to create an app registration for the CLIENT that is trying to access the API. In the case of the referenced guide, this client is for Postman.
You should already have an app registration for your deployed service (API)
You will need to get the SCOPE for the API (your deployed service) to set as the scope.
As others have indicated int he comments in the article, I had to remove "api://" portion to resolve the "Invalid audience error".

Here is a working example:

HJopefully this helps you get started with the API and OAuth 2.0.

@BryanSoltis I am going to try this out now.

@BryanSoltis I am trying a small python script to get names. I am able to get Bearer Token from Azure. However, it's not working when passing the Token to Naming Tool API. The error is as follows : -
{'code': 401, 'message': "IDX10214: Audience validation failed. Audiences: 'https://management.azure.com/'. Did not match: validationParameters.ValidAudience: 'XXXX-XXXX' or validationParameters.ValidAudiences: 'api://XXXX-XXXX-XXXX'."}

The code is as follows -
import json
import os
import requests

# Login and get access token
LOGIN_URL = "https://login.microsoftonline.com/"+os.environ.get('ARM_TENANT_ID')+"/oauth2/token"
PARAMS_MGMT = {'grant_type':'client_credentials','client_id': ''+os.environ.get('ARM_CLIENT_ID')+'','client_secret':''+os.environ.get('ARM_CLIENT_SECRET')+'', 'resource':'https://management.azure.com/'}
login = requests.post(url = LOGIN_URL, data = PARAMS_MGMT)
jsonfy = login.json()
token = jsonfy["access_token"]
query_header = {"Content-Type": "application/json", "accept": "*/*", "Authorization": "Bearer "+token}
print("Login Successful")

# Payload
json_data = {
    "resourceEnvironment": ""+os.environ.get('ENVIRONMENT')+"",
    "resourceFunction": ""+os.environ.get('FUNCTION')+"",
    "resourceInstance": "001",
    "resourceLocation": ""+os.environ.get('LOCATION')+"",
    "resourceOrg": ""+os.environ.get('ORGANIZATION')+"",
    "resourceProjAppSvc": ""+os.environ.get('PROJECT')+"",
    "resourceType": "st"
}

# Get Names
URL = os.environ.get('API_URL')
get_names = requests.post(url = URL, headers = query_header, data = json.dumps(json_data))
get_names_json = get_names.json()
print(get_names_json)

You probably need to supply the scope, as well. You can find it in your App Registration (for the tool) / Expose an API:

-Bryan

@BryanSoltis The error remains same with scope and without scope. Also tried with "api://" and without "api://" in scope.

Hi @ArlanNugara ,
I suspect it's something related to your implementation of OAuth 2.0 and Azure AD. The tool itself does not have any code that enables/implements the authentication. If Authentication is implemented (Example: Enabling Authentication on the Azure App Service), this will inject headers into the request (for the user information). The tool looks for these headers and, if they are present, identifies the user/admin. I was able to get it working using Postman, but that required some very specific Azure AD App Registration.
I would recommend continuing to work on your OAuth 2.0 implementation and how Azure AD implement this. This link may provide some additional guidance:
https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols
-Bryan

Closed due to inactivity.


move dependency to locationtech JTS

com.vividsolutions.jts will continue further development at https://github.com/locationtech/jts. Geospark should consider moving to that library.

Gettos 0.20.x was just released. They already migrated to the new location tech JTS. As Geospark is based on both JTS and geotools one should strongly consider this upgrade in both JTS and geotools.


Docusaurus Issue

Not sure what is going on but when I navigate to pnpm.io this is what is displayed:

We were discussing on the chat, I think is not docusaurus (I am using also the latest one in verdaccio), either the Cloudfare cache or something else, if you refresh or use VPN the assets are fetched correctly. @zkochan could you check those settings?

Try to put something to the URL query https://pnpm.io/?1

Not reproducible, site shows up fine for me. Issue can be closed.


Update quick start in user guide

Resolves #156
For quickstart:

Add image on startup
Specify product version
Provide java installation instructions
Provide instructions on how to check java version
Update example

Codecov Report

Merging #220 (c1a97cd) into master (095dac5) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #220   +/-   ##
=========================================
  Coverage     56.83%   56.83%
  Complexity      499      499
=========================================
  Files            93       93
  Lines          2129     2129
  Branches        254      254
=========================================
  Hits           1210     1210
  Misses          849      849
  Partials         70       70

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Add script_fields context  to KNNAllowlist

Description
Include script_fields context to existing
supported context for knn methods.
Added test cases for method and doc values.
Related Issues
#1878
Check List

[x] New functionality includes testing.
[ ] New functionality has been documented.
[ ] API changes companion pull request created.
[x] Commits are signed per the DCO using --signoff.
[ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Can you reformat PR description? Just trying to understand why we are making this change

Can you reformat PR description? Just trying to understand why we are making this change

Updated description, for background please check linked github issue on why this PR is added


Progress Update 3

Almost thought I was done and ready to commit but there appears to still be a few things missing so instead I thought a progress update to let you know what's going on. This update will not be as exciting as some of the others, this is more architectural but perhaps you will also enjoy it as much as I have.
If you've been following the updates you'll recall at #64 item 3. I talked about the one thing that was not merged from the beanshell2 fork..  the refactoring of Reflect class. I always thought if we are going to make the effort then we need a solutions that starts using MethodHandlers at the very least.
Honestly I was trying to avoid it for this release because this is no small task, but while cleaning up the property implementation and realising the #444 issue is right on the money properties are not being cached at all. There is no more room for hacking around the issues we need to address the design, so time to bite the bullet and get this over and done with. At least I've had since May pondering an action plan so I had a pretty good idea where I wanted to go with this.
The first item on my wish list is a kick ass cache, WeakHashMap is all fine and well in blissful ignorance but if you stop to look at it closer it leaves much to be desired.

Both keys and values must be referenceable
Ability to choose the type of reference independently
Asynchronous value creation
Reference clean-up as a separate process

Ask and you shall receive... meet our boss cache.
                         +----------+  CreateValueRef +-----------------+
+-------------------+    | Future   +-----------------+ReferenceFactory |
| MemberCache       |    |          |                 |                 |
|                   |    +------+-+-+                 +-+--+--^---------+
| resolveMembers    |           | |        CreateKeyRef |  |
| cacheMember       |           | |        +------------+  |
| findField         |create(key)| |        |       +-------+---------------+
| findMethod        +->>>>>-----+ |        |       |HardReferenceFactory   |
| findProperty      |             |        |       |  +--------------------+-+
|                   |             |        |       |  | SoftReferencFactory  |
+-------------------+             |        |       +--+   +------------------+---+
                             +----+--------v--+       |   | WeakReferenceFactory |
                             | ReferenceCache |       +---+                      |
                             |                |           |                      |
                        <----+ get(key)       |           +----------------------+
                             |                |
                        +----> init(key)      |           +----------------------+
                             +--------^-------+           |ReferenceQueueMonitor |
                                      +-------------------+      implements      |
                                       Remove Cleared Key |             Runnable |
                                                          +----------------------+

Supports weak, soft and hard (strongly referenced) references which can be independently chosen. Key references capture the hashcode of the referenced key which is then used for equality without needing to dereference. Values are wrapped as Future which satisfies the async requirements too. The queue monitor uses a blocking call to remove garbage collected entries and clear the cached references so basically it just sits idly by when there is nothing to do.
There is no putting values in the cache when you implement the cache you provide it with a create value method, in the diagram above we are creating MemberCache instances for the supplied key which in our case is the Class for which we will then resolve and cache members for, it also provides the functionality to find members. To use you simply call get with a key. if it doesn't exist it will be created while we wait. If you don't immediately need the entry you can use innit which will start the process and the value should be ready for you the next time you call get.
Works like a charm I am very pleased. =)
For the member implementation I decided to go with following delegate model, Invocable.
                       +-------------+
                       |Invocable    |
                       |  implements |
                       |      Member |
                       |             |
                       |             |
                       +------^--^---+
                              |  |
                              |  +------------------------+
                              |                           |
                       +------+---------------+    +------+-------------+
                       | ExecutingInvocable   |    | FieldAccess        |
                       |                      |    |                    |
                       |                      |    | invoke(base, args) |
                       +--------^-----^-------+    +--------------------+
                                |     |
                                |     |
                                |     +---------------------+
                                |                           |
                                |                           |
                       +--------+-------------+    +--------+-----------+
                       | ConstructorInvocable |    | MethodInvocable    |
                       |                      |    |                    |
                       |  invoke(base, args)  |    | invoke(base, args) |
                       +----------------------+    +--------------------+

Everything is now a single type there's no need to treat constructors different than methods, even fields act as an Invocable. An Invocable also serves as drop in replacement for reflect members so we can have a minimum impact approach. MethodHandle lookup is an inherited cascade chain where each relative may add to the handle creation. Parameter collection works the same where Invocable does the basic parameters, Executable is mainly concerned with the varargs collection and the children basically only needs to supply the class or instance based on isStatic or not.
I went with unreflected method handles as first exposure because the current design expects to find a member that we know of in advance, requires members to query their method types and signatures and so for so this was the logical approach. As we clean up more of the legacy code we simply extend the Invocable model with support for different method handles.
Success!
Everything is working exceptionally well, accessibility for starters now works as expected you can access anything from anyone anywhere, (given there is no default security manager) which for some reason was not possible before. Since everything just seems to work I managed to find several new items that weren't implemented or not working properly.

We had no support to instantiate java inner classes (generated types well)
Import static field doesn't work (import static methods do)
ran into a major memory leak, which I will create another issue for, and now had the perfect tools to fix it once.

Everything works as expected there are just a few cosmetics that still give me creeps, I'll just need another day or two, thanx for patience.
nJoy!

Property types
Forgot to mention this which I thought was really awesome. The biggest design flaw with the bean specification, coupled with java's String hangups, is in the complexity of property name to method name conversion. The process of stripping get, set or is from the method name and then having te lower case the first char to find the property name as well as the reverse process of having to uppercase the first char and the concatenating the string prefix get, set or is is exorbitantly expensive.
In the new design MethodInvocable has the ability to identify itself as a getter or setter so we only have no find the property name once only, which we additionally associate with the same member by reference in cache so you can lookup properties by property name. Sweet! =)


missing break.

when used with web caching strategy, it forces to enable cacheonly policy, which makes it impossible to use the option.

That is not very good. Thank you!


Autodetect meta when creating lazy data

🚀 Pull Request
Description

Try if not specifying meta solves #5800.

Consult Iris pull request check list

Add any of the below labels to trigger actions on this PR:

https://github.com/SciTools/iris/labels/benchmark_this

Hi @bouweandela
The existing test breakage is a fairly minor matter, but I think it indicates that removing line 287b is causing the da.from_array to do trial access on the data -- so I think it is probably better to continue to provide a meta key in this case.
However, I am also concerned that ideally we should also be addressing #5800 if possible (!)
In which case, I think this should have meta=np.ma.MaskedArray in place of np.ndarray, when appropriate.
From my reading of dask docs, I think it may also be better to pass a zero-shaped instance instead of a class, since the from_array docs don't actually mandate that usage.
And it also doesn't actually work if you pass np.ma.MaskedArray instead : you get a TypeError: MaskedArray.__new__() got an unexpected keyword argument 'shape'.
So I think something like da.from_array(real, meta=real.__class__((), dtype=real.dtype)) works here, and is probably a better way.
Does this make sense to you ?

@pp-mo I also updated the concatenate function so it avoids the slicing issue you mentioned in https://github.com/SciTools/iris/issues/5800#issuecomment-2025165305. This is now ready for review.
I suspect that the pre-commit failure is unrelated to this pull request. When I run pre-commit run --all-files on the main branch I get the same error.

@bouweandela I suspect that the pre-commit failure is unrelated to this pull request. When I run pre-commit run --all-files on the main branch I get the same error.

Yes, puzzling. We are looking into it...

Ok I think we know what this is now  -- fix coming soon.
It's only "odd" because the flake8 error code is bugged !

Thank you for reviewing @pp-mo! I just merged the latest main into this branch and the pre-commit test is passing fine now :partying_face:

However, I think we need to rethink the merge, since some changes from https://github.com/SciTools/iris/pull/5588 seem to have got lost.
I guess that means that merging back into this will cause problems
( I expect at least some tests ought then to fail, otherwise where is the testing of those new keywords ?? )
So I haven't updated with main, as you may want to resolve this in your own way.

I refactored the as_lazy_data function a bit, but the functionality is still there (see https://github.com/SciTools/iris/pull/5801#discussion_r1580022494). If you prefer, I can undo that?

If you prefer, I can undo that?

Thanks no, I just hadn't understood fully what you were doing.  It's good !


Reinstate Courts and Tribunals publishing with correct URLs

This reinstates Courts and HMCTS Tribunals (aka judicial bodies) into the Publishing and Indexing process with the correct URLs. These were set up in #2114.

Is it possible to get courts working with the publishing and indexing process without overriding organisation_url and organisation_path? The overridden organisation_url method is quite hairy...

The overridden helper is used in views too. In theory I could just override #search_link with the same logic if this wasn't the case. This seems like a cleanish solution. Having looked back at it now though, I can add more test cases for the helper. I'll do that now.

I think it would be safer to push the logic closer to the places where it is needed. The overridden url helper method is highly complex, with multiple nested if/else's and multiple return calls, making it hard to reason about and also more likely to introduce bugs. The need for overridden url helpers for documents and editions is an unhappy side-effect of the way documents and editions are modelled with STI. If we can avoid doing the same for other models, that would be great.
How about using the custom organisation_or_court_path helper in the views as before, overriding search_ink in Organisation, and updating PublishingApiPresenters::Placeholder to use the search_link instead of Whitehall.url_maker.polymorphic_path(item) to get the base_path? Would that cover all bases?

The organisation_or_court_path helper would have to have the same logic, to cover all bases. It didn't before. Putting the logic into #search_link would mean either including the OrganisationsHelper in the model, or duplicating the code.
I followed the pattern of overriding _path and _url helpers in public_document_routes_helper. The code is now a bit tidier, would you mind having another look please @tekin?

This is marked Do Not Merge as we're waiting for owner approval on the courts and tribunals pages. Merging this will make those pages discoverable on the site.

:+1: A few minor points – this looks good otherwise.

Updated.

:+1: Code looks good


[backup] Update paired region for southeastus

Related command

az backup
Description

Adding a paired region (westus3) for the southeastus region.
Testing Guide

History Notes

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

Thank you for your contribution! We will review the pull request and get back to you soon.


Fix MTL in Transformer encoder

Codecov Report

Merging #256 (fdc8525) into master (0f3c6e2) will decrease coverage by 0.03%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master     #256      +/-   ##
==========================================
- Coverage   64.28%   64.24%   -0.04%
==========================================
  Files          85       85
  Lines       10289    10290       +1
==========================================
- Hits         6614     6611       -3
- Misses       3675     3679       +4

Impacted Files
Coverage Δ

neural_sp/models/seq2seq/encoders/transformer.py
71.26% <33.33%> (-0.21%)
:arrow_down:

neural_sp/models/seq2seq/decoders/ctc.py
47.33% <0.00%> (-0.95%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f3c6e2...fdc8525. Read the comment docs.

Codecov Report

Merging #256 (fdc8525) into master (0f3c6e2) will decrease coverage by 0.03%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master     #256      +/-   ##
==========================================
- Coverage   64.28%   64.24%   -0.04%
==========================================
  Files          85       85
  Lines       10289    10290       +1
==========================================
- Hits         6614     6611       -3
- Misses       3675     3679       +4

Impacted Files
Coverage Δ

neural_sp/models/seq2seq/encoders/transformer.py
71.26% <33.33%> (-0.21%)
:arrow_down:

neural_sp/models/seq2seq/decoders/ctc.py
47.33% <0.00%> (-0.95%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f3c6e2...fdc8525. Read the comment docs.


curl.setOption(CURLOPT_POSTFIELDS) causes use-after-free of the fields parameter

It seems that Curl doesn't make a copy of the post fields, so once setOption returns the CString is gone and curl reads garbage from the heap instead.

It appears that the only place this problem is encountered is PerfectLibTests.swift. Therefore, it's not a critical security flaw, though it might cause tests to fail in certain cases. Obviously it needs to be fixed. The data should either be copied or made global so it's not invalidated upon return from the function.

I added a comment to the code regarding how this operation works. I don't think it's practical to make copies of all such pointers so this is maybe the only thing to be done for now. I adjusted the test case accordingly.


Exercise indicator doesn't turn yellow on unsaved changes (text, programming)

Describe the bug
If you edit a text or programming exercise the exercise indicator at the top doesn't turn yellow like it should on unsaved changes.
To Reproduce

Edit an text and programming exercise in the exam
Indicator doesn't turn yellow

Expected behavior
Indicator turns yellow and blue when saved
Screenshots
Didn't made a screenshot during the test exam but I think it's pretty straight forward.
Environment

 - OS: Windows 10 Pro
 - Browser: Chrome v83.0.4103.116

This is currently intended behaviour as far as I understand. Only when the connection is lost it will turn yellow.

Then this is bad design in my opinion since the user should know all the time what is saved an what not to see one a problem araises with saving.
What are your thoughts on this @filip-gregurevic as Dev Lead, @Varnika-Tyagi as Test Lead

should be fixed now


Regex Error in Safari / WebKit browsers (including Gnome Web)

The comment regex in the Postgres column compiler has a look-behind. This isn't supported by Safari or WebKit, and so causes any web bundled app to crash.
This line:
const commentEscapeRegex = /(?<!')'(?!')/g;

This file:
https://github.com/knex/knex/blob/master/lib/dialects/postgres/schema/pg-columncompiler.js#L7
Here's an error output from Gnome Web (Midori):

+1, crash on Safari. Please fix it.


Showing right answer of Rebus

Expected Behavior
Adding a button to show the answer if one cannot guess the rebus after looking at the hint.
...
Actual Behavior
Answers can't be seen. Only confetti is shown after the correct answer.
...

I can take this issue

What frameworks are being used in this project?

#737

I want to contribute to this issue.


Queue all apps eligible for redeployment on each run

maybe relax the PeriodicApplicationMaintainer interval a little bit? Currently at 1 minute.

Not sure that's needed?  The interval just controls how often we check for and queue up eligible applications. There will still be minimum 30 minutes between deployments of an application.

Its not needed, just no point in running it that often since the maintain() will iterate over several applications in pendingDeployments every time it is called, so making frequent changes to pendingDeployments is pointless as they wont be picked up until the current maintain() finishes and next one is called. If the maintain() takes more than 30 minutes, almost all applications will be in the pendingDeployments by the time maintain() is called again.
This also affects job control. If we just disable PeriodicApplicationMaintainer, it will keep on deploying all already scheduled applications before stopping.

maintain() will iterate over several applications in pendingDeployments every time it is called

maintain() doesn't iterate over pendingDeployments, the former is only used to ensure uniqueness of the queue in deploymentExecutor (the executor is backed by a BlockingQueue).

If the maintain() takes more than 30 minutes, almost all applications will be in the pendingDeployments by the time maintain() is called again.

maintain() will never block for long periods as deployments are async in this maintainer. The whole idea is too fill the queue with unique applications as often as possible.

it will keep on deploying all already scheduled applications before stopping

This behavior is true for all our maintainers.

Test hangs on Travis for some reason. Checking.


Cannot build: mqtt_payload not declared in scope

Running the second build command from the readme: https://github.com/PLCnext/MqttClient#building-from-source results in two errors, the second being fatal:
../../libs/MqttClientService/src/MqttClientManager.cpp:161:11: error: reference to ‘byte’ is ambiguous
const byte * mqtt_payload = payload.GetDataAddress();

../../libs/MqttClientService/src/MqttClientManager.cpp:165:74: error: ‘mqtt_payload’ was not declared in this scope; did you mean ‘setpayload’?
  165 |         this->client->Publish((Client::client_id_t)clientId, mqtt_topic, mqtt_payload, length, qos, retained);

The byte problem is a known issue that was introduced in version 2021.6 of the firmware:
https://www.plcnext.help/te/Known_issues.htm
(in the "Programming" section).
Unfortunately we have not updated this project for some time, so I am afraid that it has fallen behind developments in the PLCnext Control firmware and SDKs.
We have decided not to maintain this project, since (a) the general concepts demonstrated here are now also demonstrated in other repositories, (b) it will not be possible in the forseeable future for users to develop their own RSC services for the PLCnext Runtime, and (c) there are now a number of alternatives for users who require MQTT Client functionality in their PLCnext Control projects.
I have now added a note to the top of the README, which we should have done some time ago (sorry).
If there is a particular reason that you need this specific component for your project, please let know.


Local Node rescan only finds first 20 addresses

I'm submitting a…

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request

Expected Behavior
When doing a rescan with a local bitcoind, I expect that all used addresses in the wallet will be found (at least if they were used sequentially and don't have gaps larger than the address gap limit). This is necessary to show the correct balance for the wallet.
Current Behavior
It has been reported that only the first 20 addresses (and change addresses) are found:
https://twitter.com/dlawnroac/status/1269229613282201600
This means the wallet balance isn't correct. Additionally, I don't think the user is notified
that an additional rescan may be needed, leaving the unaware user in an incorrect state.
Possible Solution
I haven't used it myself, but I read that importmulti now accepts multisig descriptors, and start-end indices for searching.
If that doesn't work, we could expand the import to at least a larger number (1000?).
And/or notify the user that that they may not have found all their used addresses.
Steps to Reproduce (for bugs)
I didn't reproduce this one yet, but I suspect these are the steps:

Set up a wallet and use > 20 addresses.
Load that same wallet into a fresh caravan session.
Connect to a local bitcoind node and start a rescan.
When the rescan is complete, the balance will not be correct, and there won't be a notification that something may be incorrect.

Environment
This was reported for a myNode user on June 6, 2020.

Issue ACK. Tested out with a wallet on testnet with over 50 addresses used.
While this is no means an acceptable long term solution, a temporary workaround that did work for me is to import + rescan multiple times. This is obviously faster on testnet and a relatively heavy lift on mainnet, but at least the wallet balance will show correctly eventually.
importmulti with descriptors seems like a really great long term solution. There are plans for adding descriptor support into unchained-bitcoin, so once that's available this should be an easy update to make.

So you would re-rescan until you saw a number of unused addresses (or you see the last address you remember using), right?

Yep, exactly. You can also just keep doing it until you didn't see any new addresses via the lookahead. Since Caravan is following Core's 20 address lookahead policy and there's currently no way to customize this, it will keep updating on each new rescan until there are 20 addresses without any history or balance. If for some reason, a deposit was made to an address with a gap from the previously used address of 21+, then there is no way for Caravan to catch that. For example, if there was a deposit at path */0/40 and then the next one is at */0/70, then it won't ever get imported. Sending to address */0/59 would bump that up though as a way to get around that particular situation.

I've seen this bug too but multiple imports/rescans hasn't worked for me. Another idea could be let user pick date/blocks ranges to rescan instead of full one that way takes less time with this command:
bitcoin-cli rescanblockchain<PHONE_NUMBER>00

The Caravan Library is transitioning to a new home, and as part of this move current issues will be closed and archived. Our engineers believe this issue is now resolve, if you are still experiencing this you are welcome to re-open the issue in the new location: https://github.com/caravan-bitcoin/caravan/issues