Simulate prod test

Done:
Here I am including some missing tests to check whether the files for either corsika or simtel_array output were generated by the production tool. We are constantly breaking this tool because the tests are not checking whether the output files exist. The integration tests for simulate_prod.py will fail at the moment.
I removed the test flag for sim_showers_for_trigger_rates because it was preventing us to simulate and de facto test the tool. To minimize computational time of the simulation, I reduced the number of events to 2.
To do:
The best would be to make the corrections to the tool before merging this and have the integration tests passing.

I've looked into
pytest --no-cov "tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg]"
and files (although very small ones are generated for me at /tmp/pytest-of-root/pytest-1/test-data0/simtools-simulate-prod-gamma_20_deg/simtools-output/test-production//simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst
The log files look reasonable and it runs through without errors.
There are issues, many statements like
Warning: problematic transmission between telescope and observation level.
Evaluation was between observation layer at 21.560000 m a.s.l.
and 31.000000 m above, plus 16.000000 m in the telescope.
Lookup in transmission table in wavelength range from 200 to 1000 nm,
   altitude range from 2.156000 to 100.000000 km a.s.l.,
   for wavelength 400 nm from 0.052560 km a.s.l., for airmass factor 1.000000
Starting altitude of 0.052560 km is out of range (2.156000 to 100.000000 km).
This is either an underground installation or the wrong transmission table.
At 400 nm we get a transm. of 0.000000 between tel. and obs.lev., and 0.000000 inside tel.
Your transmission table might not be appropriate for the given site.
We are ignoring this problematic part now but you should fix your set-up.

in the log file.

I've looked into
pytest --no-cov "tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg]"
and files (although very small ones are generated for me at /tmp/pytest-of-root/pytest-1/test-data0/simtools-simulate-prod-gamma_20_deg/simtools-output/test-production//simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst
The log files look reasonable and it runs through without errors.
There are issues, many statements like
Warning: problematic transmission between telescope and observation level.
Evaluation was between observation layer at 21.560000 m a.s.l.
and 31.000000 m above, plus 16.000000 m in the telescope.
Lookup in transmission table in wavelength range from 200 to 1000 nm,
   altitude range from 2.156000 to 100.000000 km a.s.l.,
   for wavelength 400 nm from 0.052560 km a.s.l., for airmass factor 1.000000
Starting altitude of 0.052560 km is out of range (2.156000 to 100.000000 km).
This is either an underground installation or the wrong transmission table.
At 400 nm we get a transm. of 0.000000 between tel. and obs.lev., and 0.000000 inside tel.
Your transmission table might not be appropriate for the given site.
We are ignoring this problematic part now but you should fix your set-up.

in the log file.

Hi @GernotMaier, thanks for checking. This also works for me when running pytest in the container as you suggested. However, it does not work when running the application in the terminal (even in the master branch): simtools-simulate-prod --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml. The simtools-output/test-production/simtel-data/North/gamma/data/ directory is empty because of an error in the corsika part of the script as pointed out in the corsika log file in simtools-output/test-production/simtel-data/North/gamma/log/:
Setting up pipe 1 as: /workdir/sim_telarray/sim_telarray/bin/sim_telarray -c /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg -I/workdir/external/simtools/simtools-output/test-production/model -C telescope_theta=20.0 -C telescope_phi=0.0 -C power_law=2.0 -C histogram_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.hdata.zst -C output_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst -C random_state=none -C show=all - | gzip > /workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.log.gz 2>&1 || exit
Configuration file is '/workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg'.
Preprocessor command is '/workdir/sim_telarray/sim_telarray/bin/pfp -v -I. -DWITH_LOW_GAIN_CHANNEL -DMAX_GAINS=2 -DWITH_GSL_RNG -DSIMTEL_VERSION=1635199717 -DSIMTEL_RELEASE=20211027 -I/workdir/external/simtools/simtools-output/test-production/model -I. -I/workdir/sim_telarray/sim_telarray/cfg -I/workdir/sim_telarray/sim_telarray/cfg/common -I/workdir/sim_telarray/sim_telarray/cfg/hess -I/workdir/sim_telarray/sim_telarray/cfg/hess2 -I/workdir/sim_telarray/sim_telarray/cfg/hess3 -I/workdir/sim_telarray/sim_telarray/cfg/hess5000 -I/workdir/sim_telarray/sim_telarray/cfg/CTA /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg /tmp/cfg269.tmp'.
Unknown configuration item 'site_config_name'.
Unknown configuration item 'site_config_variant'.
Unknown configuration item 'site_config_version'.
Syntax error for configuration item 'EFFECTIVE_FOCAL_Length': Index out of range
Unknown configuration item 'fadc_compensate_pedestal'.
Unknown configuration item 'fadc_err_compensate_pedestal'.
Unknown configuration item 'fadc_lg_compensate_pedestal'.
Unknown configuration item 'fadc_lg_err_compensate_pedestal'.
Error in configuration data or incomplete configuration
Error in initialisation of the configuration
Error in configuration

Setup not complete.
Received signal 13
Broken pipe (unit 1) detected. Trying to handle this gracefully.
Closing pipe 1
No process left listening to me. Giving up.
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer

Could you please try to reproduce this?

Also the file you pointed out that exists is already being checked for in the integration test in this branch, which lead to a failure in the test. I am not entirely sure what is going on.
https://github.com/gammasim/simtools/actions/runs/8723725721/job/23932598219?pr=875#step:5:554

@VictorBarbosaMartins , I can run python simtools/applications/simulate_prod.py --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml without any error messages.
Are you sure you are using a recent installation of simtools?
(there are some warnings regarding observation levels, which are related to something else)

@VictorBarbosaMartins - everything runs fine with me locally without any errors. Can you check again?
Also please check the integration test which is failing on github (tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_pack_for_grid]). It is probably mostly related to the paths where output is located.

@VictorBarbosaMartins , I can run python simtools/applications/simulate_prod.py --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml without any error messages.
Are you sure you are using a recent installation of simtools?
(there are some warnings regarding observation levels, which are related to something else)

Hi @GernotMaier. The problem is not visible at first, one has to dig in. Naturally there is no error printed out, and that is the problem(!) The application runs smoothly (simtools-simulate-prod --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml), but there is no simtel file produced. If one checks the log file it doesnt look too bad, then when one looks at the corsika log file, a number of unknown configuration items break the process. I already posted the error here, but here it is again (in log_corsika_run000002_gamma_North_TestLayout_test-production.log):
Setting up pipe 1 as: /workdir/sim_telarray/sim_telarray/bin/sim_telarray -c /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg -I/workdir/external/simtools/simtools-output/test-production/model -C telescope_theta=20.0 -C telescope_phi=0.0 -C power_law=2.0 -C histogram_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.hdata.zst -C output_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst -C random_state=none -C show=all - | gzip > /workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.log.gz 2>&1 || exit
Configuration file is '/workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg'.
Preprocessor command is '/workdir/sim_telarray/sim_telarray/bin/pfp -v -I. -DWITH_LOW_GAIN_CHANNEL -DMAX_GAINS=2 -DWITH_GSL_RNG -DSIMTEL_VERSION=1635199717 -DSIMTEL_RELEASE=20211027 -I/workdir/external/simtools/simtools-output/test-production/model -I. -I/workdir/sim_telarray/sim_telarray/cfg -I/workdir/sim_telarray/sim_telarray/cfg/common -I/workdir/sim_telarray/sim_telarray/cfg/hess -I/workdir/sim_telarray/sim_telarray/cfg/hess2 -I/workdir/sim_telarray/sim_telarray/cfg/hess3 -I/workdir/sim_telarray/sim_telarray/cfg/hess5000 -I/workdir/sim_telarray/sim_telarray/cfg/CTA /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg /tmp/cfg217.tmp'.
Unknown configuration item 'site_config_name'.
Unknown configuration item 'site_config_variant'.
Unknown configuration item 'site_config_version'.
Syntax error for configuration item 'EFFECTIVE_FOCAL_Length': Index out of range
Unknown configuration item 'fadc_compensate_pedestal'.
Unknown configuration item 'fadc_err_compensate_pedestal'.
Unknown configuration item 'fadc_lg_compensate_pedestal'.
Unknown configuration item 'fadc_lg_err_compensate_pedestal'.
Error in configuration data or incomplete configuration
Error in initialisation of the configuration
Error in configuration

Setup not complete.
Received signal 13
Broken pipe (unit 1) detected. Trying to handle this gracefully.
Closing pipe 1
No process left listening to me. Giving up.
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer

To worsen even more, I found out that the integration test of the output file (pytest --no-cov "tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg]") also does not raise an error but it is indeed not working. Tests are passing regardless of the name of the file one puts there.
I will look into these issues now but I will need help with the issue with the parameters. Please follow my suggestion above and check the corsika log file.

The message above is quite clear on where the errors are coming. Wrong names of parameters. I also could not find those parameters in the DB. I am convinced these names were added/changed in the last updates (note also at the effective focal length error, which we also just changed the code there):
Unknown configuration item 'site_config_name'.
Unknown configuration item 'site_config_variant'.
Unknown configuration item 'site_config_version'.
Syntax error for configuration item 'EFFECTIVE_FOCAL_Length': Index out of range
Unknown configuration item 'fadc_compensate_pedestal'.
Unknown configuration item 'fadc_err_compensate_pedestal'.
Unknown configuration item 'fadc_lg_compensate_pedestal'.
Unknown configuration item 'fadc_lg_err_compensate_pedestal'.
I found some of these variables in the
https://gitlab.cta-observatory.org/cta-science/simulations/simulation-model/model_parameters/-/issues/16
They seem not to be part of the model and are written to the config file anyways. Do you see any straightforward solution @GernotMaier ?

The command simtools-simulate-prod --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml is working fine for me (there are some warnings in the log files which needs to be addressed, but generally both CORSIKA and sim_telarray are working fine).
How is your setup?
I am running it in a container (podman or docker; doesn't matter):
podman run --rm -it -v "$(pwd)/:/workdir/external" ghcr.io/gammasim/simtools-dev:latest bash -c "source /workdir/env/bin/activate && cd /workdir/external/simtools && pip install -e . && bash

With a container from 2024-04-17T18:05:27.34498581Z (not the latest; but reasonable new).
Not sure what else we can check.

Ok, I think we are slowly clarifying things here. I fixed the test_applications_from_config[simtools-simulate-prod_gamma_20_deg_pack_for_grid] test, which is now working in my container. However, #887 will have to be merged first, otherwise the test will not run for checking the output files

@tobiaskleiner, same here :D I guess we are ready to go?

Thanks again!

Merging here as well as this is approved and it is complementing the #906


Code Architecture For Contributors wiki page : needs update

Since I love what you guys are doing with RA, I wanted to contribute the project but was a pushed back by the app level of complexity. That is when I found the page [Code Architecture For Contributors] (https://github.com/sferik/rails_admin/wiki/Code-Architecture-For-Contributors) in the wiki started by @mlightner. Unfortunately, it has not been touched in nearly 6 months.
It would be great if someone experienced with RA could fill the doc. Even a high level view would be good and decrease the barrier of entry for contributors.

Moving toward an API (in Fields Base class; Actions Base class).
This needs detailing, but not needed for first gem. Moving to 0.1


Crash in FoundationStream

Starscream version: 3.1.1
iOS version: 13.3.0
Device: iPhone 11 Pro
Stack trace:
Crashed: NSOperationQueue 0x127d32140 (QOS: UNSPECIFIED)
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x00000000327b5b10

0  libobjc.A.dylib                0x1987b0fb0 objc_msgSend + 16
1  Starscream                     0x10602dc7c $s10Starscream16FoundationStreamC5write4dataSi0B04DataV_tF + 264
2  Starscream                     0x10602eb1c $s10Starscream16FoundationStreamCAA8WSStreamA2aDP5write4dataSi0B04DataV_tFTW + 20
3  Starscream                     0x106035b48 $s10Starscream9WebSocketC12dequeueWrite33_E91840E7069A0D2D302D042308E9D928LL_4code15writeCompletiony10Foundation4DataV_AC6OpCodeOyycSgtFyycfU_ + 1800
4  Starscream                     0x10602c370 $sIeg_IeyB_TR + 28
5  Foundation                     0x198e6ca44 __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 24
6  Foundation                     0x198d6e518 -[NSBlockOperation main] + 104
7  Foundation                     0x198e6eda0 __NSOPERATION_IS_INVOKING_MAIN__ + 24
8  Foundation                     0x198d6e194 -[NSOperation start] + 740
9  Foundation                     0x198e6f7cc __NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ + 24
10 Foundation                     0x198e6f28c __NSOQSchedule_f + 184
11 libdispatch.dylib              0x19874e39c _dispatch_block_async_invoke2 + 144
12 libdispatch.dylib              0x198741fd8 _dispatch_client_callout + 20
13 libdispatch.dylib              0x198744758 _dispatch_continuation_pop + 408
14 libdispatch.dylib              0x198743eb8 _dispatch_async_redirect_invoke + 596
15 libdispatch.dylib              0x198750bd4 _dispatch_root_queue_drain + 348
16 libdispatch.dylib              0x198751384 _dispatch_worker_thread2 + 120
17 libsystem_pthread.dylib        0x1987a7690 _pthread_wqthread + 216
18 libsystem_pthread.dylib        0x1987ad9e8 start_wqthread + 8

I received this crash from Crashlytics so not sure how to reproduce it, it has only happened once so far.

I will close this as the long term plan is to move to Network.framework or native URLSessionWebSocketTask.
I have seen these crashes and I think there are some memory issues with FoundationStreams.

Thanks for the answer @fassko, do you have an ETA on the move to Network.framework or URLSessionWebSocketTask? No rush, just curious :)


Push notifications overwrite the most recent one (Android)

Expected Behaviour
All notifications for my app are displayed in the notification panel.
Actual Behaviour
Just shows the latest notification for my app because it overwrites the most recent notification.
Reproduce Scenario (including but not limited to)
Push notifications on Android.
Steps to Reproduce

Open the app.
Send it to background.
Send a notification through the API.
Send a second notification through the API.

Platform and Version (eg. Android 5.0 or iOS 9.2.1)
Android >= 5.0
(Android) What device vendor (e.g. Samsung, HTC, Sony...)
No success on any device.
Cordova CLI version and cordova platform version
cordova --version  6.5.0
cordova platform version android  6.1.2

Plugin version
cordova plugin version | grep phonegap-plugin-push 1.10.1

Sample Push Data Payload
{
    "tokens": ["0c1de133b0beffb8e6d9584566ecb73f3d39d52cad4529549d44855bc127d238"],
    "profile": "profilesaludmovil",
    "notification": {
        "message": "Individual",
        "ios": {
            "title": "Titulo",
            "message": "Prueba individual salud movil",
            "sound":"ping.aiff"
        },
        "android": {
            "message": "Individual"
        }
    }
}

@Davegb that is actually the expected behaviour. Please read: https://github.com/phonegap/phonegap-plugin-push/blob/master/docs/PAYLOAD.md#stacking

@Davegb that is actually the expected behaviour. Please read: https://github.com/phonegap/phonegap-plugin-push/blob/master/docs/PAYLOAD.md#stacking

Thank you. I tried this and now I'm testing with the following JSON, but happens the same thing.
{
    "tokens": ["TOKEN"],
    "profile": "profilesaludmovil",
    "notification": {
        "message": "Individual",
	"notId": 1, //Then I send another with 2 or 3 and replaces the previous one
        "ios": {
            "title": "Titulo iOs",
            "message": "Prueba individual salud movil",
            "sound":"ping.aiff"
        },
        "android": {
            "message": "Individual"
        }
    }
}

@Davegb right, well it seems like you are sending via the Ionic push service and I'm not sure if it supports the notId property but I'd try:
{
    "tokens": ["TOKEN"],
    "profile": "profilesaludmovil",
    "notification": {
        "message": "Individual",
        "ios": {
            "title": "Titulo iOs",
            "message": "Prueba individual salud movil",
            "sound":"ping.aiff"
        },
        "android": {
            "notId": 1, //Then I send another with 2 or 3 and replaces the previous one
            "message": "Individual"
        }
    }
}

@macdonst Thanks a lot! You set me in the right track. This is the structure:
{
    "tokens": ["TOKEN"],
    "profile": "profilesaludmovil",
    "notification": {
        "message": "Individual ",
        "ios": {
            "title": "Titulo iOs",
            "message": "Prueba individual salud movil",
            "sound":"ping.aiff"
        },
        "android": {
	     "data": {
                  "notId": 6, //notId goes inside android.data
            	  "message": "Individual"
	     }
        }
    }
}


Show task progress in visualizer workers tab.

Description
This PR adds the task progress percentage as a bootstrap progress bar in a new column to the table of running tasks in the "Workers" tab:

The value is even updated when the message modal of the specific task is open (which already had the ability to update task messages and progress via polling).
I didn't include the new column in the actual "Task List" tab, because it is probably only important for currently running tasks.
Motivation and Context
When dealing with a couple long-running tasks, it is a nice feature to see the progress per task already in the worker tab.
Have you tested this? If so, how?
The behavior is rather static, and I couldn't find existing tests that check similar features. However, I verified that the automatic updating via the refreshInterval of the opened modal is working properly and handles also null values.

I've never dealt with the UI portion of Luigi. Do you have a colleague you can tag for review? Or perhaps a previous Luigi UI contributor?
I like the idea here.

I actually contributed a few UI related PRs such as task messages, worker processes customization, scheduler → worker communication, etc, but I think avoiding self-review is the point here ;)
@yrath @bfis Could you maybe give it a try? I prepared some test tasks to run in this gist.

Sorry for the late reply. I've checked out the PR based on the gist, everything worked as expected and looks good to me.

@yrath Thanks for testing!

@dlstadther Is this ok for you or should I provide additional tests?

Fellow review and testing works for me here. I'll merge.


Encription.php reported as malware

When I host panada to a shared host, Encription.php was detected as malware. It caused my hosting account  suspended.
HOST:      xxx.xxx.net
SCAN ID:<PHONE_NUMBER>.525815
STARTED:   Oct 21 2015 03:32:59 +0700
COMPLETED: Oct 21 2015 03:35:24 +0700
ELAPSED:   145s [find: 20s]

PATH:          /home*/*/public_html/ /var/www/html/ /usr/local/apache/htdocs/
RANGE:         1 days
TOTAL FILES:   61587
TOTAL HITS:    1
TOTAL CLEANED: 0

SUSPENDED ACCOUNTS:
xxx

FILE HIT LIST:
{HEX}php.base64.v23au.185  :  /home/xxx/public_html/xxx/panada/Resources/Encryption.php  =>  /usr/local/maldetect/quarantine/Encryption.php.187503914
===============================================
Linux Malware Detect v1.5 <<EMAIL_ADDRESS>>

If you didn't use this library at any your controller or didn't use cookie base session with encryption enabled, you can remove this file.
And thanks for your report. We will change the way encryption handling on this class.
On Oct 21, 2015 5:01 AM, Yusuf Ali<EMAIL_ADDRESS>wrote:When I host panada to a shared host, Encription.php was detected as malware. It caused my hosting account  suspended.
HOST:      xxx.xxx.net
SCAN ID:<PHONE_NUMBER>.525815
STARTED:   Oct 21 2015 03:32:59 +0700
COMPLETED: Oct 21 2015 03:35:24 +0700
ELAPSED:   145s [find: 20s]
PATH:          /home*/*/public_html/ /var/www/html/ /usr/local/apache/htdocs/
RANGE:         1 days
TOTAL FILES:   61587
TOTAL HITS:    1
TOTAL CLEANED: 0
SUSPENDED ACCOUNTS:
xxx
FILE HIT LIST:
{HEX}php.base64.v23au.185  :  /home/xxx/public_html/xxx/panada/Resources/Encryption.php  =>  /usr/local/maldetect/quarantine/Encryption.php.187503914
Linux Malware Detect v1.5 <<EMAIL_ADDRESS>>
—Reply to this email directly or view it on GitHub.


Potential memory leak in 'shallow_clone_vec' when concurrently upgrading from vec to arc

Hi I was interested in how this crate actually works. While reading the source I stumbled about this piece of code:
https://github.com/tokio-rs/bytes/blob/050d65b2cee8b2272687d798dc209dc03fe92719/src/bytes.rs#L1093
    match atom.compare_exchange(ptr as _, shared as _, Ordering::AcqRel, Ordering::Acquire) {
        Ok(actual) => {
            debug_assert!(actual as usize == ptr as usize);
            // The upgrade was successful, the new handle can be
            // returned.
            Bytes {
                ptr: offset,
                len,
                data: AtomicPtr::new(shared as _),
                vtable: &SHARED_VTABLE,
            }
        }
        Err(actual) => {
            // The upgrade failed, a concurrent clone happened. Release
            // the allocation that was made in this thread, it will not
            // be needed.
            let shared = Box::from_raw(shared);
            mem::forget(*shared);

            // Buffer already promoted to shared storage, so increment ref
            // count.
            shallow_clone_arc(actual as _, offset, len)
        }
    }

I am pretty sure the mem::forget should be a mem::drop so the behaviour matches the comment. forget causes the Box to never be dropped and thus the allocations will not be released.

Because of the dereference operation, the thing passed to mem::forget is a Shared without a box around it. This is to avoid running the destructor of fields in the Shared.
The box allocation is released by the dereference operation.


"Login failed" with "generic_request_error"

So since a couple of days ago, I've been unable to run my script. I get the login failed error, which is then usually followed by the reason why the error failed. So I modified the script to print out the whole login_text variable (line 257) and this is what I get:
{u'status': u'ok', u'error_type': u'generic_request_error', u'errors': {u'error': [u'Sorry, there was a problem with your request.']}}
Any ideas what might be going on here?

A weird update:
Everytime I'd see this error, I'd just ctrl + c. Today I forgot to do that, and everyone got scraped properly. Also, after 2 iterations (I have it on an infinite loop), the error stopped appearing.
Could this be because I added the cookie-jar option? I added it once this error started appearing, so this is basically the first time I'm using that option.
I'll keep this thread open, because we still don't know what seems to be the issue here.

Were stories scraped? Because that's the main reason to login. Without logging in, everything else will still scrape

The stories were scraped.
Non-story media on the private accounts won't be scraped without logging in either I think, so that would be another reason to log in.

I have been getting this error starting yesterday (after a long time of successful scrapes).  I noticed this comment in #383:

It had something to do with the ip. Either it was blacklisted or something as when i changed the vm's ip it started working again.

and combining that with the fact that I started getting this error right after I had been trying to log in and scrape around once a minute, rather than the usual of no less than a couple hours between scrapes, I figured that might have been what happened.
I tried switching the IP by just running my script with torify, but I still got the same error.  Since I wasn't sure if I had previously logged in with the same cookiejar file on that IP, I refreshed the Tor route (service tor restart) and tried again.  That time it worked: I got the verification prompt, and was logged in after that.
So it seems that (at least sometimes) this error means the given IP/session has been banned.
I might try again at some point to see if the ban expires eventually.


[FR] -Display "Producer" or "Relay"

https://github.com/input-output-hk/cardano-node/issues/1694
Describe the feature you'd like
Add an indicator as to whether the node is a producer or relay.

Closed because of duplication: we already have a similar task: https://github.com/input-output-hk/cardano-rt-view/issues/17


Won't click using Xpath

Expected behavior
Click on a given div by XPath

Actual behavior
 ✘ Error: CustomSelector with query not found
Steps to reproduce
Have a site with

...

await click($("//*[@id=`js-lk-modal-product-rating`]/div/div/div/div/form/div[3]/div/div/div[3]")) // copied form browser XPath
await click($("//div[data-rate=`2`]")) // and changed all types of quotes

Versions
Version: 1.0.16 (Chromium: 85.0.4168.0)

Would be very nice if there would be a chance to click on a browser and see what happens in Taiko.
like element //div[class="ClassName"] being clicked.
In order for me to reverse behavior from the UI into the code.
I didn't find anything by searching.

await click($("//div[data-rate=2]")) // and changed all types of quotes

@kaldown Your Xpath seems wrong. Try click($("//div[@data-rate='2']"))

Would be very nice if there would be a chance to click on a browser and see what happens in Taiko.
like element //div[class="ClassName"] being clicked.

Unfortunately there's no easy way to do this.
Closing this issue for now as @BugDiver pointed it looks like an XPath issue. Taiko just uses the browser's XPATH implementation.
If you see an Xpath not working it is best to check the XPATH on the browser console before using it in Taiko.


fix(createViewer): Do not use colorMapSelector with RGB images

:tada: This PR is included in version 5.4.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


feat: AMISelector uses most recently built AMI

Fixes #
Description
In this PR AMISelector will create a launch template with the most recently built AMI found matching using tags and/or name. If aws-ids key is used it will create a launch template for each matching AMI.
Relates to https://github.com/aws/karpenter/pull/2326#discussion_r950551896
How was this change tested?

Unit tests
Running make apply in cluster and ensuring that the most recently built AMI is chosen when using tags and/or name

Does this change impact docs?

[x] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[ ] No

Release Note

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

 Let me know if you can fix those, and I can help review this and get it merged in. In the meantime, I'll pull your code down and start making changes on my own branch.

Hey @andrewhibbert, thanks for the contribution! I pulled your code down and made some changes here. Since we haven't heard from you in a bit, I'll close this out. Feel free to leave a review.


US 1583733 Add missing language IDs - 23

US 1583733 - Fix CATS P1 issues by adding missing language IDs to code blocks.

Used "console" for command-line tool syntax and examples (Svcutil.exe, ComSvcConfig.exe).
Used single-tick for examples of WCF Performance Counter naming syntax.
Used single-tick for regular expression examples.

Contributes to #2192

@v-thpra can you also make sure that the PR descriptions include the sentence "Contributes to #2192" (I'm adding to the past ones that are missing this)


Windows update not working after running this script

I am not able to enable windows update from services app. It stays in disable state only. How can I fix that?

Try with
Get-Service -Name wuauserv | Set-Service -StartupType Automatic
Get-Service -Name wuauserv | Start-Service

What dou you get?

Windows update should not be getting disabled after everything is said and done. Is this happening also on multiple machines?
Sent from my iPhone

On Jun 13, 2021, at 6:19 PM, Dmitry Nefedov @.***> wrote:

Try with
Get-Service -Name wuauserv | Set-Service -StartupType Automatic
Get-Service -Name wuauserv | Start-Service
What dou you get?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Try with this as admin in PowerShell
Get-Service -Name wuauserv | Set-Service -StartupType Automatic
Get-Service -Name wuauserv | Start-Service

What dou you get?

Start-Service : Service 'Windows Update (wuauserv)' cannot be started due to the following error: Cannot start service  wuauserv on computer '.'.                                                                                At line:1 char:30                                                                                + Get-Service -Name wuauserv | Start-Service                                                                            +                              ~~~~~~~~~~~~~                                                                                + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],      ServiceCommandException                                                                                + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

@yogikatba attach a screenshot of the registry page Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wuauserv

@yogikatba attach a screenshot of the registry page Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wuauserv

@farag2

@yogikatba, sorry for the late reply. I was in flames those days. Can you set "3" to the "Start"? And try that command again. If you still have an error, please write to the Telegram group https://t.me/sophia_chat, and ping me: @sanctuary_d, cus I think I'll have to connect to you remotely via TeamViewer.


\n in kick message.

Please allow us to use \n in the kick message so we can add text to another line in a clean way <3

Agreed, a good feature that I haven't quite figured out yet. The text API keeps changing on me.

Second line is a must!
#FirstWorldProblems lol :)

Any updates on this @egg82 ?

Nothing yet, but I'll have an update at some point.

RIP.

No RIP yet - education is simply placing massive demands on our time and these are still slated to be done when possible 😄

Fixed in c33419e

Fixed in c33419e


Public room - Flickering skeleton animation displayed when navigating to  public room "#exfy-roadmap"

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: v1.4.48-0
Reproducible in staging?: yes
Reproducible in production?: yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by:
Slack conversation:
Action Performed:

Log in
Go to #exfy-roadmap public room: https://new.expensify.com/r/868417741568295

Note: If you fail to reproduce (room loads fine), log out and log in, then try again to open the room
Note2: If I open the link to the room without being logged in (anonymous account), it opens fine
Expected Result:
The room should have loaded just fine
Actual Result:
The skeleton animation stays flickering for ever and we start spamming the server with OpenReport requests
Workaround:
N/A
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/87341702/f5290b99-f996-42a5-8f04-abf499237610

View all open jobs on GitHub

This may be related to https://github.com/Expensify/App/issues/37661 or not (similar symptom at least)

I'm applying the retest-weekly label so we can check to see if this issue persists now that the linked issue is closed.

This should be retested tomorrow.

Bumped to have it retested. If it still exists, I will need to figure out which project it fits into.

Coming from the Road Map doc, it looks like it fits into #vip-vsb

This still happens for me in staging. I logged out and logged in before testing to make sure I didn't have bad data in the client:
https://github.com/Expensify/App/assets/87341702/ff17f435-3b32-4cdc-b898-0edcc536e06a
The client keeps sending OpenReport requests in a loop even after leaving the room that didn't load.

Commenting because I have the same issue

This appears to be a duplicate of https://github.com/Expensify/App/issues/35620, and we're addressing it in #fireroom-2024-03-19-spam-openreport. Going to close this


[Bug]: The residual function failed at the first call.

PyBaMM Version
23.9
Python Version
3.9.13
Describe the bug
2024-02-01 18:03:47.086 - [WARNING] simulation.solve(620): Ignoring t_eval as solution times are specified by the experiment
2024-02-01 18:03:47.087 - [NOTICE] callbacks.on_cycle_start(172): Cycle 1/1 (51.500 us elapsed) --------------------
2024-02-01 18:03:47.088 - [NOTICE] callbacks.on_step_start(180): Cycle 1/1, step 1/1: Rest for 30 min
The residual function failed at the first call.
The residual function failed at the first call.
The residual function failed at the first call.
The residual function failed at the first call.
The residual function failed at the first call.
The residual function failed at the first call.
2024-02-01 18:03:47.527 - [ERROR] callbacks.on_experiment_error(224): Simulation error: Maximum number of decreased steps occurred at t=0.0 (final SolverError: 'Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:1401:
Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:330:
.../casadi/interfaces/sundials/idas_interface.cpp:596: IDACalcIC returned "IDA_FIRST_RES_FAIL". Consult IDAS documentation.'). For a full solution try reducing dt_max (currently, dt_max=600) and/or reducing the size of the time steps or period of the experiment. Set return_solution_if_failed_early=True to return the solution object up to the point where failure occured.
Steps to Reproduce
import pybamm
import numpy as np
import matplotlib.pyplot as plt
model = pybamm.lithium_ion.DFN(
{
"thermal": "x-full",
"surface form": "differential",
"SEI":"solvent-diffusion limited",
"SEI porosity change": "true",
"lithium plating": "partially reversible",
"lithium plating porosity change": "true",
"particle mechanics":("swelling and cracking", "swelling only"),
"SEI on cracks": "true",
"loss of active material": "stress-driven",
"calculate discharge energy": "true",
}
)
var_pts = {"x_n": 5, "x_s": 5, "x_p": 5, "r_n": 30, "r_p": 30,}
def get_basic_parameters():
parameter_values = pybamm.ParameterValues("OKane2022")
parameters = parameter_values
return parameters
def get_thermal_parameters():
thermal_parameters = {
"Negative current collector surface heat transfer coefficient [W.m-2.K-1]": 0,
"Positive current collector surface heat transfer coefficient [W.m-2.K-1]": 0,
"Negative tab width [m]": 0,
"Positive tab width [m]": 0,
"Negative tab heat transfer coefficient [W.m-2.K-1]": 0,
"Positive tab heat transfer coefficient [W.m-2.K-1]": 0,
"Edge heat transfer coefficient [W.m-2.K-1]": 0,
}
return thermal_parameters
def update_parameters():
basic_parameters = get_basic_parameters()
thermal_parameters = get_thermal_parameters()
basic_parameters.update(thermal_parameters, check_already_exists = False)
return basic_parameters
parameter_values = update_parameters()
def negative_ocp(sto):
u_eq = (
0.6379
+ 0.5416 * np.exp(-305.5309 * sto)
+ 0.044 * np.tanh(-(sto - 0.1958) / 0.1088)
- 0.1978 * np.tanh((sto - 1.0571) / 0.0854)
- 0.6875 * np.tanh((sto + 0.0117) / 0.0529)
- 0.0175 * np.tanh((sto-0.5692) / 0.0875)
)
return u_eq
def negative_OCP_entropic_change(sto, c_s_max):
dEdT = (
2.8957
- 5.2972 * pybamm.cos(pybamm.arctan(1/(sto2) - 1))
+ 4.0370 * pybamm.cos(2 * pybamm.arctan(1/(sto2) - 1))
- 2.5304 * pybamm.cos(3 * pybamm.arctan(1/(sto2) - 1))
+ 1.2717 * pybamm.cos(4 * pybamm.arctan(1/(sto2) - 1))
- 0.4884 * pybamm.cos(5 * pybamm.arctan(1/(sto2) - 1))
+ 0.1301 * pybamm.cos(6 * pybamm.arctan(1/(sto2) - 1))
- 0.1086 * pybamm.cos(7 * pybamm.arctan(1/(sto**2) - 1))
)
return dEdT
def negative_exchange_current_density(c_e, c_s_surf, c_s_max, T):
cl_0 = pybamm.Parameter("Initial concentration in electrolyte [mol.m-3]")
k = 6
k_ref = k * 2 / (c_s_max * cl_0 ** 0.5)
E_ac = 20e3
arrhenius = np.exp(E_ac / pybamm.constants.R * (1 / 298.15 - 1 / T))
i0 = k_ref * arrhenius * c_e ** 0.5 * c_s_surf ** 0.5 * (c_s_max - c_s_surf) ** 0.5
return i0
def negative_diffusivity(sto, T):
D_ref = 6e-14
E_Ds = 3.5e4
arrhenius = np.exp(E_Ds / pybamm.constants.R * (1 / 298.15 - 1 / T))
Ds = D_ref * arrhenius
return Ds
def positive_ocp(sto):
u_eq = (
3.4323
- 0.8428 * np.exp(-80.2493 * (1 - sto) ** 1.3198)
- 3.2474e-6 * np.exp(20.2645 * (1 - sto) ** 3.8003)
+ 3.2482e-6 * np.exp(20.2646 * (1 - sto) ** 3.7995)
)
return u_eq
def positive_OCP_entropic_change(sto, c_s_max):
dEdT = (
-0.3538 * sto ** 8
+ 1.3902 * sto ** 7
- 2.2585 * sto ** 6
+ 1.9635 * sto ** 5
- 0.9872 * sto ** 4
+ 0.2886 * sto ** 3
- 0.0463 * sto ** 2
+ 0.0032 * sto
- 1.9186e-5
)
return dEdT
def positive_exchange_current_density(c_e, c_s_surf, c_s_max, T):
cl_0 = pybamm.Parameter("Initial concentration in electrolyte [mol.m-3]")
k = 2
k_ref = k * 2 / (c_s_max * cl_0 ** 0.5)
E_ac = 53e3
arrhenius = np.exp(E_ac / pybamm.constants.R * (1 / 298.15 - 1 / T))
i0 = k_ref * arrhenius * c_e ** 0.5 * c_s_surf ** 0.5 * (c_s_max - c_s_surf) ** 0.5
return i0
def positive_diffusivity(sto, T):
D_ref = 4e-13
E_Ds = 2e4
arrhenius = np.exp(E_Ds / pybamm.constants.R * (1 / 298.15 - 1 / T))
Ds = D_ref * arrhenius
return Ds
def electrolyte_diffusivity(c_e, T):
D_c_e = (
5.2969e-20 * c_e ** 3
- 2.7984e-16 * c_e ** 2
+ 2.8086e-13 * c_e
+ 3.4857e-10
)
E_D_c_e = 16500
arrhenius = np.exp(E_D_c_e / pybamm.constants.R * (1 / 298.15 - 1 / T))
Dl = D_c_e * arrhenius
return Dl
def electrolyte_conductivity(c_e, T):
sigma_e = (
3.2948e-10 * c_e ** 3
- 1.6463e-6 * c_e ** 2
+ 2.2550e-3 * c_e
+ 2.5977e-2
)
E_sigma_e = 4000
arrhenius = np.exp(E_sigma_e / pybamm.constants.R * (1 / 298.15 - 1 / T))
sigmal = sigma_e * arrhenius
return sigmal
def cation_transference_number(c_e, T):
t_plus_ref = (
5.7052e-17 * c_e ** 5
- 4.7979e-13 * c_e ** 4
+ 1.2203e-9 * c_e ** 3
- 1.1791e-6 * c_e ** 2
+ 2.8155e-4 * c_e
+ 3.5182e-1
)
E_t_plus = 16500
arrhenius = np.exp(E_t_plus / pybamm.constants.R * (1 / 298.15 - 1 / T))
t_plus =  t_plus_ref * arrhenius
return t_plus
def thermodynamic_factor(c_e, T):
dlnfdlnc_ref = (
-4.4566e-10 * c_e ** 3
+ 1.5644e-6 * c_e ** 2
- 7.4218e-5 * c_e
- 3.1908e-2
)
E_dlnfdlnc = 16500
arrhenius = np.exp(E_dlnfdlnc / pybamm.constants.R * (1 / 298.15 - 1 / T))
dlnfdlnc =  dlnfdlnc_ref * arrhenius
return dlnfdlnc
parameter_values["Negative electrode OCP [V]"] = negative_ocp
parameter_values["Negative electrode OCP entropic change [V.K-1]"] = negative_OCP_entropic_change
parameter_values["Negative electrode exchange-current density [A.m-2]"] = negative_exchange_current_density
parameter_values["Negative electrode diffusivity [m2.s-1]"] = negative_diffusivity
parameter_values["Positive electrode OCP [V]"] = positive_ocp
parameter_values["Positive electrode OCP entropic change [V.K-1]"] = positive_OCP_entropic_change
parameter_values["Positive electrode exchange-current density [A.m-2]"] = positive_exchange_current_density
parameter_values["Positive electrode diffusivity [m2.s-1]"] = positive_diffusivity
parameter_values["Electrolyte conductivity [S.m-1]"] = electrolyte_conductivity
parameter_values["Electrolyte diffusivity [m2.s-1]"] = electrolyte_diffusivity
parameter_values["Cation transference number"] = cation_transference_number
parameter_values["Thermodynamic factor"] = thermodynamic_factor
parameter_values.set_initial_stoichiometries(0)
pybamm.set_logging_level("NOTICE")
cycle_number = 10
experiment = pybamm.Experiment(["Rest for 30 min"])
safe_solver = pybamm.CasadiSolver(return_solution_if_failed_early=True)
simulation = pybamm.Simulation(model, parameter_values = parameter_values, experiment = experiment, var_pts = var_pts)
solution = simulation.solve(safe_solver)
Relevant log output
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[d:\yhl\pyproject\pybamm-param-develop\.venv\lib\site-packages\pybamm\solvers\casadi_solver.py](file:///D:/yhl/pyproject/pybamm-param-develop/.venv/lib/site-packages/pybamm/solvers/casadi_solver.py) in ?(self, model, y0, inputs_dict, inputs, t_eval, use_grid, extract_sensitivities_in_solution)
    685                 # If it doesn't work raise error
    686                 pybamm.logger.debug(f"Casadi integrator failed with error {error}")
--> 687                 raise pybamm.SolverError(error.args[0])
    688             pybamm.logger.debug("Finished casadi integrator")

[d:\yhl\pyproject\pybamm-param-develop\.venv\lib\site-packages\casadi\casadi.py](file:///D:/yhl/pyproject/pybamm-param-develop/.venv/lib/site-packages/casadi/casadi.py) in ?(self, *args, **kwargs)
  23388       else:
  23389     # Named inputs -> return dictionary
> 23390         return self.call(kwargs)

[d:\yhl\pyproject\pybamm-param-develop\.venv\lib\site-packages\casadi\casadi.py](file:///D:/yhl/pyproject/pybamm-param-develop/.venv/lib/site-packages/casadi/casadi.py) in ?(self, *args)
  20037
  20038         """
> 20039         return _casadi.Function_call(self, *args)

RuntimeError: Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:1401:
Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:330:
.../casadi/interfaces/sundials/idas_interface.cpp:596: IDACalcIC returned "IDA_FIRST_RES_FAIL". Consult IDAS documentation.

During handling of the above exception, another exception occurred:

SolverError                               Traceback (most recent call last)
...
    [292](file:///D:/yhl/pyproject/pybamm-param-develop/.venv/lib/site-packages/pybamm/solvers/casadi_solver.py:292)     break

SolverError: Maximum number of decreased steps occurred at t=0.0 (final SolverError: 'Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:1401:
Error in Function::call for 'F' [IdasInterface] at .../casadi/core/function.cpp:330:
.../casadi/interfaces/sundials/idas_interface.cpp:596: IDACalcIC returned "IDA_FIRST_RES_FAIL". Consult IDAS documentation.'). For a full solution try reducing dt_max (currently, dt_max=600) and/or reducing the size of the time steps or period of the experiment. Set `return_solution_if_failed_early=True` to return the solution object up to the point where failure occured.

Hi @flag2fish Solver errors happen when the simulations are not converging. It could be that there is a problem with your custom OCP functions, it could be the var_pts grid you are using, or any number of issues. For instance, on MacOS the SEI with cracks is sensitive to the grid.
It could be difficult to debug your particular issue since you are setting the grid, using a custom parameter set, using thermal models, and using degradation models all at once. You will probably need to play around with the settings a bit to narrow down what is happening. For instance, if you use your settings with an existing parameter set, do you see the same issues?

Hi @flag2fish Solver errors happen when the simulations are not converging. It could be that there is a problem with your custom OCP functions, it could be the var_pts grid you are using, or any number of issues. For instance, on MacOS the SEI with cracks is sensitive to the grid.
It could be difficult to debug your particular issue since you are setting the grid, using a custom parameter set, using thermal models, and using degradation models all at once. You will probably need to play around with the settings a bit to narrow down what is happening. For instance, if you use your settings with an existing parameter set, do you see the same issues?

thank you for your reply. I try to add position ocp function only.
positive ocp function:
def positive_ocp(sto):
u_eq = (
3.4323
- 0.8428 * np.exp(-80.2493 * (1 - sto) ** 1.3198)
- 3.2474e-6 * np.exp(20.2645 * (1 - sto) ** 3.8003)
+ 3.2482e-6 * np.exp(20.2646 * (1 - sto) ** 3.7995)
)
return u_eq
and I need to change two parameter:
parameter_values["Upper voltage cut-off [V]"] = 3.65
parameter_values["Open-circuit voltage at 100% SOC [V]"] = 3.65
and then it worked.


Add multiset type and basic casting

"Multiset" is one word, so the "Set" should not be capitalized

changed casing to treat multiset as one word

@yangyangyyy are you still working on this? If so, could you rebase and then I can take a look

Chris yes I'll rebase .
On Wed, May 4, 2016 at 4:16 PM, Christopher Berner<EMAIL_ADDRESS>
wrote:

@yangyangyyy https://github.com/yangyangyyy are you still working on
this? If so, could you rebase and then I can take a look
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/prestodb/presto/pull/4102#issuecomment-217035587

Closing this due to lack of response from author. Please rebase and open another PR you would like this merged.


Assign SNPs on chromosome 0

Some newer Family Tree DNA famfinder files seem to have SNPs assigned to chromosome 0. Similar to SNPs._assign_par_snps, use the RefSNP API to assign each of these SNPs to a chromosome and update the SNP's position to be consistent with the assembly.

I have noticed that for AncestryDNA files, the _assign_par_snps step can take a substantial amount of time (> 1 minute), because of repeated API requests. I have currently bypassed this issue by setting assign_par_snps=False. Is there a way we can speed this up with the VCF_endpoint like you suggest in #19.

Yeah, I agree that having a resource like discussed in #19 to lookup SNPs would help here as well and would reduce repeated API calls. I'll start looking into this as part of #42.


describe the API in Open API format

as was done with the BibleGet endpoint, it would be useful, other than the description provided in the README.md in the repo, to describe the API in Open API / Swagger / Postman format. WIP!

What happened? The integration with SwaggerHub was enabled, but SwaggerHub created an empty PR?

SwaggerHub was instructed to create openapi.json if it doesn't exist. However seeing that SwaggerHub issued an empty PR, let's go ahead and create an empty openapi.json just to exclude this is the problem?
f73ba1984ff4be8f0e954234405c07e1622cd920


Parallelism

Add multithreading and multiprocessing to Base Optimizer

After testing, I can conclude that there is no significant improvements when we used parallelizing techniques in PSO, because it was used only for population creation. Also, possible to rewrite numpy vector operations using parallelizing, but I think that wont give any results.
The best way to decrease computational time is using numba and numexpr.
I left this code for future improvements


feat: Add ability to use Fargate Ephemeral Storage

Description
Add the ability to use Fargate Ephemeral Storage
Motivation and Context
With multiple large plans being ran, it's possible to run the container out of disk space.  This enables the use of Fargate's ephemeral storage option to reduce that chance.
Breaking Changes
None
How Has This Been Tested?

[ ] I have tested and validated these changes using one or more of the provided examples/* projects
Been running it successfully for a couple months.

I missed that this had been reviewed for about a month, sorry for the delay.  Requested changes have been made. 😄

Thanks, @raizyr !
v3.5.0 has been just released.


Bashisms in scripts

In the liveness script, which is quite properly a POSIX shell one, there are a number of bashisms. I'll see if I can make a PR.
@krig This is fantastic! Does it work? Do you have good logging? If so, I am going to use it for my new project. Thanks man!

Hi! Oh, I haven't updated this project in ages so who knows if it is still viable :D But as far as I know it is still being used in production in at least one place.
I'm sure it's full of bashisms, any fixes you have are more than welcome!

Hey Kristoffer, if the project is still somewhere in production then it must be viable for me too! Many thanks for creating this! I think you saved my life, not to have to deal with the huge thing that Pacemaker has become.
As for bashisms, yes, there will be something soonish, but my Linux computers are sort of out of order (a Thinkpad laptop with a flaky battery) and I find it difficult to deal with it on my MacBook Air.
Cheers,
Dejan
P.S.  How is life in general, let’s talk, I can contact you on WhatsApp if that’s OK with you!

On 13 Sep 2023, at 11:20, Kristoffer Grönlund @.***> wrote:
Hi! Oh, I haven't updated this project in ages so who knows if it is still viable :D But as far as I know it is still being used in production in at least one place.
I'm sure it's full of bashisms, any fixes you have are more than welcome!
—
Reply to this email directly, view it on GitHub https://github.com/krig/koi/issues/14#issuecomment-1717262016, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAE323SEKLQRVLZGEZ73CCTX2F3EDANCNFSM6AAAAAA4THQUGA.
You are receiving this because you authored the thread.

Sounds good, no hurries or expectations from my side! Happy to take patches when or if they appear :)

P.S.  How is life in general, let’s talk, I can contact you on WhatsApp if that’s OK with you!

Please do! Life is hectic, our baby just turned one, but it's great all the same :)


'Get all' could sometimes result in a crash

Bug description
Sometimes when I click on 'Get all' it could result in a crash but not always. So for example when it crashes and restore it, it will just work normal and not crash after it when I click on 'Get all'. (Which is why I can't yet provide steps to reproduce it and since it isn't generating a crash report I can't provide it).
I would like if you maybe could look inside the code and check what could get a crash like this to happen. (Maybe it is usefull to note that I have multipele sources enabled, used the post filter and restore my previous tabs always when I open grabber)
Expected behavior
Not crash and just give the sources which have images by the downloads tab
System information
Windows 10 with the latest nightly version

Did you add/remove sources before the crash?
Or change the selected ones?
That's pretty much the only thing that might possible pause a problem from what I see in the code, but while fiddling around I couldn't reproduce any crash. 🤔

it isn't generating a crash report I can't provide it

Only "official" releases generate crash reports since I can't analyze nightly reports because I don't have their symbol files.

No I didn't add or remove sources before the crash or change the selected ones. But also got sometimes a crash just by clicking on 'Download selected'. So I don't know if they are seperate chrases or if certain action behind each other will in the end result in a crash at the end.

@Bionus For me, its behavior is a lot like the memory overflow issues, on both nightly and release versions. No crash reports are produced on release version either, it just fades out and process killed. It always happens when I click 'Get All' for the first time in a session after 2-3 hours.


rectangular selections?

Hi folks, is it possible to select rectangular (image) selections and have them exported to markdown (links to graphics files for example)?

no folks, just me :)
I'd like to have such a feature as well, however such a feature is dependent on improvements to pdfannots enabling it. You can make a feature request there!
Best I could do is implement the annotation code for images, which unfortunately only works with Obsidian as output.

Thanks for the quick response.   I will explore the annotation code for images idea and file a feature request with pdfannots.
Sincerely,
Dr. Ronald D. Haynes
Professor, Department of Mathematics and Statistics
Chair, MSc and Phd Scientific Computing Programs
Memorial University of Newfoundland
We acknowledge that the lands on which Memorial University’s campuses are situated are in the traditional territories of diverse Indigenous groups, and we acknowledge with respect the diverse histories and cultures of the Beothuk, Mi’kmaq, Innu, and Inuit of this province.
On Jan 4, 2022, 9:48 AM -0330, pseudometa @.***>, wrote:

no folks, just me :)
I'd like to have such a feature as well, however such a feature is dependent on improvements to pdfannots enabling it. You can make a feature request there!
Best I could do is implement the annotation code for images, which unfortunately only works with Obsidian as output.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID: @.***>


Improved NSFW Functionality

Describe your suggested feature
Currently, the NSFW filter understands Content Rating: Erotica, and Sources marked NSFW, but this functionality could be improved by allowing simple SFW/NSFW tags to be added in the library entry, overwriting the default value. Additionally, a NSFW Badge would be useful for library views
Other details
It may also be useful to add an option to set what triggers an entry to be marked NSFW (i.e. what tags count, and if 18+ sources should be automatically selected)
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] If this is an request for a feature in the original Tachiyomi, I am opening because multiple versions of Yōkai have been released without this feature.
[X] If this is an issue with an extension, or a request for an extension, I should be contacting the extensions repository's maintainer/support for help.
[X] I have updated the app to version <IP_ADDRESS>.
[X] I have checked through the app settings for my feature.
[X] I will fill out all of the requested information in this form.

See the following picture for the tags that are counted as NSFW, just like SY. You can already edit manga entries & add/edit tags to said entry, just add one of the NSFW tags & it'll filter it out.

Howdy, so I have bunch of Nsfw content that doesn't detected as nsfw content, really hard to add every single tags for hundreds title like that 💀
For suggestion maybe add some feature that if the source/extension is in 18+ category, that means the content is automatically detected as nsfw content. Or add feature to add tags for multiple selected content at once.
Thanks!! I hope there's improvement for future


[CDK - Overlay] Cannot read properties of null (reading 'parentElement')

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
For some reason when showing a MatTooltip attaching the host is returning an error Cannot read properties of null (reading 'parentElement') since the _host is null.
tooltip.show()
Suggestion:

Add a validation for _host like should prevent the issue (method attach  on class OverlayRef:

if (this._host && !this._host.parentElement && this._previousHostParent) {             this._previousHostParent.appendChild(this._host);         }

Reproduction
StackBlitz link: -
Steps to reproduce: Wasn't able to reproduce
Expected Behavior
Should not return an error.
Actual Behavior
Returns an error (Cannot read properties of null (reading 'parentElement')) when this._host is null.
Environment

Angular: ^13.3.9
CDK/Material: ^13.3.9
Browser(s): Chrome 121 on Linux
Operating System (e.g. Windows, macOS, Ubuntu): Ubuntu

We can't do much without a reproduction. Also you're using an old version of the CDK/Material and the issue might not be present in the newer ones.


storage: Alternative representation of the genesis file.

The genesis file is stored as a value in CometBFTs kv store. As the file can be large it can hit the internal file size limitations of databases.
We need to figure out an alternative representation of this file.
Existing PRs targetting this :

#1017
Historically, this issue was mentioned in the original Tendermint repository under:

tendermint/tendermint#8416
tendermint/tendermint#6563

A potentially related problem is that at the moment the Genesis file is retrievable via RPC. Even though it is retrieved in chunks, it is still loaded entirely into memory before it is chunked up. We could improve on this if we take it into consideration.
DoD

[ ] Genesis file is not stored as one entry in the state store
[ ] Tests written to confirm no errors are generated for large genesis files
[ ] (stretch) Improve RPC retrieval of Genesis file.

Not sure if this would be also part of this issue, but I believe we need to better understand the use cases for people retrieving the genesis file via RPC, maybe we can provide another way to query parts of it only or something that returns like a hash that people can compare to another hash from a file retrieved from a different source (e.g. Github)

It's sad that the /genesis_chunked endpoint has to exist in the form it is. Because it's JSON-RPC, the protocol cannot do the usual HTTP thing: use the Range header on a genesis GET endpoint. That can also enable off-the-shelf caching/CDN solutions that abundantly exist for HTTP web services.
Or maybe, we can serve the genesis endpoint over plain HTTP and deprecate the JSON-RPC one? It could even be a start of converting the entire RPC to plain REST, which you can route and load-balance with off-the-shelf cloud components when things like ADR 102 are realized.
What value does the JSON-RPC framing deliver in our service?

Alternatives to Handle the Genesis File
Given the discussion on #1286, here are some thoughts on how to change how we handle the genesis file.
In both cases, the GenesisDoc isn't in memory anymore; the GC will collect its only instance after exiting the NewNode function.
Note: because we would be fetching the genesis file from disk, should we implement a sanity check to ensure it hasn't been tampered with after the Node was initialized? For example, we could verify the file's hash against the one stored in the state DB.
Option 1: Store GenesisDocProvider
Store the GenesisDocProvider in the Node during initialization, rather than the genesis file's contents as we do now. As mentioned in #1286, the primary use of the genesis file post-initialization is for serving RPC APIs. Any other, less frequent, use can rely on the GenesisDocProvider to fetch the file from disk.
Cons: GenesisDocProvider not only retrieves the genesis file from disk but also computes its hash. Re-hashing the genesis file on every access introduces a performance overhead, which could degrade the responsiveness of the RPC APIs. However, this isn't a downside if we need the hash to do a sanity check as mentioned above.
Option 2: Use genesis file's path on disk
Since the Node always retains the genesis file's disk path, we could fetch the genesis file when needed leveraging the Config.
Cons (?): this approach bypasses any additional operations performed by a custom GenesisDocProvider that users may define to process the GenesisDoc after it is read from the file.

Chunking
To serve the genesis_chunked endpoint, we store a slice of 16MB chunks in memory. This results in two copies of the GenesisDoc being held in memory: one is the original GenesisDoc, and the other is the reassembled version from the split chunks.
We could still use a simple cache to keep chunks in memory for a short time to serve the RPC requests.
Option 1: Store the chunks in the file system
As proposed in #1289, we could store the chunks on disk and fetch the appropriate chunk upon an RPC request.
Cons: This still results in two copies of the genesis file, but we shift the burden of the second copy from memory to disk.
Option 2: Store the chunks' offsets
As @jmalicevic points out here, we could store only the file offsets for each chunk, eliminating the need for additional storage.
Cons: Fetching a chunk would require an extra seek operation to move to the correct position in the file, slightly impacting performance. However, the potential storage savings are significant with genesis files that can be several GBs in size.

Overall I think, we might have to keep it in memory for serving RPCs but do we have to keep two copies (chunked, and not chunked)? If the solution with pointers is not great, could we then keep only chunks and combine them when the RPC requests wants the whole file - or is that too expensive?

One solution is to implement logic that leverages the Range header, you can have the chunks on disk and if the range matches you return that or if not you do a file seek for a hybrid approach if needed. Remember that if you implement a custom way to serve chunks, the client has to implement logic to request the chunks the way the server expects.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range

One solution is to implement logic that leverages the Range header, you can have the chunks on disk and if the range matches you return that or if not you do a file seek for a hybrid approach if needed. Remember that if you implement a custom way to serve chunks, the client has to implement logic to request the chunks the way the server expects.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range

We can consider that for the current JSON-RPC APIs, but since HTTP headers aren't available with gRPC endpoints, we'll need to rethink how to serve chunks when we implement the gRPC GetGenesis endpoint.

@jmalicevic @andynog
A thought: why don't we just use gRPC streaming? gRPC supports both client-side and server-side streaming. This allows us to break the genesis file into smaller chunks and send them piece by piece. We wouldn't need to load the entire genesis file into memory at once. Additionally, the client doesn't need to make N separate gRPC calls to retrieve N chunks; instead, the entire transmission is handled within a single gRPC call.
The endpoint's code will look something like this:
// GetGenesis streams the file content in chunks.
func (s *NetworkServiceServer) GetGenesis(/* params */) error {
    // e.g., 4MB chunk size
    chunkSize = 1024 * 1024 * 4

    file, err := os.Open(genesisFile)
    if err != nil {
        return err
    }
    defer file.Close()

    buffer := make([]byte, chunkSize)
    chunkNumber := 0
    for {
        // Read the file in chunks of size chunkSize
        n, err := file.Read(buffer)
        if err != nil {
            if errors.Is(err, io.EOF) {
	        break
            }
	    return err
        }

        // Stream the chunk back to the client 'stream' is a placeholder name.
        // The streaming server to use here will be automatically generated
        // by make-proto. It'll be something like NetworkService_GetGenesisServer
        resp := Response {
            Content:     buffer[:n],
	    ChunkNumber: int32(chunkNumber),
        }
        err = stream.Send(resp)
        if err != nil {
            return err
        }

        chunkNumber++
    }

    return nil
}

Pros:

We don't need to store chunks on disk or in memory.
We don't need to store the genesis in memory, just open the file and stream it.
No need for the client to keep track of multiple gRPC calls to reassemble the genesis file

What do you think?

We can implement a streaming gRPC for that sure but we’d still need to support the endpoint in JSONRPC until it's deprecated

Our JSON-RPC implementation supports WebSocket connections, allowing us to stream the genesis file in a single RPC call. We would need to refactor the /genesis and /genesis_chunked APIs, though. These changes would be breaking, as clients currently need to make a separate RPC request for each chunk they want to download.
Alternatively, I propose keeping the JSON-RPC APIs as they are but implementing the 'store the offsets into the file' strategy under the hood.
For the gRPC endpoint, I would implement streaming from the outset.

@alesforz I like your proposal. Maybe just name the endpoint with Streaming as part of the name. And then yes, we can keep the current APIs that we have to keep with an optimized implementation. If you think that the current /genesis api can be written in a streaming way without breaking anything, that would also be acceptable.

@jmalicevic As you know, the /genesis endpoint checks if the genesis chunks slice length is greater than one. If it is, it returns an error pointing to the /genesis_chunked endpoint; otherwise, it returns the genesis file.
Converting the /genesis endpoint into a WebSocket that streams the file would be ideal but would break existing clients. Alternatively, instead of returning "please use /genesis_chunked" when the genesis file is too large, we could return "please use /genesis_streamed" directing clients to a new JSON-RPC endpoint that streams the genesis file over a WebSocket.
The proposed changes would be:

/genesis: Returns the genesis file if it's small; otherwise, directs clients to the /genesis_streamed endpoint. If you're a client using this endpoint to retrieve your genesis file, it means the file is smaller than the chunking threshold; otherwise, you would be using /genesis_chunked. Therefore, this proposal wouldn't change the behavior of the /genesis endpoint for you, and you could continue using it without any disruption.
/genesis_chunked: Continues using the "one chunk at a time" method but refactored internally to avoid storing data in memory.
/genesis_streamed: A new endpoint that streams the genesis file over a WebSocket.

This approach allows existing clients to continue using /genesis and /genesis_chunked as they currently do, while encouraging new clients to adopt the /genesis_streamed endpoint. Existing clients might also transition to the new endpoint over time, as it offers a more efficient alternative to the less efficient /genesis_chunked.
One question is whether it's worthwhile to do this work if we plan to deprecate the JSON-RPC endpoints in the future. Maybe we should focus on improving the underlying implementation of /genesis_chunked and leave it at that. But I do think streaming is a QoS improvement for JSON-RPC APIs users.

Alternatives to Handle the Genesis File
Given the discussion on #1286, here are some thoughts on how to change how we handle the genesis file.
In both cases, the GenesisDoc isn't in memory anymore; the GC will collect its only instance after exiting the NewNode function. Note: because we would be fetching the genesis file from disk, should we implement a sanity check to ensure it hasn't been tampered with after the Node was initialized? For example, we could verify the file's hash against the one stored in the state DB.
Option 1: Store GenesisDocProvider
Store the GenesisDocProvider in the Node during initialization, rather than the genesis file's contents as we do now. As mentioned in #1286, the primary use of the genesis file post-initialization is for serving RPC APIs. Any other, less frequent, use can rely on the GenesisDocProvider to fetch the file from disk.
Cons: GenesisDocProvider not only retrieves the genesis file from disk but also computes its hash. Re-hashing the genesis file on every access introduces a performance overhead, which could degrade the responsiveness of the RPC APIs. However, this isn't a downside if we need the hash to do a sanity check as mentioned above.
Option 2: Use genesis file's path on disk
Since the Node always retains the genesis file's disk path, we could fetch the genesis file when needed leveraging the Config.
Cons (?): this approach bypasses any additional operations performed by a custom GenesisDocProvider that users may define to process the GenesisDoc after it is read from the file.

@jmalicevic @andynog Any comments on this? Anything you would do differently? Have I forgotten viable options?

Hi @alesforz I think this is good, concerning file chunks I expressed some ideas here
For the hashing, we should not do it every time we access the file, I thought this was only needed when the server starts and we need to save the hash in the db if we don't have one

With the merging of #4349, we have completed all the work in this tracking issue 🥳

With the merging of #4349, we have completed all the work in this tracking issue 🥳


Add hooks for plugins in service form errors

In the case the plugins add fields to the service form and it errors in the backend, we allow the plugins to catch that

Reviewer: @jfurrow


[Bug] Jax v0.4.36 Breaks Cola Solve

It appears that the new release of JAX v0.4.36 breaks cola.linalg.solve with Cholesky solver. It results in Nan gradients.
  k_zz_inv_U = cola.linalg.solve(K, self.U.reshape(self.M, -1), alg=Cholesky())
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/cola/linalg/inverse/inv.py", line 39, in solve
    return inv(A, alg) @ b
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/plum/dispatcher.py", line 93, in new_fn
    return fn(*new_args)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/plum/function.py", line 444, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/cola/linalg/inverse/inv.py", line 98, in inv
    L = cholesky(A)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/plum/function.py", line 444, in __call__
    return _convert(method(*args, **kw_args), return_type)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/cola/linalg/decompositions/decompositions.py", line 154, in cholesky
    return Triangular(A.xnp.cholesky(A.to_dense()), lower=True)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/cola/ops/operator_base.py", line 77, in to_dense
    return self @ self.xnp.eye(self.shape[-1], self.shape[-1], dtype=self.dtype, device=self.device)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/cola/backends/jax_fns.py", line 154, in eye
    return jnp.eye(N=n, M=m, dtype=dtype)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/jax/_src/numpy/lax_numpy.py", line 6636, in eye
    output = _eye(N, M=M, k=k, dtype=dtype)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/jax/_src/numpy/lax_numpy.py", line 6659, in _eye
    return (i + offset == j).astype(dtype)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/jax/_src/numpy/array_methods.py", line 1049, in op
    return getattr(self.aval, f"_{name}")(self, *args)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/jax/_src/numpy/array_methods.py", line 573, in deferring_binary_op
    return binary_op(*args)
  File "/home/adam/anaconda3/envs/jax_torch_latest/lib/python3.12/site-packages/jax/_src/numpy/ufunc_api.py", line 179, in __call__
    return call(*args)
jax._src.source_info_util.JaxStackTraceBeforeTransformation: ValueError: Cannot lower jaxpr with verifier errors:
	type of return operand 0 ('tensor<50x50xi64>') doesn't match function result type ('tensor<50x50xi32>') in function @main
		at loc(unknown)
	see current operation: "func.return"(%1) : (tensor<50x50xi64>) -> ()
		at loc(unknown)
Define JAX_DUMP_IR_TO to dump the module.

The code that generated this is a very simple sparse GP using inducing points, which is well tested and works fine on versions on JAX 0.4.35 and below.
def gp_fit(self, t):
    kern = partial(matern52_kernel, length_scale=self.length_scale, variance=self.variance)
    k_zz = cola.ops.Dense(kern(self.Z, self.Z))
    k_xz = cola.ops.Dense(kern(t, self.Z))
    K = cola.PSD(k_zz + ((self.noise + self.jitter) * cola.ops.I_like(k_zz)))
    k_zz_inv_U = cola.linalg.solve(K, self.U.reshape(self.M, -1), alg=Cholesky())
    return k_xz @ k_zz_inv_U

@adam-hartshorne, thanks for bringing this discrepancy introcuded in JAX v0.4.36 to our attention. I'll do some analysis on the Cholesky function on JAX and report back.
On the meantime, could you share more code to replicate the exact error above?

I figured out what is causing this behaviour,
if cola.ops.I_like(k_zz))  is changed to jnp.eye(k_zz.shape[0], dtype=self.Z.dtype)) fixes the NaN problem. So it isn't actually the solve, it is something to do with I_like.


Add temporary fix to overview resampling

Codecov Report
Base: 84.87% // Head: 84.83% // Decreases project coverage by -0.03% :warning:

Coverage data is based on head (2d7c8be) compared to base (7d14461).
Patch coverage: 50.00% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #155      +/-   ##
===========================================
- Coverage    84.87%   84.83%   -0.04%
===========================================
  Files           50       50
  Lines         4171     4175       +4
===========================================
+ Hits          3540     3542       +2
- Misses         631      633       +2

Impacted Files
Coverage Δ

eogrow/utils/map.py
86.95% <50.00%> (-2.28%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


LexigramCount counts each element of alternatives

Alternatives might be a single unit so should have a value of 1(?)

Alt should function like Opt in that LexigramCount should map over alts and take the min and max interval. Will need more sophistication than that to handle things like Opt in the alts. Nested Interval probably won't make sense.


Drop on "schedule" from CI

It disables Actions after a while of inactivity

This appears to conflict now but otherwise happy to merge this in.

Yup, resolved


Some Landsat imports are improperly imported

L8 L8/192/030/LC81920302013215LGN00/

L8 L8/192/033/LC81920332013215LGN00/

dataFootprint is properly set while tileFootprint is not

As a test, @alkamin and I looked at where the scenes were that were squished, selected with this query:
SELECT id from scenes where ST_Contains(tile_footprint, data_footprint) = false AND datasource = '697a0b91-b7a8-446e-842c-97cda155554d';

Here are the locations of the scenes that came out of that query:

For comparison, here are the locations of the complement of those scenes (note the stripes):

We tried checking whether the bug was in geotrellis's projection with one of the bad scenes by:

get the lat/lon coordinates for the scenes tile footprint manually
project a multipolygon of those coordinates from EPSG:4326 to EPSG:3857 and back again
make sure it was the same polygon

That worked fine.
We think then that the bug might be in in our projection of Landsat 8 scenes before we post them to /api/scenes/ in the importer. We're going from 4326 --> 4326 there though, so that's also confusing.

Thinking on this more, I don't think this is a projection problem. While the bad tile footprints are incorrect, they are still composed of coordinates from the data footprint; when the min/max values of the bounds are calculated (https://github.com/azavea/raster-foundry/blob/develop/app-tasks/rf/src/rf/uploads/landsat8/create_footprint.py#L29) the script is coming away with the wrong coordinate, not a random value.

This is a total shot in the dark, but I'm wondering if the input to the min/max determinations that @alkamin mentioned is somehow typed improperly. For example if it was strings, you'd get weird ordering issues with coordinate pairs that crossed certain thresholds such as 9-10 because the 10 would sort alphabetically before the 9. I'm not totally convinced that's the problem, but given that we're seeing issues around 1, 10, and 100 degrees, it might be something similar.


Reasoning with the sql output

Hi,
I have been trying sqlcoder 34b model on my large database using pruning, but model cannot handle out of domain queries well as expected. It always generate sql no matter whether that column is present in the schema or not. I have two questions:

how to make model to handle out of domain queries via prompting instructions. as it tried appending in prompt.md but its not that accurate.
If I can get the reasoning as to how  generated a SQL query, that would also help to mitigate some of out of domain queries. How can we do it via prompting?

Can you please help with above?
Thanks
Aditya

Hi Aditya, could you give us a couple of examples of the full prompt that you're using? That'll help us understand the problem you're facing better.
Also, could you clarify what you mean by out of domain queries? Thanks!

Hi ,
I am using the same prompt as provided in the repo, just added instructions section after :
"""## Task
                    Generate a SQL query to answer the following question:""" + "\n"+\
                    user_query["query"] + \
                    """\n\n### Instructions
                    - If not sure with the query response, return I don't know."""+\
                    """\n\n ### Database Schema
                    This query will run on a database whose schema is represented in this string:\n"""+\
                    schema +"\n\n"+\
                    """### SQL
                      Given the database schema, here is the SQL query that answers """+user_query["query"] +"\n\n"+\
                    """```sql"""

I will rather call it in-domain out of domain queries like:

Which schools have science labs? : though there is no column in my table for science labs, it assumed a column e.g science stream present or not and provided the answer. it should either give reasoning or reply with it don't know.
how many schools are there in say jamnagar? : it miss such queries as well as it don't understand what jamnagar it so map it to state column or sometimes district columns, even after  adding "jamnagar" as part of column district_name.

Any help would be very useful.
Thanks
Aditya

Thanks! Could you please share your schema as well please? That will help us understand this better and recommend a better prompt

Yes.
following is indomain-out of domain query
query : Are there any mobile schools in district Jamnagar for nomadic communities?
schema: CREATE TABLE sch_master (district_name text, --Name of the district where the school is located    sch_loc_rural_urban text, --Type of school location (RURAL, URBAN, Possible Mistake)   pincode bigint, --Postal code of the school location   village_ward_name text, --Name of the village or village ward   school_name text, --Name of the school for the udise code   block_name text, --Name of the block where the school is located   longitude bigint, --Longitude coordinate of the school location, sch_code VARCHAR(25) -- Primary Key, Unique code for every school in database);   CREATE TABLE sch_profile (resi_sch_type text, --Type of residential school [Not Defined, Ashram (Govt.), Model School, Non-Ashram (Govt.), Jawahar Navodaya Vidyalaya, Other-Boarding, Private-Boarding, KGBV, Eklavya Model Residential School, Netaji Subhash Residential School)   mthr_tongue_pri text, --Mother tongue of primary students   distance_upr_to_govt_sch bigint, --Distance from upper primary school to government school   distance_pri_to_govt_sch bigint, --Distance from primary school to government school   spltrg_py_enrol_girls bigint, --Special training program enrollment for girls   distance_sec_to_govt_sch bigint, --Distance from secondary school to government school   num_upr_sch_in_cmplx bigint, --Number of upper primary schools in complex   distance_hsec_to_govt_sch bigint, --Distance from higher secondary school to government school   num_pri_sch_in_cmplx bigint, --Number of primary schools in complex   spltrg_py_enrol_boys bigint, --Special training program enrollment for boys   resi_sch_yn text, --Residential school type (Non-Residential, Completely Residential, Partially Residential)   num_hsec_sch_in_cmplx bigint, --Number of higher secondary schools in complex   num_sec_sch_in_cmplx bigint, --Number of secondary schools in complex   sch_hrs_stu_upr bigint, --School hours per student in upper primary school   sch_hrs_stu_hsec bigint, --School hours per student in higher secondary school   sch_hrs_teacher_upr bigint, --School hours per teacher in upper primary school, sch_code VARCHAR(25) -- Primary Key, Unique code for every school in database);   CREATE TABLE sch_facility (   bld_blk bigint, --Number of school buildings/blocks   sch_code text, --school code   clsrms_upr bigint, --Number of upper primary classrooms   clsrms_pri bigint, --Number of primary classrooms   clsrms_hsec bigint, --Number of higher secondary classrooms   cls_rm_for_instrc bigint, --Number of classrooms used for instruction   clsrms_pre_pri bigint, --Number of pre-primary classrooms   num_cls_rms_gd_cond_pucca bigint, --Number of pucca classrooms in good condition   cls_rm_need_min_repr_pucca bigint, --Number of pucca classrooms needing minor repairs   ict_instr_type text, --Type of instruction for ICT (9, Full Time, Not Available, Part Time)   cls_rm_under_cstrc bigint, --Number of classrooms under construction   clsrms_sec bigint, --Number of secondary classrooms   cls_rm_need_maj_repr bigint, --Number of classrooms needing major repairs   mobile_fun bigint, --Number of functional mobile devices, sch_code VARCHAR(25) -- Primary Key, Unique code for every school in database); -- sch_master.sch_code can be joined with sch_profile.sch_code   -- sch_master.sch_code can be joined with sch_facility.sch_code   -- sch_profile.sch_code can be joined with sch_facility.sch_code
SELECT COUNT(*) FROM sch_master JOIN sch_profile ON sch_master.sch_code = sch_code.udise_sch_code JOIN sch_facility ON sch_code.udise_sch_code = sch_facility.sch_code WHERE sch_master.district_name ilike '%JAMNAGAR%' AND sch_profile.resi_sch_type ilike '%Mobile%' AND sch_facility.mobile_fun > 0;
in sql query : resi_sch_type has no value mobile and mobile_fun is no. of phone functions, not mobile schools.
thanks
aditya

I see – you should use the defog/sqlcoder-7b-2 model for this. The 34B model will soon get these capabilities, but they have not been ported over just yet.
The new 7b model seems to handle this like you would expect! The image below is from our gradio app, which you can run here

Closing this for now, but please feel free to reopen if you still have issues!

Hi,
I have tested sqlcoder 7b-2 model on this query. Sometimes it gives the answer as above, sometime its hallucinates. I have also tried few in-domain queries with this new model which were working fine with 34b model. For those queries, it sometimes give  SELECT 'I do not know' AS answer;
Prompt I am using from here
Inference parameters:
```
"do_sample": False,
"max_new_tokens": 300,
"temperature": 0.1,
"repetition_penalty": 1.1,
"return_full_text":False,
"num_beams":4,
Can you please help on how to prevent this model from hallucinating.

Thanks
Aditya

Ah keep your temperature to 0 and repetition_penalty to 1.1. With temperature 0.1, you will always get some randomness in the results.
Would recommend using the run_inference function here for best results.

So i have removed temperature from my sagemaker inference endpoint as it ask for it be positive, not 0.
I can try running using the run_inference function that you have shared instead of SageMaker endpoint, any specific observation from you on using transformer based inference rather then SageMaker inference.
Also my metadata.sql with column descriptions,  is more than accepted input limit of 2048 tokens, so I am using pruning. Do you think by pruning as I am not giving the model more context of other columns to compare, it might be given random responses?
Thanks
Aditya

Try setting temperature to a tiny number (like 0.0001) – it will have roughly the same effect and remove randomness.
The Codellama context limit is 100k, so not sure if the input limit is a Sagemaker specific thing. But with temperature=0 (or a number close enough to 0), you will always get deterministic responses that you can then fix with better prompting!

@rishsriv By input limit I mean input token length which is 2048 tokens for sqlcoder,  so i need to apply pruning for large tables that go beyond token limit of 2048.
Regarding prompting you mean adding more instructions, right? Also is there a way to get explanations as how the sql generated, because this way i can enhance the instructions?
Thanks
Aditya


Event is appearing in two days for an all day event

I have an event with a startTime set to Wed May 18 2016 00:00:00 GMT+1000 (AUS Eastern Standard Time) and the endTime set to Wed May 18 2016 23:59:00 GMT+1000 (AUS Eastern Standard Time).
When the calendar loads, the all day event is showing on May 17 and 18. I have tried this with the allday flag set to true and false and I get the same result

@captainmorgan0 As I documented in README, you need to use UTC date for all day events.
If allDay is set to true, the startTime has to be as a UTC date which time is set to 0:00 AM, because in an allDay event, only the date is considered, the exact time or timezone doesn't matter.
For example, if an allDay event starting from 2014-05-09, then startTime is

var startTime = new Date(Date.UTC(2014, 4, 8));

Thanks. That works, and the event shows when viewing by month. However, I don't see the event in week or day view.

@captainmorgan0 You can't see the allDay events or even normal events? Is the view rendered ok? If so, could you check if the events are passed correctly to the selected date range?

In month view, I can see normal events and all day events. In week view and day view I can see normal events, but I can't see all day events.

In you documentation, you say the startTime has to be as a UTC date which time is set to 0:00 AM, However, If I do console.log(new Date(Date.UTC(2014, 4, 8)));, I get the output "Thu May 08 2014 10:00:00 GMT+1000 (AUS Eastern Standard Time)", the time is no 0:00 AM.  Does your example need to specify the time as well?

You don't need to specify the time part.
The local time Thu May 08 2014 10:00:00 GMT+1000 (AUS Eastern Standard Time) is exactly 2014-05-08 00:00 AM in UTC time.
Also in Date.UTC method, month parameter is 0-11. That's why you pass 4, but actually it's May.

My bad. My css was hiding the all day event in the day and week view.


Improve check result input validation

This pull-request improves check result input validation for the Sensu API POST /results endpoint and Client socket. Validation leverages existing attributes rules in sensu-settings.
Closes #1886

Need more test coverage.

seems like interesting improvements but not really related to the ticket

@davideauee we still need to address #1885 (I tried to add some exception rescue blocks as a "catch-all", however, they did not catch everything)


Listening on shared queues

It would be great to include support for SQS queues owned by another AWS account.
The boto3 docs indicate that QueueOwnerAWSAccountId needs to be supplied to get_queue_url() for this to work.
Without that parameter, I'm seeing an exception:
botocore.errorfactory.QueueDoesNotExist: An error occurred (AWS.SimpleQueueService.NonExistentQueue) when calling the GetQueueUrl operation: The specified queue does not exist for this wsdl version.

Feature merged with with PR #2, thank you for your contribution! ✅


Cannot find module @aws-sdk/types with yarn pnpm linker

Both the sst and sst-aws-cdk package have missing dependencies that cause errors with yarn's pnpm linker. Unlike pnpm yarn doesn't allow dependencies to access other dependencies they haven't explicitly declared. This is roughly equivalent to setting hoist=false on pnpm.
For example, sst imports @aws-sdk/types in credentials.ts, but doesn't declare it as a dependency. When run with yarn it results in the error Error: Cannot find module '@aws-sdk/types'.
I can work around this by manually specifying the dependencies in .yarnrc.yaml, but ideally the package.json would be updated to appropriately declare the dependencies.
packageExtensions:
  "sst@*":
    peerDependencies:
      "@aws-sdk/types": "*"
  "sst-aws-cdk@*":
    peerDependencies:
      "@aws-cdk/cx-api": "*"
      "@aws-cdk/cloud-assembly-schema": "*"
      "@aws-cdk/cloudformation-diff": "*"
      "aws-sdk": "*"
    dependencies:
      "fs-extra": "*"

Can you post this on #help on Discord? Need some more info on this.

We are transition this repo to https://github.com/sst/v2 and are closing this issue for now. If it is still relevant please reopen it there - sorry for the trouble


Add location /ws for ssl

To be used after wss is enabled in goshimmer. This config is tested and working at my own host, but you should test it also.

Hi @TeeVeeEss !
I don't know how I have missed your PR :(
Sorry about it.
I have already updated nginx today https://github.com/nuriel77/goshimmer-playbook/commit/e1a5f8c02fd68011b3cb4fa624b5795e4dceae14
Thanks!

No problem, it was not a very complicated PR :)


Add PDU session setup test

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ x ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

[ ] I have updated the documentation accordingly.
[ x ] I have added tests to cover my changes.
[ x ] All new and existing tests passed.

[ x ] I have read the CONTRIBUTING document.
[ x ] Each of my commits messages include Signed-off-by: Author Name<EMAIL_ADDRESS>to accept the DCO.

Hi,
Here is the PCAP when running the test with 1 UE: pduSetupTest.zip
I updated GitHub Action workflow and the test so it runs with 10 UE now.

Hi, Here is the PCAP when running the test with 1 UE: pduSetupTest.zip I updated GitHub Action workflow and the test so it runs with 10 UE now.

Look all good to me, thanks a lot for your PR!


Depreciated imports

LangChainDeprecationWarning: Importing HuggingFaceEmbeddings from langchain.embeddings is deprecated.
LangChainDeprecationWarning: Importing Chroma from langchain.vectorstores is deprecated.

@abuzahid Should be good now.


WaspError should expose response body

There should be a method for getting response body. Moreover, it would be good if WaspError could provide a method which will get a type and return an instance of that type created from response body.

Will be available with version 1.5


draw.io is reaching out to js.pusher.com similiar to #407

Preflight Checklist

[x] I agree to follow the Code of Conduct that this project adheres to.
[x] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Describe the bug
draw.io is reaching out to js.pusher.com similiar to #407
To Reproduce
We installed draw.io inside a dark site without internet connection. Inside this site we also have a on-premise installation of gitlab.com. Integration of draw.io and our local installation of gitlab was without issue. However accessing the local installation of draw.io (running using the docker-image) it is apparent, that draw.io tries to reach out to js.pusher.com but will fail eventually (because no internet).
Like in #407 we just can't set stealth=1 as this disables the storage option for our on-premise gitlab installation as well.
Expected behavior
Would like to see an option to disable each external service via the server-side configuration options (like in PreConfig.js). So that i can disable unwanted storage options and everything else stealth=1 does (like disabling js.pusher.com) on an per-option base.
draw.io version (In the Help->About menu of the draw.io editor):

draw.io version 13.6.2

Desktop:

OS: Ubuntu 20LTS
Browser chrome
Version 84.0.4147.105

Additional context
Using the docker-image jgraph/drawio:latest with digest: sha256:7561cd562fc1b8935faff5886c635f7614a71d57b3eae5a3d9680972e05fa0e4

Do you have DrawioFile.SYNC = 'none'; //Disable real-time sync in PostConfig.js?
Did you use https://github.com/jgraph/docker-drawio/tree/master/self-contained?

Sorry, missed the mail notification.

Do you have DrawioFile.SYNC = 'none'; //Disable real-time sync in PostConfig.js?
No, this isn't set

Did you use https://github.com/jgraph/docker-drawio/tree/master/self-contained?
Partially. I just added DRAWIO_SELF_CONTAINED: 1 to my docker-compose.yml and moved to a newer image. Problem went away, so closing this