Attempt at fixing stale deprovision underway metrics.

We've seen these get stuck in prod, metric keeps reporting a value, but
the cluster is long gone, and the problem will not correct itself until
hive pod is restarted at which point we no longer report.
Cannot reproduce, or say for sure why, but it's possibly a race
condition between the two places this metric was managed. This change
moves it to one controller, and makes sure we clear it if we process a
deleted cluster deployment that doesn't have our finalizer.

/lgtm


Requesting merging of the wiki

I request the owner of the repo to review the changes i made to the wiki and, if correct, please merge them with the current one. I fixed some grammatical errors and added an entry on SonarQube.
The link to my repo is https://github.com/deerter/Edited-wiki
Many thanks.

What? Can't you create a pull request to merge the changes? @fjlopez any ideas about this?

Yes, but all I see is that you canged description.md, not the wiki itself.
Let's wait for @fjlopez to reply, but as far as I understand, you should fork this repository, perform the changes you want to do on description.md, commit, push to your fork and create a pull request to this repository.

Btw @fjlopez, I think the contents in the section How to update the Wiki are totally misleading for new contributions.
Maybe it would make sense to create a CONTRIBUTION template like many other GitHub repos do?

Yes. It is misleading. I think that it is better to remove it rather than fix.

LGTM


Add debug with breakpoint onFailure to TaskRun Spec

Changes
Add debug with breakpoint onFailure to TaskRun Spec

Add -breakpoint_on_failure to entrypointer which disables exit of container upon failure
If debug is enabled (as shown below) mount /tekton/debug/scripts and /tekton/debug/info/<step-no> to TaskRun Pod.

spec:
  debug:
    breakpoint: ["onFailure"]

Add debug helper scripts to /tekton/debug/scripts
Add /tekton/debug/info/ mount which is used by helper scripts to understand which step they are running in where  denotes the step number. First step = 0, Second step = 1 and so on.

In addition, categorize the changes you're making using the "/kind" Prow command, example:
/kind feature
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[x] Includes tests (if functionality changed/added)
[x] Includes docs (if user facing)
[x] Commit messages follow commit message best practices
[x] Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.
Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes
If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.
Release Notes

Add -breakpoint_on_failure to entrypointer which disables exit of container upon failure
Add debug helper scripts to /tekton/debug/scripts
Add /tekton/debug/info/ mount which is used by helper scripts to understand which step they are running in where  denotes the step number. First step = 0, Second step = 1 and so on.

/kind feature

/assign

/test tekton-pipeline-unit-tests

/test tekton-pipeline-unit-tests

cc @vdemeester @bobcatfish

/retest

/test pull-tekton-pipeline-integration-tests
/test pull-tekton-pipeline-alpha-integration-tests
/test pull-tekton-pipeline-build-tests

It looks like you might need to rerun the ./hack/update-openapigen.sh:
 						SchemaProps: spec.SchemaProps{
 							Default: map[string]interface{}{},
-							Ref:     ref("k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta"),
+							Ref:     ref("k8s.io/apimachinergithub.com/tektoncd/pipeline/pkg/apis/meta/v1.ObjectMeta"),
 						},

This Ref is very strange - looks like some string joining went wrong somewhere and we ended up with apimachinergithub.com.  Hopefully it'll be fixed by the update script.

It would be great to keep the test coverage from slipping too much - particularly with the entrypoint package it looks like a relatively large drop.

Forgot to mention yesterday but we have some developer docs on writing new features with the "alpha" feature gate: https://github.com/tektoncd/pipeline/tree/main/docs/developers#adding-feature-gated-api-fields
This should help when writing integration tests / examples / api field validations

It looks like you might need to rerun the ./hack/update-openapigen.sh:
 						SchemaProps: spec.SchemaProps{
 							Default: map[string]interface{}{},
-							Ref:     ref("k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta"),
+							Ref:     ref("k8s.io/apimachinergithub.com/tektoncd/pipeline/pkg/apis/meta/v1.ObjectMeta"),
 						},

This Ref is very strange - looks like some string joining went wrong somewhere and we ended up with apimachinergithub.com. Hopefully it'll be fixed by the update script.

oh my bad..

/test pull-tekton-pipeline-integration-tests
/test pull-tekton-pipeline-alpha-integration-tests
/test pull-tekton-pipeline-build-tests

/test pull-tekton-pipeline-integration-tests
/test pull-tekton-pipeline-alpha-integration-tests
/test pull-tekton-pipeline-build-tests

/test pull-tekton-pipeline-integration-tests
/test pull-tekton-pipeline-alpha-integration-tests
/test pull-tekton-pipeline-build-tests

/test pull-tekton-pipeline-integration-tests
/test pull-tekton-pipeline-alpha-integration-tests

integration tests should be back in working order once https://github.com/tektoncd/pipeline/pull/3960 is merged.  You'll need to rebase once that's in.

@sbwsg thanks for the heads ups !
working on getting some more tests done till then

@sbwsg I just updated the tests, let me know what you think

/retest

/kind feature

/retest

/retest

/retest

/retest

/test tekton-pipeline-unit-tests

/test tekton-pipeline-unit-tests

/test pull-tekton-pipeline-alpha-integration-tests

/test pull-tekton-pipeline-integration-tests

/test pull-tekton-pipeline-alpha-integration-tests

/test pull-tekton-pipeline-alpha-integration-tests

/test pull-tekton-pipeline-alpha-integration-tests
/test tekton-pipeline-unit-tests
/test pull-tekton-pipeline-build-tests

/test pull-tekton-pipeline-alpha-integration-tests
/test pull-tekton-pipeline-build-tests

/test pull-tekton-pipeline-build-tests
/test pull-tekton-pipeline-alpha-integration-tests

Adding lgtm since the remaining changes I've suggested are small doc fixes.
Suggest also adding an entry about debug to the "Alpha Features" section of install.md..
/lgtm

@sbwsg I will create a separate PR for the doc changes. Thank you for taking time to review this.


Boolean in Figma "hide on collapse" is working the wrong way

Prerequisites

[X] I have read the Contributing Guidelines.
[X] I have not leaked any internal/restricted information like screenshots, videos, code snippets, links etc.

What happened?
I am using the Figma library and yesterday I recognized that the Pane Options are working a littlebit confusing

I have to set the "Hide on collapse" to off to get the X instead of << but it should be the other way round i guess since it is completely hidden if I close the pane.

What type of frontend framework are you seeing the problem on?
Angular
Which version of iX do you use?
Figma lib
Code to produce this issue.
It seems like it is working correctly in the ix-angular library within code.

Hi @katjaKoppensteiner, thanks for submitting the issue.
You are right. Has been fixed. Update should be available.


Update sql-database-disaster-recovery.md

Reviewed and updated for freshness.

Hi @elfisher, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;


Hopefully fixes #13

Test first though!

In the little bit of testing I've done (which, I will say, was pretty minimal) it seemed to work perfectly!
There were a few situations that were a good example of what this was trying to fix, and it got out just fine.


CUDA test failures on Tegra TX2 (only a few!)

Much to my surprise, Numba's CUDA target mostly "just works" on the Tegra TX2 (there are some CPU failures that I'll post in another issue).  Note that a Tegra TX2 is ARMv8 combined with a GP10B CUDA device, which is Pascal (but compute capability 6.2, different from GTX 10XX).  When using the Jetson TX2 dev board, the most common operating system is Ubuntu 16.04, where Numba can run without any modification (not even the small patch we need for ARMv7).
After compiling LLVM 5.0.1, llvmlite master, and Numba master, you can run the CUDA unit tests with a command like this:
NUMBAPRO_LIBDEVICE=/usr/local/cuda/nvvm/libdevice/ NUMBAPRO_NVVM=/usr/local/cuda/nvvm/lib/libnvvm.so NUMBAPRO_CUDALIB=/usr/local/cuda/lib64/ python -m numba.runtests -v numba.cuda.tests

(we really need to change those environment variable names, and maybe make a convenience NUMBA_CUDA_SDK that lets you specify the top-level CUDA directory and infers the rest of this.)
This command produces the following failing tests:

======================================================================
ERROR: test_pinned (numba.cuda.tests.cudadrv.test_pinned.TestPinned)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudadrv/test_pinned.py", line 27, in test_pinned
    with cuda.pinned(A):
  File "/home/nvidia/jetconda3/lib/python3.6/contextlib.py", line 82, in __enter__
    return next(self.gen)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/api.py", line 203, in pinned
    mapped=False)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 737, in mempin
    allocator()
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 732, in allocator
    driver.cuMemHostRegister(pointer, size, flags)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 288, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 323, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [801] Call to cuMemHostRegister results in CUDA_ERROR_NOT_SUPPORTED

======================================================================
ERROR: test_pinned_memory (numba.cuda.tests.cudadrv.test_cuda_memory.TestCudaMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudadrv/test_cuda_memory.py", line 43, in test_pinned_memory
    mapped=True)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 735, in mempin
    self._attempt_allocation(allocator)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 656, in _attempt_allocation
    allocator()
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 732, in allocator
    driver.cuMemHostRegister(pointer, size, flags)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 288, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 323, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [801] Call to cuMemHostRegister results in CUDA_ERROR_NOT_SUPPORTED

======================================================================
FAIL: test_ipc_array (numba.cuda.tests.cudapy.test_ipc.TestIpcMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 157, in test_ipc_array
    self.fail(out)
AssertionError: Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 61, in ipc_array_test
    with ipcarr as darr:
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 568, in __enter__
    return self.open()
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 558, in open
    dptr = self._ipc_handle.open(devices.get_context())
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 944, in open
    mem = context.open_ipc_handle(self.handle, self.size)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 770, in open_ipc_handle
    driver.cuIpcOpenMemHandle(byref(dptr), handle, flags)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 288, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 323, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to call_cuIpcOpenMemHandle results in CUDA_ERROR_INVALID_VALUE

======================================================================
FAIL: test_ipc_handle (numba.cuda.tests.cudapy.test_ipc.TestIpcMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 109, in test_ipc_handle
    self.fail(out)
AssertionError: Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 22, in core_ipc_handle_test
    arr = the_work()
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 38, in the_work
    dtype=dtype) as darr:
  File "/home/nvidia/jetconda3/lib/python3.6/contextlib.py", line 82, in __enter__
    return next(self.gen)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/api.py", line 140, in open_ipc_array
    strides=strides, dtype=dtype)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 960, in open_array
    dptr = self.open(context)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 944, in open
    mem = context.open_ipc_handle(self.handle, self.size)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 770, in open_ipc_handle
    driver.cuIpcOpenMemHandle(byref(dptr), handle, flags)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 288, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 323, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to call_cuIpcOpenMemHandle results in CUDA_ERROR_INVALID_VALUE

======================================================================
FAIL: test_ipc_handle_serialization (numba.cuda.tests.cudapy.test_ipc.TestIpcMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 138, in test_ipc_handle_serialization
    self.fail(out)
AssertionError: Traceback (most recent call last):
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 22, in core_ipc_handle_test
    arr = the_work()
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/tests/cudapy/test_ipc.py", line 50, in the_work
    dtype=dtype)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 960, in open_array
    dptr = self.open(context)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 944, in open
    mem = context.open_ipc_handle(self.handle, self.size)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 770, in open_ipc_handle
    driver.cuIpcOpenMemHandle(byref(dptr), handle, flags)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 288, in safe_cuda_api_call
    self._check_error(fname, retcode)
  File "/home/nvidia/jetconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 323, in _check_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to call_cuIpcOpenMemHandle results in CUDA_ERROR_INVALID_VALUE

----------------------------------------------------------------------
Ran 415 tests in 236.222s

FAILED (failures=3, errors=2, skipped=4, expected failures=1)

All of the failing tests should be skipped because the GP10B is missing a few features that are present on the desktop Pascal cards, such as:

CUDA IPC (limitation of the OS drivers?)
Pinned memory (not really needed since GPU and CPU share the same memory bus)

Also, to clarify, it look like specifically cuMemHostRegister is not supported on Tegra.  The cuMemAllocHost call that directly allocates page-locked memory should work.
Not sure yet what the best programmatic way to detect these limitations is.

@gmarkall is CUDA on the TX2 now working?

@stuartarchibald The IPC tests are now skipped as unsupported and the pinned memory test has a Jetson-specific configuration that it now uses, so I believe this is resolved.


Self-looped vertices

Currently the self-looped vertices are indicated with a star in the corner. I think this is somewhat unclear and unintuitive. An arrow which starts and ends in the same vertex would be better IMO. It also makes it possible to display label on these kind of edge.

Hi, you can define **<Path x:Name="PART_SelfLoopedEdge" ** in EdgeControl template and set any form you like, for ex. try setting **  Data="F1 M 18,42.5L 18,49L 27,57L 36,48.5L 36,42.5L 30,47.7903L 30,35.5C 30,29.701 34.701,25 40.5,25C 46.299,25 51,29.701 51,35.5C 51,41.1314 46.5668,45.7273 41,45.9883L 41,51.9926C 49.8814,51.7283 57,44.4454 57,35.5C 57,26.3873 49.6127,19 40.5,19C 31.3873,19 24,26.3873 24,35.5L 24,47.875L 18,42.5 Z " ** As i remeber correctly edge label in this case should be aligned to vertex.

Well, to be honest, I'm not familiar with WPF, so I ended up tinkering with the core logic's properties. It would be great if there are some properties that affects these behaviors too.


Feature/1449/checker launch auth

Endpoint for getting published workflow no longer requires auth. Checker workflow client calls can be properly done if entry is published or the user owns.
https://github.com/ga4gh/dockstore/issues/1449

Coverage decreased (-23.8%) to 36.04% when pulling 004fc6851018a08fb30e020f5139d79bae9c2dfd on feature/1449/checker-launch-auth into bff2eed271bd28e0f6daf9e792aa1931993d9f39 on develop.


Fix requiring geerlingguy.epel role only when necessary

Thanks @noplanman for pointing out the requirement was wrong in https://github.com/fiaasco/borgbackup/pull/25#issuecomment-611190431.
My bad, this is what I get for trying to slip quick changes into my day. ;)
Haven't tested it because I'm still trash.

This still doesn't work as the syntax / spacing is wrong.
(lines after "when" need to be indented)
dependencies:
  - name: geerlingguy.repo-epel
    when: >
      ansible_os_family == 'Redhat'
      and borgbackup_install_epel

Probably easier to just put it on 1 line:
dependencies:
  - name: geerlingguy.repo-epel
    when: ansible_os_family == 'Redhat' and borgbackup_install_epel

@noplanman That should be better now.

Ok, merged the previous one too fast, sorry.


Add markEager and update FlexList docs

Description
This adds an escape hatch for using FlexLists when the items in the list are functions.
It also improves some of the documentation for FlexList and renames normalizeFlexList to normalizeFlexListLazy for clarity.

Changes make sense to me.
My only feedback is around depth of documentation for "eager" vs "lazy". If this new infra is intended for public consumption, I think we need to expand our code comments to include more details about what the differences between the modes are, or link to documentation explaining it.

We are not publicly exporting these, but if we were I would agree they should have more docs and probably some examples. Since we're not, I think the current documentation is sufficient.


[CLI-457] Removed error when matplotlib plots do not contain data

Ensures that wandb.Image constructor can accept a pyplot reference in which the figure has multiple axes. Importantly, there is no requirement that any of the axes have plotted data.

Added a test case to ensure that multiple axes plots will log appropriately
Removed the check which was causing the false error condition

Fixes: https://github.com/wandb/client/issues/1319
Reference: https://wandb.atlassian.net/browse/CLI-457

Ok, so in order to do this the right way, there was a bit of scope creep. Happy to make this a simpler change if that is desired, but I think the tests add to the robustness. Overall changes:

Solved the original bug by removing check for data in plots
Added plotly dev dependency
Added helper functions (and refactored associated methods) to reduce duplication between wandb.Plotly and wandb.Plotly.make_plot_media:

util.matplotlib_to_plotly
util.matplotlib_contains_images

Added additional test cases to test for the following:

wandb.Image w/multiple axes optional data
wandb.Plotly w/multiple axes optional data
wandb.Plotly w/ image
wandb.Image w/ image
wandb.Plotly.make_plot_media w/ data
wandb.Plotly.make_plot_media w/ image

Added test cases for:

util.matplotlib_contains_images w/ and w/o images
util.matplotlib_to_plotly

Added new tests/utils/dummy_data.py file to help build mock data for tests:

matplotlib_multiple_axes_figures
matplotlib_with_image
matplotlib_without_image

Ultimately there are two reasons for the complexity here: 1) plotly does not support converting matplotlib figures to plotly which contain images (this means we still need to error out if the plotly constructor gets a figure with an image); 2) there is a very similar code path in wandb.Plotly.make_plot_media. I got feedback that there was some non-DRY code in the last commit, so in order to properly refactor, I had to introduce 2 new helper methods and associated tests.
Let me know your thoughts,
Tim


Logstash Helm Chart log4j: Execute command to remove the JndiLookup class

Hey everybody,
is there an option to remove the JndiLookup class by using the Helm Chart from the log4j2 core jar as described here to solve the security vulnerability:
https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476
Thx in advance!

FYI, 7.16.1 and 6.8.21 releases just got released.
These are the versions to use for the log4j fix.

Hi All,
Any update on removing JNLP class from logstash Helm charts ? Or if anyone has figured out a workaround for the same ?
Thx in advance!

@ritubajaj170889 I think you have to update to the latest 6.x / 7.x Version to get the fix.

@dk-do - I am using Opendistro 1.13.3 and to make logstash 7.16.1 compatible, we need to install opensearch-output-plugin.
Does creating custom image as below will work?
FROM docker.elastic.co/logstash/logstash-oss:7.16.1
RUN bin/logstash-plugin install logstash-output-opensearch
Thanks!

hi! does this include the 2nd log4j issue fixed too? can you indicate in the chart release change that it does (or not)?
https://www.lunasec.io/docs/blog/log4j-zero-day-update-on-cve-2021-45046/

The change done in the 7.16.1 chart release is: use 7.16.1 Logstash Docker image.
You can look at the Logstash release note or create an issue to Logstash repo if you need more details.


[Feature]  Visual comparisons port to java library

Playwright node.js already has "Visual comparisons" functionality which can be used for visual regression testing, we need the same in playwright java library as well.

I am totally ++++. This would be great to have it for all other languages.

This is a part of playwright test runner in node.js world. The test runner manages the screenshot files and directories, generating names for them etc. We can expose the image matching functions but JUnit/TestNG don't have built in support for managing test artifacts and we don't have our own runner in Java. Do you think that having something like assertThat(page).hasScreenshot(path) would be sufficient?

The issue here is that people programing on the other langusges are seeing the great features that are being added to the node js runner and we get envious. Would be nice to have a Playwright runner in all other langusges :)
As for the hasScreenshot(path) not sure how it would help, we need visual comparison feature.

In our framework we are using listeners to capture screenshots and manage file names, so it would be good to have atleast the functionality already present in node.js version.
From documentation i see there are assertions involving pixel threshold and content.
await expect(page).toHaveScreenshot({ maxDiffPixels: 100 });
expect(await page.textContent('.hero__title')).toMatchSnapshot('hero.txt');

Selenium has ashot library which supports visual testing on below parameters:
comparison of screenshots of pages;
comparing screenshots of pages + ignoring specified elements (dynamic content, etc.);
comparison of screenshots of elements;
breakpoints – you can compare screenshots of pages or elements with different widths. The default is 3 breakpoints – 1920px, 768px, 360px;
auto-generation of expected screenshots;
you can write tests as ordinary functional automation tests, there are no restrictions – this is important because sometimes it may be necessary to perform some complex actions before taking a screenshot.

@yury-s
Could the following features be added?
We would appreciate it if you could add them.
assertThat(page).hasScreenshot(path)

Hello @yury-s - assertThat(page).hasScreenshot(path) would be great feature to have, exposing the image matching functions is the important thing here.
Is there any timeline for that feature to be developed? :)

Hi @yury-s,
assertThat(page).hasScreenshot(path) should suffice for our framework's needs. Eagerly waiting for this feature.

Also giving it a star. Would be really good to have this also in java.

We would really need this. We would like to avoid having java and nodejs separate modules just because this feature is not available in java atm.

Would also love to see this in Java

Would love to see this feature!

assertThat(page).hasScreenshot(path) would be nice to have. Thanks

Is there any update on this topic? I would love to have it in Java.

I would really like this feature to be ported, to avoid importing extra 3rd party libraries. +1!

+1

We also need to be able do visual comparisons in playwright-java and it would be really nice to not have to resort to 3rd party libraries.


Feature: Ability to save a report

It would be nice if a current state of a pivot table could be stored (and shared). The current state should include the dimensions and measures on each of the axis and the filters.

Rudimentary support is now available as the url hash gets updated everytime a query is successfully rendered.
Saving the query is done simply by bookmarking it.

Rudimentary support is now available as the url hash gets updated everytime a query is successfully rendered. Saving the query is done simply by bookmarking it.

I like this solution a lot to save and share reports.
Now after #144 I thought I could try sharing a report as a URL, but found this doesn't work
https://rpbouman.github.io/huey/src/index.html#JTdCJTIycXVlcnlNb2RlbCUyMiUzQSU3QiUyMmRhdGFzb3VyY2VJZCUyMiUzQSUyMmZpbGUlM0ElNUMlMjJodHRwcyUzQSUyRiUyRmR1Y2tkYi5vcmclMkZkYXRhJTJGaG9sZGluZ3MucGFycXVldCU1QyUyMiUyMiUyQyUyMmNlbGxzSGVhZGVycyUyMiUzQSUyMmNvbHVtbnMlMjIlMkMlMjJheGVzJTIyJTNBJTdCJTIyY2VsbHMlMjIlM0ElNUIlN0IlMjJjb2x1bW4lMjIlM0ElMjJzaGFyZXMlMjIlMkMlMjJjb2x1bW5UeXBlJTIyJTNBJTIyRE9VQkxFJTIyJTJDJTIyZGVyaXZhdGlvbiUyMiUzQW51bGwlMkMlMjJhZ2dyZWdhdG9yJTIyJTNBJTIyc3VtJTIyJTdEJTVEJTJDJTIycm93cyUyMiUzQSU1QiU3QiUyMmNvbHVtbiUyMiUzQSUyMndoZW4lMjIlMkMlMjJjb2x1bW5UeXBlJTIyJTNBJTIyVElNRVNUQU1QJTIyJTJDJTIyZGVyaXZhdGlvbiUyMiUzQSUyMnllYXIlMjIlMkMlMjJhZ2dyZWdhdG9yJTIyJTNBbnVsbCU3RCU1RCU3RCU3RCU3RA==


enh: more mature support for external (non-SQLite) databases

Is your feature request related to a problem? Please describe.
Currently, there is support for specifying an external database via the DATABASE_URL environment variable instead of the default SQLite (added in https://github.com/open-webui/open-webui/issues/1397).
However, it seems that OpenWebUI (at least in the main branch) doesn't fully manage the database connection lifecycle, likely because SQLite is the default option, and connection management is often less critical in this context.

"Standard" approach to handling database connections in web applications looks something like

Init database connection pool

Start web server (so (1) should be done via something like lifespan in FastAPI terms)

On every incoming request - take connection from pool, execute required sql, return response to the request - and return connection to pool

Before application shutdown - gracefully close all connections to database

When using SQLite, these steps (outlined in the collapsible section) may seem overcomplicated, so OpenWebUI currently creates a single connection to the database at application start.
However, if we specify a Postgresql DSN via the DATABASE_URL env var, OpenWebUI works fine only for a limited time.
The single connection will eventually be closed by the database since connections are not immortal. Currently, neither Peewee nor OpenWebUI handles reconnections in such scenarios.
Describe the solution you'd like
OpenWebUI is build using FastAPI framework and uses Peewee ORM.
FastAPI documentation includes tutorial / how-to work with Peewee from asynchronous application, and Peewee documentation points to the same tutorial.
In short - this tutorial is about ensuring that

PeeWee functions correctly in asynchronous environment
For every request [to our FastAPI application] there is open and ready-to-be-used connection to the database.

First part can be achieved by adding the following code to the backend/apps/web/internal/db.py file (adapted from the tutorial linked above):
...
from contextvars import ContextVar

from peewee import InterfaceError as _ConnectionState
...

db_state_default = {"closed": None, "conn": None, "ctx": None, "transactions": None}
db_state = ContextVar("db_state", default=db_state_default.copy())

class PeeweeConnectionState(_ConnectionState):
    def __init__(self, **kwargs):
        super().__setattr__("_state", db_state)
        super().__init__(**kwargs)

    def __setattr__(self, name, value):
        self._state.get()[name] = value

    def __getattr__(self, name):
        return self._state.get()[name]

...

DB = connect(DATABASE_URL)
DB._state = PeeweeConnectionState()  # overrride _state property after connection

...

The second part is about ensuring an active connection. This can be done using following ad-hoc method:
Peewee has an almost ready-made solution: mixin playhouse.shortcuts.ReconnectMixin.
We just need to adapt this to psycopg2 exceptions (in the same backend/apps/web/internal/db.py file):
...
from peewee import InterfaceError as PeeWeeInterfaceError
from psycopg2 import OperationalError
from psycopg2.errors import InterfaceError
from playhouse.db_url import PooledPostgresqlDatabase, register_database
...

class PGReconnectMixin(ReconnectMixin):
    reconnect_errors = (
        (OperationalError, 'termin'),
        (InterfaceError, 'closed'),
        (PeeWeeInterfaceError, 'closed'),
    )

class ReconnectingPostgresqlDatabase(PGReconnectMixin, PooledPostgresqlDatabase):
    pass

register_database(ReconnectingPostgresqlDatabase, 'postgres+pool', 'postgresql+pool')

...

I acknowledge that this approach may not be ideal, but it should be an improvement.
Describe alternatives you've considered
Migrating from Peewee ORM to SQLAlchemy ORM (or SQLAlchemy Core) could greatly benefit the project. However, this requires substantial refactoring, which means a significant investment of person-hours to implement and even more time to properly review and test the changes.
Additional context
Q&A:

Will OpenWebUI stop working with a "connection closed" error after these changes? Hopefully not, as we will intercept such exceptions and reconnect to the database.
Is this solution complete? I am not entirely sure.
Will it provide isolation between different asynchronous requests to OpenWebUI? Again, I am not sure.

Any feedback or guidance on this would be greatly appreciated. Thank you for your hard work and dedication to this project.

Great suggestion! As long as it won't break the existing default sqlite db, PR welcome!

Is this the reason why I get 404 not found errors after a certain period of time after initial startup, when using postgresql? Works fine on restart but logins fail after some time.


Contribute via test case link is broken in readme page

Describe the bug
Link is broken results 404.
Link: https://github.com/datawire/ambassador/tree/master/ambassador/tests
To Reproduce
Steps to reproduce the behavior:

Go to '...' https://github.com/datawire/ambassador/tree/master/ambassador/tests and you should find a 404 issue.

Expected behavior
The link should open to the issues tab where users can find the issues with the coverage flag. There should be a flag suggesting coverage required in the issues tag, and the above link should point to that.
It seems like the datawire team changed the tags and it is referencing a broken link now.
I would like to contribute to this once the tag is finalized.

At version 2.x the repos were split into Emissary and Edge Stack repos so I don't think this link is still relevant.


Switch to no_std and 2018 edition

Resolves #54
Thanks!

seems that the doc tests complain about the lack of global allocator, and the only way to solve it is like:
diff --git a/src/lib.rs b/src/lib.rs
index 59a3e5b..482cab9 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -1,7 +1,8 @@
 #![doc(html_root_url = "https://docs.rs/slab/0.4.2")]
+#![feature(doc_cfg)]
 #![deny(warnings, missing_docs, missing_debug_implementations)]
 #![cfg_attr(test, deny(warnings, unreachable_pub))]
-#![no_std]
+#![cfg_attr(not(any(test, rustdoc)), no_std)]

 //! Pre-allocated storage for a uniform data type.
 //!

but feature doc_cfg is still unstable.


OpenSRP FHIR Gateway repository diff to upstream main

Describe the Issue
We need to create the accurate diff of chances between the OpenSRP repo and the FHIR Gateway upstream repo. This will help determing which changes can be contributed back upstream, which could be refactored in to plugins or which changes can be made onto upstream to enhance the Gateway.
Currently the PR introduces extra changes that are not required leading to a bloated PR. Cherry picking of the correct changes is required to get the correct diff.
https://github.com/google/fhir-gateway/pull/179

This is DONE


MySQL/PostgreSQL VNET with API Version 2017-12-01 for Public Preview.

Note : This API has only version change compared to Previous check in .
Ref PRs :
Original PR
https://github.com/Azure/azure-rest-api-specs/pull/1773
Revert due to Private Preview Feature
https://github.com/Azure/azure-rest-api-specs/pull/2411
API version change for Public Preview
https://github.com/Azure/azure-rest-api-specs-pr/pull/478
@QingqingYuan  , @rohit-joy  FYI .
@marstr  , @sarangan12 for review , Please do not merge until the label is removed  , I will notify proactively .
This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
PR information

[ ] The title of the PR is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[ ] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[ ] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[ ] I have read the contribution guidelines.
[ ] My spec meets the review criteria:

[ ] The spec conforms to the Swagger 2.0 specification.
[ ] The spec follows the guidelines described in the Swagger checklist document.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-python

Automation for azure-sdk-for-node

Automation for azure-libraries-for-java

Automation for azure-sdk-for-go

Waiting for ARM (Gaurav) approval for the new version...

Signing off!


Documentation for using rspc with Next.JS

I feel like this will be a common question. Some documentation and maybe an example project would be good.

We have an example which should be good enough for now.


Problems with clob columns that are null

Does anybody else experience troubles with null in clob-columns?
It seems to me that the driver is working perfectly with all other types and
when there are values in the clobs.
When i hit clob-columns wih null, the drivers seems to hang for å long while and then crash with the error-message "index out of bounds".
I am using ora.OraS in the prepare-statement and ora.String as Go-types, Go v. 1.8 and Oracle Client Shared Library 64-bit - <IP_ADDRESS>.0 on macOS 10.12.3

Can you provide a minimal test case? That would help the fix!
Trond Kandal<EMAIL_ADDRESS>ezt írta (időpont: 2017. febr. 19.,
V, 18:16):

Does anybody else experience troubles with null in clob-columns?
It seems to me that the driver is working perfectly with all other types
and
when there are values in the clobs.
When i hit clob-columns wih null, the drivers seems to hang for å long
while and then crash with the error-message "index out of bounds".
I am using ora.OraS in the prepare-statement and ora.String as Go-types,
Go v. 1.8 and Oracle Client Shared Library 64-bit - <IP_ADDRESS>.0 on macOS
10.12.3
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/rana/ora/issues/156, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPoSlFLwiVlTaJnT8VdIVefGk1ygykWks5reHjcgaJpZM4MFha-
.

  The database is a legacy database and protected.  :) Where do You want the Go code?

here :)
I need only the CREATE TABLE, the INSERT for one record, and the code of
select which reproduces the problem.
 The database is a legacy database and protected. :) Where
do You want the Go code?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rana/ora/issues/156#issuecomment-281054966, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPoSkJnyPf_sh-nya8QeB-m_p9U1OKtks5reXhJgaJpZM4MFha-
.

CREATE TABLE "TIASTAGE"."TIA_EMNEINFO"
(	"INSTITUSJONSNR" NUMBER(8,0) NOT NULL ENABLE,
"EMNEKODE" VARCHAR2(12 CHAR) NOT NULL ENABLE,
"VERSJONSKODE" VARCHAR2(3 CHAR) NOT NULL ENABLE,
"INFOTYPEKODE" VARCHAR2(10 CHAR) NOT NULL ENABLE,
"SPRAKKODE" VARCHAR2(10 CHAR) NOT NULL ENABLE,
"TERMINKODE_FRA" VARCHAR2(4 CHAR) NOT NULL ENABLE,
"ARSTALL_FRA" NUMBER(4,0) NOT NULL ENABLE,
"TERMINKODE_TIL" VARCHAR2(4 CHAR),
"ARSTALL_TIL" NUMBER(4,0),
"INFOTEKST" CLOB,
"INFOTEKST_ORIGINAL" CLOB,
"INSTITUSJONSNR_EIER" NUMBER(8,0) NOT NULL ENABLE,
CONSTRAINT "TIA_EMNEINFO_PK" PRIMARY KEY ("INSTITUSJONSNR", "EMNEKODE", "VERSJONSKODE", "INFOTYPEKODE", "SPRAKKODE", "TERMINKODE_FRA", "ARSTALL_FRA")
USING INDEX PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS<PHONE_NUMBER>
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
TABLESPACE "TIASTAGE"  ENABLE
) SEGMENT CREATION IMMEDIATE
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
NOCOMPRESS LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS<PHONE_NUMBER>
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
TABLESPACE "TIASTAGE"
LOB ("INFOTEKST") STORE AS BASICFILE (
TABLESPACE "TIASTAGE" ENABLE STORAGE IN ROW CHUNK 8192 RETENTION
NOCACHE LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS<PHONE_NUMBER>
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT))
LOB ("INFOTEKST_ORIGINAL") STORE AS BASICFILE (
TABLESPACE "TIASTAGE" ENABLE STORAGE IN ROW CHUNK 8192 RETENTION
NOCACHE LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS<PHONE_NUMBER>
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)) ;

type EmneInfo struct {
InstitusjonsNr     ora.Int64  json:"institusjonsnr"
EmneKode           ora.String json:"emnekode"
VersjonsKode       ora.String json:"versjonskode"
InfoTypeKode       ora.String json:"infotypekode"
SprakKode          ora.String json:"sprakkode"
TerminKodeFra      ora.String json:"terminkode_fra"
ArstallFra         ora.Int64  json:"arstall_fra"
TerminKodeTil      ora.String json:"terminkode_til"
ArstallTil         ora.Int64  json:"arstall_til"
InfoTekst          ora.String json:"infotekst"
InfoTekstOriginal  ora.String json:"infotekst_original"
InstitusjonsNrEier ora.Int64  json:"institusjonsnr_eier"
}
var sqlQuery string = "SELECT * FROM TIA_EMNEINFO"
stmt, err := ses.Prep(sqlQuery, ora.OraI64, ora.OraS, ora.OraS, ora.OraS, ora.OraS, ora.OraS,
ora.OraI64, ora.OraS, ora.OraI64, ora.OraS, ora.OraS, ora.OraI64)
defer stmt.Close()
if err != nil {
log.Printf("Prepare statement failed; error = %s\n", err.Error())
w.WriteHeader(500)
json.NewEncoder(w).Encode(JsonError{"Internal error"})
return
}
rst, err := stmt.Qry()
if err != nil {
	log.Printf("Execute failed; error = %s\n", err.Error())
	w.WriteHeader(500)
	json.NewEncoder(w).Encode(JsonError{"Internal error"})
	return
}

results := make([]EmneInfo, 0)
for rst.Next() {
	results = append(results, EmneInfo{rst.Row[0].(ora.Int64), rst.Row[1].(ora.String),
		rst.Row[2].(ora.String), rst.Row[3].(ora.String), rst.Row[4].(ora.String),
		rst.Row[5].(ora.String), rst.Row[6].(ora.Int64),
		rst.Row[7].(ora.String), rst.Row[8].(ora.Int64), rst.Row[9].(ora.String),
		rst.Row[10].(ora.String), rst.Row[11].(ora.Int64)})
}

index 850d84b..05ad2b8 100644
--- a/defLob.go
+++ b/defLob.go
@@ -242,7 +242,7 @@ func (lr *lobReader) Close() error {

 // Read into p, the next chunk.
 func (lr *lobReader) Read(p []byte) (n int, err error) {
-       if lr.ociLobLocator == nil {
+       if lr.ociLobLocator == nil || lr.Length == 0 {
                return 0, io.EOF
        }
        defer func() {

It's on v4.

You have already patched the code? ;)

The symptoms said "infinite loop", and you dogged on and presented nice informative description, even provided a test case, so I felt this should be rewarded by an instant fix :)
So I've expanded the already existing TestLOBRead with an empty case, and in the infinite loop hit Ctrl+\ and saw it's stuck in defLob with OCILobRead2.
So added the length check and this fixed the test case, so profit.

Thank You very much, Sir! :) Very good work!

Excuse me, Sir
Still have the same problem.  Am I doing something wrong?
I have changed the type to:
titusjonsNr ora.Int64 json:"institusjonsnr"
EmneKode ora.String json:"emnekode"
VersjonsKode ora.String json:"versjonskode"
InfoTypeKode ora.String json:"infotypekode"
SprakKode ora.String json:"sprakkode"
TerminKodeFra ora.String json:"terminkode_fra"
ArstallFra ora.Int64 json:"arstall_fra"
TerminKodeTil ora.String json:"terminkode_til"
ArstallTil ora.Int64 json:"arstall_til"
InfoTekst ora.Lob json:"infotekst"
InfoTekstOriginal ora.Lob json:"infotekst_original"
InstitusjonsNrEier ora.Int64 json:"institusjonsnr_eier"
}
Prepare with:
stmt, err := ses.Prep(sqlQuery, ora.OraI64, ora.OraS, ora.OraS, ora.OraS, ora.OraS, ora.OraS,
ora.OraI64, ora.OraS, ora.OraI64, ora.OraS, ora.OraS, ora.OraI64)
And get with:
rst, err := stmt.Qry()
results := make([]EmneInfo, 0)
for rst.Next() {
results = append(results, EmneInfo{rst.Row[0].(ora.Int64), rst.Row[1].(ora.String),
rst.Row[2].(ora.String), rst.Row[3].(ora.String), rst.Row[4].(ora.String),
rst.Row[5].(ora.String), rst.Row[6].(ora.Int64),
rst.Row[7].(ora.String), rst.Row[8].(ora.Int64), rst.Row[9].(ora.Lob),
rst.Row[10].(ora.Lob), rst.Row[11].(ora.Int64)})
}
Does the prepare make troubles since I use ora.OraS for clobs?

cd $GOPATH/src/gopkg.in/rana/ora.v4
git checkout -f master
git pull

then please modify TestLobSelectString or add a new test case which reproduces your problem!
I admit I haven't tested your exact case yet, I'll try it later.

PLEASE DELETE ME FROM THIS THREAD, I AM NOT SURE HOW I GOT HERE

The easiest way is clicking on the "mute the thread" link at the bottom of
the email.

Reply to this email directly, view it on GitHub
https://github.com/rana/ora/issues/156#issuecomment-281104599, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPoSn_cjC3iwyeMchwbNUMoxRbsJdR0ks5rea3ogaJpZM4MFha-
.

Check out z_lob_test.go, TestLobIssue156 (v4 branch), and

use ora.D in Prep if you want to receive Lob (and ora.S for string),
ensure that ses.Cfg().Clob == ora.D (use ses.SetCfg(ses.Cfg().SetClob(ora.D)))

Sorry for the second point, D is the default value, in which case the general config is used.
Maybe if we'd have a L (Lob) type, that'd made things easier, I don't know.

Thank You, Sir!
My program does not crash with the latest fix, if I use ses.SetCfg(ses.Cfg().SetClob(ora.D)), ora.D in Prep and ora.Lob in the type,- but I  cannot marshal my object to JSON and I need that.
If I use ses.SetCfg(ses.Cfg().SetClob(ora.D)), ora.S in the Prep and string in type, the whole program crashes with index out of range.
Have I misunderstood something?

I've pushed v4.0.1 with ora.L as a Lob column type.
Check out TestLobIssue156 in z_lob_test.go.
If still panics, please post the code and the full stack trace!

Thank You again, Sir!
I have done as You advised.
I have removed:
ses.SetCfg(ses.Cfg().SetClob(ora.D))
Use ora.L in the Prep and ora.Lob in type
When I marshal to JSON, I get:
...
{
"institusjonsnr": 194,
"emnekode": "TBBY3001",
"versjonskode": "A",
"infotypekode": "E-URL",
"sprakkode": "BOKMÅL",
"terminkode_fra": "HØST",
"arstall_fra": 2017,
"terminkode_til": null,
"arstall_til": null,
"infotekst": {
"Reader": {
"Length": 31
},
"Closer": null,
"C": false
},
"infotekst_original": {
"Reader": null,
"Closer": null,
"C": false
},
"institusjonsnr_eier": 194
}
...
What is not quite what I expected.  ;)

There is a MarshalJSON/UnmarshalJSON defined on *Lob, but that's not Lob.
Please re-download 4.0.1
Trond Kandal<EMAIL_ADDRESS>ezt írta (időpont: 2017. febr. 22.,
Sze, 20:43):

Thank You again, Sir!
I have done as You advised.
I have removed:
ses.SetCfg(ses.Cfg().SetClob(ora.D))
Use ora.L in the Prep and ora.Lob in type
When I marshal to JSON, I get:
...
{
"institusjonsnr": 194,
"emnekode": "TBBY3001",
"versjonskode": "A",
"infotypekode": "E-URL",
"sprakkode": "BOKMÅL",
"terminkode_fra": "HØST",
"arstall_fra": 2017,
"terminkode_til": null,
"arstall_til": null,
"infotekst": {
"Reader": {
"Length": 31
},
"Closer": null,
"C": false
},
"infotekst_original": {
"Reader": null,
"Closer": null,
"C": false
},
"institusjonsnr_eier": 194
}
...
What is not quite what I expected. ;)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rana/ora/issues/156#issuecomment-281780224, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPoSuptZC60Pb8VAgiaTQBA_IYqbGjwks5rfI_WgaJpZM4MFha-
.

Thank You very much, Sir! It works like charm!  I very much appreciate your work and help!


Node NPM Cannot find module 'npmlog'

After installing node.js by typing:
scoop install nodejs

I get the following error when trying to run NPM:
module.js:328
    throw err;
    ^

Error: Cannot find module 'npmlog'
    at Function.Module._resolveFilename (module.js:326:15)
    at Function.Module._load (module.js:277:25)
    at Module.require (module.js:354:17)
    at require (internal/module.js:12:17)
    at C:\Users\rpolo\AppData\Local\scoop\apps\nodejs\5.3.0\nodejs\node_modules\npm\bin\npm-cli.js:20:13
    at Object.<anonymous> (C:\Users\rpolo\AppData\Local\scoop\apps\nodejs\5.3.0\nodejs\node_modules\npm\bin\npm-cli.js:76:3)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)

Can confirm that I have the same issue. Using the
scoop install nodejs4

command works fine but gives v4 instead of v5.

I had the same issue around the same time, but with the current releases it seems to be resolved. I did some digging and this is more of an NPM issue than a scoop issue. This error is common when NPM can't update itself.

Got this issue today:
=== Logging started: 11/11/2016  23:55:44 ===
Error 1320. The specified path is too long: C:\Users\felipe\AppData\Local\scoop\apps\nodejs\7.1.0\_scoop_extract\nodejs\node_modules\npm\node_modules\npm-registry-client\node_modules\npmlog\node_modules\gauge\node_modules\string-width\node_modules\is-fullwidth-code-point\node_modules\number-is-nan
=== Logging stopped: 11/11/2016  23:56:30 ===


Remove pkg_resources imports and use packaging instead

📝 Description
The pkg_resources module is deprecated and does not export the required symbols in newer versions. It is advised to use packaging or importlib instead. Because we only rely on functionality present in packaging, the change from pkg_resources to packaging should be straightforward.
✨ Changes
Select what type of change your PR is:

[ x ] 🐞 Bug fix (non-breaking change which fixes an issue)
[ ] 🔨 Refactor (non-breaking change which refactors the code base)
[ ] 🚀 New feature (non-breaking change which adds functionality)
[ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] 📚 Documentation update
[ ] 🔒 Security update

✅ Checklist
Before you submit your pull request, please make sure you have completed the following steps:

[ x ] 📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
[ - ] 📚 I have made the necessary updates to the documentation (if applicable).
[ - ] 🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

Our CI is currently down. Once fixed, we'll prioritise this PR


Autosave would be nice (so I don't need to push save for each item)

This is an important MVP feature for usability. Need to make sure that the comments & status entered for a given checklist item are not lost when the user clicks on the next item in the list.

This is an important MVP feature for usability. Need to make sure that the comments & status entered for a given checklist item are not lost when the user clicks on the next item in the list.


[US-1013585] Add samples to TOC

Exactly what it says it does on the cover.

@sptramer,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-conceptual/TOC.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-autogen/TOC.md
:white_check_mark:Succeeded
View

docs-ref-conceptual/TOC.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

#sign-off

Reviewed and links are in good order, merging.


Refactor scheduled CVE scanning of our released images

Why it needs to get done
We have several image scanning CI runs with some overlap:

some rocks repos do daily scanning (ex: seldonio-rocks).  In this workflow we rebuild and scan the rocks.
bundle-kubeflow does daily scanning of images for some charmed kubeflow releases

We should define where periodic scanning should occur and apply that consistently across our repos so that we can provide a clear CVE scanning message.
What needs to get done
One proposed solution is to remove the daily scanning from the rocks repos and keep the bundle-kubeflow daily scanning.  This would keep visibility on CVE progress in our released images in one central place
When is the task considered done
When a clear image scanning procedure is defined and implemented

If we keep the per-repo scheduled job, we should revisit the workflow design.  The save-space job in the existing workflow doesn't do anything.  Because it runs on a separate job it is in its own vm, so what happens is:

save-space starts.  That vm has some files deleted.
save-space ends and that VM is discarded
build-scan-rocks executes on a new vm

Also, regarding scanning in rocks repos like seldonio-rocks,
I think we should not build and scan since building may result in a different image than the one we 've published. We should instead scan published images, like what we do in bundle-kubeflow.


Support checking loops without spinning off a separate query

Currently, Verus always spins off a separate SMT query to verify a loop.  This pull request adds the option of not spinning off a query, and instead verifying the loop within the original query.  Both approaches are useful in different situations:

Verifying the loop in the original query allows the loop verification to share the context with the original query, so that fewer loop invariants are needed.  However, the query may become slow for medium and large functions.
Spinning off a separate query gives the loop its own private context, allowing for faster, more modular verification.  However, more loop invariants may be needed to share information between the outer query and the loop query.

In general, verifying the loop in the original query is more convenient for small functions, while spinning off a separate query is more practical for medium and large functions.
A new attribute verifier::spinoff_loop controls which of these two options is chosen, with verifier::spinoff_loop and verifier::spinoff_loop(true) turning on the separate query and verifier::spinoff_loop(false) turning it off.  The default behavior is still to spin off a separate query, so that by default programmers don't run into trouble with timeouts.  However, this default can be changed by placing verifier::spinoff_loop(false) on a function, a module, or the entire crate.  For example, if someone wants the default to be verifier::spinoff_loop(false) for the whole crate, they can add the #![verifier::spinoff_loop(false)] attribute to the root of the crate, and then if needed use verifier::spinoff_loop(true) on individual modules, functions, or loops.

Aawww yiss, livin' the dream. This is great. Thank you. I just fought this pain for the last time. In the function I'm working on, I could remove (count 'em) four obvious invariants. I look forward to never having to stumble through that again.

Naming suggestion:
[verifier::spinoff_prover] at the function level, from a user perspective, doesn't change anything about what I "have to do", it just changes the performance of the verifier (and thus can tickle stability issues).
In the loop context, spinoff now has a very different meaning: context is being explicitly forgotten where otherwise (inside a function) context generally accumulates.
For the while loops, I'd suggest an attribute name that says more about how it affects the user, rather than how it's implemented. Perhaps isolate_loop?

Note that the name ended up being verifier::loop_isolation(false) (rather than verifier::spinoff_loop(false)).

Hah! That explains why my crate-level use of spinoff_loop(false) feature stopped working, driving me batty for a while. :vD


[FR] - Increase network throughput

Internal/External
Internal
Area
Other   Any other topic (Delegation, Ranking, ...).
Note: this is partially related to cardano-node, partially related to ouroboros-network
Progress update
It's been a few months since this issue was created, so if you're reading this after-the-fact, you may be interested in what progress was made since this issue was raised. Here is the progress so far

[x] add ability for nodes to query mempool state #3413
[x] add ability for nodes to query pending txs #3404
[x] blockchain load is now tracked so people can monitor when congestion occurs link
[x] On epoch 306 (December 1, 2021), block size was increased by 8kb (64kb->72kb - 12.5% increase)
[x] On epoch 306 (December 1, 2021), Plutus script memory units per transaction by 1.4 million (9.85->11.25 - 12.5% increase)
[x] Peer-to-peer was launched on testnet link - mainnet date not decided
[x] Milkomeda public launch on testnet - public mainnet for Q1
[ ] Improve UTXO parallelizability with reference inputs CIP-31 - scheduled for the next hardfork
[ ] Reduce block space taken my smart contract txs by introducing reference scripts CIP-33 - scheduled for the next hardfork
[ ] IOHK announced they are working on "input endorsers" for scalability (still not much public detail on this other than old documentation about the idea from a few years ago)
[ ] Hydra testnet launch (scheduled for Q1)
[ ] Milkomeda will introduce rollups (no date announced)
[ ] Rough plans for tiered network traffic blog post
[ ] Work starting on pipelining (coupling validation & block validation to speed it up) to improve block propagation times

Personal Note
Hello! If you're reading this from Twitter / Reddit and you want more technical content, check out my personal twitter or the twitter account of my company dcspark. We do more than just Cardano -- we're also involved in Solana and Urbit as well!
Background
Currently the Cardano network has hit 100% capacity at least twice. That means that mempools are full and this is causing tooling-dependent issues
You can find the latest 1hr & 24hr loads from pool.pm
Is this related to the "Congestion issue" I keep hearing about?
No. The "congestion" issue you may have heard about in the past few weeks is about how Cardano smart contracts are deterministic and so only one transaction can consume a utxo. It can be tackled by re-designing smart contracts to work better in parallel across multiple UTXO entries
This issue is about how Cardano itself has reached the max tps the network can support
How is the mempool handled currently?
The Cardano mempool is designed to be "fair". Transactions are processed in a FIFO order regardless or how much in fees they pay (the ledger spec does support a fee market, but cardano-node doesn't take this into account)
Cardano-node uses a fixed-size mempool is currently only 2 blocks in size (~280 txs). If transactions are submitted when the mempool is full, they may be dropped (this is partially tool-dependent. The cardano-node just holds agency with txsubmit protocol until there is room)
What is the max throughput for Cardano currently
Cardano tps is between 0.2tps to max 6.5tps
Cardano block time is ~20seconds
Cardano is currently hitting the maximum due to NFT drops and so the tps is usually on the higher end of that range (since Plutus script execution is not to common currently)
Can we increase tx size and/or block size?
Unfortunately not really. Plutus is still being optimized and increasing the block size could cause Plutus script execution to take too long and cause downstream problems with the network.
That being said, currently Plutus script are not only expensive in execution, but also large in size (a single script is 5-8kbs and so a single transaction in Cardano can usually only actually use 1-2 scripts before you hit the limit). This means increasing the tx size limit could allow larger batching of NFT drops while still not allowing for much extra Plutus computation (until Plutus gets optimized in the future at least)
Can we increase the mempool size?
I don't know about why 2 blocks was picked as the fixed size. Would appreciate any background info on this. Possibly this was just to try and get Cardano to stay under 8GBs of RAM?
Additionally, this would degrade the user experience for light wallets because currently light wallets have no way of querying the mempool so they can't show pending transactions. If we keep a fat mempool, we would need to prioritize mempool querying for light wallets as a feature
Can we tackle this issue at the wallet level?
To a certain extent. Wallets can implement tx resubmission so that even if the mempool being full caused the tx to be dropped initially, the wallet will just resend it for the user.
Currently there is no way of querying the node's mempool for light wallet tooling which means a wallet can't know if a tx that was submitted got dropped, failed or is still pending. This makes it tricky to implement proper tx resubmission since it means the only way to know if you should resubmit is by seeing if the ttl for your tx has expired. Lowering the ttl is not ideal either because if network congestion makes it longer for a tx to go through, decreasing the ttl lowers the chance for your tx to make it into a block before it times out.
Can we sort the mempool?
One way we can alleviate the problem without increasing the tx and/or block size is to sort transactions in the mempool such as using a fee market.
This is non-trivial to do correctly because

Transactions in the mempool could depend on each other (do you replace 2 txs with a 1tx that pays a higher fee than both combined?)
If mempool sorting takes too long, it can affect the tps of the system negatively
It allows attack vectors related to the mempool (ex: submitting txs that are only 1 lovelace better to include in the transaction repeatedly to increase computation costs. Other blockchains have suffered this kind of attacks in the past). You would need the fee gained from including the tx to be higher than the computation cost of re-sorting your mempool (but it's hard to calculate the lovelace cost of this)
It makes debugging of mempool issues harder since it's no longer as predictable
The network layer code wasn't written with sorting the mempool in mind and therefore would require a fair amount of development work and iteration on the networking models
It slightly complicated babel fees

Sorting methods
The easiest field for sorting the mempool is the ratio of tx fee / cost of tx.
Another option is sorting by the age of oldest UTXO entry in the transaction. This prevents spam attacks (because they will run out of old UTXOs eventually), but will have terrible consequences on usage of popular smart contracts as all their UTXOs will be relatively fresh
Can we prioritize transactions without having to make modifications to the mempool?
Yes, to a limited extend we can implement transaction sorting to choose which transactions to pass to your peers (so they their mempools are better than yours). This requires P2P which Cardano doesn't have yet, but can help in the future.
This would also be slightly flexible because you could choose to accept transactions from your certain peers even if they don't provide you the best tx fee return if you have some special agreement with them.
Can we rely on Hydra or Milkomeda?
One popular sentiment is that L2 solutions like Hydra or Milkomeda can help us.
Milkomeda isn't meant as a throughput solution -- it's an interoperability solution so it won't help with this.
Hydra head is still in development so we need a solution sooner than later. Additionally, the first version of Hydra deployed will be fairly limited (you need to open a state channel with a fixed set of actors, the actors all have to be online 24/7, you can't add or remove assets from the channel other than closing the channel entirely). Given this set of limitations, we can't relay on Hydra head v1 to fully alleviate the problem. It's possible to have non-custodial head-as-a-service solutions to bypass some of these limitations (but nobody is working on this as far as I know) and some future iterations of Hydra improve or get rid of these restrictions.
Even if we have Hydra, it still doesn't solve the problem entirely because states still have to be opened & closed on the main chain eventually.
Is a fee market avoidable?
No. Blockchains have always fundamentally work on a "pay up or shut up" system. If there is network congestion, the only objective way to decide who to allow through is to prioritize the transactions with the most financial value which is proxy-defined by how much in fees they are willing to pay for the transaction to go through.
Additionally, a fee market is something all Cardano users should want. Cardano has a fixed supply and so stake pool returns will keep going down (it's already below 5% return on staking), but the good news is that transaction fees are distributed to all delegators. That means the only way for people to have an incentive to continue delegating is for tx fees to increase until they eventually surpass rewards from the reserve
You can read IOG's thoughts on fees & tiered network traffic here
What does Ethereum look like?
Other blockchains like Ethereum take full advantage of the fee market (you can see live information here) and wallets look at the state of the network to know how much in tx fees they have to pay.

Ethereum does around 10~15 tps
Ethereum block time is ~13seconds

Currently transaction fees on Ethereum actually provides more to miners than the base block rewards (this is despite the fact that in Ethereum, unlike Cardano, block rewards do not go down overtime algorithmically)
Related tickets

Improve cardano-wallet handling of congestion
Have wallets implement re-submission and consider reducing TTL
Reach out to NFT tools and ask them to space out token minting

At the moment the spikes in usage have been caused via NFT drops - while I agree there needs to be a preferential treatment in mempool, I am not very keen to see simply a fee-based queue.
Having seen some (not all) of the drop mechanism, they use pre-deposits from users , which is used to cater for minimum UTxO + fees + margin.
Thus, this simply means the impact of sorting it on fees will mean users who want to send simple ADA tx will end up paying having to pay very high fees during spike, while the drops will simply continue to use very high fees specified for tx - as those are from user deposits.
Also - while doing so, the change also onboards the complications and risks as you highlighted. I almost feel the handling of queue - if not FIFO - needs to be more mature in differentiating potentially actual load versus drops, given that latter could potentially cause DDoS if based on fees using funds that's kinda borrowed).
Also, the increase in tx handling will cause an increased impact on resources (bandwidth, processing, and downstream to components like dbsync/explorers), such a change should be taken up with very careful research indeed.

@rdlrt I agree it's definitely far from ideal. The point of this issue was mostly to try and have a publicly-visible summary of the state of things that can be shared and discussed as opposed to proposing a specific concrete solution.

There are a few misunderstandings here.
Firstly, note that the system is not congested as a whole. There are simply periods when some users (e.g. doing NFT) drops have unbounded demand. That is not in itself a problem. It is fine for an NFT drop to take a while to complete.
What would be a problem is if "ordinary" users (i.e. not those doing the NFT drops) are finding that they face unacceptably long times to get their txs included onto the chain.
If that was the problem, note that increasing the system capacity would not solve it. To see why consider a hypothetical example: suppose the NFT drop takes 2 hours at the current system capacity, and otherwise normally the system is 30% loaded. Then doubling the system capacity would simply reduce the time for the NFT drop from 2 hours to 1 hours, and make the system normally 15% loaded. So all it would do is shorten the time duration in which "ordinary" users face unacceptably long confirmation times.
If this were a problem then the solution is not capacity but priority. The NFT drop can be the lowest priority since it is fine if it takes longer to complete.
Note that the system already has some natural fairness so that we do not expect the NFT drops to crowd out ordinary users. We have not yet seen any evidence that this is the case. If we do see such evidence, there are further measures we can take with priorities so that typical user transactions will get priority over single users with unbounded demand (like large NFT drops).

For users, this is a pain point (txs submitted through a wallet won't go through with no error message given)

This is not really the case at the level of the node. Submitting txs to a relay node can block until there is capacity in the mempool. This means there is notification.

for people doing NFT drops it is disastrous (a random percentage of their drop randomly doesn't go through and they have to re-submit the txs that didn't make it).

This is conflating two unrelated issues. The first issue is the same as for ordinary wallets. Submitting a transaction can induce back-pressure. This is a feature not a bug. The submitting agent does not need to do anything special to handle this: just keep submitting transactions. As and when there is space, the protocol will make forward progress.
The second issue is transactions that are successfully submitted to a local node or relay but that never make it to the chain. The system as a whole has never guaranteed the reliability of transaction submission. It is impossible to provide such a guarantee in a distributed system like this. As such, submitting agents have always been required to handle re-submission logic in an appropriate way. The TTL mechanism is provided as a way to know for sure when it is safe to re-submit a transaction.

Cardano-node uses a fixed-size mempool is currently only 2 blocks in size (~280 txs). If transactions are submitted when the mempool is full, they may be dropped (this is partially tool-dependent. The cardano-node just holds agency with txsubmit protocol until there is room)

No, they are not dropped. They are not accepted until there is room. This is the back-pressure.

To a certain extent. Wallets can implement tx resubmission so that even if the mempool being full caused the tx to be dropped initially, the wallet will just resend it for the user.

The txs are not dropped. This is a misunderstanding. They are simply not accepted yet. If you keep trying to submit then it will eventually be accepted.
But yes, totally separately to that point. It is possible for txs to fail to make it to the chain, and re-submssion is a necessary feature anyway, but it is not specifically related to congestion.

I don't know about why 2 blocks was picked as the fixed size. Would appreciate any background info on this. Possibly this was just to try and get Cardano to stay under 8GBs of RAM?

The mempool is as big as needed to ensure throughput, but no larger. Bigger buffers just create more work and bigger latency. The system is designed not to buffer but to put back-pressure on the edges of the system. This is better than using big buffers since it provides better feedback, and uses less resources to manage pending work. See also "buffer bloat" for a discussion on this in networking in general.

Can we prioritize transactions

Yes. As I say above, the issue with NFT drops is that it's a period of essentially infinite demand. Increasing capacity does not change that. The solution is to prioritise. We already do that to some extent. There are ways to do it more systematically and on more criteria.
The way to do it is not to reorder the mempool. The way to do it is to pick which txs from immediate peers to accept into the mempool in the first place. Here are some of the criteria we would probably want to use.
Characteristics/behaviour of the peer

At the level of a node in the graph that is requesting txs from its immediate peers: make fair (weighted) random selections of the txs from the immediate peers when there is space in the mempool to fill. This is a kind of bandwidth sharing.
Another scheduling trick is to prioritise the peer that has recently used least of the resource.
Give lower weighting to peers that have only very recently connected. Equivalently this prioritises peers that have established connections for a long time.
Prioritise/weight based on known peers: we sometimes have peers we know about from local config, which correspond to other nodes we control, or nodes we have peering arrangements with.
Prioritise/weight based on the stake of the remote peer, which prioritises the peers of other SPO relays over anonymous 3rd parties

Characteristics of the transaction

Age of the oldest UTxO entry.
Total value transferred in the transaction.

And finally

Fee multiplier vs min fee.

Note that we already do 1 to some extent. That's the current pseudo-random "fairness". And also note that there's lots of things to consider prioritising on before we get to a fee market.
If we see evidence that fairness is a problem with these NFT drops then we can do more of 1--5 and that should solve the problem for NFT drops and similar spikes from individual users.
When the system is more generally loaded from continuous demand rather than these spikes then it makes sense to increase capacity incrementally.
Summary

We need to make sure wallets and other submitting agents are dealing properly with back-pressure in tx submission.
We need to make sure wallets are properly handling re-submission for cases where txs are accepted but never make it to the chain.
We need to look for evidence of unfairness in tx submission (i.e. NFT drops actually crowding out normal users) and if so then plan to do some more of the prioritisation features.

We need to make sure wallets and other submitting agents are dealing properly with back-pressure in tx submission

Without triggering this, is there a way I can confirm what cardano-submit-api's behaviour is here? Assuming my HTTP Client will wait indefinitely, would cardano-submit-api's web server also wait, or will it time my request out if the node blocks waiting to take the transaction?
I tried reviewing the cardano-submit-api source for clues but couldn't find anything about timeouts. It seems like it may be using a package named Servant, but I also couldn't find anything about timeouts in their docs or repo that seemed to indicate if there is a default timeout.

@DanTup the behaviour should be that the HTTP server side will block without any timeout. I say "should" because I've also not verified that the HTTP server doesn't have a timeout. I don't think it should have any, but I've not verified it, and you're right that we should. As you say, one thing to watch out for is timeouts in your HTTP client: you want to avoid that.
So when using the submit api, the behaviour we want (both of the api and the client using it) is indeed simply to block until the node has space and gets back to us and accepts our tx.
My advice for applications doing large NFT drops would be to use the Daedalus wallet backend via its API, rather than the cardano-submit-api. The submit API does what it does ok, but it's not a wallet. It does not track pending txs. It does not resubmit, and all agents have to have some capacity to resubmit (even if that means kicking it back to a human to decide). If you're using the submit API then you must implement your own wallet functionality such as:

making txs with appropriate TTLs so you know when txs have definitely failed to make it to the chain and so can be resubmitted
actually resubmitting txs (which would involve re-signing if you use the policy of waiting for the TTL)
tracking the pending/in-flight set of UTxOs (so we don't re-use UTxOs that are being spent by earlier txs, but make them available again for failed txs)

cc @newhoggy @Jimbo4350 we should verify this blocking behaviour in the REST API.

So when using the submit api, the behaviour we want (both of the api and the client using it) is indeed simply to block until the node has space and gets back to us and accepts our tx.

I think this should also be very clear in the documentation (for the API endpoint for the web API, and the description for cardano-cli transaction submit for the CLI) that this is the behaviour as a prompt/reminder for anyone using them to consider this case. And if the results if your client ties out (or you hit Ctrl+C in the CLI) is not deterministic (eg. you might cancel right as the tx is processed, or it might not be accepted and probably would therefore be dropped), I think it's worth calling that out too. Docs should help guide people into writing robust software and they might not always be aware of infrequent edge cases like this.

My advice for applications doing large NFT drops would be to use the Daedalus wallet backend via its API, rather than the cardano-submit-api. The submit API does what it does ok, but it's not a wallet.

Assuming you mean cardano-wallet, I tried to use that first. I spent many weekends discovering that it could not mint (it has APIs that just throws as not implemented?), and it's submit-proxy API always rejected transactions I built locally (saying they weren't encoded correctly - there's little documentation about this and questions about this in the forums were all responded to with "don't use that API, it's not for you").
cardano-submit-api was my very last choice, and the only one I was able to make work 🤷♂

making txs with appropriate TTLs so you know when txs have definitely failed to make it to the chain and so can be resubmitted

If you use a time-locked policy, you have to use an exact matching TTL on the transaction (at least, that seemed to be my experience), so I'm not sure this works. My plan is just to keep the txs around on disk as a backup, so I can periodically re-submit them if required (rebuilding the txs would be complicated because to increase throughput I spend utxos from un-confirmed transactions.. resubmitting the same txs is therfore much simpler).

I agree this ought to be clear in the docs.

And if the results if your client ties out (or you hit Ctrl+C in the CLI) is not deterministic (eg. you might cancel right as the tx is processed, or it might not be accepted and probably would therefore be dropped), I think it's worth calling that out too.

It's worth noting here that application authors that care about this may want to use the tx submission protocol at a lower level where they can in fact resolve that race condition. At the lower level, if you talk directly to a relay, the way it works is like this:

The relay asks you for up to N TxIds that you might wish to send.
You reply with up to N TxIds
The relay then can ask for the full Tx for any of the TxIds that you told it about.
You reply with the requested Txs.

The relay will only do step 3 and ask for the full Tx once there is space in its mempool. So in the state in which you're blocking waiting to submit, we do in fact know that there's no danger of loosing track of whether a tx was submitted or not. It is only once the relay asks for and we reply with the full Tx that we've really fired the tx off into the system.
Now we don't currently make it very easy to interact with the node-to-node tx submission protocol, but if we all conclude that would be useful then that might be the way to go.
As for your frustration with the wallet backend, I take your point. It has the right basic infrastructure to do well in this use case, but yes it would need to have the right APIs to support it.

If you use a time-locked policy, you have to use an exact matching TTL on the transaction (at least, that seemed to be my experience), so I'm not sure this works.

Ah, fortunately that's not the case. The rule there is simply that the validity interval of the tx has to be within the time required by the script. Suppose you have a time lock script that says "the time must be before the 1st of October and signed with key X or after 1st October and signed with key Y". Then you can submit a tx signed with key X with a validity interval that ends well before the deadline. It does not need to extend to the deadline exactly. And the same goes for the other end of the validity interval, for requiring the time be after some point.
So yes you can still use TTLs for knowing txs didn't make it, and also use time locks. You can verify this for yourself in the Allegra ledger specification and in practice on the testnet.

My plan is just to keep the txs around on disk as a backup, so I can periodically re-submit them if required (rebuilding the txs would be complicated because to increase throughput I spend utxos from un-confirmed transactions.. resubmitting the same txs is therfore much simpler).

That's a perfectly reasonable policy too.

Why not just increase the mempool? It's extremely tiny at this moment and 99% of time cardano network is almost idle. These spikes would be easily handled if mempool was bigger, most of blocks that follow the spike are near 0 txs, so this way it would just redistribute into following blocks and users wouldn't need to resubmit anything.

really this problem is so easy and trivial to fix that I don't understand why there is such a lengthy complicated discussion about it :)

@KtorZ is implementing mempool query API over here, which is relevant to this discussion...

Neil Burgess here, I am working on CAD-3504 to improve the docs. I want to create a best practices document for API users submitting transactions to the Cardano network.
I gather
cardano-submit-api is the correct endpoint to address.
When a transaction is submitted, the submitter should block until it is accepted before submitting another one.
Recommended method of confirming acceptance?

I have prepared a document that summarizes my understanding of this discussion. Please review and comment. All of Input Output has editor access.

@dcoutts
I think this can be a part of the solution:
If anybody is having questions why, feel free to ask.
--- a/ouroboros-consensus/src/Ouroboros/Consensus/Mempool/Impl/Pure.hs
+++ b/ouroboros-consensus/src/Ouroboros/Consensus/Mempool/Impl/Pure.hs
@@ -130,7 +130,10 @@ pureTryAddTxs
 pureTryAddTxs cfg txSize wti tx is
   | let size    = txSize tx
         curSize = msNumBytes  $ isMempoolSize is
-  , curSize + size > getMempoolCapacityBytes (isCapacity is)
+        capacity = getMempoolCapacityBytes (isCapacity is)
+  , case wti of
+      Intervene -> curSize + size > (3 * capacity `div` 2)
+      DoNotIntervene -> curSize + size > capacity
   = NoSpaceLeft
   | otherwise
   = case eVtx of

Submitting txs to a relay node can block until there is capacity in the mempool

Can anyone tell me what the ordering of accepting new txs into the mempool is? If a mempool is full and two peers want to send transactions, are they always accepted in-order? If the mempool frees up 10kb and the first peer wants to give a 15kb tx but the second peer has a 5kb tx, will the node wait until it has enough space to take the first tx, or will it accept the second smaller one to full up the space?
I assumed they were taken in order, but some of the descriptions I've seen recently - like the image shown at https://youtu.be/EGwJCiMy4_E?t=125 seem to suggest that smaller txs can jump the queue while large ones are held back.

@dcoutts could you elaborate on how the following can be prioritized ideally:

Characteristics of the transaction
Age of the oldest UTxO entry.
Total value transferred in the transaction.

There has been some changes here right?
I mean except peer to peer and hydra all of the above points should be implemented?


Simplify/unify the config format of workspaces and repositories

Hi,
I've been using tab for a good while now, and I find it really useful. Recently however, I realised I haven't really been using the workspaces feature at all, and reading the related docs left me with some doubts (maybe it's just a matter of nomenclature), so I thought I'd ask before suggesting any changes.
When defining a workspace, it'll be available to the subdirectories of the directory where it was defined, however, the examples include child workspaces and a similar mechanism for repositories. With that in mind:

Is there any advantage to having nested tab.ymls instead of adding the relevant metadata to the root/parent workspace?

What's the advantage of having a separate system for repositories? Doesn't having a tab.yml in the root of the repo work the same?

Is there any reason tab uses tab.yml instead of a hidden .tab.yml?

Once again, thank you for all the work you've put into this project!

Hey @savq,
Workspaces help when you have a ton of tab configurations, and you want to keep tabs visible within the workspace directory.  Maybe the best way to explain this is an example of how I use it.  Here is my ~/open-source/tab.yml workspace config:
doc: "open source repositories"

workspace:
  - repo: lifeline-rs
  - repo: tab-rs
  - repo: postage-rs
  - repo: tiwi-rs
  - repo: not-exists
  - repo: homebrew-taps
  - repo: this-week-in-rust
  - tab: bash
    shell: bash
  - tab: zsh
    shell: zsh
  - tab: screen
  - tab: tmux
  - tab: any
    doc: "random tasks"

And in ~/.config/tab.yml, I have:
workspace:
  - workspace: open-source
  - workspace: other-repos

tab -l from my home directory is nice and clean:
❯ tab -l
Available tabs:
    austinjones/    (austin's home directory)
    open-source/    (open source repositories)
    other-repos/      (my other repositories at ~/other-repos)

Due to the link in ~/.config/tab.yml, I can always activate the open-source workspace on the command-line or fuzzy finder:
❯ tab open-source/
super-terminal with alacrity, fish, powerline, `tab` multiplexer, and navi/zoxide.  try `help`

tab open-source/  ❯ tab --list
Available tabs:
    any/              (random tasks)
    austinjones/      (austin's home directory)
    bar/
    bash/
    foo/
    homebrew-taps/
    lifeline/         (lifeline-rs, a dependency injection library for async channels)
    open-source/      (open source repositories)
    postage-rs/
    screen/           (screen, the GNU terminal multiplexer)
    tab/              (the tab-rs terminal multiplexer)
    tmux/             (tmux, the next-gen terminal multiplexer)
    twir/             (this week in rust, the weekly rust newsletter)
    workspace/        (threatx repositories)
    zsh/

Repo configs are nice if you need a lot of tabs for a particular project.  project/git, project/run-frontend, project/run-backend, project/logs, etc.  And if you need to share config (shells/env), that can be configured for all the project tabs in one place.  Without a workspace, you'd have to cd into the project to get those tabs.  But workspaces 'pin' the tabs, even in sibling directories.
You can definitely define all your tabs at the workspace level, or even at ~/.config/tab.yml.  But child repo configs save repeated relative paths, and they are accessible in code editors when you have a project open.
I don't have a great use case for workspaces within workspaces - maybe someone with a lot more tabs than I do will find a use :D  The support for nested workspaces is how ~/.config/tab.yml works with ~/open-source though, so when I implemented that for the global config, I added a bit of documentation (because it does work).
Oh, and tab actually supports .tab.yml for local config files.  The global config would require setting $TAB_CONFIG though.

Hey @savq,
Workspaces help when you have a ton of tab configurations, and you want to keep tabs visible within the workspace directory.  Maybe the best way to explain this is an example of how I use it.  Here is my ~/open-source/tab.yml workspace config:
doc: "open source repositories"

workspace:
  - repo: lifeline-rs
  - repo: tab-rs
  - repo: postage-rs
  - repo: tiwi-rs
  - repo: not-exists
  - repo: homebrew-taps
  - repo: this-week-in-rust
  - tab: bash
    shell: bash
  - tab: zsh
    shell: zsh
  - tab: screen
  - tab: tmux
  - tab: any
    doc: "random tasks"

And in ~/.config/tab.yml, I have:
workspace:
  - workspace: open-source
  - workspace: other-repos

tab -l from my home directory is nice and clean:
❯ tab -l
Available tabs:
    austinjones/    (austin's home directory)
    open-source/    (open source repositories)
    other-repos/      (my other repositories at ~/other-repos)

Due to the link in ~/.config/tab.yml, I can always activate the open-source workspace on the command-line or fuzzy finder:
❯ tab open-source/
super-terminal with alacrity, fish, powerline, `tab` multiplexer, and navi/zoxide.  try `help`

tab open-source/  ❯ tab --list
Available tabs:
    any/              (random tasks)
    austinjones/      (austin's home directory)
    bar/
    bash/
    foo/
    homebrew-taps/
    lifeline/         (lifeline-rs, a dependency injection library for async channels)
    open-source/      (open source repositories)
    postage-rs/
    screen/           (screen, the GNU terminal multiplexer)
    tab/              (the tab-rs terminal multiplexer)
    tmux/             (tmux, the next-gen terminal multiplexer)
    twir/             (this week in rust, the weekly rust newsletter)
    workspace/        (threatx repositories)
    zsh/

Repo configs are nice if you need a lot of tabs for a particular project.  project/git, project/run-frontend, project/run-backend, project/logs, etc.  And if you need to share config (shells/env), that can be configured for all the project tabs in one place.  Without a workspace, you'd have to cd into the project to get those tabs.  But workspaces 'pin' the tabs, even in sibling directories.
You can definitely define all your tabs at the workspace level, or even at ~/.config/tab.yml.  But child repo configs save repeated relative paths, and they are accessible in code editors when you have a project open.
I don't have a great use case for workspaces within workspaces - maybe someone with a lot more tabs than I do will find a use :D  The support for nested workspaces is how ~/.config/tab.yml works with ~/open-source though, so when I implemented that for the global config, I added a bit of documentation (because it does work).
Oh, and tab actually supports .tab.yml for local config files.  The global config would require setting $TAB_CONFIG though.

Also, thanks so much for using tab!  If you have any thoughts on how to make the config easier to use, I'd be really interested :)

Also, thanks so much for using tab!  If you have any thoughts on how to make the config easier to use, I'd be really interested :)

I recently found tab-rs and had this exact question.
I think the difference between repos and workspaces rather unclear. Why don't have only repos that can include other repos like that?
❯ tab -l
Available tabs:
    austinjones/    (austin's home directory)
    austinjones/open-source/    (open source repositories)
    austinjones/open-source/tab-rs
    austinjones/other-repos/      (my other repositories at ~/other-repos)

I recently found tab-rs and had this exact question.
I think the difference between repos and workspaces rather unclear. Why don't have only repos that can include other repos like that?
❯ tab -l
Available tabs:
    austinjones/    (austin's home directory)
    austinjones/open-source/    (open source repositories)
    austinjones/open-source/tab-rs
    austinjones/other-repos/      (my other repositories at ~/other-repos)

Interesting.  I'll rethink this and see if there is a way to unify the config formats.

Interesting.  I'll rethink this and see if there is a way to unify the config formats.

Hey @austinjones,
Your example does clear some things up, but @jeizsm comment makes me think this is a common source of confusion.
My first intution/draft proposal for a different config system would be:

1 tab.yml corresponds to 1 workspace
workspace == repo == bundle of tabs + docstring + env. variables

(remove the distinction between workspaces and repos)

tab = session + docstring + env. variables
tabs inherit env. variables from their workspace (unless overwritten)
links are allowed, but should be named link rather than workspace
workspaces can't overwrite their names (to make linking consistent)

The big difference is that workspaces would be abstract, i.e. any tab instace in the workspace has to be explicit (no default tab), which might not be how other people understand the idea of a workspace. So obviously it's not the best alternative right now.
Since this is mostly a design decision, rather than an implementation problem, I think that pinning it and letting other folks give their opinion would be really useful.

Hey @austinjones,
Your example does clear some things up, but @jeizsm comment makes me think this is a common source of confusion.
My first intution/draft proposal for a different config system would be:

1 tab.yml corresponds to 1 workspace
workspace == repo == bundle of tabs + docstring + env. variables

(remove the distinction between workspaces and repos)

tab = session + docstring + env. variables
tabs inherit env. variables from their workspace (unless overwritten)
links are allowed, but should be named link rather than workspace
workspaces can't overwrite their names (to make linking consistent)

The big difference is that workspaces would be abstract, i.e. any tab instace in the workspace has to be explicit (no default tab), which might not be how other people understand the idea of a workspace. So obviously it's not the best alternative right now.
Since this is mostly a design decision, rather than an implementation problem, I think that pinning it and letting other folks give their opinion would be really useful.

Hey @savq,
I like it.  I'm going to think about it a bit more.  And if anyone has thoughts, please chime in :)

Hey @savq,
I like it.  I'm going to think about it a bit more.  And if anyone has thoughts, please chime in :)

I found a way to make this change in a backwards-compatible way.  I'm thinking of including this in v0.6.0, and deprecating the old format in v0.7.0.

I found a way to make this change in a backwards-compatible way.  I'm thinking of including this in v0.6.0, and deprecating the old format in v0.7.0.


fix:solana: use durable nonce and rotate node connections

Signed-off-by: Andrei Lebedev<EMAIL_ADDRESS>What issue # is this merge request related to?
Closes #20
Overview of merge request
(Provide an overview of the merge request)
Proposed Changes
(What files does this add/remove/change, and how? Brief summary)
Additional information
(Please provide any additional information. For example, future considerations or information useful for reviewers.)

@gauthier-voron finally...

I'm testing it, for now with 4+1 c5.xlarge instances in a single region. It looks good.
I see the primary is doing a lot of work before the workload generation. Is it generating accounts at this time?
What is the reason for changes to DefaultTimeout and waitingTicker? Could it affect the results produced (and make previous results invalid)?

I'm testing it, for now with 4+1 c5.xlarge instances in a single region. It looks good.
I see the primary is doing a lot of work before the workload generation. Is it generating accounts at this time?
What is the reason for changes to DefaultTimeout and waitingTicker? Could it affect the results produced (and make previous results invalid)?

@gauthier-voron Yes, now it creates accounts for the nonces, because they have to be initialized anyway, so it does not help to generate them in the genesis block.
I broke waitingTicker in the initial Solana PR, and here I fixed it, because it should be in seconds. For timeout, I'd say it just does not hurt, but it should work without that change too! I can revert the default value.