Hugging Face Hub Integration Upgrades

Mainly makes the uploading and downloading of media files in push_to_hub() and load_from_hub() much more efficient, smart, and informative. For compatibility with Hugging Face repos, this PR creates a new dataset type in FiftyOne called FiftyOneChunkedDataset which allows you to split media files into subdirectories of specified size rather than putting all files directly into one data directory.
(Details)
Release Notes
Is this a user-facing change that should be mentioned in the release notes?

[ ] No. You can skip the rest of this section.
[ ] Yes. Give a description of this change to be included in the release
notes for FiftyOne users.

(Details in 1-2 sentences. You can just refer to another PR with a description
if this PR is part of a larger change.)
What areas of FiftyOne does this PR affect?

[ ] App: FiftyOne application changes
[ ] Build: Build and test infrastructure changes
[x] Core: Core fiftyone Python library changes
[x] Documentation: FiftyOne documentation changes
[x] Other

Summary by CodeRabbit

New Features

Introduced the FiftyOneChunkedDataset for representing datasets split into subdirectories, enhancing data management and accessibility.
Added export functionality for FiftyOneChunkedDataset, allowing datasets to be exported in chunks with customizable parameters.
Enhanced import capabilities to support the new chunked dataset format, including options for shuffling and limiting data samples.

Documentation

Updated user guides to detail the structure, usage, and interaction with the new FiftyOneChunkedDataset.

Refactor

Improved dataset export methods to handle different scenarios and support new chunking functionality.

Style

Standardized terminology in dataset handling functions to improve clarity and consistency.

How about instead just adding an optional chunk_size parameter to FiftyOneDatasetExporter that enables the batched data/ folder that you're after?
This way of exporting media would be equally applicable to all dataset formats, not just FiftyOneDataset. I'd just pass the chunk_size parameter through to the existing MediaExporter class so that any other data format could support nested subfolders just by passing this chunk_size parameter through to their MediaExporter instance as well.
As you noticed, FiftyOneDatasetImporter already "knows" how to import media regardless of how it is organized within data/, so it's really still "FiftyOneDataset" format. This also applies to all other builtin importers: the way they are generally implemented, they'll gracefully handle nested subdirectories too.

Good idea. Let me try this :)


NLTK Tokenizer not working in Latest Tag [ 3.9.1 ]

Hi,
I have been using NLTK version 3.8.1 for some time without any issues. However, after recently updating to version 3.9.1, I encountered an error when using the word_tokenize function. I would appreciate any assistance in resolving this issue.
Here is the code snippet that is causing the issue:
import nltk
nltk.download('punkt')
nltk.word_tokenize('over 25 years ago and 5^"w is her address')
This is the error I which is shown on execution:

Thank you for your support.

change to nltk.download('punkt_tab') see https://github.com/nltk/nltk/issues/3293


[CAMEL-18472] lock 'model' in 'removeRoute' first to avoid deadlock

Obtain the lock on model before locking camelContext in removeRoute to avoid a potential deadlock with concurrent addRoutes calls.

Let's wait for @davsclaus

The method disposeModel is used when in lightweight mode to clear the definition classes for runtimes where you do not dynamic add/remove routes.
So even if model is null then an exception would be thrown so that should be okay

LGTM


Finalized Version for Adding a Habit to Habit List

This PR will add the floating action button to DashboardFragment which will be used to open the dialog to add a new habit to our habit list. Verification of all user input is done with dialog before being sent back to dashboard to be added the the habit list, as well as to firebase. This PR closes #31.
Dialog:

Floating action button

Tried adding a habit, got this error. Checked DB, didn't add to DB. E/AndroidRuntime: FATAL EXCEPTION: grpc-okhttp-0 Process: com.example.habitsmasher, PID: 26441 java.lang.AssertionError: Method getAlpnSelectedProtocol not supported for object SSL socket over Socket[address=firestore.googleapis.com/<IP_ADDRESS>,port=443,localPort=41090] at io.grpc.okhttp.internal.OptionalMethod.invoke(OptionalMethod.java:114) at io.grpc.okhttp.internal.OptionalMethod.invokeWithoutCheckedException(OptionalMethod.java:135) at io.grpc.okhttp.OkHttpProtocolNegotiator$AndroidNegotiator.getSelectedProtocol(OkHttpProtocolNegotiator.java:183) at io.grpc.okhttp.OkHttpProtocolNegotiator$AndroidNegotiator.negotiate(OkHttpProtocolNegotiator.java:145) at io.grpc.okhttp.OkHttpTlsUpgrader.upgrade(OkHttpTlsUpgrader.java:63) at io.grpc.okhttp.OkHttpClientTransport$4.run(OkHttpClientTransport.java:571) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:923)

This was directly above it, not sure what it means
D/AddHabitDialog: Confirm W/GooglePlayServicesUtil: Google Play Store is missing. E/GooglePlayServicesUtil: GooglePlayServices not available due to error 9 W/Firestore: (23.0.4) [GrpcCallProvider]: Failed to update ssl context: com.google.android.gms.common.GooglePlayServicesNotAvailableException D/OpenGLRenderer: endAllActiveAnimators on 0xf0e4d250 (RippleDrawable) with handle 0xc7e91cb0 W/le.habitsmashe: Accessing hidden method Lcom/android/org/conscrypt/OpenSSLSocketImpl;->getAlpnSelectedProtocol()[B (greylist-max-q,core-platform-api, reflection, denied)     Accessing hidden method Lcom/android/org/conscrypt/AbstractConscryptSocket;->getAlpnSelectedProtocol()[B (greylist-max-q, reflection, denied) E/AndroidRuntime: FATAL EXCEPTION: grpc-okhttp-0     Process: com.example.habitsmasher, PID: 26441     java.lang.AssertionError: Method getAlpnSelectedProtocol not supported for object SSL socket over Socket[address=firestore.googleapis.com/<IP_ADDRESS>,port=443,localPort=41090]

Tried adding a habit, got this error. Checked DB, didn't add to DB. E/AndroidRuntime: FATAL EXCEPTION: grpc-okhttp-0 Process: com.example.habitsmasher, PID: 26441 java.lang.AssertionError: Method getAlpnSelectedProtocol not supported for object SSL socket over Socket[address=firestore.googleapis.com/<IP_ADDRESS>,port=443,localPort=41090] at io.grpc.okhttp.internal.OptionalMethod.invoke(OptionalMethod.java:114) at io.grpc.okhttp.internal.OptionalMethod.invokeWithoutCheckedException(OptionalMethod.java:135) at io.grpc.okhttp.OkHttpProtocolNegotiator$AndroidNegotiator.getSelectedProtocol(OkHttpProtocolNegotiator.java:183) at io.grpc.okhttp.OkHttpProtocolNegotiator$AndroidNegotiator.negotiate(OkHttpProtocolNegotiator.java:145) at io.grpc.okhttp.OkHttpTlsUpgrader.upgrade(OkHttpTlsUpgrader.java:63) at io.grpc.okhttp.OkHttpClientTransport$4.run(OkHttpClientTransport.java:571) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:923)

This was directly above it, not sure what it means D/AddHabitDialog: Confirm W/GooglePlayServicesUtil: Google Play Store is missing. E/GooglePlayServicesUtil: GooglePlayServices not available due to error 9 W/Firestore: (23.0.4) [GrpcCallProvider]: Failed to update ssl context: com.google.android.gms.common.GooglePlayServicesNotAvailableException D/OpenGLRenderer: endAllActiveAnimators on 0xf0e4d250 (RippleDrawable) with handle 0xc7e91cb0 W/le.habitsmashe: Accessing hidden method Lcom/android/org/conscrypt/OpenSSLSocketImpl;->getAlpnSelectedProtocol()[B (greylist-max-q,core-platform-api, reflection, denied) Accessing hidden method Lcom/android/org/conscrypt/AbstractConscryptSocket;->getAlpnSelectedProtocol()[B (greylist-max-q, reflection, denied) E/AndroidRuntime: FATAL EXCEPTION: grpc-okhttp-0 Process: com.example.habitsmasher, PID: 26441 java.lang.AssertionError: Method getAlpnSelectedProtocol not supported for object SSL socket over Socket[address=firestore.googleapis.com/<IP_ADDRESS>,port=443,localPort=41090]

@Nova-Drive Is this on the latest version of his branch? Try pulling one more time. I'll try as well.


Remove content-type from presigned query requests

This deletes the content-type header from query requests which are
being presigned. Specifically it removes the auto-set value
"application/x-www-form-urlencoded; charset=utf-8" because that
value is not valid for presigned urls.

Codecov Report

Merging #1336 into develop will decrease coverage by 1.36%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop    #1336      +/-   ##
===========================================
- Coverage    98.12%   96.76%   -1.37%
===========================================
  Files           46       46
  Lines         7694     7703       +9
===========================================
- Hits          7550     7454      -96
- Misses         144      249     +105

Impacted Files
Coverage Δ

botocore/signers.py
98.43% <ø> (ø)
:arrow_up:

botocore/handlers.py
95.92% <100%> (-0.41%)
:arrow_down:

botocore/compat.py
68.5% <0%> (-25.42%)
:arrow_down:

botocore/awsrequest.py
74.09% <0%> (-24.55%)
:arrow_down:

botocore/hooks.py
94.52% <0%> (-1%)
:arrow_down:

botocore/utils.py
95.79% <0%> (-0.21%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06f34bb...d2e8570. Read the comment docs.

rebased

Looks good to me. Though it does still bother me a little that we're making a customization to remove a header that we manually add to the request. Have we thought about how viable the opposite approach is? (Adding the header in a customization instead of unconditionally).

@joguSD we could, though it would be a bit weird since we would be moving something we always need for normal requests out of the serializer in favor of the special case of pre-signing.

@jamesls The signer was rejecting the request because the header wasn't included in the request. When you actually do send the header, you get a different error:
import botocore

sess = botocore.session.get_session()
sns = sess.create_client('sts')
url = sns.generate_presigned_url('get_caller_identity', {})
response = requests.post(
    url, headers={
        'content-type': 'application/x-www-form-urlencoded; charset=utf-8'
    }
)
print(response.text)

<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    IncompleteSignature
    <Message>When Content-Type:application/x-www-form-urlencoded, URL cannot include query-string parameters (after '?'): '/?Action=GetCallerIdentity&Version=2011-06-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=redacted%2F20171212%2Fus-east-1%2Fsts%2Faws4_request&X-Amz-Date=20171212T220209Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Signature=signature'</Message>
  </Error>
  <RequestId>1a16614a-df88-11e7-af03-a144f0113aaf</RequestId>
</ErrorResponse>

While that error does come from the remote signer, it is fundamentally a serialization error.

One alternative is to move it up into generate_presigned_url. That would eliminate the need for a handler as well as the need for adding to the context.

The above idea is growing on me. I think this whole remove content type header should be the concern of the client layer and not necessarily the general signer. Not sure what everyone else thinks?

test added


Allow row class to use initializer list or getters and be reflected in table

Simply, If we've the following table:
@UseRowClass(User)
class Users extends Table {
  IntColumn get id => integer().autoIncrement()();
  TextColumn get some => text()();
  TextColumn get some2 => text()();
}

we must have all arguments initialized at User's constructor, so we can't do:
class User {
  User({required this.id, required this.some}): some2 = some;

  final int id;
  final String some;
  final String some2;
}

or:
class User {
  User({required this.id, required this.some});

  final int id;
  final String some;
  String get some2 => some;
}

Nvm.
I don't think that's reasonable. If we update some2 value at the database, it won't reflect when we get User value.

It's broken for the reason you've mentioned, but I think it should already work nonetheless (you can write row classes that don't have every column of the table in their constructor). Drift will simply ignore the some2 column when mapping rows to Dart in that case.


example doesn't compile for me

I'm getting a type error when trying to compile the example.
$ nim r --d:release --d:threadsafe example.nim
Hint: used config file '/home/itwrx/.choosenim/toolchains/nim-2.0.0/config/nim.cfg' [Conf]
Hint: used config file '/home/itwrx/.choosenim/toolchains/nim-2.0.0/config/config.nims' [Conf]
Hint: used config file '/var/www/betterweb-guildenstern/betterweb/nim.cfg' [Conf]
......................................................................................................................................................
/var/www/betterweb-guildenstern/betterweb/example.nim(13, 32) Error: type mismatch
Expression: getOrDefault(readData(getBody()), "say")
  [1] readData(getBody()): StringTableRef
  [2] "say": string

Expected one of (first mismatch at [position]):
[1] func getOrDefault(headers: HttpHeaders; key: string;
                  default = @[""].HttpHeaderValues): HttpHeaderValues

In previous versions, strtabs was automatically exported to user. But because I want the server to be just a very unmagical and unopinionated building block, I removed such export (you only need StringTableRef with certain header operations). So, the fix is to explicitly import strtabs when required, and I have modified the example accordingly. Need to document this also at some point...


Databricks-sqlalchemy-Use System tables to fetch metadata

Current Databricks Sql Alchemy uses describe command/show schema/table to get metadata
Can we use system tables to get metadata

Hi @SomanathSankara! Sorry, but I don't think I understand what the question is. Can you please elaborate it a bit?

hi @kravets-levko correct my understanding.
Currently sqlalchemy (databricks ) package is using show table comand , show columns command to get table ,column details.
Cant we use UC system table(information schema) to fetch these details which will be faster

Adding some signal here: yes the system tables can be used to reflect the database schema. This would require Unity Catalog on the target instance, so it would be good to maintain a fallback to the existing SHOW TABLES behavior. But there's no technical limitation of SQLAlchemy that blocks this manner of implementation.

Hi all. I was looking at potentially implementing this and submitting a PR. For some background, I'm running into an issue using alembic to manage migrations for Databricks and if the schema does not exist, alembic throws an error because it's passing through the error from SHOW TABLES FROM <invalid_schema>.
I had a couple comments/questions:

information_schema only contains actual structures. It will not contain temporary tables or views. I see the get_view_names has a specific parameter for this, but do we see this as a limitation?
If we do need the temporary tables/views (and thus have to use SHOW TABLES/VIEWS), is there any objection to modifying the code to catch any DDL errors and returning an empty set with a logged warning instead?


TASK Problem with Memory ESP32 - Unavaliable Memory - Corruption  (IDFGH-3883)

Lastest Master as of 8/24/2020 has a serious problem with memory, it is using a lot of memory and we cant use with tasks.
please advise that is a serious faulty
xPortGetFreeHeapSize: 4244228 before_count:12 before_heap_32:4260724 before_heap_8:4246532
Heap summary for capabilities 0x00000004:
At 0x3ffc8de4 len 32767 free 2304 allocated 30408 min_free 2304
largest_free_block 2304 alloc_blocks 4 free_blocks 1 total_blocks 5
At 0x3f8037b0 len 4180047 free 4122000 allocated 57808 min_free 4092260
largest_free_block 4121984 alloc_blocks 49 free_blocks 2 total_blocks 51
At 0x3ffaff10 len 240 free 0 allocated 180 min_free 0
largest_free_block 0 alloc_blocks 7 free_blocks 0 total_blocks 7
At 0x3ffb6388 len 7288 free 0 allocated 6968 min_free 0
largest_free_block 0 alloc_blocks 72 free_blocks 0 total_blocks 72
At 0x3ffb9a20 len 16648 free 4 allocated 15556 min_free 0
largest_free_block 4 alloc_blocks 263 free_blocks 1 total_blocks 264
At 0x3ffc8dc8 len 94776 free 0 allocated 94336 min_free 0
largest_free_block 0 alloc_blocks 102 free_blocks 0 total_blocks 102
At 0x3ffe0440 len 15072 free 8420 allocated 5972 min_free 8420
largest_free_block 8420 alloc_blocks 161 free_blocks 1 total_blocks 162
At 0x3ffe4350 len 113840 free 113804 allocated 0 min_free 88200
largest_free_block 113804 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 4246532 allocated 211228 min_free 4191184 largest_free_block 4121984
Heap summary for capabilities 0x00000002:
At 0x3ffc8de4 len 32767 free 2304 allocated 30408 min_free 2304
largest_free_block 2304 alloc_blocks 4 free_blocks 1 total_blocks 5
At 0x3f8037b0 len 4180047 free 4122000 allocated 57808 min_free 4092260
largest_free_block 4121984 alloc_blocks 49 free_blocks 2 total_blocks 51
At 0x3ffaff10 len 240 free 0 allocated 180 min_free 0
largest_free_block 0 alloc_blocks 7 free_blocks 0 total_blocks 7
At 0x3ffb6388 len 7288 free 0 allocated 6968 min_free 0
largest_free_block 0 alloc_blocks 72 free_blocks 0 total_blocks 72
At 0x3ffb9a20 len 16648 free 4 allocated 15556 min_free 0
largest_free_block 4 alloc_blocks 263 free_blocks 1 total_blocks 264
At 0x3ffc8dc8 len 94776 free 0 allocated 94336 min_free 0
largest_free_block 0 alloc_blocks 102 free_blocks 0 total_blocks 102
At 0x3ffe0440 len 15072 free 8420 allocated 5972 min_free 8420
largest_free_block 8420 alloc_blocks 161 free_blocks 1 total_blocks 162
At 0x3ffe4350 len 113840 free 113804 allocated 0 min_free 88200
largest_free_block 113804 alloc_blocks 0 free_blocks 1 total_blocks 1
At 0x4009c86c len 14228 free 14192 allocated 0 min_free 14184
largest_free_block 14192 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 4260724 allocated 211228 min_free 4205368 largest_free_block 4121984
Task Name       Status  Prio    HWM     Task#   Affinity
Sentinel       R       15      5216    15      0
IDLE1           R       0       984     7       1
IDLE0           R       0       988     6       0
tiT             B       18      2596    12      -1
MultiCast   B       10      18524   21      -1
DigitalTRX B       5       3584    22      -1
Tmr Svc         B       1       1564    8       0
ipc1            B       24      416     2       1
sys_evt         B       20      1568    13      0
wifi            B       23      1868    14      0
esp_timer       B       22      3396    3       0
ipc0            B       24      492     1       0
so when trying to create any other task now the system just halt.
Please advise;

Add caps internal to functions checking free heap. Task stack cant be allocated in spiram.

Hi @filzek,
As Chegewara mentions, task stacks have to be allocated in internal memory. The heap dumps you have posted are for all 8-bit capable memory (0x04) and all 32-bit capable memory (0x02). Both these include PSRAM so there is a lot of memory free which is not available for use as a task stack.
Can you please try generate the same output for MALLOC_CAP_8BIT|MALLOC_CAP_INTERNAL? Also, suggest running idf.py size or make size to see a summary of how much static DRAM & IRAM is in use by the app.
You can try making more small memory allocations from PSRAM first by reducing the value of this config item:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/kconfig.html#config-spiram-malloc-alwaysinternal
My colleague has also written some more tips for reducing memory usage here:
https://medium.com/the-esp-journal/esp32-memory-analysis-case-study-eacc75fe5431

Hi, @projectgus
I have made what you request and it shows:
Task Name       Status  Prio    HWM     Task#   Affinity
Sentinel   R       15      1504    15      0
IDLE1           R       0       532     7       1
IDLE0           R       0       632     6       0
tiT             B       18      2220    12      -1
Digital      B       5       3580    25      -1
Tmr Svc         B       1       1528    8       0
hciT            B       22      1644    23      0
sys_evt         B       20      900     13      0
btuT            B       20      2728    24      0
BTC_TASK        B       19      28596   22      0
btController    B       23      2048    21      0
wifi            B       23      4088    14      0
esp_timer       B       22      3320    3       0
ipc1            B       24      1520    2       1
ipc0            B       24      1528    1       0
Task Create Failed, trying to go to recreate task again, erro returned:[-1]
xPortGetFreeHeapSize: 4129024 before_count:15 before_heap_32:4137548 before_heap_8:4129200 before_heap_INTERNAL:18952
So I just have 2 tasks and its something eating all the memory and I cant find out what is using it.
Is there any code that show exactly what is running and using memory on the SDK?
As, right now this is the more agressive problem.
Canary bytes doesnt show a heap corruption, char *task_list_buffer = malloc(uxTaskGetNumberOfTasks() * bytes_per_task); cant show the full usage, so still lost to understand memory usage,

BTC_TASK B 19 28596 22 0
this task is having way too big stack size. You can easy decrease it by 25000. It is bluetooth task, so you increased it stack size in menuconfig i think.

Well spotted, @chegewara . The default size for this task is 4096 so setting the config item back to this should help greatly.

Is there any code that show exactly what is running and using memory on the SDK?

In addition to the functions you're already using, there is the idf.py size, idf.py size-components, and idf.py size-files commands which print some information about the static memory allocations in an ESP-IDF project.
Also please check the other things linked in my previous reply, in particular changing the ratio of PSRAM to internal memory allocated by malloc can free up some internal memory.
But you may not need to do any of this after the BTC_TASK stack size is reduced.

We have move some code off the btc_task area as if the callback executes  any function it shall need memory allocated to it  so, we have pushed it to another task and create a queue pool area so share among calls beside process within the callback, it shrink the btc_task but moves the code to the new task, so result is the same.
Also we found out that BLE, WIFI and HTTP sockets are consuming a lot of memory to be executed.
We are trying to identify why this huge blocks of memory are being pulled out from the heap as well.

Maybe you need to switch to nimble.
It is "normal" that bluedroid is using a lot of heap, HTTP, especially HTTPS also need some memory to make TLS, and wifi is also not very light module, but you should be able to run all 3 easy and stay with 50kB free heap.

fizer, are you using .cpp and enable CONFIG_COMPILER_CXX_EXCEPTIONS in your program? It takes more then 1M RAM

We are not using it, but it seens some unexpected behavior for some default tasks, as we use some functions to write to fat inside the BLE call we need to increase the task size as it needs heap space to run, so we cant reduce that much, but seens that memory still a huge problem in the esp world.

Some components (e.g. nvs_flash) are written on CPP and in conjunction with CONFIG_COMPILER_CXX_EXCEPTIONS I've lost a huge heap of RAM. This features or bag appeared only in latest version. I had to roll back.

Hi @dima0905 ,

are you using .cpp and enable CONFIG_COMPILER_CXX_EXCEPTIONS in your program? It takes more then 1M RAM

This sounds like a bug. Could you please open a new issue with the requested details including exact ESP-IDF version, etc?
As this option is disabled by default it sounds like it's unrelated to the issue reported here, at least.
Hi @filzek ,

as we use some functions to write to fat inside the BLE call we need to increase the task size as it needs heap space to run, so we cant reduce that much, but seens that memory still a huge problem in the esp world.

A FATFS operation shouldn't require an extra 24KB of stack overhead (4KB stack size increased to 28KB stack size). Is it possible you're doing something here which is inadvertently blowing out the stack requirement? Is there any code you can share?

We have move some code off the btc_task area as if the callback executes any function it shall need memory allocated to it so, we have pushed it to another task and create a queue pool area so share among calls beside process within the callback

If you're doing relatively heavy operations (like filesystem operations) in the BTC callbacks then moving them off to a queue + worker task design like this is still probably a good idea. Depending on what else your firmware does, you may be able to consolidate some other operations here as well.

So, can you guys advise what is the real memory available?
MALLOC_CAP_8BIT:
Heap summary for capabilities 0x00000004:
At 0x3ffb2730 len 15448 free 0 allocated 15080 min_free 0
largest_free_block 0 alloc_blocks 20 free_blocks 1 total_blocks 21
At 0x3ffca5dc len 32767 free 356 allocated 31976 min_free 16
largest_free_block 356 alloc_blocks 24 free_blocks 1 total_blocks 25
At 0x3f804ad4 len 4175147 free 3258184 allocated 913908 min_free 3246496
largest_free_block 3248820 alloc_blocks 185 free_blocks 12 total_blocks 197
At 0x3ffaff10 len 240 free 0 allocated 96 min_free 0
largest_free_block 0 alloc_blocks 7 free_blocks 0 total_blocks 7
At 0x3ffb6388 len 7288 free 0 allocated 6988 min_free 0
largest_free_block 0 alloc_blocks 16 free_blocks 1 total_blocks 17
At 0x3ffb9a20 len 16648 free 40 allocated 14580 min_free 0
largest_free_block 0 alloc_blocks 122 free_blocks 8 total_blocks 130
At 0x3ffca5b8 len 88648 free 0 allocated 86944 min_free 0
largest_free_block 0 alloc_blocks 104 free_blocks 1 total_blocks 105
At 0x3ffe0440 len 15072 free 0 allocated 13360 min_free 0
largest_free_block 0 alloc_blocks 105 free_blocks 0 total_blocks 105
At 0x3ffe4350 len 113840 free 2716 allocated 109488 min_free 100
largest_free_block 2136 alloc_blocks 98 free_blocks 6 total_blocks 104
Totals:
free 3261296 allocated 1192420 min_free 3246612 largest_free_block 3248820
MALLOC_CAP_32BIT:
Heap summary for capabilities 0x00000002:
At 0x3ffb2730 len 15448 free 0 allocated 15080 min_free 0
largest_free_block 0 alloc_blocks 20 free_blocks 1 total_blocks 21
At 0x3ffca5dc len 32767 free 356 allocated 31976 min_free 16
largest_free_block 356 alloc_blocks 24 free_blocks 1 total_blocks 25
At 0x3f804ad4 len 4175147 free 3258184 allocated 913908 min_free 3246496
largest_free_block 3248820 alloc_blocks 185 free_blocks 12 total_blocks 197
At 0x3ffaff10 len 240 free 0 allocated 96 min_free 0
largest_free_block 0 alloc_blocks 7 free_blocks 0 total_blocks 7
At 0x3ffb6388 len 7288 free 0 allocated 6988 min_free 0
largest_free_block 0 alloc_blocks 16 free_blocks 1 total_blocks 17
At 0x3ffb9a20 len 16648 free 40 allocated 14580 min_free 0
largest_free_block 0 alloc_blocks 122 free_blocks 8 total_blocks 130
At 0x3ffca5b8 len 88648 free 0 allocated 86944 min_free 0
largest_free_block 0 alloc_blocks 104 free_blocks 1 total_blocks 105
At 0x3ffe0440 len 15072 free 0 allocated 13360 min_free 0
largest_free_block 0 alloc_blocks 105 free_blocks 0 total_blocks 105
At 0x3ffe4350 len 113840 free 2716 allocated 109488 min_free 100
largest_free_block 2136 alloc_blocks 98 free_blocks 6 total_blocks 104
At 0x4009d100 len 12032 free 11984 allocated 0 min_free 11964
largest_free_block 11984 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 3273280 allocated 1192420 min_free 3258576 largest_free_block 3248820
MALLOC_CAP_INTERNAL:
Heap summary for capabilities 0x00000800:
At 0x3ffb2730 len 15448 free 0 allocated 15080 min_free 0
largest_free_block 0 alloc_blocks 20 free_blocks 1 total_blocks 21
At 0x3ffca5dc len 32767 free 356 allocated 31976 min_free 16
largest_free_block 356 alloc_blocks 24 free_blocks 1 total_blocks 25
At 0x3ffaff10 len 240 free 0 allocated 96 min_free 0
largest_free_block 0 alloc_blocks 7 free_blocks 0 total_blocks 7
At 0x3ffb6388 len 7288 free 0 allocated 6988 min_free 0
largest_free_block 0 alloc_blocks 16 free_blocks 1 total_blocks 17
At 0x3ffb9a20 len 16648 free 40 allocated 14580 min_free 0
largest_free_block 0 alloc_blocks 122 free_blocks 8 total_blocks 130
At 0x3ffca5b8 len 88648 free 0 allocated 86944 min_free 0
largest_free_block 0 alloc_blocks 104 free_blocks 1 total_blocks 105
At 0x3ffe0440 len 15072 free 0 allocated 13360 min_free 0
largest_free_block 0 alloc_blocks 105 free_blocks 0 total_blocks 105
At 0x3ffe4350 len 113840 free 2716 allocated 109488 min_free 100
largest_free_block 2136 alloc_blocks 98 free_blocks 6 total_blocks 104
At 0x4009d100 len 12032 free 11984 allocated 0 min_free 11964
largest_free_block 11984 alloc_blocks 0 free_blocks 1 total_blocks 1
Totals:
free 15096 allocated 278512 min_free 12080 largest_free_block 11984
MALLOC_CAP_8BIT|MALLOC_CAP_INTERNAL:
Heap summary for capabilities 0x00000804:
At 0x3ffb2730 len 15448 free 0 allocated 15080 min_free 0
largest_free_block 0 alloc_blocks 20 free_blocks 1 total_blocks 21
At 0x3ffca5dc len 32767 free 356 allocated 31976 min_free 16
largest_free_block 356 alloc_blocks 24 free_blocks 1 total_blocks 25
At 0x3ffaff10 len 240 free 0 allocated 96 min_free 0
largest_free_block 0 alloc_blocks 7 free_blocks 0 total_blocks 7
At 0x3ffb6388 len 7288 free 0 allocated 6988 min_free 0
largest_free_block 0 alloc_blocks 16 free_blocks 1 total_blocks 17
At 0x3ffb9a20 len 16648 free 40 allocated 14580 min_free 0
largest_free_block 0 alloc_blocks 122 free_blocks 8 total_blocks 130
At 0x3ffca5b8 len 88648 free 0 allocated 86944 min_free 0
largest_free_block 0 alloc_blocks 104 free_blocks 1 total_blocks 105
At 0x3ffe0440 len 15072 free 0 allocated 13360 min_free 0
largest_free_block 0 alloc_blocks 105 free_blocks 0 total_blocks 105
At 0x3ffe4350 len 113840 free 2716 allocated 109488 min_free 100
largest_free_block 2136 alloc_blocks 98 free_blocks 6 total_blocks 104
Totals:
free 3112 allocated 278512 min_free 116 largest_free_block 2136
Is Just 3112??
or is it 15096??

Hi @filzek ,
Sorry for the slow reply. The internal DRAM is MALLOC_CAP_INTERNAL_MALLOC_CAP_8BIT and the available RAM is 3112 bytes.
If you count all of the internal IRAM & DRAM together then you have 15096 bytes. But the internal IRAM can only be allocated with MALLOC_CAP_INTERNAL|MALLOC_CAP_32BIT and this memory can only be read/written with 32-bit aligned reads/writes, so you can't use this as a general purpose RAM only for specific purposes with custom written code.
To try and move more of the internal DRAM allocations into PSRAM and free up some internal DRAM then you can tweak the things mentioned here.

Closing this as it seems like the issue is to do with task internal memory usage in the firmware rather than something we can fix in ESP-IDF.
@filzek if you have notice some excessive memory use by ESP-IDF itself then please a new issue with the details of how we can reproduce this.

Closing this as it seems like the issue is to do with task internal memory usage in the firmware rather than something we can fix in ESP-IDF.
@filzek if you have notice some excessive memory use by ESP-IDF itself then please a new issue with the details of how we can reproduce this.


Updates htmldiff to work with ruby 2.1.1 and rspec 3.2

I'd like to use this gem on an upcoming project and I noticed that it hadn't been updated in a while. I've updated the Rakefile and htmldiff_spec in order to get the specs to run on my system.
Tanner J.

It's been too long so I'm closing this. I'm archiving this repo and it should remain in it's current state IMO.


Android | Shader error

I get the following error on Android(Samsung TabA), and I managed to patch it. However I'm unable to create a pull request as I'm not able to test it an another platform than Android at the moment to check that it doesn't break anything else. So I've created this here for anyone else who will find it useful.
Line 36: ERROR: 'assign' :  cannot convert from ' const int' to ' temp highp float'
Line 36: ERROR: '' : compilation terminated
ERROR: 2 compilation errors.  No code generated.

Traceback

[C]: in function 'newShader'
lib/g3d/g3d/init.lua:49: in main chunk
[C]: in function 'require'
...

Solution:
line 36 of g3d/g3d.vert change screenPosition.y *= -1; to screenPosition.y *= -1.0;

Thanks for pointing this out!
It's kinda annoying that different systems have different type coercion rules, and I don't catch them sometimes because I only test on Windows and Mac. I just committed this change, it'll be in the next major version.


Improve text wrapping

This change applies the suggestions from @micahilbery in #56
Closes #56

I'll merge that in for now. Please comment if there's anythong wrong or missing.


HRTIM - Move EEVC3 patch to common.yaml since it affects all devices

@burrbull , here is a not so great attempt at solving #1041
As is before this PR, EECR3 is wrong for h7 devices. For some reason h7's EECR3 has the same value as EECR2.
Since EECR3 seems to have the same patch for both g4x4 and f3x4 i thought moving it to perpherals/hrtim/common.yaml would fix h7.

you should also add missing registers to peripheral in "common_patches" first

you should also add missing registers to peripheral in "common_patches" first

Something like cc79769?

Removing the brackets seems to make CI happy. However I do not know what the consequences of that may be


Update Office-365-URLs-IP-address-ranges.md

https://github.com/MicrosoftDocs/OfficeDocs-SkypeForBusiness/issues/5791
Replaced link.

Thank you for submitting feedback and contributing to the docs. We are currently investigating this.

@JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@SerdarSoysal PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Create S3bucket block

Summary

Created S3Bucket block
Added S3Bucket tests for MinIOCredentials and AwsCredentials blocks

Relevant Issue(s)

Checklist

[x] Summarized PR's changes in the CHANGELOG.md
[ ] Updated _logo_url with #noqa
[x] Updated _block_type_name
[x] Function calls get_secret_value() where applicable
[x] Used SecretStr/SecretBytes if applicable
[x] Included example of loading block
[x] Docstring updated (block mentioned)
[x] Included entry point in setup.py
[x] Add testing for S3Bucket block
[ ] Make the pytests dry

= Coverage for /home/runner/work/prefect-aws/prefect-aws/prefect_aws/ =
- Detailed Coverage -
| Name                              |  Status |
|-----------------------------------|---------|
| s3bucket.py (module)              |  MISSED |
|   S3Bucket (L12)                  | COVERED |
|     S3Bucket._get_s3_client (L43) |  MISSED |
|     S3Bucket.read_path (L60)      |  MISSED |
|     S3Bucket._read_sync (L63)     |  MISSED |
|     S3Bucket.write_path (L71)     |  MISSED |
|     S3Bucket._write_sync (L77)    |  MISSED |
|-----------------------------------|---------|

- Summary -
| Name        | Total | Miss | Cover | Cover% |
|-------------|-------|------|-------|--------|
| s3bucket.py |     7 |    6 |     1 |    14% |
|-------------|-------|------|-------|--------|
| TOTAL       |    27 |    6 |    21 |  77.8% |
|-------------------------------------------------|
| (5 of 6 files omitted due to complete coverage) |
- RESULT: FAILED (minimum: 95.0%, actual: 77.8%) -

I think you need a short description of s3bucket.py at the top of the file to fix the static analysis, e.g.
"""
Module for ...
"""

Should s3bucket.py be renamed to s3_bucket.py?

Okay so I just realized if we rename to s3_bucket.py, the variable is s3_bucket too.
from s3_bucket import S3Bucket
s3_bucket = S3Bucket()

Will this cause circular issues? maybe s3_bucket.py should just be bucket.py

Thanks for yet another contribution! Wondering, how does this module work with s3.py. Are we deprecating s3.py?

I think you'd have to ask Alex about that.

I think this is ready once:

Static analysis is fixed (add something similar to setup.cfg as https://github.com/PrefectHQ/prefect-ray/blob/main/setup.cfg#L28)
For tests to speed up, can you do git fetch && git pull origin fix_reqs
Can you elaborate on why _resolve_path would break read_path? Maybe we need to add to docstring that basepath here only applies to write_path.

I think this is ready once:

Static analysis is fixed by adding short one line docstrings to the methods read_path, write_path, etc.
Can you elaborate on why _resolve_path would break read_path? Maybe we need to add to docstring that basepath here only applies to write_path.

Added to docstring.

Here's mkdocs.yml and docs/
Thanks so much for the revisions!

LGTM! Can you please select 'Squash and Merge' when you merge since this PR has so many commits?

@desertaxle @ahuang11 both reviews were dismissed


Add Oui to Openwrt General Packages Feed

Please add Oui for inclusion in Openwrt/packages (https://github.com/openwrt/packages). This enables ease of deployment to wider openwrt devices.
Here is an example of package inclusion commit.
https://github.com/openwrt/packages/commit/0871991d8d732fc79c04259ec80da0f4a3d14fc6#diff-90530134f0e9c6482a9ceb7ba0bbed2a

I'd love this too, It's an amazing interface, very well designed.
I can help if needed

It would be great to have it as a general package!

It seems that it needs to be split into a single package to merge into openwrt / packages.

Can we do that?

The best way to join OpenWrt is as feeds, like Luci.

But will the OpenWRT team allow another feed? I believe they only accept the merge of packages into the existing feeds.
I didn't take a look but is it that complicated to separate oui into several packages and get them merged?

@zhaojh329 have you ever tried to merge this as a feed? Have you ever seen anyone do that? I have been reading the packages and makefiles anda I believe that they don't fit any of the folders, aside from uhttpd, wich they could argue that would only be another version, I believe this requires another folder just like LuCi, since this is a "replacement".
Also, is there any way to make it run on nginx?

@zhaojh329 have you ever tried to merge this as a feed? Have you ever seen anyone do that? I have been reading the packages and makefiles anda I believe that they don't fit any of the folders, aside from uhttpd, wich they could argue that would only be another version, I believe this requires another folder just like LuCi, since this is a "replacement".
Also, is there any way to make it run on nginx?

No plan to support nginx.

And have you ever tried to merge your feed to OpenWRT?


Alpha.3 repository package code contract change

Hi. First - I LOVE THIS ADDON! Great work Team!!
We noticed that our development checkout of alpha.3 had one contract and then a yesterday checkout of alpha.3 on staging had a seperate contract. Ie: a function went missing!
May I please suggest to use a patch or version bump on any code changes and not republish an existing version!
Best!

I am going to close this issue for now, feel free to reopen this issue if you feel like it's still relevant and we should take a look at it.


Adding a Coming Soon Page For broken and not Working Links

Issue Report
Title
Proposal of Creating a Cooming Soon Page
Description
There Are many Section in the Websties which are not fully Functional,there are many buttons, hyperlinks, and all which are not functional so we can create a Common Page for all Such elements in the page and then redirect all the elements to the same page as it will Create Positive Impact on the Users of the Web site . The Website Will feel Like Broken.
Steps to Reproduce the Issue

Action 1: Make  a Coming Soon Page for all the upcoming Features
Action 2: Identify all the elements in the page that are not present now.
Action 3: Redirect all the elements to the same page

Expected Behavior

Actual Behavior
doing nothing

@gauravsingh1281 pls assign me this task. i will work on it.

@kuldip40  I am working on the issue and will submit my work by today's evening


Record changes in links through patch_links

After each call to Patch Links, the Links added and Removed are recorded in a new LinkChange model. The source- and target_content_id are recorded, the link type and if the change was an addition (change=1) or a removal (change=-1), the index of the record and the related Action
The LinkChanges are exposed through a new API call (/v2/link-changes?start=) which outputs the LinkChanges in JSON. The number of records can be limited through the start parameter which takes the ID of the first LinkChange to fetch
Trello: https://trello.com/c/162iqEul/185-update-change-history-every-minute-from-publishing-api

I'm splitting this PR into 2 for easier review. https://github.com/alphagov/publishing-api/pull/1026 implements the change history, with @kevindew's comments about Action, enums and indexes addressed.


java.lang.ClassCastException: javax.swing.plaf.FontUIResource cannot be cast to javax.swing.Painter

error report uploaded. Id: 3368c6e
MultiBit HD version: 0.1.3
Operating system: 64 6.3 Windows 8.1
No user notes
Number of stack traces: 1
First stack trace:
java.lang.ClassCastException: javax.swing.plaf.FontUIResource cannot be cast to javax.swing.Painter
at javax.swing.plaf.nimbus.NimbusStyle.getBackgroundPainter(Unknown Source) ~[na:1.7.0_80]
at javax.swing.plaf.nimbus.SynthPainterImpl.paintBackground(Unknown Source) ~[na:1.7.0_80]
at javax.swing.plaf.nimbus.SynthPainterImpl.paintLabelBackground(Unknown Source) ~[na:1.7.0_80]
at javax.swing.plaf.synth

Looks like another "pure Java so can't trap it" scenario.

Added to list of benign exceptions.
Awaiting review and closing.

Code review looks good.
Closing.


Husky testing first pass

This PR incorporates the UW Reopening REDCap project (Husky Coronavirus Testing) into the SCAN Switchboard.
This is the first step in pulling HCT records -- we are not yet deep linking to particular repeat instances or dropping users into the PCDEQC form yet.
Tom suggested that we get this step out the door first, and both I and the lab agreed that this was a good idea.

Because we're adding new columns to the sqlite record_barcodes table, we'll need to delete the old .sqlite file before deploying.

Oops, some of my testing code snuck into the second commit. I force re-pushed to drive it out of there.

 Add Husky Testing REDCap to SCAN Switchboard

Thanks for pushing this for review Kairsten! The other thing I was considering including with this first change was renaming "SCAN switchboard" to something that better reflects it includes non-SCAN projects. Perhaps just at the UI layer, if not more deeply throughout the repo. What do you think about "SFS Switchboard", or "Barcode Switchboard"?

Thanks for pushing this for review Kairsten! The other thing I was considering including with this first change was renaming "SCAN switchboard" to something that better reflects it includes non-SCAN projects. Perhaps just at the UI layer, if not more deeply throughout the repo. What do you think about "SFS Switchboard", or "Barcode Switchboard"?

I agree, we probably want to rename this repo and our mentions of it to one of the options you suggested, or simply "Switchboard".

Not sure if anyone checks this page, but duplicate_record_ids is filled with the UW-reopening records because the repeated instances having the same record_id.

Ooh good catch. We may want to drop that view if it's no longer useful, or filter it to only projects with non-repeating instances. Thoughts there?
Also, does anyone consider the duplicate_record_ids table no longer being useful or the "SCAN" in "SCAN Switchboard" naming to be a blocker here, or can I merge this ASAP for the lab?

Also, does anyone consider the duplicate_record_ids table no longer being useful or the "SCAN" in "SCAN Switchboard" naming to be a blocker here, or can I merge this ASAP for the lab?

I think these are definitely not blockers! Just things to think about in the long-term.

Also, does anyone consider the duplicate_record_ids table no longer being useful or the "SCAN" in "SCAN Switchboard" naming to be a blocker here, or can I merge this ASAP for the lab?

I think these are definitely not blockers! Just things to think about in the long-term.

I absolutely agree.

 Remove 'duplicate_record_ids' table from the Switchboard

Yep, I dont think those need to be blocking. Can we add a note to https://trello.com/c/vsYwqHXS/778-add-husky-testing-redcap-to-scan-switchboard so we make sure to do them in the second stage of the task?


fix: Wrong position of grid content when resize column width #335

This is the temporary solution I used.

good. The current code base has changed a lot, so I will refer to it and merge it when applying. thank you.

Version 2.9.0 has been released.
https://github.com/bosskmk/pluto_grid/releases/tag/2.9.0


Evaluate coefficients columns in pickles

This is the Mina counterpart to https://github.com/o1-labs/proof-systems/pull/820.
Still to do:

[x] fix compilation of WASM bindings (and update JS bindings to handle the new field).

!ci-build-me

!ci-build-me

!ci-build-me


The argument type 'void Function(TapPosition)' can't be assigned to the parameter type 'void Function(TapPosition, LatLng)'.

Hi, I'm getting this error as noted in the title, for onTap: (_) => _popupLayerController.hideAllPopups()
Any idea how I could solve this issues? Many thanks in advance.

@nep2099 Based on the error message it looks like it should be onTap: (_, __) => _popupLayerController.hideAllPopups().


[BUG] - Tabs component type error "Type 'Key' is not assignable to type 'Key | null | undefined'."

NextUI Version
2.2.8
Describe the bug
Seems like some type error
Your Example Website or App
https://stackblitz.com/edit/stackblitz-starters-jfhwnj?file=app%2Fpage.tsx
Steps to Reproduce the Bug or Issue
open /app/page.tsx (Line 22)
Expected behavior
I expect no type errors :)
Screenshots or Videos

Operating System Version
macOS
Browser
Chrome

Hey @MeonisRP looks like the react-aria tabs key is different from the rest which uses React.Key, as a workaround while this is fixed, you can import the Key type from '@react-types/shared'; instead of using the react one.
https://stackblitz.com/edit/stackblitz-starters-rtm2h5?file=app%2Fpage.tsx,node_modules%2F%40react-types%2Fshared%2Fsrc%2Fselection.d.ts


(Refactoring) Renaming function to finish removal of the old code

This PR renames

tune_new->tune
perform_utility_analysis_new ->perform_utility_analysis

Thanks for review!


Add iter search for big queries

Related to issue https://github.com/neocl/jamdict/issues/22

Available in release 0.1a11
Documentation: https://jamdict.readthedocs.io/en/latest/recipes.html#iteration-search


dynamic viewports and scissors do not allow negative x- and y-values

In the viewport and scissors parsing of the code generator, for dynamic you do a check for negative x- and y-values
https://github.com/mikolalysenko/regl/blob/gh-pages/lib/core.js#L653
          check.optional(function () {
            env.assert(scope,
              BOX_X + '>=0&&' +
              BOX_Y + '>=0&&' +
              BOX_W + '>0&&' +
              BOX_H + '>0',
              'invalid ' + param)
          })

but you do not do the corresponding check for static. You only do this:
https://github.com/mikolalysenko/regl/blob/gh-pages/lib/core.js#L600
        var x = box.x | 0
        var y = box.y | 0

so you do not check for negative x- and y-values for static. This is weird and inconsistent, and should be looked over.
Also, you do not seem to have any test cases for viewports with negative x- and y-values. That's probably why the above bug slipped through.

Yeah, this stuff needs better testing.  I'm planning on rolling some of this in with the upcoming fbo tweaks since viewports and scissor boxes are FBO dependent features.

Why are we enforcing positive only x and y for viewports? Only width and height are required to be positive https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/viewport

It shouldn't do that any more.  With the latest version you can now have negative viewports.


Compounding .where() and .orderBy()

A Table has .orderBy() and returns a Collection.
A table also has .where() and returns a Collection.
How do I combine the two?
For example, this works:
let table = db.issues
let collection = table.orderBy('updated_at')
// optionally I can add the chainable `collection = collection.reverse()`

And this works:
let table = db.issues
let collection = table.where('key').anyOf('a', 'b')

But this doesn't works:
let table = db.issues
let collection = table.where('key').anyOf('a', 'b')
collection = collection.orderBy('updated_at')

Ideally, I should be able to chain freely, like in the Django ORM:
let table = db.issues
let results = table
if (some condition) {
  results = results.where(...).equals(...)
}
results = results.orderBy('somekey').reverse().limit(10).offset(20)
if (other condition) {
  // more filtering!
  results = results.where(...).anyOf(...)
}
// eventually, send the query definition to the database
results.toArray().then(records => ...)
// and I should be able to make a count too
results.count().then(number => ...)

True, but the implementation would have to choose whether to use the index for sorting or filtering. With Dexie (as of current version at least), you will have to make that choice yourself and not rely on a qualified performance guess done by the system. An SQL system that gets that kind of query will make a choice whether to use the index for sorting or filtering. If sorting is choosed, it will do a full-table scan sorting with the index. If filtering is choosed, it will sort the resulting array after retrieval. With Dexie you could choose strategy depending on how narrow the filter is, and how large the result set would be. If the filter is narrow and the result set is small, choose where() to utilize the index for filtering. If the result set is large or filter is broad, you would probably be better of using orderBy() to utilize the index for sorting. Especially if you use limit() on the Collection.
That said. I am tempted to change the next major version of Dexie so that there is only a Collection class and no Table class, with possibility to chain where() and orderBy() clauses as with real ORMs. Today Dexie pretty much maps to the indexedDB API with not that much magic put on top of it.

Thank you for the answer.
Unfortunately, it means I can't use Dexie to combine where, limit and orderBy. Granted my database isn't enormous but if I pick orderBy I have to filter out almost ALL other records in memory. If I chose where I have to first sort ALL other records in memory. Either way, if I have 10,000 records in the database (and 1,000 belonging to some constraint) and I only want the top 10 (order by something), I would have to extract 1,000 records from the database do the sortBy or filter, show the top 10, then "drop" the other 990 objects extracted.
I'm not sure I have the capacity to help the project but I would LOVE for there to be an ability to do ordering AND constraints in 1 query.

If you only want the top X items you'd normally be better off using the index for sorting and filtering out the data in memory. That way your performance wont be affected as the database size increases. If I'd ever implement the possibility to combine orderBy() with where(), I'd initially build a simple query planner that would always utilize the query for sorting rather than filtering in case Collection.limit() was used on the collection.

Hello David,
Thank you for the awesome job with both building Dexie and maintaining the "community" here and on stackoverflow.
I am also wondering when might the ability to combine where with orderBy/sortBy be coming to Dexie?

I'm sorry to say that I cannot give an estimate for that, though I wish I could. If the Collection class where rewritten to be more abstract (not tied to indexedDB operations), we could do such things on it as orderBy(). That is part of the "Vision for Dexie". If everything goes as plan, I will be able to release some new architecture in the winter/spring, including allowing arbritary backends (to use the Dexie API towards a postgres database for example). But as the project is not yet financed at all, these are just hopes and visions so far.
The work done so far on this is in the branch next-expression-engine but is still totally untested.

Hi David,
1. Does Dexie do Case-insensitive sorting
2. Does Dexie supports contains/indexOf
3. Why does Dexie processing latest request while cancelling previous pending request.
For Instance : while searching on dexie db with search string "p" if this request is under process
meantime if search string updated with "pe" then previous request("p") cancelled and updated
search string request with "pe" is processing and it returns single response which is related to
"pe" but i supposed to  get two response one is for "p"  and second is for "pe"
NOTE : This issue cause live search inaccurate result.
Is there any workaround?

Is this still not possible?

@aprixon01112017 The state of Dexie 3 is still that you need to choose what you want to use the index for - if used for sorting, use orderBy(), if used for filtering, use where(). If you build up a compound index, you may combine the two by having the second part representing the orderBy while using the first part in your query, such as querying all friends named 'Matthew' and order the result by their age: where('[name+age]').between(['Matthew', Dexie.minKey], ['Matthew', Dexie.maxKey]). However, this works only for equality comparision, and the way to use it is not obvious.
The plan is to release a new major (Dexie 4) at some time, with a lot of news. One is that the query engine will accept arbitrary filter-expressions and ordering, even when it cannot utilize an index. Pretty much the same approach as other databases use (SQL / MongoDB, etc) and try to do a best-effort choice automatically instead. I cannot promise the time schedule for this as I'm working as a consultant most part of my work time, so it has taken a great deal of time to complete the design of this new API, but I feel I've landed in a great API now that I'm looking much forward to get the time to release. Some of the news I'm planning for, are:

React Suspense support
No separation between Table and Collection.
Nested where clauses and allow orderBy on Collection.
Let Collections be compatible with ES-Observable and have a subscribe() method, also let any database querying expression be observable.
Will use an API that is less IndexedDB-specific and can be implemented using other resolvers.
The possibility to implement a server for it and have the two support conflict-free sync (Strong Eventual Consistency).
Easier typescript integration
Simplified migration
Support for arbritary expressions (complex, nested expressions, partially mongodb compatible)
Better testability (in-memory resolver for the API)

I'm trying this aproach: https://dexie.org/docs/Collection/Collection.offset()#paged-or-queries
It says that "Will have equal performance for page 1 and page 100."
But I can't understant how that is possible if I need go straith to a given page.
How to get the lastEntry without run by all the pages until the one I want?

That piece of code in the docs is conceptual. It represents a way of using the index for sorting on other key than filtering. I don't think it scales for result sets larger than 100,000 as the Set of primary keys needs to fit in memory all the time, but the time to retrieve next page would always be constant. Given that you have the lastEntry from the previous page, you will get the same amount of data fetching to retrieve the next page, but you cannot start at page 100 since you wouldn't have the lastEntry from page 99 unless you've navigated there before.

Get all primary keys from the evaluation of the expression.
Put those keys into a Set
Enumerate the index from prevKey and forward until page is filled. get the actual value for each primary key in the page.

So basically there are two data fetchings which would be equal all the time:

Getting all the primary keys for the entire result.
Enumerating index from a certain key and forward.

Right. So, I think I'm stuck with a memory bloat issue, but thanks for the clarification!

Hi @dfahlander, thanks for giving us Dexie.js, it is totally amazing. Just wanted to check on v4 plan

The plan is to release a new major (Dexie 4) at some time, with a lot of news.

Is there any plan in near future for it?

I originally had the plan to release it at the end of 2022 but my current focus is getting dexie-cloud up to production so dexie@4 release will have to be delayed. It is possible to install the current beta using npm install dexie@next - it's an easy and backward compatible upgrade, but it can be disappointing since most features mentioned in the road map are not in place yet. My current guess would be it be fully implemented at some point in the summer or autumn 2023.

@dfahlander I am trying to use this suggestion you made above:

If you build up a compound index, you may combine the two by having the second part representing the orderBy while using the first part in your query, such as querying all friends named 'Matthew' and order the result by their age: where('[name+age]').between(['Matthew', Dexie.minKey], ['Matthew', Dexie.maxKey]). However, this works only for equality comparision, and the way to use it is not obvious.

I am in a situation where this was useful to me and I would like to share the code with others in hope it will help them, too, and you, in case you spot an issue with it I might have missed.
I am in a need of where+orderBy because in my case I am paging the database table and optionally skipping entries which are marked a certain value in one of their cells. I used an index on that column to be able to say where('isValid').notEqual(skipValid ? 'true' : ''), but with this code, the order of the items returned was not the same as the order in which they were loaded in the database.
At your suggestion I used a compound index and used the primary key as the second component so that the entries would always come out in the original order. I have prototype a solution based on this and it seems pretty fast for my use case. Here it is with 10k rows, both lookups performing well. In production I am working with datasets of 100k to 1M. That's why I am keen on not doing the sorting on the main thread. It seems to me with this compound index that is avoided? The performance seems to suggest so at least…
https://stackblitz.com/edit/dexie-playground-strc7n?file=index.ts
import { Dexie } from 'dexie';

const db = new Dexie(Date.now().toString());
db.version(1).stores({
  test: '++id,[isValid+id]',
});

void (async function () {
  try {
    for (let index = 0; index < 1000; index++) {
      await db['test'].add({ index });
    }

    console.log(await db['test'].count());

    console.log('preview before changes:');
    for (const entry of await db['test'].limit(5).toArray()) {
      console.log(entry);
    }

    await db['test']
      .toCollection()
      .modify((entry) => (entry.isValid = (entry.index % 2 === 0).toString()));

    console.log('preview after changes:');
    for (const entry of await db['test'].limit(5).toArray()) {
      console.log(entry);
    }

    function getEntries(skipValid: boolean) {
      let query = db['test'];
      if (skipValid) {
        query = query
          .where('[isValid+id]')
          .between(['false', Dexie.minKey], ['false', Dexie.maxKey]);
      }

      return query;
    }

    const invalidStamp = Date.now();
    console.log('preview of invalid ascending:');
    for (const entry of await getEntries(true).limit(5).toArray()) {
      console.log(entry);
    }
    console.log('====>', Date.now() - invalidStamp, 'ms');

    const invalidReductionStamp = Date.now();
    console.log('reduction of invalid ascending:');
    const invalidStats = { valid: 0, invalid: 0 };
    await getEntries(true).each((entry) => {
      if (entry.isValid === 'true') {
        invalidStats.valid++;
      } else {
        invalidStats.invalid++;
      }
    });
    console.log(invalidStats);
    console.log('====>', Date.now() - invalidReductionStamp, 'ms');

    const allStamp = Date.now();
    console.log('preview of all ascending:');
    for (const entry of await getEntries(false).limit(5).toArray()) {
      console.log(entry);
    }
    console.log('====>', Date.now() - allStamp, 'ms');

    const allReductionStamp = Date.now();
    console.log('reduction of all ascending:');
    const allStats = { valid: 0, invalid: 0 };
    await getEntries(false).each((entry) => {
      if (entry.isValid === 'true') {
        allStats.valid++;
      } else {
        allStats.invalid++;
      }
    });
    console.log(allStats);
    console.log('====>', Date.now() - allReductionStamp, 'ms');
  } catch (error) {
    console.log(error.message ?? error);
  }
})();

I am also trying to do .where() and .orderBy() combined but it seems I can do only on of them and do the rest using simple js. Please let me know if there is a build in way to do this in dexie.js

@dfahlander : The plan is to release a new major (Dexie 4) at some time, with a lot of news. One is that the query engine will accept arbitrary filter-expressions and ordering, even when it cannot utilize an index.

As of 2024 with Dexie 4.0.7 compounding .where() and .orderBy() is still not possible.

@dfahlander  (2019): The plan is to release a new major (Dexie 4) at some time, with a lot of news. One is that the query engine will accept arbitrary filter-expressions and ordering, even when it cannot utilize an index.

As of 2024 with Dexie 4.0.7 compounding .where() and .orderBy() is still not possible.
It is however possible with .filter()
In my code this doesn't work:
const mostVisited = await table.where({ show: 1 }).orderBy('visitedCount').reverse().limit(3).toArray();
However using .filter() it's possible:
const showFilter = c => c.show == 1;
const mostVisited = await table.orderBy('visitedCount').filter(showFilter).reverse().limit(3).toArray();

In this case, you can combine the criteria and the order using a compound index '[show+visitedCount]':
const db = new Dexie('dbName');
db.version(1).stores({
  yourTable: 'yourPK, [show+visitedCount]'
});
...
const mostVisited = await db.yourTable
  .where('[show+visitedCount]')
  .between([1, -Infinity], [1, Infinity])
  .reverse()
  .limit(3)
  .toArray();

The reason is that the order is always defined by the index used in the where-clause, and in this case we both limit the criteria and order the result on [show+visitedCount] using the index only. reverse() will make it reverse the order.

The reason is that the order is always defined by the index used in the where-clause, and in this case we both limit the criteria and order the result on [show+visitedCount] using the index only. reverse() will make it reverse the order.

This should be added to documentation as an example. I was scratching my head how to solve this because I have "heavy" entries and .toArray() takes very long if entires are not ordered AND filtered by an index. Thank you!


feat: Orbit interpolation

Orbit interpolation!
@jnibauer, you can use this to model InterpolatedMockStream. That will be really awesome to have!

Sigh. Some vectorization issue.


Computed properties cannot with different types in getter/setter in TypeScript

Vue version
3.2.45
Link to minimal reproduction
https://sfc.vuejs.org/#eNqVUk1LxEAM/StxLt2FbYseS7coIijoRQQvA9Jds7XS+WAmVaT2v5uZ6bpePbWZl7y8vGQSV9YWHyOKStR+73pL4JFGC0Oru60U5KVopO6VNY5gAoeHDeyNsiPhK8xwcEZBxgSZ1FLvjfYEL8p3sA2ptR7VDl2zOl8fwYQdGVaT1AAdUgWrNWybWFt8tMOIm4CwFkZiXEEig2/w5HrdxfxYL+lUxuTxy3JolnrmxmUJ9GXRHELzCp5dT+1uwOtFw+NJZ5jhL1N2wWMt5ZClthn0HrQhaL3vOx2YgMySkni4qC6Tm5GyJlR2aAk5AqjfzptpikbMc11yFF97zWqgiq3ZeIalgMv4yiGraRbvVlhQ69gzVgC3Tw/3dyHnZkCFmtZJe1wak57leRr6vxNAnjNDXf4qFxuRjiBXrS3evdF8Msn9BeBLqdI+whufRIileCOyvipLf9iHQ3v3hXFdyX+FGzX1Cgv0Kt858+nRMbEUgSKsTsw/487lnQ==
Steps to reproduce
Define a computed property, getter type is number, setter type is number | string.
The computed property type is WritableComputedRef<number>.
And cannot set a string to its value.
What is expected?
Can set both a string and a number to the value of computed property.
What is actually happening?
type 'string' is not assignable to type 'number'
System Info
System:
    OS: Windows 10 10.0.22621
    CPU: (8) x64 Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
    Memory: 1.66 GB / 11.80 GB
  Binaries:
    Node: 19.0.1 - D:\Program Files\nodejs\node.EXE
    Yarn: 1.22.17 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 8.19.2 - D:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.22621.819.0), Chromium (107.0.1418.62)
    Internet Explorer: 11.0.22621.1

Any additional comments?
TypeScript is support different types for getter and setter now.
A suggested solution is to define the type of Writable Computed Ref as WritableComputedRef<number, number | string>.

It will work if you set const _msg = ref<number | string>(1);

@otomad You need to do computed<string | number>({...)
This is fully documented: https://vuejs.org/guide/typescript/composition-api.html#typing-computed

You need to do computed<string | number>({...)

@ferferga If you do computed<string | number>({...) then computed's value will have type string | number and not number, so it's not a solution

@jacekkarczmarczyk What's the use case for this though? I think allowing this is error-prone.
Computed are derived state, and as such, the getter should have some relation with the setter.
With pure (non-reactive) variables, TypeScript doesn't allow to define a type for "getting" and another for "setting". Neither in classes. It's a bad design pattern. Why should this be in core?
Computed should not be treated like functions, but as variables (more precisely, getters)
I understand that the original poster's issue is that the original get value is just a number, so TypeScript just infers it as a number and needs additional info (in this case a type annotation) to know it can be a string too.

I've only pointed out that your "solution" is not a solution for the problem. Whether the feature can be implemented or not or what are the use cases is a different thing
Anyway, most probably it can't be implemented because of ts limitations, here's a long discussion on this topic: https://github.com/microsoft/TypeScript/issues/2521

I understand that the original poster's issue is that the original get value is just a number, so TypeScript just infers it as a number and needs additional info (in this case a type annotation) to know it can be a string too.

@jacekkarczmarczyk This is why I said it's a solution in my first message, since I believed it's a misconception (though probably I shouldn't have done that in the first place 😊).

The issue has been resolved by PR #11472


[Feature Request]Add a Text Stream is over Event to SSE in the docker deployment

Is your feature request related to a problem? Please describe.
Currently there is no clear way to determine when the Text transmission when using SSE (streaming) is over. It is mostly guess work.
Describe the solution you'd like
Add an event that is sent after the response from the LLM is complete.
Describe alternatives you've considered
Guessing.

Hello @mymro,
Thanks for reaching out for us.
For stream mode, the HTTP response is sent back using chunked transfer encoding. The client side knows when the data is fully sent from the server.
Additionally, there are 3rd party libraries (like sseclient-py) which can iterate over the http response as events.
The client code can simply consume the events using a for loop like below:
client = sseclient.SSEClient(response)
for event in client.events():
    pprint.pprint(json.loads(event.data))

The for loop ends automatically when there are no more events. Under the hood, it is determined by the end of the trunced stream. In other words, the end of event stream is determined by the server side, not a guess work.

We will close this issue as no further action is required.


Add .prefix() expression to {name}

It would be very helpful to have an expression that is able to retrieve the prefix of an EOSIO name type.
Currently only .suffix() is available.
Example 1
name account = name{"my.account"};
name suffix = account.suffix(); // => "account"
name prefix = account.prefix(); // => "my"

Example 2
For names with multiple prefixes, split name at the first . dot to the right.
name account = name{"a.my.account"};
name suffix = account.suffix(); // => "account"
name prefix = account.prefix(); // => "a.my"

name subsuffix = prefix.suffix(); // => "my"
name subprefix = prefix.prefix(); // => "a"

References

eosiolib/name.hpp

Added an additional example in the case of multiple prefixes (ex: a.my.account)

Can we close this since it seems to be completed?

🎉 awesome, completed indeed
Thanks @victorj8 for the implementation!


is there any change i can get the example images without caption on it?

thanks for your wonderful work! The sample shown in the readme is amazing too.
I am wondering if i can get these example images without caption on it? Thanks.

Do you want the images?

https://github.com/user-attachments/assets/93a62148-5fba-45ed-81ee-26252eeb176e

thanks!


Upgrade to latest<EMAIL_ADDRESS>and others

and others (strum, tower-http).
Notes about upgrading Leptos:

Release notes + changes https://github.com/leptos-rs/leptos/releases/tag/v0.4.0
leptos::ServerFn::register is deprecated nowwarning: use of deprecated associated function `leptos::ServerFn::register`: Explicit server function registration is no    longer required on most platforms (including Linux, macOS, iOS, FreeBSD, Android, and Windows). If you are on another    platform and need to explicitly register server functions, call ServerFn::register_explicit() instead.

Closes #182
Closes #181
Closes #178
Close #177

Thanks for keeping this up to date!


Contact link doesn't work

https://www.soren-mindermann.com/ in the README returns a 404 error.

Updated


generates new comscore panels stratified on race census (ACS) data an…

…d precompute cohorts

all panel generation code is within notebook. minor bug fix in stratify_data_without_replacement function
this supercedes code that stratified on both race and income distributions from CPS data
generating 10 panels for each of the 52 weeks of data, creating 520 total panels
also precomputed simhash and cohort IDs for comscore panel
precomputed simhash intermediary file in output/comscore_acs_race_panels_simhash.csv
computed cohorts in file output/comscore_acs_race_panels_cohorts.csv
compare values for min k for cohorts, using ratio of google chrome OT k:|C| as baseline
cohorts with k=32 (ish) most similar ratio
also computes cohorts for a variety of k

comparison data:

computes comparison data using 2 different methods
comparison data represents panel data cohort assignments if browsing histo
ries were not related to race
tests comparison data
for t-closeness analysis compare the true panel cohorts data and the compa
rison data that shuffles the simhash values from the true data

This replaces https://github.com/aberke/floc-analysis/pull/8

I think randomly assigning cohort IDs instead of shuffling browsing history does make sense and results in panels with the right sizes for our analysis
ratio (the 16.93) used to source the $k=32$ makes sense to me, I back this
Of course, weeks are not independent but we decided a long time ago to pretend they are :)

@dcalacci please merge when you think it's ready.


add initial inscriptions for the ordinal screenz collection

Adding the first 13 of 100 Screenz Ordinals for collection listing

adding two new collections. Note: @ordinalscreenz please make a new pr for each collection next time. thank you for your understanding.


ys = torch.cat((g_xs, Gs_joint), 1  this line of code give Cuda out of memory error .how to fix it ?

Do you have soled the problem yet? I meet the same problem.

You can decrease the batchsize in .sh


Code coverage

We will also be adding automated code coverage to help enforce testing practices.

Reached 94%


Instant View on Telegram Broken

fx and fixupx on Telegram doesn't show any media on Telegram unless you add d. or g. to the beginning. Instant View is broken.

I've determined it only is broken on links listed as sensitive media. Others seem fine.

At the moment, sensitive posts do work in Instant View for me. To help us narrow down the problem, could you please provide:

An example Tweet where the issue occurs
Information about which Telegram client you are using. For example, iOS, Android, Mac, Telegram Desktop, Unigram, etc.

Warning, the below image/link is NSFW and contains partial nudity.
https://fixupx.com/washlet_mskm/status/1738944540990722313?s=52&t=NSLOJXNT0GY7aWKwVzq9sA
This did not produce an instant view. Only way to produce a preview was with g.fixupx or g.fxtwitter

Thank you so much for this. I can reproduce the issue with that link.

Trying to debug it with Telegram's Instant View templates website and it is getting stuck on "Processing links...". That's bizarre, I didn't know that was something that could even happen.


Switching activates an unexpected tab.

I have two tab groups, and I press the hotkey to go to the right tab group.  When I do, the tab that was visible there is deactivated and another tab is activated.  When I switch back to the first tab group, the formerly active tab in the right becomes hidden, and some other tab comes to the front on the right side.

I don't have a clear picture of what is happening. Could you capture take a screencapture (video) and also indicate what keys / mapping you are pressing?

In the readme, I added a video showing expected behavior; I also added some written explanations of behavior. Please share a video if the plugin is not behaving as I've indicated it should.

I uninstalled it because it wasn’t usable.  The key doesn’t matter, but let’s say it was Ctrl-T.On the left I have document B.md active.  On the right group I have C.md is visible.  B is in focus.  I press Ctrl-T.  On the right, X.md becomes active and focussed.  I press Ctrl-Shif-T to go back.  On the right C.md becomes visible again..  B regains keyboard focus.Obviously it would be more reasonable if C stayed visible always through this process.On Mar 17, 2024, at 13:34, Josh @.***> wrote:
I don't have a clear picture of what is happening. Could you capture take a screencapture (video) and also indicate what keys / mapping you are pressing?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

Hmm it does sound weird! I have not have any trouble using this plugin and I've been using it for a few weeks. If you shared a video, then I could begin to troubleshoot it. Do you have tabs pinned in side windows?
In any case it should just be opening the file that was in focus when the command was pressed. I cannot replicate the issue you describe.
I'm on 1.5.8 and most recent is 1.5.9. What version are you on?

I tried again and I think I was misusing it, or it just happens to work this time.  There is the Move tab to other group command which I didn't realize was a thing, but might have been what I was accidentally using.  Now that I see it, that's very useful.  I've realigned keys and it all seems to work well currently.  I will report back if I see misbehavior again.
Alan @.***
On Sunday, March 17, 2024 at 07:57:51 p.m. EDT, Josh<EMAIL_ADDRESS>wrote:

Hmm it does sound weird! I have not have any trouble using this plugin and I've been using it for a few weeks. If you shared a video, then I could begin to troubleshoot it. Do you have tabs pinned in side windows?
In any case it should just be opening the file that was in focus when the command was pressed. I cannot replicate the issue you describe.
I'm on 1.5.8 and most recent is 1.5.9. What version are you on?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Wonderful! I am glad it's useful. Please reopen if you observe any misbehaviors


setup a "land" bot so we can use ghstack

We could iterate faster with smaller PR's if we could "stack" them using ghstack. To enable this, while still having branch protection rules, we'd need to add some kind of "land bot" as described here:
https://github.com/ezyang/ghstack/issues/50#issuecomment-1455712692

WIP

I'd be happy to work on this - I can setup the workflows files.
Currently I don't have the privileges to configure the repo, invite a new bot account, setup the PAT secret.
I would suggest to use a genetic name for the bot account as we might want to use the same bot for further automation in future.

I created the @fms-cibot, which I will setup to run /land commands on PR via comment

this was added in https://github.com/foundation-model-stack/foundation-model-stack/pull/112


[RoadMap][Year 2022 Q4] Community RoadMap

Description

Notes of the community roadmap discussions from the Oct. 12th, 2022 bi-weekly community conference.
For some topics which need further discussions, we temporarily note the owner as Community.

RoadMap

Type
Description
Link
Priority
Owner

Feature
Add jRaft registry plugin
#11144
High
@zhuxt2015

Feature
K8S Executor
#13316
High
@EricGao888

Feature
Configuration Center
#10283
High
Community

Feature
Automated API Test
---
High
wen

Feature
Remote Logging
#13017
High
@rickchengx

Feature
Remote Task
#11652
High
@DarkAssassinator

Feature
Dynamic Task Plugin
---
Middle
@Amy0104

Feature
Independent Release of Task Plugin
---
Middle
@caishunfeng

Feature
Support Streaming Task
#11352
Middle
@caishunfeng

Feature
Task Sharing Storage
---
Middle
@jieguangzhou

Improvement
UT Improvement
#10573
High
@EricGao888

Improvement
pydolphinscheduler task plugin enrichment
#11793
High
@jieguangzhou

Improvement
Task Parameters Sharing Optimization
---
Middle
@zhongjiajie

Improvement
K8S Operator
---
Middle
@kezhenxu94

Improvement
Workflow Failure Strategy Optimization
#11793
Middle
@EricGao888

Related issues

WIP

Code of Conduct

[X] I agree to follow this project's Code of Conduct

hi @EricGao888, could we add https://github.com/apache/dolphinscheduler/issues/11652 in 2022 Q4 RoadMap?

hi @EricGao888, could we add #11652 in 2022 Q4 RoadMap?

done

if you'd like to implement some features for DolphinScheduler, please feel free to leave a message.

Closed by #13459
For WIP tasks in 2022 Q4 roadmap, we will keep a new record in #13459. For tasks which haven't received updates for long, we will not record them in the new roadmap. Thanks.


Add healthcheck to featureflagservice

Changes
In order to assess the health of this central service, compose should perform a check on the service.
Merge Requirements
For new features contributions please make sure you have completed the following
essential items:

[x] CHANGELOG.md updated to document new feature additions

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

I believe we have decided on monitoring the health using Kubernetes and on not supporting doing through docker like your proposal.

Why is that? If a certain service fails in the docker compose scenario, how do you know (other than fallout)?
Does that mean the primary intention of this project is to be run on k8s? Then we should probably remove the docker compose file.

@julianocosta89 thanks for raising awareness!

@puckpuck should we do something regarding helm in this one?
Otherwise, I think we are good to merge this one


Suggest changing the identifier of a model

We currently store the IPFS hash of our model split into two parts.
PySonar currently uses it as such:
deploy_trans = self.get_transaction(model.owner,value=self.web3.toWei(model.bounty,'ether')).addModel([ipfs_address[0:32],ipfs_address[32:]],model.initial_error,model.target_error)
return self.call.getNumModels()-1

self.call.getNumModels()-1

leads to race conditions.
e.g. Bob and Alice add a Model at the same time, both their local nodes will return them the same id. but the definite order is decided by how the miner orders the transactions in the block. In the end either the Bob or the Alice model will get the index they got from self.call.getNumModels()-1
Suggestion:
We keccak/sha3 the ipfs hash and store it in a single bytes32 instead of an array of two bytes32, this makes it easier to work with the ids and the calle knows the id of his model before he persists it with the contract.

Great, another step towards an actual federated prototype :)
I like the idea of reusing an existing property as unique ID. There two questions that come to mind:

Are IPFS addresses unique enough or should we add sth else to make sure we can deal with identical models?
Storing the ipfs address is still relevant and the hash would be an additional property right?

Are IPFS addresses unique enough [...]

Yes, in case there will ever be collision I guess ipfs will switch their hashing algorithm

[...] make sure we can deal with identical models?

Will we support users adding identical models?

Storing the ipfs address is still relevant and the hash would be an additional property right?

Depends, where would I promote my model? There i would probably provide the IPFS hash which can then be used to calculate its ID in the contract. Meaning the IPFS hash would not have to be stored on-chain. Only ethereumHash(IPFSHash(myModel)).


[entropy] reduce reseed interval in cryptolib

This reduces the reseed interval for EDN0 and 1 which was causing OTBN program execution to appear to hang in the ft_personalize binary during an attestation key generation operation.
Specifically, when working on #22714, I noticed that OTBN execution would hang during the UDS attestation (ECC P256) key generation. It would not error out, rather it would just hang, causing //sw/device/silicon_creator/manuf/skus/earlgrey_a0/sival_bringup:ft_provision_fpga_cw310_sival to timeout eventually. (After attaching a JTAG/OpenOCD/GDB setup, I was able to determine the location of the hang.)
A few things we noticed were:

moving the loading of the OTBN Boot Services program to before the entropy complex was reconfigured for keymgr operations inside the ft_personalize binary made the problem go away, and
making the modification made in this PR also made the problem go away.

As result, the thought is that entropy requests are getting dropped somewhere in the pipeline of FIFOs between entropy_src --> CSRNG --> EDN. This should probably be investigated further via waveforms. If we increase the reseed intervals (opposite of what is done in this PR) we may be able to trigger some DV testcases to fail. Alternatively, I could try to create a failing DV testcase based on parts of the ft_personalize test mentioned above.
CC: @vogelpi

CC: @h-filali this may fix the issue you saw in #22114. Could you try applying this on top to see if it fixes it?

Thanks for your investigations @timothytrippel and for reporting your findings!
Dropping entropy requests really shouldn't be happening. We have quite some block-level DV in place now to properly verify all interfaces between the blocks. Based on your findings and after having a look at the software code I believe what's really happening is that the entropy complex is being reconfigured (using entropy_complex_init()) without being shutdown first: The entropy_complex_stop_all() defined in the cryptolib is called nowhere in the code. I have a strong suspicion that ROM_EXT configures the entropy complex in auto mode and during the test it's reconfigured without a proper shutdown first. This is when requests of the EDNs get into CSRNG and are left hanging once CSRNG is reconfigured. Changing the reseed interval changes the frequency of requests but doesn't solve the problem properly - we might just be lucky as the reconfiguration now happens between the issuing of requests between the EDNs and CSRNG.
Later today, I will try to push a fix into @h-filali 's PR and see if it really works. I'll report back here.

Oh, no! I just noticed that we indeed shutdown the entropy complex before re-configuring it:
status_t entropy_complex_init(void) {
  entropy_complex_stop_all();

  const entropy_complex_config_t *config =
      &kEntropyComplexConfigs[kEntropyComplexConfigIdContinuous];
  if (launder32(config->id) != kEntropyComplexConfigIdContinuous) {
    return OTCRYPTO_RECOV_ERR;
  }

  HARDENED_TRY(entropy_src_configure(&config->entropy_src));
  csrng_configure();
  HARDENED_TRY(edn_configure(&config->edn0));
  return edn_configure(&config->edn1);
}

One thing to clarify @vogelpi is the test I was observing this behavior on (//sw/device/silicon_creator/manuf/skus/earlgrey_a0/sival_bringup:ft_provision_fpga_cw310_sival) does not have a ROM_EXT in place. It runs at the ROM_EXT stage itself (using the OTTF). So the entropy complex configuration that is in place at the start of the test should be that of what the ROM sets up (i.e. in boot mode).
Did this change help with #22114?

Thanks a lot @timothytrippel for this. @h-filali checked #22114 with this and it works. I am merging the PR now and will create an issue to debug the underlying issue before M4.

Note: I've merged this PR despite the following CI failures:

CW310 SiVal non-ROM_EXT Tests: i2c_target_test_fpga_cw310_sival failed, PR #22729 marks this as broken.
EDN smoke test failed: this got fixed by https://github.com/lowRISC/opentitan/pull/22736


connect to 'postgres' instead of a database whose name is admin's name

solve the problem #2363
When an admin user connects to a database to create a database for wallets, the database is not given. Then, the database name is the same as the admin user; see https://www.postgresql.org/docs/current/libpq-connect.html 3.11.2. It is required to create a database whose name is the same as the admin user.
By using the database 'postgres' instead of admin user name's database, we can create a database for wallets without creating the database whose name is the same as the admin user.

(ci) test this please

@toshirin33  can you check the conflict?  (Probably caused when I merged your other PR)

@ianco I am sorry that the #2366 contains the following line. I noticed that there is the variable _POSTGRES_DB before issuing this pull request and rewrote, but forgot to drop it in #2368 .
https://github.com/hyperledger/indy-sdk/blob/dbd89cf94a73e7a62611c4150a874c38b810ff8d/experimental/plugins/postgres_storage/src/postgres_storage.rs#L1048


Bundle PMML Evaluator Library with PMML File Resource into a single jar file

This jar can be imported into larger application with custom business logic based on the prediction - or combination of predictions (wide and deep ensemble)

defer for now


Is it possible to release the camera_preview.frag?

Hi, Michael.
I was wondering if you can release the camera_preview.frag in the android demo part. I am trying to build a real-time hdrnet demo. It would be great helpful that you could share the code. Thank you!

Hi, I unfortunately cannot release this piece of the code. But the algorithm is this one:
https://github.com/mgharbi/hdrnet_legacy/blob/b06d0119e2fb22c62c757161e6d351a304720544/hdrnet/ops/bilateral_slice.cu.cc#L224


3Legged OAuth feature using the RC-2.0.0 Pre Release candidate to cache the tokens

Changes included are :

authUrl method of the platform object lets you create the URL needed for the redirect
parseAuthRedirectUrl method of the platform to parse the redirectUri for the Authorization Code

Once authorized using the platform->login() , the platform object stores the tokens in the auth object.
Note : We are using the 2.0.0-rc1 version of ringcentral-php SDK at the moment.

Closing this as Duplicate


[Review] Enhanced logging and more tests

Add test and method to logging.

@nilsen340 please review :)

@onurakpolat could I get some more context of this?

This adds the HTTP method to the logging


Add delete in basic lsh.

I needed this so I thought someone else might. I don't know if this is correct though? Let me know.

It looks right to me.


feat: add australia region support

This PR adds Australia region support. It's effectively a copy paste of the China + EU regions, with the stamps updated for the Australia CFB and everything else irrelevant removed.
I've tested most of it but can't test all of it myself, so I think this is good enough to merge now and we can merge fixes in as we I test it more day-to-day and get some other people testing as well :)

There are a couple things that I noticed that I'd like to clean up, but I'll merge them in as separate PRs later 👍

Amazing work! Thanks for doing this. A few quick comments.

I still get "Kia is not supported in Australia yet" when I try - has AU support only been added for Hyundai?


Data Viewer Not Working in Debug Mode

Applies To

[ ] Notebooks (.ipynb files)
[X] Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?
There is no longer a "View Value in Data Viewer" when I am debugging a .py file.

VS Code Version
1.86.0
Jupyter Extension Version
2024.1.0
Jupyter logs
No response
Coding Language and Runtime Version
Python 3.12.1
Language Extension Version (if applicable)
Python Extension 2024.0.0
Anaconda Version (if applicable)
No response
Running Jupyter locally or remotely?
Local

I've faced the same issue, I've done a clean install of VS Code, tried previous versions of Python, Python debugger and Jupyter; however, the problem still persists. This has only become an issue last week (Friday).

Same issue here.

Same here. I've twice uninstalled and reinstalled python, vs code, the jupyter extension, and still no change...

same issue here. can't get the data viewer to show up as an option. i've tried uninstall/reinstall, different configurations, etc.

same issue here. can't get the data viewer to show up as an option. i've tried uninstall/reinstall, different configurations, etc. no data viewer here either

Potentially related: #15077, #15128
Potentially related on Stack Overflow:

View value in Data Viewer stopped showing - view Dataframes
Unable to See "View Value in Data Viewer" Option in VS Code Debugger
"View in Data Viewer" in VS Code not available (without jupyter)

This is not related to this specific version, but for those who just need this to work right now, I managed to get it to work by rolling back versions.

Uninstall Python debugger Extension
Roll back python extension (I rolled back to v2023.18.0)
Roll back jupyter (I did to v2023.9.1102792234)
I dont think it really matters specifically which version you roll back to, but I just rolled back 4 months just to be safe.

thanks for the investigation @askar-fabricnano, that sounds right.
Rolling back to last month's Python release and disabling the python debug extension seems to be enough to workaround the issue while we figure out what to do here.

the fix for this is currently in the pre-release of the Jupyter extension, and we are pushing out a release patch with the fix this week

release v2024.1.1 has been published with the fix


Updates to Readme file

Right now we've 10+ contributors, but in the contributors section in readme file, it is only showing 9 contributors.
We need to correct this, like whenever there is a new contributor the list is updated automatically.

hi @ShehrozIrfan please assign this to me

hi @ShehrozIrfan I would like to contribute to this

@navaneethkp36
Assigned.

hi @ShehrozIrfan Seems like this is already added using contrib.rocks. Can you please check and confirm.

Yes @navaneethkp36 it is already added.
Thanks.


MSIX Installs to different locations on each new version

The install path for the MSIX version from the private preview from the Microsoft store changes between versions, making subsequent use in VSCode & Win Terminal, which require the paths be hardcoded in settings files, another manual user update task.
This ideally should keep the same path on update - however this may a feature of the store based installs

As a general note, Terminal doesn't require that paths be hardcoded. It will use %PATH% for spawning shells, and will launch the store version of powershell if its execution alias shows up earlier in %PATH% than any other version.
There is also a version-agnostic alias at %LOCALAPPDATA%\Microsoft\WindowsApps\pwsh.exe and a preview/non-preview-agnostic alias at %LOCALAPPDATA%\Microsoft\WindowsApps\Microsoft.PowerShellPreview_8wekyb3d8bbwe\pwsh.exe.

The above provides a suitable and easily workable work around


Fix style

Changes Unknown when pulling d5601d5f52523a508f7e8561ef71a22c30043c33 on kashefy:fix_style into ** on nigroup:master**.

Coverage increased (+0.01%) to 96.0% when pulling d5601d5f52523a508f7e8561ef71a22c30043c33 on kashefy:fix_style into df995e476bcdef1a35ea475b28d050c6065b2da7 on nigroup:master.

Coverage increased (+0.01%) to 96.0% when pulling f59cb52decf909ec25759f6aef96725816e8a551 on kashefy:fix_style into df995e476bcdef1a35ea475b28d050c6065b2da7 on nigroup:master.


Support general pruning based on <col> = 'const' in PruningPredicate

Is your feature request related to a problem or challenge?
In IOx in certain cases we know that a "container" (parquet file, or set of record batches) has only a single value for some computed quantity (in our case hash(column) % N for some constant N (like 100).
For this case, we want to be able to quickly determine, given an arbitrary predicate, if that container could not possible contain the value.
So for example, if we know the container has hash(column) % 100 = 27, given a predicate that includes an expression like column = 'foo', we can compute the quantity hash(column) % 100 and if it is not 27 we can skip the entire container.
We could implement this directly in our codebase, and we may do so temporarily. However, I think the usecase is common enough that we would like to improve the support upstream in DataFusion so others can both benefit and help optimize for it.
For example, applying BloomFilters to prune out parquet row groups (added in https://github.com/apache/arrow-datafusion/pull/7821 by @hengfeiyang) has the same pattern.
Since we also have other information such as min/max and null counts for certain columns that we prune using PruningPredicate, having this ability be part of PruningPredicate is compelling
Describe the solution you'd like
DataFusion's PruningPredicate  can already use information on ranges (min/max values). I would like to extend its capabilities to incorporate knowledge about certain specific values to take advantage of column = <constant> predicates.
I propose we extend PruningStatistics with some way to pass knowledge on about the contents of data structures like Bloom Filters. For example:
pub trait PruningStatistics {
    // PROPOSED ADDITIONS

    // Returns an array where each element is
    // * `true` if the value of column CERTAINLY DOES contain `value`
    // * `false` if the value of column CERTAINLY DOES NOT contain `value`
    // * `null` if the value of column may or may not contain the value
    fn contains(&self, column: &Column, value: &ScalarValue) -> Result<Option<BooleanArray>>;

    // EXISTING METHODS
    fn min_values(&self, column: &Column) -> Option<ArrayRef>;
    fn max_values(&self, column: &Column) -> Option<ArrayRef>;
    fn num_containers(&self) -> usize;
    fn null_counts(&self, column: &Column) -> Option<ArrayRef>;
}

We could then implement the bloom filter pruning in DataFusion with this API as well as use the same thing for our downstream usecase
Example for equality predicate col = 'foo'
The PruningPredicate would call
PruningStatistics::contains(Column(col), 'foo')

and could prune all containers that returned false
Example for inequality predicate col != 'foo'
The PruningPredicate would call
PruningStatistics::contains(Column(col), 'foo')

and could prune all containers that returned true
Note I don't think the contains API could be used for other inequality predicates like col < 'foo' for example. The existing min/max statistics would have to be used
Describe alternatives you've considered
I also thought about trying to rewrite equality predicates to take advantage of the existing min/max statistics (which can represent where a column has only a single value). However, that API doesn't allow for information like Bloom filters which simply can say for sure if the value may be present  or not.
Additional context
Here is the code that does bloom filtering: https://github.com/apache/arrow-datafusion/blob/2a692446f46ef96f48eb9ba19231e9576be9ff5a/datafusion/core/src/datasource/physical_plan/parquet/row_groups.rs#L203-L252

@waynexia  / @haohuaijin  I wonder if you have time to review the PRs that are part of this issue (would make parquet statistics pruning more effective, among other things):

https://github.com/apache/arrow-datafusion/pull/8437
https://github.com/apache/arrow-datafusion/pull/8440
https://github.com/apache/arrow-datafusion/pull/8397

cc @Dandandan and @thinkharderdev  as I think you use this API as well and perhaps may have an interest as well

Technically speaking this was closed with https://github.com/apache/arrow-datafusion/pull/8440
https://github.com/apache/arrow-datafusion/pull/8442 is a cleanup but I am going to close this ticket as done!